merge 3.25 heads

[hg] Add generated slapd.conf to hgignore

Fix flake8 errors in unittest_cwctl

[cwctl] Drop 'cubicweb_' prefix when displaying cubes with 'cubicweb-ctl list' consistently with newcube / create commands. This is an alternate implementation to backed out changeset fe995d56c949. Closes #17054738

Fix flake8 errors in unittest_cwconfig

[config] Test loading of ccplugin modules to ensure ccplugin modules of cube as package aren't loaded twice, once as `cubicweb_<cube>.ccplugin`, once as `cubes.<cube>.ccplugin`. This test used to fail until fe995d56c949 has been backout. It requires calling `cleanup_sys_module` in `CubicWebConfigurationTC.tearDown` to remove any cube's submodule loaded during tests, so content of `sys.modules` may be tested.

[test] Extract out method-which-is-not-a-method mocking iter_entry_points to ease reuse and improve readability.

[cwconfig] Add a docstring on available_cubes method a renaming could be better at some point, but that's a start.

Backed out changeset fe995d56c949 available_cubes should not strip cubicweb_prefix but return actual package name for cube as package, since its output is also used to e.g. get ccplugin or site_cubicweb module name. Original aim of this set was to fix output of the "cubicweb-ctl list" command. This will be done by an alternate implementation in a later cset.

[hooks] Drop "logstats" hook It is now useless as its looping task would not run on a web instance because respective repository has no scheduler.

[hooks] Do not register "logstats" if repository has no scheduler

[server] Introduce an `has_scheduler` method on Repository This is to be used by client application to determine if looping tasks may be registered in the current process. By checking this, one will avoid the warning in looping_task method when the repository has no scheduler.

[server] Warn instead of failing when a looping task is registered and repo has no scheduler We should provide a way for client code to detect if they should register the looping task or not. See next patch for that.

[server] Exit quickly when a looping task is registered in maintenance mode In such cases the repository will not have a scheduler on purpose because the repository will not be kept running and will quickly shutdown after migration so that it's undesirable to have looping tasks being executed.

[config] Drop unused init_available_cubes method

[doc] Move content of former README.pyramid into relevant modules

Drop README.pyramid.rst file It has been kept since 3.24 when we merged pyramid-cubicweb and cubicweb-pyramid repositories into cubicweb's one waiting for someone to move the information in some canonical place but this never happened and this information is probably not so useful or already existing elsewhere. So just drop the file.

Make entypo link a true URL in README

Refer to cubicweb.readthedocs.io in README docs.cubicweb.org contains outdated documentation at the moment (and for quite some time); until it gets update, refer to the up-to-date documentation on readthedocs.

[server] don't catch exception from postcommit_event() if we are in test mode The historic behavior is to hide potential exception occurring in postcommit_event(). Unfortunately logging statements are hidden by default during tests and it become very hard to debug. At least raise if we are in test mode.

[pkg] Version 3.25.0rc1

[tox] Simplify command line to build the doc

[pyramid] Drop warning about pyramid_debugtoolbar not found in debug mode Some people way want to run a pyramid server in debug mode without the toolbar. This warning is annoying for them. Others should just install and *include* the toolbar by hand.

[pkg] Add development.ini.tmpl to "cubicweb.pyramid" package data Otherwise the file is not installed and `cubicweb-ctl create -c pyramid` does not work.

[skeleton] Don't use system site packages in tox environment As we changed this in cubicweb, there is a consensus that this is not the best practice so let us change it as well for skeleton.

[doc] Update "settings" pyramid documentation Document that without "bwcompat" we can use a development.ini file directly and add a dedicated section to list configuration options.

[doc] Update the "quickstart" page of pyramid documentation Again mentioning the two modes of operations and dropping references to prior implementation.

[doc] Rewrite "pyramid" index page in the book We drop references to pyramid-cubicweb and cubicweb-pyramid, remaining of the merge in 3.24. We now mention the two possible operation modes (with bwcompat being true or false).

[doc] Drop documentation for cubicweb.pyramid.tools module Which, as documented, mostly consists of internals.

[doc] Drop reference to "make_cubicweb_application" in pyramid doc This function got renamed into config_from_cwconfig and this is arguably an internal detail so should not appear in the API documentation.

[doc] Remove trailing whitespaces in doc/book/pyramid/index.rst

[tox] Install "pyramid" dependencies in "doc" environment Otherwise most of the cubicweb.pyramid API doc fails to build. For this we skip installation in the "doc" environment and source the new requirements/doc.txt which will install cubicweb[pyramid], thus pulling cubicweb plus pyramid's dependencies. Furthermore this requirements/doc.txt would be useful to build to documentation on readthedocs.

[web/views] fix bad html in deleteconf with show_composite=True On an entity with no composite an additional item "None" was displayed.

[web/views] display unique composite entities in deleteconf When the entity has two relations to the same composite, display it only once.

Really make rqlrewrite flake8 compliant

[pyramid] disable bwcompat by default in PyramidCWTest To avoid cubicweb exceptions handling.

Fix flake8 errors in rqlrewrite.py

[rqlrewrite/schema] Avoid parsing computed relations for each query The RQLRelationRewriter is instanciated for each RQL query, it should avoid parsing computed relations formula by using a cache, which seems rightly located on the instance's schema. This brings a *huge* performance boost to some pages on application with a few computed relations (x4 observed on a client app). Kudos to Adrien, David and Sylvain. Closes #17059828

[pkg] Restrict yams version to be less than 0.45.0 This is a backport of 10d3bf220be2 on 3.23 branch

[pyramid] Remove pyramid_debugtoolbar from "pyramid.includes" Most of the times, the package is not installed (not pulled by cubicweb[pyramid] nor pyramid itself) in a fresh environment so that first start of a new instance will fail.

[doc] Add some details about rtag derivation and server api cleanup in release notes

[doc] Simplify release notes section Those are all about backward incompatibilities, keep only one section about that.

[doc] Fill release notes for 3.25

[devtools] Create "newcube" in current directory by default Instead of using "cubes path", which in many cases will be something wrong (like <prefix>/share/cubicweb/cubes) now that cubes are regular Python packages.

[pyramid] Do not register repository.shutdown atexit in test mode It seems that there already is a special shutdown step for repository in test mode and pyramid's atexit handler interferes with it leading to assertion error about 'shutting_down' attribute being True in Repository.shutdown().

[cwconfig] don't rely on _CubeImporter while loading cubes _CubeImporter can load cube subpackages twice (there is a test for submodules in cubicweb.test.unittest_cubes.test_no_double_import() but not with a subpackage). Since _CubeImporter is for backward compatibility and is supposed to be deprecated as soon as possible, don't rely on it when loading cube schema or appobjects and always load the real modname (eg. cubicweb_<cube> if exists else cubes.<cube>).

Added tag 3.24.7, debian/3.24.7-1, centos/3.24.7-1 for changeset 58f1e8545a77

[pkg] Version 3.24.7

[pkg] Restrict yams version to be less than 0.45.0 Because of reader's API changes x and other subtle but hard to debug changes in appobjects registration in version 0.45.0. Closes #17065626.

[pkg] Version 3.25.0rc0 Let's make a release candidate version of CubicWeb 3.25.0 (only for the Python package), in the hope that some issues that we usually discover after release will be sorted out before.

[pyramid] Rename paste app factory to pyramid_main Making it clearer that this would build a Pyramid application, just in case we eventually provide other means to create a WSGI application than Pyramid.

[tox] Do not use site packages in testenv This is the normal way of running tox. We used to set this because some people liked to run tests using packages installed by other means (i.e. system-wide). Now, there seems to be a consensus that running tox without site packages is a better idea, in particular since many of our dependencies are now available as wheels and thus would install quickly provided one uses a recent pip/virtualenv.

[test] Add user-site to PYTHONPATH before installation of new cube in test_newcube_install Otherwise, setuptools complains that the target installation directory (i.e. "user site" is not in PYTHONPATH). This is probably because we use a fake HOME with --user install option. Failure shows up when running tests through tox without system site packages (see next patch).

[pkg] Depends on yams >= 0.45.0

[pkg] Depends on logilab-common >= 1.4.0

Merge with 3.24 branch

Added tag 3.24.6, debian/3.24.6-1, centos/3.24.6-1 for changeset 4029fb83a67d

[pkg] Version 3.24.6

Merge public heads

[web/views] deleteconf: ignore workflow when displaying composite entities This hide TrInfo composite entities when deleting an workflowable entity.

[rset] Use RQL cache in syntax_tree method This avoid several unnecessary parsings per HTTP request, since in most case rset's rql is in the rql cache (since it has just been executed), while yapps based parsing is known to be slow.

[repo] Introduce a clear_caches method on the Querier class that is called from repository's clear_caches.

[repo] Consistent API for cache clearing clear_caches, reset_caches, clear_eid_caches are now all named 'clear_caches' with optional eids/etypes arguments to clear eid specific cache entry,

[repo] Extract rql cache handling to a dedicated class

[sobjects] Fix a trivial flake8 error

[repo] Drop cache clearing methods from the source interface Those are actually system source specific.

[repo] Move and rename repo._clear_planning_cache to make it clearer it's clearing the @cache of source_defs method.

[sobjects] Fix flake8 errors in services.py

[test] Rename BaseQuerierTC._access to BaseQuerierTC.admin_access so it's consistent with CubicWebTC and avoid access to a protected attribute which has been exposed by the removal of the 'session' property a few csets earlier.

[repo] Fix flake8 error

[pyramid] Add a "pyramid" instance configuration type In a new module 'cubicweb.pyramid.config' we define a "pyramid" instance configuration type. The noticeable feature of this configuration is that it manages a 'development.ini' file that gets installed in application home (along with `.conf` file). This file is templated and includes generated values for secrets of session and authtk tokens. This means that we can just call: pserve etc/cubicweb.d/<appname>/development.ini or gunicorn --paste etc/cubicweb.d/<appname>/development.ini -b :8080 just after instance creation to get a pyramid instance running without having to hack around a 'pyramid.ini' file. This patch drops 'development.ini' from skeleton and moves it in cubicweb/pyramid so that it gets installed at instance creation which is more appropriate than in cube creation. The new configuration class sets "cubicweb.bwcompat" setting to false so it is not intended to replace the "all-in-one" configuration type (which would require a bit more work). This configuration is close to the the 'repository' configuration type with just a couple of options from WebConfiguration that are needed for Pyramid (anonymous user/password plus some miscellaneous options that I'm not so sure are really needed). Note, in particular, that we do not pull CORS settings to be injected as a WSGI middleware like in wsgi_application_from_cwconfig() since I believe this should be left as an end-user responsibility and since this can be defined in a standard way in paste configuration. This configuration inherits from ServerConfiguration but registers the same appobjects as WebConfiguration. In cubicweb.web.request._CubicWebRequestBase, we guard against access to "uiprops" and "datadir_url" of the config because this new "pyramid" config does not have these (this does not make sense without bwcompat mode). At some point, we should either avoid using `cw_request`'s pyramid request attribute or make cubicweb's web request really independant of existing implementation and drop these assumptions.

[pyramid] Call repository's shutdown method atexit This way when run through a WSGI server (pserve, gunicorn, etc.) we make sure that the repository is properly shut down. In particular, this would wait for threads to terminate (otherwise, they'll be killed).

[querier] Turn repo.querier_cache_key into a private function of the querier module, only place where it's used.

[cleanup] Fix flake8 errors in web/application.py and associated test

[test] Simplify CWTC's init_authentication and assertAuthSuccess which shouldn't rely on removed .session attribute while it's not necessary and could be written in a simpler way.

[cleanup] Drop non sense sentence in docstring

[repo] Drop Session at last This is not used at all anymore.

Fix flake8 some errors

[test] Use existing reference to repo_source

[repo] Drop repo.new_session method we should not go through Session to create Connection anymore. This is unofficial API, no backward compat for now.

Stop using Session on the repository side Only expect session on web request, and let the web session/authentication managers provide them. Access to cnx.data, which used to return session data, is deprecated: there is no more access to session data from the repository side, and they should be access from req.session.data from the web side.

[test] Drop unnecessary call to new_session This is not unittest, the above call is enough.

[user] Drop ugly hack on CWUser entity class which should be not necessary anymore, beside two remaining usage in our own test suite.

[test] Use user_session_cache_key function to generate cache key.

[session+test] Stop storing / accessing session when it's not necessary

[devtools] Stop relying on Session to create connection in RepoAccess

[devtools] Stop using repo.new_session

[session] Kill support for a 'session_open' hook event To be consistent with dropping of the 'session_close' event, and also because this is already not called when using pyramid.

[devtools] Rename FakeSession into FakeConnection because it is what it is.

[cleanup] Drop references to the old internal_session method dropped a while ago, no need to talk about it in a docstring nor to fake it.

[session] Drop unused class

[cleanup] Drop no more used parse_repo_uri function

[server] Fix flake8 error of test/data/hooks.py

[repo] Drop repo.connect backward compat use new_session instead (introduced in 3.19 iirc)

[session] Drop no more necessary Session.sessionid and Connection.connectionid

[repo] Kill repo._sessions and with it the need to have a looping task to clean it up, and with that all the session's timestamping machinery.

[repo] Stop closing session The only point in "closing" session is to call the `session_close` event. Since I'm not aware of any application relying on it, I think this is too costly (in term of code) to maintain and propose to drop it.

[views] Stop giving stats / debug information relying on _sessions This is already broken when using pyramid anyway.

[session] Stop relying on _sessions to synchronize living session by doing what we currently do in pyramid. Also, attempt to synchronize CWUser's class sounds like a dark corner case handling that should be removed in favor of short lived user objects (beside it shouldn't occur except during migration).

[test] Drop useless access to repo._sessions test still succeed afterwards.

[repo] Fix flake8 errors in server/__init__.py

[repo] Drop no more used ShuttingDown exception

[repo] Drop _get_session method it's not actually necessary but holds underlying _sessions dict handling. Drop backward compat relying on it (which IMO should not be a problem). Drop import of unused QueryError along the way.

[test] Drop call to CubicWebConfiguration.load_cwctl_plugins in unittest_cwctl.py This is no longer need now that we call load_available_configurations() in the "list" command (see 5e7282bdf140).

[web] Extract a base class out WebConfiguration This will be used to build a CubicWebPyramidConfiguration in following changeset.

[tox] Drop reference to removed init_instance.py file in flake8-ok-files.txt The file got removed in 1b9fd8bda1cd.

[web] Put 'use-uicache' option in level 2 So that users do not get prompted with this. Follow-up on c34590161082.

[server] Add a "scheduler" command to run repository scheduler This commands starts the repository scheduler as a standalone process that should complement a CubicWeb web instance running as a WSGI application. Added a log message in repository's shutdown method to help testing the command (i.e. make sure the method is called after the scheduler stopped). Related to #17057223.

[cwconfig] Do not instantiate a repo with a scheduler in repository method None of the caller of this method need the repository to have a scheduler and having a scheduler should be something that's explicitly requested.

[server] Make cubicweb/server/utils.py flake8-clean

[server] Drop utils's LoopTask and TasksManager classes not used anymore Related to 17057223.

[server] replace repository's tasks manager by a scheduler Repository() does not accept anymore a 'tasks_manager' argument but rather a 'scheduler' argument which is expected to be an instance of sched.scheduler class. The drop the _tasks_manager attribute of the repository and adjust all internal usages of it. In particular, in the 'repo_stats' service we do not export 'looping_tasks' statistics anymore as there's no way to retrieve this anymore from a web instance. Closes #17057223.

[server] introduce a scheduler class to run repository "looping tasks" We just use the sched module from the standard library and introduce a tiny Python2/3 compatibility layer (more for convenience actually). The "looping" aspect of tasks (previously in LoopTask class) is re-implemeted as a `schedule_periodic_task` function. This is a reasonably thin layer as compared to LoopTask/TasksManager classes. Only the "restart" aspect of LoopTask is no longer present as I'm not sure it's worth keeping. The advantage of using this (in addition to eventually dropping our custom code) is that this scheduler class provides a `run` method that blocks the process while running tasks in its queue. So we can rely on this to have a 'scheduler' ctl command (see forthcoming patch) that would only run "looping tasks" without having to implement the "blocking" aspect ourself. Related to #17057223.

web: add options to ignore css compilation and uicache

[skeleton] call includeme from new cube

[etwist] Do not call repository's start_looping_tasks anymore and warn about this We are about to drop this method from Repository class and replace it by a blocking alternative. This is not compatible with how things currently work in a Twisted server implementation. So do not start repository "looping tasks" in Twisted server anymore and issue a warning about this. If someone is interested in restoring the "all-in-one" behavior where the repository runs within a Twisted server, they may start by implementing repository looping tasks using a Twisted mechanism such as, e.g., http://twistedmatrix.com/documents/current/core/howto/time.html and eventually provide the repository with a compatible scheduler instance so that is can register its periodic tasks. At the moment, we lack resources to do this (and maintain the Twisted server of CubicWeb in general). Related to #17057223.

[test] Flake8-clean and use stdlib unittest in cubicweb/server/test/unittest_utils.py

[web] Don't raise ProcessFormError if there is no value specified a03376213747 introduced a slight modification in None / empty string values and with it a regression where an exception is raised on empty values, while we should simply consider no value is specified and return None.

[test] Cleanup imports in unittest_form Remove unused and reorder, + use bare unittest along the way.

Fix log level parameter not taken into account in cwsource log table An error in the jquery selector of the html widget controlling the log level was the cause of this bug. Closes #15772634 (grafted from c9129aae884b476455c8ed1ad802efc3a26503a5)