Merge public heads

[web/views] deleteconf: ignore workflow when displaying composite entities This hide TrInfo composite entities when deleting an workflowable entity.

[rset] Use RQL cache in syntax_tree method This avoid several unnecessary parsings per HTTP request, since in most case rset's rql is in the rql cache (since it has just been executed), while yapps based parsing is known to be slow.

[repo] Introduce a clear_caches method on the Querier class that is called from repository's clear_caches.

[repo] Consistent API for cache clearing clear_caches, reset_caches, clear_eid_caches are now all named 'clear_caches' with optional eids/etypes arguments to clear eid specific cache entry,

[repo] Extract rql cache handling to a dedicated class

[sobjects] Fix a trivial flake8 error

[repo] Drop cache clearing methods from the source interface Those are actually system source specific.

[repo] Move and rename repo._clear_planning_cache to make it clearer it's clearing the @cache of source_defs method.

[sobjects] Fix flake8 errors in services.py

[test] Rename BaseQuerierTC._access to BaseQuerierTC.admin_access so it's consistent with CubicWebTC and avoid access to a protected attribute which has been exposed by the removal of the 'session' property a few csets earlier.

[repo] Fix flake8 error

[pyramid] Add a "pyramid" instance configuration type In a new module 'cubicweb.pyramid.config' we define a "pyramid" instance configuration type. The noticeable feature of this configuration is that it manages a 'development.ini' file that gets installed in application home (along with `.conf` file). This file is templated and includes generated values for secrets of session and authtk tokens. This means that we can just call: pserve etc/cubicweb.d/<appname>/development.ini or gunicorn --paste etc/cubicweb.d/<appname>/development.ini -b :8080 just after instance creation to get a pyramid instance running without having to hack around a 'pyramid.ini' file. This patch drops 'development.ini' from skeleton and moves it in cubicweb/pyramid so that it gets installed at instance creation which is more appropriate than in cube creation. The new configuration class sets "cubicweb.bwcompat" setting to false so it is not intended to replace the "all-in-one" configuration type (which would require a bit more work). This configuration is close to the the 'repository' configuration type with just a couple of options from WebConfiguration that are needed for Pyramid (anonymous user/password plus some miscellaneous options that I'm not so sure are really needed). Note, in particular, that we do not pull CORS settings to be injected as a WSGI middleware like in wsgi_application_from_cwconfig() since I believe this should be left as an end-user responsibility and since this can be defined in a standard way in paste configuration. This configuration inherits from ServerConfiguration but registers the same appobjects as WebConfiguration. In cubicweb.web.request._CubicWebRequestBase, we guard against access to "uiprops" and "datadir_url" of the config because this new "pyramid" config does not have these (this does not make sense without bwcompat mode). At some point, we should either avoid using `cw_request`'s pyramid request attribute or make cubicweb's web request really independant of existing implementation and drop these assumptions.

[pyramid] Call repository's shutdown method atexit This way when run through a WSGI server (pserve, gunicorn, etc.) we make sure that the repository is properly shut down. In particular, this would wait for threads to terminate (otherwise, they'll be killed).

[querier] Turn repo.querier_cache_key into a private function of the querier module, only place where it's used.

[cleanup] Fix flake8 errors in web/application.py and associated test

[test] Simplify CWTC's init_authentication and assertAuthSuccess which shouldn't rely on removed .session attribute while it's not necessary and could be written in a simpler way.

[cleanup] Drop non sense sentence in docstring

[repo] Drop Session at last This is not used at all anymore.

Fix flake8 some errors

[test] Use existing reference to repo_source

[repo] Drop repo.new_session method we should not go through Session to create Connection anymore. This is unofficial API, no backward compat for now.

Stop using Session on the repository side Only expect session on web request, and let the web session/authentication managers provide them. Access to cnx.data, which used to return session data, is deprecated: there is no more access to session data from the repository side, and they should be access from req.session.data from the web side.

[test] Drop unnecessary call to new_session This is not unittest, the above call is enough.

[user] Drop ugly hack on CWUser entity class which should be not necessary anymore, beside two remaining usage in our own test suite.

[test] Use user_session_cache_key function to generate cache key.

[session+test] Stop storing / accessing session when it's not necessary

[devtools] Stop relying on Session to create connection in RepoAccess

[devtools] Stop using repo.new_session

[session] Kill support for a 'session_open' hook event To be consistent with dropping of the 'session_close' event, and also because this is already not called when using pyramid.

[devtools] Rename FakeSession into FakeConnection because it is what it is.

[cleanup] Drop references to the old internal_session method dropped a while ago, no need to talk about it in a docstring nor to fake it.

[session] Drop unused class

[cleanup] Drop no more used parse_repo_uri function

[server] Fix flake8 error of test/data/hooks.py

[repo] Drop repo.connect backward compat use new_session instead (introduced in 3.19 iirc)

[session] Drop no more necessary Session.sessionid and Connection.connectionid

[repo] Kill repo._sessions and with it the need to have a looping task to clean it up, and with that all the session's timestamping machinery.

[repo] Stop closing session The only point in "closing" session is to call the `session_close` event. Since I'm not aware of any application relying on it, I think this is too costly (in term of code) to maintain and propose to drop it.

[views] Stop giving stats / debug information relying on _sessions This is already broken when using pyramid anyway.

[session] Stop relying on _sessions to synchronize living session by doing what we currently do in pyramid. Also, attempt to synchronize CWUser's class sounds like a dark corner case handling that should be removed in favor of short lived user objects (beside it shouldn't occur except during migration).

[test] Drop useless access to repo._sessions test still succeed afterwards.

[repo] Fix flake8 errors in server/__init__.py

[repo] Drop no more used ShuttingDown exception

[repo] Drop _get_session method it's not actually necessary but holds underlying _sessions dict handling. Drop backward compat relying on it (which IMO should not be a problem). Drop import of unused QueryError along the way.

[test] Drop call to CubicWebConfiguration.load_cwctl_plugins in unittest_cwctl.py This is no longer need now that we call load_available_configurations() in the "list" command (see 5e7282bdf140).

[web] Extract a base class out WebConfiguration This will be used to build a CubicWebPyramidConfiguration in following changeset.

[tox] Drop reference to removed init_instance.py file in flake8-ok-files.txt The file got removed in 1b9fd8bda1cd.

[web] Put 'use-uicache' option in level 2 So that users do not get prompted with this. Follow-up on c34590161082.

[server] Add a "scheduler" command to run repository scheduler This commands starts the repository scheduler as a standalone process that should complement a CubicWeb web instance running as a WSGI application. Added a log message in repository's shutdown method to help testing the command (i.e. make sure the method is called after the scheduler stopped). Related to #17057223.

[cwconfig] Do not instantiate a repo with a scheduler in repository method None of the caller of this method need the repository to have a scheduler and having a scheduler should be something that's explicitly requested.

[server] Make cubicweb/server/utils.py flake8-clean

[server] Drop utils's LoopTask and TasksManager classes not used anymore Related to 17057223.

[server] replace repository's tasks manager by a scheduler Repository() does not accept anymore a 'tasks_manager' argument but rather a 'scheduler' argument which is expected to be an instance of sched.scheduler class. The drop the _tasks_manager attribute of the repository and adjust all internal usages of it. In particular, in the 'repo_stats' service we do not export 'looping_tasks' statistics anymore as there's no way to retrieve this anymore from a web instance. Closes #17057223.

[server] introduce a scheduler class to run repository "looping tasks" We just use the sched module from the standard library and introduce a tiny Python2/3 compatibility layer (more for convenience actually). The "looping" aspect of tasks (previously in LoopTask class) is re-implemeted as a `schedule_periodic_task` function. This is a reasonably thin layer as compared to LoopTask/TasksManager classes. Only the "restart" aspect of LoopTask is no longer present as I'm not sure it's worth keeping. The advantage of using this (in addition to eventually dropping our custom code) is that this scheduler class provides a `run` method that blocks the process while running tasks in its queue. So we can rely on this to have a 'scheduler' ctl command (see forthcoming patch) that would only run "looping tasks" without having to implement the "blocking" aspect ourself. Related to #17057223.

web: add options to ignore css compilation and uicache

[skeleton] call includeme from new cube

[etwist] Do not call repository's start_looping_tasks anymore and warn about this We are about to drop this method from Repository class and replace it by a blocking alternative. This is not compatible with how things currently work in a Twisted server implementation. So do not start repository "looping tasks" in Twisted server anymore and issue a warning about this. If someone is interested in restoring the "all-in-one" behavior where the repository runs within a Twisted server, they may start by implementing repository looping tasks using a Twisted mechanism such as, e.g., http://twistedmatrix.com/documents/current/core/howto/time.html and eventually provide the repository with a compatible scheduler instance so that is can register its periodic tasks. At the moment, we lack resources to do this (and maintain the Twisted server of CubicWeb in general). Related to #17057223.

[test] Flake8-clean and use stdlib unittest in cubicweb/server/test/unittest_utils.py

[web] Don't raise ProcessFormError if there is no value specified a03376213747 introduced a slight modification in None / empty string values and with it a regression where an exception is raised on empty values, while we should simply consider no value is specified and return None.

[test] Cleanup imports in unittest_form Remove unused and reorder, + use bare unittest along the way.

Fix log level parameter not taken into account in cwsource log table An error in the jquery selector of the html widget controlling the log level was the cause of this bug. Closes #15772634 (grafted from c9129aae884b476455c8ed1ad802efc3a26503a5)

[pyramid] Drop call to repository's start_looping_tasks method and warn about this We are about to drop the start_looping_tasks method and running "looping tasks" along the WSGI application is not a very good idea. So issue a warning pointing to the forthcoming "scheduler" command. Related to #17057223.

[test] Make unittest_serverctl.py flake8-clean

[wsgi] Drop calls to repo.start_looping_tasks It's unlikely that these implementations were ever used so we do not bother with deprecation or a warning. We just drop the call to this method which should not be called from within a WSGI application. Related to #17057223.

[pyramid] Drop useless "init_instance" module

[server] Drop inexistent "bootstrap" argument in Repository initialization This is a missing bit from 5de78b6fff2e. Unfortunately this code appears to be uncovered by our test suite.

[server] fix a typo in a docstring

[ctl] Load available configurations in "cubicweb-ctl list" Otherwise we rely on associated ctl plugins to be loaded which is arguably orthogonal to configurations.

[web/views] Fix a docstring

[server] Use looping_task method in Repository._prepare_startup()

[skeleton] Drop logging configuration from development.ini This is actually not useful as logging is already initialized and configure "somewhere else" where the repository is started.

[pyramid] Drop retrieval of auth/session secret in all-in-one.conf We actually never supported this in cubicweb, but only in pyramid-cubicweb.

[cwctl] Recommend 'pyramid' before 'wsgi' when twisted is not available Not sure anyone ever really used the latter and the former is certainly more reliable/tested.

[cwctl] Drop message about 'the pyramid cube' in "start" command This is shown when twisted is not importable and we suggest to use 'cubicweb pyramid <appid>' instead. Since 3.24, cubicweb-pyramid got merge in cubicweb so drop this reference in the message.

[cwconfig] Fix two NameError in load_available_configs() This comes from a copy-paste in d92d75b17a85.

[etwist] Drop try/except around definitions of 'all-in-one' configuration and command handlers cubicweb.server is no longer conditionally shipped, neither in python package nor in debian package since 3.24.

[server] Call 'server_startup' hooks at the end of repository "bootstrap" Now that we have guarded all calls to Repository.looping_task() in 'server_startup' hooks we can execute this hooks category at bootstrap step. This way, repositories running without a tasks manager (i.e. those embedded into a WSGI application) will have these hooks triggered.

[pyramid] Drop module-level cache and cleanup looping tasks in tools And use a LRU cache over cached_build_user function. This looping task is problematic because it would not be run from within a WSGI application which does not have a repository with a tasks manager. This pulls an explicit dependency on 'repoze.lru' but it's not a big deal since pyramid already depends on this. RPM spec file not update since it does not even mention pyramid...

[hooks] Return early in server_startup hooks if the repository has not tasks manager

[server] Only register "clean_sessions" looping tasks if repository has a tasks manager This prepares for calling Repository's _prepare_startup() in bootstrap() so that repository instances without a tasks manager running can still have their 'server_startup' hooks triggered (this is particularly useful for BFSS storages).

[utils] Do not pass a tasks manager to Repository in admincnx() We do not call repo.start_looping_tasks() for the instantiated repository, so a tasks manager is useless.

[server] Separate repository bootstrap from initialization Thus we now call 'bootstrap' here and there explicitly and remove call of this method in Repository.__init__(). This way instantiation of a Repository does not *implicitly* triggers the "bootstrap" step, which is arguably not a trivial thing and thus deserves to be independent. In __init__, set 'shutting_down' attribute to None and then to False in bootstrap as a mean to indicate that, when initialized, a repository is neither shutting down nor started (not sure where this is used though).

[server] Also remove tmpdir in case of error during restore database command

[server] Rename Repository's init_cnxset_pool method bootstrap See the (removed) docstring for a rationale...

[server] Replace server config's init_cnxset_pool attribute by a "bootstrap" parameter in Repository This 'init_cnxset_pool' class attribute is actually used to control whether a Repository instance should be "bootstraped" (see Repository.init_cnxset_pool() for a definition of "bootstrap") or not. I seems clearer to me to have this controlled by a boolean "bootstrap" initialization parameter in Repository.

[server] Drop close/open steps in NativeSQLSource.restore() The only place where this is called is in ServerMigrationHelper.restore_database() where config.init_cnxset_pool is set to False. So these steps appear to be useless.

[skel] Fix rpm's .spec file skeleton It's broken for new-style packages since it doesn't consider (even remove) python packages.

[server] Move "starting repository..." message where this actually happens

[server] Initialize Repository without a tasks manager in ServerMigrationHelper.restore_database() We don't need a tasks manager here I think. Thus instantiate the Repository from its class directly instead of using config.repository() which set a tasks manager.

[server] Fix a typo in log message in Repository.init_cnxset_pool()

[cwconfig] Issue a warning if a configuration module cannot be loaded We already do this when loading ctl plugin modules, so make things symmetrical for config modules.

[ctl] Load available configurations in "cubicweb-ctl list" Otherwise we rely on associated ctl plugins to be loaded which is arguably orthogonal to configurations.

[skeleton] Set cubicweb.bwcompat to False in development.ini Quite a few things from the default web UI do not work when running the application through pserve (in particular, data files are not apparently served). So better disable the "bwcompat" mode until we are sure it works properly (or it gets dropped ;)).

[pyramid] Use existing repo in PyramidCWTest We thus do not make use of config_from_cwconfig() and build the configurator instance by hand prior to include 'cubicweb.pyramid'.

[pyramid] Make it possible to setup CubicWeb instance from an existing repo in includeme()

[pyramid] Move loading of pyramid_debugtoolbar later in includeme() To gather cwconfig and repository instantiations.

[skeleton,pyramid] Move pyramid app definition in cubicweb.pyramid module The application definition is actually not specific to the final "cube" being bootstrapped from skeleton. This patch thus move the pyramid application function into cubicweb.pyramid module and let cubicweb register the "paste.app_factory" entry point (instead of the bootstrapped cube). Useless call to `config.scan` is dropped along the way.

[pyramid] Add a copyright and docstring to all modules We add copyright statements for both UNLISH (original author) and LOGILAB.

[cwconfig] Strip "cubicweb_" prefix from cube name in available_cubes() Closes #17054738.

[web] fix os.rename usage on windows when destination exists os.rename on windows will raise OSError (or WindowsError subclass) when if destination file already exists. Also check that exception is EEXIST. There is an attempt to fix in f6ba947c but using IOError instead of OSError. Closes #14214794

[web/views] show composite entities in delete view Disabled by default, can be activated with show_composite = True Display first level of composite entities in a treeview, limited to the current (page_size - 1), so the maximum of displayed entities bump to page_size * (page_size - 1).

[workflow] Utilities for declarative definition of workflows Closes #5337897

[server] Remove pseudo-handling of exceptions in Repository._delete_cascade_multi() It seems to me that we never want to pass on an exception here be it in "test" mode or not, so remove the last `except Exception:` clause. For instance, the actual implementation would pass on an IntegrityError and thus possibly hide it in a log message. Concerning the `except Unauthorized:`, as said by the log message this should not happen because we are within a `security_enabled(write=False)` context. So remove it as well. As far as I can tell, this strange handling of exceptions dates from 37668bf302f5 and there is no clear justification to it.

[web] fix os.rename usage on windows when destination exists os.rename on windows will raise OSError (or WindowsError subclass) when if destination file already exists. Also check that exception is EEXIST. There is an attempt to fix in f6ba947c but using IOError instead of OSError. Closes #14214794

[skeleton] Add "cubicweb.session.secret" to development.ini Commented as other "secret" settings. At least, now all settings are present in the development.ini file. [ci skip]

[pyramid] Include 'cubicweb.pyramid.predicates' where it is used It's only used in "rest_api" module.

[pyramid] Rename make_cubicweb_application function as config_from_cwconfig And update its docstring. This function does not actually "make a cubicweb application", it just builds a pyramid.config.Configurator instance from a CubicWeb config object.

[server] Reprase messages of db-check to avoid using "system" and "sources" And rather mention "entity type table" and "entities" table, which are clearer references.

[predicates] Fix reference to has_related_entities in partial_has_related_entities docstring

Fix possible double import of cubes modules When cubes using the new layout are imported with 'cubicweb_<cube>' and with 'cubes.<cube>', the same module is imported twice. Handle this by adding 'cubes.<cube>' to sys.modules when importing from 'cubicweb_<cube>'. Move load_module() to a sub class _CubesLoader to share informations computed in find_modules(). Don't handle subpackages in _CubesImporter and rely on normal import machinery instead. Add a test and use unittest from cubicweb.devtools.testlib which resolve to unittest2 on PY2 with assertLogs() method.

Make load_module() return existing module if present in sys.modules Otherwise the reload() builtin will not work correctly: https://www.python.org/dev/peps/pep-0302/#specification-part-1-the-importer-protocol

[server] Ignore computed relations in "relations" integrity checker

[server] Use global registry variable of integrity checker functions Instead of querying globals() in checkintegrity.check(), we maintain a _CHECKERS dict mapping checker name to function in checkintegrity module. This is later used to build the list of available checkers in 'db-check' command help.

[doc] Add 3.25 release notes [ci skip]

[repository] possibility to disable connections pooling Add a new configuration option 'connections-pooler-enabled' (default yes). If set to no, disable connection pooling and open new connections to the database on demand. This allow to use CubicWeb with other poolers (such as pgbouncer).

[js] Fix loading image URL

[migration] Move pyramid-related migration after "entities" table modifications Otherwise, we getting into `add_entity_type('CWSession')` we try to insert a CWEType without any 'asource' column (per 3.24's schema) whereas the backend still wants it. Closes #17054035.

Merge 3.24.5 into default branch

[entities] Introduce an IDublinCore adapter Move all dc_ methods from AnyEntity to the new IDublinCore adapter and proxy them through a __getattr__ method on AnyEntity. Current test suite should be enough. Closes #3119992.

[skeleton] Add an entry point and configuration for running a Pyramid application We add a `pyramid_main` function in __init__.py that instantiate the WSGI application using "cubicweb.pyramid"; this is defined as an entry point (in setup.py) so that pserve_ can find it. Alongside comes a development.ini file which includes basic settings so that running a Pyramid+CubicWeb application works (only the "instance=<appid>" may be passed as a command-line argument). Logging is also configured there, but only includes the cube at stake and cubicweb (others could be added if needed). .. _perse: \ http://docs.pylonsproject.org/projects/pyramid/en/1.8-branch/pscripts/pserve.html

[cwconfig] Issue a warning when a "core" ctl plugin failed to load This is often because of a missing dependency, the warning would indicate this hopefully.

[dataimport] Fix "existing_relations" parameter name in ExtEntitiesImporter

[ical] Create ICal events or todos depending on component type returned by adapter

Added tag 3.24.5, debian/3.24.5-1, centos/3.24.5-1 for changeset 70d28e632206

Prepare 3.24.5

[devtools] Skip computed relations when attempting to auto-populate test database Fix the automatic database population underlying automatic test to avoid attempt to create computed relations.

[migration] Stop asking confirm to commit

[migration] Fix crash on 3.24 migration Some sql using a column droped the line above crash migration of cw < 3.24.

[devtools] Skip computed relations when attempting to auto-populate test database Fix the automatic database population underlying automatic test to avoid attempt to create computed relations.

[test] Use our BaseTestCase in unittest_rtags.py We need it DeprecatedInstanceWithoutModule for assertWarns method. Use this instead of unittest2 directly with a try/except block to get rid of flake8 error about import not being one top of file.

[server] Add a blank line before _CnxSetPool to fix PEP8 error

[pkg] Use logilab-common's master repository in dev requirements Should follow-up on 7b2247098f58 after integration on logilab-common side.

[rtags] pass module name to RegistrableInstance Instantiate a RegistrableInstance without passing the module name is deprecated in https://www.logilab.org/patch/10047069

[repository] move cnxset pool handling to a helper class The class has responsability to handle connections pool operations in a single public attribute 'cnxsets'. On Repository _get_cnxset() and _free_cnxset() are replaced by cnxsets.get() and cnxsets.release(). Drop multiple access to private attributes and methods from outside of Repository.

[pkg] Fix some error on building documentation

Merge 3.24.4 into default branch

Added tag 3.24.4, debian/3.24.4-1, centos/3.24.4-1 for changeset 35fd54c0065d

Prepare 3.24.4

[source,native] Generalize the IntegrityError constraint parsing regex This allows catching constraint-related errors that have been translated by psycopg2, replacing surrounding english quotes by french quotes + space for instance. Closes #17047951.

[pyramid] use pyramid.request `path_info` property instead of `path` path is concatenation of SCRIPT_NAME and PATH_INFO we just need PATH_INFO here

[pyramid] fix login route with language-mode = url-prefix

[devtools] Make timeout error in qunit tests configurable and use SkipTest in test_jscript.py This is grafted from 87443f279b0f in default branch (got tired of CI failing because of these unreliable tests).

[web] Handle unspecified value in TZDatetimeField `field._ensure_correctly_typed(self, form, value)` should return None if value is None. Closes #17050181.

[web] Handle unspecified 'date' or 'time' in JQueryDateTimePicker.process_field_data() dict.get() may return None, which has no attribute strip.

[cwconfig] create a virtual "cubes" package _CubesImporter now handle import of "cubes" and return a virtual module so that "cubes" is always importable without having to create a __init__.py in CUBES_DIR and add CUBES_DIR/.. to sys.path. Drop custom code in setup.py used to create CUBES_DIR/__init__.py and drop now useless warning when cubicweb is installed in develop mode Update test_cubes_path() and ensure 'cubes.file' is not loaded before testing its import. This test seems to have mistakes because cubes are loaded automatically when discovering cw commands (cubes.<x>.ccplugin), not sure how this should be fixed definitely.

Change hooks control (deny_all_hooks_but / allow_all_hooks_but) to be more predictable Prior to this, if one execute code like: with cnx.hooks.deny_all_hooks_but('metadata'): with cnx.hooks.deny_all_hooks_but(): # mycode 'metadata' hooks will be activated anyway in the inner block, which is rather unexpected (of course in real life you only see the latest hooks control statement, the former being higher in the call stack). This is due to the underlying usage of old `enable_hook_categories` / `disable_hook_categories` methods, which were introduced much before the now official context manager based API (with `cnx.[deny|all]_all_hooks_but(...)`). To move on, this patch drop the two legacy methods, rename and privatize related internal state on the connection (`hooks_mode` becomes `_hooks_mode`, `disabled_hook_cats` and `enabled_hook_cats` become `_hooks_categories`) and reimplement the `_hooks_control` context manager to simply update them. See the added unit test for details.

[cwconfig] make appobjects_cubes_modnames() public The method is used in cubicweb.cwvreg without underscore and outside of the config class, so let's make this method public. Fix autoreload with twisted.

Fix str(Unauthorized) which is really problematic on python 3 where __unicode__ is never called anymore. This was caused because in PY3, CubicWebException.__str__ was referencing __unicode__ implementation of CubicWebException, not of its subclass that implements it. Fix a flake8 style warning about lambda assignment as a bonus.

[test] avoid using a deprecated feature of yams handle_file() now require a module name (yams@d9120d7)

[test] add some tests to web.views.baseviews

[tox] Drop "touch {envdir}/share/cubicweb/cubes/__init__.py" command on py34 env This was useful when we installed cubicweb in "develop" mode but we do not do this anymore.

Drop support for https-url in all-in-one.conf This feature allowing an instance to have both anonymous and authenticated (https) url is not used anymore, let's drop it.

[test] use TemporaryDirectory context manager For consistency with others tests, use TemporaryDirectory from cubicweb.devtools.testlib.

[tox] Drop "exclude" option from flake8 config and adjust flake8-ok-files.txt accordingly Some files listed in flake8-ok-files.txt were excluded of flake8 run because of the "exclude" option in tox.ini. Some of them were non-existent files (moved), some others were actually non-flake8-compliant. In the latter case, we adjust trivial errors (blank lines, module import not on top of file) but remove others (like cubicweb/test/unittest_utils.py) from flake8-ok-files.txt.

[autoform] Go through guess_field even when field class is specified using uicfg guess_field does not only find the proper field class for some attribute/relation, but also do some basic configuration depending on e.g. constraints. Before this patch, if one was specifying explicitly the field class using uicfg.autoform_field, this automatic configuration wasn't happening. Change form.field_by_name and ff.guess_field so explicit class are also automatically configured now. Closes #14474840

[uilib] Fix incorrect serialization of python dicts into javascript objects Valid javascript object keys must be surrounded by quotes unless they are valid javascript identifiers. Valid identifiers are a defined by complex and changing specs, so it is much simpler to always use quotes. Closes #17046704.

[repository] Fix connection-pool-size not set to 1 with quick_start enabled config.load_schema() and config.init_cube() reload configuration options from config file, so a manually set connections-pool-size (eg. when quick_start is enabled) wasn't working.

[migration] Enhance assertion message to get a chance to fix the problem

[migration] Skip virtual rtypes when adding relation definition through add_cube Virtual rtypes should be skipped as they are added dynamically on loading schema. Those are skipped by e.g. schema serialisation or add_entity_type, do the same thing here.

[migration] Avoid unnecessary intermediary commit when migrating a schema Those are low-hanging fruit following changes in ac74476d686c (Fix addition of entity type including boundary constraints on its own attributes): this is no more necesary to commit the get a new relation type definition in the schema. At some point we could/should probably do more on this topic to avoid intermediary commit on e.g. entity type addition, but this requires more work.

[migration] Fix addition of entity type including boundary constraints on its own attributes This was failing because of the sequence of sql executing when adding an attribute: 1. add entity type <commit> 2. add relation type for attribute 1 <commit> 3. add attribute 1 and associated constraints <commit> etc. In the case of e.g. start/end constraint, we were trying to add the constraint before addition of the constrained attribute (e.g. add constraint on 'start' referencing 'end', but only 'start' has been added yet, not 'end'). This patch fix this by: * adding the relation type to the schema without having to commit, but keeping the operation to revert the addition if necessary - this allows to a single commit for all attributes of the entity type ; * using a LateOperation on constraint operation, so we ensure attributes are actually added before any constraint is added.

[schema sync] Refactor AfterAddCWRTypeHook and AfterAddCWComputedRTypeHook so the latter inherit from the former and enhance their docstring.

Drop embed-allowed option, gone away for a long time

Never record undo information for session or data import for session, it may causes unlimited database size expansion while it makes no sense to undo this kind of changes. For data import, they are created programatically hence we may not want to undo this either.

[cwvreg] load registry using modules names instead of directories Introspect cubicweb, cubes and apphome using pkgutil to generate the full list of modules names for loading registries. Avoiding using bogus logilab.common.modutils.modpath_from_file().

[schema] load schema from modules names instead of directories Introspect cubicweb, cubes and apphome using pkgutil to generate the full list of modules names for loading the schema. Keep historical behavior and check if source .py file exists if a module is found using python bytecode file (.pyc and .pyo) Loading schema from apphome require apphome to be present in sys.path and that "schema" module resolve to a file located in apphome. Update migraction tests to explicitely update sys.path when loading schema from different apps, use a contextmanager for this so it's more readable. Require updated logilab-common and yams

[devtools/test] Skip qunit tests in case of timeout Follow-up on 87443f279b0f where other similar tests got adjusted not to fail (in particular in our CI environment).

[autoform] Avoid two calls to field.process_form for the same field in some cases when some entity is being created and data include non-inlined relations, the values for this relation are now stored for later usage, avoiding two calls to field's process_form method, which may be unexpected for custom fields. This has been discovered in saem_ref#f5444b1f9770. Reorganize imports in the test along the way.

[pyramid] Don't use unsafe_cnx_context_manager for write queries we may have clumsy error that hide previous problems on attempting to commit the transaction. Closes #16753531

Repair database wrt indexes / unique constraints * recreate indexes dropped by 3.23 migration (but it's still unclear why) * attempt drop remaining extra indexes (there may be a bunch of these on old instances) * warn about missing expected indexes Closes #16666137

Enhance postgres index filter with values from real life This has been found on fairly old instances of ours (cwo/elo). Dunno if we really want this in, it may lead to false negatives.

Extract functions to list expected/found database indexes from check_indexes so they may be used from other contexts. Use consistent naming along the way: always use indexes, not indices.

Simplify and fix _cw.drop_entity_cache * it's never called with an eid as argument, beside in a useless case in test (removed) * the only place where it's called from outside the tests is in full-text reindexation in server.checkintegrity: we could removed the request implementation and move it in unittest_rset, byt I decided to keep it for consistency with all other entity cache handling methods * get back a fix from Julien Cristau for the connection's implementation, quoting is commit message: When removing an entity from the transaction's cache, clear the entity's own cache May avoid issues where an entity object is still accessible somewhere else (e.g. an operation) after dropping it from the transaction's cache, with a stale attribute or relation cache.

[entity/optimization] Cache rset when entity.related is called with entities=False If fail to see why we couldn't cache in this case, while this is important for optimization reason: when doing a single HTTP request, some queries are done two or three times because of predicates / uicfg or alike. Also always store tuple and not list in the cache, because: * else we get some regression * and inconsistent result type (tuple or list) * and it simply feels better to cache an unmutable object.

Ensure in 3.24 migration that entities from LDAP have consistent cwuri Closes #16666157

[doc/book] fix error in markup

[rtags] Allow to 'derive' rtags Since some releases, rtags (structure underlying uicfg) have selector and may be copied using something like: new_rtags = deepcopy(original_rtags) new_rtags.__module__ = __name__ new_rtags.__select__ = custom_selector The problem is that starting from that, both rtags wil diverge and changes in original_rtags won't be considered, while we usually want to set a few specific rules only in new_rtags. To fix this problem, this cset introduces the notion of "derivated/parent" rtag, eg: new_rtags = original_rtags.derive(__name__, custom_selector) Beside easier copying, when using the above method changes in original_rtags which are not overriden by new_rtags will be considered since it only hold its specific rules but look among its parent chain for non-found keys. Along the way, flake8 unittest_rtags. Closes #16164880

[test] Use bare unittest in unittest_rtags

[rtags] Docstring fix

[cleanup] flake8 rtags.py

[pkg] Require passlib >= 1.7.0 Since bc9d901cb9e6 we use 1.7.0 API, update packaging accordingly.

[web/http_headers] do not crash when IfModifiedSince is empty string (closes #16527954)

[pkg] Set an upper bound to passlib We're using a deprecated API and are getting the following deprecation warning: :: the method passlib.context.CryptContext.encrypt() is deprecated as of Passlib 1.7, and will be removed in Passlib 2.0, use CryptContext.hash() instead. So let's make sure we're getting a working library. [ci skip]

[cwconfig] Retrieve real path of cubes directory Since logilab-common 1.3.0, we have real path expansion modutils.modpath_from_file. But the extrapath parameter that is passed from cubicweb.schema reader may still have symlinks, so expand them here to hopefully have consistent comparison in modpath_from_file.

[server] Use CryptContext's hash method instead of deprecated encrypt method Getting rid of the following deprecation warning: :: the method passlib.context.CryptContext.encrypt() is deprecated as of Passlib 1.7, and will be removed in Passlib 2.0, use CryptContext.hash() instead.

Merge public heads

[cwctl] on upgrade, clear instance_md5_version cache The generate-static-datadir depends on a correct config.instance_md5_version(), and we just invalidated it by upgrading the instance.

[twisted] add request error handler to avoid finishing it twice And avoid stack traces like:: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 783, in __bootstrap self.__bootstrap_inner() File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 763, in run self.__target(*self.__args, **self.__kwargs) --- <exception caught here> --- [...] File "/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 196, in _worker result = context.call(ctx, function, *args, **kwargs) File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext return func(*args,**kw) File "/home/me/envs/grshell-cw/cubicweb/statsd_logger.py", line 121, in __call__ return self.callable(*args, **kw) File "/home/me/envs/grshell-cw/cubicweb/etwist/server.py", line 131, in render_request code=500, twisted_request=request) File "/home/me/envs/grshell-cw/cubicweb/etwist/http.py", line 22, in __init__ self._finalize() File "/home/me/envs/grshell-cw/cubicweb/etwist/http.py", line 46, in _finalize self._twreq.finish() File "/usr/lib/python2.7/dist-packages/twisted/web/server.py", line 228, in finish return http.Request.finish(self) File "/usr/lib/python2.7/dist-packages/twisted/web/http.py", line 931, in finish "Request.finish called on a request after its connection was lost; " exceptions.RuntimeError: Request.finish called on a request after its connection was lost; use Request.notifyFinish to keep track of this.

[web/tests] Hide DeprecationWarnings

[web] remove next_tabindex() It's hard to see it work reliably in view of multiple server processes. If something like that is needed it should probably be built on the client (js) side.

[views/optimization] Ensure we call rset.possible_actions with the same argument rset.possible_actions (which should definitly not be an ResultSet method, but that's another story) has been designed to hold a cache to compute possible actions for a only once, since this may be a fairly costly operation (notably because of the 'has_editable_relations' predicates). But this has been broken by introduction of a new 'view' parameter which is not given by every call. To fix this, this cset adds the missing view argument where necessary and reimplements the rset's method to assert it's always called with the same key. Unfortunatly, those changes have to be ported to squareui and bootstrap cubes as well.

[schema/optimization] Ensure read permissions are deactivated when we're checking a permission When called from a web request, since we *are* checking some permission, read permissions don't have to be introduced in this query. We may avoid that since there now more any differences between web and repo connections, so let's do it.

[web] Enhance query log file * Add an overall page generation time * Don't log URL without any query (e.g. static files)

[massive store] Don't store eids_seq_range as a store attribute since it's not considered after object's initialization.

Flake8 bits with some copyright updates

[test] Use plain unittest in unittest_views_forms.py

Some py3k related fixes: use text_type instead of unicode

[cwctl] Kill deprecated ordered_instances method no need to keep bw compat for this.

[test/cleanup] Use plain unittest

[test/cleanup] flake8 unittest_rqlannotation.py

[debian] logilab >= 1.2.2 in requires (only in build depends) Related to #16404515.

[debian] move unittest2 to requires not recommends closes #16404515

[devtools] named context for boxes subTest for consistency

Added tag 3.24.3, debian/3.24.3-1, centos/3.24.3-1 for changeset bb5904cd284e

[pkg] 3.24.3

[cwconfig] ensure CubicWebNoAppConfiguration returns a typed value (closes #16364459)

[views/optimization] Makes has_editable_relations predicate less costly by using generator The `has_editable_relations` predicate is used to say if the 'modify' action should appear. To do so, it checks if anything is editable (ie. something in the attributes, relations or inlined section of the automatic form). This may be costly since it may have to check several permissions. To optimize this a bit, this cset turns list into generator so that we'll avoid unnecessary work as soon as we find a match.

[cwconfig] Trim down allowed exceptions in gettext language setting I can't see any valid reason to pass on ImportError and AttributeError. In particular, the latter shadowed a Python 3 error until 4f43e64603ef.

[pkg] Configure "universal" wheel build According to https://packaging.python.org/distributing/#wheels, we should create universal wheels (no 2to3 step, no C extension). Configure this in setup.cfg. [ci skip]

[rqlrewrite] Test and fix potential NameError We were referencing a loop variable, which may lead to a name error and show a potential error if there are several matching variables. To avoid this, introduce a list to hold every encountered variable and process all of them later.

flake8 uicfg

[uicfg] Remove unpredictability from autoform_section initialization It was attempting to guess smart default values by looking at the opposite relation (neg_role), but depending on the (random) ordering of the schema, it may or may not have been set yet, leading to unpredictable result with varying hash seed. Remove those fuzzy lookup all at once, it may change some (unreliable) things in your app but at least it's still a bit easier to explain. Closes #16272968

[schema/optimization] Wrap rql expressions into EXISTS node when checking individual permissions Some RQL expression may retrieve several results. It makes sense to wrap them into an EXISTS node to optimized things a bit. This is already done by security insertion of 'read' rql expressions.

[views/optimization] Prefetch state'names and comment format in wfhistory component This will avoid later queries to retrieve them.

[devtools] Make timeout error in qunit tests configurable and use SkipTest in test_jscript.py The reason is that the latter tests quite often fail in our CI environment, possibly due to concurrent execution of the browser, and this makes it hard to identify a truly failing build there. So skip tests with a timeout error.

Added tag 3.24.2, debian/3.24.2-1, centos/3.24.2-1 for changeset 296077513782 And drop erroneous 0.34.2 tag.

Added tag 0.34.2, debian/0.34.2-1, centos/0.34.2-1 for changeset 296077513782

[pkg] Version 3.24.2

Merge with 3.23 head

[migration] Fix 3.23.0 migration script, broken by a703f00718c2 * first query use "IN" where it should use "=" -> crash * second query is missing an interpolation -> crash * third query is attempting to remove constraint that have already been removed by the second query -> crash

[doc] Add 3.24 release date and reference in index pages [ci skip]

Closing "oldstable" branch We now release-based branch naming.

Closing "stable" branch We now release-based branch naming.

Merge 3.24 branch

[pkg] Bump version to 3.25.0.dev0

Added tag 3.24.1, debian/3.24.1-1, centos/3.24.1-1 for changeset da52fda6f15b

[debian] update changelog's time tag

[pyramid] Drop reference to pyramid_cubicweb in ctl command docstring [ci skip]

[debian] Rework split of cubicweb-ctl package It does not appear simple to have the .install files work. So just copy cubicweb-ctl script in DESDIR of cubicweb-ctl binary package and remove .install files. Related to #16133259.

[skeleton] Rephrase long description of Debian package into something meaningful [ci skip]

[skeleton] Depends on python-cubicweb in debian packaging [ci skip]

[debian] Have a single python-cubicweb binary for the library But keep distinct meta-packages for dependencies of the HTTP server (Twisted/Pyramid) and database backend. Related to #16133259.

[debian] Update debian packaging (closes #16133259) - use dh_python, pybuild and debhelper>=9, - refactor and simplify the debian/rules, - rename binary packages (but cubicweb-ctl) to python-xxx - remove daemon handling stuff (initscripts...) from cubicweb-ctl (one should now use a standard WSGI delivery method),

[pkg] Version 3.24.1

[tox] Use `python -m check_manifest` instead of check-manifest program To make sure we use Python from tox's virtualenv.

add debian buildpackage-generated files to hgignore

[test] Fix language negotiation test At least when executed alone (there seems to be a test isolation problem which I could not figure out).

[pyramid] No more need for an internal connection when user is cached

[web] req.lang should be None or a non-empty string