[devtools] Make timeout error in qunit tests configurable and use SkipTest in test_jscript.py This is grafted from 87443f279b0f in default branch (got tired of CI failing because of these unreliable tests).

[web] Handle unspecified value in TZDatetimeField `field._ensure_correctly_typed(self, form, value)` should return None if value is None. Closes #17050181.

[web] Handle unspecified 'date' or 'time' in JQueryDateTimePicker.process_field_data() dict.get() may return None, which has no attribute strip.

[cwconfig] create a virtual "cubes" package _CubesImporter now handle import of "cubes" and return a virtual module so that "cubes" is always importable without having to create a __init__.py in CUBES_DIR and add CUBES_DIR/.. to sys.path. Drop custom code in setup.py used to create CUBES_DIR/__init__.py and drop now useless warning when cubicweb is installed in develop mode Update test_cubes_path() and ensure 'cubes.file' is not loaded before testing its import. This test seems to have mistakes because cubes are loaded automatically when discovering cw commands (cubes.<x>.ccplugin), not sure how this should be fixed definitely.

Change hooks control (deny_all_hooks_but / allow_all_hooks_but) to be more predictable Prior to this, if one execute code like: with cnx.hooks.deny_all_hooks_but('metadata'): with cnx.hooks.deny_all_hooks_but(): # mycode 'metadata' hooks will be activated anyway in the inner block, which is rather unexpected (of course in real life you only see the latest hooks control statement, the former being higher in the call stack). This is due to the underlying usage of old `enable_hook_categories` / `disable_hook_categories` methods, which were introduced much before the now official context manager based API (with `cnx.[deny|all]_all_hooks_but(...)`). To move on, this patch drop the two legacy methods, rename and privatize related internal state on the connection (`hooks_mode` becomes `_hooks_mode`, `disabled_hook_cats` and `enabled_hook_cats` become `_hooks_categories`) and reimplement the `_hooks_control` context manager to simply update them. See the added unit test for details.

[cwconfig] make appobjects_cubes_modnames() public The method is used in cubicweb.cwvreg without underscore and outside of the config class, so let's make this method public. Fix autoreload with twisted.

Fix str(Unauthorized) which is really problematic on python 3 where __unicode__ is never called anymore. This was caused because in PY3, CubicWebException.__str__ was referencing __unicode__ implementation of CubicWebException, not of its subclass that implements it. Fix a flake8 style warning about lambda assignment as a bonus.

[test] avoid using a deprecated feature of yams handle_file() now require a module name (yams@d9120d7)

[test] add some tests to web.views.baseviews

[tox] Drop "touch {envdir}/share/cubicweb/cubes/__init__.py" command on py34 env This was useful when we installed cubicweb in "develop" mode but we do not do this anymore.

Drop support for https-url in all-in-one.conf This feature allowing an instance to have both anonymous and authenticated (https) url is not used anymore, let's drop it.

[test] use TemporaryDirectory context manager For consistency with others tests, use TemporaryDirectory from cubicweb.devtools.testlib.

[tox] Drop "exclude" option from flake8 config and adjust flake8-ok-files.txt accordingly Some files listed in flake8-ok-files.txt were excluded of flake8 run because of the "exclude" option in tox.ini. Some of them were non-existent files (moved), some others were actually non-flake8-compliant. In the latter case, we adjust trivial errors (blank lines, module import not on top of file) but remove others (like cubicweb/test/unittest_utils.py) from flake8-ok-files.txt.

[autoform] Go through guess_field even when field class is specified using uicfg guess_field does not only find the proper field class for some attribute/relation, but also do some basic configuration depending on e.g. constraints. Before this patch, if one was specifying explicitly the field class using uicfg.autoform_field, this automatic configuration wasn't happening. Change form.field_by_name and ff.guess_field so explicit class are also automatically configured now. Closes #14474840

[uilib] Fix incorrect serialization of python dicts into javascript objects Valid javascript object keys must be surrounded by quotes unless they are valid javascript identifiers. Valid identifiers are a defined by complex and changing specs, so it is much simpler to always use quotes. Closes #17046704.

[repository] Fix connection-pool-size not set to 1 with quick_start enabled config.load_schema() and config.init_cube() reload configuration options from config file, so a manually set connections-pool-size (eg. when quick_start is enabled) wasn't working.

[migration] Enhance assertion message to get a chance to fix the problem

[migration] Skip virtual rtypes when adding relation definition through add_cube Virtual rtypes should be skipped as they are added dynamically on loading schema. Those are skipped by e.g. schema serialisation or add_entity_type, do the same thing here.

[migration] Avoid unnecessary intermediary commit when migrating a schema Those are low-hanging fruit following changes in ac74476d686c (Fix addition of entity type including boundary constraints on its own attributes): this is no more necesary to commit the get a new relation type definition in the schema. At some point we could/should probably do more on this topic to avoid intermediary commit on e.g. entity type addition, but this requires more work.

[migration] Fix addition of entity type including boundary constraints on its own attributes This was failing because of the sequence of sql executing when adding an attribute: 1. add entity type <commit> 2. add relation type for attribute 1 <commit> 3. add attribute 1 and associated constraints <commit> etc. In the case of e.g. start/end constraint, we were trying to add the constraint before addition of the constrained attribute (e.g. add constraint on 'start' referencing 'end', but only 'start' has been added yet, not 'end'). This patch fix this by: * adding the relation type to the schema without having to commit, but keeping the operation to revert the addition if necessary - this allows to a single commit for all attributes of the entity type ; * using a LateOperation on constraint operation, so we ensure attributes are actually added before any constraint is added.

[schema sync] Refactor AfterAddCWRTypeHook and AfterAddCWComputedRTypeHook so the latter inherit from the former and enhance their docstring.

Drop embed-allowed option, gone away for a long time

Never record undo information for session or data import for session, it may causes unlimited database size expansion while it makes no sense to undo this kind of changes. For data import, they are created programatically hence we may not want to undo this either.

[cwvreg] load registry using modules names instead of directories Introspect cubicweb, cubes and apphome using pkgutil to generate the full list of modules names for loading registries. Avoiding using bogus logilab.common.modutils.modpath_from_file().

[schema] load schema from modules names instead of directories Introspect cubicweb, cubes and apphome using pkgutil to generate the full list of modules names for loading the schema. Keep historical behavior and check if source .py file exists if a module is found using python bytecode file (.pyc and .pyo) Loading schema from apphome require apphome to be present in sys.path and that "schema" module resolve to a file located in apphome. Update migraction tests to explicitely update sys.path when loading schema from different apps, use a contextmanager for this so it's more readable. Require updated logilab-common and yams

[devtools/test] Skip qunit tests in case of timeout Follow-up on 87443f279b0f where other similar tests got adjusted not to fail (in particular in our CI environment).

[autoform] Avoid two calls to field.process_form for the same field in some cases when some entity is being created and data include non-inlined relations, the values for this relation are now stored for later usage, avoiding two calls to field's process_form method, which may be unexpected for custom fields. This has been discovered in saem_ref#f5444b1f9770. Reorganize imports in the test along the way.

[pyramid] Don't use unsafe_cnx_context_manager for write queries we may have clumsy error that hide previous problems on attempting to commit the transaction. Closes #16753531

Repair database wrt indexes / unique constraints * recreate indexes dropped by 3.23 migration (but it's still unclear why) * attempt drop remaining extra indexes (there may be a bunch of these on old instances) * warn about missing expected indexes Closes #16666137

Enhance postgres index filter with values from real life This has been found on fairly old instances of ours (cwo/elo). Dunno if we really want this in, it may lead to false negatives.

Extract functions to list expected/found database indexes from check_indexes so they may be used from other contexts. Use consistent naming along the way: always use indexes, not indices.

Simplify and fix _cw.drop_entity_cache * it's never called with an eid as argument, beside in a useless case in test (removed) * the only place where it's called from outside the tests is in full-text reindexation in server.checkintegrity: we could removed the request implementation and move it in unittest_rset, byt I decided to keep it for consistency with all other entity cache handling methods * get back a fix from Julien Cristau for the connection's implementation, quoting is commit message: When removing an entity from the transaction's cache, clear the entity's own cache May avoid issues where an entity object is still accessible somewhere else (e.g. an operation) after dropping it from the transaction's cache, with a stale attribute or relation cache.

[entity/optimization] Cache rset when entity.related is called with entities=False If fail to see why we couldn't cache in this case, while this is important for optimization reason: when doing a single HTTP request, some queries are done two or three times because of predicates / uicfg or alike. Also always store tuple and not list in the cache, because: * else we get some regression * and inconsistent result type (tuple or list) * and it simply feels better to cache an unmutable object.

Ensure in 3.24 migration that entities from LDAP have consistent cwuri Closes #16666157

[doc/book] fix error in markup

[rtags] Allow to 'derive' rtags Since some releases, rtags (structure underlying uicfg) have selector and may be copied using something like: new_rtags = deepcopy(original_rtags) new_rtags.__module__ = __name__ new_rtags.__select__ = custom_selector The problem is that starting from that, both rtags wil diverge and changes in original_rtags won't be considered, while we usually want to set a few specific rules only in new_rtags. To fix this problem, this cset introduces the notion of "derivated/parent" rtag, eg: new_rtags = original_rtags.derive(__name__, custom_selector) Beside easier copying, when using the above method changes in original_rtags which are not overriden by new_rtags will be considered since it only hold its specific rules but look among its parent chain for non-found keys. Along the way, flake8 unittest_rtags. Closes #16164880

[test] Use bare unittest in unittest_rtags

[rtags] Docstring fix

[cleanup] flake8 rtags.py

[pkg] Require passlib >= 1.7.0 Since bc9d901cb9e6 we use 1.7.0 API, update packaging accordingly.

[web/http_headers] do not crash when IfModifiedSince is empty string (closes #16527954)

[pkg] Set an upper bound to passlib We're using a deprecated API and are getting the following deprecation warning: :: the method passlib.context.CryptContext.encrypt() is deprecated as of Passlib 1.7, and will be removed in Passlib 2.0, use CryptContext.hash() instead. So let's make sure we're getting a working library. [ci skip]

[cwconfig] Retrieve real path of cubes directory Since logilab-common 1.3.0, we have real path expansion modutils.modpath_from_file. But the extrapath parameter that is passed from cubicweb.schema reader may still have symlinks, so expand them here to hopefully have consistent comparison in modpath_from_file.

[server] Use CryptContext's hash method instead of deprecated encrypt method Getting rid of the following deprecation warning: :: the method passlib.context.CryptContext.encrypt() is deprecated as of Passlib 1.7, and will be removed in Passlib 2.0, use CryptContext.hash() instead.

Merge public heads

[cwctl] on upgrade, clear instance_md5_version cache The generate-static-datadir depends on a correct config.instance_md5_version(), and we just invalidated it by upgrading the instance.

[twisted] add request error handler to avoid finishing it twice And avoid stack traces like:: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 783, in __bootstrap self.__bootstrap_inner() File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 763, in run self.__target(*self.__args, **self.__kwargs) --- <exception caught here> --- [...] File "/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 196, in _worker result = context.call(ctx, function, *args, **kwargs) File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext return func(*args,**kw) File "/home/me/envs/grshell-cw/cubicweb/statsd_logger.py", line 121, in __call__ return self.callable(*args, **kw) File "/home/me/envs/grshell-cw/cubicweb/etwist/server.py", line 131, in render_request code=500, twisted_request=request) File "/home/me/envs/grshell-cw/cubicweb/etwist/http.py", line 22, in __init__ self._finalize() File "/home/me/envs/grshell-cw/cubicweb/etwist/http.py", line 46, in _finalize self._twreq.finish() File "/usr/lib/python2.7/dist-packages/twisted/web/server.py", line 228, in finish return http.Request.finish(self) File "/usr/lib/python2.7/dist-packages/twisted/web/http.py", line 931, in finish "Request.finish called on a request after its connection was lost; " exceptions.RuntimeError: Request.finish called on a request after its connection was lost; use Request.notifyFinish to keep track of this.

[web/tests] Hide DeprecationWarnings