[cwvreg] load registry using modules names instead of directories Introspect cubicweb, cubes and apphome using pkgutil to generate the full list of modules names for loading registries. Avoiding using bogus logilab.common.modutils.modpath_from_file().

[schema] load schema from modules names instead of directories Introspect cubicweb, cubes and apphome using pkgutil to generate the full list of modules names for loading the schema. Keep historical behavior and check if source .py file exists if a module is found using python bytecode file (.pyc and .pyo) Loading schema from apphome require apphome to be present in sys.path and that "schema" module resolve to a file located in apphome. Update migraction tests to explicitely update sys.path when loading schema from different apps, use a contextmanager for this so it's more readable. Require updated logilab-common and yams

[devtools/test] Skip qunit tests in case of timeout Follow-up on 87443f279b0f where other similar tests got adjusted not to fail (in particular in our CI environment).

[autoform] Avoid two calls to field.process_form for the same field in some cases when some entity is being created and data include non-inlined relations, the values for this relation are now stored for later usage, avoiding two calls to field's process_form method, which may be unexpected for custom fields. This has been discovered in saem_ref#f5444b1f9770. Reorganize imports in the test along the way.

[pyramid] Don't use unsafe_cnx_context_manager for write queries we may have clumsy error that hide previous problems on attempting to commit the transaction. Closes #16753531

Repair database wrt indexes / unique constraints * recreate indexes dropped by 3.23 migration (but it's still unclear why) * attempt drop remaining extra indexes (there may be a bunch of these on old instances) * warn about missing expected indexes Closes #16666137

Enhance postgres index filter with values from real life This has been found on fairly old instances of ours (cwo/elo). Dunno if we really want this in, it may lead to false negatives.

Extract functions to list expected/found database indexes from check_indexes so they may be used from other contexts. Use consistent naming along the way: always use indexes, not indices.

Simplify and fix _cw.drop_entity_cache * it's never called with an eid as argument, beside in a useless case in test (removed) * the only place where it's called from outside the tests is in full-text reindexation in server.checkintegrity: we could removed the request implementation and move it in unittest_rset, byt I decided to keep it for consistency with all other entity cache handling methods * get back a fix from Julien Cristau for the connection's implementation, quoting is commit message: When removing an entity from the transaction's cache, clear the entity's own cache May avoid issues where an entity object is still accessible somewhere else (e.g. an operation) after dropping it from the transaction's cache, with a stale attribute or relation cache.

[entity/optimization] Cache rset when entity.related is called with entities=False If fail to see why we couldn't cache in this case, while this is important for optimization reason: when doing a single HTTP request, some queries are done two or three times because of predicates / uicfg or alike. Also always store tuple and not list in the cache, because: * else we get some regression * and inconsistent result type (tuple or list) * and it simply feels better to cache an unmutable object.

Ensure in 3.24 migration that entities from LDAP have consistent cwuri Closes #16666157

[doc/book] fix error in markup

[rtags] Allow to 'derive' rtags Since some releases, rtags (structure underlying uicfg) have selector and may be copied using something like: new_rtags = deepcopy(original_rtags) new_rtags.__module__ = __name__ new_rtags.__select__ = custom_selector The problem is that starting from that, both rtags wil diverge and changes in original_rtags won't be considered, while we usually want to set a few specific rules only in new_rtags. To fix this problem, this cset introduces the notion of "derivated/parent" rtag, eg: new_rtags = original_rtags.derive(__name__, custom_selector) Beside easier copying, when using the above method changes in original_rtags which are not overriden by new_rtags will be considered since it only hold its specific rules but look among its parent chain for non-found keys. Along the way, flake8 unittest_rtags. Closes #16164880

[test] Use bare unittest in unittest_rtags

[rtags] Docstring fix

[cleanup] flake8 rtags.py

[pkg] Require passlib >= 1.7.0 Since bc9d901cb9e6 we use 1.7.0 API, update packaging accordingly.

[web/http_headers] do not crash when IfModifiedSince is empty string (closes #16527954)

[pkg] Set an upper bound to passlib We're using a deprecated API and are getting the following deprecation warning: :: the method passlib.context.CryptContext.encrypt() is deprecated as of Passlib 1.7, and will be removed in Passlib 2.0, use CryptContext.hash() instead. So let's make sure we're getting a working library. [ci skip]

[cwconfig] Retrieve real path of cubes directory Since logilab-common 1.3.0, we have real path expansion modutils.modpath_from_file. But the extrapath parameter that is passed from cubicweb.schema reader may still have symlinks, so expand them here to hopefully have consistent comparison in modpath_from_file.

[server] Use CryptContext's hash method instead of deprecated encrypt method Getting rid of the following deprecation warning: :: the method passlib.context.CryptContext.encrypt() is deprecated as of Passlib 1.7, and will be removed in Passlib 2.0, use CryptContext.hash() instead.

Merge public heads

[cwctl] on upgrade, clear instance_md5_version cache The generate-static-datadir depends on a correct config.instance_md5_version(), and we just invalidated it by upgrading the instance.

[twisted] add request error handler to avoid finishing it twice And avoid stack traces like:: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 783, in __bootstrap self.__bootstrap_inner() File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 763, in run self.__target(*self.__args, **self.__kwargs) --- <exception caught here> --- [...] File "/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 196, in _worker result = context.call(ctx, function, *args, **kwargs) File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext return func(*args,**kw) File "/home/me/envs/grshell-cw/cubicweb/statsd_logger.py", line 121, in __call__ return self.callable(*args, **kw) File "/home/me/envs/grshell-cw/cubicweb/etwist/server.py", line 131, in render_request code=500, twisted_request=request) File "/home/me/envs/grshell-cw/cubicweb/etwist/http.py", line 22, in __init__ self._finalize() File "/home/me/envs/grshell-cw/cubicweb/etwist/http.py", line 46, in _finalize self._twreq.finish() File "/usr/lib/python2.7/dist-packages/twisted/web/server.py", line 228, in finish return http.Request.finish(self) File "/usr/lib/python2.7/dist-packages/twisted/web/http.py", line 931, in finish "Request.finish called on a request after its connection was lost; " exceptions.RuntimeError: Request.finish called on a request after its connection was lost; use Request.notifyFinish to keep track of this.

[web/tests] Hide DeprecationWarnings

[web] remove next_tabindex() It's hard to see it work reliably in view of multiple server processes. If something like that is needed it should probably be built on the client (js) side.

[views/optimization] Ensure we call rset.possible_actions with the same argument rset.possible_actions (which should definitly not be an ResultSet method, but that's another story) has been designed to hold a cache to compute possible actions for a only once, since this may be a fairly costly operation (notably because of the 'has_editable_relations' predicates). But this has been broken by introduction of a new 'view' parameter which is not given by every call. To fix this, this cset adds the missing view argument where necessary and reimplements the rset's method to assert it's always called with the same key. Unfortunatly, those changes have to be ported to squareui and bootstrap cubes as well.

[schema/optimization] Ensure read permissions are deactivated when we're checking a permission When called from a web request, since we *are* checking some permission, read permissions don't have to be introduced in this query. We may avoid that since there now more any differences between web and repo connections, so let's do it.

[web] Enhance query log file * Add an overall page generation time * Don't log URL without any query (e.g. static files)

[massive store] Don't store eids_seq_range as a store attribute since it's not considered after object's initialization.