Fix possible double import of cubes modules When cubes using the new layout are imported with 'cubicweb_<cube>' and with 'cubes.<cube>', the same module is imported twice. Handle this by adding 'cubes.<cube>' to sys.modules when importing from 'cubicweb_<cube>'. Move load_module() to a sub class _CubesLoader to share informations computed in find_modules(). Don't handle subpackages in _CubesImporter and rely on normal import machinery instead. Add a test and use unittest from cubicweb.devtools.testlib which resolve to unittest2 on PY2 with assertLogs() method.

Make load_module() return existing module if present in sys.modules Otherwise the reload() builtin will not work correctly: https://www.python.org/dev/peps/pep-0302/#specification-part-1-the-importer-protocol

[server] Ignore computed relations in "relations" integrity checker

[server] Use global registry variable of integrity checker functions Instead of querying globals() in checkintegrity.check(), we maintain a _CHECKERS dict mapping checker name to function in checkintegrity module. This is later used to build the list of available checkers in 'db-check' command help.

[doc] Add 3.25 release notes [ci skip]

[repository] possibility to disable connections pooling Add a new configuration option 'connections-pooler-enabled' (default yes). If set to no, disable connection pooling and open new connections to the database on demand. This allow to use CubicWeb with other poolers (such as pgbouncer).

[js] Fix loading image URL

[migration] Move pyramid-related migration after "entities" table modifications Otherwise, we getting into `add_entity_type('CWSession')` we try to insert a CWEType without any 'asource' column (per 3.24's schema) whereas the backend still wants it. Closes #17054035.

Merge 3.24.5 into default branch

[entities] Introduce an IDublinCore adapter Move all dc_ methods from AnyEntity to the new IDublinCore adapter and proxy them through a __getattr__ method on AnyEntity. Current test suite should be enough. Closes #3119992.

[skeleton] Add an entry point and configuration for running a Pyramid application We add a `pyramid_main` function in __init__.py that instantiate the WSGI application using "cubicweb.pyramid"; this is defined as an entry point (in setup.py) so that pserve_ can find it. Alongside comes a development.ini file which includes basic settings so that running a Pyramid+CubicWeb application works (only the "instance=<appid>" may be passed as a command-line argument). Logging is also configured there, but only includes the cube at stake and cubicweb (others could be added if needed). .. _perse: \ http://docs.pylonsproject.org/projects/pyramid/en/1.8-branch/pscripts/pserve.html

[cwconfig] Issue a warning when a "core" ctl plugin failed to load This is often because of a missing dependency, the warning would indicate this hopefully.

[dataimport] Fix "existing_relations" parameter name in ExtEntitiesImporter

[ical] Create ICal events or todos depending on component type returned by adapter

Added tag 3.24.5, debian/3.24.5-1, centos/3.24.5-1 for changeset 70d28e632206

Prepare 3.24.5

[devtools] Skip computed relations when attempting to auto-populate test database Fix the automatic database population underlying automatic test to avoid attempt to create computed relations.

[migration] Stop asking confirm to commit

[migration] Fix crash on 3.24 migration Some sql using a column droped the line above crash migration of cw < 3.24.

[devtools] Skip computed relations when attempting to auto-populate test database Fix the automatic database population underlying automatic test to avoid attempt to create computed relations.

[test] Use our BaseTestCase in unittest_rtags.py We need it DeprecatedInstanceWithoutModule for assertWarns method. Use this instead of unittest2 directly with a try/except block to get rid of flake8 error about import not being one top of file.

[server] Add a blank line before _CnxSetPool to fix PEP8 error

[pkg] Use logilab-common's master repository in dev requirements Should follow-up on 7b2247098f58 after integration on logilab-common side.

[rtags] pass module name to RegistrableInstance Instantiate a RegistrableInstance without passing the module name is deprecated in https://www.logilab.org/patch/10047069

[repository] move cnxset pool handling to a helper class The class has responsability to handle connections pool operations in a single public attribute 'cnxsets'. On Repository _get_cnxset() and _free_cnxset() are replaced by cnxsets.get() and cnxsets.release(). Drop multiple access to private attributes and methods from outside of Repository.

[pkg] Fix some error on building documentation

Merge 3.24.4 into default branch

Added tag 3.24.4, debian/3.24.4-1, centos/3.24.4-1 for changeset 35fd54c0065d

Prepare 3.24.4

[source,native] Generalize the IntegrityError constraint parsing regex This allows catching constraint-related errors that have been translated by psycopg2, replacing surrounding english quotes by french quotes + space for instance. Closes #17047951.

[pyramid] use pyramid.request `path_info` property instead of `path` path is concatenation of SCRIPT_NAME and PATH_INFO we just need PATH_INFO here

[pyramid] fix login route with language-mode = url-prefix

[devtools] Make timeout error in qunit tests configurable and use SkipTest in test_jscript.py This is grafted from 87443f279b0f in default branch (got tired of CI failing because of these unreliable tests).

[web] Handle unspecified value in TZDatetimeField `field._ensure_correctly_typed(self, form, value)` should return None if value is None. Closes #17050181.

[web] Handle unspecified 'date' or 'time' in JQueryDateTimePicker.process_field_data() dict.get() may return None, which has no attribute strip.

[cwconfig] create a virtual "cubes" package _CubesImporter now handle import of "cubes" and return a virtual module so that "cubes" is always importable without having to create a __init__.py in CUBES_DIR and add CUBES_DIR/.. to sys.path. Drop custom code in setup.py used to create CUBES_DIR/__init__.py and drop now useless warning when cubicweb is installed in develop mode Update test_cubes_path() and ensure 'cubes.file' is not loaded before testing its import. This test seems to have mistakes because cubes are loaded automatically when discovering cw commands (cubes.<x>.ccplugin), not sure how this should be fixed definitely.

Change hooks control (deny_all_hooks_but / allow_all_hooks_but) to be more predictable Prior to this, if one execute code like: with cnx.hooks.deny_all_hooks_but('metadata'): with cnx.hooks.deny_all_hooks_but(): # mycode 'metadata' hooks will be activated anyway in the inner block, which is rather unexpected (of course in real life you only see the latest hooks control statement, the former being higher in the call stack). This is due to the underlying usage of old `enable_hook_categories` / `disable_hook_categories` methods, which were introduced much before the now official context manager based API (with `cnx.[deny|all]_all_hooks_but(...)`). To move on, this patch drop the two legacy methods, rename and privatize related internal state on the connection (`hooks_mode` becomes `_hooks_mode`, `disabled_hook_cats` and `enabled_hook_cats` become `_hooks_categories`) and reimplement the `_hooks_control` context manager to simply update them. See the added unit test for details.

[cwconfig] make appobjects_cubes_modnames() public The method is used in cubicweb.cwvreg without underscore and outside of the config class, so let's make this method public. Fix autoreload with twisted.

Fix str(Unauthorized) which is really problematic on python 3 where __unicode__ is never called anymore. This was caused because in PY3, CubicWebException.__str__ was referencing __unicode__ implementation of CubicWebException, not of its subclass that implements it. Fix a flake8 style warning about lambda assignment as a bonus.

[test] avoid using a deprecated feature of yams handle_file() now require a module name (yams@d9120d7)

[test] add some tests to web.views.baseviews

[tox] Drop "touch {envdir}/share/cubicweb/cubes/__init__.py" command on py34 env This was useful when we installed cubicweb in "develop" mode but we do not do this anymore.

Drop support for https-url in all-in-one.conf This feature allowing an instance to have both anonymous and authenticated (https) url is not used anymore, let's drop it.

[test] use TemporaryDirectory context manager For consistency with others tests, use TemporaryDirectory from cubicweb.devtools.testlib.

[tox] Drop "exclude" option from flake8 config and adjust flake8-ok-files.txt accordingly Some files listed in flake8-ok-files.txt were excluded of flake8 run because of the "exclude" option in tox.ini. Some of them were non-existent files (moved), some others were actually non-flake8-compliant. In the latter case, we adjust trivial errors (blank lines, module import not on top of file) but remove others (like cubicweb/test/unittest_utils.py) from flake8-ok-files.txt.

[autoform] Go through guess_field even when field class is specified using uicfg guess_field does not only find the proper field class for some attribute/relation, but also do some basic configuration depending on e.g. constraints. Before this patch, if one was specifying explicitly the field class using uicfg.autoform_field, this automatic configuration wasn't happening. Change form.field_by_name and ff.guess_field so explicit class are also automatically configured now. Closes #14474840

[uilib] Fix incorrect serialization of python dicts into javascript objects Valid javascript object keys must be surrounded by quotes unless they are valid javascript identifiers. Valid identifiers are a defined by complex and changing specs, so it is much simpler to always use quotes. Closes #17046704.

[repository] Fix connection-pool-size not set to 1 with quick_start enabled config.load_schema() and config.init_cube() reload configuration options from config file, so a manually set connections-pool-size (eg. when quick_start is enabled) wasn't working.

[migration] Enhance assertion message to get a chance to fix the problem

[migration] Skip virtual rtypes when adding relation definition through add_cube Virtual rtypes should be skipped as they are added dynamically on loading schema. Those are skipped by e.g. schema serialisation or add_entity_type, do the same thing here.

[migration] Avoid unnecessary intermediary commit when migrating a schema Those are low-hanging fruit following changes in ac74476d686c (Fix addition of entity type including boundary constraints on its own attributes): this is no more necesary to commit the get a new relation type definition in the schema. At some point we could/should probably do more on this topic to avoid intermediary commit on e.g. entity type addition, but this requires more work.

[migration] Fix addition of entity type including boundary constraints on its own attributes This was failing because of the sequence of sql executing when adding an attribute: 1. add entity type <commit> 2. add relation type for attribute 1 <commit> 3. add attribute 1 and associated constraints <commit> etc. In the case of e.g. start/end constraint, we were trying to add the constraint before addition of the constrained attribute (e.g. add constraint on 'start' referencing 'end', but only 'start' has been added yet, not 'end'). This patch fix this by: * adding the relation type to the schema without having to commit, but keeping the operation to revert the addition if necessary - this allows to a single commit for all attributes of the entity type ; * using a LateOperation on constraint operation, so we ensure attributes are actually added before any constraint is added.

[schema sync] Refactor AfterAddCWRTypeHook and AfterAddCWComputedRTypeHook so the latter inherit from the former and enhance their docstring.

Drop embed-allowed option, gone away for a long time

Never record undo information for session or data import for session, it may causes unlimited database size expansion while it makes no sense to undo this kind of changes. For data import, they are created programatically hence we may not want to undo this either.

[cwvreg] load registry using modules names instead of directories Introspect cubicweb, cubes and apphome using pkgutil to generate the full list of modules names for loading registries. Avoiding using bogus logilab.common.modutils.modpath_from_file().

[schema] load schema from modules names instead of directories Introspect cubicweb, cubes and apphome using pkgutil to generate the full list of modules names for loading the schema. Keep historical behavior and check if source .py file exists if a module is found using python bytecode file (.pyc and .pyo) Loading schema from apphome require apphome to be present in sys.path and that "schema" module resolve to a file located in apphome. Update migraction tests to explicitely update sys.path when loading schema from different apps, use a contextmanager for this so it's more readable. Require updated logilab-common and yams

[devtools/test] Skip qunit tests in case of timeout Follow-up on 87443f279b0f where other similar tests got adjusted not to fail (in particular in our CI environment).

[autoform] Avoid two calls to field.process_form for the same field in some cases when some entity is being created and data include non-inlined relations, the values for this relation are now stored for later usage, avoiding two calls to field's process_form method, which may be unexpected for custom fields. This has been discovered in saem_ref#f5444b1f9770. Reorganize imports in the test along the way.

[pyramid] Don't use unsafe_cnx_context_manager for write queries we may have clumsy error that hide previous problems on attempting to commit the transaction. Closes #16753531

Repair database wrt indexes / unique constraints * recreate indexes dropped by 3.23 migration (but it's still unclear why) * attempt drop remaining extra indexes (there may be a bunch of these on old instances) * warn about missing expected indexes Closes #16666137

Enhance postgres index filter with values from real life This has been found on fairly old instances of ours (cwo/elo). Dunno if we really want this in, it may lead to false negatives.

Extract functions to list expected/found database indexes from check_indexes so they may be used from other contexts. Use consistent naming along the way: always use indexes, not indices.

Simplify and fix _cw.drop_entity_cache * it's never called with an eid as argument, beside in a useless case in test (removed) * the only place where it's called from outside the tests is in full-text reindexation in server.checkintegrity: we could removed the request implementation and move it in unittest_rset, byt I decided to keep it for consistency with all other entity cache handling methods * get back a fix from Julien Cristau for the connection's implementation, quoting is commit message: When removing an entity from the transaction's cache, clear the entity's own cache May avoid issues where an entity object is still accessible somewhere else (e.g. an operation) after dropping it from the transaction's cache, with a stale attribute or relation cache.

[entity/optimization] Cache rset when entity.related is called with entities=False If fail to see why we couldn't cache in this case, while this is important for optimization reason: when doing a single HTTP request, some queries are done two or three times because of predicates / uicfg or alike. Also always store tuple and not list in the cache, because: * else we get some regression * and inconsistent result type (tuple or list) * and it simply feels better to cache an unmutable object.

Ensure in 3.24 migration that entities from LDAP have consistent cwuri Closes #16666157

[doc/book] fix error in markup

[rtags] Allow to 'derive' rtags Since some releases, rtags (structure underlying uicfg) have selector and may be copied using something like: new_rtags = deepcopy(original_rtags) new_rtags.__module__ = __name__ new_rtags.__select__ = custom_selector The problem is that starting from that, both rtags wil diverge and changes in original_rtags won't be considered, while we usually want to set a few specific rules only in new_rtags. To fix this problem, this cset introduces the notion of "derivated/parent" rtag, eg: new_rtags = original_rtags.derive(__name__, custom_selector) Beside easier copying, when using the above method changes in original_rtags which are not overriden by new_rtags will be considered since it only hold its specific rules but look among its parent chain for non-found keys. Along the way, flake8 unittest_rtags. Closes #16164880

[test] Use bare unittest in unittest_rtags

[rtags] Docstring fix

[cleanup] flake8 rtags.py

[pkg] Require passlib >= 1.7.0 Since bc9d901cb9e6 we use 1.7.0 API, update packaging accordingly.

[web/http_headers] do not crash when IfModifiedSince is empty string (closes #16527954)

[pkg] Set an upper bound to passlib We're using a deprecated API and are getting the following deprecation warning: :: the method passlib.context.CryptContext.encrypt() is deprecated as of Passlib 1.7, and will be removed in Passlib 2.0, use CryptContext.hash() instead. So let's make sure we're getting a working library. [ci skip]

[cwconfig] Retrieve real path of cubes directory Since logilab-common 1.3.0, we have real path expansion modutils.modpath_from_file. But the extrapath parameter that is passed from cubicweb.schema reader may still have symlinks, so expand them here to hopefully have consistent comparison in modpath_from_file.

[server] Use CryptContext's hash method instead of deprecated encrypt method Getting rid of the following deprecation warning: :: the method passlib.context.CryptContext.encrypt() is deprecated as of Passlib 1.7, and will be removed in Passlib 2.0, use CryptContext.hash() instead.

Merge public heads

[cwctl] on upgrade, clear instance_md5_version cache The generate-static-datadir depends on a correct config.instance_md5_version(), and we just invalidated it by upgrading the instance.

[twisted] add request error handler to avoid finishing it twice And avoid stack traces like:: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 783, in __bootstrap self.__bootstrap_inner() File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 763, in run self.__target(*self.__args, **self.__kwargs) --- <exception caught here> --- [...] File "/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 196, in _worker result = context.call(ctx, function, *args, **kwargs) File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext return func(*args,**kw) File "/home/me/envs/grshell-cw/cubicweb/statsd_logger.py", line 121, in __call__ return self.callable(*args, **kw) File "/home/me/envs/grshell-cw/cubicweb/etwist/server.py", line 131, in render_request code=500, twisted_request=request) File "/home/me/envs/grshell-cw/cubicweb/etwist/http.py", line 22, in __init__ self._finalize() File "/home/me/envs/grshell-cw/cubicweb/etwist/http.py", line 46, in _finalize self._twreq.finish() File "/usr/lib/python2.7/dist-packages/twisted/web/server.py", line 228, in finish return http.Request.finish(self) File "/usr/lib/python2.7/dist-packages/twisted/web/http.py", line 931, in finish "Request.finish called on a request after its connection was lost; " exceptions.RuntimeError: Request.finish called on a request after its connection was lost; use Request.notifyFinish to keep track of this.

[web/tests] Hide DeprecationWarnings

[web] remove next_tabindex() It's hard to see it work reliably in view of multiple server processes. If something like that is needed it should probably be built on the client (js) side.

[views/optimization] Ensure we call rset.possible_actions with the same argument rset.possible_actions (which should definitly not be an ResultSet method, but that's another story) has been designed to hold a cache to compute possible actions for a only once, since this may be a fairly costly operation (notably because of the 'has_editable_relations' predicates). But this has been broken by introduction of a new 'view' parameter which is not given by every call. To fix this, this cset adds the missing view argument where necessary and reimplements the rset's method to assert it's always called with the same key. Unfortunatly, those changes have to be ported to squareui and bootstrap cubes as well.

[schema/optimization] Ensure read permissions are deactivated when we're checking a permission When called from a web request, since we *are* checking some permission, read permissions don't have to be introduced in this query. We may avoid that since there now more any differences between web and repo connections, so let's do it.

[web] Enhance query log file * Add an overall page generation time * Don't log URL without any query (e.g. static files)

[massive store] Don't store eids_seq_range as a store attribute since it's not considered after object's initialization.

Flake8 bits with some copyright updates

[test] Use plain unittest in unittest_views_forms.py

Some py3k related fixes: use text_type instead of unicode

[cwctl] Kill deprecated ordered_instances method no need to keep bw compat for this.

[test/cleanup] Use plain unittest

[test/cleanup] flake8 unittest_rqlannotation.py

[debian] logilab >= 1.2.2 in requires (only in build depends) Related to #16404515.

[debian] move unittest2 to requires not recommends closes #16404515

[devtools] named context for boxes subTest for consistency

Added tag 3.24.3, debian/3.24.3-1, centos/3.24.3-1 for changeset bb5904cd284e

[pkg] 3.24.3

[cwconfig] ensure CubicWebNoAppConfiguration returns a typed value (closes #16364459)

[views/optimization] Makes has_editable_relations predicate less costly by using generator The `has_editable_relations` predicate is used to say if the 'modify' action should appear. To do so, it checks if anything is editable (ie. something in the attributes, relations or inlined section of the automatic form). This may be costly since it may have to check several permissions. To optimize this a bit, this cset turns list into generator so that we'll avoid unnecessary work as soon as we find a match.

[cwconfig] Trim down allowed exceptions in gettext language setting I can't see any valid reason to pass on ImportError and AttributeError. In particular, the latter shadowed a Python 3 error until 4f43e64603ef.

[pkg] Configure "universal" wheel build According to https://packaging.python.org/distributing/#wheels, we should create universal wheels (no 2to3 step, no C extension). Configure this in setup.cfg. [ci skip]

[rqlrewrite] Test and fix potential NameError We were referencing a loop variable, which may lead to a name error and show a potential error if there are several matching variables. To avoid this, introduce a list to hold every encountered variable and process all of them later.

flake8 uicfg

[uicfg] Remove unpredictability from autoform_section initialization It was attempting to guess smart default values by looking at the opposite relation (neg_role), but depending on the (random) ordering of the schema, it may or may not have been set yet, leading to unpredictable result with varying hash seed. Remove those fuzzy lookup all at once, it may change some (unreliable) things in your app but at least it's still a bit easier to explain. Closes #16272968

[schema/optimization] Wrap rql expressions into EXISTS node when checking individual permissions Some RQL expression may retrieve several results. It makes sense to wrap them into an EXISTS node to optimized things a bit. This is already done by security insertion of 'read' rql expressions.

[views/optimization] Prefetch state'names and comment format in wfhistory component This will avoid later queries to retrieve them.

[devtools] Make timeout error in qunit tests configurable and use SkipTest in test_jscript.py The reason is that the latter tests quite often fail in our CI environment, possibly due to concurrent execution of the browser, and this makes it hard to identify a truly failing build there. So skip tests with a timeout error.

Added tag 3.24.2, debian/3.24.2-1, centos/3.24.2-1 for changeset 296077513782 And drop erroneous 0.34.2 tag.

Added tag 0.34.2, debian/0.34.2-1, centos/0.34.2-1 for changeset 296077513782

[pkg] Version 3.24.2

Merge with 3.23 head

[migration] Fix 3.23.0 migration script, broken by a703f00718c2 * first query use "IN" where it should use "=" -> crash * second query is missing an interpolation -> crash * third query is attempting to remove constraint that have already been removed by the second query -> crash

[doc] Add 3.24 release date and reference in index pages [ci skip]

Closing "oldstable" branch We now release-based branch naming.

Closing "stable" branch We now release-based branch naming.

Merge 3.24 branch

[pkg] Bump version to 3.25.0.dev0

Added tag 3.24.1, debian/3.24.1-1, centos/3.24.1-1 for changeset da52fda6f15b

[debian] update changelog's time tag

[pyramid] Drop reference to pyramid_cubicweb in ctl command docstring [ci skip]

[debian] Rework split of cubicweb-ctl package It does not appear simple to have the .install files work. So just copy cubicweb-ctl script in DESDIR of cubicweb-ctl binary package and remove .install files. Related to #16133259.

[skeleton] Rephrase long description of Debian package into something meaningful [ci skip]

[skeleton] Depends on python-cubicweb in debian packaging [ci skip]

[debian] Have a single python-cubicweb binary for the library But keep distinct meta-packages for dependencies of the HTTP server (Twisted/Pyramid) and database backend. Related to #16133259.

[debian] Update debian packaging (closes #16133259) - use dh_python, pybuild and debhelper>=9, - refactor and simplify the debian/rules, - rename binary packages (but cubicweb-ctl) to python-xxx - remove daemon handling stuff (initscripts...) from cubicweb-ctl (one should now use a standard WSGI delivery method),

[pkg] Version 3.24.1

[tox] Use `python -m check_manifest` instead of check-manifest program To make sure we use Python from tox's virtualenv.

add debian buildpackage-generated files to hgignore

[test] Fix language negotiation test At least when executed alone (there seems to be a test isolation problem which I could not figure out).

[pyramid] No more need for an internal connection when user is cached

[web] req.lang should be None or a non-empty string

[web] Simplify a bit language handling Drop `set_default_language` and `set_user_language` by considering that: * before being bound to the user, request will have the site's default language * cnx has the user's preferred language, hence get it back to the request when it's bound to a connection The first point requires some change to cubicweb's vreg faking so it doesn't break. That should be enough.

Fix AttributeError for "lang" on repo/client connections This is a regression introduced by b48020a80dc3, which removed call to set_language from the `req._set_user` method. This is fine, but we still want a language on connections, so we've to handle it if necesary in Connection.__enter__ (i.e. once connection is properly open). When using pyramid front-end, which has a users cache, we've to cache its language as well because we must not access its preferred_language method since it's not bound to a proper connection.

Alias ugettext to gettext method of cwGNUTranslations in Python 3 The former does not exist in Python 3, but we rely on it. This fixes test_login_bad_password in cubicweb/pyramid/test/test_login.py which has never passed since being introduced in 6392f34fcdad.

[devctl] make i18n custom message extractors work with legacy layouts distname needs to be "cubicweb_<cubename>", even with legacy layouts for pkg.load_entry_point() to work closes #16272177

Fix (new) flake8 errors They showed up on upgrade of flake8/pep8.

[pyramid] Try "eid" first when retrieving a subresource of ETypeResource So that Pyramid traversal works as existing CubicWeb URL publishing.

Fix AttributeError on postcommit when db-create adds non-system CWSource entities In this case, `Repository.add_source` is called from an operation, not from `Repository.init_sources_from_database`, thus the attribute `sources_by_eid` does not exist. Closes #16251078.

[devctl] Optimize cleanup_sys_modules a bit by moving out of a loop a costly computation that is constant. This at least benefits the `i18ncube` command. Related to #15789486.

[pyramid] Install file listing monitored files in application home Instead of using a temporary file, that never gets deleted. Closes #16159807.

[pyramid] Fix login error message language Use http negotiation unless language is explicitly set for the site. Closes #16236485.

i18n update Pyramid-related messages were not translated. Closes #16236243.

[pyramid] Add action verb used in some messages displayed by the command For instance we'll see 'instance started' instead of 'instance None', which is nicer.

[pyramid] No more need to check CW version since it's now shipped with it

[pyramid] Fix 404 handling Avoid seeing a traceback in the UI by catching it before it reaches pyramid and restore usage of the '404' view. Closes #16159863

Fix broken flake8 configuration and flake8 errors which were hidden by this breakage. flake8 --filename options doesn't work as expected: * it's expected to be a shell pattern, using stdlib's fnmatch.fnmatch function internally. This funciton thinks that 'cubicweb/x.py' doesn't match 'cubicweb/x.py' (there must be a reason but that's not the point), hence no file was actually checked ; * as this is a list of pattern, each encountered file is checked against each pattern, leading to run time explosion. So maintain list of files to check in a separated file and give this list to flake8 using unix's xarg command.

[entities] Fix typo in wfobjs debug message

[doc] Add my 3.24 release notes

[migration] Drop cw_schema relation first without this, we ends up with the traceback shown at https://www.cubicweb.org/ticket/16130960. This is not the proper fix, which I have not been able to find. It seems due to this very rare case of deletion of such relation linked to CWRType vs order of execution of operation (in this case, the operation deleting the entity table is run before some other queries using it). As forcing this relation to be deleted before the entity type fixes the problem while this case seems rare enough, IMO this patch is "good enough". Closes #16130960

[hooks] Delete some properties cached on entities schema on schema updates cubicweb.schema add two additional @cachedproperty that should be cleared when in-memory schema is modified. This may leads to e.g. attempt to delete some already dropped relations. Related to #16130960

[doc] Add some words in changelog about pyramid support in core + fix its title Closes #16121152.

[pkg] Fix extra packages not installable For instance `pip install -e ".[pyramid]"` should install the pyramid dependencies. A typo caused all these extra packages to not be found by pip. Closes #16121322.

Added tag 3.24.0, centos/3.24.0-1 for changeset 9f7949b63ab2

[views] simplify and fix the TreeView (closes #16055814) Simplify the (internal) TreeView._init_params() method to get rid of the useless 'initial_thru_ajax' argument. The ajax tree view was not working properly when loaded via ajax, for instance from a lazy tab. This was due to the buggy protection against double javascript/ css inclusion, which is no more needed server-side as it is now implemented client-side. Deprecate use of the initial_thru_ajax argument of the TreeView.call() method.

[massive store] remove unused code

Drop a cw 3.24 warning

Fix unexpected req.relative_path() when calling CWTC.app_handle_request with path argument The _deprecated_path_arg decorator can not be used in this case, since it won't ensure backward compat if path value has been specified and is not similar as req's path (which is generally the case). This problem has been introduced by 904ee9cd0cf9.

[req,web/request] Move language prefix handling into web request This changes part of d8830e2bd2e0 by unsetting "lang" attribute of RequestSessionBase in particular (because it appears to break many downstream applications). The new implementation relies on a new "build_url_path" method on RequestSessionBase that is overridden in _CubicWebRequestBase to handle the language prefix in URL. As a consequence build_url in RequestSessionBase is not aware of this prefix anymore (contrary previous implementation from d8830e2bd2e0). Move respected tests from unittest_req.py into unittest_request.py. Commenting out part of test_handle_request_no_lang_negotiation_fixed_language in unittest_application.py because it now fails because of some obscure bug that cannot be resolved at the moment. Related to #15743487.

[test] Use stdlib unittest instead of lgc.testlib in unittest_request.py

[test] Distinguish "request" tests from accept parser ones in unittest_request.py

[pkg] Add setuptools to RPM build requirements

[pkg] Version 3.24.0

[req,web] Make it possible to handle page language from URL prefix Adding a short language prefix to URL (like "/en" or "/fr") changes the language the pages are displayed in. This prefix is kept during navigation. This way it is not necessary to do language negotiation, nor to use user preferences to determine which language to apply. This behavior is controlled through a new configuration option "language-mode", which replaces "language-negociation" option and which values may be "http-negotiation", "url-prefix" or "" (to disable language setting and force using "ui.language" property). Migration from previous option is not handled because I could not manage to have it working (users will get prompted with the configuration file diff anyways). Add some tests checking various scenarios. Closes #15743487

[config] fix the load_site_cubicweb() method for to 'new-style' cubes (closes #16059402) We first try to load the site_cubicweb module from the cubicweb_<cube> package, and if it fails, revert back to old cube path.

[massive store] remove on_commit / on_rollback parameters No actual use of this was found in client code. If one wants to control this behaviour, he can always override the commit method and add a try / except around flush_entities in its own store

[massive store] master_init should commit by default Default client usage is to create a master store to control slave ones. They need to access cwmassive_initialized from another transaction

[massive store] isolate SQL statements that one may want to customize i.e. provide public methods for meta data insertions

[massive store] Store entities in temporary table as well * on some entity type is encountered by a slave, create a dedicated table for insertion of entities of this type by this slave, similarly to what is done for relation - this should lower changes of conflicts in master/slaves mode ; * delay drop of constraints and indexes to `finish` method, where copy from temporary tables to regular table is done ; * insertion of metadata is done by scanning temporary tables, which may be way shorter than theier associated regular table ; * drop drop_metadata_constraints with its constraint_dropped friend attribute, there are no more necessary since this is done once in the `finish`. Related to #15538303

[massive store] Drop useless check for empty buffer If something went wrong, an error will be raised.

[massive store] Docstring / comment cleanups

[massive store] Turn a runtime error into an assertion geting there is a bug.

[massive store] Delay metadata attributes and default values handling to flush_entities This may be done once for all there, instead of for each entity in prepare_insert_entity.

[massive store] Use a slave specific table for relation insertion in the massive store * attribute a random id to the store * add a column containing this id in the cwmassive_initialized table * separate generic `master_init` which create the cwmassive_initialized table from rtype specific initialization which must be done in each slave now * delay removal of table indexes to `finish` * get back relations from each table on `finish` Similar work still has to be done for entities insertion - which will allow more cleanup to the constraints handling which is still rough for now. Related to #15538303

[massive store] docstring and __init__ cleanup

[massive store] Drop dead code This function is never used

[massive store] Reintroduce methods that are necessary to properly handle master/slave configuration Related to #15538303

[massive store] Follow configuration of the metadata generator Don't drop constraints and indexes for tables that are ignored by the metadata generator given to the store. One may now easily disable insertion of e.g. created_by / owned_by by removing them from the MetadataGenerator.META_RELATIONS set, in which case indexes for associated table won't be removed by the massive store.

[dataimport] make MetadataGenerator.META_RELATIONS customizable This should be done on the instance rather than on the class

[massive store] Rework constraint/index handling The basic idea is to keep the primary constraint on entities.eid since it's heavily used in metadata insertions. Other option would have been to drop / recreate but its very costly on big database, and the index is used for insertion into the entities table itself, so it's not worth droping it at a first glance. Also, keeping it avoids to systematically drop all constraints which depends on it. We may thus now lazily drop constraints, only on insertion of some etype/rtype for the related table. Related to #15538359

[massive store] Lazy removal of constraints and metadata indexes They should not be removed during store's init, because we may want to query the database with its index between store creation and call to prepare_insert* (e.g. to build the extid2eid map). Along the way: * rename drop_metadata_constraints into drop_metadata_indexes, because that's what it does * rework a bit impacted tests Closes #15538359

[repository] Drop the index on entities.type It should be now only used in maintenance queries in checkintegrity. Those are not worth maintaining it. Closes #15538317

[source] Drop source mapping handling It was only used by the cwxmlparser which has been deleted. This is too complex for litle benefit, we don't want to maintain that in cubicweb.

[repository] Drop the entities.extid column and associated cache This was not necessary anymore with promoted usage of the new data import API. Turn repository's _type_extid_cache to _type_cache with only the entity's type as key. This introduces an backward incompatible change: entity_metas dict doesn't contains anymore the extid key, but it doesn't seem used at all anywhere, so this sounds acceptable. Closes #15538317

[repository] Drop the entities.asource column It not used anymore thanks to previous patches. Closes #15538288

[integrity check] Stop using entities.asource column in integrity checks Simply check if entities have a cw_source, and link it to the system source when not found. Related to #15538288

[rql2sql/test] Ensure test results predictability Without this, test were randomly failing within tox due to different order of solutions.

[rql2sql] Stop generating SQL query from RQL using entities.type This was necessary in the "true multi-sources" time, it's not anymore, while maintaining this index is costly. Related to #15538317

[test] Remove some usages of lgc.testlib

[test] Avoid pytest discovery warnings Having TestServerConfiguration and alike in the test module namespace causes pytest discovery errors, because it thinks it's a test class.

flake8 and all * update some copyright * drop most __docformat__ declaration * fix some flake8 warnings / errors

[schema2sql] Drop deprecated comment

[cnx] Use entity_type instead of entity_metas()['type'] The latter is deprecated in favor of the former which has been introduced on cnx to replace it.

[entity] Reimplement and deprecate entity.cw_metainformation It shouldn't rely anymore on cnx.entity_metas which is itself deprecated, and the cw_metainformation API is similarly not needed anymore, access directly to cwuri / cw_source if needed. Related to #15538288

[repository] Drop type_and_source_from_eid and rename related cache We don't want to handle anymore the entities.asource column so we shouldn't use those anymore. Also rename repository's _type_source_cache into _type_extid_cache as this is what it's containing now. Do similar renaming to the system source API. Related to #15538288

[entity] Stop returning source information in cw_metainformation() We should not need that. It has been introduced to handle things we should not do and probably don't do anymore (e.g. attempting to grasp recursive import of several sites). Related to #15538288.

[json] Stop serializing cw_source into default json representation of an entity Related to #15538288

Drop deprecated LDAP related script both are relying on the 'entities.source' column which has been dropped in 3.19. They have been written with the old ldapsource in mind, which has been dropped at that time.

[entity] Stop linking to external site for external entities This behaviour was ok when we had "true" multi-sources but its rather painful with datafeed sources. Also, it makes absolute_url() costlier than what it should. Besides, it relies on cw_metainformation()['source'] that is pending for removal. Instead, add a link to the original object in the metadata view (the one that displays eid and source at the bottom right corner of the primary view). Related to #15538288.

[datafeed] Drop entity deletion handling in the default source / parser This should be handled by specific parser to avoid using "entities.asource". Reimplement it in the ldap parser. Related to #15538288

[sources] Drop 'moved_entities' table handling This was clumsy and could be handled using exturi / cwsource, as the ldap source now does.

[repository] Drop deprecated extid2eid API and friends This will break cwxmlparser based sources. They should be rewritten using a specific parser, based on xml representation or on rqlio. This is harsh but allows a so big cleanup of the code base. Furthermore, it's necessary for asource/extid handling in the entities table which is costly for most app that don't care at all about that... In this cset, delete: * all extid2eid methods * repo._extid_cache handling * [before/after]_entity_insertion source callback * the cwxmlparser and update related tests, notably unittest_datafeed where 'repull' testing has been removed, since it's now handled by the dataimport API and should not be retested there. Related to #15538288 Closes #15538383

[ldap] Stop using entities table in ldap source authentication and parser We may used cwuri for the same purpose, and do one more step towards deletion of entities.extid column. Related to #15538288

[test] Use stdlib unittest in unittest_webconfig.py

[tox] Require ldap3 < 2 for server tests Apparently we are using some constants that got dropped/moved in ldap3 >= 2.0.0 (which came out last week). So stay with version 1.4 until this gets sorted out.

[setup] fix CWDevelop to keep compat with python 2.7 in Python, 2.7, setuptools.command.develop.develop is an old style class, thus it cannot be called with super.

[devtools] Make dependency on backports.tempfile (Python2) optional The package may not be available in all systems (e.g. no Debian package exist at the moment), and we should not crash with ImportError when importing testlib from client code. Follow up on a6dc650bc230 where the dependency was introduced.

[sources] Drop source support_entity / support_relation source API this was used in the pre-datafeed area, it's not worth it anymore. Its only valid usage was in authentication to detect if the source was supporting CWUser, hence we now call it systematically and catch NotImplementedError.

[dataimport] Add a new NullStore This is useful to do some import tests where you only want to test steps prior to the database insertion, as well as "dry-run" commands. Use it as a documenting base class.

[devtools] Properly reraise the exception with previous implementation, the original exception wasn't visible in python 2.

[pkg] Do not install sql scripts in <prefix>/share/cubicweb/schemas These are already site-packages along with cubicweb package.

[devtools/qunit] Retain firefox process stdout to eventually print it The bare RuntimeError that gets emitted upon timeout does not help much to diagnose the problem. Let's see if we can get more information from stdout by not dropping it.

[devtools/qunit] Turn FirefoxHelper into a context manager So that stop() method always gets called. Also ensure temporary directory created on __init__ gets deleted as well upon exit.

[test] Replace logilab-common's with_tempdir by tempfile.TemporaryDirectory logilab-common's with_tempdir decorator, which is considered harmful (see https://www.logilab.org/ticket/8267966). Rely on backports.tempfile to provide TemporaryDirectory on Python 2. Re-export it in testlib module for convenience.

[test] Use `with open` syntax in cwconfig "prefix" tests

[test] Use stdlib unittest.main() in qunit

[web] Clear "pending_others" session key after all relations got processed The RelationField appends data to "pending_others" key while processing its posted information, but this key may already have been removed by the edit controller. So remove the key from session data only when all "others" relations have been processed using a while loop. A test case would be nice, but I could not build one easily unfortunately. Closes #4354551.

[datafeed] Complete the import log even if parser could not be found It happens that if the parser could not be found, _pull_data() would just return an empty dict without taking care to close the import log which it just opened. This leads to misleading information in the user interface where CWDataImport entities kept accumulating in the "imports" tab of CWSource primary view without anything else happening. So: * log an error message when parser cannot be found * always close (write logs and set "end_timestamp" attribute) import log when leaving _pull_data(). Closes #15505460.

[twconfig] Set default value for "interface" to 0.0.0.0 This value means "all IP addresses on this host", which seems to fit with "default to everywhere" that's documented. Closes #15135610.

[session] Allow self._open to be False in __enter__ I do not understand why this assertion was there in the first place. And currently, it makes running any migration impossible. Hopefully, someone may be able to explain. See also commit b48020a80dc3 which mentions this assertion and suggests to remove it.

[pkg] Warn about "develop" command not installing legacy cubes directory Not sure we can do better than a warning, which is better than nothing anyways. Closes #15785635.

[pkg] Cleanup MANIFEST.in from non-matching entries This cleans up output of setup.py executions.

[devtools] make i18ncube customizable in a cube closes #15613724

[rql2sql] Drop deprecated __docformat__ it causes flake8 error on some of its versions.

[rql2sql] Handle comparison of eid on variables from an subquery We don't want to force using 'identity' in this case which may be easily handled by detecting variable is a `ColumnAlias` and relation is 'eid'. Closes #15393583

Take care that cnx may not be set in error view