[pkg] Version 3.25.4

[pyramid] Let logging be configured through .ini file We do not initialize logging in CubicWebPyramidConfiguration thus preventing logging to be started with cubicweb's configuration. On the other hand, any logging configuration defined in ``development.ini`` file will be loaded by whatever start the instance using this file (i.e. pserve, gunicorn, etc.). The benefit is that logging can now be configured per "qualified name" in this file (i.e. one can easily set the DEBUG level for the application cube while keeping all dependencies' level to WARNING). In the development.ini template, we add logger configurations for "logilab" and "cubicweb" qualified names (along with those of the application cube).

Disable pytest's logs reporting CubicWeb already prints captured logs in stderr upon test failure, so we get them twice. Until we fix our logging management, disabling pytest capture and restore brings the output level back to sanity.

[pyramid] Do not issue security warnings in test mode When some session or authtk secret is missing in Pyramid settings, scary "!! SECURITY WARNING !!" are issued. This is arguably pointless in tests. So disable them in this case.

[ssplanner] Prevent execution of write queries involving computed relations Previously, setting a computed relation upon entity creation or modification (using the ORM or an RQL query) would usually fail with an operational error in the backend ("no such table"). However, under some mysterious circumstances (like passing a string as value in cw_set for a computed relation), the RQL to SQL transformation would simply drop the clause. To prevent this to happen, we add a check for computed relation before adding a relation to an execution plan. This check raises a QueryError. It happens in several places: * in querier.InsertPlan.add_relation_def() (called from several places in ssplanner steps) for INSERT queries, * in ssplanner.UpdateStep.execute() for SET queries and, * in ssplanner.SSplanner.build_delete_plan() for DELETE queries. Tests added to unittest_querier.py because unittest_sslplanner.py looked inappropriate (it has only unit tests) and the former already had a NonRegressionTC class.

[querier] Expand 'rdef' argument in InsertPlan.add_relation_def() Expand the 'rdef' tuple to improve readability. The 'rtype' value is not used at the moment, but will in the next changeset.

[server] Use dict.setdefault() instead of if/else in repo.glob_add_relations()

[cleanup] Fix some flake8 errors

[test] Stop using lgc.testlib in unittest_migractions

[test] Drop no more used "maxeid" based deletion in BaseQuerierTC and derived tests This is probably only necessary for QuerierTC itself, move it there and drop incantation from other derived classes to deactivate this feature.

[rql2sql/test] Remove dead setUpModule tearDownModule they are overwritten a few lines below , hence this code is dead and not necessary for test success, drop it.

[rql2sql/test] Move base class from repotest to unittest_rql2sql It's only used there, so move it closer to its usage. Along the way, stop using lgc.testlib.

[rql2sql/test] Set maxDiff to None to see full diffs this is usually useful to compare SQL strings

[rql2sql] Use string_type instead of unicode for py3k support

[gcdebug] Only ignore weakref.WeakKeyDictionary class on Python 2 From Python 3.5, this class fails isinstance check with the following error: :: cls = <class 'weakref.WeakKeyDictionary'>, instance = <functools._lru_list_elem object at 0x7f4331859a48> def __instancecheck__(cls, instance): """Override for isinstance(instance, cls).""" # Inline the cache checking > subclass = instance.__class__ E AssertionError: [<class 'AttributeError'> in gc] 'functools._lru_list_elem' object has no attribute '__class__' /usr/lib/python3.5/abc.py:181: AssertionError I have no clue why this happens, but it makes cubicweb.web.test.test_views.AutomaticWebTest.test_startup_views fail on Python 3.5. So only consider this class for Python 2.

[wsgi] Convert "code" passed to WSGIResponse as integer WSGIResponse is passed a Request.status_out in CubicWebWSGIApplication._render(), which, starting from Python 3.5, is an instance of HTTPStatus. However, webtest still expects a 3-digits value as status string. Fortunately, calling int() on an HTTPStatus works.

[test] Add a "version" to cubicweb_mycube/__pkginfo__.py Starting from Python 3.5, unittest_cwctl.py::CubicWebCtlTC.test_list() would fail without this.

[tox] Use py3 environment instead of py34 The former would select any Python 3 interpreter.

[querier] Allow usage of computed relation into WHERE clause of write queries This used to crash because of an attempt to access unexisting table. Closes #17113286

[cleanup] Fix some flake8 errors

[tox] Bump flake8 version to 3.5 Previous patches fixed new errors raised in this version.

[pyramid] Get rid of a couple of bare "except"

[pyramid] Get rid of a bare "except" in UpdateLoginTimeAuthenticationPolicy Also refactor a bit the method to restrict the try/except block to the code for which we actually want to catch an exception (i.e. the cnx.execute() + cnx.commit()).

[web] Avoid bare "except" in ConcatFilesHandler.concat_cached_filepath()

[test] Use ignore_errors=True in shutil.rmtree instead of try/except/pass

[server] Get rid of a bare except in sqlutils

[server] Do not use logilab.common.date.strptime in sqlutils This function is aliased to datetime.datetime.strptime() and exists only for compatibility with Python <= 2.5 which we no longer support.

[dataimport/test] Ignore flake8 error E741 in test_pgstore.py This error is about "l" being an ambiguous variable name. Here, it's arguably acceptable.

[test] Avoid ambiguous variable name "l" in unittest_req.py This is an error (E741) from flake8 3.5.

[tox] Bound flake8 version to >=3.4, <3.5 The code base is not clean with 3.5, so keep using 3.4 in *stable* branches.

[pyramid] set waitress url_scheme to https when base-url scheme is https See https://docs.pylonsproject.org/projects/waitress/en/latest/#using-behind-a-reverse-proxy Since we do not configure trusted_proxy waitress ignore X-Forwarded-Proto header and may generate http urls (pyramid request.url) when the instance is behind a https reverse proxy. This cause cubicweb-signredrequest to not work with since it rely on the url (including scheme). Set url_scheme to 'https' when CubicWeb base-url scheme is https as a workaround.

Added tag 3.25.3, centos/3.25.3-1, debian/3.25.3-1 for changeset d238badfc268

[pkg] version 3.25.3

[uicfg] Fix autoform_section rtags wrt derivation cw 3.25 introduced the 'derived rtags' feature, but missed that `autoform_section` implementation was overriding several methods where this was implemented in the base class. The following problems are tested and fixed: * during init, we should not attempt to call `_initfunc_step2` if there is some parent, * but we should handle expansion of wildcard ('*' used as subject or object of a tag) since this is handled during init step for this class, unlike others which handle this in `get` method (i.e. at lookup time), * fix overrided `get` method to consider parent rtag if any. Closes #17107020

[rtags] Enhance repr of rtags * we don't care of the class, the registry id is enough (never seen a case of sub-classing) * add the module from which the rtag is coming as well as parent rtag if any.

[web/test] Use bare unittest main instead of lgc.main

[cleanup] Fix some flake8 errors and drop upper bound of copyright along the way. autoform module is not added to flake8-ok-files since there are some remaining errors to be handled, but that's still a start.

[doc] Update the advanced tutorial to follow the new style layout of cubes among others.

[views] Respond with 400 Bad Request in case of validation error in ajax request Previously, we raised a plain RemoteCallFailed without a status code, in which case it defaults to "500 Internal Server Error". Now, for validation errors, we issue a 400 Bad Request since these are clearly client errors.

[web] fix session /cnx mix in anonymized_request

[test] Pin some test dependencies We currently have CI failures because cubes used as test dependencies have been updated to new-style cube layout. To avoid this, pin them to previous released. Those dependencies should be removed but in the mean time this should be enough (and backported in all active branches).

[test] Pin some test dependencies We currently have CI failures because cubes used as test dependencies have been updated to new-style cube layout. To avoid this, pin them to previous released. Those dependencies should be removed but in the mean time this should be enough (and backported in all active branches).

[web/form] Relax condition for non-translating field names we should test true condition instead of None, so if one raise a ValidationError using an empty string as 'no-field' marker instead of None, we don't get the PO file header that gettext likes to return when one attempt to translate an empty string.

Merge with 3.25 branch

[entities] Restore dc_ methods instead of __getattr__ proxy In 8de62610cea2, a __getattr__ proxy was introduced to handle compatibility with dc_* methods being called on entity instances instead of through the IDublinCore adapter. Unfortunately, __getattr__ does not play well with super() and since the latter is often used by, we here restore all dc_ methods with an explicit proxy to the adapter. More verbose but should work. Closes #17103999.

[pkg] add missing unittest2 dependency to spec file

Added tag 3.25.2, debian/3.25.2-1, centos/3.25.2-1 for changeset 5010381099f1

[pkg] Prepare version 3.25.2

[cubicweb/pyramid] take into account two ignored CORS configuration items from all-in-one.conf

[cwctl] upgrade: do delete the static data dir if verbosity is 0 (closes #17069749)

[doc] Fix option_*() migration commands in doc The function signature in the doc is not the same as in the code. Also option_removed should probably be removed from the doc since the function in migration.py is empty

[massive store] ensure the cwmassive_initialized table is deleted if needed This may be necessary if a previous import did badly crached, leaving the cwmassive_initialized behind.

[server] fix deserialize_schema to handle properly pg schema (closes 17076486) Postgres behaviour is to look first for a table in the configured schema then fall back to 'public' if no table is found. This must be implemented in deserialize_schema when listing existing tables to check for computed relations (cw_cwcomputedrtype) otherwise computed relations may not work when using non-default pg schema.

[server/utils] update passlib API to 1.7 (closes #17054805) while keeping bw-compat with 1.6 - 2.7 is not available for all distributions, e.g. jessie. The dependancy has been bumped in 6186cd15a46d because of a change in bc9d901cb9e6. We may easily use the appropriate method depending on the version installed.

Merge with 3.25 branch

Added tag 3.25.1, debian/3.25.1-1, centos/3.25.1-1 for changeset 5fe62978801a

[pkg] Version 3.25.1

[facets] Fix disappearance of navtop component on facet filtering which is because facets are replacing the whole #pageContent div, while this one contains other stuff than the view: * a type selector component that should be dropped for a while, * a computed title, * the page navigation. Then the view content itself is in a #contentmain div. The thing is that the navigation should be rebuilded on filtering (this is not the case for other bullets in the list above). This is currently handled specifically in the ajaxcontroller (except for the type selector which will disappear... who said it should be dropped at once?). So to fix this we: * put the page navigation into the "contentmain" div * don't replace anymore "pageContent" but "contentmain" After that we can even remove from the ajax controller the code that reimplements title handling similarly to the main template. Notice the part that changes the main template has to be ported to squareui. Closes #17074195

[web] Set response status to 400 when appropriate in ajax controller When this is clearly a client error, set status to 400. Otherwise, keep the default value for RemoteCallFailed (status=500).

[web] Only log exceptions in debug mode in Ajax controllers When we raise a RemoteCallFailed error (erroneously turned into a '500 Internal Server Error' response, but that's another business). In production environment, we do not want to log the exception in most cases where it's actually a client error. So only log the exception in debug mode.

[pyramid] Only expose 'cubicweb.bwcompat' setting for "all-in-one" configuration type The "pyramid" instance configuration does not work with "cubiwceb.bwcompat" mode (on purpose). Yet, having the setting exposed in development.ini file (generate by `cubicweb-ctl create --config pyramid <cube> <instance>` command) is misleading and we want to remove it. Thus, we only query this setting when cubicweb configuration is "all-in-one" and drop the setting line from templated development.ini file. If the option is found and True for any other configuration type, we issue a user warning (and ignore the option).

[migration] Only drop moved table entities if it does not exists For very old instances (namely docaster), this table has never been created.

[web/views] delete: show composite entities in predictible order Iterate over relations type in alphabetical order, so the order is predictible dans does not depend on PYTHONHASHSEED.

[req] fix find() generating non-rewritable rql on non final relations When filtering on a relation, find() was generating rql like 'Any X WHERE X is ETYPE, X relation EID' which work without being rewritten (it should probably not), but when applying some rewrite (eg. permissions) it raise in rqlrewrite code. def _use_orig_term(self, snippet_varname, term): ... > self.rewritten[key] = term.name E AttributeError: 'Constant' object has no attribute 'name' Generate valid rql instead 'Any X WHERE X is ETYPE, X relation Y, Y eid EID'.

[req] raise KeyError instead of AssertionError in req.find()

[req] use format() instead of % substitution For readability, avoid escaping using %% and use format() instead.

[test] make assertions on exception message

[req] clearer exception message Display what attribute trigger the NotImplementedError

[cleanup] Drop useless fake function actually not necessary.

[cleanup] Use plain unittest main in unittest_rqlrewrite

[cleanup] Fix flake8 error unittest_rqlrewrite hard one!

[rqlrewrite] Enhance detection of need for Exists node We actually want one only if this is not a (`And` / `Or`) binary tree of `Not` or `Exists` nodes, so write a recursive function to tell so. Related to #17074119

[rqlrewrite] Fix rewrite on ambiguities introduced by NOT relation or "is IN" type restriction When some inserted RQL snippet generate more solutions than the original RQL, the rewriter attempt to duplicate the snippet for each newly introduced solution. There are though some cases where we do not want this behaviour in case of ambiguities introduced by: * NOT(X relation Y) expression, since it won't be equivalent to NOT(X relation Y1, Y1 is Type1) OR NOT(X relation Y2, Y2 is Type2) ; * EXISTS(X relation Y, Y is IN (Type1, Type2) expression, since it's not actually necessary to split an explicitly introduced ambiguity (and it crash if we attempt to do so, so...). In test, we've to modify the `rewrite()` function because in the newly introduced test we need the same constraint to be applied to two variables in the original query, and this was not supported before. Notice the generated RQL in test is still *NOT CORRECT* "(EXISTS(NOT EXISTS() OR EXISTS(...))", or at least isn't optimal. This will be fixed in a forthcoming changeset. Related to #17074119

[dataimport] Add explanation about why external entities can't be inserted By default after the import processed the importer indicates which external entities can't be inserted because they are missing dependency data (other entities, used in inlined or mandatory relations). It usually helps there to find out which extids / relations are missing, so add this to the log.

[querier] Fix typo in Unauthorized message exception

[massive store] Ensure temporary metadata table get dropped even if there is some integrity errors while inserting entity values or setting back index/constraints, or any other errors. In case of exception, rollback to ensure we're not committing undesired intermediate state.

[test] Stop using lgc.testlib.unittest_main in dataimport's test_sqlgenstore.py

[test/cleanup] Cleanup flake8 errors in dataimport's test_sqlgenstore

[pkg] Fix path error in flake8-ok-files 'sources' is a subpackage of 'server'.

[cleanup] Fix undetected pep8 error

[doc] Set release date of 3.25 in changelog

[cwconfig] load appobjects subpackages recursively This is a regression appeared in 3.25.0. Appobjects (entities, views etc) subpackages should be loaded recursively, this is the historical behavior, relevant code is in logilab.common.registry._toload_info(). To keep historical behavior schema subpackages should not be loaded, relevant code is in yams.reader.get_schema_files(). We may want to have a consistent behavior later on... Closes #17073706.

[pkg] Set version to 3.26.0.dev0

Merge with 3.25 branch

Added tag 3.25.0, debian/3.25.0-1, centos/3.25.0-1 for changeset dacc5b168e29

[pkg] Version 3.25.0

[entities] Fix backward compat of IDublinCore adapter wrt dc_long_title `entity.dc_long_title()` used to fallback to `dc_title()`, and most entity types were relying on this, thus only implementing the later. Since introduction of the IDublinCore adapter, if one call `entity.dc_long_title()` on an entity that only implements `dc_title()`, it will retrieve the adapter which will then call its own `title()` method instead of the `dc_title()` method of the entity as expected. Fix this by calling instead `entity.dc_title()` which will eventually kick in the backward compat layer if necessary.

[web] Do not try to rmtree symlinks in rmtreecontent() There is a symlink in data directory (created by generate_static_dir() method) and trying to rmtree() it will fail with an OSError. So we unlink() it instead.

[pkg] Version 3.25.0rc3

[test] Stop using lgc.testlib

Fix some flake8 errors

[sources] Turn classmethod into standard method No need for a class method and easier to get proper logger including source's name.

[test] Add some tests for hooks.syncsources for parts which are not located in some source's check_config / check_urls method.

[sources] Check source's url attribute value on creation/modification Similarly as for config. Now, ldap source validation is properly done there instead of at initialization time.

[sources] Check sources configuration is fine on creation/modification Reintroduce usage of 'source.check_config' which had almost disappeared, as well as tests for the currently detected errors. Part of the system source specific checking done in syncsources reimplemented in a specific check_config implementation. Tests are dispatched among ldap / datafeed and syncsources tests but are not strictly correctly located (notably syncsources tests behaviour of the native source's check_config). The system source url checking part which disappears from syncsources will be reintroduced in a follow-up.

[test] Fix option name in ldap wrong group test which wasn't actually testing what it expected (and nothing guarantee it does now, but that's another story). Fixing this will avoid failure once we properly check the configuration.

[sources] Enhance prototype of check_conf_dict * mark it private * explicit name * update docstring Private function is still called from syncsources but this will be updated in a follow-up.

[sources] Stop translating validation error this should be done later in the web ui, where lang is properly set.

[sources] Simplify source's init method Only call it when enabled instead of giving a boolean flag indicating whether it is or not (which were not correctly considered).

[server] Deprecate Repository.sources_by_eid It's not used anymore within cubicweb itself.

[server] Add source_by_eid and source_by_uri methods to repository Most of the times we only need to retrieve one source (either by uri or eid) and querying sources_by_eid and sources_by_uri properties on repository just for one item is costly. So these methods query what's needed. We issue a ValueError (instead of KeyError for sources_by_{eid,uri} dict) in case the key is not found.

[hooks] Remove list() around repo.sources_by_uri There's no need to convert it as a list anymore since sources_by_uri is a property and will not be modified.

[server] Inline _entity_update method into init method of AbstractSource This _entity_update method does not make sense now that we do not update source from database information but always build them afresh.

[server] Drop update_config method of source It does not make sense anymore to update the config of a source instance (subclass of cubicweb.server.sources.AbstractSource) now that they are always built from database information (CWSource). In datafeed and ldapfeed, we move all code from "update_config" method in "init" method. This changeset fixes LDAPFeedUserDeletionTC.test_a_filter_inactivate() failure (unittest_ldapsource.py) introduces in previous changeset.

[server] Make "sources_by_uri" and "sources_by_eid" properties of repository I.e. do not populate these dict as repo initialization (bootstrap step) but always use information from database. This is needed because when multiple instances of the same application run, if one instance adds a CWSource the other ones will not see it. In particular, when using a scheduler instance, new CWSource will be added by the web instance and not seen by the scheduler which is supposed to update them. We thus define properties for sources_by_eid and sources_by_uri instead attributes on repository instance. CWSource entities are thus retrieved from database every time these properties are accessed. We factor out initialization of the "source" instance (subclass of cubicweb.server.source.AbstractSource) in a _sources() method. Note that this method takes care of calling "init" method on the source as well as "set_schema" (previously done in repo.set_schema(), which now only touches system_source). Accordingly the "init_sources_from_database" method is dropped along with "add_source"/"remove_source" methods. In syncsources hook, we thus drop: * SourceAddedOp operation which called repo.add_source() so that the SourceAddedHook only cares about checking source configuration now; * SourceRemovedOp and SourceRenamedOp operations for the same reason; * SourceConfigUpdatedOp as updating the live config of source is meaningless once we rely on them being retrieved from the database; * SourceHostConfigUpdatedHook hook which is now useless without call to SourceConfigUpdatedOp; In 3.10 migration script, remove usage of sources_by_uri repo attribute which, unless I'm missing something, appears useless (at least now). In tests: * unittest_datafeed: remove test_update_url method since we dropped respective hook; * unittest_ldapsource: LDAPFeedUserDeletionTC.test_a_filter_inactivate() currently fails because it still relies on live config being updated, this will be fixed in the next changeset once all "live source" logic will be removed.

Add a "Contributing" section to README with patch submission guidelines For the CubicWeb project and its dependencies, we now prefer patches submission and review by email on a public mailing list. We are thus moving away from the previous vcreview-based workflow taking place on the forge. This change is motivated by the following points: - the current reviewer assignment mechanism (pick a random reviewer, rely on reviewer availability rather than on willingness to review, send related patches to distinct people, etc.) is inefficient if not counter-productive; - most of the times, discussion only happens between the patch submitter and a reviewer with no easy way to increase the audience; - cubicweb-vcreview has no concept of patch series; - cubicweb-vcreview is not actively maintained anymore and its usability keeps deteriorating. We expect that email-based submission and review of patches will circumvent these limitations. Anybody interested in the project is welcome to subscribed to the mailing list and participate to the review process. This patch documents the basic workflow of patches submissions by email.

[cwctl] do only clean static data dir content (closes #17069762) deleting the directory itself is useless and may be a problem in automated deployment environments (user may not have permissions to delete or create that directory).

[cwctl] add a no-config-update option to the upgrade command (closes #17069607)

[cwconfig] Add "pyramid" config type to MCOMPAT dict I don't know what this is for exactly, but the accept_mode method below queries this dict with the configuration type name. And if the latter is "pyramid", it crashes since there's no key in MCOMPAT dict. So add one with the same value as "all-in-one". This code is called during "upgrade" migration in particular.

[pyramid] Avoid shutting down the repository at exit if it's already shutting down For install `cubicweb-ctl pyramid` command will explicitly call `repo.shutdown()` so the atexit hook is redundant. Prevent the latter to be executed when the repository is already shutting down (otherwise one gets an assertion error).

[server/test] Extract a "ldapsource" function in unittest_ldapsource.py We will add more use of this pattern in forthcoming changesets, so avoid repetition.

[server] Move "system" source initialization into its own method

[entities] Make CWSource's dictconfig method clearer

[entities] Move CWSource update_config method as a function in 3.15 migration script This script is the only caller of this method. In the code of the function, drop "skip_unknown" argument and continue accordingly upon OptionError.

[server/test] Turn "pull" method of LDAPFeedTestBase into a staticmethod Because it does not use reference to "self" (which should be "cls" for classmethod).

[py3] Use six.text_type instead of unicode in _CWSourceCfgMixIn.update_config()

[server] Drop condition on "3.10" migration in repo.init_sources_from_database() We'll duplicate the code of this method in forthcoming changeset so clean it up a bit of old stuff.

[pkg] Version 3.25.0rc2

[doc] More release note about session data handling mentionning deprecation of _cw.data and removal of get/set_shared_data.

[session] Drop long deprecated [get/set]_shared_data methods they are deprecated since 3.19 and access to the .data attribute itself deprecated in 3.25, so it's time to drop them.

[session] Drop the user session synchronization machinery which should not be necessary anymore since groups and properties are fetched for each request (cache lives in transaction_data).

[session] Avoid deprecation warning on access to Connection.data this attribute has been deprecated during the session related rework, remove warning by explicitly storing data in .transaction_data since session data is not anymore reachable through Connection object. This will allows simplification of session synchronization hooks in a later cset. Also since we do not rely anymore on session data here, we may at some point rewrite those properties to rely on the ORM instead of such custom caching.

[doc] Add release note about change in uicfg

merge 3.25 heads

[hg] Add generated slapd.conf to hgignore

Fix flake8 errors in unittest_cwctl

[cwctl] Drop 'cubicweb_' prefix when displaying cubes with 'cubicweb-ctl list' consistently with newcube / create commands. This is an alternate implementation to backed out changeset fe995d56c949. Closes #17054738

Fix flake8 errors in unittest_cwconfig