[server] Fixed issue with the adapters cache of a cloned EditedEntity The adapters' cache of the entity within an EditedEntity used to be copied along the entity itself during a cloning operation of the EditedEntity. This raises an issue when the adapters' cache is not empty and contains cached values linking to transaction-specific data (or the transaction itself). This prevents the clone EditedEntity being used in a transaction outside the one that created it. This change resets the adapters' cache of the new copy so that no cached value remain. The new copy can then be used in a new transaction afterwards.

[py37] `async` is now a reserved keyword

[skeleton] Use "py3" as tox environment instead of "py34"

[views/startup] drop a duplicate rql query in manage view Counting entities in `nb` is already done above this code, regression introduced by a85db6fa

Depend on cubicweb-file >= 2.0.0 I applied V1 of the patch, but V2 contained this extra hunk...

[test] Update webconfig's tests accounting for cubicweb-file being now a package WebconfigTC.test_locate_all_files and WebconfigTC.test_locate_resource assumed that cubicweb-file was installed in <share>/cubes/file whereas, from the latest release, it is now a standard Python package thus installed in site-packages as cubicweb_file. We simply update base directory of these two tests.

[widgets.js] fix obvious bug detected by sonar

[views/uicfg] obvious bug detected by sonar

[style] Make toolsutils.py flake8-clean Run `autopep8 --max-line-length 100 -i cubicweb/toolsutils.py` and fixed unused imports.

[test/rset] typo

[test/rset] flake8

[py3] import configparser from six.moves instead of using a try/except Using six is better than the try/except because it clearly indicates compatibility code. Also, when using tools such as vermin which relies on AST, it helps getting rid of false compatibility errors.

[doc] didn't had any questions

[doc] missing import, cw now require this explicitely

[doc] add actual link to previous post

[doc] missing import in code example

[doc] add a note because uiprops.py is not generated anymore by default

[doc] recommend pip instead of apt for installing things According to a discussion on XMPP we should recommend pip instead.

[doc] fix, link to internal doc since it's not the blog post anymore

[doc] also show command with sqlite since it's recommended

[doc] fix, the cube name was used, it was the instance named that was needed

[doc] it's a tutorial, not a blog post anymore

[doc] make example more redable

[doc] uses :file:`name` everywhere + indicate file when it's not present

[doc] english typos

[doc] pep8/retab in code examples

[doc] correct import for i18n

[doc] pep8 and retab in tutorial code example

[doc] correct pip command to have needed dependancies to use the start command I know we want to remove this command but in the meantime having this documentation working seems like something good.

[doc] explain the difference between a relation and an attribute

[doc] plural

[doc] misc, put every reference to something in the code between backquotes

[doc] bold to make reading more obvious

[doc] style, switch to bullet points list to make things easier to read

[doc] explains what the <X.Y.Z> means

[doc] command output style has changed

[doc] always put file name for code example and uses :file:`path` syntax everywhere

[doc] explicitely give the local server url to make this step more obvious

[doc] start the server in debug mode, it's easier during learning

[doc] recommend sqlite during tutorial because it's simplier to use

[doc] add a link to every cube documentation page

[doc] make this step more obvious

[doc] move to latest CW version for cube creation

[doc] missng import for i18n in example

[doc] unused import in example

[doc] output has changed

[doc] blanks spaces to allow example to breath and be more lisible

[doc] default tests template has changed

[doc] default generated filename as changed

[doc] rst syntax bug, blank line needed before item list

[doc] add link to cubes contribution document

Merge 3.26

[doc] link to current documentation, not old cubicweb.org doc

Explicitly require repoze.lru for Pyramid tests This used to be pulled by pyramid but the dependency got dropped in pyramid version 1.10, see: https://docs.pylonsproject.org/projects/pyramid/en/latest/whatsnew-1.10.html#backward-incompatibilities We already have an explicit dependency in setup.py, but it was missing for tests. Perhaps we could simply use '.[pyramid]' in this requirements file? Not sure what side effect it'd have though...

[cubicweb-ctl] make uid option obvious

[README] point to xmpp room for contributors

[doc] link to README regarding how to contribute to the code base

[README] improve contribution section with links and command examples

[README] point to logilab's jenkins in tests section

[pyramid] improve pyramid warning with link to doc on how to write pyramid.ini file

[doc] translate french to english

[doc] typo, 'CW' doesn't exist in the example

[doc] always indicate the file in which the code should be put

[doc] respect pep8 in code examples

[doc] serie a small improvments

[doc] typo, the field is "name" not "title"

[doc] fix, command "create" is missing base cube

[doc] improve blog tutorial first steps and explain common errors

[doc] how to install pyramid or twisted application server

[doc] new behavior in virtualenv since 3.26

[skeleton] use python3 flake8 In tox "flake8" environment, use python 3 virtualenv, so we can check for python3 incompatible syntax.

[skeleton] drop whitelist_externals for flake8 whitelist_externals makes sense when used with sitepackages=true. In this case it will always rely on $VIRTUAL_ENV/bin/flake8 so, the whitelist_externals has no effect.

Add changelog note for running tests with pytest-xdist

Make test database template creation concurrent build_db_cache() is used in tests to create test database templates, i.e. DEFAULT_EMPTY_DB_ID (which is __default_empty_db__) and custom template database using CubicwebTC test_db_id/pre_setup_database API. When running tests in parallel using multiple processes, build_db_cache() may try to build the same database twice. Avoid this by adding synchronisation of process by using a file lock. So when two processes require the same template database, one build the database and others wait it to be created. Use filelock (https://github.com/benediktschmitt/py-filelock) library to have a portable (unix / windows) way for handling locks. Also filelock is packaged in debian: https://packages.debian.org/source/python-filelock

Make test database cache lookup concurrent has_cache() is used to dertermine if the template database already exists or not. When running tests accross multiple process, has_cache() wasn't aware of databases created by other processes because it used a cache to generate the cache... Since the discovering discover_cached_db() time is quite reasonable (it's just about listing file in a directory), let re-generate the cache on cache miss.

Fix flake8 issues since release 3.6.0 Flake8 had a new release which raise new issues, namely: W504: line break after binary operator F841: local variable 'ex' is assigned to but never used W605: invalid escape sequence F821: undefined name 'buffer' (noqa seems the only way to avoid this false positive) Also pin flake8>=3.6 in our tests and make explicit that we use python3 to run flake8 tests.

Stabilize flake8 tests on 3.26 branch Use basepython=python2 or flake8 installed with python3 will raise "[F821] undefined name 'buffer'" Pin flake8 < 3.6 which has new checks that doesn't pass on our current code base.

Fix typo in RDFS URI

[doc] Do not rely on lgc's sphinx_ext This module is broken (getting an ImportError) and I don't want to fix it. The only usage we make of it is for the "autodocstring" directive, and I don't know what it does. Based on the context, replacing autodocstring with either automodule or autoclass and removing sphinx_ext from required Sphinx extensions.

Added tag 3.26.7 for changeset ee4ad63c91e5

[pkg] Version 3.26.7

Fix possible double import of cubes For a new-style cube, if we import "cubicweb_<name>" and then import "cubes.<name>", the cube will be imported twice. cubes.<name> and then cubicweb_name is ok though... When using pyramid, we try to find which cube define a "includeme" by importing them with the name "cubes.<name>", so we (possibly ?) double import all new-style cube. This case may also occur in tests with PyramidCWTest. Touching the import loader to fix this scares me, so let's fix this by testing "cubicweb_<name>" *before* "cubes.name" (for old style cubes, importing cubicweb_<name> raise ImportError).

[autoform] Make RelationField.fromcardinality a class method factory Instead of having the "RelationField" class hard-coded in this factory method (thus preventing any subclass to properly use it), we make that method a classmethod and instantiate the actual class instead of the base one.

[autoform] Use fieldclass from kwargs in guess_field for non-final relations This follows up on af969080e7e6 in which a "fieldclass" kwarg has been added in EntityFieldsForm.field_by_name() before calling guess_field(). In this changeset, this value was only used for final relation but the way non-final relation actually also changed since we also use guess_field from these relation (previously, the field class specified through uicfg was instantiated directly in EntityFieldsForm.field_by_name()). So in the non-final case in guess_field() we simply try to retrieve "fieldclass" from kwargs and fall back to RelationField.

[py37] `async` is now a reserved keyword

Merge 3.26

Added tag 3.26.6, debian/3.26.6-1 for changeset 8362503a9248

[pkg] version 3.26.6

[py3] Replace unicode keyword which is unknown in python3

[skeleton] Use "py3" as tox environment instead of "py34"

Added tag 3.26.5 for changeset 76d7b0aa94d2

[pkg] Version 3.26.5

Merge with 3.26

[py3] Use "utf-8" as input encoding for docutils in rest extension Docutils says that "unicode" is an unknown encoding on Python3. Not sure where "unicode" as an encoding comes (it's there since showtime) but "utf-8" seems more appropriate. From now on, rendering of CubicWeb's rst directive (e.g. ..winclude) works on Python 3. Accordingly, we extend wdoc's test to demonstrate this. For this we need to install docutils in tox's web environment.

[py3] Avoid setting attribute on Element instance in wdoc view This change is needed to make "wdoc" view work on Python 3. Indeed, before this patch, we used to set a "parent" attribute on Element instances, which is not permitted since Python 3.3 as xml.etree.ElementTree uses the C implementation by default. To get rid of this "parent" attribute, we rework the XML nodes processing logic to return and retrieve a node along with its parent in several place of the code. Namely, build_toc_index() now accepts the "parent" value for "node" and inserts it in the "index" dict (thus avoid `node.parent = <value>`). Respectively, every query to the "index" dict is updated to also retrieve the "parent" (thus avoiding `<value> = node.parent`). Later in InlineHelpView class, the "index" dict is bound to "tocindex" attribute so we adjust how it is queried in respective methods. Tests cubicweb/web/test/unittest_views_wdoc.py now pass on Python 3.

[test] Add a trivial test for "wdoc" view We just make sure it gets rendered without errors (which wasn't the case before the previous patch). It does not work on Python 3; this will be fixed in next changeset.

[py3] Open rst file in text mode in wdoc view Otherwise, UStringIO complains that it receives bytes.

[ext/rest] Drop "handle_io_errors" argument to docutils' FileInput This argument is deprecated and ignore in docutils since version 0.10. Besides, its default value is None which is the same as we passed.

[wdoc] Remove fallback about xml.etree module Module xml.etree exists since python 2.5.

[wdoc] Avoid usage of deprecated xlm elementree getchildren() method Use list(node) as suggested by the deprecation warning.

[style] Make cubicweb/web/views/wdoc.py flake8-clean

[py3] Use six.text_type instead of unicode in wdoc views

[views/startup] drop a duplicate rql query in manage view Counting entities in `nb` is already done above this code, regression introduced by a85db6fa

Merge with 3.26

Added tag 3.26.4, debian/3.26.4-1 for changeset e1c1af68db86

[pkg] Version 3.26.4

[py3] ensure AjaxController return bytes publish() must return bytes, when the @ajaxfunc decorated method return None, this raised: TypeError: You cannot set Response.body to a text object (use Response.text)

[py3] Let InstanceCommand.run_arg() always return an int The caller of this method (which is method "run_args") does `max(status, self.run_arg(appid))` and run_arg might return None if the underlying <cmdname>_instance method returns None. On python3 max(1, None) breaks with "TypeError: unorderable types: int() > NoneType()". So we enforce run_arg() method to return an integer even if the underlying method returns None. This fixes call of "cubicweb-ctl i18ninstance" with Python3.

[test] Use cubicweb-file < 2.0.0 From version 2.0.0, cubicweb-file moved to new-style layout (Python package) but we still expect it to be an old-style cube in CubicWeb's tests (WebconfigTC in web/test/unittest_webconfig.py).

[py3] Decode subprocess stdout in toolsutils.show_diffs() On Python 3, subprocess' pipes are bytes stream so we must decode them before printing. Without this change, diff of configuration files (e.g. all-in-one.conf as displayed upon instance upgrade) will rendered as a single line on Python 3.

[python3] make sure staticcontroller returns a bytes instance The ``publish()`` method of controller is supposed to return a bytes intsance. Returning a text_type instance will crash later on in the Pyramid response handler when assigning content to ``response.body`` (at least in Python3):: >>> from pyramid.response import Response >>> r = Response() >>> r.body = '' Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/adim/.virtualenvs/…/response.py", line 562, in _body__set raise TypeError(msg) TypeError: You cannot set Response.body to a text object (use Response.text)

[python3] use inspect.getfullargspec instead of inspect.getargspec The latter is now deprecated

[py3] ensure AjaxController return bytes publish() must return bytes, when the @ajaxfunc decorated method return None, this raised: TypeError: You cannot set Response.body to a text object (use Response.text)

[py3] Let InstanceCommand.run_arg() always return an int The caller of this method (which is method "run_args") does `max(status, self.run_arg(appid))` and run_arg might return None if the underlying <cmdname>_instance method returns None. On python3 max(1, None) breaks with "TypeError: unorderable types: int() > NoneType()". So we enforce run_arg() method to return an integer even if the underlying method returns None. This fixes call of "cubicweb-ctl i18ninstance" with Python3.

Depend on cubicweb-file >= 2.0.0 I applied V1 of the patch, but V2 contained this extra hunk...

[test] Update webconfig's tests accounting for cubicweb-file being now a package WebconfigTC.test_locate_all_files and WebconfigTC.test_locate_resource assumed that cubicweb-file was installed in <share>/cubes/file whereas, from the latest release, it is now a standard Python package thus installed in site-packages as cubicweb_file. We simply update base directory of these two tests.

[widgets.js] fix obvious bug detected by sonar

[views/uicfg] obvious bug detected by sonar

[style] Make toolsutils.py flake8-clean Run `autopep8 --max-line-length 100 -i cubicweb/toolsutils.py` and fixed unused imports.

[py3] Decode subprocess stdout in toolsutils.show_diffs() On Python 3, subprocess' pipes are bytes stream so we must decode them before printing. Without this change, diff of configuration files (e.g. all-in-one.conf as displayed upon instance upgrade) will rendered as a single line on Python 3.

[python3] make sure staticcontroller returns a bytes instance The ``publish()`` method of controller is supposed to return a bytes intsance. Returning a text_type instance will crash later on in the Pyramid response handler when assigning content to ``response.body`` (at least in Python3):: >>> from pyramid.response import Response >>> r = Response() >>> r.body = '' Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/adim/.virtualenvs/…/response.py", line 562, in _body__set raise TypeError(msg) TypeError: You cannot set Response.body to a text object (use Response.text)

[python3] use inspect.getfullargspec instead of inspect.getargspec The latter is now deprecated

[utils] Fixes compatiblity of QueryCache with expected dict interface QueryCache is expected to mimick the interface of the dict built-in. The current __iter__ implementation breaks this expectation by iterating over key, value pairs instead of only the keys. This changeset fixes this issue by changing the __iter__ implementation to iterate over the keys in the cache and providing an implementation of the items method with a contract identical to the dict build-in (return a copy of the key- value pairs within the dictionary as a list).

[test/statsd] add tests for cubicweb.statsd_logger

[utils] Implements __iter__ on QueryCache In order to be able to iterate over the currently cached queries (so that they may be inspected and invalidated), this changeset implementes __iter__ on QueryCache.

[utils] Added new get method to QueryCache The new get method enables the retrieval an item in the cache by specifying a default value so that in the absence of an item for the key, the default value is returned.

[test/rset] typo

[utils] Fixed issue in clearing QueryCache When trying to add an item in a full QueryCache, the cache tries to make room. It tries to list and remove non-permanent items with a transient counter (number of times it has been requested). However, there is a pathological case where items could be non-permanent but still not have a transient counter because they were added, but never requested. In some cases, the full cache could be flushed, including the permanent items. This changeset attempts to fix this issue by only dropping the non-permanent items that did not hav a transient counter.

[statsd] add a statsd_timethis context manager This context manager allows to report to statsd the execution time of any piece of code, without the need for it to be a function or a method, like:: def some_function(): do_stuff() with statsd_timethis('some context message'): do_that() ensure_this()

[test/rset] add tests for ResultSet.first() and .last() methods

[test/rset] flake8

[rset] update copyright

[rset] add convenient first() and last() methods on ResultSet These make life easier for the heavy user of the CW shell. Also aliases entities() with all(); so we end with a pretty consistant set: - .one() - .first() - .last() - .all()

[server] Fixed issue with the adapters cache of a cloned EditedEntity The adapters' cache of the entity within an EditedEntity used to be copied along the entity itself during a cloning operation of the EditedEntity. This raises an issue when the adapters' cache is not empty and contains cached values linking to transaction-specific data (or the transaction itself). This prevents the clone EditedEntity being used in a transaction outside the one that created it. This change resets the adapters' cache of the new copy so that no cached value remain. The new copy can then be used in a new transaction afterwards.

[rql2sql] fix handling of the ORDERBY clause when the aggregation function is "embedded" ie. a situation like: Any X GROUPBY X ORDERBY COUNT(Y) WHERE ... works fine since the aggregation function (COUNT) if the first one, whereas a situation like the proposed test: Any X GROUPBY X ORDERBY ABS(COUNT(Y)) WHERE ... failed due to the fact we only checked the top aggratation function (ABS). Closes #17108671

Merge 3.26

[pkg] Set version to 3.27.0.dev0

Added tag 3.26.3, debian/3.26.3-1 for changeset f7067be5f69c

[pkg] Version 3.26.3

[utils/statsd_logger] flake8

[source/native] do NOT do anything with fti if do_fti is False ensure no fti related process is done when fti is disabled on a db source.

[test/web] flake8 unittest_propertysheet.py

[web] fix css file generation permission bug (closes #17143773) PropertySheet.process_resource() uses mkstemp which enforce a 0600 mode on the output file, which is not the file permission we want for css files.

[test/web] add a test for css generation file permission problem (related to #17143773)

[doc] include the 3.26 changelog in changes's index

[server] set PGPASSWORD when running pg commands for backup and restore For db-dump (implied by upgrade) and db-restore with postgres with non-trusted authentication, the db password is asked by pg tools (pg_dump, pg_restore, createdb, dropdb). Avoid this behavior by setting the PGPASSWORD environment variable when database driver is postgres and db-password is set in sources file.

Added tag 3.26.2, centos/3.26.2-1, debian/3.26.2-1 for changeset 9bee3134d304

[pkg] Version 3.26.2

Flake8 cubicweb/web/webctl.py Having the file flake8 clean makes it easier to catch basic programming errors (e.g. name errors).

[web] Add missing import of _DATA_DIR in webctl Followup on d84bc85f7f70 which introduced usage of this name in _datadirs method of GenStaticDataDirMixIn but resulted into a NameError at runtime and makes migrations crash.

[server] unicode → text_type in utils

[views] unicode → six.text_type in views/authentication.py

[pyramid] Recreate CWSession entity when eid no longer exists As noted in comment, it might occur that we receive a "sessioneid" from a valid cookie while respective CWSession got dropped (typical case is db being recreated while users got preserved). In such case, we just recreate the CWSession entity as if it did not exist.

Added tag 3.26.1, debian/3.26.1-1, centos/3.26.1-1 for changeset 027676243aaa

[pkg] Drop non-existant files from MANIFEST.in

[pkg] Version 3.26.1

[test] Drop an unused function in cwconfig tests

[cwconfig] Always use "sys.prefix" to determine installation prefix We drop all legacy logic implemented in _find_prefix() which now happens to be buggy since we cleaned up setup.py in 3.26 (noticeably because virtualenv is no longer properly detected). Closes #17132990.

[cwconfig] Look for migration scripts directory relatively to code Thus getting rid of the last piece of data in "shared" directory. As a consequence, we can drop data_files from __pkginfo__.py. Related to #17132990.

[pkg] Get rid of "pydir" in __pkginfo__.py Nothing uses this.

[pkg] Get rid of "shared" data These are not used anymore per previous changesets. Related to #17132990.

[cwconfig] Drop shared_dir method There's no more users of this method. Related to #17132990.

[config] Avoid relying on shared_dir method when possible From 3.26, some "data" directories (like web/data and i18n or wdoc) are installed along with CubicWeb in site packages. So we can retrieve them relatively to the module that needs them (namely "cubicweb" for "i18n" and "cubicweb.web" for "web/data" and "wdoc"). Reason for doing this is that "share_dir()" is not working when cubicweb is installed in "develop" mode and we are about to drop this method. Related to #17132990.

[test] Add a test for WebConfiguration.locate_all_files() We are going to change the implementation, make sure there's no regression.

[cwconfig] Get rid of _forced_mode global And compute CubicWebNoAppConfiguration.mode within the class namespace. Related to #17132990.

[cwconfig] Get rid of CWDEV This is in preparation of a fix for #17132990 to cleanup legacy (and probably no longer working) behaviors.

[doc] Set release date of 3.26

[web] Fix UnboundLocalError about "exc" in ajax controller Introduced in 159dce89a145.

Link to "3.26" docs in README

Added tag 3.26.0, debian/3.26.0-1, centos/3.26.0-1 for changeset 199851fcddd4

[pkg] Version 3.26.0

[pkg] Version 3.26.0rc0

[schema] Define a decorator to handle methods override There is a number of external classes (from yams/rql) methods overridden "by hand" in cubicweb/schema.py. Define a single decorator to factor out the pattern. It handles specifying the method name (to avoid conflict with the local namespace) and passing the original method to the new definition when needed.

[pkg] Use setuptools.find_packages() We drop the custom get_packages() function in setup.py that used to look for test files (not discovered automatically since test directories are not packages) and replace it by explicit entries in MANIFEST.in.

[pkg] Drop SandboxViolation verification cracking Dunno what this is for. Let's see if this work without.

[pkg] Drop custom "install_data" command Not sure what this is for, the comment mentions "egg" directory, but this is quite likely an old hack that should be unnecessary when using a recent setuptools.

[pkg] Replace custom "scripts" handling by a setuptools entry points http://setuptools.readthedocs.io/en/latest/setuptools.html#automatic-script-creation

Set a default value for arguments of cubicweb.cwctl:run() This will help making a console script entry point from this to get rid of the custom installation of scripts in setup.py. Accordingly, update all calls of this function.

[pkg] Drop custom install_lib command and use include_package_data instead The custom install_lib command appeared to be meant to install package data, as declared in __pkginfo__.py. Let's rely on setuptools' include_package_data[1] option to handle this. [1]: http://setuptools.readthedocs.io/en/latest/setuptools.html#including-data-files

[py3] Replace a unicode() occurrence in workflow views

[py3] Drop a couple of unicode() in devtools.testlib

[py3] Alias long to int in pgstore tests The previous version of handling PY2/PY3 made flake8 complain.

Merge with 3.25

Added tag 3.25.4, centos/3.25.4-1, debian/3.25.4-1 for changeset b8567725c473

[pkg] Version 3.25.4

[pyramid] Let logging be configured through .ini file We do not initialize logging in CubicWebPyramidConfiguration thus preventing logging to be started with cubicweb's configuration. On the other hand, any logging configuration defined in ``development.ini`` file will be loaded by whatever start the instance using this file (i.e. pserve, gunicorn, etc.). The benefit is that logging can now be configured per "qualified name" in this file (i.e. one can easily set the DEBUG level for the application cube while keeping all dependencies' level to WARNING). In the development.ini template, we add logger configurations for "logilab" and "cubicweb" qualified names (along with those of the application cube).

Disable pytest's logs reporting CubicWeb already prints captured logs in stderr upon test failure, so we get them twice. Until we fix our logging management, disabling pytest capture and restore brings the output level back to sanity.

[pyramid] Do not issue security warnings in test mode When some session or authtk secret is missing in Pyramid settings, scary "!! SECURITY WARNING !!" are issued. This is arguably pointless in tests. So disable them in this case.

[ssplanner] Prevent execution of write queries involving computed relations Previously, setting a computed relation upon entity creation or modification (using the ORM or an RQL query) would usually fail with an operational error in the backend ("no such table"). However, under some mysterious circumstances (like passing a string as value in cw_set for a computed relation), the RQL to SQL transformation would simply drop the clause. To prevent this to happen, we add a check for computed relation before adding a relation to an execution plan. This check raises a QueryError. It happens in several places: * in querier.InsertPlan.add_relation_def() (called from several places in ssplanner steps) for INSERT queries, * in ssplanner.UpdateStep.execute() for SET queries and, * in ssplanner.SSplanner.build_delete_plan() for DELETE queries. Tests added to unittest_querier.py because unittest_sslplanner.py looked inappropriate (it has only unit tests) and the former already had a NonRegressionTC class.

[querier] Expand 'rdef' argument in InsertPlan.add_relation_def() Expand the 'rdef' tuple to improve readability. The 'rtype' value is not used at the moment, but will in the next changeset.

[server] Use dict.setdefault() instead of if/else in repo.glob_add_relations()

[cleanup] Fix some flake8 errors

[test] Stop using lgc.testlib in unittest_migractions

[test] Drop no more used "maxeid" based deletion in BaseQuerierTC and derived tests This is probably only necessary for QuerierTC itself, move it there and drop incantation from other derived classes to deactivate this feature.

[rql2sql/test] Remove dead setUpModule tearDownModule they are overwritten a few lines below , hence this code is dead and not necessary for test success, drop it.

[rql2sql/test] Move base class from repotest to unittest_rql2sql It's only used there, so move it closer to its usage. Along the way, stop using lgc.testlib.

[rql2sql/test] Set maxDiff to None to see full diffs this is usually useful to compare SQL strings

[rql2sql] Use string_type instead of unicode for py3k support

[gcdebug] Only ignore weakref.WeakKeyDictionary class on Python 2 From Python 3.5, this class fails isinstance check with the following error: :: cls = <class 'weakref.WeakKeyDictionary'>, instance = <functools._lru_list_elem object at 0x7f4331859a48> def __instancecheck__(cls, instance): """Override for isinstance(instance, cls).""" # Inline the cache checking > subclass = instance.__class__ E AssertionError: [<class 'AttributeError'> in gc] 'functools._lru_list_elem' object has no attribute '__class__' /usr/lib/python3.5/abc.py:181: AssertionError I have no clue why this happens, but it makes cubicweb.web.test.test_views.AutomaticWebTest.test_startup_views fail on Python 3.5. So only consider this class for Python 2.

[wsgi] Convert "code" passed to WSGIResponse as integer WSGIResponse is passed a Request.status_out in CubicWebWSGIApplication._render(), which, starting from Python 3.5, is an instance of HTTPStatus. However, webtest still expects a 3-digits value as status string. Fortunately, calling int() on an HTTPStatus works.

[test] Add a "version" to cubicweb_mycube/__pkginfo__.py Starting from Python 3.5, unittest_cwctl.py::CubicWebCtlTC.test_list() would fail without this.

[tox] Use py3 environment instead of py34 The former would select any Python 3 interpreter.

[querier] Allow usage of computed relation into WHERE clause of write queries This used to crash because of an attempt to access unexisting table. Closes #17113286

[cleanup] Fix some flake8 errors

[tox] Bump flake8 version to 3.5 Previous patches fixed new errors raised in this version.

[pyramid] Get rid of a couple of bare "except"

[pyramid] Get rid of a bare "except" in UpdateLoginTimeAuthenticationPolicy Also refactor a bit the method to restrict the try/except block to the code for which we actually want to catch an exception (i.e. the cnx.execute() + cnx.commit()).

[web] Avoid bare "except" in ConcatFilesHandler.concat_cached_filepath()

[test] Use ignore_errors=True in shutil.rmtree instead of try/except/pass

[server] Get rid of a bare except in sqlutils

[server] Do not use logilab.common.date.strptime in sqlutils This function is aliased to datetime.datetime.strptime() and exists only for compatibility with Python <= 2.5 which we no longer support.

[dataimport/test] Ignore flake8 error E741 in test_pgstore.py This error is about "l" being an ambiguous variable name. Here, it's arguably acceptable.

[test] Avoid ambiguous variable name "l" in unittest_req.py This is an error (E741) from flake8 3.5.

[tox] Bound flake8 version to >=3.4, <3.5 The code base is not clean with 3.5, so keep using 3.4 in *stable* branches.

[pyramid] set waitress url_scheme to https when base-url scheme is https See https://docs.pylonsproject.org/projects/waitress/en/latest/#using-behind-a-reverse-proxy Since we do not configure trusted_proxy waitress ignore X-Forwarded-Proto header and may generate http urls (pyramid request.url) when the instance is behind a https reverse proxy. This cause cubicweb-signredrequest to not work with since it rely on the url (including scheme). Set url_scheme to 'https' when CubicWeb base-url scheme is https as a workaround.

Added tag 3.25.3, centos/3.25.3-1, debian/3.25.3-1 for changeset d238badfc268

[pkg] version 3.25.3

[uicfg] Fix autoform_section rtags wrt derivation cw 3.25 introduced the 'derived rtags' feature, but missed that `autoform_section` implementation was overriding several methods where this was implemented in the base class. The following problems are tested and fixed: * during init, we should not attempt to call `_initfunc_step2` if there is some parent, * but we should handle expansion of wildcard ('*' used as subject or object of a tag) since this is handled during init step for this class, unlike others which handle this in `get` method (i.e. at lookup time), * fix overrided `get` method to consider parent rtag if any. Closes #17107020

[rtags] Enhance repr of rtags * we don't care of the class, the registry id is enough (never seen a case of sub-classing) * add the module from which the rtag is coming as well as parent rtag if any.

[web/test] Use bare unittest main instead of lgc.main

[cleanup] Fix some flake8 errors and drop upper bound of copyright along the way. autoform module is not added to flake8-ok-files since there are some remaining errors to be handled, but that's still a start.

[doc] Update the advanced tutorial to follow the new style layout of cubes among others.

[views] Respond with 400 Bad Request in case of validation error in ajax request Previously, we raised a plain RemoteCallFailed without a status code, in which case it defaults to "500 Internal Server Error". Now, for validation errors, we issue a 400 Bad Request since these are clearly client errors.

[web] fix session /cnx mix in anonymized_request

[test] Pin some test dependencies We currently have CI failures because cubes used as test dependencies have been updated to new-style cube layout. To avoid this, pin them to previous released. Those dependencies should be removed but in the mean time this should be enough (and backported in all active branches).

[test] Pin some test dependencies We currently have CI failures because cubes used as test dependencies have been updated to new-style cube layout. To avoid this, pin them to previous released. Those dependencies should be removed but in the mean time this should be enough (and backported in all active branches).

[web/form] Relax condition for non-translating field names we should test true condition instead of None, so if one raise a ValidationError using an empty string as 'no-field' marker instead of None, we don't get the PO file header that gettext likes to return when one attempt to translate an empty string.

Merge with 3.25 branch

[entities] Restore dc_ methods instead of __getattr__ proxy In 8de62610cea2, a __getattr__ proxy was introduced to handle compatibility with dc_* methods being called on entity instances instead of through the IDublinCore adapter. Unfortunately, __getattr__ does not play well with super() and since the latter is often used by, we here restore all dc_ methods with an explicit proxy to the adapter. More verbose but should work. Closes #17103999.

[pkg] add missing unittest2 dependency to spec file

Added tag 3.25.2, debian/3.25.2-1, centos/3.25.2-1 for changeset 5010381099f1

[pkg] Prepare version 3.25.2

[cubicweb/pyramid] take into account two ignored CORS configuration items from all-in-one.conf

[cwctl] upgrade: do delete the static data dir if verbosity is 0 (closes #17069749)

[doc] Fix option_*() migration commands in doc The function signature in the doc is not the same as in the code. Also option_removed should probably be removed from the doc since the function in migration.py is empty

[massive store] ensure the cwmassive_initialized table is deleted if needed This may be necessary if a previous import did badly crached, leaving the cwmassive_initialized behind.

[server] fix deserialize_schema to handle properly pg schema (closes 17076486) Postgres behaviour is to look first for a table in the configured schema then fall back to 'public' if no table is found. This must be implemented in deserialize_schema when listing existing tables to check for computed relations (cw_cwcomputedrtype) otherwise computed relations may not work when using non-default pg schema.

[server/utils] update passlib API to 1.7 (closes #17054805) while keeping bw-compat with 1.6 - 2.7 is not available for all distributions, e.g. jessie. The dependancy has been bumped in 6186cd15a46d because of a change in bc9d901cb9e6. We may easily use the appropriate method depending on the version installed.

Merge with 3.25 branch

Added tag 3.25.1, debian/3.25.1-1, centos/3.25.1-1 for changeset 5fe62978801a

[pkg] Version 3.25.1

[facets] Fix disappearance of navtop component on facet filtering which is because facets are replacing the whole #pageContent div, while this one contains other stuff than the view: * a type selector component that should be dropped for a while, * a computed title, * the page navigation. Then the view content itself is in a #contentmain div. The thing is that the navigation should be rebuilded on filtering (this is not the case for other bullets in the list above). This is currently handled specifically in the ajaxcontroller (except for the type selector which will disappear... who said it should be dropped at once?). So to fix this we: * put the page navigation into the "contentmain" div * don't replace anymore "pageContent" but "contentmain" After that we can even remove from the ajax controller the code that reimplements title handling similarly to the main template. Notice the part that changes the main template has to be ported to squareui. Closes #17074195

[web] Set response status to 400 when appropriate in ajax controller When this is clearly a client error, set status to 400. Otherwise, keep the default value for RemoteCallFailed (status=500).

[web] Only log exceptions in debug mode in Ajax controllers When we raise a RemoteCallFailed error (erroneously turned into a '500 Internal Server Error' response, but that's another business). In production environment, we do not want to log the exception in most cases where it's actually a client error. So only log the exception in debug mode.

[pyramid] Only expose 'cubicweb.bwcompat' setting for "all-in-one" configuration type The "pyramid" instance configuration does not work with "cubiwceb.bwcompat" mode (on purpose). Yet, having the setting exposed in development.ini file (generate by `cubicweb-ctl create --config pyramid <cube> <instance>` command) is misleading and we want to remove it. Thus, we only query this setting when cubicweb configuration is "all-in-one" and drop the setting line from templated development.ini file. If the option is found and True for any other configuration type, we issue a user warning (and ignore the option).

[migration] Only drop moved table entities if it does not exists For very old instances (namely docaster), this table has never been created.

[web/views] delete: show composite entities in predictible order Iterate over relations type in alphabetical order, so the order is predictible dans does not depend on PYTHONHASHSEED.

[req] fix find() generating non-rewritable rql on non final relations When filtering on a relation, find() was generating rql like 'Any X WHERE X is ETYPE, X relation EID' which work without being rewritten (it should probably not), but when applying some rewrite (eg. permissions) it raise in rqlrewrite code. def _use_orig_term(self, snippet_varname, term): ... > self.rewritten[key] = term.name E AttributeError: 'Constant' object has no attribute 'name' Generate valid rql instead 'Any X WHERE X is ETYPE, X relation Y, Y eid EID'.

[req] raise KeyError instead of AssertionError in req.find()

[req] use format() instead of % substitution For readability, avoid escaping using %% and use format() instead.

[test] make assertions on exception message

[req] clearer exception message Display what attribute trigger the NotImplementedError

[cleanup] Drop useless fake function actually not necessary.

[cleanup] Use plain unittest main in unittest_rqlrewrite

[cleanup] Fix flake8 error unittest_rqlrewrite hard one!

[rqlrewrite] Enhance detection of need for Exists node We actually want one only if this is not a (`And` / `Or`) binary tree of `Not` or `Exists` nodes, so write a recursive function to tell so. Related to #17074119

[rqlrewrite] Fix rewrite on ambiguities introduced by NOT relation or "is IN" type restriction When some inserted RQL snippet generate more solutions than the original RQL, the rewriter attempt to duplicate the snippet for each newly introduced solution. There are though some cases where we do not want this behaviour in case of ambiguities introduced by: * NOT(X relation Y) expression, since it won't be equivalent to NOT(X relation Y1, Y1 is Type1) OR NOT(X relation Y2, Y2 is Type2) ; * EXISTS(X relation Y, Y is IN (Type1, Type2) expression, since it's not actually necessary to split an explicitly introduced ambiguity (and it crash if we attempt to do so, so...). In test, we've to modify the `rewrite()` function because in the newly introduced test we need the same constraint to be applied to two variables in the original query, and this was not supported before. Notice the generated RQL in test is still *NOT CORRECT* "(EXISTS(NOT EXISTS() OR EXISTS(...))", or at least isn't optimal. This will be fixed in a forthcoming changeset. Related to #17074119

[dataimport] Add explanation about why external entities can't be inserted By default after the import processed the importer indicates which external entities can't be inserted because they are missing dependency data (other entities, used in inlined or mandatory relations). It usually helps there to find out which extids / relations are missing, so add this to the log.

[querier] Fix typo in Unauthorized message exception

[massive store] Ensure temporary metadata table get dropped even if there is some integrity errors while inserting entity values or setting back index/constraints, or any other errors. In case of exception, rollback to ensure we're not committing undesired intermediate state.

[test] Stop using lgc.testlib.unittest_main in dataimport's test_sqlgenstore.py

[test/cleanup] Cleanup flake8 errors in dataimport's test_sqlgenstore

[pkg] Fix path error in flake8-ok-files 'sources' is a subpackage of 'server'.

[cleanup] Fix undetected pep8 error

[doc] Set release date of 3.25 in changelog

[cwconfig] load appobjects subpackages recursively This is a regression appeared in 3.25.0. Appobjects (entities, views etc) subpackages should be loaded recursively, this is the historical behavior, relevant code is in logilab.common.registry._toload_info(). To keep historical behavior schema subpackages should not be loaded, relevant code is in yams.reader.get_schema_files(). We may want to have a consistent behavior later on... Closes #17073706.

[pkg] Set version to 3.26.0.dev0

Merge with 3.25 branch

Added tag 3.25.0, debian/3.25.0-1, centos/3.25.0-1 for changeset dacc5b168e29

[pkg] Version 3.25.0

[entities] Fix backward compat of IDublinCore adapter wrt dc_long_title `entity.dc_long_title()` used to fallback to `dc_title()`, and most entity types were relying on this, thus only implementing the later. Since introduction of the IDublinCore adapter, if one call `entity.dc_long_title()` on an entity that only implements `dc_title()`, it will retrieve the adapter which will then call its own `title()` method instead of the `dc_title()` method of the entity as expected. Fix this by calling instead `entity.dc_title()` which will eventually kick in the backward compat layer if necessary.

[web] Do not try to rmtree symlinks in rmtreecontent() There is a symlink in data directory (created by generate_static_dir() method) and trying to rmtree() it will fail with an OSError. So we unlink() it instead.

[pkg] Version 3.25.0rc3

[test] Stop using lgc.testlib

Fix some flake8 errors

[sources] Turn classmethod into standard method No need for a class method and easier to get proper logger including source's name.

[test] Add some tests for hooks.syncsources for parts which are not located in some source's check_config / check_urls method.

[sources] Check source's url attribute value on creation/modification Similarly as for config. Now, ldap source validation is properly done there instead of at initialization time.

[sources] Check sources configuration is fine on creation/modification Reintroduce usage of 'source.check_config' which had almost disappeared, as well as tests for the currently detected errors. Part of the system source specific checking done in syncsources reimplemented in a specific check_config implementation. Tests are dispatched among ldap / datafeed and syncsources tests but are not strictly correctly located (notably syncsources tests behaviour of the native source's check_config). The system source url checking part which disappears from syncsources will be reintroduced in a follow-up.

[test] Fix option name in ldap wrong group test which wasn't actually testing what it expected (and nothing guarantee it does now, but that's another story). Fixing this will avoid failure once we properly check the configuration.

[sources] Enhance prototype of check_conf_dict * mark it private * explicit name * update docstring Private function is still called from syncsources but this will be updated in a follow-up.

[sources] Stop translating validation error this should be done later in the web ui, where lang is properly set.

[sources] Simplify source's init method Only call it when enabled instead of giving a boolean flag indicating whether it is or not (which were not correctly considered).

[server] Deprecate Repository.sources_by_eid It's not used anymore within cubicweb itself.

[server] Add source_by_eid and source_by_uri methods to repository Most of the times we only need to retrieve one source (either by uri or eid) and querying sources_by_eid and sources_by_uri properties on repository just for one item is costly. So these methods query what's needed. We issue a ValueError (instead of KeyError for sources_by_{eid,uri} dict) in case the key is not found.

[hooks] Remove list() around repo.sources_by_uri There's no need to convert it as a list anymore since sources_by_uri is a property and will not be modified.

[server] Inline _entity_update method into init method of AbstractSource This _entity_update method does not make sense now that we do not update source from database information but always build them afresh.

[server] Drop update_config method of source It does not make sense anymore to update the config of a source instance (subclass of cubicweb.server.sources.AbstractSource) now that they are always built from database information (CWSource). In datafeed and ldapfeed, we move all code from "update_config" method in "init" method. This changeset fixes LDAPFeedUserDeletionTC.test_a_filter_inactivate() failure (unittest_ldapsource.py) introduces in previous changeset.

[server] Make "sources_by_uri" and "sources_by_eid" properties of repository I.e. do not populate these dict as repo initialization (bootstrap step) but always use information from database. This is needed because when multiple instances of the same application run, if one instance adds a CWSource the other ones will not see it. In particular, when using a scheduler instance, new CWSource will be added by the web instance and not seen by the scheduler which is supposed to update them. We thus define properties for sources_by_eid and sources_by_uri instead attributes on repository instance. CWSource entities are thus retrieved from database every time these properties are accessed. We factor out initialization of the "source" instance (subclass of cubicweb.server.source.AbstractSource) in a _sources() method. Note that this method takes care of calling "init" method on the source as well as "set_schema" (previously done in repo.set_schema(), which now only touches system_source). Accordingly the "init_sources_from_database" method is dropped along with "add_source"/"remove_source" methods. In syncsources hook, we thus drop: * SourceAddedOp operation which called repo.add_source() so that the SourceAddedHook only cares about checking source configuration now; * SourceRemovedOp and SourceRenamedOp operations for the same reason; * SourceConfigUpdatedOp as updating the live config of source is meaningless once we rely on them being retrieved from the database; * SourceHostConfigUpdatedHook hook which is now useless without call to SourceConfigUpdatedOp; In 3.10 migration script, remove usage of sources_by_uri repo attribute which, unless I'm missing something, appears useless (at least now). In tests: * unittest_datafeed: remove test_update_url method since we dropped respective hook; * unittest_ldapsource: LDAPFeedUserDeletionTC.test_a_filter_inactivate() currently fails because it still relies on live config being updated, this will be fixed in the next changeset once all "live source" logic will be removed.

Add a "Contributing" section to README with patch submission guidelines For the CubicWeb project and its dependencies, we now prefer patches submission and review by email on a public mailing list. We are thus moving away from the previous vcreview-based workflow taking place on the forge. This change is motivated by the following points: - the current reviewer assignment mechanism (pick a random reviewer, rely on reviewer availability rather than on willingness to review, send related patches to distinct people, etc.) is inefficient if not counter-productive; - most of the times, discussion only happens between the patch submitter and a reviewer with no easy way to increase the audience; - cubicweb-vcreview has no concept of patch series; - cubicweb-vcreview is not actively maintained anymore and its usability keeps deteriorating. We expect that email-based submission and review of patches will circumvent these limitations. Anybody interested in the project is welcome to subscribed to the mailing list and participate to the review process. This patch documents the basic workflow of patches submissions by email.

[cwctl] do only clean static data dir content (closes #17069762) deleting the directory itself is useless and may be a problem in automated deployment environments (user may not have permissions to delete or create that directory).

[cwctl] add a no-config-update option to the upgrade command (closes #17069607)

[cwconfig] Add "pyramid" config type to MCOMPAT dict I don't know what this is for exactly, but the accept_mode method below queries this dict with the configuration type name. And if the latter is "pyramid", it crashes since there's no key in MCOMPAT dict. So add one with the same value as "all-in-one". This code is called during "upgrade" migration in particular.

[pyramid] Avoid shutting down the repository at exit if it's already shutting down For install `cubicweb-ctl pyramid` command will explicitly call `repo.shutdown()` so the atexit hook is redundant. Prevent the latter to be executed when the repository is already shutting down (otherwise one gets an assertion error).

[server/test] Extract a "ldapsource" function in unittest_ldapsource.py We will add more use of this pattern in forthcoming changesets, so avoid repetition.

[server] Move "system" source initialization into its own method

[entities] Make CWSource's dictconfig method clearer

[entities] Move CWSource update_config method as a function in 3.15 migration script This script is the only caller of this method. In the code of the function, drop "skip_unknown" argument and continue accordingly upon OptionError.

[server/test] Turn "pull" method of LDAPFeedTestBase into a staticmethod Because it does not use reference to "self" (which should be "cls" for classmethod).

[py3] Use six.text_type instead of unicode in _CWSourceCfgMixIn.update_config()

[server] Drop condition on "3.10" migration in repo.init_sources_from_database() We'll duplicate the code of this method in forthcoming changeset so clean it up a bit of old stuff.

[pkg] Version 3.25.0rc2

[doc] More release note about session data handling mentionning deprecation of _cw.data and removal of get/set_shared_data.

[session] Drop long deprecated [get/set]_shared_data methods they are deprecated since 3.19 and access to the .data attribute itself deprecated in 3.25, so it's time to drop them.

[session] Drop the user session synchronization machinery which should not be necessary anymore since groups and properties are fetched for each request (cache lives in transaction_data).

[session] Avoid deprecation warning on access to Connection.data this attribute has been deprecated during the session related rework, remove warning by explicitly storing data in .transaction_data since session data is not anymore reachable through Connection object. This will allows simplification of session synchronization hooks in a later cset. Also since we do not rely anymore on session data here, we may at some point rewrite those properties to rely on the ORM instead of such custom caching.

[doc] Add release note about change in uicfg

merge 3.25 heads

[hg] Add generated slapd.conf to hgignore

Fix flake8 errors in unittest_cwctl

[cwctl] Drop 'cubicweb_' prefix when displaying cubes with 'cubicweb-ctl list' consistently with newcube / create commands. This is an alternate implementation to backed out changeset fe995d56c949. Closes #17054738

Fix flake8 errors in unittest_cwconfig

[config] Test loading of ccplugin modules to ensure ccplugin modules of cube as package aren't loaded twice, once as `cubicweb_<cube>.ccplugin`, once as `cubes.<cube>.ccplugin`. This test used to fail until fe995d56c949 has been backout. It requires calling `cleanup_sys_module` in `CubicWebConfigurationTC.tearDown` to remove any cube's submodule loaded during tests, so content of `sys.modules` may be tested.

[test] Extract out method-which-is-not-a-method mocking iter_entry_points to ease reuse and improve readability.

[cwconfig] Add a docstring on available_cubes method a renaming could be better at some point, but that's a start.

Backed out changeset fe995d56c949 available_cubes should not strip cubicweb_prefix but return actual package name for cube as package, since its output is also used to e.g. get ccplugin or site_cubicweb module name. Original aim of this set was to fix output of the "cubicweb-ctl list" command. This will be done by an alternate implementation in a later cset.

[hooks] Drop "logstats" hook It is now useless as its looping task would not run on a web instance because respective repository has no scheduler.

[hooks] Do not register "logstats" if repository has no scheduler

[server] Introduce an `has_scheduler` method on Repository This is to be used by client application to determine if looping tasks may be registered in the current process. By checking this, one will avoid the warning in looping_task method when the repository has no scheduler.

[server] Warn instead of failing when a looping task is registered and repo has no scheduler We should provide a way for client code to detect if they should register the looping task or not. See next patch for that.

[server] Exit quickly when a looping task is registered in maintenance mode In such cases the repository will not have a scheduler on purpose because the repository will not be kept running and will quickly shutdown after migration so that it's undesirable to have looping tasks being executed.

[config] Drop unused init_available_cubes method

[doc] Move content of former README.pyramid into relevant modules

Drop README.pyramid.rst file It has been kept since 3.24 when we merged pyramid-cubicweb and cubicweb-pyramid repositories into cubicweb's one waiting for someone to move the information in some canonical place but this never happened and this information is probably not so useful or already existing elsewhere. So just drop the file.

Make entypo link a true URL in README

Refer to cubicweb.readthedocs.io in README docs.cubicweb.org contains outdated documentation at the moment (and for quite some time); until it gets update, refer to the up-to-date documentation on readthedocs.

[server] don't catch exception from postcommit_event() if we are in test mode The historic behavior is to hide potential exception occurring in postcommit_event(). Unfortunately logging statements are hidden by default during tests and it become very hard to debug. At least raise if we are in test mode.

[pkg] Version 3.25.0rc1

[tox] Simplify command line to build the doc

[pyramid] Drop warning about pyramid_debugtoolbar not found in debug mode Some people way want to run a pyramid server in debug mode without the toolbar. This warning is annoying for them. Others should just install and *include* the toolbar by hand.

[pkg] Add development.ini.tmpl to "cubicweb.pyramid" package data Otherwise the file is not installed and `cubicweb-ctl create -c pyramid` does not work.

[skeleton] Don't use system site packages in tox environment As we changed this in cubicweb, there is a consensus that this is not the best practice so let us change it as well for skeleton.

[doc] Update "settings" pyramid documentation Document that without "bwcompat" we can use a development.ini file directly and add a dedicated section to list configuration options.

[doc] Update the "quickstart" page of pyramid documentation Again mentioning the two modes of operations and dropping references to prior implementation.

[doc] Rewrite "pyramid" index page in the book We drop references to pyramid-cubicweb and cubicweb-pyramid, remaining of the merge in 3.24. We now mention the two possible operation modes (with bwcompat being true or false).

[doc] Drop documentation for cubicweb.pyramid.tools module Which, as documented, mostly consists of internals.

[doc] Drop reference to "make_cubicweb_application" in pyramid doc This function got renamed into config_from_cwconfig and this is arguably an internal detail so should not appear in the API documentation.

[doc] Remove trailing whitespaces in doc/book/pyramid/index.rst

[tox] Install "pyramid" dependencies in "doc" environment Otherwise most of the cubicweb.pyramid API doc fails to build. For this we skip installation in the "doc" environment and source the new requirements/doc.txt which will install cubicweb[pyramid], thus pulling cubicweb plus pyramid's dependencies. Furthermore this requirements/doc.txt would be useful to build to documentation on readthedocs.

[web/views] fix bad html in deleteconf with show_composite=True On an entity with no composite an additional item "None" was displayed.

[web/views] display unique composite entities in deleteconf When the entity has two relations to the same composite, display it only once.

Really make rqlrewrite flake8 compliant

[pyramid] disable bwcompat by default in PyramidCWTest To avoid cubicweb exceptions handling.

Fix flake8 errors in rqlrewrite.py

[rqlrewrite/schema] Avoid parsing computed relations for each query The RQLRelationRewriter is instanciated for each RQL query, it should avoid parsing computed relations formula by using a cache, which seems rightly located on the instance's schema. This brings a *huge* performance boost to some pages on application with a few computed relations (x4 observed on a client app). Kudos to Adrien, David and Sylvain. Closes #17059828

[pkg] Restrict yams version to be less than 0.45.0 This is a backport of 10d3bf220be2 on 3.23 branch

[pyramid] Remove pyramid_debugtoolbar from "pyramid.includes" Most of the times, the package is not installed (not pulled by cubicweb[pyramid] nor pyramid itself) in a fresh environment so that first start of a new instance will fail.

[doc] Add some details about rtag derivation and server api cleanup in release notes

[doc] Simplify release notes section Those are all about backward incompatibilities, keep only one section about that.

[doc] Fill release notes for 3.25

[devtools] Create "newcube" in current directory by default Instead of using "cubes path", which in many cases will be something wrong (like <prefix>/share/cubicweb/cubes) now that cubes are regular Python packages.

[pyramid] Do not register repository.shutdown atexit in test mode It seems that there already is a special shutdown step for repository in test mode and pyramid's atexit handler interferes with it leading to assertion error about 'shutting_down' attribute being True in Repository.shutdown().

[cwconfig] don't rely on _CubeImporter while loading cubes _CubeImporter can load cube subpackages twice (there is a test for submodules in cubicweb.test.unittest_cubes.test_no_double_import() but not with a subpackage). Since _CubeImporter is for backward compatibility and is supposed to be deprecated as soon as possible, don't rely on it when loading cube schema or appobjects and always load the real modname (eg. cubicweb_<cube> if exists else cubes.<cube>).

Added tag 3.24.7, debian/3.24.7-1, centos/3.24.7-1 for changeset 58f1e8545a77

[pkg] Version 3.24.7

[pkg] Restrict yams version to be less than 0.45.0 Because of reader's API changes x and other subtle but hard to debug changes in appobjects registration in version 0.45.0. Closes #17065626.

[pkg] Version 3.25.0rc0 Let's make a release candidate version of CubicWeb 3.25.0 (only for the Python package), in the hope that some issues that we usually discover after release will be sorted out before.

[pyramid] Rename paste app factory to pyramid_main Making it clearer that this would build a Pyramid application, just in case we eventually provide other means to create a WSGI application than Pyramid.

[tox] Do not use site packages in testenv This is the normal way of running tox. We used to set this because some people liked to run tests using packages installed by other means (i.e. system-wide). Now, there seems to be a consensus that running tox without site packages is a better idea, in particular since many of our dependencies are now available as wheels and thus would install quickly provided one uses a recent pip/virtualenv.

[test] Add user-site to PYTHONPATH before installation of new cube in test_newcube_install Otherwise, setuptools complains that the target installation directory (i.e. "user site" is not in PYTHONPATH). This is probably because we use a fake HOME with --user install option. Failure shows up when running tests through tox without system site packages (see next patch).

[pkg] Depends on yams >= 0.45.0

[pkg] Depends on logilab-common >= 1.4.0

Merge with 3.24 branch

Added tag 3.24.6, debian/3.24.6-1, centos/3.24.6-1 for changeset 4029fb83a67d

[pkg] Version 3.24.6

Merge public heads

[web/views] deleteconf: ignore workflow when displaying composite entities This hide TrInfo composite entities when deleting an workflowable entity.

[rset] Use RQL cache in syntax_tree method This avoid several unnecessary parsings per HTTP request, since in most case rset's rql is in the rql cache (since it has just been executed), while yapps based parsing is known to be slow.

[repo] Introduce a clear_caches method on the Querier class that is called from repository's clear_caches.

[repo] Consistent API for cache clearing clear_caches, reset_caches, clear_eid_caches are now all named 'clear_caches' with optional eids/etypes arguments to clear eid specific cache entry,

[repo] Extract rql cache handling to a dedicated class

[sobjects] Fix a trivial flake8 error

[repo] Drop cache clearing methods from the source interface Those are actually system source specific.

[repo] Move and rename repo._clear_planning_cache to make it clearer it's clearing the @cache of source_defs method.

[sobjects] Fix flake8 errors in services.py

[test] Rename BaseQuerierTC._access to BaseQuerierTC.admin_access so it's consistent with CubicWebTC and avoid access to a protected attribute which has been exposed by the removal of the 'session' property a few csets earlier.

[repo] Fix flake8 error

[pyramid] Add a "pyramid" instance configuration type In a new module 'cubicweb.pyramid.config' we define a "pyramid" instance configuration type. The noticeable feature of this configuration is that it manages a 'development.ini' file that gets installed in application home (along with `.conf` file). This file is templated and includes generated values for secrets of session and authtk tokens. This means that we can just call: pserve etc/cubicweb.d/<appname>/development.ini or gunicorn --paste etc/cubicweb.d/<appname>/development.ini -b :8080 just after instance creation to get a pyramid instance running without having to hack around a 'pyramid.ini' file. This patch drops 'development.ini' from skeleton and moves it in cubicweb/pyramid so that it gets installed at instance creation which is more appropriate than in cube creation. The new configuration class sets "cubicweb.bwcompat" setting to false so it is not intended to replace the "all-in-one" configuration type (which would require a bit more work). This configuration is close to the the 'repository' configuration type with just a couple of options from WebConfiguration that are needed for Pyramid (anonymous user/password plus some miscellaneous options that I'm not so sure are really needed). Note, in particular, that we do not pull CORS settings to be injected as a WSGI middleware like in wsgi_application_from_cwconfig() since I believe this should be left as an end-user responsibility and since this can be defined in a standard way in paste configuration. This configuration inherits from ServerConfiguration but registers the same appobjects as WebConfiguration. In cubicweb.web.request._CubicWebRequestBase, we guard against access to "uiprops" and "datadir_url" of the config because this new "pyramid" config does not have these (this does not make sense without bwcompat mode). At some point, we should either avoid using `cw_request`'s pyramid request attribute or make cubicweb's web request really independant of existing implementation and drop these assumptions.

[pyramid] Call repository's shutdown method atexit This way when run through a WSGI server (pserve, gunicorn, etc.) we make sure that the repository is properly shut down. In particular, this would wait for threads to terminate (otherwise, they'll be killed).

[querier] Turn repo.querier_cache_key into a private function of the querier module, only place where it's used.

[cleanup] Fix flake8 errors in web/application.py and associated test

[test] Simplify CWTC's init_authentication and assertAuthSuccess which shouldn't rely on removed .session attribute while it's not necessary and could be written in a simpler way.

[cleanup] Drop non sense sentence in docstring

[repo] Drop Session at last This is not used at all anymore.

Fix flake8 some errors

[test] Use existing reference to repo_source

[repo] Drop repo.new_session method we should not go through Session to create Connection anymore. This is unofficial API, no backward compat for now.

Stop using Session on the repository side Only expect session on web request, and let the web session/authentication managers provide them. Access to cnx.data, which used to return session data, is deprecated: there is no more access to session data from the repository side, and they should be access from req.session.data from the web side.

[test] Drop unnecessary call to new_session This is not unittest, the above call is enough.

[user] Drop ugly hack on CWUser entity class which should be not necessary anymore, beside two remaining usage in our own test suite.

[test] Use user_session_cache_key function to generate cache key.

[session+test] Stop storing / accessing session when it's not necessary

[devtools] Stop relying on Session to create connection in RepoAccess

[devtools] Stop using repo.new_session

[session] Kill support for a 'session_open' hook event To be consistent with dropping of the 'session_close' event, and also because this is already not called when using pyramid.

[devtools] Rename FakeSession into FakeConnection because it is what it is.

[cleanup] Drop references to the old internal_session method dropped a while ago, no need to talk about it in a docstring nor to fake it.

[session] Drop unused class

[cleanup] Drop no more used parse_repo_uri function

[server] Fix flake8 error of test/data/hooks.py

[repo] Drop repo.connect backward compat use new_session instead (introduced in 3.19 iirc)

[session] Drop no more necessary Session.sessionid and Connection.connectionid

[repo] Kill repo._sessions and with it the need to have a looping task to clean it up, and with that all the session's timestamping machinery.

[repo] Stop closing session The only point in "closing" session is to call the `session_close` event. Since I'm not aware of any application relying on it, I think this is too costly (in term of code) to maintain and propose to drop it.

[views] Stop giving stats / debug information relying on _sessions This is already broken when using pyramid anyway.

[session] Stop relying on _sessions to synchronize living session by doing what we currently do in pyramid. Also, attempt to synchronize CWUser's class sounds like a dark corner case handling that should be removed in favor of short lived user objects (beside it shouldn't occur except during migration).

[test] Drop useless access to repo._sessions test still succeed afterwards.

[repo] Fix flake8 errors in server/__init__.py

[repo] Drop no more used ShuttingDown exception

[repo] Drop _get_session method it's not actually necessary but holds underlying _sessions dict handling. Drop backward compat relying on it (which IMO should not be a problem). Drop import of unused QueryError along the way.

[test] Drop call to CubicWebConfiguration.load_cwctl_plugins in unittest_cwctl.py This is no longer need now that we call load_available_configurations() in the "list" command (see 5e7282bdf140).

[web] Extract a base class out WebConfiguration This will be used to build a CubicWebPyramidConfiguration in following changeset.

[tox] Drop reference to removed init_instance.py file in flake8-ok-files.txt The file got removed in 1b9fd8bda1cd.

[web] Put 'use-uicache' option in level 2 So that users do not get prompted with this. Follow-up on c34590161082.

[server] Add a "scheduler" command to run repository scheduler This commands starts the repository scheduler as a standalone process that should complement a CubicWeb web instance running as a WSGI application. Added a log message in repository's shutdown method to help testing the command (i.e. make sure the method is called after the scheduler stopped). Related to #17057223.

[cwconfig] Do not instantiate a repo with a scheduler in repository method None of the caller of this method need the repository to have a scheduler and having a scheduler should be something that's explicitly requested.

[server] Make cubicweb/server/utils.py flake8-clean

[server] Drop utils's LoopTask and TasksManager classes not used anymore Related to 17057223.

[server] replace repository's tasks manager by a scheduler Repository() does not accept anymore a 'tasks_manager' argument but rather a 'scheduler' argument which is expected to be an instance of sched.scheduler class. The drop the _tasks_manager attribute of the repository and adjust all internal usages of it. In particular, in the 'repo_stats' service we do not export 'looping_tasks' statistics anymore as there's no way to retrieve this anymore from a web instance. Closes #17057223.

[server] introduce a scheduler class to run repository "looping tasks" We just use the sched module from the standard library and introduce a tiny Python2/3 compatibility layer (more for convenience actually). The "looping" aspect of tasks (previously in LoopTask class) is re-implemeted as a `schedule_periodic_task` function. This is a reasonably thin layer as compared to LoopTask/TasksManager classes. Only the "restart" aspect of LoopTask is no longer present as I'm not sure it's worth keeping. The advantage of using this (in addition to eventually dropping our custom code) is that this scheduler class provides a `run` method that blocks the process while running tasks in its queue. So we can rely on this to have a 'scheduler' ctl command (see forthcoming patch) that would only run "looping tasks" without having to implement the "blocking" aspect ourself. Related to #17057223.

web: add options to ignore css compilation and uicache

[skeleton] call includeme from new cube

[etwist] Do not call repository's start_looping_tasks anymore and warn about this We are about to drop this method from Repository class and replace it by a blocking alternative. This is not compatible with how things currently work in a Twisted server implementation. So do not start repository "looping tasks" in Twisted server anymore and issue a warning about this. If someone is interested in restoring the "all-in-one" behavior where the repository runs within a Twisted server, they may start by implementing repository looping tasks using a Twisted mechanism such as, e.g., http://twistedmatrix.com/documents/current/core/howto/time.html and eventually provide the repository with a compatible scheduler instance so that is can register its periodic tasks. At the moment, we lack resources to do this (and maintain the Twisted server of CubicWeb in general). Related to #17057223.

[test] Flake8-clean and use stdlib unittest in cubicweb/server/test/unittest_utils.py

[web] Don't raise ProcessFormError if there is no value specified a03376213747 introduced a slight modification in None / empty string values and with it a regression where an exception is raised on empty values, while we should simply consider no value is specified and return None.

[test] Cleanup imports in unittest_form Remove unused and reorder, + use bare unittest along the way.

Fix log level parameter not taken into account in cwsource log table An error in the jquery selector of the html widget controlling the log level was the cause of this bug. Closes #15772634 (grafted from c9129aae884b476455c8ed1ad802efc3a26503a5)

[pyramid] Drop call to repository's start_looping_tasks method and warn about this We are about to drop the start_looping_tasks method and running "looping tasks" along the WSGI application is not a very good idea. So issue a warning pointing to the forthcoming "scheduler" command. Related to #17057223.

[test] Make unittest_serverctl.py flake8-clean

[wsgi] Drop calls to repo.start_looping_tasks It's unlikely that these implementations were ever used so we do not bother with deprecation or a warning. We just drop the call to this method which should not be called from within a WSGI application. Related to #17057223.

[pyramid] Drop useless "init_instance" module

[server] Drop inexistent "bootstrap" argument in Repository initialization This is a missing bit from 5de78b6fff2e. Unfortunately this code appears to be uncovered by our test suite.

[server] fix a typo in a docstring

[ctl] Load available configurations in "cubicweb-ctl list" Otherwise we rely on associated ctl plugins to be loaded which is arguably orthogonal to configurations.

[web/views] Fix a docstring

[server] Use looping_task method in Repository._prepare_startup()

[skeleton] Drop logging configuration from development.ini This is actually not useful as logging is already initialized and configure "somewhere else" where the repository is started.

[pyramid] Drop retrieval of auth/session secret in all-in-one.conf We actually never supported this in cubicweb, but only in pyramid-cubicweb.

[cwctl] Recommend 'pyramid' before 'wsgi' when twisted is not available Not sure anyone ever really used the latter and the former is certainly more reliable/tested.

[cwctl] Drop message about 'the pyramid cube' in "start" command This is shown when twisted is not importable and we suggest to use 'cubicweb pyramid <appid>' instead. Since 3.24, cubicweb-pyramid got merge in cubicweb so drop this reference in the message.

[cwconfig] Fix two NameError in load_available_configs() This comes from a copy-paste in d92d75b17a85.

[etwist] Drop try/except around definitions of 'all-in-one' configuration and command handlers cubicweb.server is no longer conditionally shipped, neither in python package nor in debian package since 3.24.

[server] Call 'server_startup' hooks at the end of repository "bootstrap" Now that we have guarded all calls to Repository.looping_task() in 'server_startup' hooks we can execute this hooks category at bootstrap step. This way, repositories running without a tasks manager (i.e. those embedded into a WSGI application) will have these hooks triggered.

[pyramid] Drop module-level cache and cleanup looping tasks in tools And use a LRU cache over cached_build_user function. This looping task is problematic because it would not be run from within a WSGI application which does not have a repository with a tasks manager. This pulls an explicit dependency on 'repoze.lru' but it's not a big deal since pyramid already depends on this. RPM spec file not update since it does not even mention pyramid...

[hooks] Return early in server_startup hooks if the repository has not tasks manager

[server] Only register "clean_sessions" looping tasks if repository has a tasks manager This prepares for calling Repository's _prepare_startup() in bootstrap() so that repository instances without a tasks manager running can still have their 'server_startup' hooks triggered (this is particularly useful for BFSS storages).

[utils] Do not pass a tasks manager to Repository in admincnx() We do not call repo.start_looping_tasks() for the instantiated repository, so a tasks manager is useless.

[server] Separate repository bootstrap from initialization Thus we now call 'bootstrap' here and there explicitly and remove call of this method in Repository.__init__(). This way instantiation of a Repository does not *implicitly* triggers the "bootstrap" step, which is arguably not a trivial thing and thus deserves to be independent. In __init__, set 'shutting_down' attribute to None and then to False in bootstrap as a mean to indicate that, when initialized, a repository is neither shutting down nor started (not sure where this is used though).

[server] Also remove tmpdir in case of error during restore database command

[server] Rename Repository's init_cnxset_pool method bootstrap See the (removed) docstring for a rationale...

[server] Replace server config's init_cnxset_pool attribute by a "bootstrap" parameter in Repository This 'init_cnxset_pool' class attribute is actually used to control whether a Repository instance should be "bootstraped" (see Repository.init_cnxset_pool() for a definition of "bootstrap") or not. I seems clearer to me to have this controlled by a boolean "bootstrap" initialization parameter in Repository.

[server] Drop close/open steps in NativeSQLSource.restore() The only place where this is called is in ServerMigrationHelper.restore_database() where config.init_cnxset_pool is set to False. So these steps appear to be useless.

[skel] Fix rpm's .spec file skeleton It's broken for new-style packages since it doesn't consider (even remove) python packages.

[server] Move "starting repository..." message where this actually happens

[server] Initialize Repository without a tasks manager in ServerMigrationHelper.restore_database() We don't need a tasks manager here I think. Thus instantiate the Repository from its class directly instead of using config.repository() which set a tasks manager.

[server] Fix a typo in log message in Repository.init_cnxset_pool()

[cwconfig] Issue a warning if a configuration module cannot be loaded We already do this when loading ctl plugin modules, so make things symmetrical for config modules.

[ctl] Load available configurations in "cubicweb-ctl list" Otherwise we rely on associated ctl plugins to be loaded which is arguably orthogonal to configurations.

[skeleton] Set cubicweb.bwcompat to False in development.ini Quite a few things from the default web UI do not work when running the application through pserve (in particular, data files are not apparently served). So better disable the "bwcompat" mode until we are sure it works properly (or it gets dropped ;)).

[pyramid] Use existing repo in PyramidCWTest We thus do not make use of config_from_cwconfig() and build the configurator instance by hand prior to include 'cubicweb.pyramid'.

[pyramid] Make it possible to setup CubicWeb instance from an existing repo in includeme()

[pyramid] Move loading of pyramid_debugtoolbar later in includeme() To gather cwconfig and repository instantiations.

[skeleton,pyramid] Move pyramid app definition in cubicweb.pyramid module The application definition is actually not specific to the final "cube" being bootstrapped from skeleton. This patch thus move the pyramid application function into cubicweb.pyramid module and let cubicweb register the "paste.app_factory" entry point (instead of the bootstrapped cube). Useless call to `config.scan` is dropped along the way.

[pyramid] Add a copyright and docstring to all modules We add copyright statements for both UNLISH (original author) and LOGILAB.

[cwconfig] Strip "cubicweb_" prefix from cube name in available_cubes() Closes #17054738.

[web] fix os.rename usage on windows when destination exists os.rename on windows will raise OSError (or WindowsError subclass) when if destination file already exists. Also check that exception is EEXIST. There is an attempt to fix in f6ba947c but using IOError instead of OSError. Closes #14214794

[web/views] show composite entities in delete view Disabled by default, can be activated with show_composite = True Display first level of composite entities in a treeview, limited to the current (page_size - 1), so the maximum of displayed entities bump to page_size * (page_size - 1).

[workflow] Utilities for declarative definition of workflows Closes #5337897

[server] Remove pseudo-handling of exceptions in Repository._delete_cascade_multi() It seems to me that we never want to pass on an exception here be it in "test" mode or not, so remove the last `except Exception:` clause. For instance, the actual implementation would pass on an IntegrityError and thus possibly hide it in a log message. Concerning the `except Unauthorized:`, as said by the log message this should not happen because we are within a `security_enabled(write=False)` context. So remove it as well. As far as I can tell, this strange handling of exceptions dates from 37668bf302f5 and there is no clear justification to it.

[web] fix os.rename usage on windows when destination exists os.rename on windows will raise OSError (or WindowsError subclass) when if destination file already exists. Also check that exception is EEXIST. There is an attempt to fix in f6ba947c but using IOError instead of OSError. Closes #14214794

[skeleton] Add "cubicweb.session.secret" to development.ini Commented as other "secret" settings. At least, now all settings are present in the development.ini file. [ci skip]

[pyramid] Include 'cubicweb.pyramid.predicates' where it is used It's only used in "rest_api" module.

[pyramid] Rename make_cubicweb_application function as config_from_cwconfig And update its docstring. This function does not actually "make a cubicweb application", it just builds a pyramid.config.Configurator instance from a CubicWeb config object.

[server] Reprase messages of db-check to avoid using "system" and "sources" And rather mention "entity type table" and "entities" table, which are clearer references.

[predicates] Fix reference to has_related_entities in partial_has_related_entities docstring

Fix possible double import of cubes modules When cubes using the new layout are imported with 'cubicweb_<cube>' and with 'cubes.<cube>', the same module is imported twice. Handle this by adding 'cubes.<cube>' to sys.modules when importing from 'cubicweb_<cube>'. Move load_module() to a sub class _CubesLoader to share informations computed in find_modules(). Don't handle subpackages in _CubesImporter and rely on normal import machinery instead. Add a test and use unittest from cubicweb.devtools.testlib which resolve to unittest2 on PY2 with assertLogs() method.

Make load_module() return existing module if present in sys.modules Otherwise the reload() builtin will not work correctly: https://www.python.org/dev/peps/pep-0302/#specification-part-1-the-importer-protocol

[server] Ignore computed relations in "relations" integrity checker

[server] Use global registry variable of integrity checker functions Instead of querying globals() in checkintegrity.check(), we maintain a _CHECKERS dict mapping checker name to function in checkintegrity module. This is later used to build the list of available checkers in 'db-check' command help.

[doc] Add 3.25 release notes [ci skip]