[doc] improve blog tutorial first steps and explain common errors

[doc] how to install pyramid or twisted application server

[doc] new behavior in virtualenv since 3.26

[skeleton] use python3 flake8 In tox "flake8" environment, use python 3 virtualenv, so we can check for python3 incompatible syntax.

[skeleton] drop whitelist_externals for flake8 whitelist_externals makes sense when used with sitepackages=true. In this case it will always rely on $VIRTUAL_ENV/bin/flake8 so, the whitelist_externals has no effect.

Add changelog note for running tests with pytest-xdist

Make test database template creation concurrent build_db_cache() is used in tests to create test database templates, i.e. DEFAULT_EMPTY_DB_ID (which is __default_empty_db__) and custom template database using CubicwebTC test_db_id/pre_setup_database API. When running tests in parallel using multiple processes, build_db_cache() may try to build the same database twice. Avoid this by adding synchronisation of process by using a file lock. So when two processes require the same template database, one build the database and others wait it to be created. Use filelock (https://github.com/benediktschmitt/py-filelock) library to have a portable (unix / windows) way for handling locks. Also filelock is packaged in debian: https://packages.debian.org/source/python-filelock

Make test database cache lookup concurrent has_cache() is used to dertermine if the template database already exists or not. When running tests accross multiple process, has_cache() wasn't aware of databases created by other processes because it used a cache to generate the cache... Since the discovering discover_cached_db() time is quite reasonable (it's just about listing file in a directory), let re-generate the cache on cache miss.

Fix flake8 issues since release 3.6.0 Flake8 had a new release which raise new issues, namely: W504: line break after binary operator F841: local variable 'ex' is assigned to but never used W605: invalid escape sequence F821: undefined name 'buffer' (noqa seems the only way to avoid this false positive) Also pin flake8>=3.6 in our tests and make explicit that we use python3 to run flake8 tests.

Stabilize flake8 tests on 3.26 branch Use basepython=python2 or flake8 installed with python3 will raise "[F821] undefined name 'buffer'" Pin flake8 < 3.6 which has new checks that doesn't pass on our current code base.

Fix typo in RDFS URI

[doc] Do not rely on lgc's sphinx_ext This module is broken (getting an ImportError) and I don't want to fix it. The only usage we make of it is for the "autodocstring" directive, and I don't know what it does. Based on the context, replacing autodocstring with either automodule or autoclass and removing sphinx_ext from required Sphinx extensions.

Added tag 3.26.7 for changeset ee4ad63c91e5

[pkg] Version 3.26.7

Fix possible double import of cubes For a new-style cube, if we import "cubicweb_<name>" and then import "cubes.<name>", the cube will be imported twice. cubes.<name> and then cubicweb_name is ok though... When using pyramid, we try to find which cube define a "includeme" by importing them with the name "cubes.<name>", so we (possibly ?) double import all new-style cube. This case may also occur in tests with PyramidCWTest. Touching the import loader to fix this scares me, so let's fix this by testing "cubicweb_<name>" *before* "cubes.name" (for old style cubes, importing cubicweb_<name> raise ImportError).

[autoform] Make RelationField.fromcardinality a class method factory Instead of having the "RelationField" class hard-coded in this factory method (thus preventing any subclass to properly use it), we make that method a classmethod and instantiate the actual class instead of the base one.

[autoform] Use fieldclass from kwargs in guess_field for non-final relations This follows up on af969080e7e6 in which a "fieldclass" kwarg has been added in EntityFieldsForm.field_by_name() before calling guess_field(). In this changeset, this value was only used for final relation but the way non-final relation actually also changed since we also use guess_field from these relation (previously, the field class specified through uicfg was instantiated directly in EntityFieldsForm.field_by_name()). So in the non-final case in guess_field() we simply try to retrieve "fieldclass" from kwargs and fall back to RelationField.

[py37] `async` is now a reserved keyword

Merge 3.26

Added tag 3.26.6, debian/3.26.6-1 for changeset 8362503a9248

[pkg] version 3.26.6

[py3] Replace unicode keyword which is unknown in python3

[skeleton] Use "py3" as tox environment instead of "py34"

Added tag 3.26.5 for changeset 76d7b0aa94d2

[pkg] Version 3.26.5

Merge with 3.26

[py3] Use "utf-8" as input encoding for docutils in rest extension Docutils says that "unicode" is an unknown encoding on Python3. Not sure where "unicode" as an encoding comes (it's there since showtime) but "utf-8" seems more appropriate. From now on, rendering of CubicWeb's rst directive (e.g. ..winclude) works on Python 3. Accordingly, we extend wdoc's test to demonstrate this. For this we need to install docutils in tox's web environment.

[py3] Avoid setting attribute on Element instance in wdoc view This change is needed to make "wdoc" view work on Python 3. Indeed, before this patch, we used to set a "parent" attribute on Element instances, which is not permitted since Python 3.3 as xml.etree.ElementTree uses the C implementation by default. To get rid of this "parent" attribute, we rework the XML nodes processing logic to return and retrieve a node along with its parent in several place of the code. Namely, build_toc_index() now accepts the "parent" value for "node" and inserts it in the "index" dict (thus avoid `node.parent = <value>`). Respectively, every query to the "index" dict is updated to also retrieve the "parent" (thus avoiding `<value> = node.parent`). Later in InlineHelpView class, the "index" dict is bound to "tocindex" attribute so we adjust how it is queried in respective methods. Tests cubicweb/web/test/unittest_views_wdoc.py now pass on Python 3.

[test] Add a trivial test for "wdoc" view We just make sure it gets rendered without errors (which wasn't the case before the previous patch). It does not work on Python 3; this will be fixed in next changeset.

[py3] Open rst file in text mode in wdoc view Otherwise, UStringIO complains that it receives bytes.

[ext/rest] Drop "handle_io_errors" argument to docutils' FileInput This argument is deprecated and ignore in docutils since version 0.10. Besides, its default value is None which is the same as we passed.

[wdoc] Remove fallback about xml.etree module Module xml.etree exists since python 2.5.

[wdoc] Avoid usage of deprecated xlm elementree getchildren() method Use list(node) as suggested by the deprecation warning.

[style] Make cubicweb/web/views/wdoc.py flake8-clean

[py3] Use six.text_type instead of unicode in wdoc views

[views/startup] drop a duplicate rql query in manage view Counting entities in `nb` is already done above this code, regression introduced by a85db6fa

Merge with 3.26

Added tag 3.26.4, debian/3.26.4-1 for changeset e1c1af68db86

[pkg] Version 3.26.4

[py3] ensure AjaxController return bytes publish() must return bytes, when the @ajaxfunc decorated method return None, this raised: TypeError: You cannot set Response.body to a text object (use Response.text)

[py3] Let InstanceCommand.run_arg() always return an int The caller of this method (which is method "run_args") does `max(status, self.run_arg(appid))` and run_arg might return None if the underlying <cmdname>_instance method returns None. On python3 max(1, None) breaks with "TypeError: unorderable types: int() > NoneType()". So we enforce run_arg() method to return an integer even if the underlying method returns None. This fixes call of "cubicweb-ctl i18ninstance" with Python3.

[test] Use cubicweb-file < 2.0.0 From version 2.0.0, cubicweb-file moved to new-style layout (Python package) but we still expect it to be an old-style cube in CubicWeb's tests (WebconfigTC in web/test/unittest_webconfig.py).

[py3] Decode subprocess stdout in toolsutils.show_diffs() On Python 3, subprocess' pipes are bytes stream so we must decode them before printing. Without this change, diff of configuration files (e.g. all-in-one.conf as displayed upon instance upgrade) will rendered as a single line on Python 3.

[python3] make sure staticcontroller returns a bytes instance The ``publish()`` method of controller is supposed to return a bytes intsance. Returning a text_type instance will crash later on in the Pyramid response handler when assigning content to ``response.body`` (at least in Python3):: >>> from pyramid.response import Response >>> r = Response() >>> r.body = '' Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/adim/.virtualenvs/…/response.py", line 562, in _body__set raise TypeError(msg) TypeError: You cannot set Response.body to a text object (use Response.text)

[python3] use inspect.getfullargspec instead of inspect.getargspec The latter is now deprecated

[py3] ensure AjaxController return bytes publish() must return bytes, when the @ajaxfunc decorated method return None, this raised: TypeError: You cannot set Response.body to a text object (use Response.text)

[py3] Let InstanceCommand.run_arg() always return an int The caller of this method (which is method "run_args") does `max(status, self.run_arg(appid))` and run_arg might return None if the underlying <cmdname>_instance method returns None. On python3 max(1, None) breaks with "TypeError: unorderable types: int() > NoneType()". So we enforce run_arg() method to return an integer even if the underlying method returns None. This fixes call of "cubicweb-ctl i18ninstance" with Python3.

Depend on cubicweb-file >= 2.0.0 I applied V1 of the patch, but V2 contained this extra hunk...

[test] Update webconfig's tests accounting for cubicweb-file being now a package WebconfigTC.test_locate_all_files and WebconfigTC.test_locate_resource assumed that cubicweb-file was installed in <share>/cubes/file whereas, from the latest release, it is now a standard Python package thus installed in site-packages as cubicweb_file. We simply update base directory of these two tests.

[widgets.js] fix obvious bug detected by sonar

[views/uicfg] obvious bug detected by sonar

[style] Make toolsutils.py flake8-clean Run `autopep8 --max-line-length 100 -i cubicweb/toolsutils.py` and fixed unused imports.

[py3] Decode subprocess stdout in toolsutils.show_diffs() On Python 3, subprocess' pipes are bytes stream so we must decode them before printing. Without this change, diff of configuration files (e.g. all-in-one.conf as displayed upon instance upgrade) will rendered as a single line on Python 3.

[python3] make sure staticcontroller returns a bytes instance The ``publish()`` method of controller is supposed to return a bytes intsance. Returning a text_type instance will crash later on in the Pyramid response handler when assigning content to ``response.body`` (at least in Python3):: >>> from pyramid.response import Response >>> r = Response() >>> r.body = '' Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/adim/.virtualenvs/…/response.py", line 562, in _body__set raise TypeError(msg) TypeError: You cannot set Response.body to a text object (use Response.text)

[python3] use inspect.getfullargspec instead of inspect.getargspec The latter is now deprecated

[utils] Fixes compatiblity of QueryCache with expected dict interface QueryCache is expected to mimick the interface of the dict built-in. The current __iter__ implementation breaks this expectation by iterating over key, value pairs instead of only the keys. This changeset fixes this issue by changing the __iter__ implementation to iterate over the keys in the cache and providing an implementation of the items method with a contract identical to the dict build-in (return a copy of the key- value pairs within the dictionary as a list).

[test/statsd] add tests for cubicweb.statsd_logger

[utils] Implements __iter__ on QueryCache In order to be able to iterate over the currently cached queries (so that they may be inspected and invalidated), this changeset implementes __iter__ on QueryCache.

[utils] Added new get method to QueryCache The new get method enables the retrieval an item in the cache by specifying a default value so that in the absence of an item for the key, the default value is returned.

[test/rset] typo

[utils] Fixed issue in clearing QueryCache When trying to add an item in a full QueryCache, the cache tries to make room. It tries to list and remove non-permanent items with a transient counter (number of times it has been requested). However, there is a pathological case where items could be non-permanent but still not have a transient counter because they were added, but never requested. In some cases, the full cache could be flushed, including the permanent items. This changeset attempts to fix this issue by only dropping the non-permanent items that did not hav a transient counter.

[statsd] add a statsd_timethis context manager This context manager allows to report to statsd the execution time of any piece of code, without the need for it to be a function or a method, like:: def some_function(): do_stuff() with statsd_timethis('some context message'): do_that() ensure_this()

[test/rset] add tests for ResultSet.first() and .last() methods

[test/rset] flake8

[rset] update copyright

[rset] add convenient first() and last() methods on ResultSet These make life easier for the heavy user of the CW shell. Also aliases entities() with all(); so we end with a pretty consistant set: - .one() - .first() - .last() - .all()

[server] Fixed issue with the adapters cache of a cloned EditedEntity The adapters' cache of the entity within an EditedEntity used to be copied along the entity itself during a cloning operation of the EditedEntity. This raises an issue when the adapters' cache is not empty and contains cached values linking to transaction-specific data (or the transaction itself). This prevents the clone EditedEntity being used in a transaction outside the one that created it. This change resets the adapters' cache of the new copy so that no cached value remain. The new copy can then be used in a new transaction afterwards.

[rql2sql] fix handling of the ORDERBY clause when the aggregation function is "embedded" ie. a situation like: Any X GROUPBY X ORDERBY COUNT(Y) WHERE ... works fine since the aggregation function (COUNT) if the first one, whereas a situation like the proposed test: Any X GROUPBY X ORDERBY ABS(COUNT(Y)) WHERE ... failed due to the fact we only checked the top aggratation function (ABS). Closes #17108671

Merge 3.26

[pkg] Set version to 3.27.0.dev0

Added tag 3.26.3, debian/3.26.3-1 for changeset f7067be5f69c

[pkg] Version 3.26.3

[utils/statsd_logger] flake8

[source/native] do NOT do anything with fti if do_fti is False ensure no fti related process is done when fti is disabled on a db source.

[test/web] flake8 unittest_propertysheet.py

[web] fix css file generation permission bug (closes #17143773) PropertySheet.process_resource() uses mkstemp which enforce a 0600 mode on the output file, which is not the file permission we want for css files.

[test/web] add a test for css generation file permission problem (related to #17143773)

[doc] include the 3.26 changelog in changes's index

[server] set PGPASSWORD when running pg commands for backup and restore For db-dump (implied by upgrade) and db-restore with postgres with non-trusted authentication, the db password is asked by pg tools (pg_dump, pg_restore, createdb, dropdb). Avoid this behavior by setting the PGPASSWORD environment variable when database driver is postgres and db-password is set in sources file.

Added tag 3.26.2, centos/3.26.2-1, debian/3.26.2-1 for changeset 9bee3134d304

[pkg] Version 3.26.2

Flake8 cubicweb/web/webctl.py Having the file flake8 clean makes it easier to catch basic programming errors (e.g. name errors).

[web] Add missing import of _DATA_DIR in webctl Followup on d84bc85f7f70 which introduced usage of this name in _datadirs method of GenStaticDataDirMixIn but resulted into a NameError at runtime and makes migrations crash.

[server] unicode → text_type in utils

[views] unicode → six.text_type in views/authentication.py

[pyramid] Recreate CWSession entity when eid no longer exists As noted in comment, it might occur that we receive a "sessioneid" from a valid cookie while respective CWSession got dropped (typical case is db being recreated while users got preserved). In such case, we just recreate the CWSession entity as if it did not exist.

Added tag 3.26.1, debian/3.26.1-1, centos/3.26.1-1 for changeset 027676243aaa

[pkg] Drop non-existant files from MANIFEST.in

[pkg] Version 3.26.1

[test] Drop an unused function in cwconfig tests

[cwconfig] Always use "sys.prefix" to determine installation prefix We drop all legacy logic implemented in _find_prefix() which now happens to be buggy since we cleaned up setup.py in 3.26 (noticeably because virtualenv is no longer properly detected). Closes #17132990.

[cwconfig] Look for migration scripts directory relatively to code Thus getting rid of the last piece of data in "shared" directory. As a consequence, we can drop data_files from __pkginfo__.py. Related to #17132990.

[pkg] Get rid of "pydir" in __pkginfo__.py Nothing uses this.

[pkg] Get rid of "shared" data These are not used anymore per previous changesets. Related to #17132990.

[cwconfig] Drop shared_dir method There's no more users of this method. Related to #17132990.

[config] Avoid relying on shared_dir method when possible From 3.26, some "data" directories (like web/data and i18n or wdoc) are installed along with CubicWeb in site packages. So we can retrieve them relatively to the module that needs them (namely "cubicweb" for "i18n" and "cubicweb.web" for "web/data" and "wdoc"). Reason for doing this is that "share_dir()" is not working when cubicweb is installed in "develop" mode and we are about to drop this method. Related to #17132990.

[test] Add a test for WebConfiguration.locate_all_files() We are going to change the implementation, make sure there's no regression.

[cwconfig] Get rid of _forced_mode global And compute CubicWebNoAppConfiguration.mode within the class namespace. Related to #17132990.

[cwconfig] Get rid of CWDEV This is in preparation of a fix for #17132990 to cleanup legacy (and probably no longer working) behaviors.

[doc] Set release date of 3.26

[web] Fix UnboundLocalError about "exc" in ajax controller Introduced in 159dce89a145.

Link to "3.26" docs in README

Added tag 3.26.0, debian/3.26.0-1, centos/3.26.0-1 for changeset 199851fcddd4

[pkg] Version 3.26.0

[pkg] Version 3.26.0rc0

[schema] Define a decorator to handle methods override There is a number of external classes (from yams/rql) methods overridden "by hand" in cubicweb/schema.py. Define a single decorator to factor out the pattern. It handles specifying the method name (to avoid conflict with the local namespace) and passing the original method to the new definition when needed.

[pkg] Use setuptools.find_packages() We drop the custom get_packages() function in setup.py that used to look for test files (not discovered automatically since test directories are not packages) and replace it by explicit entries in MANIFEST.in.

[pkg] Drop SandboxViolation verification cracking Dunno what this is for. Let's see if this work without.

[pkg] Drop custom "install_data" command Not sure what this is for, the comment mentions "egg" directory, but this is quite likely an old hack that should be unnecessary when using a recent setuptools.

[pkg] Replace custom "scripts" handling by a setuptools entry points http://setuptools.readthedocs.io/en/latest/setuptools.html#automatic-script-creation

Set a default value for arguments of cubicweb.cwctl:run() This will help making a console script entry point from this to get rid of the custom installation of scripts in setup.py. Accordingly, update all calls of this function.

[pkg] Drop custom install_lib command and use include_package_data instead The custom install_lib command appeared to be meant to install package data, as declared in __pkginfo__.py. Let's rely on setuptools' include_package_data[1] option to handle this. [1]: http://setuptools.readthedocs.io/en/latest/setuptools.html#including-data-files

[py3] Replace a unicode() occurrence in workflow views

[py3] Drop a couple of unicode() in devtools.testlib

[py3] Alias long to int in pgstore tests The previous version of handling PY2/PY3 made flake8 complain.

Merge with 3.25

Added tag 3.25.4, centos/3.25.4-1, debian/3.25.4-1 for changeset b8567725c473

[pkg] Version 3.25.4

[pyramid] Let logging be configured through .ini file We do not initialize logging in CubicWebPyramidConfiguration thus preventing logging to be started with cubicweb's configuration. On the other hand, any logging configuration defined in ``development.ini`` file will be loaded by whatever start the instance using this file (i.e. pserve, gunicorn, etc.). The benefit is that logging can now be configured per "qualified name" in this file (i.e. one can easily set the DEBUG level for the application cube while keeping all dependencies' level to WARNING). In the development.ini template, we add logger configurations for "logilab" and "cubicweb" qualified names (along with those of the application cube).

Disable pytest's logs reporting CubicWeb already prints captured logs in stderr upon test failure, so we get them twice. Until we fix our logging management, disabling pytest capture and restore brings the output level back to sanity.

[pyramid] Do not issue security warnings in test mode When some session or authtk secret is missing in Pyramid settings, scary "!! SECURITY WARNING !!" are issued. This is arguably pointless in tests. So disable them in this case.

[ssplanner] Prevent execution of write queries involving computed relations Previously, setting a computed relation upon entity creation or modification (using the ORM or an RQL query) would usually fail with an operational error in the backend ("no such table"). However, under some mysterious circumstances (like passing a string as value in cw_set for a computed relation), the RQL to SQL transformation would simply drop the clause. To prevent this to happen, we add a check for computed relation before adding a relation to an execution plan. This check raises a QueryError. It happens in several places: * in querier.InsertPlan.add_relation_def() (called from several places in ssplanner steps) for INSERT queries, * in ssplanner.UpdateStep.execute() for SET queries and, * in ssplanner.SSplanner.build_delete_plan() for DELETE queries. Tests added to unittest_querier.py because unittest_sslplanner.py looked inappropriate (it has only unit tests) and the former already had a NonRegressionTC class.

[querier] Expand 'rdef' argument in InsertPlan.add_relation_def() Expand the 'rdef' tuple to improve readability. The 'rtype' value is not used at the moment, but will in the next changeset.

[server] Use dict.setdefault() instead of if/else in repo.glob_add_relations()

[cleanup] Fix some flake8 errors

[test] Stop using lgc.testlib in unittest_migractions

[test] Drop no more used "maxeid" based deletion in BaseQuerierTC and derived tests This is probably only necessary for QuerierTC itself, move it there and drop incantation from other derived classes to deactivate this feature.

[rql2sql/test] Remove dead setUpModule tearDownModule they are overwritten a few lines below , hence this code is dead and not necessary for test success, drop it.