[pkg] Run all unit tests in autopkgtest We previously forgot to copy `tox.ini` alongside the tests. As this file configures pytest pattern matching rules, we were not running quite a few tests. This is now fixed. The added tests required some adjustments in the Debian dependencies. Sadly, not all tests currently pass. It seems there are some ordering dependencies between the tests in regards to how data are loaded. Fixing this probably would probably be better debugged by ensuring the test suite does not fail when using pytest random-order plugin.

[pkg] Stop running unit tests from skeleton The test files shipped as part as the skeleton for `cubeweb-ctl newcube` are not meant to be run as part of our test suite!

[tests] Use 16 bytes key when testing Blowfish The version of Cryptodome currently in Debian only accepts Blowfish keys with at least 5 bytes. This is a bug (it should accept 4 as it did before) that has since been fixed upstream: https://github.com/Legrandin/pycryptodome/commit/93912230f3c39e09f4038cb17e301468687c6538 So we work around this issue by using a 16 bytes key instead (which happens to be the minimum recommended length).

Use time.process_time() instead of deprecated clock() This fixes: DeprecationWarning: time.clock has been deprecated in Python 3.3 and will be removed from Python 3.8: use time.perf_counter or time.process_time instead

[web] Make the time separator configurable in JQueryDateTimePicker We add a "separator" argument to JQueryDateTimePicker which will be used when creating the underlying JQueryTimePicker instance so that the user can control how time is displayed by the widget. This is very similar to the "timesteps" argument that's already present.

Added tag 3.26.12, debian/3.26.12-1 for changeset 4531a266f8b2

[pkg] Version 3.26.12

[autoform] Fix display_fields handling This follows-up on changeset b2ceb483e056 (present in 3.25 as well), in which AutomaticEntityForm.editable_attributes() was turned into a generator. In this method, in case display_fields is not None, we previously returned a list of (rtype, role); this was changed into a yield, but the change was missing a return after the loop in order not to go through the remainder of the function. Add tests for editable_attributes() method.

Flake8 cubicweb/web/views/autoform.py Fixes: cubicweb/web/views/autoform.py:703: [E122] continuation line missing indentation or outdented cubicweb/web/views/autoform.py:704: [E122] continuation line missing indentation or outdented cubicweb/web/views/autoform.py:819: [W504] line break after binary operator cubicweb/web/views/autoform.py:820: [E125] continuation line with same indent as next logical line cubicweb/web/views/autoform.py:961: [W504] line break after binary operator cubicweb/web/views/autoform.py:962: [E126] continuation line over-indented for hanging indent

[py3] unicode() -> six.text_type in autoform

Added tag debian/3.26.11-2 for changeset e7eb914df71d

[debian] Release 3.26.11-2

[test] Require pycryptodomex, not pycryptodome This follows up on changeset 5b0ce10a7046.

[debian] Mention Lintian warnings fixes from previous changesets

[debian] Remove unused/redundant entries from debian/cubicweb-ctl.dirs * bash completion entry produces a lintian warning "package-installs-into-obsolete-dir etc/bash_completion.d/", and it does not install bash completion. * doc directory should be handled automatically.

[debian] Make descriptions of python-cubicweb and python3-cubicweb uniform Thus solving lintian warning "description-synopsis-starts-with-article".

[debian] Mention that -postgresql-support binaries are dependency package Fixing lintian warning "empty-binary-package".

[debian] Add ${misc:Depends} where missing Fixes lintian warning "debhelper-but-no-misc-depends".

[debian] Set Section: python for python- binary package This is to fix a Lintian warning.

[debian] Install cubicweb-ctl from python2 build Previously, the binary package was empty. Now it is bound to the python2 package as support for python3 is "fresh". In order python2 and python3 packages to be co-installable, we drop the cubicweb-ctl file from both packages.

[debian] Introduce python3-cubicweb and python3-cubicweb-pyramid binary packages This is the bare minimum to have a working cubicweb on python3. Further work is done in default branch.

[skeleton] Update Debian packaging template Here are multiple updates to the Debian packaging template provided when running `cubicweb-ctl newcube`: * Switch to source format 3.0 (quilt). * Switch to debhelper 9. * Switch to dh-python. * Switch to Python 3. * Name the binary package `python3-cubicweb-CUBENAME` instead of just `cubicweb-CUBENAME` (which is still the source package). * Populate Depends using dh_python3 support for Python requirements. * Run test suite at build time using pytest. * Add autopkgtest to run test suite against the installed package. * Bump Standards-Version to 4.3.0.

[test] Use --short-description when testing newcube command

[devtools] Allow to specify short desc on `newcube` command line In order to implement automated testing of `cubicweb-ctl newcube`, we need to avoid any user interaction. So we add a `-s` switch to specify a short description on the command-line instead of requiring an input.

[pkg] Remove build dependency on dh-systemd As suggested by Lintian, remove build dependency on obsolete package dh-systemd. Instead, specify a more precise version of debhelper 9 that added support for systemd.

[config] Raise in case of failure to send emails when in test mode Since, for some reason, exception log messages are not visible when running tests, it's very hard to debug when sendmails() method fails during tests. So, as in some other places, let's raise the exception when in test mode.

[doc] Update documentation to match current Debian packages

[pkg] Remove transitional packages We can take the opportunity of the switch to Python 3 to get rid of old transitional packages. Let's do it. :)

[pkg] Use sections from requires.txt to populate Recommends and Suggests As Denis Laxalde pointed out, dh_python can also generate Recommends and Suggests from Python package names. So let's use that instead of manually populating these fields in `debian/control`. Optional dependencies are currently specified in `setup.py` grouped by feature. These dependency groups are turned into sections in `requires.txt`. Thankfully `dh_python3` has options to populate Recommends or Suggests with all package from a given section. `debian/rules` now contains a list of which sections should go to Recommends and which section should go to Suggests. Because such extra list easily gets out-of-sync, we add a third list for ignored sections, and ensure that all sections currently in `requires.txt` get a mentioned in `debian/rules`. Here are the results compared to the previous version with explicit Recommends and Suggests (only listing Python packages): |==============================================================| | only in previous | common | only in new | |==============================================================| | Recommends | |--------------------------------------------------------------| | | python3-docutils | | | python3-fyzz | | | | python3-imaging | | | | | python3-pycryptodome | | | | python3-pyramid | | | | python3-pyramid-multiauth | | | python3-pysqlite2 | | | | | python3-rdflib | | | | python3-repoze.lru | | | python3-simpletal | | | | | python3-vobject | | | | python3-waitress | | | python3-werkzeug | | | | | python3-wsgicors | | |--------------------------------------------------------------| | Suggests | |--------------------------------------------------------------| | | | python3-pil | We also lose versioned dependencies which should not really be an issue for Recommends and Suggests.

[crypto] Use Cryptodome namespace instead of Crypto PyCryptodome comes in two flavors: “an almost drop-in replacement for the old PyCrypto library” and “a library independent of the old PyCrypto”. The former uses the Crypto namespace, and is shipped as `pycryptodome` while the latter uses Cryptodome instead and lies in the `pycryptodomex` package. Given the reason to switch to PyCryptodome is that PyCrypto in unmaintained, its probably better to avoid any mistake and mandate the specific usage of the Cryptodome namespace by requiring `pycryptodomex` instead of `pycryptodome`. A more present reason is that Debian buster will only provide a package with the separate namespace flavor. The current Recommends is not working with the current code. Although it's important to note that the package name will probably have to be changed to `python3-pycryptodomex` once https://bugs.debian.org/886291 is solved.

[pkg] Remove python3-six from Build-Depends The default branch only provides support for Python 3.

[pkg] Merge python3-cubicweb-pyramid into python3-cubicweb As suggested by Denis Laxalde, we can merge python3-cubicweb-pyramid into python3-cubicweb as all other web backend in the default branch.

[pkg] Remove explicit Depends on Python packages for python3-cubicweb As suggested by Denis Laxalde, we can let pybuild generate the list of Depends from the egg requirements. The list we get is slightly less specific in terms of needed versions, but given the age of the previous ones, it's unlikely to cause issues. Also we lose python3-six from the list… which is correct as it was removed from the dependencies in 26744ad37953.

[pkg] Fix spelling mistake in doc-base abstract

[pkg] Fix doc-base section As pointed by Lintian, the “Apps” section is only used in menu, not in doc-base. We can just remove it.

[pkg] Use “dependency package” instead of “virtual package” to describe python3-cubicweb-postgresql-support The term “virtual” was too imprecise and is not accepted for empty packages by Lintian anymore.

[pkg] Move lintian-overrides file to debian/source directory Just use the now preferred location.

[pkg] Stop using /etc/bash_completion.d cubicweb-ctl shipped an empty /etc/bash_completion.d as a leftover of shipping a bash completion script. As pointed by Lintian, this directory is not obsolete, so let's get rid of it for good.

[pkg] Tidy substvars usage in control file The documentation package was missing a Built-Using field to record the provenance of the templates. Empty packages or meta packages do not require Depends on a specific Python version, so we can remove the relevant substvars.

[pkg] Specify priority “optional” instead of the obsolete “extra” Quoting Lintian, “Since Debian Policy version 4.0.1, the priority extra has been deprecated.” So let's use “optional” instead.

[pkg] Remove build dependency on dh-systemd As suggested by Lintian, remove build dependency on obsolete package dh-systemd. Instead, specify a more precise version of debhelper 9 that added support for systemd.

[pkg] Run test suite as part of autopkgtest Add support for autopkgtest. The test “unittest” will run the test suite using pytest. As we need PostgreSQL in the test environment and open network connection, the test is marked with `isolation-container` and requires at least LXC to be run, e.g: autopkgtest -s -- lxc -s autopkgtest-sid

[pkg] Switch to Debian source format 3.0 (quilt) This forces us to base the Debian package on the source of the Python package (as produced by running `python3 setup.py sdist`). While it might seem cumbersome, this greatly reduces likelyhood of mismatch between an installation via `pip` and one using the Debian package. `dpkg-source` will also document for us what is currently in the source repository but not in the Python package. Currently the missing file are: PKG-INFO cubicweb/sobjects/test/data/cubicweb_comment/schema.py cubicweb/sobjects/test/data/cubicweb_comment/__init__.py cubicweb/sobjects/test/data/cubicweb_comment/__pkginfo__.py cubicweb/sobjects/test/data/cubicweb_card/schema.py cubicweb/sobjects/test/data/cubicweb_card/entities.py cubicweb/sobjects/test/data/cubicweb_card/__init__.py cubicweb/sobjects/test/data/cubicweb_card/__pkginfo__.py cubicweb/server/test/data-migractions/cubicweb_comment/schema.py cubicweb/server/test/data-migractions/cubicweb_comment/__init__.py cubicweb/server/test/data-migractions/cubicweb_comment/__pkginfo__.py cubicweb/server/test/data-migractions/cubicweb_localperms/schema.py cubicweb/server/test/data-migractions/cubicweb_localperms/__init__.py cubicweb/server/test/data-migractions/cubicweb_localperms/__pkginfo__.py cubicweb/server/test/data-migractions/cubicweb_basket/schema.py cubicweb/server/test/data-migractions/cubicweb_basket/__init__.py cubicweb/server/test/data-migractions/cubicweb_basket/__pkginfo__.py cubicweb/server/test/data-migractions/migratedapp/cubicweb_comment/schema.py cubicweb/server/test/data-migractions/migratedapp/cubicweb_comment/__init__.py cubicweb/server/test/data-migractions/migratedapp/cubicweb_comment/__pkginfo__.py cubicweb/server/test/data-migractions/migratedapp/cubicweb_localperms/schema.py cubicweb/server/test/data-migractions/migratedapp/cubicweb_localperms/__init__.py cubicweb/server/test/data-migractions/migratedapp/cubicweb_localperms/__pkginfo__.py cubicweb/server/test/data-migractions/migratedapp/cubicweb_basket/schema.py cubicweb/server/test/data-migractions/migratedapp/cubicweb_basket/__init__.py cubicweb/server/test/data-migractions/migratedapp/cubicweb_basket/__pkginfo__.py cubicweb/server/test/data-migractions/migratedapp/cubicweb_tag/schema.py cubicweb/server/test/data-migractions/migratedapp/cubicweb_tag/__init__.py cubicweb/server/test/data-migractions/migratedapp/cubicweb_tag/__pkginfo__.py cubicweb/server/test/data-migractions/migratedapp/cubicweb_card/schema.py cubicweb/server/test/data-migractions/migratedapp/cubicweb_card/__init__.py cubicweb/server/test/data-migractions/migratedapp/cubicweb_card/__pkginfo__.py cubicweb/server/test/data-migractions/migratedapp/cubicweb_file/schema.py cubicweb/server/test/data-migractions/migratedapp/cubicweb_file/__init__.py cubicweb/server/test/data-migractions/migratedapp/cubicweb_file/__pkginfo__.py cubicweb/server/test/data-migractions/cubicweb_tag/schema.py cubicweb/server/test/data-migractions/cubicweb_tag/__init__.py cubicweb/server/test/data-migractions/cubicweb_tag/__pkginfo__.py cubicweb/server/test/data-migractions/cubicweb_card/schema.py cubicweb/server/test/data-migractions/cubicweb_card/__init__.py cubicweb/server/test/data-migractions/cubicweb_card/__pkginfo__.py cubicweb/server/test/data-migractions/cubicweb_file/schema.py cubicweb/server/test/data-migractions/cubicweb_file/__init__.py cubicweb/server/test/data-migractions/cubicweb_file/__pkginfo__.py cubicweb/test/data-rewrite/cubicweb_localperms/schema.py cubicweb/test/data-rewrite/cubicweb_localperms/__init__.py cubicweb/test/data-rewrite/cubicweb_localperms/__pkginfo__.py cubicweb/test/data-rewrite/cubicweb_card/schema.py cubicweb/test/data-rewrite/cubicweb_card/entities.py cubicweb/test/data-rewrite/cubicweb_card/__init__.py cubicweb/test/data-rewrite/cubicweb_card/__pkginfo__.py The current manifest will duplicate several files that are stored as symlinks in the Mercurial repository, hence the long list of ignored files in `extend-diff-ignore`.

[pkg] Switch all Debian packages to Python 3

Fix sorting key for rdefs in schema viewer With changeset 234ca3cbbb46, clicking in the schema of an entity with cubicweb in py3 on "vue en boite" will probably result in an infinite spinner (which implies cw > 3.26) What happened ? This "vue en boite" used to work at least until... hg diff -c a8c1ea390400 cubicweb/schema.py @@ -993,10 +992,6 @@ class CubicWebRelationSchema(PermissionM return False return True - @deprecated('use .rdef(subjtype, objtype).role_cardinality(role)') - def cardinality(self, subjtype, objtype, target): - return self.rdef(subjtype, objtype).role_cardinality(target) - class CubicWebSchema(Schema): """set of entities and relations schema defining the possible data sets But, wait ... If I open a shell on an instance of cw 3.24 something seems off >>> list(schema['CWUniqueTogetherConstraint'].relation_definitions())[0][0].cardinality # <bound method CubicWebRelationSchema.wrapped of <constraint_of [CWUniqueTogetherConstraint,CWEType]>> We have been sorting on a method the whole time ? Is it possible what were the effects ? 1) We cannot sort function can't we ? >>> def adder(i): return lambda x: x+i >>> sorted(map(adder,range(10))) [<function __main__.<lambda>>, <function __main__.<lambda>>, ... Yes we can. 2) what does it means. >>> { adder(1) : 1 } Out[19]: {<function __main__.<lambda>>: 1} In fact the function object as a __hash__ method (which is practical for making memoizers (cache)), and return truly random results (pseudo random). My take on this patch is relations have NEVER been sorted by cardinality. No one never ever noticed. Hence, I propose to not fix a bug that never was reported.

Flake8 cubicweb/web/schemaviewer.py

[py3] changing unicode to str py2 support being dropped, passing the code in full py3 str/bytes syntax

[py3] Use sorted(key=...) instead of sorted(cmp=...)

Merge with 3.26

Added tag 3.26.11, debian/3.26.11-1 for changeset 633ca84bea58

[pkg] Version 3.26.11

Added tag debian/3.26.10-1 for changeset 0ce26fb5ab48

[debian] New upstream release

[debian] Drop override_dh_auto_install about cubicweb-ctl The script should be installed normally as it is a setuptools' entry point.

[debian] Remove test/doc Build-Depends We are not actually running tests during package build. (This was disabled for python2, now it is also for python3.) And having these build-depends makes it hard to rebuild the package.

[fix] parse cube version number without space in them

Install patched yapps for test from an http archive Instead of git which require git to be installed. Also downloading archive is faster than cloning the repository.

[server/test] make test filename uniques To avoid these pytest error when collecting the whole test suite: import file mismatch: imported module 'unittest_utils' has this __file__ attribute: cubicweb/cubicweb/server/test/unittest_utils.py which is not the same as the test file we want to collect: cubicweb/cubicweb/test/unittest_utils.py Move cubicweb/server/test/unittest_security.py to cubicweb/server/test/unittest_security.py and cubicweb/test/unittest_utils.py to cubicweb/test/unittest_server_utils.py

Merge 3.26

[dataimport] Fix case when extid is text in use_extid_as_cwuri() Extid can be bytes or text. This avoid AttributeError: 'str' object has no attribute 'decode'

[doc] fix key used in the hook example The hook's regid is check_no_subsidiary_cycle.

[doc] set_operation has been replaced by add_data See the changelog https://cubicweb.readthedocs.io/en/3.26/changes/changelog/#id14

[doc] check_cycle is a standard function

[doc] fix several typos

[pkg] make cubicweb[crypto] depends on pycryptodome instead of pycrypto pycrypto isn't maintained anymore, the last release 2.6.1 is from 2014. Use the drop-in replacement fork pycryptodome instead. pycryptodome is packaged in debian starting from stretch-backports.

[crypto] Encode unicode strings in _cypherer() pycrypto accept both bytes or string but pycryptodome, which will be introduced in next changeset, doesn't accept this. For backward compatibility, encode unicode strings in _cypherer()

[ldapfeed] FIX: Unique Key violation when synchronizing with LDAPfeed What was happening ================== The bug appears when ldapfeed tries to insert a user while another user exists exists with a different source. Simple use case to reproduce: - create a local user in cubicweb (source=system) - sync with ldap - ldapfeed will stop complaining user already exists. Without next patch the test MUST fail with message: cubicweb/server/sources/native.py:714: UniqueTogetherError The ldapfeed is thus stopped ignoring any further ldap entries. The proposal ============ Prior to this patch, the insertion was trying to create all CWUser with the computed login from eeimporter.extid2eid (method process of DataFeedLDAPAdapter). When a CWUser existed with a different cw_source ("system" for user created with cubicweb for instance), it created a conflict. To avoid collisions, in the extentities_generator method a bypass was added at the insertion layer. Prior to insertion the absence of the computed login is checked on a list of all existing login from a different source. If collision is detected, we skip the ldap record. By short circuiting at the CWUser entity insertion level we also avoid to treat CWGroup and EmailAddress related to this user. Hence ensuring a behaviour that will not break existing instances. (conservative approach: faced with ambiguity better do nothing than guess). An error message is added stating explicitly the conflict.

Skip tests for ldapsource with python >= 3.7 Until someone works on fixing these, this should make our CI green again. I tried to use setupModule() to check for python version, but pre_setup_database() is apparently called even when a SkipTest exception is raised there. So handle this in that method. (grafted from 4d68d20427dee4b6751a0f1f5511fec2a04f4782)

Flake8 crypto module

Make crypto module python3-compatible * Remove usage of unicode() and decode the base64-encoded string in encrypt(); * Encode the string received in decrypt() as (I supposed) it should come from the result of encrypt(). Add tests for this module along the way.

[mod] make ViolatedConstraint actually display useful information Move from: (Pdb++) raise some_exception *** ViolatedConstraint: To: (Pdb++) raise some_exception *** ViolatedConstraint: constraint 'cstr56c2ab4b3154f21d08b067742ce5bd9d' is being violated by the query 'ALTER TABLE cw_Bibliography ADD CONSTRAINT cstr56c2ab4b3154f21d08b067742ce5bd9d CHECK(cw_item_type IN ('journalArticle', 'note', 'book', 'thesis', 'film', 'web page', 'manuscrit', 'tapuscrit'))'. You can run the inverted constraint on the database to list the problematic rows. And save hours of debugging to actually understand what is going on.

Fix flake8 errors/warnings about import in cubicweb/_exceptions.py We either ignore errors in case of name re-exports and drop re-exports from old backwards compatibility. We fix the only occurrence of NoSelectableObject being imported from cubicweb in tests and update the import to use lgc instead.

add comment and # noqa on used import

remove unused import

[autopep8] E305 - Expected 2 blank lines after end of function or class

[autopep8] E303 - Remove extra blank lines

[autopep8] E302 - Add missing 2 blank lines

[autopep8] E301 - Add missing blank line

[enh] display CWConstraint type in its repr This will change CWConstraint repr from something like: <Entity CWConstraint 314282 [] at 140127700141904> To: <Entity CWConstraint 314282 of type UniqueConstraint [] at 140127700141904> To help improve debugging UX a little bit.

[entities] display CWConstraint type in its repr

[changelog] document pyramid.ini automatic generation

Flake8 cubicweb/pyramid/test/test_hooks.py

[pyramid/doc] bad docstring

[pyramid/misc] remove useless variable

[pyramid/test] add test for pyramid.ini generation

Added tag 3.26.10 for changeset 74cc5de1ee6c

[pkg] Version 3.26.10

Move CWSchemaTC to RQLExpressionTC The test actually doesn't require a cubicweb schema, except for testing EmailAdress which is out of scope of the test (testing RQLExpression.transform_has_permission()). Just move the test in existing RQLExpressionTC inheriting from TestCase and rename the tests to be more explicit.

[skeleton] Drop py27 tox environment Since we dropped py2 support from cubicweb, it does not make much sense to keep it in skeleton from now on.

[skeleton] Do not ship Debian and RPM packaging files in sdist

[skeleton] Add a check-manifest tox environment

Fix pyramid tests Followup fa292e9 which require cubicweb.session.secret to be set.

Fix flake8 and check-manifest Followup fa292e9

Merge 3.26 This fixes tests with psycopg2

DeprecationWarning: In future versions of Waitress clear_untrusted_proxy_headers will be set to True by default. You may opt-out by setting this value to False, or opt-in explicitly by setting this to True. Source: https://docs.pylonsproject.org/projects/waitress/en/stable/arguments.html?highlight=clear_untrusted_proxy_headers > This tells Waitress to remove any untrusted proxy headers ("Forwarded", > "X-Forwared-For", "X-Forwarded-By", "X-Forwarded-Host", "X-Forwarded-Port", > "X-Forwarded-Proto") not explicitly allowed by trusted_proxy_headers. According to grep we don't use any of those headers so let's turn it on for security reasons.

DeprecationWarning: The default pickle serializer is deprecated as of Pyramid 1.9 and it will be changed to use pyramid.session.JSONSerializer in version 2.0. Explicitly set the serializer to avoid future incompatibilities . See "Upcoming Changes to ISession in Pyramid 2.0" for more information about this change. As describe here https://docs.pylonsproject.org/projects/pyramid/en/1.10-branch/narr/sessions.html#pickle-session-deprecation use a serializer that fallback on pickle if needed to avoid impacting users.

[pyramid/enh] generate pyramid.ini "create" and on "pyramid" command if needed

DeprecationWarning: The SafeConfigParser class has been renamed to ConfigParser in Python 3.2. This alias will be removed in future versions. Use ConfigParser directly instead.

Account for new psycopg2 exception classes mapping From psycopg2 >= 2.8, specific exceptions are raised corresponding to postgresql errors. E.g. a CheckViolation exception is raised instead of a generic IntegrityError previously when a constraint violation occurs. The way we intercept database errors, especially for constraint violation, is not compliant with that because we do not catch subclasses of IntegrityError in native source's doexec() method. We fix this by checking for the presence of IntegrityError error in exception class's mro. This is still overcomplicated and clumsy, because we still use string comparison, but this is the best we can do as far as I know. (A better fix would be 'isinstance(ex, IntegrityError)' but we have no engine-independent error classes, so this is not possible. Something like sqlalchemy's DBAPI Errors [1] might help: https://docs.sqlalchemy.org/en/latest/errors.html#dbapi-errors)

[cubicweb-ctl] remove "cubicweb-ctl wsgi" command following pyramid standardization

Use secure hash algorithm in WebConfiguration.sign_text Fix: PendingDeprecationWarning: HMAC() without an explicit digestmod argument is deprecated. The default hash algorithm used by hmac.new is md5. As of today, md5 is so weak that it's the equivalent of plaintext and can't be considered to be secured at all. Therefor, we switch to a secure hash algorithm. The rational for choosing sha3_512 is: * the recommended algorithm is at least sha_256 * the stronger, the more secured and sha3_512 is the stronger available * thinking about the future this should keep this part of the code safe long enough before people think about checking it again You can read more about choosing a secure hash algorithm in the NIST recommendations https://csrc.nist.gov/Projects/Hash-Functions/NIST-Policy-on-Hash-Functions This code modification should normally be transparent since check_text_sign is exactly this code 'self.sign_text(text) == signature' and that sign_text is only used in combination with it. The only impact is that the hash is going to move from 32 char to 128 which might make html page a bit bigger and that sha3_512 is slow to compute (which is a good thing for security)

[enh] don't catch all exceptions in notification hooks during tests

[doc] add a reference to static-messages.pot usage

[doc] Clarify sentences

[doc] Fix format and typo

Fix DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated

Fix DeprecationWarning: invalid escape sequence

[devtools/testlib] avoid hidding AttributeError in create_user() commit() might raise a AttributeError too. Use getattr(req, 'cnx', req) instead, which is a form already used to get the real cnx in some code: cubicweb/rset.py:577: cnx = getattr(self.req, 'cnx', self.req) cubicweb/schema.py:353: with getattr(_cw, 'cnx', _cw).security_enabled(read=False): We could use if hasattr(req, 'commit') here too but it lead to 3 additionals lines. Maybe we should have commit() and rollback() on cubicweb.web.request.ConnectionCubicWebRequestBase too ?

Fix DeprecationWarning: invalid escape sequence \

DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead

[mod] remove backward compatible code for passlib and force modern version

DeprecationWarning: Please use assertEqual instead.

DeprecationWarning: Please use assertRaisesRegex instead.

DeprecationWarning: inspect.getargspec() is deprecated since Python 3.0, use inspect.signature() or inspect.getfullargspec()

[README] add testing instructions

Add pytest-subtests to dev requirements This pytest plugin adds support for unittest's TestCase.subTest (for Python >= 3.4). Since we we now only support python3 and since we use a fair amount of these, let's benefit of better reporting by using this plugin.

[test] use unittest.mock instead of mock library Now that we use Python 3 only, this is possible.

Drop python2 support This mostly consists in removing the dependency on "six" and updating the code to use only Python3 idioms. Notice that we previously used TemporaryDirectory from cubicweb.devtools.testlib for compatibility with Python2. We now directly import it from tempfile.

Merge with 3.26

Remove unused imports in cubicweb.pyramid.config

Added tag 3.26.9 for changeset 4d6909de765a

[pkg] Version 3.26.9

[pyramid/security] use cryptographically secure random generator Based on django source code in case SystemRandom is not available. According to python documentation: https://docs.python.org/2/library/random.html > Warning > The pseudo-random generators of this module should not be used for security > purposes. Use os.urandom() or SystemRandom if you require a cryptographically > secure pseudo-random number generator.

Reclosing branch after merge

Merging heads of old, closed branch

Reclosing branch after merge

Merging heads of old, closed branch

[doc/changes] document that legacy cube support has been dropped

[doc] replace legacy import to new style cube import in various places

[web/views] drop deprecated module cubicweb.web.views.timeline

[web/views] drop deprecated module cubicweb.web.views.massmailing

[web/views] drop deprecated module cubicweb.web.views.isioc

[web/views] drop deprecated module cubicweb.web.views.embedding

[web/views] drop deprecated module cubicweb.web.views.igeocodable

[pyramid/core] drop fallback import of legacy cubes

[cwconfig] drop importing legacy cubes This finally drop support for legacy cubes. In cwconfig, don't load modules in the "cubes" namespace. In cube_pkginfo() handle CW_MIGRATION_MAP which wasn't working with new style cube. Drop all method specific to legacy cubes importer: cubes_search_path(), extrapath(), cls_adjust_sys_path() Drop environment variables related to legacy cubes: CW_CUBES_PATH and CW_CUBES_DIR

[doc] better documentation for "cubicweb-ctl newcube" cubicweb-ctl newcube create cubicweb-<name> in current directory.

[test] make unittest_cwconfig.py does not depend on legacy cube machinery By dropping legacy cube tests in various places.

[devtools/test] drop legacy cube i18n tests test_i18ncube_legacy_layout() is droped and make others tests uses a new style cube "cubicweb_i18ntestcube", so it doesn't depends on legacy cube machinery.

[server/test] make unittest_migractions.py use new style cubes MigrationTC used old cube layout in cubicweb/server/test/data-migractions/cubes, move them to new style cubes as "cubicweb_<cubename>" in "apphome" (cubicweb/server/test/data-migractions/) which is added to PYTHONPATH by CubicWebTC.

[cwconfig] drop _cube_modname Since we drop support for legacy cubes, this is equivalent to _cube_pkgname()

[cwconfig] drop legacy cube importer We don't want to support legacy cubes anymore.

Drop more deprecated code This follows up on changeset a8c1ea390400, in which code deprecated using logilab.common.deprecation got dropped. Now we also drop code deprecated using stdlib's warn(<msg>, DeprecationWarning). Notice that, as a consequence of dropping old/new etypes aliases in cubicweb/schema.py, we drop the import ETYPE_NAME_MAP (no longer needed); but since other modules imported that name from cubicweb.schema, we need to update the import statement to use "cubicweb" directly.

Flake8 cubicweb/cwvreg.py

Drop deprecated "boxes" and "contentnavigation" registries

Drop deprecated CWCache entity type

Update i18n after "recent" changes

Drop repair_file_1-9_migration.py

Drop a left-over 3.22 deprecation warning

[doc] fix reference to ~src/cubes in tutorial ~src/cubes refer to the $HOME/cubes directory of "src" user while ~/src/cubes refer to $HOME/src/cubes directory of current user.

[doc] replace cubicweb-ctl start by cubicweb-ctl pyramid Since twisted support is gone.

[doc] make the tutorial use a python3 virtualenv

[doc] Update README "Getting started" * install using pip and with pyramid in a virtualenv * link to pyramid.ini documentation * Use pyramid to start application since twisted support is gone

Flake8 cubicweb/devtools/httptest.py

Remove Twisted web server Twisted web server is not used anymore and has been superseded by pyramid many years ago. Furthermore, our usage is not compatible with Python 3. So we drop the "etwist" sub-package. As a consequence, "all-in-one" configuration type gets dropped as it was Twisted-specific. We resurrect it in cubicweb/pyramid/config.py by only keeping options used by the "pyramid". Similarly, we introduce a AllInOneCreateHandler in cubicweb/pyramid/pyramidctl.py that is basically the one that lived in cubicweb/etwist/twctl.py and is used to create the "all-in-one" instance. Added a TODO here about "pyramid.ini" that could be generated at the end of bootstrap() method. In cubicweb/devtools/httptest.py, CubicWebServerTC is now equivalent to CubicWebWsgiTC and the latter is dropped.

Separate twisted-specific configuration from WebConfigurationBase class This is in preparation of dropping the Twisted web server backend. We thus draw a clearer line between what's specific to twisted from what's generic by moving WebConfigurationBase class into cubicweb/web/webconfig.py and only keeping twisted-related in AllInOneConfiguration (still living in cubicweb/etwist/twconfig.py).

Flake8 cubicweb/web/webconfig.py

Flake8 cubicweb/etwist/twconfig.py

Remove ctl commands using the Twisted backend This backend is not maintained, very likely no longer used in any recent instance, and not supported in Python 3.

Flake8 cubicweb/cwctl.py * run autopep8 on the file * fixed unused imports * rename "version" import as "cw_version" in order not to shadow local variables

Remove _handle_win32() method from "create" command class This method generates code in a file using etwist module that we are about to drop.

Skip tests for ldapsource with python >= 3.7 Until someone works on fixing these, this should make our CI green again. I tried to use setupModule() to check for python version, but pre_setup_database() is apparently called even when a SkipTest exception is raised there. So handle this in that method.

[doc] Update cubicweb-ctl page about how to run an instance * remove mention of start/stop/restart/status commands that do not exist anymore * briefly mention the "pyramid" and point to respective documentation page * mention that a WSGI server should be used in production

[doc] Drop mention of the "pyramid" cube This cube got merged into cubicweb several versions ago.

[cli/enh] on 'shell' or migration command, use ipython if available

[web/test] drop dependency on third party cubes Drop dependency on cubicweb-file, cubicweb-blog and cubicweb-tag for cubicweb/web/test Copy required parts of cubes (schema, entities, views and hooks) into cubicweb/web/test/data/cubicweb-<cube> that make tests pass.

[test] drop dependency on third party cubes Drop dependency on cubicweb-file, cubicweb-localperms, cubicweb-tag and cubicweb-comment for cubicweb.test and cubicweb.sobjects.test Move cubicweb/test/data/libpython/cubicweb_* to cubicweb/test/data, so it's included in PYTHONPATH. Add missing parts of schema, entities, views, hooks that make tests pass.

[test] drop CubicWebConfigurationWithLegacyCubesTC This is used to test legacy cube import which is planned for removal in cubicweb 3.27, so drop the code.

[test] drop unittest_cubes.py This is used to test legacy cube import which is planned for removal in cubicweb 3.27, so drop the code.

[server/test] do not depend on third party cubes Drop requirement of basket, card, comment, file, localperms and tag cubes for cubicweb.server tests. Copy needed part of schema, entities, views, hooks into the cubicweb/server/test/data directory (which is included in PYTHONPATH for CubicWebTC tests). For migration tests which use different "datapath", use symlinks to have dependent cubes importables.

[server/test] drop unused variable initialization in OperationsTC.setUp() Variable "self.hm" is unused

[server/test] bootstrap config in setUpModule function This is test initialization, so use the setUpModule function for this

[schema] drop monkeypatch "name_for" on BaseConstraint BaseConstraint doesn't have "name_for" method, this is just an addition for cubicweb needs. Implement this as a function instead of a monkeypatch to avoid having "name_for" not defined because cubicweb.schema hasn't been imported. In particular this fix running cubicweb/server/test/unittest_schema2sql.py::SQLSchemaTC::test_known_values alone.

[fix] this check was too restrictive and broke tests

[fix] exception type has changed

[server.rqlannotation] refactor to make class SQLGenAnnotator consistent _annotate_select() takes SQLGenAnnotator as a first argument and is not used anywhere except in SQLGenAnnotator. This looks like a method to me. Let us fold it into its class.

Drop most of deprecated code As in previous changesets, most code has been deprecated since a very long time. This changeset concerns "core" cubicweb modules (e.g. "server", "appobjects") for which it's not easy to handle change atomically. In cubicweb/server/querier.py, we adjust empty_rset() function as a result of "rqlst" argument of ResultSet being dropped. (There was no use of the keyword argument anyways.)

Drop deprecated code in cubicweb.hooks Changelog entry will come in a later changeset.

Drop deprecated code in cubicweb.entities Also dropped an unused import along the way. Changelog entry will come in a later changeset.

Drop deprecated code in cubicweb.devtools Changelog entry will come in a later changeset.

Drop deprecated code in cubicweb.dataimport We actually drop *most* of deprecated code in this subpackage. What's left is the MetaGenerator class in (cubicweb/dataimport/stores.py) which is tricky to remove as it's still internally used by cubicweb. Changelog entry will come in a later changeset.

Drop most deprecated code in cubicweb.web This includes both Python and JavaScript code. Most code has been deprecated since a very long time (versions 3.1x mostly, more than 5 years ago, and a few things until 3.24). There are still a few deprecated things, most of which are actually used within cubicweb (or in cubes we depend on for tests). I don't know what to do with these... Removed a couple of unused imports along the way. Changelog entry will come in a later changeset.

[web] use Separator "component" instead of deprecated BoxSeparator widget

Flake8 cubicweb/web/box.py

[test] Remove upper bound for cubicweb-localperms Version 0.4.0 does not use a deprecated API that we are planning to drop in forthcoming changesets. This requires changing expected values in unittest_cwconfig.py where we scan install cubes. * In CubicWebConfigurationTC.test_available_cubes(), localperms disappears from the list of expected cubes because this test method will look for new style cubes in located in cubicweb/test/data/libpython/ (we patch pkg_resources.iter_entry_points()) as well as old style cubes installed in $PREFIX/share/cubicweb/cubes; since the new cubicweb-localperms version is new-style, it won't be catched by this method. * In CubicWebConfigurationWithLegacyCubesTC.test_available_cubes(), localperms is still listed, but now with the package name, because this method will simply look at installed cubes whether old or new style.

[test] Make the expected value explicit in CubicWebConfigurationWithLegacyCubesTC.test_available_cubes() This will help with the next changeset, and avoiding computation in tests is a good thing in general.

[test] Inline _test_available_cubes() method in cwconfig tests Improves readability.

[DX] adding error messages to {ResultSet,ActionsRegistry}.possible_actions Try to make it more accessible, when we first encountered these errors it it didn't made any sense to us.

[server/querier] remove useless indirection QuerierHelper._annotate

[server] remove useless indirection ExecutionPlan.sqlannotate Simpler is better. Use directly ExecutionPlan.querier.sqlgen_annotate.

[web] removed unused imports in basecomponents

[test] Depends on psycopg2-binary Avoids the following warning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi

[doc/changelog] the removal of next_tabindex produce an error not in the changelog When one tried to migrate to 3.25, the error "class X initialization doesn't have a settabindex argument" can happen but the word "settabindex" is not present in the changelog which makes it hard to find to which modification this error is related.

Merge with branch 3.26

[py3.7,test] Use cubicweb-file >= 2.2.2 This includes changeset aa4a3a3d0c18 fixing import of os.errno.

[flake8] Fix errors in cubicweb/ext/test/unittest_rest.py

[test] Shorten prefix string in test_rql_role_with_unknown_vid With Python3.7, this test fails as the error message returned by rest_publish() is: <p>an error occurred while interpreting this rql directive: ObjectNotFound('toto')</p> while we previously assumed a comma after 'toto'. Fixing this by only shortening the prefix used in out.startswith().

[flake8] fix errors in cubicweb_i18ntestcube/views.py test file

[test] Pass __module__ upon instantiation of RegistrableInstance This gets rid of the deprecation warning from logilab.common.registry: DeprecationWarning: instantiate MyAFS with __module__=__name__ and solves a test error on Python 3.7.

Added tag 3.26.8 for changeset 91f75319a726

[pkg] Version 3.26.8

[doc] didn't had any questions

[doc] missing import, cw now require this explicitely

[doc] add actual link to previous post

[doc] missing import in code example

[doc] add a note because uiprops.py is not generated anymore by default

[doc] recommend pip instead of apt for installing things According to a discussion on XMPP we should recommend pip instead.

[doc] fix, link to internal doc since it's not the blog post anymore

[doc] also show command with sqlite since it's recommended

[doc] fix, the cube name was used, it was the instance named that was needed

[doc] it's a tutorial, not a blog post anymore

[doc] make example more redable

[doc] uses :file:`name` everywhere + indicate file when it's not present

[doc] english typos

[doc] pep8/retab in code examples

[doc] correct import for i18n

[doc] pep8 and retab in tutorial code example

[doc] correct pip command to have needed dependancies to use the start command I know we want to remove this command but in the meantime having this documentation working seems like something good.

[doc] explain the difference between a relation and an attribute

[doc] plural

[doc] misc, put every reference to something in the code between backquotes

[doc] bold to make reading more obvious

[doc] style, switch to bullet points list to make things easier to read

[doc] explains what the <X.Y.Z> means

[doc] command output style has changed

[doc] always put file name for code example and uses :file:`path` syntax everywhere

[doc] explicitely give the local server url to make this step more obvious

[doc] start the server in debug mode, it's easier during learning

[doc] recommend sqlite during tutorial because it's simplier to use

[doc] add a link to every cube documentation page

[doc] make this step more obvious

[doc] move to latest CW version for cube creation

[doc] missng import for i18n in example

[doc] unused import in example

[doc] output has changed

[doc] blanks spaces to allow example to breath and be more lisible

[doc] default tests template has changed

[doc] default generated filename as changed

[doc] rst syntax bug, blank line needed before item list

[doc] add link to cubes contribution document

[doc] link to current documentation, not old cubicweb.org doc

[doc] translate french to english

[doc] typo, 'CW' doesn't exist in the example

[doc] always indicate the file in which the code should be put

[doc] respect pep8 in code examples

[doc] serie a small improvments

[doc] typo, the field is "name" not "title"

[doc] fix, command "create" is missing base cube

[doc] improve blog tutorial first steps and explain common errors

[doc] new behavior in virtualenv since 3.26

[skeleton] use python3 flake8 In tox "flake8" environment, use python 3 virtualenv, so we can check for python3 incompatible syntax.

[skeleton] drop whitelist_externals for flake8 whitelist_externals makes sense when used with sitepackages=true. In this case it will always rely on $VIRTUAL_ENV/bin/flake8 so, the whitelist_externals has no effect.

Fix flake8 issues since release 3.6.0 Flake8 had a new release which raise new issues, namely: W504: line break after binary operator F841: local variable 'ex' is assigned to but never used W605: invalid escape sequence F821: undefined name 'buffer' (noqa seems the only way to avoid this false positive) Also pin flake8>=3.6 in our tests and make explicit that we use python3 to run flake8 tests.

Fix typo in RDFS URI

[server] Fixed issue with the adapters cache of a cloned EditedEntity The adapters' cache of the entity within an EditedEntity used to be copied along the entity itself during a cloning operation of the EditedEntity. This raises an issue when the adapters' cache is not empty and contains cached values linking to transaction-specific data (or the transaction itself). This prevents the clone EditedEntity being used in a transaction outside the one that created it. This change resets the adapters' cache of the new copy so that no cached value remain. The new copy can then be used in a new transaction afterwards.

[py37] `async` is now a reserved keyword

[skeleton] Use "py3" as tox environment instead of "py34"

[views/startup] drop a duplicate rql query in manage view Counting entities in `nb` is already done above this code, regression introduced by a85db6fa

Depend on cubicweb-file >= 2.0.0 I applied V1 of the patch, but V2 contained this extra hunk...

[test] Update webconfig's tests accounting for cubicweb-file being now a package WebconfigTC.test_locate_all_files and WebconfigTC.test_locate_resource assumed that cubicweb-file was installed in <share>/cubes/file whereas, from the latest release, it is now a standard Python package thus installed in site-packages as cubicweb_file. We simply update base directory of these two tests.

[widgets.js] fix obvious bug detected by sonar

[views/uicfg] obvious bug detected by sonar

[style] Make toolsutils.py flake8-clean Run `autopep8 --max-line-length 100 -i cubicweb/toolsutils.py` and fixed unused imports.

[test/rset] typo

[test/rset] flake8

[py3] import configparser from six.moves instead of using a try/except Using six is better than the try/except because it clearly indicates compatibility code. Also, when using tools such as vermin which relies on AST, it helps getting rid of false compatibility errors.

[doc] didn't had any questions

[doc] missing import, cw now require this explicitely

[doc] add actual link to previous post

[doc] missing import in code example

[doc] add a note because uiprops.py is not generated anymore by default

[doc] recommend pip instead of apt for installing things According to a discussion on XMPP we should recommend pip instead.

[doc] fix, link to internal doc since it's not the blog post anymore

[doc] also show command with sqlite since it's recommended

[doc] fix, the cube name was used, it was the instance named that was needed

[doc] it's a tutorial, not a blog post anymore

[doc] make example more redable

[doc] uses :file:`name` everywhere + indicate file when it's not present

[doc] english typos

[doc] pep8/retab in code examples

[doc] correct import for i18n

[doc] pep8 and retab in tutorial code example

[doc] correct pip command to have needed dependancies to use the start command I know we want to remove this command but in the meantime having this documentation working seems like something good.

[doc] explain the difference between a relation and an attribute

[doc] plural

[doc] misc, put every reference to something in the code between backquotes

[doc] bold to make reading more obvious

[doc] style, switch to bullet points list to make things easier to read

[doc] explains what the <X.Y.Z> means

[doc] command output style has changed

[doc] always put file name for code example and uses :file:`path` syntax everywhere

[doc] explicitely give the local server url to make this step more obvious

[doc] start the server in debug mode, it's easier during learning

[doc] recommend sqlite during tutorial because it's simplier to use

[doc] add a link to every cube documentation page

[doc] make this step more obvious

[doc] move to latest CW version for cube creation

[doc] missng import for i18n in example

[doc] unused import in example

[doc] output has changed

[doc] blanks spaces to allow example to breath and be more lisible

[doc] default tests template has changed

[doc] default generated filename as changed

[doc] rst syntax bug, blank line needed before item list

[doc] add link to cubes contribution document

Merge 3.26

[doc] link to current documentation, not old cubicweb.org doc

Explicitly require repoze.lru for Pyramid tests This used to be pulled by pyramid but the dependency got dropped in pyramid version 1.10, see: https://docs.pylonsproject.org/projects/pyramid/en/latest/whatsnew-1.10.html#backward-incompatibilities We already have an explicit dependency in setup.py, but it was missing for tests. Perhaps we could simply use '.[pyramid]' in this requirements file? Not sure what side effect it'd have though...

[cubicweb-ctl] make uid option obvious

[README] point to xmpp room for contributors

[doc] link to README regarding how to contribute to the code base

[README] improve contribution section with links and command examples

[README] point to logilab's jenkins in tests section

[pyramid] improve pyramid warning with link to doc on how to write pyramid.ini file

[doc] translate french to english

[doc] typo, 'CW' doesn't exist in the example

[doc] always indicate the file in which the code should be put

[doc] respect pep8 in code examples

[doc] serie a small improvments

[doc] typo, the field is "name" not "title"

[doc] fix, command "create" is missing base cube

[doc] improve blog tutorial first steps and explain common errors

[doc] how to install pyramid or twisted application server

[doc] new behavior in virtualenv since 3.26

[skeleton] use python3 flake8 In tox "flake8" environment, use python 3 virtualenv, so we can check for python3 incompatible syntax.

[skeleton] drop whitelist_externals for flake8 whitelist_externals makes sense when used with sitepackages=true. In this case it will always rely on $VIRTUAL_ENV/bin/flake8 so, the whitelist_externals has no effect.

Add changelog note for running tests with pytest-xdist

Make test database template creation concurrent build_db_cache() is used in tests to create test database templates, i.e. DEFAULT_EMPTY_DB_ID (which is __default_empty_db__) and custom template database using CubicwebTC test_db_id/pre_setup_database API. When running tests in parallel using multiple processes, build_db_cache() may try to build the same database twice. Avoid this by adding synchronisation of process by using a file lock. So when two processes require the same template database, one build the database and others wait it to be created. Use filelock (https://github.com/benediktschmitt/py-filelock) library to have a portable (unix / windows) way for handling locks. Also filelock is packaged in debian: https://packages.debian.org/source/python-filelock

Make test database cache lookup concurrent has_cache() is used to dertermine if the template database already exists or not. When running tests accross multiple process, has_cache() wasn't aware of databases created by other processes because it used a cache to generate the cache... Since the discovering discover_cached_db() time is quite reasonable (it's just about listing file in a directory), let re-generate the cache on cache miss.

Fix flake8 issues since release 3.6.0 Flake8 had a new release which raise new issues, namely: W504: line break after binary operator F841: local variable 'ex' is assigned to but never used W605: invalid escape sequence F821: undefined name 'buffer' (noqa seems the only way to avoid this false positive) Also pin flake8>=3.6 in our tests and make explicit that we use python3 to run flake8 tests.

Stabilize flake8 tests on 3.26 branch Use basepython=python2 or flake8 installed with python3 will raise "[F821] undefined name 'buffer'" Pin flake8 < 3.6 which has new checks that doesn't pass on our current code base.

Fix typo in RDFS URI

[doc] Do not rely on lgc's sphinx_ext This module is broken (getting an ImportError) and I don't want to fix it. The only usage we make of it is for the "autodocstring" directive, and I don't know what it does. Based on the context, replacing autodocstring with either automodule or autoclass and removing sphinx_ext from required Sphinx extensions.

Added tag 3.26.7 for changeset ee4ad63c91e5

[pkg] Version 3.26.7

Fix possible double import of cubes For a new-style cube, if we import "cubicweb_<name>" and then import "cubes.<name>", the cube will be imported twice. cubes.<name> and then cubicweb_name is ok though... When using pyramid, we try to find which cube define a "includeme" by importing them with the name "cubes.<name>", so we (possibly ?) double import all new-style cube. This case may also occur in tests with PyramidCWTest. Touching the import loader to fix this scares me, so let's fix this by testing "cubicweb_<name>" *before* "cubes.name" (for old style cubes, importing cubicweb_<name> raise ImportError).

[autoform] Make RelationField.fromcardinality a class method factory Instead of having the "RelationField" class hard-coded in this factory method (thus preventing any subclass to properly use it), we make that method a classmethod and instantiate the actual class instead of the base one.

[autoform] Use fieldclass from kwargs in guess_field for non-final relations This follows up on af969080e7e6 in which a "fieldclass" kwarg has been added in EntityFieldsForm.field_by_name() before calling guess_field(). In this changeset, this value was only used for final relation but the way non-final relation actually also changed since we also use guess_field from these relation (previously, the field class specified through uicfg was instantiated directly in EntityFieldsForm.field_by_name()). So in the non-final case in guess_field() we simply try to retrieve "fieldclass" from kwargs and fall back to RelationField.

[py37] `async` is now a reserved keyword

Merge 3.26

Added tag 3.26.6, debian/3.26.6-1 for changeset 8362503a9248

[pkg] version 3.26.6

[py3] Replace unicode keyword which is unknown in python3

[skeleton] Use "py3" as tox environment instead of "py34"

Added tag 3.26.5 for changeset 76d7b0aa94d2

[pkg] Version 3.26.5

Merge with 3.26

[py3] Use "utf-8" as input encoding for docutils in rest extension Docutils says that "unicode" is an unknown encoding on Python3. Not sure where "unicode" as an encoding comes (it's there since showtime) but "utf-8" seems more appropriate. From now on, rendering of CubicWeb's rst directive (e.g. ..winclude) works on Python 3. Accordingly, we extend wdoc's test to demonstrate this. For this we need to install docutils in tox's web environment.

[py3] Avoid setting attribute on Element instance in wdoc view This change is needed to make "wdoc" view work on Python 3. Indeed, before this patch, we used to set a "parent" attribute on Element instances, which is not permitted since Python 3.3 as xml.etree.ElementTree uses the C implementation by default. To get rid of this "parent" attribute, we rework the XML nodes processing logic to return and retrieve a node along with its parent in several place of the code. Namely, build_toc_index() now accepts the "parent" value for "node" and inserts it in the "index" dict (thus avoid `node.parent = <value>`). Respectively, every query to the "index" dict is updated to also retrieve the "parent" (thus avoiding `<value> = node.parent`). Later in InlineHelpView class, the "index" dict is bound to "tocindex" attribute so we adjust how it is queried in respective methods. Tests cubicweb/web/test/unittest_views_wdoc.py now pass on Python 3.

[test] Add a trivial test for "wdoc" view We just make sure it gets rendered without errors (which wasn't the case before the previous patch). It does not work on Python 3; this will be fixed in next changeset.

[py3] Open rst file in text mode in wdoc view Otherwise, UStringIO complains that it receives bytes.

[ext/rest] Drop "handle_io_errors" argument to docutils' FileInput This argument is deprecated and ignore in docutils since version 0.10. Besides, its default value is None which is the same as we passed.

[wdoc] Remove fallback about xml.etree module Module xml.etree exists since python 2.5.

[wdoc] Avoid usage of deprecated xlm elementree getchildren() method Use list(node) as suggested by the deprecation warning.

[style] Make cubicweb/web/views/wdoc.py flake8-clean

[py3] Use six.text_type instead of unicode in wdoc views

[views/startup] drop a duplicate rql query in manage view Counting entities in `nb` is already done above this code, regression introduced by a85db6fa

Merge with 3.26

Added tag 3.26.4, debian/3.26.4-1 for changeset e1c1af68db86

[pkg] Version 3.26.4

[py3] ensure AjaxController return bytes publish() must return bytes, when the @ajaxfunc decorated method return None, this raised: TypeError: You cannot set Response.body to a text object (use Response.text)

[py3] Let InstanceCommand.run_arg() always return an int The caller of this method (which is method "run_args") does `max(status, self.run_arg(appid))` and run_arg might return None if the underlying <cmdname>_instance method returns None. On python3 max(1, None) breaks with "TypeError: unorderable types: int() > NoneType()". So we enforce run_arg() method to return an integer even if the underlying method returns None. This fixes call of "cubicweb-ctl i18ninstance" with Python3.

[test] Use cubicweb-file < 2.0.0 From version 2.0.0, cubicweb-file moved to new-style layout (Python package) but we still expect it to be an old-style cube in CubicWeb's tests (WebconfigTC in web/test/unittest_webconfig.py).

[py3] Decode subprocess stdout in toolsutils.show_diffs() On Python 3, subprocess' pipes are bytes stream so we must decode them before printing. Without this change, diff of configuration files (e.g. all-in-one.conf as displayed upon instance upgrade) will rendered as a single line on Python 3.

[python3] make sure staticcontroller returns a bytes instance The ``publish()`` method of controller is supposed to return a bytes intsance. Returning a text_type instance will crash later on in the Pyramid response handler when assigning content to ``response.body`` (at least in Python3):: >>> from pyramid.response import Response >>> r = Response() >>> r.body = '' Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/adim/.virtualenvs/…/response.py", line 562, in _body__set raise TypeError(msg) TypeError: You cannot set Response.body to a text object (use Response.text)

[python3] use inspect.getfullargspec instead of inspect.getargspec The latter is now deprecated

[py3] ensure AjaxController return bytes publish() must return bytes, when the @ajaxfunc decorated method return None, this raised: TypeError: You cannot set Response.body to a text object (use Response.text)

[py3] Let InstanceCommand.run_arg() always return an int The caller of this method (which is method "run_args") does `max(status, self.run_arg(appid))` and run_arg might return None if the underlying <cmdname>_instance method returns None. On python3 max(1, None) breaks with "TypeError: unorderable types: int() > NoneType()". So we enforce run_arg() method to return an integer even if the underlying method returns None. This fixes call of "cubicweb-ctl i18ninstance" with Python3.

Depend on cubicweb-file >= 2.0.0 I applied V1 of the patch, but V2 contained this extra hunk...

[test] Update webconfig's tests accounting for cubicweb-file being now a package WebconfigTC.test_locate_all_files and WebconfigTC.test_locate_resource assumed that cubicweb-file was installed in <share>/cubes/file whereas, from the latest release, it is now a standard Python package thus installed in site-packages as cubicweb_file. We simply update base directory of these two tests.

[widgets.js] fix obvious bug detected by sonar

[views/uicfg] obvious bug detected by sonar

[style] Make toolsutils.py flake8-clean Run `autopep8 --max-line-length 100 -i cubicweb/toolsutils.py` and fixed unused imports.

[py3] Decode subprocess stdout in toolsutils.show_diffs() On Python 3, subprocess' pipes are bytes stream so we must decode them before printing. Without this change, diff of configuration files (e.g. all-in-one.conf as displayed upon instance upgrade) will rendered as a single line on Python 3.

[python3] make sure staticcontroller returns a bytes instance The ``publish()`` method of controller is supposed to return a bytes intsance. Returning a text_type instance will crash later on in the Pyramid response handler when assigning content to ``response.body`` (at least in Python3):: >>> from pyramid.response import Response >>> r = Response() >>> r.body = '' Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/adim/.virtualenvs/…/response.py", line 562, in _body__set raise TypeError(msg) TypeError: You cannot set Response.body to a text object (use Response.text)

[python3] use inspect.getfullargspec instead of inspect.getargspec The latter is now deprecated

[utils] Fixes compatiblity of QueryCache with expected dict interface QueryCache is expected to mimick the interface of the dict built-in. The current __iter__ implementation breaks this expectation by iterating over key, value pairs instead of only the keys. This changeset fixes this issue by changing the __iter__ implementation to iterate over the keys in the cache and providing an implementation of the items method with a contract identical to the dict build-in (return a copy of the key- value pairs within the dictionary as a list).

[test/statsd] add tests for cubicweb.statsd_logger

[utils] Implements __iter__ on QueryCache In order to be able to iterate over the currently cached queries (so that they may be inspected and invalidated), this changeset implementes __iter__ on QueryCache.

[utils] Added new get method to QueryCache The new get method enables the retrieval an item in the cache by specifying a default value so that in the absence of an item for the key, the default value is returned.

[test/rset] typo

[utils] Fixed issue in clearing QueryCache When trying to add an item in a full QueryCache, the cache tries to make room. It tries to list and remove non-permanent items with a transient counter (number of times it has been requested). However, there is a pathological case where items could be non-permanent but still not have a transient counter because they were added, but never requested. In some cases, the full cache could be flushed, including the permanent items. This changeset attempts to fix this issue by only dropping the non-permanent items that did not hav a transient counter.

[statsd] add a statsd_timethis context manager This context manager allows to report to statsd the execution time of any piece of code, without the need for it to be a function or a method, like:: def some_function(): do_stuff() with statsd_timethis('some context message'): do_that() ensure_this()

[test/rset] add tests for ResultSet.first() and .last() methods

[test/rset] flake8

[rset] update copyright

[rset] add convenient first() and last() methods on ResultSet These make life easier for the heavy user of the CW shell. Also aliases entities() with all(); so we end with a pretty consistant set: - .one() - .first() - .last() - .all()

[server] Fixed issue with the adapters cache of a cloned EditedEntity The adapters' cache of the entity within an EditedEntity used to be copied along the entity itself during a cloning operation of the EditedEntity. This raises an issue when the adapters' cache is not empty and contains cached values linking to transaction-specific data (or the transaction itself). This prevents the clone EditedEntity being used in a transaction outside the one that created it. This change resets the adapters' cache of the new copy so that no cached value remain. The new copy can then be used in a new transaction afterwards.

[rql2sql] fix handling of the ORDERBY clause when the aggregation function is "embedded" ie. a situation like: Any X GROUPBY X ORDERBY COUNT(Y) WHERE ... works fine since the aggregation function (COUNT) if the first one, whereas a situation like the proposed test: Any X GROUPBY X ORDERBY ABS(COUNT(Y)) WHERE ... failed due to the fact we only checked the top aggratation function (ABS). Closes #17108671

Merge 3.26

[pkg] Set version to 3.27.0.dev0

Added tag 3.26.3, debian/3.26.3-1 for changeset f7067be5f69c

[pkg] Version 3.26.3

[utils/statsd_logger] flake8

[source/native] do NOT do anything with fti if do_fti is False ensure no fti related process is done when fti is disabled on a db source.

[test/web] flake8 unittest_propertysheet.py

[web] fix css file generation permission bug (closes #17143773) PropertySheet.process_resource() uses mkstemp which enforce a 0600 mode on the output file, which is not the file permission we want for css files.

[test/web] add a test for css generation file permission problem (related to #17143773)

[doc] include the 3.26 changelog in changes's index

[server] set PGPASSWORD when running pg commands for backup and restore For db-dump (implied by upgrade) and db-restore with postgres with non-trusted authentication, the db password is asked by pg tools (pg_dump, pg_restore, createdb, dropdb). Avoid this behavior by setting the PGPASSWORD environment variable when database driver is postgres and db-password is set in sources file.

Added tag 3.26.2, centos/3.26.2-1, debian/3.26.2-1 for changeset 9bee3134d304

[pkg] Version 3.26.2

Flake8 cubicweb/web/webctl.py Having the file flake8 clean makes it easier to catch basic programming errors (e.g. name errors).

[web] Add missing import of _DATA_DIR in webctl Followup on d84bc85f7f70 which introduced usage of this name in _datadirs method of GenStaticDataDirMixIn but resulted into a NameError at runtime and makes migrations crash.

[server] unicode → text_type in utils

[views] unicode → six.text_type in views/authentication.py

[pyramid] Recreate CWSession entity when eid no longer exists As noted in comment, it might occur that we receive a "sessioneid" from a valid cookie while respective CWSession got dropped (typical case is db being recreated while users got preserved). In such case, we just recreate the CWSession entity as if it did not exist.

Added tag 3.26.1, debian/3.26.1-1, centos/3.26.1-1 for changeset 027676243aaa

[pkg] Drop non-existant files from MANIFEST.in

[pkg] Version 3.26.1

[test] Drop an unused function in cwconfig tests

[cwconfig] Always use "sys.prefix" to determine installation prefix We drop all legacy logic implemented in _find_prefix() which now happens to be buggy since we cleaned up setup.py in 3.26 (noticeably because virtualenv is no longer properly detected). Closes #17132990.

[cwconfig] Look for migration scripts directory relatively to code Thus getting rid of the last piece of data in "shared" directory. As a consequence, we can drop data_files from __pkginfo__.py. Related to #17132990.

[pkg] Get rid of "pydir" in __pkginfo__.py Nothing uses this.

[pkg] Get rid of "shared" data These are not used anymore per previous changesets. Related to #17132990.

[cwconfig] Drop shared_dir method There's no more users of this method. Related to #17132990.

[config] Avoid relying on shared_dir method when possible From 3.26, some "data" directories (like web/data and i18n or wdoc) are installed along with CubicWeb in site packages. So we can retrieve them relatively to the module that needs them (namely "cubicweb" for "i18n" and "cubicweb.web" for "web/data" and "wdoc"). Reason for doing this is that "share_dir()" is not working when cubicweb is installed in "develop" mode and we are about to drop this method. Related to #17132990.

[test] Add a test for WebConfiguration.locate_all_files() We are going to change the implementation, make sure there's no regression.

[cwconfig] Get rid of _forced_mode global And compute CubicWebNoAppConfiguration.mode within the class namespace. Related to #17132990.

[cwconfig] Get rid of CWDEV This is in preparation of a fix for #17132990 to cleanup legacy (and probably no longer working) behaviors.

[doc] Set release date of 3.26

[web] Fix UnboundLocalError about "exc" in ajax controller Introduced in 159dce89a145.

Link to "3.26" docs in README

Added tag 3.26.0, debian/3.26.0-1, centos/3.26.0-1 for changeset 199851fcddd4

[pkg] Version 3.26.0

[pkg] Version 3.26.0rc0

[schema] Define a decorator to handle methods override There is a number of external classes (from yams/rql) methods overridden "by hand" in cubicweb/schema.py. Define a single decorator to factor out the pattern. It handles specifying the method name (to avoid conflict with the local namespace) and passing the original method to the new definition when needed.

[pkg] Use setuptools.find_packages() We drop the custom get_packages() function in setup.py that used to look for test files (not discovered automatically since test directories are not packages) and replace it by explicit entries in MANIFEST.in.

[pkg] Drop SandboxViolation verification cracking Dunno what this is for. Let's see if this work without.

[pkg] Drop custom "install_data" command Not sure what this is for, the comment mentions "egg" directory, but this is quite likely an old hack that should be unnecessary when using a recent setuptools.

[pkg] Replace custom "scripts" handling by a setuptools entry points http://setuptools.readthedocs.io/en/latest/setuptools.html#automatic-script-creation

Set a default value for arguments of cubicweb.cwctl:run() This will help making a console script entry point from this to get rid of the custom installation of scripts in setup.py. Accordingly, update all calls of this function.

[pkg] Drop custom install_lib command and use include_package_data instead The custom install_lib command appeared to be meant to install package data, as declared in __pkginfo__.py. Let's rely on setuptools' include_package_data[1] option to handle this. [1]: http://setuptools.readthedocs.io/en/latest/setuptools.html#including-data-files

[py3] Replace a unicode() occurrence in workflow views

[py3] Drop a couple of unicode() in devtools.testlib

[py3] Alias long to int in pgstore tests The previous version of handling PY2/PY3 made flake8 complain.

Merge with 3.25

Added tag 3.25.4, centos/3.25.4-1, debian/3.25.4-1 for changeset b8567725c473

[pkg] Version 3.25.4

[pyramid] Let logging be configured through .ini file We do not initialize logging in CubicWebPyramidConfiguration thus preventing logging to be started with cubicweb's configuration. On the other hand, any logging configuration defined in ``development.ini`` file will be loaded by whatever start the instance using this file (i.e. pserve, gunicorn, etc.). The benefit is that logging can now be configured per "qualified name" in this file (i.e. one can easily set the DEBUG level for the application cube while keeping all dependencies' level to WARNING). In the development.ini template, we add logger configurations for "logilab" and "cubicweb" qualified names (along with those of the application cube).

Disable pytest's logs reporting CubicWeb already prints captured logs in stderr upon test failure, so we get them twice. Until we fix our logging management, disabling pytest capture and restore brings the output level back to sanity.

[pyramid] Do not issue security warnings in test mode When some session or authtk secret is missing in Pyramid settings, scary "!! SECURITY WARNING !!" are issued. This is arguably pointless in tests. So disable them in this case.

[ssplanner] Prevent execution of write queries involving computed relations Previously, setting a computed relation upon entity creation or modification (using the ORM or an RQL query) would usually fail with an operational error in the backend ("no such table"). However, under some mysterious circumstances (like passing a string as value in cw_set for a computed relation), the RQL to SQL transformation would simply drop the clause. To prevent this to happen, we add a check for computed relation before adding a relation to an execution plan. This check raises a QueryError. It happens in several places: * in querier.InsertPlan.add_relation_def() (called from several places in ssplanner steps) for INSERT queries, * in ssplanner.UpdateStep.execute() for SET queries and, * in ssplanner.SSplanner.build_delete_plan() for DELETE queries. Tests added to unittest_querier.py because unittest_sslplanner.py looked inappropriate (it has only unit tests) and the former already had a NonRegressionTC class.

[querier] Expand 'rdef' argument in InsertPlan.add_relation_def() Expand the 'rdef' tuple to improve readability. The 'rtype' value is not used at the moment, but will in the next changeset.

[server] Use dict.setdefault() instead of if/else in repo.glob_add_relations()

[cleanup] Fix some flake8 errors

[test] Stop using lgc.testlib in unittest_migractions

[test] Drop no more used "maxeid" based deletion in BaseQuerierTC and derived tests This is probably only necessary for QuerierTC itself, move it there and drop incantation from other derived classes to deactivate this feature.

[rql2sql/test] Remove dead setUpModule tearDownModule they are overwritten a few lines below , hence this code is dead and not necessary for test success, drop it.

[rql2sql/test] Move base class from repotest to unittest_rql2sql It's only used there, so move it closer to its usage. Along the way, stop using lgc.testlib.

[rql2sql/test] Set maxDiff to None to see full diffs this is usually useful to compare SQL strings

[rql2sql] Use string_type instead of unicode for py3k support

[gcdebug] Only ignore weakref.WeakKeyDictionary class on Python 2 From Python 3.5, this class fails isinstance check with the following error: :: cls = <class 'weakref.WeakKeyDictionary'>, instance = <functools._lru_list_elem object at 0x7f4331859a48> def __instancecheck__(cls, instance): """Override for isinstance(instance, cls).""" # Inline the cache checking > subclass = instance.__class__ E AssertionError: [<class 'AttributeError'> in gc] 'functools._lru_list_elem' object has no attribute '__class__' /usr/lib/python3.5/abc.py:181: AssertionError I have no clue why this happens, but it makes cubicweb.web.test.test_views.AutomaticWebTest.test_startup_views fail on Python 3.5. So only consider this class for Python 2.

[wsgi] Convert "code" passed to WSGIResponse as integer WSGIResponse is passed a Request.status_out in CubicWebWSGIApplication._render(), which, starting from Python 3.5, is an instance of HTTPStatus. However, webtest still expects a 3-digits value as status string. Fortunately, calling int() on an HTTPStatus works.

[test] Add a "version" to cubicweb_mycube/__pkginfo__.py Starting from Python 3.5, unittest_cwctl.py::CubicWebCtlTC.test_list() would fail without this.

[tox] Use py3 environment instead of py34 The former would select any Python 3 interpreter.

[querier] Allow usage of computed relation into WHERE clause of write queries This used to crash because of an attempt to access unexisting table. Closes #17113286

[cleanup] Fix some flake8 errors

[tox] Bump flake8 version to 3.5 Previous patches fixed new errors raised in this version.

[pyramid] Get rid of a couple of bare "except"

[pyramid] Get rid of a bare "except" in UpdateLoginTimeAuthenticationPolicy Also refactor a bit the method to restrict the try/except block to the code for which we actually want to catch an exception (i.e. the cnx.execute() + cnx.commit()).

[web] Avoid bare "except" in ConcatFilesHandler.concat_cached_filepath()

[test] Use ignore_errors=True in shutil.rmtree instead of try/except/pass

[server] Get rid of a bare except in sqlutils

[server] Do not use logilab.common.date.strptime in sqlutils This function is aliased to datetime.datetime.strptime() and exists only for compatibility with Python <= 2.5 which we no longer support.

[dataimport/test] Ignore flake8 error E741 in test_pgstore.py This error is about "l" being an ambiguous variable name. Here, it's arguably acceptable.

[test] Avoid ambiguous variable name "l" in unittest_req.py This is an error (E741) from flake8 3.5.

[tox] Bound flake8 version to >=3.4, <3.5 The code base is not clean with 3.5, so keep using 3.4 in *stable* branches.

[pyramid] set waitress url_scheme to https when base-url scheme is https See https://docs.pylonsproject.org/projects/waitress/en/latest/#using-behind-a-reverse-proxy Since we do not configure trusted_proxy waitress ignore X-Forwarded-Proto header and may generate http urls (pyramid request.url) when the instance is behind a https reverse proxy. This cause cubicweb-signredrequest to not work with since it rely on the url (including scheme). Set url_scheme to 'https' when CubicWeb base-url scheme is https as a workaround.

Added tag 3.25.3, centos/3.25.3-1, debian/3.25.3-1 for changeset d238badfc268

[pkg] version 3.25.3

[uicfg] Fix autoform_section rtags wrt derivation cw 3.25 introduced the 'derived rtags' feature, but missed that `autoform_section` implementation was overriding several methods where this was implemented in the base class. The following problems are tested and fixed: * during init, we should not attempt to call `_initfunc_step2` if there is some parent, * but we should handle expansion of wildcard ('*' used as subject or object of a tag) since this is handled during init step for this class, unlike others which handle this in `get` method (i.e. at lookup time), * fix overrided `get` method to consider parent rtag if any. Closes #17107020

[rtags] Enhance repr of rtags * we don't care of the class, the registry id is enough (never seen a case of sub-classing) * add the module from which the rtag is coming as well as parent rtag if any.

[web/test] Use bare unittest main instead of lgc.main

[cleanup] Fix some flake8 errors and drop upper bound of copyright along the way. autoform module is not added to flake8-ok-files since there are some remaining errors to be handled, but that's still a start.

[doc] Update the advanced tutorial to follow the new style layout of cubes among others.

[views] Respond with 400 Bad Request in case of validation error in ajax request Previously, we raised a plain RemoteCallFailed without a status code, in which case it defaults to "500 Internal Server Error". Now, for validation errors, we issue a 400 Bad Request since these are clearly client errors.

[web] fix session /cnx mix in anonymized_request

[test] Pin some test dependencies We currently have CI failures because cubes used as test dependencies have been updated to new-style cube layout. To avoid this, pin them to previous released. Those dependencies should be removed but in the mean time this should be enough (and backported in all active branches).

[test] Pin some test dependencies We currently have CI failures because cubes used as test dependencies have been updated to new-style cube layout. To avoid this, pin them to previous released. Those dependencies should be removed but in the mean time this should be enough (and backported in all active branches).