[doc] Update documentation to match current Debian packages

[pkg] Remove transitional packages We can take the opportunity of the switch to Python 3 to get rid of old transitional packages. Let's do it. :)

[pkg] Use sections from requires.txt to populate Recommends and Suggests As Denis Laxalde pointed out, dh_python can also generate Recommends and Suggests from Python package names. So let's use that instead of manually populating these fields in `debian/control`. Optional dependencies are currently specified in `setup.py` grouped by feature. These dependency groups are turned into sections in `requires.txt`. Thankfully `dh_python3` has options to populate Recommends or Suggests with all package from a given section. `debian/rules` now contains a list of which sections should go to Recommends and which section should go to Suggests. Because such extra list easily gets out-of-sync, we add a third list for ignored sections, and ensure that all sections currently in `requires.txt` get a mentioned in `debian/rules`. Here are the results compared to the previous version with explicit Recommends and Suggests (only listing Python packages): |==============================================================| | only in previous | common | only in new | |==============================================================| | Recommends | |--------------------------------------------------------------| | | python3-docutils | | | python3-fyzz | | | | python3-imaging | | | | | python3-pycryptodome | | | | python3-pyramid | | | | python3-pyramid-multiauth | | | python3-pysqlite2 | | | | | python3-rdflib | | | | python3-repoze.lru | | | python3-simpletal | | | | | python3-vobject | | | | python3-waitress | | | python3-werkzeug | | | | | python3-wsgicors | | |--------------------------------------------------------------| | Suggests | |--------------------------------------------------------------| | | | python3-pil | We also lose versioned dependencies which should not really be an issue for Recommends and Suggests.

[crypto] Use Cryptodome namespace instead of Crypto PyCryptodome comes in two flavors: “an almost drop-in replacement for the old PyCrypto library” and “a library independent of the old PyCrypto”. The former uses the Crypto namespace, and is shipped as `pycryptodome` while the latter uses Cryptodome instead and lies in the `pycryptodomex` package. Given the reason to switch to PyCryptodome is that PyCrypto in unmaintained, its probably better to avoid any mistake and mandate the specific usage of the Cryptodome namespace by requiring `pycryptodomex` instead of `pycryptodome`. A more present reason is that Debian buster will only provide a package with the separate namespace flavor. The current Recommends is not working with the current code. Although it's important to note that the package name will probably have to be changed to `python3-pycryptodomex` once https://bugs.debian.org/886291 is solved.

[pkg] Remove python3-six from Build-Depends The default branch only provides support for Python 3.

[pkg] Merge python3-cubicweb-pyramid into python3-cubicweb As suggested by Denis Laxalde, we can merge python3-cubicweb-pyramid into python3-cubicweb as all other web backend in the default branch.

[pkg] Remove explicit Depends on Python packages for python3-cubicweb As suggested by Denis Laxalde, we can let pybuild generate the list of Depends from the egg requirements. The list we get is slightly less specific in terms of needed versions, but given the age of the previous ones, it's unlikely to cause issues. Also we lose python3-six from the list… which is correct as it was removed from the dependencies in 26744ad37953.

[pkg] Fix spelling mistake in doc-base abstract

[pkg] Fix doc-base section As pointed by Lintian, the “Apps” section is only used in menu, not in doc-base. We can just remove it.

[pkg] Use “dependency package” instead of “virtual package” to describe python3-cubicweb-postgresql-support The term “virtual” was too imprecise and is not accepted for empty packages by Lintian anymore.

[pkg] Move lintian-overrides file to debian/source directory Just use the now preferred location.

[pkg] Stop using /etc/bash_completion.d cubicweb-ctl shipped an empty /etc/bash_completion.d as a leftover of shipping a bash completion script. As pointed by Lintian, this directory is not obsolete, so let's get rid of it for good.

[pkg] Tidy substvars usage in control file The documentation package was missing a Built-Using field to record the provenance of the templates. Empty packages or meta packages do not require Depends on a specific Python version, so we can remove the relevant substvars.

[pkg] Specify priority “optional” instead of the obsolete “extra” Quoting Lintian, “Since Debian Policy version 4.0.1, the priority extra has been deprecated.” So let's use “optional” instead.

[pkg] Remove build dependency on dh-systemd As suggested by Lintian, remove build dependency on obsolete package dh-systemd. Instead, specify a more precise version of debhelper 9 that added support for systemd.

[pkg] Run test suite as part of autopkgtest Add support for autopkgtest. The test “unittest” will run the test suite using pytest. As we need PostgreSQL in the test environment and open network connection, the test is marked with `isolation-container` and requires at least LXC to be run, e.g: autopkgtest -s -- lxc -s autopkgtest-sid

[pkg] Switch to Debian source format 3.0 (quilt) This forces us to base the Debian package on the source of the Python package (as produced by running `python3 setup.py sdist`). While it might seem cumbersome, this greatly reduces likelyhood of mismatch between an installation via `pip` and one using the Debian package. `dpkg-source` will also document for us what is currently in the source repository but not in the Python package. Currently the missing file are: PKG-INFO cubicweb/sobjects/test/data/cubicweb_comment/schema.py cubicweb/sobjects/test/data/cubicweb_comment/__init__.py cubicweb/sobjects/test/data/cubicweb_comment/__pkginfo__.py cubicweb/sobjects/test/data/cubicweb_card/schema.py cubicweb/sobjects/test/data/cubicweb_card/entities.py cubicweb/sobjects/test/data/cubicweb_card/__init__.py cubicweb/sobjects/test/data/cubicweb_card/__pkginfo__.py cubicweb/server/test/data-migractions/cubicweb_comment/schema.py cubicweb/server/test/data-migractions/cubicweb_comment/__init__.py cubicweb/server/test/data-migractions/cubicweb_comment/__pkginfo__.py cubicweb/server/test/data-migractions/cubicweb_localperms/schema.py cubicweb/server/test/data-migractions/cubicweb_localperms/__init__.py cubicweb/server/test/data-migractions/cubicweb_localperms/__pkginfo__.py cubicweb/server/test/data-migractions/cubicweb_basket/schema.py cubicweb/server/test/data-migractions/cubicweb_basket/__init__.py cubicweb/server/test/data-migractions/cubicweb_basket/__pkginfo__.py cubicweb/server/test/data-migractions/migratedapp/cubicweb_comment/schema.py cubicweb/server/test/data-migractions/migratedapp/cubicweb_comment/__init__.py cubicweb/server/test/data-migractions/migratedapp/cubicweb_comment/__pkginfo__.py cubicweb/server/test/data-migractions/migratedapp/cubicweb_localperms/schema.py cubicweb/server/test/data-migractions/migratedapp/cubicweb_localperms/__init__.py cubicweb/server/test/data-migractions/migratedapp/cubicweb_localperms/__pkginfo__.py cubicweb/server/test/data-migractions/migratedapp/cubicweb_basket/schema.py cubicweb/server/test/data-migractions/migratedapp/cubicweb_basket/__init__.py cubicweb/server/test/data-migractions/migratedapp/cubicweb_basket/__pkginfo__.py cubicweb/server/test/data-migractions/migratedapp/cubicweb_tag/schema.py cubicweb/server/test/data-migractions/migratedapp/cubicweb_tag/__init__.py cubicweb/server/test/data-migractions/migratedapp/cubicweb_tag/__pkginfo__.py cubicweb/server/test/data-migractions/migratedapp/cubicweb_card/schema.py cubicweb/server/test/data-migractions/migratedapp/cubicweb_card/__init__.py cubicweb/server/test/data-migractions/migratedapp/cubicweb_card/__pkginfo__.py cubicweb/server/test/data-migractions/migratedapp/cubicweb_file/schema.py cubicweb/server/test/data-migractions/migratedapp/cubicweb_file/__init__.py cubicweb/server/test/data-migractions/migratedapp/cubicweb_file/__pkginfo__.py cubicweb/server/test/data-migractions/cubicweb_tag/schema.py cubicweb/server/test/data-migractions/cubicweb_tag/__init__.py cubicweb/server/test/data-migractions/cubicweb_tag/__pkginfo__.py cubicweb/server/test/data-migractions/cubicweb_card/schema.py cubicweb/server/test/data-migractions/cubicweb_card/__init__.py cubicweb/server/test/data-migractions/cubicweb_card/__pkginfo__.py cubicweb/server/test/data-migractions/cubicweb_file/schema.py cubicweb/server/test/data-migractions/cubicweb_file/__init__.py cubicweb/server/test/data-migractions/cubicweb_file/__pkginfo__.py cubicweb/test/data-rewrite/cubicweb_localperms/schema.py cubicweb/test/data-rewrite/cubicweb_localperms/__init__.py cubicweb/test/data-rewrite/cubicweb_localperms/__pkginfo__.py cubicweb/test/data-rewrite/cubicweb_card/schema.py cubicweb/test/data-rewrite/cubicweb_card/entities.py cubicweb/test/data-rewrite/cubicweb_card/__init__.py cubicweb/test/data-rewrite/cubicweb_card/__pkginfo__.py The current manifest will duplicate several files that are stored as symlinks in the Mercurial repository, hence the long list of ignored files in `extend-diff-ignore`.

[pkg] Switch all Debian packages to Python 3

Fix sorting key for rdefs in schema viewer With changeset 234ca3cbbb46, clicking in the schema of an entity with cubicweb in py3 on "vue en boite" will probably result in an infinite spinner (which implies cw > 3.26) What happened ? This "vue en boite" used to work at least until... hg diff -c a8c1ea390400 cubicweb/schema.py @@ -993,10 +992,6 @@ class CubicWebRelationSchema(PermissionM return False return True - @deprecated('use .rdef(subjtype, objtype).role_cardinality(role)') - def cardinality(self, subjtype, objtype, target): - return self.rdef(subjtype, objtype).role_cardinality(target) - class CubicWebSchema(Schema): """set of entities and relations schema defining the possible data sets But, wait ... If I open a shell on an instance of cw 3.24 something seems off >>> list(schema['CWUniqueTogetherConstraint'].relation_definitions())[0][0].cardinality # <bound method CubicWebRelationSchema.wrapped of <constraint_of [CWUniqueTogetherConstraint,CWEType]>> We have been sorting on a method the whole time ? Is it possible what were the effects ? 1) We cannot sort function can't we ? >>> def adder(i): return lambda x: x+i >>> sorted(map(adder,range(10))) [<function __main__.<lambda>>, <function __main__.<lambda>>, ... Yes we can. 2) what does it means. >>> { adder(1) : 1 } Out[19]: {<function __main__.<lambda>>: 1} In fact the function object as a __hash__ method (which is practical for making memoizers (cache)), and return truly random results (pseudo random). My take on this patch is relations have NEVER been sorted by cardinality. No one never ever noticed. Hence, I propose to not fix a bug that never was reported.

Flake8 cubicweb/web/schemaviewer.py

[py3] changing unicode to str py2 support being dropped, passing the code in full py3 str/bytes syntax

[py3] Use sorted(key=...) instead of sorted(cmp=...)

Merge with 3.26

Added tag 3.26.11, debian/3.26.11-1 for changeset 633ca84bea58

[pkg] Version 3.26.11

Added tag debian/3.26.10-1 for changeset 0ce26fb5ab48

[debian] New upstream release

[debian] Drop override_dh_auto_install about cubicweb-ctl The script should be installed normally as it is a setuptools' entry point.

[debian] Remove test/doc Build-Depends We are not actually running tests during package build. (This was disabled for python2, now it is also for python3.) And having these build-depends makes it hard to rebuild the package.

[fix] parse cube version number without space in them

Install patched yapps for test from an http archive Instead of git which require git to be installed. Also downloading archive is faster than cloning the repository.

[server/test] make test filename uniques To avoid these pytest error when collecting the whole test suite: import file mismatch: imported module 'unittest_utils' has this __file__ attribute: cubicweb/cubicweb/server/test/unittest_utils.py which is not the same as the test file we want to collect: cubicweb/cubicweb/test/unittest_utils.py Move cubicweb/server/test/unittest_security.py to cubicweb/server/test/unittest_security.py and cubicweb/test/unittest_utils.py to cubicweb/test/unittest_server_utils.py

Merge 3.26

[dataimport] Fix case when extid is text in use_extid_as_cwuri() Extid can be bytes or text. This avoid AttributeError: 'str' object has no attribute 'decode'

[doc] fix key used in the hook example The hook's regid is check_no_subsidiary_cycle.

[doc] set_operation has been replaced by add_data See the changelog https://cubicweb.readthedocs.io/en/3.26/changes/changelog/#id14

[doc] check_cycle is a standard function

[doc] fix several typos

[pkg] make cubicweb[crypto] depends on pycryptodome instead of pycrypto pycrypto isn't maintained anymore, the last release 2.6.1 is from 2014. Use the drop-in replacement fork pycryptodome instead. pycryptodome is packaged in debian starting from stretch-backports.

[crypto] Encode unicode strings in _cypherer() pycrypto accept both bytes or string but pycryptodome, which will be introduced in next changeset, doesn't accept this. For backward compatibility, encode unicode strings in _cypherer()

[ldapfeed] FIX: Unique Key violation when synchronizing with LDAPfeed What was happening ================== The bug appears when ldapfeed tries to insert a user while another user exists exists with a different source. Simple use case to reproduce: - create a local user in cubicweb (source=system) - sync with ldap - ldapfeed will stop complaining user already exists. Without next patch the test MUST fail with message: cubicweb/server/sources/native.py:714: UniqueTogetherError The ldapfeed is thus stopped ignoring any further ldap entries. The proposal ============ Prior to this patch, the insertion was trying to create all CWUser with the computed login from eeimporter.extid2eid (method process of DataFeedLDAPAdapter). When a CWUser existed with a different cw_source ("system" for user created with cubicweb for instance), it created a conflict. To avoid collisions, in the extentities_generator method a bypass was added at the insertion layer. Prior to insertion the absence of the computed login is checked on a list of all existing login from a different source. If collision is detected, we skip the ldap record. By short circuiting at the CWUser entity insertion level we also avoid to treat CWGroup and EmailAddress related to this user. Hence ensuring a behaviour that will not break existing instances. (conservative approach: faced with ambiguity better do nothing than guess). An error message is added stating explicitly the conflict.

Skip tests for ldapsource with python >= 3.7 Until someone works on fixing these, this should make our CI green again. I tried to use setupModule() to check for python version, but pre_setup_database() is apparently called even when a SkipTest exception is raised there. So handle this in that method. (grafted from 4d68d20427dee4b6751a0f1f5511fec2a04f4782)

Flake8 crypto module

Make crypto module python3-compatible * Remove usage of unicode() and decode the base64-encoded string in encrypt(); * Encode the string received in decrypt() as (I supposed) it should come from the result of encrypt(). Add tests for this module along the way.

[mod] make ViolatedConstraint actually display useful information Move from: (Pdb++) raise some_exception *** ViolatedConstraint: To: (Pdb++) raise some_exception *** ViolatedConstraint: constraint 'cstr56c2ab4b3154f21d08b067742ce5bd9d' is being violated by the query 'ALTER TABLE cw_Bibliography ADD CONSTRAINT cstr56c2ab4b3154f21d08b067742ce5bd9d CHECK(cw_item_type IN ('journalArticle', 'note', 'book', 'thesis', 'film', 'web page', 'manuscrit', 'tapuscrit'))'. You can run the inverted constraint on the database to list the problematic rows. And save hours of debugging to actually understand what is going on.

Fix flake8 errors/warnings about import in cubicweb/_exceptions.py We either ignore errors in case of name re-exports and drop re-exports from old backwards compatibility. We fix the only occurrence of NoSelectableObject being imported from cubicweb in tests and update the import to use lgc instead.

add comment and # noqa on used import

remove unused import

[autopep8] E305 - Expected 2 blank lines after end of function or class

[autopep8] E303 - Remove extra blank lines

[autopep8] E302 - Add missing 2 blank lines

[autopep8] E301 - Add missing blank line

[enh] display CWConstraint type in its repr This will change CWConstraint repr from something like: <Entity CWConstraint 314282 [] at 140127700141904> To: <Entity CWConstraint 314282 of type UniqueConstraint [] at 140127700141904> To help improve debugging UX a little bit.

[entities] display CWConstraint type in its repr

[changelog] document pyramid.ini automatic generation

Flake8 cubicweb/pyramid/test/test_hooks.py

[pyramid/doc] bad docstring

[pyramid/misc] remove useless variable

[pyramid/test] add test for pyramid.ini generation

Added tag 3.26.10 for changeset 74cc5de1ee6c

[pkg] Version 3.26.10

Move CWSchemaTC to RQLExpressionTC The test actually doesn't require a cubicweb schema, except for testing EmailAdress which is out of scope of the test (testing RQLExpression.transform_has_permission()). Just move the test in existing RQLExpressionTC inheriting from TestCase and rename the tests to be more explicit.

[skeleton] Drop py27 tox environment Since we dropped py2 support from cubicweb, it does not make much sense to keep it in skeleton from now on.

[skeleton] Do not ship Debian and RPM packaging files in sdist

[skeleton] Add a check-manifest tox environment

Fix pyramid tests Followup fa292e9 which require cubicweb.session.secret to be set.

Fix flake8 and check-manifest Followup fa292e9

Merge 3.26 This fixes tests with psycopg2

DeprecationWarning: In future versions of Waitress clear_untrusted_proxy_headers will be set to True by default. You may opt-out by setting this value to False, or opt-in explicitly by setting this to True. Source: https://docs.pylonsproject.org/projects/waitress/en/stable/arguments.html?highlight=clear_untrusted_proxy_headers > This tells Waitress to remove any untrusted proxy headers ("Forwarded", > "X-Forwared-For", "X-Forwarded-By", "X-Forwarded-Host", "X-Forwarded-Port", > "X-Forwarded-Proto") not explicitly allowed by trusted_proxy_headers. According to grep we don't use any of those headers so let's turn it on for security reasons.

DeprecationWarning: The default pickle serializer is deprecated as of Pyramid 1.9 and it will be changed to use pyramid.session.JSONSerializer in version 2.0. Explicitly set the serializer to avoid future incompatibilities . See "Upcoming Changes to ISession in Pyramid 2.0" for more information about this change. As describe here https://docs.pylonsproject.org/projects/pyramid/en/1.10-branch/narr/sessions.html#pickle-session-deprecation use a serializer that fallback on pickle if needed to avoid impacting users.

[pyramid/enh] generate pyramid.ini "create" and on "pyramid" command if needed

DeprecationWarning: The SafeConfigParser class has been renamed to ConfigParser in Python 3.2. This alias will be removed in future versions. Use ConfigParser directly instead.

Account for new psycopg2 exception classes mapping From psycopg2 >= 2.8, specific exceptions are raised corresponding to postgresql errors. E.g. a CheckViolation exception is raised instead of a generic IntegrityError previously when a constraint violation occurs. The way we intercept database errors, especially for constraint violation, is not compliant with that because we do not catch subclasses of IntegrityError in native source's doexec() method. We fix this by checking for the presence of IntegrityError error in exception class's mro. This is still overcomplicated and clumsy, because we still use string comparison, but this is the best we can do as far as I know. (A better fix would be 'isinstance(ex, IntegrityError)' but we have no engine-independent error classes, so this is not possible. Something like sqlalchemy's DBAPI Errors [1] might help: https://docs.sqlalchemy.org/en/latest/errors.html#dbapi-errors)

[cubicweb-ctl] remove "cubicweb-ctl wsgi" command following pyramid standardization

Use secure hash algorithm in WebConfiguration.sign_text Fix: PendingDeprecationWarning: HMAC() without an explicit digestmod argument is deprecated. The default hash algorithm used by hmac.new is md5. As of today, md5 is so weak that it's the equivalent of plaintext and can't be considered to be secured at all. Therefor, we switch to a secure hash algorithm. The rational for choosing sha3_512 is: * the recommended algorithm is at least sha_256 * the stronger, the more secured and sha3_512 is the stronger available * thinking about the future this should keep this part of the code safe long enough before people think about checking it again You can read more about choosing a secure hash algorithm in the NIST recommendations https://csrc.nist.gov/Projects/Hash-Functions/NIST-Policy-on-Hash-Functions This code modification should normally be transparent since check_text_sign is exactly this code 'self.sign_text(text) == signature' and that sign_text is only used in combination with it. The only impact is that the hash is going to move from 32 char to 128 which might make html page a bit bigger and that sha3_512 is slow to compute (which is a good thing for security)

[enh] don't catch all exceptions in notification hooks during tests

[doc] add a reference to static-messages.pot usage

[doc] Clarify sentences

[doc] Fix format and typo

Fix DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated

Fix DeprecationWarning: invalid escape sequence

[devtools/testlib] avoid hidding AttributeError in create_user() commit() might raise a AttributeError too. Use getattr(req, 'cnx', req) instead, which is a form already used to get the real cnx in some code: cubicweb/rset.py:577: cnx = getattr(self.req, 'cnx', self.req) cubicweb/schema.py:353: with getattr(_cw, 'cnx', _cw).security_enabled(read=False): We could use if hasattr(req, 'commit') here too but it lead to 3 additionals lines. Maybe we should have commit() and rollback() on cubicweb.web.request.ConnectionCubicWebRequestBase too ?

Fix DeprecationWarning: invalid escape sequence \

DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead

[mod] remove backward compatible code for passlib and force modern version

DeprecationWarning: Please use assertEqual instead.

DeprecationWarning: Please use assertRaisesRegex instead.

DeprecationWarning: inspect.getargspec() is deprecated since Python 3.0, use inspect.signature() or inspect.getfullargspec()

[README] add testing instructions

Add pytest-subtests to dev requirements This pytest plugin adds support for unittest's TestCase.subTest (for Python >= 3.4). Since we we now only support python3 and since we use a fair amount of these, let's benefit of better reporting by using this plugin.

[test] use unittest.mock instead of mock library Now that we use Python 3 only, this is possible.

Drop python2 support This mostly consists in removing the dependency on "six" and updating the code to use only Python3 idioms. Notice that we previously used TemporaryDirectory from cubicweb.devtools.testlib for compatibility with Python2. We now directly import it from tempfile.

Merge with 3.26

Remove unused imports in cubicweb.pyramid.config

Added tag 3.26.9 for changeset 4d6909de765a

[pkg] Version 3.26.9

[pyramid/security] use cryptographically secure random generator Based on django source code in case SystemRandom is not available. According to python documentation: https://docs.python.org/2/library/random.html > Warning > The pseudo-random generators of this module should not be used for security > purposes. Use os.urandom() or SystemRandom if you require a cryptographically > secure pseudo-random number generator.

Reclosing branch after merge

Merging heads of old, closed branch

Reclosing branch after merge

Merging heads of old, closed branch

[doc/changes] document that legacy cube support has been dropped

[doc] replace legacy import to new style cube import in various places

[web/views] drop deprecated module cubicweb.web.views.timeline

[web/views] drop deprecated module cubicweb.web.views.massmailing

[web/views] drop deprecated module cubicweb.web.views.isioc

[web/views] drop deprecated module cubicweb.web.views.embedding

[web/views] drop deprecated module cubicweb.web.views.igeocodable

[pyramid/core] drop fallback import of legacy cubes

[cwconfig] drop importing legacy cubes This finally drop support for legacy cubes. In cwconfig, don't load modules in the "cubes" namespace. In cube_pkginfo() handle CW_MIGRATION_MAP which wasn't working with new style cube. Drop all method specific to legacy cubes importer: cubes_search_path(), extrapath(), cls_adjust_sys_path() Drop environment variables related to legacy cubes: CW_CUBES_PATH and CW_CUBES_DIR

[doc] better documentation for "cubicweb-ctl newcube" cubicweb-ctl newcube create cubicweb-<name> in current directory.

[test] make unittest_cwconfig.py does not depend on legacy cube machinery By dropping legacy cube tests in various places.