Project Structure Related to #4291173

[tox] Have less test environments This is to reduce the load on CI server because of parallel clones of the repository in Docker containers. Only keep "server", and "web" environments separated. All other prior environments are in "misc". To avoid duplicate entries in requirements files, move them all in a "requirements" directory (this appears to be a "common" practice, see e.g. Celery). Adjustments in tests: * Rename cubicweb/hooks/test/unittest_notification.py so that it does not conflict with cubicweb/sobjects/test/unittest_notification.py during test discovery as they would have the same module name but different __file__ attribute. * Add "comment" cube to the list of expected cubes in unittest_cwconfig.py as this cube is pulled by requirements/test-mist.txt. Closes #15440662.

[repo] Don't crash on start when fs schema is missing some db schema entities This occurs usually while developping and we don't want systematically to rebuild the database to start the instance or run e.g. i18ninstance.

Let configuration option be overridden by environment variables Related to #13889793.

[tox] Add a dummy test-results.xml file in check-manifest and flake8 environments This is to work around Jenkins Junit plugin that does not apparently account for the absence of this file (though it accepts it to be empty).

[tox] Basic flake8 config

Use pkgutil.ImpLoader instead of a custom loader for cubes import redirection It just works fine (same implementation of load_module method as _CubesLoader introduced in d404fd8499dd) and is complete w.r.t. PEP 302 (all methods implemented). Related to #13001466.

[config] Make available_cubes aware of cubes installed as packages For this add a "cubicweb.cubes" entry points group which will be scanned through to find out installed cubes. Entries in this group are expected to expose the module name of a cube (i.e. `cubicweb_foo`). Note that CubicWebConfiguration's available_cubes method will return the module name of cubes as packages (cubicweb_foo), so we had to add a special "key" sorting function to keep cubes sorted as before, despite possible different distribution schemes. This makes it possible to handle loading of CTL plugins in an almost similar manner as before (just tweaking the package name from cube name in load_cwctl_plugins method). I had to tweak (again?) the test_cubes_path method in unittest_cwconfig.py but did not find out why. Apart from unforeseen bugs and pending documentation, this finishes the work on porting cubes to standard Python packages. Closes #13001466.

[skeleton] Use distname directly in setup.py distname is part of required metadata and modname cannot actually be used in place of it.

[skeleton] Fix modname in __pkginfo__

[test] Use plain unittest in unittest_cwctl.py

[devctl] Remove commented LiveServerCommand The code got commented in 2010 (changeset 5ab3f63b06ad)...

[pkg] Inline call to setup() in setup.py The point of having this "install" function, called in "main" mode is not clear. Better stick to standard practices.

[pkg] Drop old checks in setup.py

[tox] Exclude symlinks from check-manifest call These are not handled by check-manifest, see https://github.com/mgedmin/check-manifest/issues/69.

Drop the only reference to an '__insert' form key

Drop bits from the past (no more dbapi isolated installation)

[devtools] CWTC.request is gone, don't attempt to use it in CWTC.view So one will have to provide at least one of `rset` or `req` argument of the `view` method.

Merge 3.22 into 3.23 Some fixes there are necessary for client work depending on cubicweb 3.23.

[devtools] Handle i18ncube command for "cubes as packages" And add a test so that both layouts are tested. Move the test cube into a package. Related to #13001466.

Handle cubes as packages in cwconfig Rely on importlib.import_module in several places in cwconfig.py, so this would not work with python 2.6. Method available_cubes will not list cubes installed as package for now. I'm not sure it's worth implementing this method (used in `cubicweb-ctl list cubes` command) for new cubes layout as the same result can basically be obtained with `pip freeze | grep cubicweb_`. In unittest_cwconfig.py, duplicate CubicWebConfigurationTC test case to test both the "cubes as packages" layout and the "legacy cubes" layout. The former having a custom sys.path set (pointing to datapath('libpython') where all cubes' packages live) and the latter having the previous config attribute setup. All test data cubes are moved to packages in libpython directory and symlinks are introduced in the cubes directory. Related to #13001466.

[cwconfig] Reorder imports by alphabetic order

Add an import redirect hook from "cubes.<name>" to "cubicweb_<name>" The hook consists of a finder and a loader implemented following PEP-302; it is responsible for loading cubes distributed as packages (i.e. installed as ``cubicweb_<name>`` in site-packages) but imported (in client code) as ``from cubes.<name> import ...``. So this is a transitional mechanism allowing cubes following the new layout to be used by old-style cubes/applications. The importer is installed upon calling CubicWebConfiguration's cls_adjust_sys_path method (also called in cubicweb.devtools.__init__.py, which is a prerequisite for importing any "legacy" cube. The loading of old-style cubes is still handled by the CubicWeb configuration, based on adjustment of sys.path etc. Related to #13001466.

[devtools] Update skeleton's setup.py to install cube as a package Most of the prior logic of skeleton's setup.py gets dropped as installing a cube as a "classic package" (i.e. in site-packages) is just the default behavior of distutils. Also add a test checking installation of new cube. Related to #13001466.

[devtools] Adjust skeleton setup.py to have it read information from __pkginfo__.py Since the later is now installed in cubicweb_<CUBENAME> directory. Add a test ensuring that `python setup.py sdist` works on newly created cube (which should at least verify that setup.py is valid). Related to #13001466.

[devtools] Update skeleton and newcube command to cube as package layout Related to #13001466.

[pkg] Add missing entries to MANIFEST.in Detected by running `check-manifest <https://github.com/mgedmin/check-manifest>`_. Add a tox environment accordingly. Closes #15229018.

[test] Add a test for toolsutils.read_config

[test] Use plain unittest in unittest_toolsutils

[skeleton] Remove test/pytestconf.py

[pkg] Make it clearer which entries of __pkginfo__ are actually used

[test] Avoid usage of lgc.testlib TestCase in unittest_cwconfig.py

[test] Move setup of spa2rql tests into setUpClass method Thus avoiding cryptic errors upon unittest discovery.

[schema] Set CubicWebSchemaLoader's extrapath using config eponymous property

[doc] Document change of instances location in virtualenv installation Closes #14789440.

[cwconfig] Do not override user-defined CW_MODE in virtualenv setup And thus consider `_forced_mode` first. Related to #14789440.

Added tag 3.22.4, debian/3.22.4-1, centos/3.22.4-1 for changeset 92db0bb8e26d

[pkg] remove .dev0 from version

[pkg] 3.22.4

[session] Ensure access to rql varmaker always mark the session as dirty When one accesses the rql_varmaker, that's usually to use it. The pb was that when the varmaker was already in page's data (which is stored as session data), session storage such as redis won't see that the session data is dirty and has to be stored back at the end of the request. To fix this, systematically call set_page_data. (grafted from 3432f0e2540d)

[test] fix test_printable_value_bytes with current pygments An empty span was added in https://bitbucket.org/birkenfeld/pygments-main/commits/164574c13533195a555181a2b9c685fea2470403 (grafted from 74b04a88d28a)

[webctl] set uid of file produced by gen-static-datadir (closes #11298794)

[webconfig] ensure uicache content has correct uid (related to #11298794)

[cwconfig] add the CubicWebConfiguration.ensure_uid_directory() method (related to #11298794) that recursively enforce uid on files.

[cwconfig] extract uid file setting code from check_writeable_uid_directory (related to #11298794) into an ensure_uid() method

[web] Fix bug with usage of os.rename under windows environment In 7c386161ebd6 we removed cache handling from property sheet and introduced usage of a tempfile + os.rename to get atomic generation of files. The pb is that this is not portable, since under windows os.rename will raise an exception if the file already exists (because there is no way to write a file atomatically in such case). This kind of thing should be out of the CW scope anyway, so implements a quick & dirty fix in the mean time. Closes #14214794

[migration] More resilient migration script Query the information schema to retrieve existing constraint (primary key and unique) and drop them. Thus don't use anymore DROP CONSTRAINT IF EXISTS (not implemented by e.g. old versions of sqlserver). To do so a missing DISTINCT was necessary to avoid duplicates.

[session] Ensure access to rql varmaker always mark the session as dirty When one accesses the rql_varmaker, that's usually to use it. The pb was that when the varmaker was already in page's data (which is stored as session data), session storage such as redis won't see that the session data is dirty and has to be stored back at the end of the request. To fix this, systematically call set_page_data.

typo

[native] Use cnx.system_sql instead of doexec method in syntax_tree_search The latter handles connection issues and tries to reconnect. Method syntax_tree_search appears to be called from quite a few place where it seems useful to be resilient to connection issues (authentication for instance). This reconnection logic appears to have disappeared at some point when working on https://www.cubicweb.org/2919309. Could not find the exact reason though...

Remove usage of cnx.ensure_cnx_set context manager It has been deprecated since 3.21, no reason to keep using it internally.

[hooks/test/synschema] remove an unneeded skipTest. Related to #5557633.

[test] Check retcode of subprocesses in devctl tests And display decoded stdout in case of failure.

merge 3.23

[pkg] Bump version to 3.24.0.dev0

[pkg] Drop NO_SETUPTOOLS option setuptools is the defacto standard nowadays. The only place where NO_SETUPTOOLS option was used is during packaging (RPM, Debian) and even in this context, it seems that setuptools is the way to go (see https://wiki.debian.org/Python/LibraryStyleGuide).