Move the cors middleware initialisation to pyramid-cubicweb to reduce code duplication

Added tag cubicweb-pyramid-version-0.1.0, cubicweb-pyramid-debian-version-0.1.0-1 for changeset 3defbb0f147a

Prepare release

Add a --no-daemon option

Use 'wsgicors' for CORS handling. The CW CORS handling (in web/cors.py) is only able to work on cubicweb requests. When a request is not handled by bwcompat, we need a proper solution. The `wsgicors` library provides what we need as a wsgi middleware.

Watch for i18n files changes for auto-reload

Auto-reload now survives failed reload When the server stops within the reloader and let a list of files that were monitored, the reloader waits for any of these files to change and re-attempt to start the server. This work well when a syntax error is saved to a file: there is no need to restart manually the server anymore.

Implements auto-reload and daemon mode. Heavily inspired by pyramid pserve, with pieces of code taken from it. auto-reload Start the server in a subprocess that auto-stops when a file is modified, and exit with a specific code. daemon mode Uses some code from pserve, but use the cw configuratione so the command is compatible with 'status' and 'stop' commands.

Add the 'pyramid-auth-secret' option to all-in-one.conf

Add the 'pyramid-session-secret' option to all-in-one.conf This value is used by pyramid_cubicweb to initialize the cookie factory.

Add a CWSession entity. It is a subset of the CWSession that will be added to cubicweb in a future version. Closes #4317363

Implements the 'pyramid' command. It load an instance config, and load in the pyramid scaffolding provided by pyramid_cubicweb. The application is then served by a waitress server. Closes #4317312

Project structure

Merge with pyramid-cubicweb The following tasks have been done: - merge packaging files - merge documentation - move pyramid_cubicweb package at cubicweb/pyramid and update imports accordingly - rename tests directory into test - move pyramid-cubicweb README.rst into README.pyramid.rst until better idea - add a test dependency on unreleased cubicweb-pyramid to have both py27 and py34 tests pass Closes #14023058.

Port to Python3 (closes #14159555) Add py34 environments to tox configuration (only for CubicWeb >= 3.23). And depend on hg version of cubicweb-pyramid since it is not currently Python3-compatible.

Use opened connections as much as possible instead of opening a new internal connection to set the session cookie or retrieve the session data, which may exhaust the connection pools.

Test and fix status code and content of the login form on authentication error The status code was 200 and is now fixed to 403, but the content part of the test will fail until cset 02328f8cbd5c is integrated in cubicweb.

Added tag 0.7.0, debian/0.7.0-1, centos/0.7.0-1 for changeset 0cf2972d2011

[pkg] Version 0.7.0

retrieve session data in a single query instead of two (one for the session object, the other for its cwsessiondata attribute).

[tox] Update CubicWeb version targets * In -release, no need to specify cubicweb dependency, its pulled by setup.py. * Use 3.22 as -compat. * Point to "default" branch for -dev. Also remove test directory from posargs.

wsgi: clearer exception when CW_INSTANCE is missing instance_name is mandatory for cwcfg.config_for() Raise with a KeyError: 'CW_INSTANCE' instead of hard to read exception.

typo

[doc] update a bit the documentation Also convert the README in rst (so it can be properly displayed on cubicweb.org).

split collecting setting before using them, so the function can be reused when inserting WSGI middlewares

[bwcompat] log execption even when cubicweb.bwcompat.errorhandler = True (closes #13421901)

[pkg] add spec file

Added tag 0.6.1, debian/0.6.1-1 for changeset 2621daafa10c

[pkg] 0.6.1

[bwcompat] send 403 on authentication errors (closes #12219849) 200 is just wrong.

[bwcompat] set response headers on AuthenticationError (closes #12219860) Turns out sending a response without a Content-Type header (among others) is suboptimal. 5b36399b6b21 was not good enough.

Added tag 0.6.0, debian/0.6.0-1 for changeset 94d1a024b3bf

[pkg] 0.6.0

[tests] add a __main__ handler the relative import in test_rest_api.py needs to be modified to prevent a ValueError: Attempted relative import in non-package

[pkg] add a requirements-test.txt file for test dependencies on WebTest and cubicweb-pyramid

[login] fix the redirect url after login (closes #11689118) redirecting to '/' by default after login doesn't work properly when a prefix is used, whether we're sitting behind a PrefixMiddleware or not. To fix this, rely on cubicweb's build_url to turn any relative path into an absolute url.

keep track of all traceback in error handling, not just the exception message (closes #11689093)

[bwcompat] also set response headers in error cases I had this sitting around in my local copy, I don't remember why I needed this but it seems correct.

[config] move config of the secret used to encrypt session's data ID in pyramid.ini (closes #11689082) Introduce a new config entry (in pyramid.ini) for this (cubicweb.session.secret) to replace the (now deprecated) pyramid-session-secret (in all-in-one.conf). So we have now 3 secrets to configure: - cubicweb.session.secret: to encrypt session's data ID stored in a cookie, - cubicweb.auth.authtkt.session.secret: to encrypt auth cookie - cubicweb.auth.authtkt.persistent.secret: to encrypt persistent session auth cookie

Document how to configure the "secure" flag for authentication policies Closes #11376233.

[tox] Let posargs override py.test args

[tox] Use py.test Rename test view so that py.test does not consider it as a test function.

Add a tox configuration

Register predicates from the predicates module

[refactoring] Move MatchIsETypePredicate to a separate module

[refactoring] Move EntityResource and ETypeResource to a separate module

use CubicWeb request to execute RQL rset should be retrieved with cw_request, as it's then bound to it and propagate to all entities created from this rset (._cw). From there it may reach code expecting a request, not a connection (view, selector, etc).

[views] Rename entities to rest_api

[routes] Add a 'cwentities' route with traversal The route uses a factory that produces ETypeResource and EntityResource as a context. A 'delete' view serve as a test and demonstration. The module being experimental, it has to be explicitely included.

Added tag 0.5.0, debian/0.5.0-1 for changeset 61f69ac2d6bc

[pkg] 0.5.0

[auth] Authtkt http_only and secure by default The test suite is now full 'https'. Closes #4731765

[auth] Make the configuration cookies completely configurable Also transfert the secret setting from all-in-one.conf to pyramid.ini, with backward compatibility. Closes #5999625

Added tag 0.4.1, debian/0.4.1-1 for changeset 398b2c840e14

[pkg] 0.4.1

[core] Adjust cw<3.21 compatibility Most of cubibweb < 3.21 expects a ClientConnection as the main connection. Related to #5731814 Closes #5878592

[bwcompat] Make the error handler optional Closes #5739625

[doc] Document embedding in a pyramid app

[config] Move most config code to a includeme() The goal is to make it easier to use pyramid_cubicweb from a pyramid application.

[doc] Anonymous access is not mandatory anymore

Added tag 0.4.0, debian/0.4.0-1 for changeset 897a149e8208

[pkg] add debian/watch

[pkg] 0.4.0

[pkg] add python-pyramid-multiauth dependency to debian package Closes #5576182

[core] adjust cnx handling for cubicweb 3.21 Closes #5731814

set httponly on session cookie

[bwcompat] use cubicweb error views (closes #4545130)

Added tag 0.3.1, debian/0.3.1-1 for changeset 6df91cb85ecc

[pkg] 0.3.1

Handle absence of anonymous user Set cw_session and then cw_cnx request attributes to None in case anonymous connection is not allowed (i.e. no "anon" user in config). Then catch AuthenticationError in CubicWebPyramidHandler and return the 'login' view. Closes #4751862.

[debian] Add python-wsgicors dependency as it is now available

Update Changes for version 0.3.0

Added tag pyramid-cubicweb-version-0.3.0, pyramid-cubicweb-debian-version-0.3.0-1 for changeset a80e076d3f42

Fix debugtoolbar pkg name

Set version 0.3.0

[doc] update changes list

Rollback 'uncommitable' cnx Closes #5343870

[debug] The debug mode now set pyramid.reload_templates

Use pyramid flash queue for messages Use a 'cubicweb' flash queue and make sure it contains only one message so that the behavior is the same as cubicweb. Also, the 'message' property now returns both the cubicweb flash queue and the default flash queue. One big difference with the former behaviour is that messages set with set_message will survive a redirection, making set_redirect_message useless in most case. Closes #5298654

[doc] Document the new authentication stack

Allow tests to override pyramid_settings Closes #5307426

Make debug mode usable without pyramid_debugtoolbar Add the latter in Debian recommends along the way. Closes #5310434.

[auth] Use a second authtkt policy for 'rememberme' The former solution was buggy because the expire time of the auth cookie, if set through 'remember', was lost on the first cookie reissuing. The new approach, make possible thanks to multiauth, use two different cookies. One for session bounded authentication (no 'rememberme'), and one for long lasting authentication (w 'rememberme'). The choice between the two of them is done by adding a 'persistent' argument to the top-level 'security.remember' call. Passing this argument will inhibate a policy or the other. The two policies are (a little) configurable through the 'cubicweb.auth.authtkt.[session|persistent].*' variables. Related to #4985962

[auth] Use pyramid_multiauth It makes it easier to finely tune what parts of the default authentication stack we want to use or not. It also makes it possible for any cube to add its own policy in addition to the others. Related to #4985962

[auth] remove dead code (closes #5230746)

[login] Test the login views

Fix project homepage url

Replace the '_' with '-' in the package name The change was made manually on pypi (see https://sourceforge.net/p/pypi/support-requests/459/)

On exceptions from CW, copy headers Closes #4939219

[doc] fix pyramid-auth-secret conf sample

[doc] Update change list

[core] Protect session data from unwanted loading. Use specialised Session and Connection types that forward their 'data' and 'session_data' attributes to the pyramid request.session attribute. This forwarding is done with properties, instead of copying a reference, which allow to access request.session (and the session factory) if and only if Session.data or Connection.session_data is accessed. In some cases, most notably the static resources requests, it can mean no access the session during the request handling, which saves a request to the session persistence layer. Closes #4891437

[core] Use tools.cached_user_build for better performances Closes #4870347

[doc] Document tools Related to #4870347

[tools] Provide a faster build_user The main trick is to use a cache of user entities. To do so, a few tools are needed since the entities are not supposed to be copied around between connexions. Related to #4870347

Added tag pyramid_cubicweb-version-0.2.1, pyramid_cubicweb-debian-version-0.2.1-1 for changeset 1ae61c25299a

Prepare version 0.2.1

[cors] Fix 'headers' and 'methods' parameters Closes #4849874

Fix the 0.2.0 release date

Change project url

Added tag pyramid_cubicweb-version-0.2.0, pyramid_cubicweb-debian-version-0.2.0-1 for changeset cd8308245d20

set debian version

Prepare version 0.2.0

Document the changes

Fix configuration loading when 'cubicweb.includes' is not set Closes #4849314

pep8

Provides requirements for rtd Related to #4849313

Initial documentation. Closes #4849313

Remove dead code

[profile] Add a profiling tool

[config] Read pyramid settings in a 'pyramid.ini' file If a 'pyramid-debug.ini' file is present, it will be used instead when debugmode is on. Closes #4811298

Move auth-related configuration to a dedicated module.

Fix cors 'origin' parameter passing Closes #4783343

[auth] Fix the config option name in the warning message

session -> cnx

Added tag pyramid_cubicweb-debian-version-0.1.3-1, pyramid_cubicweb-version-0.1.3 for changeset 0a7769e583c2

Set version 0.1.3

Cookie 'max_age' must be a integer, not a string. If not, the value is used verbatim for the 'expires' of the cookie, which is invalid. Closes #4731764.

Added tag pyramid_cubicweb-version-0.1.2, pyramid_cubicweb-debian-version-0.1.2-1 for changeset 5eb4e27e9998

Update debian changelog

Don't rollback if exception is HTTPSuccessful or HTTPRedirection In the request finishing, the 'cleanup' callback set by _cw_cnx automatically commit the transaction except is an exception is set on the request. Problem is, redirections and successul http return code can raise exceptions. This patch detects such exceptions and avoid rolling back the transaction. Closes #4566482

Set version to 0.1.2

Provides a full wsgi cubicweb application builder

Added tag pyramid_cubicweb-version-0.1.1, pyramid_cubicweb-debian-0.1.1-1 for changeset 9f3b9e610c3d

Set version 0.1.1

[auth] Make last_login_time updated. The update is done when the user logs in, then every time the authentication is reissued. Closes #4549891

Added tag pyramid_cubicweb-version-0.1.0, pyramid_cubicweb-debian-version-0.1.0-1 for changeset 3d60138100d6

Add a description to the debian package

Set version to 0.1.0

Initial debian packaging missing at least a long description and wsgicors dependency.

Move PyramidCWTest to pyramid_cubicweb.tests

Optimise repo_connect by skipping authenticate_user The authentication being handled by pyramid itself, going through the authentication stack to recreate the temporary session at each request is very costly. On my desktop, for a mostly static front page, the total time for delivering the page drops from 100ms to 47ms.

Use lightweight sessions Provides a lightweight version of repo.connect() that does not keep track of opened sessions. The speed gain on a mostly static front page is about 5% Warning ! This means that, for now, the "session_open" and "session_close" hooks are NOT called anymore.

Use 'wsgicors' for CORS handling. The CW CORS handling (in web/cors.py) is only able to work on cubicweb requests. When a request is not handled by bwcompat, we need a proper solution. The `wsgicors` library provides what we need as a wsgi middleware.

Add requirements

Handle '__setauthcookie' '__setauthcookie' is a form parameter added by the 'rememberme' cube. If present and equals to '1', the cookie max_age will be set to 7 days instead of being a session cookie. To make sure the auth cookie is renewed, the reissue_time is set to 1h.

Fix POST handling. The issues where revealed by the unittests, which are ported from the cubicweb wsgi tests.

Use AuthTktAuthenticationPolicy

DB-saved session data Related to #4291173

Handle properly the '/https/*' urls CW uses a url prefix to detect https behing a reverse-proxy. A more proper way to do that is documented here in the waitress documentation (waitress is the default pyramid wsgi server): https://waitress.readthedocs.org/en/latest/#using-behind-a-reverse-proxy A later version should implement this, or use waitress in the 'pyramid' command. Related to #4291181

Correctly pass the multiple parameters to the cubicweb request When naively converted to a dict, the webob MultiDict will not set the multiple parameters (a same name with several values) in a way CW can understand. MultiDict.mixed() however, does exactly what CW needs: list values for keys with multiple values, and single value for the others. Related to #4291181

Add a wsgi application factory suitable for wsgi servers. This factory can generate a wsgi application for a cubicweb instance. It reads the instance name from the CW_INSTANCE environment variable, and activates the debugmode if CW_DEBUG is defined in environment. It is usable by uwsgi as the 'module' parameter : CW_INSTANCE=test uwsgi --plugins python,http --http 0.0.0.0:8080 --module pyramid_cubicweb:wsgi_application()

If any cube has a 'includeme' attribute, call config.include on it Related to #4291181

Add a make_cubicweb_application function This function will be used by the 'pyramid' cubicweb-ctl command. Related to #4291173

Fix session closing for cubicweb 3.19 Related to #4291173

Convert cubicweb.NotFound to HTTPNotFound Related to #4291173

Use registry['cubicweb.registry'] instead of registry['cubicweb.appli'].vreg because the application may not be present. Related to #4291173

Use a predicate based view selection for handling /login This will allow easy addition of login handlers from the application or cubes Related to #4291173

Use a tween application instead of a catchall route. Using a catchall route has some drawbacks. Especially, we have no mean to have a route that would match only if no other one does AND no view matches either. Said differently, our default handler cannot be plugged on the route level nor the view level, because it is has to be activated only if nothing else works in the pyramid application. Using a tween application allow to handle requests that raises a HTTPNotFound error, while having the pyramid error handler still active between our tween app and the outside world. Related to #4291173

Document the view problem hypothesis. Related to #4291173

If the postlogin_path is 'login', redirect to '/' instead Related to #4291173

Put the login view in a separate module. Related to #4291173

Separate into 4 modules * init_instance: load the cubicweb repository from the `pyramid_cubicweb.instance` configuration key * defaults: provides cw-like defaults for the authentication and session management * core: make cubicweb use the authentication and session management of pyramid. It assumes the application provides the auth policies and session factory, and that the `cubicweb.*` registry entries are correctly initialised. This is this only required module or pyramid_cubicweb, the other ones are optional if the application provides its own versions of what they do. * bwcompat: provides a catchall route that delegate the request handling to an old-fashion cubicweb publisher (ie using url_resolver and controllers). Related to #4291173

Update the TODO list Related to #4291173

Provide instructions and a requirements list to quickly start the sample application. Related to #4291173

Use the pyramid session object as the cubiweb session.data (needs a patched cw 3.19) Related to #4291173

Add comments on parts we want to reconsider later Related to #4291173

Skip core_handle, add a context manager to handle cubicweb errors The context manager is also used to catch errors in render_view. It handles the 'external' errors raised by cubicweb code. The more internal errors, the one that should occur only in url resolving and cubicweb controllers, are handled directly in CubicWebPyramidHandler. ValidationError is handled by CubicWebPyramidHandler for now, but should probably be handled by cw_to_pyramid Related to #4291173

Documents dependencies on a ubuntu system Related to #4291173

Use short-lived cubicweb sessions to let pyramid actually handle the web sessions Related to #4291173

Isolate the default handler and extend its role The handler now does the job of CubicWebPublisher.main_handle_request() and calls CubicWebPublisher.core_handle(). Instead of using config.add_notfound_view, a catchall route is defined and the handler plugged to it. Related to #4291173

Add a basic sample application Related to #4291173

Integration pyramid and cubicweb authentication. We use pyramid sessions to store the cubicweb sessionid so we can reuse it when needed, or regenerate it if it was lost. The cubicweb sessionid is obtained from a login in the repo OR directly from the user identified by pyramid. Related to #4291173

[doc] Escape _ in README.md So that the Markdown is okay.

Initial implementation Set up a default route that passes requests to a cubicweb instance. The requests are wrapped in an adequate adapter so that cubicweb works with no change. Related to #4291173

Project Structure Related to #4291173

[tox] Have less test environments This is to reduce the load on CI server because of parallel clones of the repository in Docker containers. Only keep "server", and "web" environments separated. All other prior environments are in "misc". To avoid duplicate entries in requirements files, move them all in a "requirements" directory (this appears to be a "common" practice, see e.g. Celery). Adjustments in tests: * Rename cubicweb/hooks/test/unittest_notification.py so that it does not conflict with cubicweb/sobjects/test/unittest_notification.py during test discovery as they would have the same module name but different __file__ attribute. * Add "comment" cube to the list of expected cubes in unittest_cwconfig.py as this cube is pulled by requirements/test-mist.txt. Closes #15440662.

[repo] Don't crash on start when fs schema is missing some db schema entities This occurs usually while developping and we don't want systematically to rebuild the database to start the instance or run e.g. i18ninstance.

Let configuration option be overridden by environment variables Related to #13889793.

[tox] Add a dummy test-results.xml file in check-manifest and flake8 environments This is to work around Jenkins Junit plugin that does not apparently account for the absence of this file (though it accepts it to be empty).

[tox] Basic flake8 config

Use pkgutil.ImpLoader instead of a custom loader for cubes import redirection It just works fine (same implementation of load_module method as _CubesLoader introduced in d404fd8499dd) and is complete w.r.t. PEP 302 (all methods implemented). Related to #13001466.

[config] Make available_cubes aware of cubes installed as packages For this add a "cubicweb.cubes" entry points group which will be scanned through to find out installed cubes. Entries in this group are expected to expose the module name of a cube (i.e. `cubicweb_foo`). Note that CubicWebConfiguration's available_cubes method will return the module name of cubes as packages (cubicweb_foo), so we had to add a special "key" sorting function to keep cubes sorted as before, despite possible different distribution schemes. This makes it possible to handle loading of CTL plugins in an almost similar manner as before (just tweaking the package name from cube name in load_cwctl_plugins method). I had to tweak (again?) the test_cubes_path method in unittest_cwconfig.py but did not find out why. Apart from unforeseen bugs and pending documentation, this finishes the work on porting cubes to standard Python packages. Closes #13001466.

[skeleton] Use distname directly in setup.py distname is part of required metadata and modname cannot actually be used in place of it.

[skeleton] Fix modname in __pkginfo__

[test] Use plain unittest in unittest_cwctl.py

[devctl] Remove commented LiveServerCommand The code got commented in 2010 (changeset 5ab3f63b06ad)...

[pkg] Inline call to setup() in setup.py The point of having this "install" function, called in "main" mode is not clear. Better stick to standard practices.

[pkg] Drop old checks in setup.py

[tox] Exclude symlinks from check-manifest call These are not handled by check-manifest, see https://github.com/mgedmin/check-manifest/issues/69.

Drop the only reference to an '__insert' form key

Drop bits from the past (no more dbapi isolated installation)

[devtools] CWTC.request is gone, don't attempt to use it in CWTC.view So one will have to provide at least one of `rset` or `req` argument of the `view` method.

Merge 3.22 into 3.23 Some fixes there are necessary for client work depending on cubicweb 3.23.

[devtools] Handle i18ncube command for "cubes as packages" And add a test so that both layouts are tested. Move the test cube into a package. Related to #13001466.

Handle cubes as packages in cwconfig Rely on importlib.import_module in several places in cwconfig.py, so this would not work with python 2.6. Method available_cubes will not list cubes installed as package for now. I'm not sure it's worth implementing this method (used in `cubicweb-ctl list cubes` command) for new cubes layout as the same result can basically be obtained with `pip freeze | grep cubicweb_`. In unittest_cwconfig.py, duplicate CubicWebConfigurationTC test case to test both the "cubes as packages" layout and the "legacy cubes" layout. The former having a custom sys.path set (pointing to datapath('libpython') where all cubes' packages live) and the latter having the previous config attribute setup. All test data cubes are moved to packages in libpython directory and symlinks are introduced in the cubes directory. Related to #13001466.

[cwconfig] Reorder imports by alphabetic order

Add an import redirect hook from "cubes.<name>" to "cubicweb_<name>" The hook consists of a finder and a loader implemented following PEP-302; it is responsible for loading cubes distributed as packages (i.e. installed as ``cubicweb_<name>`` in site-packages) but imported (in client code) as ``from cubes.<name> import ...``. So this is a transitional mechanism allowing cubes following the new layout to be used by old-style cubes/applications. The importer is installed upon calling CubicWebConfiguration's cls_adjust_sys_path method (also called in cubicweb.devtools.__init__.py, which is a prerequisite for importing any "legacy" cube. The loading of old-style cubes is still handled by the CubicWeb configuration, based on adjustment of sys.path etc. Related to #13001466.

[devtools] Update skeleton's setup.py to install cube as a package Most of the prior logic of skeleton's setup.py gets dropped as installing a cube as a "classic package" (i.e. in site-packages) is just the default behavior of distutils. Also add a test checking installation of new cube. Related to #13001466.

[devtools] Adjust skeleton setup.py to have it read information from __pkginfo__.py Since the later is now installed in cubicweb_<CUBENAME> directory. Add a test ensuring that `python setup.py sdist` works on newly created cube (which should at least verify that setup.py is valid). Related to #13001466.

[devtools] Update skeleton and newcube command to cube as package layout Related to #13001466.

[pkg] Add missing entries to MANIFEST.in Detected by running `check-manifest <https://github.com/mgedmin/check-manifest>`_. Add a tox environment accordingly. Closes #15229018.

[test] Add a test for toolsutils.read_config

[test] Use plain unittest in unittest_toolsutils

[skeleton] Remove test/pytestconf.py

[pkg] Make it clearer which entries of __pkginfo__ are actually used

[test] Avoid usage of lgc.testlib TestCase in unittest_cwconfig.py

[test] Move setup of spa2rql tests into setUpClass method Thus avoiding cryptic errors upon unittest discovery.

[schema] Set CubicWebSchemaLoader's extrapath using config eponymous property

[doc] Document change of instances location in virtualenv installation Closes #14789440.

[cwconfig] Do not override user-defined CW_MODE in virtualenv setup And thus consider `_forced_mode` first. Related to #14789440.

Added tag 3.22.4, debian/3.22.4-1, centos/3.22.4-1 for changeset 92db0bb8e26d

[pkg] remove .dev0 from version

[pkg] 3.22.4

[session] Ensure access to rql varmaker always mark the session as dirty When one accesses the rql_varmaker, that's usually to use it. The pb was that when the varmaker was already in page's data (which is stored as session data), session storage such as redis won't see that the session data is dirty and has to be stored back at the end of the request. To fix this, systematically call set_page_data. (grafted from 3432f0e2540d)

[test] fix test_printable_value_bytes with current pygments An empty span was added in https://bitbucket.org/birkenfeld/pygments-main/commits/164574c13533195a555181a2b9c685fea2470403 (grafted from 74b04a88d28a)

[webctl] set uid of file produced by gen-static-datadir (closes #11298794)

[webconfig] ensure uicache content has correct uid (related to #11298794)

[cwconfig] add the CubicWebConfiguration.ensure_uid_directory() method (related to #11298794) that recursively enforce uid on files.

[cwconfig] extract uid file setting code from check_writeable_uid_directory (related to #11298794) into an ensure_uid() method

[web] Fix bug with usage of os.rename under windows environment In 7c386161ebd6 we removed cache handling from property sheet and introduced usage of a tempfile + os.rename to get atomic generation of files. The pb is that this is not portable, since under windows os.rename will raise an exception if the file already exists (because there is no way to write a file atomatically in such case). This kind of thing should be out of the CW scope anyway, so implements a quick & dirty fix in the mean time. Closes #14214794

[migration] More resilient migration script Query the information schema to retrieve existing constraint (primary key and unique) and drop them. Thus don't use anymore DROP CONSTRAINT IF EXISTS (not implemented by e.g. old versions of sqlserver). To do so a missing DISTINCT was necessary to avoid duplicates.

[session] Ensure access to rql varmaker always mark the session as dirty When one accesses the rql_varmaker, that's usually to use it. The pb was that when the varmaker was already in page's data (which is stored as session data), session storage such as redis won't see that the session data is dirty and has to be stored back at the end of the request. To fix this, systematically call set_page_data.

typo

[native] Use cnx.system_sql instead of doexec method in syntax_tree_search The latter handles connection issues and tries to reconnect. Method syntax_tree_search appears to be called from quite a few place where it seems useful to be resilient to connection issues (authentication for instance). This reconnection logic appears to have disappeared at some point when working on https://www.cubicweb.org/2919309. Could not find the exact reason though...

Remove usage of cnx.ensure_cnx_set context manager It has been deprecated since 3.21, no reason to keep using it internally.

[hooks/test/synschema] remove an unneeded skipTest. Related to #5557633.

[test] Check retcode of subprocesses in devctl tests And display decoded stdout in case of failure.

merge 3.23

[pkg] Bump version to 3.24.0.dev0

[pkg] Drop NO_SETUPTOOLS option setuptools is the defacto standard nowadays. The only place where NO_SETUPTOOLS option was used is during packaging (RPM, Debian) and even in this context, it seems that setuptools is the way to go (see https://wiki.debian.org/Python/LibraryStyleGuide).

[debian] update cubicweb-dev.lintian-overrides cubicweb/skeleton/debian/rules.tmpl is now cubicweb/skeleton/debian/rules prevent lintian from complaining

[pkg] Properly export data files in setup.py and adjust "newcube" test With the new package layout (everything under "cubicweb" package), the custom install_lib rule which makes use of include_dirs defined in __pkginfo__.py did not prepend the package name to source directories to be copied. Fixing this. Also, in setup.py's export() function, the destination directories' path to be created during source tree walk was wrong. All this makes cubicweb/skeleton directory (which is not a package) properly installed by setup.py. The test in cubicweb/devtools/test/unittest_devctl.py wasn't properly implemented because it used an installation of cubicweb in "develop" mode which shadows such packaging issues. Also it used "python -m cubicweb" instead of directly "cubicweb-ctl" and the former appears to fall back to using the cubicweb package *from sources* instead of the installed one. Now that this test runs against the installed version of cubicweb, fix MANIFEST.in to include tox.ini files (cubicweb's and skeleton's) as this is expected from the test. Closes #14127941.

[pkg] Handle verbose option in setup.py's export function Make debugging easier.

[pkg] Drop unused EMPTY_FILE global variable

[tox] Do not install package in "develop" This might shadow so packaging issues. And user can still pass --develop to tox.

[pkg] Bump version to 3.23.2

pep8 bits

[schema] Add a method on yams constraints to compute its unique name used to identify it in the DB backend.

[migration] Test and fix case of addition of an attribute with some serialized constraint It was crashing on attempt to remove an unexisting constraint. This code may be removed safely as it's expected to be handled by the constraint removal hook. Notice that adding a vocabulary on shortpara makes yams change the varchar max size. This is arguable but not the point of this test, so simply adapt the max size defined in the schema coherently so we don't have to bother with that at this point.

[test] Add a migration test for update of a static vocabulary constraint This is a constraint serialized to be checked by the DB backend, hence ensure the old constraint is dropped and the new one is created.

[schema2sql] Avoid "parsing" SQL statements for database initialization A big SQL string was generated, then splitted. This caused bug if some value in the schema (eg vocabulary, default) contained the separator (';'). To properly fix this, yield each individual statement instead of generating a string. Closes #14050899

[schema2sql] Drop most of the DB DROP related code This code is dead for a while, it has been superseded by `sql_drop_all_user_tables`. Related to #14050899

[hooks] Use already imported module alias

[schema2sql] Rename as_sql to constraint_value_as_sql which is easier to grasp independantly of the context.

[schema2sql] Drop unused indent argument on aschema2sql Related to #14050899

[schema2sql] More docstrings and minor API improvements Along the way, renamed local `column` variable to `attr` where we're NOT manipulating a DB column name but a schema attribute. Related to #14050899

[schema2sql] inline eschema_attrs method * not a public API * doesn't makes things easier to grasp -> useless Related to #14050899

[schema2sql] Give a rdef to check_constraint This simplifies the API. Related to #14050899

merge 3.20.16 in 3.22