Document the changes

Fix configuration loading when 'cubicweb.includes' is not set Closes #4849314

pep8

Provides requirements for rtd Related to #4849313

Initial documentation. Closes #4849313

Remove dead code

[profile] Add a profiling tool

[config] Read pyramid settings in a 'pyramid.ini' file If a 'pyramid-debug.ini' file is present, it will be used instead when debugmode is on. Closes #4811298

Move auth-related configuration to a dedicated module.

Fix cors 'origin' parameter passing Closes #4783343

[auth] Fix the config option name in the warning message

session -> cnx

Added tag pyramid_cubicweb-debian-version-0.1.3-1, pyramid_cubicweb-version-0.1.3 for changeset 0a7769e583c2

Set version 0.1.3

Cookie 'max_age' must be a integer, not a string. If not, the value is used verbatim for the 'expires' of the cookie, which is invalid. Closes #4731764.

Added tag pyramid_cubicweb-version-0.1.2, pyramid_cubicweb-debian-version-0.1.2-1 for changeset 5eb4e27e9998

Update debian changelog

Don't rollback if exception is HTTPSuccessful or HTTPRedirection In the request finishing, the 'cleanup' callback set by _cw_cnx automatically commit the transaction except is an exception is set on the request. Problem is, redirections and successul http return code can raise exceptions. This patch detects such exceptions and avoid rolling back the transaction. Closes #4566482

Set version to 0.1.2

Provides a full wsgi cubicweb application builder

Added tag pyramid_cubicweb-version-0.1.1, pyramid_cubicweb-debian-0.1.1-1 for changeset 9f3b9e610c3d

Set version 0.1.1

[auth] Make last_login_time updated. The update is done when the user logs in, then every time the authentication is reissued. Closes #4549891

Added tag pyramid_cubicweb-version-0.1.0, pyramid_cubicweb-debian-version-0.1.0-1 for changeset 3d60138100d6

Add a description to the debian package

Set version to 0.1.0

Initial debian packaging missing at least a long description and wsgicors dependency.

Move PyramidCWTest to pyramid_cubicweb.tests

Optimise repo_connect by skipping authenticate_user The authentication being handled by pyramid itself, going through the authentication stack to recreate the temporary session at each request is very costly. On my desktop, for a mostly static front page, the total time for delivering the page drops from 100ms to 47ms.

Use lightweight sessions Provides a lightweight version of repo.connect() that does not keep track of opened sessions. The speed gain on a mostly static front page is about 5% Warning ! This means that, for now, the "session_open" and "session_close" hooks are NOT called anymore.

Use 'wsgicors' for CORS handling. The CW CORS handling (in web/cors.py) is only able to work on cubicweb requests. When a request is not handled by bwcompat, we need a proper solution. The `wsgicors` library provides what we need as a wsgi middleware.

Add requirements

Handle '__setauthcookie' '__setauthcookie' is a form parameter added by the 'rememberme' cube. If present and equals to '1', the cookie max_age will be set to 7 days instead of being a session cookie. To make sure the auth cookie is renewed, the reissue_time is set to 1h.

Fix POST handling. The issues where revealed by the unittests, which are ported from the cubicweb wsgi tests.

Use AuthTktAuthenticationPolicy

DB-saved session data Related to #4291173

Handle properly the '/https/*' urls CW uses a url prefix to detect https behing a reverse-proxy. A more proper way to do that is documented here in the waitress documentation (waitress is the default pyramid wsgi server): https://waitress.readthedocs.org/en/latest/#using-behind-a-reverse-proxy A later version should implement this, or use waitress in the 'pyramid' command. Related to #4291181

Correctly pass the multiple parameters to the cubicweb request When naively converted to a dict, the webob MultiDict will not set the multiple parameters (a same name with several values) in a way CW can understand. MultiDict.mixed() however, does exactly what CW needs: list values for keys with multiple values, and single value for the others. Related to #4291181

Add a wsgi application factory suitable for wsgi servers. This factory can generate a wsgi application for a cubicweb instance. It reads the instance name from the CW_INSTANCE environment variable, and activates the debugmode if CW_DEBUG is defined in environment. It is usable by uwsgi as the 'module' parameter : CW_INSTANCE=test uwsgi --plugins python,http --http 0.0.0.0:8080 --module pyramid_cubicweb:wsgi_application()

If any cube has a 'includeme' attribute, call config.include on it Related to #4291181

Add a make_cubicweb_application function This function will be used by the 'pyramid' cubicweb-ctl command. Related to #4291173

Fix session closing for cubicweb 3.19 Related to #4291173

Convert cubicweb.NotFound to HTTPNotFound Related to #4291173

Use registry['cubicweb.registry'] instead of registry['cubicweb.appli'].vreg because the application may not be present. Related to #4291173

Use a predicate based view selection for handling /login This will allow easy addition of login handlers from the application or cubes Related to #4291173

Use a tween application instead of a catchall route. Using a catchall route has some drawbacks. Especially, we have no mean to have a route that would match only if no other one does AND no view matches either. Said differently, our default handler cannot be plugged on the route level nor the view level, because it is has to be activated only if nothing else works in the pyramid application. Using a tween application allow to handle requests that raises a HTTPNotFound error, while having the pyramid error handler still active between our tween app and the outside world. Related to #4291173

Document the view problem hypothesis. Related to #4291173

If the postlogin_path is 'login', redirect to '/' instead Related to #4291173

Put the login view in a separate module. Related to #4291173

Separate into 4 modules * init_instance: load the cubicweb repository from the `pyramid_cubicweb.instance` configuration key * defaults: provides cw-like defaults for the authentication and session management * core: make cubicweb use the authentication and session management of pyramid. It assumes the application provides the auth policies and session factory, and that the `cubicweb.*` registry entries are correctly initialised. This is this only required module or pyramid_cubicweb, the other ones are optional if the application provides its own versions of what they do. * bwcompat: provides a catchall route that delegate the request handling to an old-fashion cubicweb publisher (ie using url_resolver and controllers). Related to #4291173

Update the TODO list Related to #4291173

Provide instructions and a requirements list to quickly start the sample application. Related to #4291173

Use the pyramid session object as the cubiweb session.data (needs a patched cw 3.19) Related to #4291173

Add comments on parts we want to reconsider later Related to #4291173

Skip core_handle, add a context manager to handle cubicweb errors The context manager is also used to catch errors in render_view. It handles the 'external' errors raised by cubicweb code. The more internal errors, the one that should occur only in url resolving and cubicweb controllers, are handled directly in CubicWebPyramidHandler. ValidationError is handled by CubicWebPyramidHandler for now, but should probably be handled by cw_to_pyramid Related to #4291173

Documents dependencies on a ubuntu system Related to #4291173

Use short-lived cubicweb sessions to let pyramid actually handle the web sessions Related to #4291173

Isolate the default handler and extend its role The handler now does the job of CubicWebPublisher.main_handle_request() and calls CubicWebPublisher.core_handle(). Instead of using config.add_notfound_view, a catchall route is defined and the handler plugged to it. Related to #4291173

Add a basic sample application Related to #4291173

Integration pyramid and cubicweb authentication. We use pyramid sessions to store the cubicweb sessionid so we can reuse it when needed, or regenerate it if it was lost. The cubicweb sessionid is obtained from a login in the repo OR directly from the user identified by pyramid. Related to #4291173

[doc] Escape _ in README.md So that the Markdown is okay.

Initial implementation Set up a default route that passes requests to a cubicweb instance. The requests are wrapped in an adequate adapter so that cubicweb works with no change. Related to #4291173

Project Structure Related to #4291173

[tox] Have less test environments This is to reduce the load on CI server because of parallel clones of the repository in Docker containers. Only keep "server", and "web" environments separated. All other prior environments are in "misc". To avoid duplicate entries in requirements files, move them all in a "requirements" directory (this appears to be a "common" practice, see e.g. Celery). Adjustments in tests: * Rename cubicweb/hooks/test/unittest_notification.py so that it does not conflict with cubicweb/sobjects/test/unittest_notification.py during test discovery as they would have the same module name but different __file__ attribute. * Add "comment" cube to the list of expected cubes in unittest_cwconfig.py as this cube is pulled by requirements/test-mist.txt. Closes #15440662.

[repo] Don't crash on start when fs schema is missing some db schema entities This occurs usually while developping and we don't want systematically to rebuild the database to start the instance or run e.g. i18ninstance.

Let configuration option be overridden by environment variables Related to #13889793.

[tox] Add a dummy test-results.xml file in check-manifest and flake8 environments This is to work around Jenkins Junit plugin that does not apparently account for the absence of this file (though it accepts it to be empty).

[tox] Basic flake8 config

Use pkgutil.ImpLoader instead of a custom loader for cubes import redirection It just works fine (same implementation of load_module method as _CubesLoader introduced in d404fd8499dd) and is complete w.r.t. PEP 302 (all methods implemented). Related to #13001466.

[config] Make available_cubes aware of cubes installed as packages For this add a "cubicweb.cubes" entry points group which will be scanned through to find out installed cubes. Entries in this group are expected to expose the module name of a cube (i.e. `cubicweb_foo`). Note that CubicWebConfiguration's available_cubes method will return the module name of cubes as packages (cubicweb_foo), so we had to add a special "key" sorting function to keep cubes sorted as before, despite possible different distribution schemes. This makes it possible to handle loading of CTL plugins in an almost similar manner as before (just tweaking the package name from cube name in load_cwctl_plugins method). I had to tweak (again?) the test_cubes_path method in unittest_cwconfig.py but did not find out why. Apart from unforeseen bugs and pending documentation, this finishes the work on porting cubes to standard Python packages. Closes #13001466.

[skeleton] Use distname directly in setup.py distname is part of required metadata and modname cannot actually be used in place of it.

[skeleton] Fix modname in __pkginfo__

[test] Use plain unittest in unittest_cwctl.py

[devctl] Remove commented LiveServerCommand The code got commented in 2010 (changeset 5ab3f63b06ad)...

[pkg] Inline call to setup() in setup.py The point of having this "install" function, called in "main" mode is not clear. Better stick to standard practices.

[pkg] Drop old checks in setup.py

[tox] Exclude symlinks from check-manifest call These are not handled by check-manifest, see https://github.com/mgedmin/check-manifest/issues/69.

Drop the only reference to an '__insert' form key

Drop bits from the past (no more dbapi isolated installation)

[devtools] CWTC.request is gone, don't attempt to use it in CWTC.view So one will have to provide at least one of `rset` or `req` argument of the `view` method.

Merge 3.22 into 3.23 Some fixes there are necessary for client work depending on cubicweb 3.23.

[devtools] Handle i18ncube command for "cubes as packages" And add a test so that both layouts are tested. Move the test cube into a package. Related to #13001466.

Handle cubes as packages in cwconfig Rely on importlib.import_module in several places in cwconfig.py, so this would not work with python 2.6. Method available_cubes will not list cubes installed as package for now. I'm not sure it's worth implementing this method (used in `cubicweb-ctl list cubes` command) for new cubes layout as the same result can basically be obtained with `pip freeze | grep cubicweb_`. In unittest_cwconfig.py, duplicate CubicWebConfigurationTC test case to test both the "cubes as packages" layout and the "legacy cubes" layout. The former having a custom sys.path set (pointing to datapath('libpython') where all cubes' packages live) and the latter having the previous config attribute setup. All test data cubes are moved to packages in libpython directory and symlinks are introduced in the cubes directory. Related to #13001466.

[cwconfig] Reorder imports by alphabetic order

Add an import redirect hook from "cubes.<name>" to "cubicweb_<name>" The hook consists of a finder and a loader implemented following PEP-302; it is responsible for loading cubes distributed as packages (i.e. installed as ``cubicweb_<name>`` in site-packages) but imported (in client code) as ``from cubes.<name> import ...``. So this is a transitional mechanism allowing cubes following the new layout to be used by old-style cubes/applications. The importer is installed upon calling CubicWebConfiguration's cls_adjust_sys_path method (also called in cubicweb.devtools.__init__.py, which is a prerequisite for importing any "legacy" cube. The loading of old-style cubes is still handled by the CubicWeb configuration, based on adjustment of sys.path etc. Related to #13001466.

[devtools] Update skeleton's setup.py to install cube as a package Most of the prior logic of skeleton's setup.py gets dropped as installing a cube as a "classic package" (i.e. in site-packages) is just the default behavior of distutils. Also add a test checking installation of new cube. Related to #13001466.

[devtools] Adjust skeleton setup.py to have it read information from __pkginfo__.py Since the later is now installed in cubicweb_<CUBENAME> directory. Add a test ensuring that `python setup.py sdist` works on newly created cube (which should at least verify that setup.py is valid). Related to #13001466.

[devtools] Update skeleton and newcube command to cube as package layout Related to #13001466.

[pkg] Add missing entries to MANIFEST.in Detected by running `check-manifest <https://github.com/mgedmin/check-manifest>`_. Add a tox environment accordingly. Closes #15229018.

[test] Add a test for toolsutils.read_config

[test] Use plain unittest in unittest_toolsutils

[skeleton] Remove test/pytestconf.py

[pkg] Make it clearer which entries of __pkginfo__ are actually used

[test] Avoid usage of lgc.testlib TestCase in unittest_cwconfig.py

[test] Move setup of spa2rql tests into setUpClass method Thus avoiding cryptic errors upon unittest discovery.

[schema] Set CubicWebSchemaLoader's extrapath using config eponymous property

[doc] Document change of instances location in virtualenv installation Closes #14789440.

[cwconfig] Do not override user-defined CW_MODE in virtualenv setup And thus consider `_forced_mode` first. Related to #14789440.

Added tag 3.22.4, debian/3.22.4-1, centos/3.22.4-1 for changeset 92db0bb8e26d

[pkg] remove .dev0 from version

[pkg] 3.22.4

[session] Ensure access to rql varmaker always mark the session as dirty When one accesses the rql_varmaker, that's usually to use it. The pb was that when the varmaker was already in page's data (which is stored as session data), session storage such as redis won't see that the session data is dirty and has to be stored back at the end of the request. To fix this, systematically call set_page_data. (grafted from 3432f0e2540d)

[test] fix test_printable_value_bytes with current pygments An empty span was added in https://bitbucket.org/birkenfeld/pygments-main/commits/164574c13533195a555181a2b9c685fea2470403 (grafted from 74b04a88d28a)

[webctl] set uid of file produced by gen-static-datadir (closes #11298794)

[webconfig] ensure uicache content has correct uid (related to #11298794)

[cwconfig] add the CubicWebConfiguration.ensure_uid_directory() method (related to #11298794) that recursively enforce uid on files.

[cwconfig] extract uid file setting code from check_writeable_uid_directory (related to #11298794) into an ensure_uid() method

[web] Fix bug with usage of os.rename under windows environment In 7c386161ebd6 we removed cache handling from property sheet and introduced usage of a tempfile + os.rename to get atomic generation of files. The pb is that this is not portable, since under windows os.rename will raise an exception if the file already exists (because there is no way to write a file atomatically in such case). This kind of thing should be out of the CW scope anyway, so implements a quick & dirty fix in the mean time. Closes #14214794

[migration] More resilient migration script Query the information schema to retrieve existing constraint (primary key and unique) and drop them. Thus don't use anymore DROP CONSTRAINT IF EXISTS (not implemented by e.g. old versions of sqlserver). To do so a missing DISTINCT was necessary to avoid duplicates.

[session] Ensure access to rql varmaker always mark the session as dirty When one accesses the rql_varmaker, that's usually to use it. The pb was that when the varmaker was already in page's data (which is stored as session data), session storage such as redis won't see that the session data is dirty and has to be stored back at the end of the request. To fix this, systematically call set_page_data.

typo

[native] Use cnx.system_sql instead of doexec method in syntax_tree_search The latter handles connection issues and tries to reconnect. Method syntax_tree_search appears to be called from quite a few place where it seems useful to be resilient to connection issues (authentication for instance). This reconnection logic appears to have disappeared at some point when working on https://www.cubicweb.org/2919309. Could not find the exact reason though...

Remove usage of cnx.ensure_cnx_set context manager It has been deprecated since 3.21, no reason to keep using it internally.

[hooks/test/synschema] remove an unneeded skipTest. Related to #5557633.

[test] Check retcode of subprocesses in devctl tests And display decoded stdout in case of failure.

merge 3.23

[pkg] Bump version to 3.24.0.dev0

[pkg] Drop NO_SETUPTOOLS option setuptools is the defacto standard nowadays. The only place where NO_SETUPTOOLS option was used is during packaging (RPM, Debian) and even in this context, it seems that setuptools is the way to go (see https://wiki.debian.org/Python/LibraryStyleGuide).

[debian] update cubicweb-dev.lintian-overrides cubicweb/skeleton/debian/rules.tmpl is now cubicweb/skeleton/debian/rules prevent lintian from complaining

[pkg] Properly export data files in setup.py and adjust "newcube" test With the new package layout (everything under "cubicweb" package), the custom install_lib rule which makes use of include_dirs defined in __pkginfo__.py did not prepend the package name to source directories to be copied. Fixing this. Also, in setup.py's export() function, the destination directories' path to be created during source tree walk was wrong. All this makes cubicweb/skeleton directory (which is not a package) properly installed by setup.py. The test in cubicweb/devtools/test/unittest_devctl.py wasn't properly implemented because it used an installation of cubicweb in "develop" mode which shadows such packaging issues. Also it used "python -m cubicweb" instead of directly "cubicweb-ctl" and the former appears to fall back to using the cubicweb package *from sources* instead of the installed one. Now that this test runs against the installed version of cubicweb, fix MANIFEST.in to include tox.ini files (cubicweb's and skeleton's) as this is expected from the test. Closes #14127941.

[pkg] Handle verbose option in setup.py's export function Make debugging easier.

[pkg] Drop unused EMPTY_FILE global variable

[tox] Do not install package in "develop" This might shadow so packaging issues. And user can still pass --develop to tox.

[pkg] Bump version to 3.23.2

pep8 bits

[schema] Add a method on yams constraints to compute its unique name used to identify it in the DB backend.

[migration] Test and fix case of addition of an attribute with some serialized constraint It was crashing on attempt to remove an unexisting constraint. This code may be removed safely as it's expected to be handled by the constraint removal hook. Notice that adding a vocabulary on shortpara makes yams change the varchar max size. This is arguable but not the point of this test, so simply adapt the max size defined in the schema coherently so we don't have to bother with that at this point.

[test] Add a migration test for update of a static vocabulary constraint This is a constraint serialized to be checked by the DB backend, hence ensure the old constraint is dropped and the new one is created.

[schema2sql] Avoid "parsing" SQL statements for database initialization A big SQL string was generated, then splitted. This caused bug if some value in the schema (eg vocabulary, default) contained the separator (';'). To properly fix this, yield each individual statement instead of generating a string. Closes #14050899

[schema2sql] Drop most of the DB DROP related code This code is dead for a while, it has been superseded by `sql_drop_all_user_tables`. Related to #14050899

[hooks] Use already imported module alias

[schema2sql] Rename as_sql to constraint_value_as_sql which is easier to grasp independantly of the context.

[schema2sql] Drop unused indent argument on aschema2sql Related to #14050899

[schema2sql] More docstrings and minor API improvements Along the way, renamed local `column` variable to `attr` where we're NOT manipulating a DB column name but a schema attribute. Related to #14050899

[schema2sql] inline eschema_attrs method * not a public API * doesn't makes things easier to grasp -> useless Related to #14050899

[schema2sql] Give a rdef to check_constraint This simplifies the API. Related to #14050899

merge 3.20.16 in 3.22

merge 3.20.16 in 3.21

Added tag 3.20.16, debian/3.20.16-1, centos/3.20.16-1 for changeset e60a8e5d29ef

[debian] add a lintian override for source-is-missing about jquery.flot.js

[pkg] 3.20.16

[dataimport] fix count_lines function (closes #14407650) ensure i allways exists whatever f content is

[captcha] improve a bit the redability of generated captcha images (closes #13500378)

3.20 branch

merge 3.23 changes

Added tag 3.23.1, centos/3.23.1-1, debian/3.23.1-1 for changeset 8f035d5d5953

prepare 3.23.1

[debian] Adjust symlink to cubicweb package so that documention builds fine Broken since source tree re-organization.

[dataimport] Restore handling of Binary in ExtEntity's values Broken since 62a7100d774b. Closes #13973741.

[tox] Ensure virtualenv's python is used to build the documentation

[doc] Reference 3.23 page in changelog

[tox] Ensure virtualenv's python is used to build the documentation

Backed out changeset b288debc6736 Only superusers can set lc_messages variable. This prevents connection in many cases. Closes #14010345.

Merge with 3.23

[pkg] Bump version to 3.22.4.dev0

Added tag 3.23.0, centos/3.23.0-1, debian/3.23.0-1 for changeset 4b07f2a79021

[doc] Fix symlinks of sphinx theme images

Added tag 3.23.0, centos/3.23.0-1, debian/3.23.0-1 for changeset 93b0f836cb07

[debian] Move python-unittest2 >= 0.7.0 as runtime dependency It seems to me that this is not used during Debian package build (tests are not run), but should be needed at runtime with the cubicweb-dev package. Also, at the moment, we have no pbuilder setting with this package available and it blocks the release.

Symlink __pkginfo__.py at root Maybe this will make apycot work again...

[3.23] More release notes

i18n update

[doc/changes] CubicWebTC.set_description no longer available

Add some release notes

[pkg] Version 3.23.0

[pkg] Bump yams version in cubicweb-common Debian package

[tox] Drop yams from dev-requirements.txt Yams 0.44.0 got released.

Merge with 3.22

Merge with 3.21

backport 3.21 changes

backport 3.20 changes

[sobjects] Drop custom unique constraint checks in user registration service This constraints (on unique CWUser's login and EmailAddress's login) are handled by the backend. This follows "[hooks/integrity] kill CheckUniqueHook". Extend test for the ValidationError along the way.

[test] Add a test for user registration unicity constraint on email

[test] Clarify user registration service tests * split permissions test appart * avoid usage of internal_cnx as this does not seem need

[test/hooks] pep8 and unittest.main instead in unittest_hooks.py

[hooks/integrity] kill CheckUniqueHook It's the backend's job. The noticeable consequence (as can be seen in tests updated here) is that error message are a bit less user-friendly.

[migration] drop identity_relation if it exists It has been generated erroneously for a while, then fixed but not properly deleted from old databases. Don't use IF EXISTS for sqlserver compat.

[migration] Reserialize all constraints since yams introduce new JSON serializing format So we may drop support for older format in yams at some point.

[migration] Add IF EXISTS on DROP CONSTRAINT problem encountered on migration of a legacy database

[schema sync] Rename index when an entity type is renamed So we may still find them later using their expected name, and avoid collisions. Closes #13822045

[c-c] Add a command to check database index List indices expected by the schema but not in the database and vice-versa Closes #13822027

[sql gen] Control size of index name using an md5 hash When using concatenation of table and column, we may hit the backend size limit on index names, which results on truncated name (64 for postgres). Then we may have conflict or be unable to find expected index on later synchronization attempt. This is done by monkeypatching lgdb for now to avoid dependancy problems. Unique and primary key indices of entity types handling is left to the backend, as they are not explicitly named. See https://www.logilab.org/ticket/4789189 Closes #13821949

[sql gen] Explicitly name unique index This patch hotfix logilab database for https://www.logilab.org/6662663 and fix database creation to stop using UNIQUE in the table schema but rather create the index later on, so its name is under control. Also consider this change in schema synchronization hooks. This is preliminary work related to #13822045

Update to yams 0.44 API

[test] Fix randomness in tests

[test] Drop unused import

[serverctl] allow to sync multiple and all sources in source-sync command Breaking change: by default source are not synced anymore if it is fresh (regards to synchronization-interval). To get the previous behavior an option --force is added. Closes #13886467

[tox] Split migractions tests from server environment These are the longest tests in this environment, so running them apart (in parallel of others) would hopefully speed up the whole suite.

[dataimport] Fix rdef loopup in importer

[py3k] left-over unicode() in authobjs.py

[server] Improve massive deletion of entities sqlserver is limited on the "IN" array size and produce an error "The query processor ran out of internal resources and could not produce a query plan". The exact limitation is unclear but can occur with > 10000 item in the array. See https://support.microsoft.com/en-us/kb/288095 Now we process the deletion by batch of 10000, some quick benchmarks show that there is no performance impact: delete_in(50000): 46.765182972 delete_in_batch(50000): 46.3242430687 delete_in(100000): 62.2969429493 delete_in_batch(100000): 61.8402349949

[hooks] Add a bunch of tests for session synchronization And change unicode into six.text_type in authobjs.py spotted by new tests.