[massive store] Rework constraint/index handling The basic idea is to keep the primary constraint on entities.eid since it's heavily used in metadata insertions. Other option would have been to drop / recreate but its very costly on big database, and the index is used for insertion into the entities table itself, so it's not worth droping it at a first glance. Also, keeping it avoids to systematically drop all constraints which depends on it. We may thus now lazily drop constraints, only on insertion of some etype/rtype for the related table. Related to #15538359

[massive store] Lazy removal of constraints and metadata indexes They should not be removed during store's init, because we may want to query the database with its index between store creation and call to prepare_insert* (e.g. to build the extid2eid map). Along the way: * rename drop_metadata_constraints into drop_metadata_indexes, because that's what it does * rework a bit impacted tests Closes #15538359

[repository] Drop the index on entities.type It should be now only used in maintenance queries in checkintegrity. Those are not worth maintaining it. Closes #15538317

[source] Drop source mapping handling It was only used by the cwxmlparser which has been deleted. This is too complex for litle benefit, we don't want to maintain that in cubicweb.

[repository] Drop the entities.extid column and associated cache This was not necessary anymore with promoted usage of the new data import API. Turn repository's _type_extid_cache to _type_cache with only the entity's type as key. This introduces an backward incompatible change: entity_metas dict doesn't contains anymore the extid key, but it doesn't seem used at all anywhere, so this sounds acceptable. Closes #15538317

[repository] Drop the entities.asource column It not used anymore thanks to previous patches. Closes #15538288

[integrity check] Stop using entities.asource column in integrity checks Simply check if entities have a cw_source, and link it to the system source when not found. Related to #15538288

[rql2sql/test] Ensure test results predictability Without this, test were randomly failing within tox due to different order of solutions.

[rql2sql] Stop generating SQL query from RQL using entities.type This was necessary in the "true multi-sources" time, it's not anymore, while maintaining this index is costly. Related to #15538317

[test] Remove some usages of lgc.testlib

[test] Avoid pytest discovery warnings Having TestServerConfiguration and alike in the test module namespace causes pytest discovery errors, because it thinks it's a test class.

flake8 and all * update some copyright * drop most __docformat__ declaration * fix some flake8 warnings / errors

[schema2sql] Drop deprecated comment

[cnx] Use entity_type instead of entity_metas()['type'] The latter is deprecated in favor of the former which has been introduced on cnx to replace it.

[entity] Reimplement and deprecate entity.cw_metainformation It shouldn't rely anymore on cnx.entity_metas which is itself deprecated, and the cw_metainformation API is similarly not needed anymore, access directly to cwuri / cw_source if needed. Related to #15538288

[repository] Drop type_and_source_from_eid and rename related cache We don't want to handle anymore the entities.asource column so we shouldn't use those anymore. Also rename repository's _type_source_cache into _type_extid_cache as this is what it's containing now. Do similar renaming to the system source API. Related to #15538288

[entity] Stop returning source information in cw_metainformation() We should not need that. It has been introduced to handle things we should not do and probably don't do anymore (e.g. attempting to grasp recursive import of several sites). Related to #15538288.

[json] Stop serializing cw_source into default json representation of an entity Related to #15538288

Drop deprecated LDAP related script both are relying on the 'entities.source' column which has been dropped in 3.19. They have been written with the old ldapsource in mind, which has been dropped at that time.

[entity] Stop linking to external site for external entities This behaviour was ok when we had "true" multi-sources but its rather painful with datafeed sources. Also, it makes absolute_url() costlier than what it should. Besides, it relies on cw_metainformation()['source'] that is pending for removal. Instead, add a link to the original object in the metadata view (the one that displays eid and source at the bottom right corner of the primary view). Related to #15538288.

[datafeed] Drop entity deletion handling in the default source / parser This should be handled by specific parser to avoid using "entities.asource". Reimplement it in the ldap parser. Related to #15538288

[sources] Drop 'moved_entities' table handling This was clumsy and could be handled using exturi / cwsource, as the ldap source now does.

[repository] Drop deprecated extid2eid API and friends This will break cwxmlparser based sources. They should be rewritten using a specific parser, based on xml representation or on rqlio. This is harsh but allows a so big cleanup of the code base. Furthermore, it's necessary for asource/extid handling in the entities table which is costly for most app that don't care at all about that... In this cset, delete: * all extid2eid methods * repo._extid_cache handling * [before/after]_entity_insertion source callback * the cwxmlparser and update related tests, notably unittest_datafeed where 'repull' testing has been removed, since it's now handled by the dataimport API and should not be retested there. Related to #15538288 Closes #15538383

[ldap] Stop using entities table in ldap source authentication and parser We may used cwuri for the same purpose, and do one more step towards deletion of entities.extid column. Related to #15538288

[test] Use stdlib unittest in unittest_webconfig.py

[tox] Require ldap3 < 2 for server tests Apparently we are using some constants that got dropped/moved in ldap3 >= 2.0.0 (which came out last week). So stay with version 1.4 until this gets sorted out.

[setup] fix CWDevelop to keep compat with python 2.7 in Python, 2.7, setuptools.command.develop.develop is an old style class, thus it cannot be called with super.

[devtools] Make dependency on backports.tempfile (Python2) optional The package may not be available in all systems (e.g. no Debian package exist at the moment), and we should not crash with ImportError when importing testlib from client code. Follow up on a6dc650bc230 where the dependency was introduced.

[sources] Drop source support_entity / support_relation source API this was used in the pre-datafeed area, it's not worth it anymore. Its only valid usage was in authentication to detect if the source was supporting CWUser, hence we now call it systematically and catch NotImplementedError.

[dataimport] Add a new NullStore This is useful to do some import tests where you only want to test steps prior to the database insertion, as well as "dry-run" commands. Use it as a documenting base class.

[devtools] Properly reraise the exception with previous implementation, the original exception wasn't visible in python 2.

[pkg] Do not install sql scripts in <prefix>/share/cubicweb/schemas These are already site-packages along with cubicweb package.

[devtools/qunit] Retain firefox process stdout to eventually print it The bare RuntimeError that gets emitted upon timeout does not help much to diagnose the problem. Let's see if we can get more information from stdout by not dropping it.

[devtools/qunit] Turn FirefoxHelper into a context manager So that stop() method always gets called. Also ensure temporary directory created on __init__ gets deleted as well upon exit.

[test] Replace logilab-common's with_tempdir by tempfile.TemporaryDirectory logilab-common's with_tempdir decorator, which is considered harmful (see https://www.logilab.org/ticket/8267966). Rely on backports.tempfile to provide TemporaryDirectory on Python 2. Re-export it in testlib module for convenience.

[test] Use `with open` syntax in cwconfig "prefix" tests

[test] Use stdlib unittest.main() in qunit

[web] Clear "pending_others" session key after all relations got processed The RelationField appends data to "pending_others" key while processing its posted information, but this key may already have been removed by the edit controller. So remove the key from session data only when all "others" relations have been processed using a while loop. A test case would be nice, but I could not build one easily unfortunately. Closes #4354551.

[datafeed] Complete the import log even if parser could not be found It happens that if the parser could not be found, _pull_data() would just return an empty dict without taking care to close the import log which it just opened. This leads to misleading information in the user interface where CWDataImport entities kept accumulating in the "imports" tab of CWSource primary view without anything else happening. So: * log an error message when parser cannot be found * always close (write logs and set "end_timestamp" attribute) import log when leaving _pull_data(). Closes #15505460.

[twconfig] Set default value for "interface" to 0.0.0.0 This value means "all IP addresses on this host", which seems to fit with "default to everywhere" that's documented. Closes #15135610.

[session] Allow self._open to be False in __enter__ I do not understand why this assertion was there in the first place. And currently, it makes running any migration impossible. Hopefully, someone may be able to explain. See also commit b48020a80dc3 which mentions this assertion and suggests to remove it.

[pkg] Warn about "develop" command not installing legacy cubes directory Not sure we can do better than a warning, which is better than nothing anyways. Closes #15785635.

[pkg] Cleanup MANIFEST.in from non-matching entries This cleans up output of setup.py executions.

[devtools] make i18ncube customizable in a cube closes #15613724

[rql2sql] Drop deprecated __docformat__ it causes flake8 error on some of its versions.

[rql2sql] Handle comparison of eid on variables from an subquery We don't want to force using 'identity' in this case which may be easily handled by detecting variable is a `ColumnAlias` and relation is 'eid'. Closes #15393583

Take care that cnx may not be set in error view

[autoform] Ease overriding of inlined form renderer. Closes #15755515

[ldapfeed] Use new MetadataGenerator api instead of deprecated MetaGenerator

Fix typos

[devctl] make sure i18ncube always delete its tempdir closes #15613724

i18ncube should ignore node_modules directory by default related to #15613724

refactor i18n messages extraction This refactoring will ease later implementation of i18n cube customization. related to #15613724

[web/application] remove `path` argument from CubicwebPublisher methods Path can actually be accessed from `req` object. This allows to avoid duplicating this information. This cset prepares the next ones that aim at adding the language as a prefix of the relative path.

Merge 3.23 branch

[pkg] Fix recursive-include missing a "*" in MANIFEST.in

Added tag 3.23.2, centos/3.23.2-1, debian/3.23.2-1 for changeset 8b287ad7b95b

Release 3.23.2

[test] Add test method required by d41832121b29

[ldapfeed] Fix crash when user's password has to be generated Binary isn't hashable hence can't be added to a set. Test added.

[test/ldapfeed] Drop unused constant

[migration] Test and fix case of addition of an attribute with some serialized constraint It was crashing on attempt to remove an unexisting constraint. This code may be removed safely as it's expected to be handled by the constraint removal hook. Notice that adding a vocabulary on shortpara makes yams change the varchar max size. This is arguable but not the point of this test, so simply adapt the max size defined in the schema coherently so we don't have to bother with that at this point. Grafted from f87da59faea1 in default branch.

[skeleton] Properly format debian/changelog using current datetime and author information Reformat "context" dict to have it PEP8 compliant along the way (only white space changes).

[utils] Encode Binary value in base64 for JSON export In unittest_utils.py, use unittest2's TestCase instead of lgc.testlib one to benefit from subTest API. Closes #15409885.

[test] Make cubicweb/test/unittest_utils.py flake8-clean

[skeleton] Set Debian distribution to "UNRELEASED" in changelog So to as to make it clear that changelog needs to be edited before the Debian package may be published.

Fix log level parameter not taken into account in cwsource log table An error in the jquery selector of the html widget controlling the log level was the cause of this bug. Closes #15772634

[config] Ignore cubicweb-pyramid in available_cubes() So that, in particular, its ccplugin.py does not get loaded (which would hide the "pyramid" command in cubicweb). Related to #14023058.

[config] Exclude "pyramid" cube when initializing application cubes Related to #14023058.

[migration] Drop cubicweb-pyramid CWProperty in 3.24.0 migration This does a partial "drop_cube('pyramid')" equivalent but only considering CWProperty. We cannot just drop the cube since we have introduced some mechanism to exclude pyramid cube from config's cubes list and that would make MigrationHelper.cmd_drop_cube() call not return "pyramid" cube. Related to #14023058.

[dataimport/test] Use minimal schema for massive import tests Drop unused stuff.

[massive store] Drop deprecated code This is not a desired part of the store API. Mapping between extid and eid should be done by the importer. This had been backported from the dataio cube, but not one seems to rely on it yet, so don't provide any backward compat.

[flake8] cubicweb/dataimport/test/test_massive_store.py is now flake8 friendly

[massive store] Drop buggy log call PGHelper has no logger, the store has.

[massive store] Drop duplicated log

[flake8] cubicweb/server/rqlannotation.py is now flake8 friendly

[pyramid] Override cubicweb.hooks.syncsession.get_user_sessions() for Pyramid Closes #13436818

[pyramid] Retrieve user's groups using a custom RQL instead of user's groups property With cubicweb 3.24, user.groups is set lazily and so retrieving it at this point will cause a traceback because the connection is not yet entered. Also, user doesn't accept anymore groups and properties arguments.

[hooks] Change get_user_sessions prototype, necessary for hijacking from other session implementations (e.g. pyramid)

Store user groups and properties as session data * stop retrieving them systematically, only when need, * reimplement session synchronization hooks with some cleanups along the way, * cleanup call to set language: not needed from the base request nor from the server side, only for the web request (on the server side, language is necessary only for notification and such code should set it explicitly). There is still a XXX remaining about one can only "enter" a connection once and this is a problem in some cases. IMO, this restriction could be removed. Closes #13500113.

[skeleton] Remove override_dh_python2 We do not ship files as "data files" anymore. Related to #13001466.

[skeleton] Have setuptools install data files Just add `include_package_data=True` as recommended by `setuptools documentation`_ as this appears to be the simplest way to handle this. Related to #13001466. .. _`setuptools documentation`: \ http://setuptools.readthedocs.io/en/latest/setuptools.html#including-data-files

[skeleton] Update MANIFEST.in following new "package" layout Related to #13001466.

[skeleton] Add python-setuptools to Debian Build-Depends Related to #13001466.

[skeleton] Add dh-python to Debian Build-Depends Related to #13001466.

[skeleton] Copy files mode bits (debian/rules in particular)

[skeleton] Do not exclude setup.py from flake8 check in tox.ini The file is flake8 compliant, actually.

Merge public heads

[doc] Document configuration override by environment variables Related to #13889793.

[doc] Update book sections about cubes being Python packages Related to #1300146006.

[doc] Start filling changes/3.24.rst with recent changes

[doc] Drop commented mention of "live-server" command Follow-up for d83676aaea21.

[pyramid] Drop guard of old cubicweb version

[pkg] Declare extras (optional) dependencies This is primary to extract "pyramid" dependencies out of install_requires as they are actually optional. Along the way, resurrect all things in __pkginfo__.__recommends__ which were not used at all and convert them into extra_requires. It appears that __depends__ and __recommends__ in __pkginfo__.py are not useful so drop them and inline dependencies as install_requires/extra_requires in setup function call. For pyramid tests to continue working, add respective dependencies to test-misc.txt requirements file.

[tox] Exclude all files in symlinked directories from check-manifest call

[tox] Get rid of "touch test-results.xml" for flake8 and check-manifest environments CI server does not rely on this anymore.

[debian] cubicweb-pyramid conflicts and replaces pyramid-cubicweb Related to #14023058.

Merge cubicweb-pyramid cube Only keep the CWSession schema definition and the ctl command, now in cubicweb/pyramid/pyramidctl.py Related to #14023058.

[ccplugin] flake8

[ccplugin] docstring clarification

[doc] improve a bit the README to explain how to allow login in dev mode

Added tag 0.4.0, debian/0.4.0-1, centos/0.4.0-1 for changeset c0c98e5ba55e

[pkg] typo

[pkg] 0.4.0

[doc] add a bit more infos in the README

[cfg] remove the pyramid-{auth,session}-secret config options (closes #11027189) these should now be set in the pyramid.ini config file.

[cc] add a dbglevel cmdline option to specify the DBG_XXX flags to set also display the list of valid loglevel values in the help message.

Added tag centos/0.3.2-1 for changeset c9daf54738fd

[pkg] add spec file

Added tag 0.3.2, debian/0.3.2-1 for changeset ab4c5509407f

[pkg] fixup debian control metadata add Homepage field, replace deprecated XS-Python-Version with X-Python-Version.

[pkg] add debian watch file

[pkg] don't remove egg_info from debian package

[pkg] 0.3.2

[ccplugin] get the cw repo from the wsgi app Otherwise with cubicweb 3.22 we end up re-creating a new repo from scratch, which ends up in tears.

[ccplugin] print_function

Move tag debian/0.3.1-1 on changeset ab68206f765b

[debian] typo in debian/changelog

Added tag 0.3.1, debian/0.3.1-1 for changeset 9d82848e82a7

[pkg] 0.3.1

[ccplugin] fix a bug introduced in e95725d7ce90 (closes #5731783)

Added tag 0.3.0, debian/0.3.0-1 for changeset 7fbeb77fe690

[pkg] 0.3.0

[debian] add missing dependency on pyramid-cubicweb and python-wsgicors

[pkg] depends on waitress (closes #5576180)

[pkg] update to dh9 and dh_python2

Allow to override config file options by cmdline arguments (closes #5724484) For the sake of consistency with cw, the 'profile' option's short name has been removed, so we can use the lower 'p' for this purpose (for consistency with cubicweb). Requires CubicWeb 3.21 to work.

Added tag cubicweb-pyramid-version-0.2.0, cubicweb-pyramid-debian-version-0.2.0-1 for changeset 4c16305d5825

set debian version 0.2.0-1

Set version 0.2.0 And requires pyramid-cubicweb 0.2.0

Add profiling options

Monitor more configuration files Closes #4811352

Set version 0.1.1

--debug now activates all debug options Added a --debug-mode so the debug toolbar can be activated without switching reload on. Closes #4783342

Move the cors middleware initialisation to pyramid-cubicweb to reduce code duplication

Added tag cubicweb-pyramid-version-0.1.0, cubicweb-pyramid-debian-version-0.1.0-1 for changeset 3defbb0f147a

Prepare release

Add a --no-daemon option

Use 'wsgicors' for CORS handling. The CW CORS handling (in web/cors.py) is only able to work on cubicweb requests. When a request is not handled by bwcompat, we need a proper solution. The `wsgicors` library provides what we need as a wsgi middleware.

Watch for i18n files changes for auto-reload

Auto-reload now survives failed reload When the server stops within the reloader and let a list of files that were monitored, the reloader waits for any of these files to change and re-attempt to start the server. This work well when a syntax error is saved to a file: there is no need to restart manually the server anymore.

Implements auto-reload and daemon mode. Heavily inspired by pyramid pserve, with pieces of code taken from it. auto-reload Start the server in a subprocess that auto-stops when a file is modified, and exit with a specific code. daemon mode Uses some code from pserve, but use the cw configuratione so the command is compatible with 'status' and 'stop' commands.

Add the 'pyramid-auth-secret' option to all-in-one.conf

Add the 'pyramid-session-secret' option to all-in-one.conf This value is used by pyramid_cubicweb to initialize the cookie factory.

Add a CWSession entity. It is a subset of the CWSession that will be added to cubicweb in a future version. Closes #4317363

Implements the 'pyramid' command. It load an instance config, and load in the pyramid scaffolding provided by pyramid_cubicweb. The application is then served by a waitress server. Closes #4317312

Project structure

Merge with pyramid-cubicweb The following tasks have been done: - merge packaging files - merge documentation - move pyramid_cubicweb package at cubicweb/pyramid and update imports accordingly - rename tests directory into test - move pyramid-cubicweb README.rst into README.pyramid.rst until better idea - add a test dependency on unreleased cubicweb-pyramid to have both py27 and py34 tests pass Closes #14023058.

Port to Python3 (closes #14159555) Add py34 environments to tox configuration (only for CubicWeb >= 3.23). And depend on hg version of cubicweb-pyramid since it is not currently Python3-compatible.

Use opened connections as much as possible instead of opening a new internal connection to set the session cookie or retrieve the session data, which may exhaust the connection pools.

Test and fix status code and content of the login form on authentication error The status code was 200 and is now fixed to 403, but the content part of the test will fail until cset 02328f8cbd5c is integrated in cubicweb.

Added tag 0.7.0, debian/0.7.0-1, centos/0.7.0-1 for changeset 0cf2972d2011

[pkg] Version 0.7.0

retrieve session data in a single query instead of two (one for the session object, the other for its cwsessiondata attribute).

[tox] Update CubicWeb version targets * In -release, no need to specify cubicweb dependency, its pulled by setup.py. * Use 3.22 as -compat. * Point to "default" branch for -dev. Also remove test directory from posargs.

wsgi: clearer exception when CW_INSTANCE is missing instance_name is mandatory for cwcfg.config_for() Raise with a KeyError: 'CW_INSTANCE' instead of hard to read exception.

typo

[doc] update a bit the documentation Also convert the README in rst (so it can be properly displayed on cubicweb.org).

split collecting setting before using them, so the function can be reused when inserting WSGI middlewares

[bwcompat] log execption even when cubicweb.bwcompat.errorhandler = True (closes #13421901)

[pkg] add spec file

Added tag 0.6.1, debian/0.6.1-1 for changeset 2621daafa10c

[pkg] 0.6.1

[bwcompat] send 403 on authentication errors (closes #12219849) 200 is just wrong.

[bwcompat] set response headers on AuthenticationError (closes #12219860) Turns out sending a response without a Content-Type header (among others) is suboptimal. 5b36399b6b21 was not good enough.

Added tag 0.6.0, debian/0.6.0-1 for changeset 94d1a024b3bf

[pkg] 0.6.0

[tests] add a __main__ handler the relative import in test_rest_api.py needs to be modified to prevent a ValueError: Attempted relative import in non-package

[pkg] add a requirements-test.txt file for test dependencies on WebTest and cubicweb-pyramid

[login] fix the redirect url after login (closes #11689118) redirecting to '/' by default after login doesn't work properly when a prefix is used, whether we're sitting behind a PrefixMiddleware or not. To fix this, rely on cubicweb's build_url to turn any relative path into an absolute url.

keep track of all traceback in error handling, not just the exception message (closes #11689093)

[bwcompat] also set response headers in error cases I had this sitting around in my local copy, I don't remember why I needed this but it seems correct.

[config] move config of the secret used to encrypt session's data ID in pyramid.ini (closes #11689082) Introduce a new config entry (in pyramid.ini) for this (cubicweb.session.secret) to replace the (now deprecated) pyramid-session-secret (in all-in-one.conf). So we have now 3 secrets to configure: - cubicweb.session.secret: to encrypt session's data ID stored in a cookie, - cubicweb.auth.authtkt.session.secret: to encrypt auth cookie - cubicweb.auth.authtkt.persistent.secret: to encrypt persistent session auth cookie

Document how to configure the "secure" flag for authentication policies Closes #11376233.

[tox] Let posargs override py.test args

[tox] Use py.test Rename test view so that py.test does not consider it as a test function.

Add a tox configuration

Register predicates from the predicates module

[refactoring] Move MatchIsETypePredicate to a separate module

[refactoring] Move EntityResource and ETypeResource to a separate module

use CubicWeb request to execute RQL rset should be retrieved with cw_request, as it's then bound to it and propagate to all entities created from this rset (._cw). From there it may reach code expecting a request, not a connection (view, selector, etc).

[views] Rename entities to rest_api

[routes] Add a 'cwentities' route with traversal The route uses a factory that produces ETypeResource and EntityResource as a context. A 'delete' view serve as a test and demonstration. The module being experimental, it has to be explicitely included.

Added tag 0.5.0, debian/0.5.0-1 for changeset 61f69ac2d6bc

[pkg] 0.5.0

[auth] Authtkt http_only and secure by default The test suite is now full 'https'. Closes #4731765

[auth] Make the configuration cookies completely configurable Also transfert the secret setting from all-in-one.conf to pyramid.ini, with backward compatibility. Closes #5999625

Added tag 0.4.1, debian/0.4.1-1 for changeset 398b2c840e14

[pkg] 0.4.1

[core] Adjust cw<3.21 compatibility Most of cubibweb < 3.21 expects a ClientConnection as the main connection. Related to #5731814 Closes #5878592

[bwcompat] Make the error handler optional Closes #5739625

[doc] Document embedding in a pyramid app

[config] Move most config code to a includeme() The goal is to make it easier to use pyramid_cubicweb from a pyramid application.

[doc] Anonymous access is not mandatory anymore

Added tag 0.4.0, debian/0.4.0-1 for changeset 897a149e8208

[pkg] add debian/watch

[pkg] 0.4.0

[pkg] add python-pyramid-multiauth dependency to debian package Closes #5576182

[core] adjust cnx handling for cubicweb 3.21 Closes #5731814

set httponly on session cookie

[bwcompat] use cubicweb error views (closes #4545130)

Added tag 0.3.1, debian/0.3.1-1 for changeset 6df91cb85ecc

[pkg] 0.3.1

Handle absence of anonymous user Set cw_session and then cw_cnx request attributes to None in case anonymous connection is not allowed (i.e. no "anon" user in config). Then catch AuthenticationError in CubicWebPyramidHandler and return the 'login' view. Closes #4751862.

[debian] Add python-wsgicors dependency as it is now available

Update Changes for version 0.3.0

Added tag pyramid-cubicweb-version-0.3.0, pyramid-cubicweb-debian-version-0.3.0-1 for changeset a80e076d3f42

Fix debugtoolbar pkg name

Set version 0.3.0

[doc] update changes list

Rollback 'uncommitable' cnx Closes #5343870

[debug] The debug mode now set pyramid.reload_templates

Use pyramid flash queue for messages Use a 'cubicweb' flash queue and make sure it contains only one message so that the behavior is the same as cubicweb. Also, the 'message' property now returns both the cubicweb flash queue and the default flash queue. One big difference with the former behaviour is that messages set with set_message will survive a redirection, making set_redirect_message useless in most case. Closes #5298654

[doc] Document the new authentication stack

Allow tests to override pyramid_settings Closes #5307426

Make debug mode usable without pyramid_debugtoolbar Add the latter in Debian recommends along the way. Closes #5310434.

[auth] Use a second authtkt policy for 'rememberme' The former solution was buggy because the expire time of the auth cookie, if set through 'remember', was lost on the first cookie reissuing. The new approach, make possible thanks to multiauth, use two different cookies. One for session bounded authentication (no 'rememberme'), and one for long lasting authentication (w 'rememberme'). The choice between the two of them is done by adding a 'persistent' argument to the top-level 'security.remember' call. Passing this argument will inhibate a policy or the other. The two policies are (a little) configurable through the 'cubicweb.auth.authtkt.[session|persistent].*' variables. Related to #4985962

[auth] Use pyramid_multiauth It makes it easier to finely tune what parts of the default authentication stack we want to use or not. It also makes it possible for any cube to add its own policy in addition to the others. Related to #4985962

[auth] remove dead code (closes #5230746)

[login] Test the login views

Fix project homepage url

Replace the '_' with '-' in the package name The change was made manually on pypi (see https://sourceforge.net/p/pypi/support-requests/459/)

On exceptions from CW, copy headers Closes #4939219

[doc] fix pyramid-auth-secret conf sample

[doc] Update change list

[core] Protect session data from unwanted loading. Use specialised Session and Connection types that forward their 'data' and 'session_data' attributes to the pyramid request.session attribute. This forwarding is done with properties, instead of copying a reference, which allow to access request.session (and the session factory) if and only if Session.data or Connection.session_data is accessed. In some cases, most notably the static resources requests, it can mean no access the session during the request handling, which saves a request to the session persistence layer. Closes #4891437

[core] Use tools.cached_user_build for better performances Closes #4870347

[doc] Document tools Related to #4870347

[tools] Provide a faster build_user The main trick is to use a cache of user entities. To do so, a few tools are needed since the entities are not supposed to be copied around between connexions. Related to #4870347

Added tag pyramid_cubicweb-version-0.2.1, pyramid_cubicweb-debian-version-0.2.1-1 for changeset 1ae61c25299a

Prepare version 0.2.1

[cors] Fix 'headers' and 'methods' parameters Closes #4849874

Fix the 0.2.0 release date

Change project url

Added tag pyramid_cubicweb-version-0.2.0, pyramid_cubicweb-debian-version-0.2.0-1 for changeset cd8308245d20

set debian version

Prepare version 0.2.0

Document the changes

Fix configuration loading when 'cubicweb.includes' is not set Closes #4849314

pep8

Provides requirements for rtd Related to #4849313

Initial documentation. Closes #4849313

Remove dead code

[profile] Add a profiling tool

[config] Read pyramid settings in a 'pyramid.ini' file If a 'pyramid-debug.ini' file is present, it will be used instead when debugmode is on. Closes #4811298

Move auth-related configuration to a dedicated module.

Fix cors 'origin' parameter passing Closes #4783343

[auth] Fix the config option name in the warning message

session -> cnx

Added tag pyramid_cubicweb-debian-version-0.1.3-1, pyramid_cubicweb-version-0.1.3 for changeset 0a7769e583c2

Set version 0.1.3

Cookie 'max_age' must be a integer, not a string. If not, the value is used verbatim for the 'expires' of the cookie, which is invalid. Closes #4731764.

Added tag pyramid_cubicweb-version-0.1.2, pyramid_cubicweb-debian-version-0.1.2-1 for changeset 5eb4e27e9998

Update debian changelog

Don't rollback if exception is HTTPSuccessful or HTTPRedirection In the request finishing, the 'cleanup' callback set by _cw_cnx automatically commit the transaction except is an exception is set on the request. Problem is, redirections and successul http return code can raise exceptions. This patch detects such exceptions and avoid rolling back the transaction. Closes #4566482

Set version to 0.1.2

Provides a full wsgi cubicweb application builder

Added tag pyramid_cubicweb-version-0.1.1, pyramid_cubicweb-debian-0.1.1-1 for changeset 9f3b9e610c3d

Set version 0.1.1

[auth] Make last_login_time updated. The update is done when the user logs in, then every time the authentication is reissued. Closes #4549891

Added tag pyramid_cubicweb-version-0.1.0, pyramid_cubicweb-debian-version-0.1.0-1 for changeset 3d60138100d6

Add a description to the debian package

Set version to 0.1.0

Initial debian packaging missing at least a long description and wsgicors dependency.

Move PyramidCWTest to pyramid_cubicweb.tests

Optimise repo_connect by skipping authenticate_user The authentication being handled by pyramid itself, going through the authentication stack to recreate the temporary session at each request is very costly. On my desktop, for a mostly static front page, the total time for delivering the page drops from 100ms to 47ms.

Use lightweight sessions Provides a lightweight version of repo.connect() that does not keep track of opened sessions. The speed gain on a mostly static front page is about 5% Warning ! This means that, for now, the "session_open" and "session_close" hooks are NOT called anymore.

Use 'wsgicors' for CORS handling. The CW CORS handling (in web/cors.py) is only able to work on cubicweb requests. When a request is not handled by bwcompat, we need a proper solution. The `wsgicors` library provides what we need as a wsgi middleware.

Add requirements

Handle '__setauthcookie' '__setauthcookie' is a form parameter added by the 'rememberme' cube. If present and equals to '1', the cookie max_age will be set to 7 days instead of being a session cookie. To make sure the auth cookie is renewed, the reissue_time is set to 1h.

Fix POST handling. The issues where revealed by the unittests, which are ported from the cubicweb wsgi tests.

Use AuthTktAuthenticationPolicy

DB-saved session data Related to #4291173

Handle properly the '/https/*' urls CW uses a url prefix to detect https behing a reverse-proxy. A more proper way to do that is documented here in the waitress documentation (waitress is the default pyramid wsgi server): https://waitress.readthedocs.org/en/latest/#using-behind-a-reverse-proxy A later version should implement this, or use waitress in the 'pyramid' command. Related to #4291181

Correctly pass the multiple parameters to the cubicweb request When naively converted to a dict, the webob MultiDict will not set the multiple parameters (a same name with several values) in a way CW can understand. MultiDict.mixed() however, does exactly what CW needs: list values for keys with multiple values, and single value for the others. Related to #4291181

Add a wsgi application factory suitable for wsgi servers. This factory can generate a wsgi application for a cubicweb instance. It reads the instance name from the CW_INSTANCE environment variable, and activates the debugmode if CW_DEBUG is defined in environment. It is usable by uwsgi as the 'module' parameter : CW_INSTANCE=test uwsgi --plugins python,http --http 0.0.0.0:8080 --module pyramid_cubicweb:wsgi_application()

If any cube has a 'includeme' attribute, call config.include on it Related to #4291181

Add a make_cubicweb_application function This function will be used by the 'pyramid' cubicweb-ctl command. Related to #4291173

Fix session closing for cubicweb 3.19 Related to #4291173

Convert cubicweb.NotFound to HTTPNotFound Related to #4291173

Use registry['cubicweb.registry'] instead of registry['cubicweb.appli'].vreg because the application may not be present. Related to #4291173

Use a predicate based view selection for handling /login This will allow easy addition of login handlers from the application or cubes Related to #4291173

Use a tween application instead of a catchall route. Using a catchall route has some drawbacks. Especially, we have no mean to have a route that would match only if no other one does AND no view matches either. Said differently, our default handler cannot be plugged on the route level nor the view level, because it is has to be activated only if nothing else works in the pyramid application. Using a tween application allow to handle requests that raises a HTTPNotFound error, while having the pyramid error handler still active between our tween app and the outside world. Related to #4291173

Document the view problem hypothesis. Related to #4291173

If the postlogin_path is 'login', redirect to '/' instead Related to #4291173

Put the login view in a separate module. Related to #4291173

Separate into 4 modules * init_instance: load the cubicweb repository from the `pyramid_cubicweb.instance` configuration key * defaults: provides cw-like defaults for the authentication and session management * core: make cubicweb use the authentication and session management of pyramid. It assumes the application provides the auth policies and session factory, and that the `cubicweb.*` registry entries are correctly initialised. This is this only required module or pyramid_cubicweb, the other ones are optional if the application provides its own versions of what they do. * bwcompat: provides a catchall route that delegate the request handling to an old-fashion cubicweb publisher (ie using url_resolver and controllers). Related to #4291173

Update the TODO list Related to #4291173

Provide instructions and a requirements list to quickly start the sample application. Related to #4291173

Use the pyramid session object as the cubiweb session.data (needs a patched cw 3.19) Related to #4291173

Add comments on parts we want to reconsider later Related to #4291173

Skip core_handle, add a context manager to handle cubicweb errors The context manager is also used to catch errors in render_view. It handles the 'external' errors raised by cubicweb code. The more internal errors, the one that should occur only in url resolving and cubicweb controllers, are handled directly in CubicWebPyramidHandler. ValidationError is handled by CubicWebPyramidHandler for now, but should probably be handled by cw_to_pyramid Related to #4291173

Documents dependencies on a ubuntu system Related to #4291173

Use short-lived cubicweb sessions to let pyramid actually handle the web sessions Related to #4291173

Isolate the default handler and extend its role The handler now does the job of CubicWebPublisher.main_handle_request() and calls CubicWebPublisher.core_handle(). Instead of using config.add_notfound_view, a catchall route is defined and the handler plugged to it. Related to #4291173

Add a basic sample application Related to #4291173

Integration pyramid and cubicweb authentication. We use pyramid sessions to store the cubicweb sessionid so we can reuse it when needed, or regenerate it if it was lost. The cubicweb sessionid is obtained from a login in the repo OR directly from the user identified by pyramid. Related to #4291173

[doc] Escape _ in README.md So that the Markdown is okay.

Initial implementation Set up a default route that passes requests to a cubicweb instance. The requests are wrapped in an adequate adapter so that cubicweb works with no change. Related to #4291173

Project Structure Related to #4291173

[tox] Have less test environments This is to reduce the load on CI server because of parallel clones of the repository in Docker containers. Only keep "server", and "web" environments separated. All other prior environments are in "misc". To avoid duplicate entries in requirements files, move them all in a "requirements" directory (this appears to be a "common" practice, see e.g. Celery). Adjustments in tests: * Rename cubicweb/hooks/test/unittest_notification.py so that it does not conflict with cubicweb/sobjects/test/unittest_notification.py during test discovery as they would have the same module name but different __file__ attribute. * Add "comment" cube to the list of expected cubes in unittest_cwconfig.py as this cube is pulled by requirements/test-mist.txt. Closes #15440662.

[repo] Don't crash on start when fs schema is missing some db schema entities This occurs usually while developping and we don't want systematically to rebuild the database to start the instance or run e.g. i18ninstance.

Let configuration option be overridden by environment variables Related to #13889793.

[tox] Add a dummy test-results.xml file in check-manifest and flake8 environments This is to work around Jenkins Junit plugin that does not apparently account for the absence of this file (though it accepts it to be empty).

[tox] Basic flake8 config

Use pkgutil.ImpLoader instead of a custom loader for cubes import redirection It just works fine (same implementation of load_module method as _CubesLoader introduced in d404fd8499dd) and is complete w.r.t. PEP 302 (all methods implemented). Related to #13001466.

[config] Make available_cubes aware of cubes installed as packages For this add a "cubicweb.cubes" entry points group which will be scanned through to find out installed cubes. Entries in this group are expected to expose the module name of a cube (i.e. `cubicweb_foo`). Note that CubicWebConfiguration's available_cubes method will return the module name of cubes as packages (cubicweb_foo), so we had to add a special "key" sorting function to keep cubes sorted as before, despite possible different distribution schemes. This makes it possible to handle loading of CTL plugins in an almost similar manner as before (just tweaking the package name from cube name in load_cwctl_plugins method). I had to tweak (again?) the test_cubes_path method in unittest_cwconfig.py but did not find out why. Apart from unforeseen bugs and pending documentation, this finishes the work on porting cubes to standard Python packages. Closes #13001466.

[skeleton] Use distname directly in setup.py distname is part of required metadata and modname cannot actually be used in place of it.

[skeleton] Fix modname in __pkginfo__

[test] Use plain unittest in unittest_cwctl.py

[devctl] Remove commented LiveServerCommand The code got commented in 2010 (changeset 5ab3f63b06ad)...

[pkg] Inline call to setup() in setup.py The point of having this "install" function, called in "main" mode is not clear. Better stick to standard practices.

[pkg] Drop old checks in setup.py

[tox] Exclude symlinks from check-manifest call These are not handled by check-manifest, see https://github.com/mgedmin/check-manifest/issues/69.

Drop the only reference to an '__insert' form key

Drop bits from the past (no more dbapi isolated installation)

[devtools] CWTC.request is gone, don't attempt to use it in CWTC.view So one will have to provide at least one of `rset` or `req` argument of the `view` method.

Merge 3.22 into 3.23 Some fixes there are necessary for client work depending on cubicweb 3.23.

[devtools] Handle i18ncube command for "cubes as packages" And add a test so that both layouts are tested. Move the test cube into a package. Related to #13001466.

Handle cubes as packages in cwconfig Rely on importlib.import_module in several places in cwconfig.py, so this would not work with python 2.6. Method available_cubes will not list cubes installed as package for now. I'm not sure it's worth implementing this method (used in `cubicweb-ctl list cubes` command) for new cubes layout as the same result can basically be obtained with `pip freeze | grep cubicweb_`. In unittest_cwconfig.py, duplicate CubicWebConfigurationTC test case to test both the "cubes as packages" layout and the "legacy cubes" layout. The former having a custom sys.path set (pointing to datapath('libpython') where all cubes' packages live) and the latter having the previous config attribute setup. All test data cubes are moved to packages in libpython directory and symlinks are introduced in the cubes directory. Related to #13001466.

[cwconfig] Reorder imports by alphabetic order

Add an import redirect hook from "cubes.<name>" to "cubicweb_<name>" The hook consists of a finder and a loader implemented following PEP-302; it is responsible for loading cubes distributed as packages (i.e. installed as ``cubicweb_<name>`` in site-packages) but imported (in client code) as ``from cubes.<name> import ...``. So this is a transitional mechanism allowing cubes following the new layout to be used by old-style cubes/applications. The importer is installed upon calling CubicWebConfiguration's cls_adjust_sys_path method (also called in cubicweb.devtools.__init__.py, which is a prerequisite for importing any "legacy" cube. The loading of old-style cubes is still handled by the CubicWeb configuration, based on adjustment of sys.path etc. Related to #13001466.

[devtools] Update skeleton's setup.py to install cube as a package Most of the prior logic of skeleton's setup.py gets dropped as installing a cube as a "classic package" (i.e. in site-packages) is just the default behavior of distutils. Also add a test checking installation of new cube. Related to #13001466.

[devtools] Adjust skeleton setup.py to have it read information from __pkginfo__.py Since the later is now installed in cubicweb_<CUBENAME> directory. Add a test ensuring that `python setup.py sdist` works on newly created cube (which should at least verify that setup.py is valid). Related to #13001466.

[devtools] Update skeleton and newcube command to cube as package layout Related to #13001466.

[pkg] Add missing entries to MANIFEST.in Detected by running `check-manifest <https://github.com/mgedmin/check-manifest>`_. Add a tox environment accordingly. Closes #15229018.

[test] Add a test for toolsutils.read_config

[test] Use plain unittest in unittest_toolsutils

[skeleton] Remove test/pytestconf.py

[pkg] Make it clearer which entries of __pkginfo__ are actually used

[test] Avoid usage of lgc.testlib TestCase in unittest_cwconfig.py

[test] Move setup of spa2rql tests into setUpClass method Thus avoiding cryptic errors upon unittest discovery.

[schema] Set CubicWebSchemaLoader's extrapath using config eponymous property

[doc] Document change of instances location in virtualenv installation Closes #14789440.

[cwconfig] Do not override user-defined CW_MODE in virtualenv setup And thus consider `_forced_mode` first. Related to #14789440.

Added tag 3.22.4, debian/3.22.4-1, centos/3.22.4-1 for changeset 92db0bb8e26d

[pkg] remove .dev0 from version

[pkg] 3.22.4

[session] Ensure access to rql varmaker always mark the session as dirty When one accesses the rql_varmaker, that's usually to use it. The pb was that when the varmaker was already in page's data (which is stored as session data), session storage such as redis won't see that the session data is dirty and has to be stored back at the end of the request. To fix this, systematically call set_page_data. (grafted from 3432f0e2540d)

[test] fix test_printable_value_bytes with current pygments An empty span was added in https://bitbucket.org/birkenfeld/pygments-main/commits/164574c13533195a555181a2b9c685fea2470403 (grafted from 74b04a88d28a)

[webctl] set uid of file produced by gen-static-datadir (closes #11298794)

[webconfig] ensure uicache content has correct uid (related to #11298794)

[cwconfig] add the CubicWebConfiguration.ensure_uid_directory() method (related to #11298794) that recursively enforce uid on files.

[cwconfig] extract uid file setting code from check_writeable_uid_directory (related to #11298794) into an ensure_uid() method

[web] Fix bug with usage of os.rename under windows environment In 7c386161ebd6 we removed cache handling from property sheet and introduced usage of a tempfile + os.rename to get atomic generation of files. The pb is that this is not portable, since under windows os.rename will raise an exception if the file already exists (because there is no way to write a file atomatically in such case). This kind of thing should be out of the CW scope anyway, so implements a quick & dirty fix in the mean time. Closes #14214794

[migration] More resilient migration script Query the information schema to retrieve existing constraint (primary key and unique) and drop them. Thus don't use anymore DROP CONSTRAINT IF EXISTS (not implemented by e.g. old versions of sqlserver). To do so a missing DISTINCT was necessary to avoid duplicates.

[session] Ensure access to rql varmaker always mark the session as dirty When one accesses the rql_varmaker, that's usually to use it. The pb was that when the varmaker was already in page's data (which is stored as session data), session storage such as redis won't see that the session data is dirty and has to be stored back at the end of the request. To fix this, systematically call set_page_data.

typo

[native] Use cnx.system_sql instead of doexec method in syntax_tree_search The latter handles connection issues and tries to reconnect. Method syntax_tree_search appears to be called from quite a few place where it seems useful to be resilient to connection issues (authentication for instance). This reconnection logic appears to have disappeared at some point when working on https://www.cubicweb.org/2919309. Could not find the exact reason though...

Remove usage of cnx.ensure_cnx_set context manager It has been deprecated since 3.21, no reason to keep using it internally.

[hooks/test/synschema] remove an unneeded skipTest. Related to #5557633.

[test] Check retcode of subprocesses in devctl tests And display decoded stdout in case of failure.

merge 3.23

[pkg] Bump version to 3.24.0.dev0

[pkg] Drop NO_SETUPTOOLS option setuptools is the defacto standard nowadays. The only place where NO_SETUPTOOLS option was used is during packaging (RPM, Debian) and even in this context, it seems that setuptools is the way to go (see https://wiki.debian.org/Python/LibraryStyleGuide).

[debian] update cubicweb-dev.lintian-overrides cubicweb/skeleton/debian/rules.tmpl is now cubicweb/skeleton/debian/rules prevent lintian from complaining

[pkg] Properly export data files in setup.py and adjust "newcube" test With the new package layout (everything under "cubicweb" package), the custom install_lib rule which makes use of include_dirs defined in __pkginfo__.py did not prepend the package name to source directories to be copied. Fixing this. Also, in setup.py's export() function, the destination directories' path to be created during source tree walk was wrong. All this makes cubicweb/skeleton directory (which is not a package) properly installed by setup.py. The test in cubicweb/devtools/test/unittest_devctl.py wasn't properly implemented because it used an installation of cubicweb in "develop" mode which shadows such packaging issues. Also it used "python -m cubicweb" instead of directly "cubicweb-ctl" and the former appears to fall back to using the cubicweb package *from sources* instead of the installed one. Now that this test runs against the installed version of cubicweb, fix MANIFEST.in to include tox.ini files (cubicweb's and skeleton's) as this is expected from the test. Closes #14127941.

[pkg] Handle verbose option in setup.py's export function Make debugging easier.

[pkg] Drop unused EMPTY_FILE global variable

[tox] Do not install package in "develop" This might shadow so packaging issues. And user can still pass --develop to tox.

[pkg] Bump version to 3.23.2

pep8 bits

[schema] Add a method on yams constraints to compute its unique name used to identify it in the DB backend.

[migration] Test and fix case of addition of an attribute with some serialized constraint It was crashing on attempt to remove an unexisting constraint. This code may be removed safely as it's expected to be handled by the constraint removal hook. Notice that adding a vocabulary on shortpara makes yams change the varchar max size. This is arguable but not the point of this test, so simply adapt the max size defined in the schema coherently so we don't have to bother with that at this point.

[test] Add a migration test for update of a static vocabulary constraint This is a constraint serialized to be checked by the DB backend, hence ensure the old constraint is dropped and the new one is created.

[schema2sql] Avoid "parsing" SQL statements for database initialization A big SQL string was generated, then splitted. This caused bug if some value in the schema (eg vocabulary, default) contained the separator (';'). To properly fix this, yield each individual statement instead of generating a string. Closes #14050899

[schema2sql] Drop most of the DB DROP related code This code is dead for a while, it has been superseded by `sql_drop_all_user_tables`. Related to #14050899

[hooks] Use already imported module alias

[schema2sql] Rename as_sql to constraint_value_as_sql which is easier to grasp independantly of the context.

[schema2sql] Drop unused indent argument on aschema2sql Related to #14050899

[schema2sql] More docstrings and minor API improvements Along the way, renamed local `column` variable to `attr` where we're NOT manipulating a DB column name but a schema attribute. Related to #14050899

[schema2sql] inline eschema_attrs method * not a public API * doesn't makes things easier to grasp -> useless Related to #14050899

[schema2sql] Give a rdef to check_constraint This simplifies the API. Related to #14050899

merge 3.20.16 in 3.22

merge 3.20.16 in 3.21

Added tag 3.20.16, debian/3.20.16-1, centos/3.20.16-1 for changeset e60a8e5d29ef

[debian] add a lintian override for source-is-missing about jquery.flot.js

[pkg] 3.20.16

[dataimport] fix count_lines function (closes #14407650) ensure i allways exists whatever f content is

[captcha] improve a bit the redability of generated captcha images (closes #13500378)

3.20 branch

merge 3.23 changes

Added tag 3.23.1, centos/3.23.1-1, debian/3.23.1-1 for changeset 8f035d5d5953

prepare 3.23.1

[debian] Adjust symlink to cubicweb package so that documention builds fine Broken since source tree re-organization.

[dataimport] Restore handling of Binary in ExtEntity's values Broken since 62a7100d774b. Closes #13973741.

[tox] Ensure virtualenv's python is used to build the documentation

[doc] Reference 3.23 page in changelog

[tox] Ensure virtualenv's python is used to build the documentation

Backed out changeset b288debc6736 Only superusers can set lc_messages variable. This prevents connection in many cases. Closes #14010345.

Merge with 3.23

[pkg] Bump version to 3.22.4.dev0

Added tag 3.23.0, centos/3.23.0-1, debian/3.23.0-1 for changeset 4b07f2a79021

[doc] Fix symlinks of sphinx theme images

Added tag 3.23.0, centos/3.23.0-1, debian/3.23.0-1 for changeset 93b0f836cb07

[debian] Move python-unittest2 >= 0.7.0 as runtime dependency It seems to me that this is not used during Debian package build (tests are not run), but should be needed at runtime with the cubicweb-dev package. Also, at the moment, we have no pbuilder setting with this package available and it blocks the release.

Symlink __pkginfo__.py at root Maybe this will make apycot work again...

[3.23] More release notes

i18n update

[doc/changes] CubicWebTC.set_description no longer available

Add some release notes

[pkg] Version 3.23.0

[pkg] Bump yams version in cubicweb-common Debian package

[tox] Drop yams from dev-requirements.txt Yams 0.44.0 got released.

Merge with 3.22

Merge with 3.21

backport 3.21 changes

backport 3.20 changes

[sobjects] Drop custom unique constraint checks in user registration service This constraints (on unique CWUser's login and EmailAddress's login) are handled by the backend. This follows "[hooks/integrity] kill CheckUniqueHook". Extend test for the ValidationError along the way.

[test] Add a test for user registration unicity constraint on email

[test] Clarify user registration service tests * split permissions test appart * avoid usage of internal_cnx as this does not seem need

[test/hooks] pep8 and unittest.main instead in unittest_hooks.py

[hooks/integrity] kill CheckUniqueHook It's the backend's job. The noticeable consequence (as can be seen in tests updated here) is that error message are a bit less user-friendly.

[migration] drop identity_relation if it exists It has been generated erroneously for a while, then fixed but not properly deleted from old databases. Don't use IF EXISTS for sqlserver compat.

[migration] Reserialize all constraints since yams introduce new JSON serializing format So we may drop support for older format in yams at some point.

[migration] Add IF EXISTS on DROP CONSTRAINT problem encountered on migration of a legacy database

[schema sync] Rename index when an entity type is renamed So we may still find them later using their expected name, and avoid collisions. Closes #13822045

[c-c] Add a command to check database index List indices expected by the schema but not in the database and vice-versa Closes #13822027

[sql gen] Control size of index name using an md5 hash When using concatenation of table and column, we may hit the backend size limit on index names, which results on truncated name (64 for postgres). Then we may have conflict or be unable to find expected index on later synchronization attempt. This is done by monkeypatching lgdb for now to avoid dependancy problems. Unique and primary key indices of entity types handling is left to the backend, as they are not explicitly named. See https://www.logilab.org/ticket/4789189 Closes #13821949

[sql gen] Explicitly name unique index This patch hotfix logilab database for https://www.logilab.org/6662663 and fix database creation to stop using UNIQUE in the table schema but rather create the index later on, so its name is under control. Also consider this change in schema synchronization hooks. This is preliminary work related to #13822045

Update to yams 0.44 API

[test] Fix randomness in tests

[test] Drop unused import

[serverctl] allow to sync multiple and all sources in source-sync command Breaking change: by default source are not synced anymore if it is fresh (regards to synchronization-interval). To get the previous behavior an option --force is added. Closes #13886467

[tox] Split migractions tests from server environment These are the longest tests in this environment, so running them apart (in parallel of others) would hopefully speed up the whole suite.

[dataimport] Fix rdef loopup in importer

[py3k] left-over unicode() in authobjs.py

[server] Improve massive deletion of entities sqlserver is limited on the "IN" array size and produce an error "The query processor ran out of internal resources and could not produce a query plan". The exact limitation is unclear but can occur with > 10000 item in the array. See https://support.microsoft.com/en-us/kb/288095 Now we process the deletion by batch of 10000, some quick benchmarks show that there is no performance impact: delete_in(50000): 46.765182972 delete_in_batch(50000): 46.3242430687 delete_in(100000): 62.2969429493 delete_in_batch(100000): 61.8402349949

[hooks] Add a bunch of tests for session synchronization And change unicode into six.text_type in authobjs.py spotted by new tests.

[entities] Drop very old backward compat from erudi time (even the standard datetime type didn't exist at that time)

pep8 + docstrings and comments improvments

[server] some pep8 in rql2sql

[dataimport] Deprecate SQLGenObjectStore Closes #10338709.

[datafeed] attempt to acquire synchronization lock even when force is given instead of the implementation in e717da3dc164, raise an error if the lock is already grabbed and catch this error in the caller. See discussion on https://www.cubicweb.org/revision/10790765 Closes #10451635

[schema] Cache RQL expressions on schema loading Add a mapping rql string to rql statement Deserializing schema is ~36% faster with a lot (> 4000) expressions like those generated by cubicweb-container.

[schema] Avoid parsing RQL expressions twice on schema loading Kill full_rql (unnecessary) Use snippet_rql rather than rqlst when possible Don't access to rqlst cachedproperty from __init__ Deserializing schema is ~40% faster with a lot (> 4000) expressions like those generated by cubicweb-container.

[dataimport] Handle Bytes data in ExtEntity and convert them to Binary Closes #13887276.

[test] Avoid assertCountEqual in unittest_syncschema.py It's not clear why this happens, but it seems that assertCountEqual does strange things with unittest2 on Python 2 for equality comparison (trying to sort constraint objects, even though BaseConstraint raise NotImplemented in __lt__ from Yams 0.43). So just avoid using it to get this test passing.

[packaging] Bump version number

[web/session] make sure not to modify an inner value of session.data (closes #11217958) When we modify a mutable value stored in the session's data (dict-like object) we prevent an alternative session data handler (eg. pyramid-redis-session) from detecting this update and synchronizing it in its backend (like redis).

[book] improve a bit the documentation of the database setup also remove the MySQL section.

[doc] add documentation for specializing translation of other cubes in an application cube

[doc] fix some URLs (cs to test jenkins autobuild )

[uicfg] Fix typo in set_fields_order

[forms] We don't want cwjs-edition-cancel class on reledit's cancel button Else it's bound to a js function that collides with onclick provided by reledit code. Closes #13120795

[test] Prefer unittest.main

[test] py.test related fix We ought not import a concrete test case from a sibling module, else it ends up partially initialized at some point and later raise assertion error in setUpClass.

[dataimport] add a bit of extra-documentation on the massive store about prepare_insert_relation limitation on inlined relation and lack of full-text indexation.

[dataimport] no need to call .keys()

[dataimport] drop massive store's flush_metadata method This method handle a temporary table which sounds useless, and even buggy in some cases (metadata for a single entity types will be only flushed once even if flush is called several time). Instead, simply call the method-doing-the job after entities have been inserted (and make it private along the way).

[dataimport] test and fix external source support for the massive store

[dataimport] add new test for nohook store importing in a custom source

[dataimport] introduce usage of MetadataGenerator into the massive store This allows more flexibility and code sharing w/ nohook store. Closes #10338621

[dataimport] rename massive store's metagen_push_relation method to _insert_meta_relation.

[dataimport] meta_data -> metadata

[dataimport] use sql shortcut

[dataimport] move cwmassive_constraint temporary table handling to the PGHelper class with consistent renaming and some rationalization to avoid doing things several times and have code slightly easier to grasp.

[dataimport] inline some methods of the massive store

[dataimport] move everything related to the "URI handling" feature of the massive store together

[dataimport] test for a value is in a set and insertion in a set should live together also inline some methods on the way, in the hope to make the public api clearer at some point, and only put valuable values in those sets so we don't have to endlessly test for consistency.

[dataimport] use a single _initialized set instead of _entities / _rtypes and rework a bit methods using them so that 'in' test and set insertion leave in the same method.

[dataimport] drop massive store etype to eid index and fix potential problem with is_instance_of relation insertion in case where the class of inserted entity has some (yams) parent.

[dataimport] add a .schema shortcut attribute on the massive store also rephrase a bit a method along the way

[dataimport] rework massive store's __init__ * _cnx is set by the parent class * count_cwuri is not used * group related things together

[dataimport] rework PGHelper class

[dataimport] flake8

[dataimport] enhance a bit sql queries readability

[dataimport] use IF EXISTS when possible it's better than try/except and avoid potential concurrency problems.

[dataimport] PGHelper should be responsible to retrieve the database schema

[dataimport] introduce a new MetadataGenerator to replace former MetaGenerator with a similar-but-different API that will allow its usage by the massive store (no more entity object manipulation), while making it a bit more flexible for customization by subclasses (experience taken from client code). Related to #10338621

[dataimport] Prefer now(pytz.utc) to utcnow

[dataimport] rename nohook store .source attribute so it's clearly not the samething as metagenerator.source. Also add a bit of doc along the way.

[dataimport] make rql store test easier to read and test a few more thing, then reuse it to test nohook start which had no tests at all

[dataimport] remove useless assignment in massive store

[dataimport] move eid attribution from the metagenerator back to the nohook store

[dataimport] rephrase minor bits of the meta generator

[pkg] Depends on logilab-common >= 1.2.2

Merge with 3.22.3

[pkg] Depends on yams >= 0.43 RPM spec file and debian/control got updated previously.

Added tag 3.22.3, debian/3.22.3-1, centos/3.22.3-1 for changeset b1e7de000536

[pkg] Version 3.22.3

[pkg] Depends on yams < 0.43

Define BASE_URL and DATA_URL js variables in the login template. Closes #13385185 Those are expected to be defined in almost every js files. They were not before this patch because login form doesn't use the regular main template.

[web] Remove some unused imports in editforms

Fix cancel action on deletion confirmation form Currently crashed with RequestError: invalid action '__action_cancel'. It's broken as of https://hg.logilab.org/master/cubicweb/rev/cb217b2b3 This fix is similar as the one done in https://hg.logilab.org/master/cubicweb/rev/9f1c89e7426d Closes #13521493

[rql2sql] Test and fix some case of sql generation when an unstable variable (ie whose type change from one solution to another) is used in an EXISTS node but belongs to an outer scope. Closes #13518646.

[test] Add missing os import in unittest_migractions.py Dropped in ^ merge.

backport 3.22 changes

[entity] Make it possible to specify sorting terms in cw_related_rqlst method Closes #12306543.

[sql gen] Properly create UNIQUE index on non string column. Closes #13728492 Unique was previously detected in a loop triggered only for String columns.

[server] some pep8 changes

pep8 schema2sql

[migration] Drop custom "readfunc" for code.interact in shell As the referenced bug http://bugs.python.org/issue1288615 got fixed in Python 2.6, this is no longer needed. Furthermore the implementation is not Python 3 compatible (raw_input, unicode).

[tox] Put development dependencies into a dev-requirements.txt file The latter being sourced in tox.ini. For now this contains references to hg repositories for some dependencies but under normal circumstances it should only contain pytest. This is useful when developping cubicweb itself without using tox but working with a virtualenv so that one can simply do: :: $ virtualenv --system-site-packages venv $ . venv/bin/activate $ pip install -r dev-requirements.txt $ python setup.py develop

[py3k] Fix config-level comparison in "cubicweb-ctl create" command dict.get -> None, and None cannot compare to an int in Python 3.

[pkg] PEP8 fixes in __pkginfo__.py

[pkg] Drop "support" of unused subpackage_of option in setup.py __pkginfo__ does not declare subpackage_of.

Add missing import, eaten by previous merge

Only access once to _cw.message in the login form else the first access may pop the message and the second won't get anything to display. This works on default message implementation but doesn't on the pyramid's one, leading to no message displayed on erroneous authentication attempt, which is not very user friendly.

backport 3.22 changes

backport changes from 3.21

[server,bfss] fix bfss migration test for python 3 (related #13519541)

[pkg] Use io.open with utf-8 encoding to read long description from README This forces decoding so that this is consistent between Python 2 and 3 as, in Python 3, decoding which would occur anyways and may fail depending on the environment locale settings.

[dataimport] order of ExtEntities should be irrelevant (closes #13117472)

Test and fix "pickleability" of Binary objects which has been recently broken while some cubes rely on this (eg fastimport). Do some licensing/pep8 cleanup along the way. Closes #13385274

[server,bfss] fix storage migration when Bytes attributes is None (closes #13519541) When a Bytes attributes is None before the BFSS storage migration, the migration crashes. A Bytes attribute can be None, when the attribute is not required and there is no default value. In this case, the attributes should remained None after migration and no file is stored on the disk.

Let hg Ignore py.test's .cache directory And drop rule for .svn along the way.

[entity] Add a cw_related_rqlst method returning the RQL select node of cw_related method This is useful if one wants to modify the query before execution. A bit a PEP8 style fix in tests along the way. Related to #12306543.

[devtools] set .config in setUpClass instead of using a cache in a classproperty This is much clearer and avoid problem with py.test introspection that triggered creation and caching of the configuration before actual test execution. This caused pb in unittest_postgres.PostgresStatementTimeoutTC where configuration was built before module's setup, hence source configuration was missing information about where to find the pg cluster.

[devtools] Simplify test configuration's init Specify module's __file__ instead of apphome. __file__ is saved on the class as it will be used later (eg to find the proper pg cluster without relying on shared dictionnary modifications). This will require some tests update, but it should be less common in cubes than in cubicweb itself.

[web] Have the LoginController return 403 on failed auth (closes #11569950)

[devtools] Drop old init_test_database backward compat

[devtools] show DeprecationWarning during test execution They are not shown by default since python 2.7

[devtools] deprecate usage of sources file for test This is old support which clutters the code and should no more be used. As this is rarely used (if it is) and test only, it feels ok to drop support at once without a deprecation phase.

[devtools] Set dumb value for postgres's db-host to avoid unexpected connection to a local server Currently, if something goes wrong (eg config built before call to startpgcluster) while a pg server is running on localhost, it may connect to this server and run the test in there unexepectedly.

[ldapfeed] ldap3 do not raise an exception in case of failure of cnx.bind() but return 'False' instead.

[test] add a test to ensure we get an AuthenticationError with ldapfeed when using (correct login, wrong password) credentials

[entity] Exclude computed relations from Entity.copy_relations Closes #12481591.

[pkg] Set version to 3.22.2.dev0 So that cubes used in test dependencies do not install a released CubicWeb.

[devtools] pep8 a bit package's __init__

[test/server] Only use configcls with statement timeout when needed Other tests in PostgresFTITC class apparently do not make use of this particular configuration.

[server/test] Add an assertion in test_statement_timeout

[datafeed] minor cleanups to cwxml parser

[datafeed] don't allow arbitrary kwargs on DatafeedParser initializer else they will be handled nicely by appobject base class, while we rather want an error in the case of a source parser (arguable).

[datafeed] gives information about moved entities to the parser Also, don't support anymore arbitrary parameters at parser initialization. Related to #5414754

[datafeed] simplify SQL query used to retrieve information about entities from an external source no need to join on cw_source

[datafeed] rename parser.sourceuris to source_uris Its more readable. Also, uniformize related method and local identifier names.

[datafeed] move process_urls to the parser

[tox] Use py.test to run tests In unittest_rqlrewrite.py and repotest.py, rename utility functions so that they are not considered as "test functions" by py.test.

[test] Fix devtools unittest_i18n when ran without pytest There is currently some weird error, requiring both to hack cubicweb.__path__ and to stop using load_module_from_name to load the registry. Introduce a dependency on to-be-released version of logilab common integrating fix for https://www.logilab.org/ticket/6085650 Do some extra import cleanups along the way.

[server] explicitly set postgres LC_MESSAGES variable to C on connection We parse some error messages so we need them to be in English. It's not clear to me exactly when that's not the case, but this fixes running unittest_postgres' "test_constraint_validationerror" with py.test for Sylvain.

[web] cleanup xmlrss view Don't trigger exceptions unnecessarily, and add line breaks to the output.

[migration] add rollback when replace_eid_sequence_with_eid_numrange fails If we get here, the current transaction is aborted, so if we are to assume everything is fine and continue we need to rollback first.

[migration] move CWComputedRType later The ordering in this file is a mess, and things don't actually work this way (when upgrading from 3.17).

docstring fix

[cwctl] make cwctl not depends on cubicweb.wsgi (closes #12210063) cubicweb.wsgi may not be available at all on Debian.

[hooks/workflow] pass eids, not entities, from hooks to operations Passing entity objects around runs the risk of getting out of date data from caches.

[web/httpcache] fix Last-Modified generation time.mktime takes a time tuple in *local* time. We have an UTC datetime coming from the last_modified method, which we were interpreting as local time, leading to hilarity. Use calendar.timegm instead. Add tests for Last-Modified generation of the 'download' and 'manage' views.

[forms] Fix edition of TZDatetime attributes Parse the submitted web form values into a timezone-aware datetime (by default, UTC) when such an attribute is edited. This is done using an automatically-selected (by `web.formfield.guess_field`) field named `TZDatetimeField`. Closes #12666598.

[doc] fix abstract and example of data import

[doc] fix abstract of data import

[repository] drop remanescence of old multi-sources code we don't use anymore this inputmap/varmap thing.

Added tag 3.20.15, debian/3.20.15-1, centos/3.20.15-1 for changeset 636a83e65870

[pkg] 3.20.15

[views] do not eat the self._cw.message in LogFormView (closes #12421177) accessing self._cw.message now(?) eat the message, so don't call "self._cw.message" twice.

[server] make InternalManager.name() return an empty string (closes #12418708) Otherwise the notification emails sent via an internal_session never honor the EMAIL.sender-name configuration option.

[test/migractions] work around drop_cube not deleting final etypes Closes #10096736

[hooks] deleting a final entity type shouldn't try dropping an sql table

[tox] Use a revision on proper branch for Yams "default" is not good when some bug fix release suddenly comes out...

[webconfig] disable generate-staticdir by default Its side effects mean it can be inconvenient, and using it requires manual (web server) config changes anyway, so there's no reason it needs to be enabled by default.

[test] Skip test_deserialization_base (unittest_repository.SchemaDeserialTC) See ticket #10096736.

[devtools] drop unused DumbOrderedDict2 class

[devtools] fix sorting in ExecutionPlan._check_permissions on python3 Sort lists instead of dict_items objects to get predictable results. With this change, unittest_querier.UtilsTC.test_preprocess_security passes every time instead of once in a blue moon. Closes #11759133.

[test] assertItemsEqual → assertCountEqual in unittest_repository.py

merge 3.20.14 in 3.21 branch

Added tag 3.20.14, debian/3.20.14-1, centos/3.20.14-1 for changeset f66a4895759e

[pkg] 3.20.14

[tox] use master URL for yams repository No need to clone from review. This should help get consistent results.

[pylint] Avoid invalid-name on schema relation class names

[pylint] more work on the pylint support * uniformize usage of ClassDef and simplify related import * don't insert in the ast a python function but an astroid representation of that function * ignore __pkginfo__ file * drop attr-rgx, variable-rgx and argument-rgx, put the name in good-names instead * drop additional-builtins * add cell_call to the list of methods for which we don't want 'abstract-method'

[pylint] remove abstract-method warning This kind of message is targeted: "Method 'entity_call' is abstract in class 'EntityView' but is not overridden (abstract-method)" For now, only "entity_call" and "render_body" are dealt with, the list can be extended.

[pylint] upgrade pylintext plugin

update and enhance pylintrc

[schema] use json to serialize constraints Require yams 0.43: constraints are serialized to json, which means we need to recreate the actual checks in the database on upgrade. Temporary deps in tox.ini to pull respective changes in yams.

[pkg] fix cubicweb.spec syntax (grafted from 3309ddb970594cd54529f5c205d1ea5b5da4131e)

[pkg] Use Twisted < 16.0.0 We have some test failures with 16.0.0, such as: - test_put_content (unittest_server.ETwistHTTPTC) - test_base_url (unittest_httptest.TwistedCWAnonTC) (grafted from 10c12f31d0fe38c2e8814176fc42bf2ea210efdf)

[pkg] fix cubicweb.spec syntax

[cwconfig] Use imp.load_source instead of exec() to load site_cubicweb.py from apphome Lets us return a module object and go back to the previous code in CubicWebConfiguration._load_site_cubicweb.

[cwconfig] Stop using lgc.modutils to load *config.py and *ctl.py files

[cwconfig] Stop using lgc.modutils to load site_cubicweb.py files config.load_site_cubicweb function now takes a list of cubes (instead of paths) and default to all known cubes.

[test] Use a Popen in i18n tests Instead of check_call, so that we clearly see the failure. A bit of flake8 along the way.

[forms] Fix HTableFormRenderer _render_fields method behavior when display_label is False. Indeed, now the extra <tr><th>...</th></tr> are not inserted anymore unless display_label is True. Closes #3811325.

[connection] remove a dead counter Related to #1381328.

[session] drop .is_repo_in_memory This is deader than a dead dog's bone buried down a blind alley off a dead-end street in a ghost town. Related to #1381328.

[session] drop session-as-a-context-manager It made little sense in the 3.19 - session-as-cnx-bw-compat era, maybe. Related to #1381328.

[devtools/repotest] use the RepoAccess object Quite better than playing tricks with the Repo._sessions. Related to #1381328.

[devtools] adapt stresstester to current APIs

[session] retire session.vreg Related to #1381328.

[session] fix super call in Connection.__init__ The vile hacks in test fixtures are fixed.

[session,repo] remove last trace of "cnxprops" Related to #1381328.

[pkg] Use Twisted < 16.0.0 We have some test failures with 16.0.0, such as: - test_put_content (unittest_server.ETwistHTTPTC) - test_base_url (unittest_httptest.TwistedCWAnonTC)

[web/views] stop using cwaction='cancel' It's broken as of https://hg.logilab.org/master/cubicweb/rev/cb217b2b3

[test] fix test_printable_value_bytes with current pygments An empty span was added in https://bitbucket.org/birkenfeld/pygments-main/commits/164574c13533195a555181a2b9c685fea2470403

[session, repository] deprecate repo.connect and move .close reponsibility to session object Repository.new_session just returns a plain session object, and `.connect` (which returns a sessionid) is deprecated. For .close:: session.close() # done ! The session will bear the responsibility to call the "session_close" event but that's better than the previous idiom:: repo.close(session.sessionid) which involves both objects. Related to #1381328. Related to #2919309.

[web/test] fix typo in doc string

[web/httpcache] send weak ETags There's no way we can provide a strong validator in general (especially considering how ETags are generated by the current EtagHTTPCacheManager and EntityHTTPCacheManager). So mark our entity-tags as weak.

[server] print backend exception if we fail to execute an sql script It's more useful to see could not open extension control file "/usr/share/postgresql/9.4/extension/postgis.control": No such file or directory rather than merely -> ERROR, skipping /usr/share/cubicweb/cubes/postgis/schema/_ext.postgres.sql

[cwctl] drop unnecessary admincnx call from 'shell' The connection is done by the migration helper.

[serverctl] allow schema-diff when db and fs are not in sync

[cwctl] remove networked remote shell This was a dbapi leftover. Related to #3933480.

[web/calendar] fix regression in previous changeset "Enables css customization in CalendarView" was untested, and introduced a crash: TypeError: not all arguments converted during string formatting

Enables css customization in CalendarView now a custom css class can be set on the calendar div (closes #3654043).

refactor a bit breadcrumbs to ease overriding of the root's display

Add migration helper function to delete entity types faster Bypasses all hooks and (for now) ignores any composite relations.

[testlib] Import nocoverage from logilab.common.testlib This depends on a change in logilab-common 1.2.0

[web/test] replace unicode with six.text_type Some uses of unicode reappeared when merging changes from 3.19.14.

[devtools/testlib] Issue a RuntimeError when a generative tests is detected Otherwise, generative tests *à la logilab-common* in CubicWebTC test cases would silently not run (at least using logilab.common.pytest runner).

[devtools] stop using lgc.testlib.TestCase It's become incompatible with unittest(2).TestCase due to generative tests. Just keep the datadir/datapath and assertCountEqual bits in our BaseTestCase.

[test] Replace remaining generative tests by subtests

[tox] Enable all applicable py34 environments All except etwist. Comes with a few tricks in tox.ini: * add ``-e.`` to force installing of cubicweb in development because dependency cubes would otherwise install cubicweb 3.22 (the latest release), which is not python3 installable. * "install" cubes/__init__.py because the file is not installed in develop mode; maybe this file should be managed by cubes themselves...

[migration/3.10] fix AttributeError 'Connection' object has no attribute 'HOOKS_DENY_ALL'

[server/sources] Fix authenticate on pre-3.10 database When migrating from a pre-3.10 database, the CWSource entity type and cw_source relation type are not known, but authenticate should still work to be able to add them.

[test] Add a test ensuring new cube built from skeleton is flake8-compliant And make skeleton files actually flake8-compliant henceforth.

[skeleton] Ship a tox.ini file upon new cube

[web] fix headers on 304 responses (closes #11466875) The response MUST include the following header fields: [...] - ETag and/or Content-Location, if the header would have been sent in a 200 response to the same request https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.5 Unfortunately we weren't copying headers from what would have been the 200 response, but from the request.

[web/httpcache] GMTOFFSET no longer exists

[web] fix headers on 304 responses (closes #11466875) The response MUST include the following header fields: [...] - ETag and/or Content-Location, if the header would have been sent in a 200 response to the same request https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.5 Unfortunately we weren't copying headers from what would have been the 200 response, but from the request.

[rset] fix infinite recursion introduced in c1eb5a676c80 The mechanism to avoid loops in c1eb5a676c80 "[rset] Always complete attribute/relation caches in ResultSet.get_entity" breaks down: we would loop forever if two entities that were already in the cache were linked by a relation with ? or 1 cardinality in both directions. To avoid that, keep a set of already-considered columns. Related to #9942503.

pep8 unittest_application and unittest_formfields

[rset] Always complete attribute/relation caches in ResultSet.get_entity RQL queries are often designed to fill up the ORM's caches when fetching entities out of the result set. Until now, if an entry already existed in the entity cache, ResultSet.get_entity would return it unchanged, not using the new ResultSet's contents to update the attribute cache, breaking expectations (if the attributes are needed, they'd then be fetched later one at a time, one entity at a time), resulting in loads of DB accesses. So we change ResultSet.get_entity so that: * if the entity is already cached and has been instantiated from the same rset, it is returned as-is (to avoid loops) * if the entity is not yet cached, it is instantiated * if the entity is cached via another rset, its attribute/relation caches are completed Closes #9942503

[rset] refactoring of _build_entity method define a method for entity instantiation and storage in cache (related to #9942503)

[entity] comparing two entities returns NotImplemented Fixes a test when using some versions of unittest2, where assertItemsEqual tries a sort and expects TypeError, which it doesn't get if we directly raise NotImplementedError. The NotImplemented value is treated specially by the interpreter: https://docs.python.org/3.4/library/constants.html#NotImplemented

[test] replace one use of lgc.testlib.unittest_main with unittest.main

[test] stop using TestCase.set_description It's a lgc-ism.

[test] convert a lgc-style generative test into unittest2-style

[web/application] fix query log handling on python3 Writing bytes to a file opened in text mode doesn't work.

[devtools/test] duplicate CubicWebServerTC tests for CubicWebWsgiTC

[devtools/test] fix bytes vs text in test_base_url Needed for python3 with the next changeset

[devtools] skip CubicWebServerTC on python3 People should migrate to wsgi if they're going to use CubicWeb on python3.

[web/test] switch unittest_web from twisted to wsgi And make it pass on python3 while at it.

[devtools] add CubicWebWsgiTC Similar to CubicWebServerTC, except using wsgiref instead of twisted as an http server.

[web/test] return the hash of uploaded files in FileUploadTC instead of their contents json must be unicode, which doesn't go well with arbitrary file contents. Compute the file hash instead, as we know the hexdigest is (ascii) text.

[web/test] open() as a context manager

[web/test] json.loads expects text, not bytes

[wsgi] fix handling of multiple posted files We were only keeping the last received file.

[web/test] Reword doc string for fileupload ajaxfunc

[skeleton] do not import the __pkginfo__.py module to make cubes installable via setuptools When using a simple "import" statement, running 'python setup.py install' for a cube fails in a fresh virtualenv due to the fact the install of the dependencies (here cubicweb iteself) is done by the python process doing the cube's installation, so the __pkginfo__ module from the cube is already loaded (thus used) when installing cubicweb... So we apply the third technique from: https://packaging.python.org/en/latest/single_source_version.html

merge changes from 3.22.2

Added tag 3.22.2, debian/3.22.2-1, centos/3.22.2-1 for changeset 1b93ff37755b

[pkg] 3.22.2

[pkg] fix centos install

[datafeed] do NOT call commit in DataFeedXMLParser.process() (closes #10928592) and so, remove more useless code. Doing a commit in DataFeedXMLParser.process() is a bad idea because this method is called for every imported/updated entity, before handling deletions. So if we commit here, odds are quite high that we're violating some integrity constraint and end up with a ValidationError.

[schema] restore constraint checking when running on old sqlite Old sqlite3 doesn't provide CHECK constraint names in error messages, preventing us from translating a backend integrity error into a ValidationError. This was added in 2012, but the sqlite3 version in RHEL6 is older; so if we run on old sqlite, keep checking the constraints in python rather than only in SQL. Closes #10927494

[web/autoform] fix regression introduced by #5758ba784ebd The changes from cset 5758ba784ebd "[autoform] when building inlined-form view for a relation, consider related entities intead of other views" lead to an unexpected 'add new' link on creation of an entity with inlined form for a relation of cardinality 1. To avoid this, we should consider formviews and not existing relations when the entity is being created. Closes #10860258

[tox] Generate test environments for Python 2.7 and 3.4 Test commands for each environment are written down explicitly since I could not find a way to extract the "package" name (e.g. "hooks") from the environment name (e.g. "py34-hooks"). For Python 3.4 interpreter, only environments (subpackages) that do not depend on cubes for their tests are listed since those test dependency cubes are not yet installable with Python 3.x. etwist is also not included since the Python 3 port is not complete at the moment. From local testing, py34 tests pass for the following subpackages: - dataimport - entities - ext - hooks - wsgi

[notification] avoid using InternalManager when we've got a CWUser This will avoid potential information leak due to missed security restriction and is more compatible with legacy code (pb encountered in some client code). Closes #10294761

[dataimport] rename .time to .now attribute of meta generator

[pkg] Set version to 3.22.2.dev0 So that cubes used in test dependencies do not install a released CubicWeb.

[dataimport] cleanup a bit nohook store, mostly protecting some attributes

[dataimport] add a filter function to not fail if some extentity has several values for an attribute of final relation the function will simply record a warning in the import log and keep a value randomly. Notice the function has to be explicitly inserted in the transformation stream.

[sources] synchronize source asynchronously when started from the UI and redirect to the forthcoming import log, for a better user experience (there is still a lot more to do there though). Closes #10468967

[test] fix unittest_devctl on python3.4

[test] Make sure that newcube command test produces expected files

[server] remove some pre 3.10 compat Let's assume nobody will try upgrading an instance from pre-3.10 cubicweb to 3.23+. If they do, they're setting themselves up for a lot of pain anyway.

[pkg] use https URLs to pypi.{python.org,debian.net}

[pkg] use https URL for www.cubicweb.org

README: use https URLs for docs.cubicweb.org

pep8 bits

[i18n] enhance text formats french translation

Merge with 3.22 branch The merge was clean, just dropped cubicweb/web/data/cubicweb.goa.js.

[dataimport] update example to use standard importer

merge changes from 3.21.6

flake8 serverctl

c-c source-sync now actually force synchronization as advertized by its help message. This was because "force" argument wasn't considered by acquire_synchronization_lock().

Added tag 3.21.6, debian/3.21.6-1, centos/3.21.6-1 for changeset b3cbbb7690b6

[pkg] 3.21.6

merge changes from 3.20.13

Added tag 3.20.13, debian/3.20.13-1, centos/3.20.13-1 for changeset 8c5dabbcd4d9

[pkg] 3.20.13

[pkg] Yams >= 0.42 breaks CubicWeb < 3.21 Closes #9720390.

merge changes from 3.19.14

Added tag 3.19.14, debian/3.19.14-1, centos/3.19.14-1 for changeset 72a0f70879ac

[pkg] 3.19.14

[sources/native] remove extra space before colon This isn't French.

[web/views] fix editcontroller regression unittest_views_basecontrollers.py's EditControllerTC.test_edit_mandatory_inlined3_object regressed with changeset dcbb64d3a1d9 because it relied on POSTing nonexisting eids. This seems like a silly idea, but let's keep the behaviour unchanged.

Added tag 3.22.1, debian/3.22.1-1, centos/3.22.1-1 for changeset d0d86803a804

fix test broken by #012e6c7d02ef

[pkg] Attempt to fix rhel6 build I don't know what I'm doing, but this seems to work.

[dataimport/test] fix python2.6 syntax error Don't use braces for set literals.

Remove unused import in entity.py

[migration] don't ask confirm for those sql queries

quick fix which closes #0673348 - see comment for more info

[migration] test if entity type has been in the deleted in the transaction should protect both clauses else we may attempt to execute the UPDATE query on a table which has been deleted. Closes #10692374

[pkg] 3.22.1

[cwctl] remove support for startorder file in /etc/cubicweb.d (closes #5618947) It's now useless.

[exceptions] fix string conversion for Unauthorized exception changeset 3cb87b61b067 changed CubicWebException to define __unicode__ as the main entry point for string conversion, but left Unauthorized's __str__ definition unchanged, leading to a mismatch. Closes #10646396.

Fix Binary on python < 2.7.4 (closes #10593811) Before the fix for http://bugs.python.org/issue10211 we can't pass a buffer object to BytesIO, so keep using StringIO in that case. And because StringIO is an old-style class, we can't use super().

[migration] fix crash on 3.22 upgrade if entity type doesn't exist in fsschema (pending removal)

[autoform] when building inlined-form view for a relation, consider related entities intead of other views by considering previous form-views, we may miss that there exists related entities but which are not editable. Closes #10495602

Add a __main__ to cubicweb This avoids calling cubicweb-ctl relatively in tests and could be useful in virtualenv. This is just a copy of bin/cubicweb-ctl file, which should probably be generated from a setuptools entry points.

merge 3.19 heads Getting an unexpected test error though: ERROR: test_edit_mandatory_inlined3_object (unittest_views_basecontrollers.EditControllerTC)

[cwvreg] Fix propertyvalues hot updates When CWProperty instances are touched, the corresponding value gets updated in vreg['propertyvalues'], but their type was ignored.

[pkg] remove useless python >= 3.4 dependency python is python2 is never >= 3.4.

[dataimport/test] fix test failures since 35b29f1eb37a The index name changed, so we need to follow suit.

[pkg] do not include toplevel directories in rpm (closes #10362119) Our spec file conflicts with the filesystem package in centos7: we set permissions to 0755 for /usr/bin and /usr/lib whereas filesystem sets them to 0555. So to be compatible with both centos7 and others we simply ignore these directories.

[devtools] fix config.repository signature Its argument is called 'vreg', not 'x'. Closes #10515113.

[predicates] fix paginated_rset when the class has no page_size_property Fixes test failure introduced by 70733b2dfd2a, which assumed paginated_rset would only be called on a class with a page_size_property attribute.

[repository] drop usage of no more necessary eschema_eid function since the previous cset, we are guaranteed that repository's entity schema will have their eid attribute properly set, so we don't have anymore to check everytime if some entity schema's .eid attributes is set or not (and retrieve it if not). Related to #10450092

[repository] set .eid on eschema when schema is loaded from the filesystem enforcing the contract that a repository's schema should have .eid attribute of entity schema set to the eid of the entity used to serialize them in the db. Before this cset, this was not true during tests or for some c-c commands where 'quick_start' is set (eg db-restore, i18ncube...). The change in server __init__ makes this assumption true during instance creation: the serializing code was actually setting eid on schema object, but a reference to a previously built schema was given instead of the one for the latest created repository. Closes #10450092

[dataimport] simplify SQLGenSourceWrapper.add_info At this point 'CWSource' is a thing (we're not in the middle of a migration), and source.eid won't be None (we read it from the db)

ensure .tox is blacklisted when recursing on the fs to get some files among others, avoid crash when using 'c-c i18ncube' on a cube using tox. Closes #10429317

drop a note about usage of sqls_create_multicol_unique_index and avoid a lgdb warning

[devtools] don't auto-populate virtual relations

[skeleton] Restore verbose output in setup.py This part of commit fdadf59be612 missing the integration window.

[hooks] Skip modification_date attribute in "notifentityupdated" hook Depending on the call order of hooks, "modification_date" may come in entity.cw_edited or not (from "metaattrsupdate" hook usually). It should however not be considered in the notification context. Closes #8842064.

[test] Gather tests about mail module From sobjects/test/unittest_notification.py to test/unittest_mail.py where other tests for mail.py are already. Some style fixes along the way.

[web/component] guard against invalid page_size form parameter Same as dff28956f37e, except the almost same code exists in two places (yay us).

[predicates] fix paginated_rset for components that change page_size_property The predicate and component had better agree on the page size.

[devtools] fix get_bounds Say an attribute has two IntervalBoundConstraints, one with minvalue=0 and the other with maxvalue=10 (not literally, or that could be a single constraint, but e.g. with references to other attributes). We don't want to override the minvalue of the former with None just because the latter happens last in the constraints list. This doesn't fix the case where two constraints both have a min or a max value. Meh.

[devtools] "fix" boundary condition for autofill maxvalue in an IntervalBoundConstraint is inclusive. If the min and max values happened to coincide, we would call random.choice on an empty list, which it didn't like. This is still broken in the case of BoundaryConstraint('>', minvalue), as we assume the minvalue is in the range, but that's something for another time.

[test] Use stdlib doctest module instead of logilab-common class The former just works fine now but the later does not play well with py.test.

[test] Re-implement generative tests with subtests

[devtools] Re-implement generative tests using subtests Generative tests as implemented in logilab.common.testib are not compatible with tests runner other than lgc.pytest and this implementation differs from the standard library, which has support for subtests_ since Python 3.4. Use unittest2 to bridge the gap. Maybe it'd be good to implement this on logilab.common.testlib side at some point. Let's see how this gets received here first. .. _subtests: https://docs.python.org/3/library/unittest.html#subtests

cleanups

[pkg] python-tz does not exist in centos should be pytz (closes #10290056)

[test] Skip LDAP tests if any of the underlying programs is not found Some programs are in ldap-utils or ldapscripts, which may not be installed despite slapd may. Follow-up on 282880f81311.

Handle virtualenv development mode Looking for VIRTUAL_ENV environment variable as a sign of virtualenv activation. This at least works for developing CubicWeb itself (not sure yet about cubes). And it makes `tox --develop` work as well. Add a test for cwconfig._find_prefix within a virtualenv. In tox.ini, use `develop option`_ accordingly. This makes tox running significantly faster since it avoids building and installing the sdist. Aside, the "develop" install is now performed *after* all test requirements are processed, which means that the cubicweb package should always be the one from source -- instead of a released version pulled by a cube through test dependencies, which may occur in "recreate" step. .. _`develop option`: <http://tox.readthedocs.org/en/latest/config.html#confval-usedevelop=BOOL

[skeleton] Use os.walk in setup.py for Python3 compatibility

[skeleton] Use unittest.main directly Instead of lgc.testlib.unittest_main as they are identical.

[devtools/testlib] Disable pause trace in setUp Closes #9856028.

[test] Remove dependency on person cube in devtools package

[pkg] add missing python-tz dependency

[edit controller] Fix composite entity reparenting when inlined This implies handling the reparenting case sooner, otherwise the child entity modifications are cancelled (see 113e9da47afc). Add a new test to ensure non-regression in all previous tests. Closes #10291977.

[forms] Do not raise errors on composite required relation being deleted The entity included in the composite will be deleted in such a case, so the field is not required even when the cardinality is '1' or '+'. Related to #8529868

[edit controller] Cancel RQL queries to be performed on entities to be deleted Composite relation removal via the EditController trigger an entity removal. Changes on such a to-be-removed entity must be cancelled as they may trigger integrity errors before the deletion occurs. Closes #8529868.