[hooks] base64-encode extid before inserting it into moved_entities If we insert the wrong value we're never going to find these entities again... Closes #5741105.

[hook] remove assumption about entity cache vs cw_edited We were assuming that if an entity is in the connection's cache, it has a cw_edited attribute. This doesn't seem to always be the case. At least in cubicweb 3.21, merging the web and repo-side caches makes this more likely, but Denis says he's also seen this on 3.20.

[hook] don't open-code Connection.entity_cache

[tox] Add missing dependency on webtest for devtools tests

[tox] Add missing dependency on webtest for wsgi tests

[tox] Remove set of PATH environment variable The : concatenation does not work actually so the resulting PATH was just wrong (it did not contain the previous PATH). Anyways, the PATH environment variable is passed by default by tox so it's probably better to let the caller set it (thus avoiding to hard-code the specific postgres version and path). http://tox.readthedocs.org/en/latest/example/basic.html#passing-down-environment-variables

[doc] generalize info about debian/ubuntu repositories That section was not kept up to date with recent distro releases; stop trying.

[doc] fix link to setup info in README ... and fix URL to the repository's gpg key.

[server/session] remove unused cnx_meth function

[server] don't issue a warning for something expected Creating an instance happens all the time in tests, there's no reason to warn every time.

[server] stick a _cw on internal connnection's user Entity._fetch_restrictions goes through user._cw.vreg :( Closes #5705769

[migration/3.21] fix stupid error in migration script str.join doesn't like a list of ints. Closes #5706359.

Added tag 3.21.0, debian/3.21.0-1, centos/3.21.0-1 for changeset 887c6eef8077

[pkg] various fixes - add missing files to MANIFEST - exclude doc/changes from dh_installchangelogs (it's a directory, which dh_installchangelogs doesn't expect) - fix doc build in debian package

[doc] fix broken symlinks

[pkg] update 3.21.0 release date

[doc/3.21] mention the postgresql-only state of the upgrade script

[datafeed] also catch EnvironmentError when trying to load the cwclientlib config file means the config file has not been found, so fall back to the old implementation instead of crashing.

[schema] Make syntax compatible with print_function yams seems to exec() us sometimes (instead of __import__()), and yams.reader uses print_function nowadays.

[pkg] 3.21.0

[test] use FakeFile instead of File in test_fulltextindex File.data is no longer fulltextindexed.

i18n update

[serverctl] fix schema-diff command It hadn't been ported away from dbapi.

[serverct] call server_{maintenance,shutdown} hooks around source-sync command

[serverctl] call server_{maintenance,shutdown} hooks around add-source

Update hgignore

delete unmaintained clone_deps script Closes #2517580.

merge 3.20.9 into 3.21 Fix conflict in unittest_postgres.py by changing sources configuration in PostgresTimeoutConfiguration.__init__ so it happens *after* setUpModule().

Added tag 3.20.9, debian/3.20.9-1, centos/3.20.9-1 for changeset d477e6447582

[pkg] 3.20.9

[forms] don't display 'remove' link of inlined form for the first entity related using a '+' cardinality to its parent entity Closes #5227394

[tox] Update doc testenv to account for documentation restructuring * Build doc from `doc` directory. * Use `sphinx-build` directly instead of `make` as this now works and allows to build within .tox temporary environment (instead of doc/_build), preferable for a "test" action. Related to #5447161.

[schema] drop old backward compat code preventing yams to be itself fully cleaned

[datafeed] allow to use cwclientlib for datafeed's queries (closes #5456849) HTTP based URLs will be handled by a CWProxy if: - cwclientlib is available and - the URL matches a cwclientlibrc config entry Otherwise, fallback to previous implementation. See https://www.cubicweb.org/project/cwclientlib :Warning: This comes with an API modification of DataFeedParser.retrieve_url: it used to accept data and headers arguments to build GET or POST (if data was given) queries, but this was not used by any known code and imply more complicated code.

[cwxml parser] refactor url handling code to use urlparse and urlencode (related to #5456849)

add use_extid_as_cwuri ext entity transform, that will be often necessary and not so easy to write at once

[dataimport] dispatch and deprecate old code previously, the whole dataimport.py content has been copied in dataimport/__init__.py, then dataimport/importer.py API has been introduced. Following these changes, this cset: * dispatch __init__ code into sub-modules * dispatch unittest_dataimport accordingly * deprecates functions and classes that are deprecated by the new API Related to #5414753.

Add tox.ini file * Factor out `commands` for most environments using {envname} substitution to build pytest commande. * Positional arguments are passed on to `pytest` (others may also be passed after a `--`). * Set PATH env variable for tests requiring PostgreSQL. * Also added a `doc` target to build the book. Also update .hgignore exclude .tox directory (where virtualenvs will be created and cubicweb.egg-info created by sdist command). Closes #5447161.

Add requirements.txt files in test directories These files, located in each test directory (when needed), list test dependencies supplementary to install requirements. Some requirements.txt files include psycopg2 because it won't get installed by logilab-database apparently. Related to #5447161.

Spelling fixes in comments and docstrings

[web/views] Clean up confusing backwards compatibility code Dead code since fbb370195199, related to #4943392.

[server/test] avoid lgdb warnings from str to unicode conversion

[web/test] Modernize unittest_propertysheet.py - remove call to os.makedirs in global scope - fold CACHEDIR as an instance attribute - bonus: use unittest directly instead of lgc.testlib

[web/webconfig] Drop pre-3.9 backward compatibility method "has_resource"

[web] Stop using negative indices with os.stat results The python doc says: More items may be added at the end by some implementations. .st_mtime is also more readable than [-2].

[doc] small rst fixes and cleanups in changelog files Related to #4832808

[doc] make the changelog chapter a one html page Related to #4832808

[doc] reformat changes files headers to remove the "what's new" phrasing and add the publication date instead. Related to #4832808

[doc] remove mentions of pyro and reposity instance type from the concepts chapter Related to #4832808

[doc] put the tutorials between the intro and the `Repository development` toc of the book Related to #4832808

[doc] move the documentations from docstring directly to the book. and improve a bit API documentations. Related to #4832808

[predicates] remove deprecated sentences in docstrings

[predicates] objectify the anonymous_user predicate for the sake of consistency.

[doc] more fixes of warnings/errors in doc build - move 3.21.rst changelog description at it's proper place - include changelogs in the main index - few typos - add :noindex: for autoxxx directives not in api/ Related to #4832808

[doc] fix warnings/errors in doc build - fix links to images - fix a couple of typos - re-add IDownloadableOneLineView doc - rename documenting.rst back to .txt, it's intended as a doc of how to write rst, not part of the rst doc Related to #4832808

[doc] Start documenting the API Related to #4832808

[doc] Remove useless indexes Related to #4832808

[doc] Insert the dev docs in the toc Related to #4832808

[doc] Restructure the documentation * Create a new index file * Move the sphinx configuration files do the documentation root * Move book/README to dev/documenting.rst * Move book/mode_plan.py to tools/ * Move book/en/images to images * Move book/en/* to book/ * Move changelogs to changes/* * Adapt the Makefile * Add a title to the javascript api index Related to #4832808

[migration/3.21.0] performance and reliability fixes - use EXCEPT instead of NOT IN, seems to improve performance dramatically on postgresql - delete rows from cw_<etype> tables whose eid is unknown - deal with missing/dangling required inline relations (by deleting the subject entities) instead of asserting - fix/improve print statements - don't ask for confirmation before running pure select queries

[serverctl] fix source-sync command Closes #5491700

[server] missing import Fixes NameError in an unlikely error case.

[server/test] stop using email cube - make unittest_migractions use a separate datadir, and add a fake email cube there (we use it to test {add,drop}_cube, so need something that looks like a cube) - for the other test modules, just copy what we need from cubicweb-email's schema to our test schema

[web/views] Remove unused variable

[server/session] bring anonymous_connection back Fixes cubicweb-comment.

[migration/3.21] idempotency fixes Drop constraints before (re)creating them This is postgresql-specific syntax, sqlserver needs work.

[migration/3.21] Make index and table creation idempotent

[migration/3.21] Add missing foreign keys when upgrading Try to delete dangling relations, since there's no point in keeping them. Related to #4846892.

[schema2sql] properly consider skip_relations parameters. This is the same bug as yams #286912. Also, introducing a unique function telling if the relation should have a table or not unhide some other problems related to computed relation, or missing skipped relations information.

[hooks/test] don't rely on RelationDefinition implicitly Its presence in globals is deprecated and going away.

[web] write css_class to attrs on rendering and not on button creation Because css_class attribute may be overriden by bootstrap after button creation and this modification has to be taken into account. closes #5457548

[statsd_logger] handle ipv6 addresses

implement a simple statsd logger (closes #5488711) and add a few timing probes on: - etwist.server.CubicWebRootResource.render_request() - server.querier.QuerierHelper.execute() - server.sources.native.NativeSQLSource: - sqlexec() - compile_rql() - authenticate() - doexec() - doexecmany() as well as a pair of counters on server.querier.QuerierHelper cache (hit/miss).

[predicates] ExpectedValuePredicate now accepts a dict parameter ``match_form_params`` and ``match_kwargs`` benefit from that. For instance, the following statements are valid:: __select__ = match_form_params('vid', 'subvid') __select__ = match_form_params(vid='list', subvid='incontext') __select__ = match_form_params(vid=('list', 'tsearch')) In the latter cases, not only the parameters (``vid``/``subvid``) must be in the request form but their corresponding value must also match the expected values (or one of possible choices if `values` is a sequence). closes #5484070

[rql2sql] Fix iter_exists_sols() excessive cleaning of _state.tables Closes #5503548

[cwctl] allow overriding config settings from the command line (closes #5557656) eg. cubicweb-ctl start -p port:8082 myapp to overload the port parameter defined in the all-in-one.conf file.

avoid sanitizing warnings

[devtools] avoid warning about source['global-db-name'] sqlite and postgres db handlers store the db template name there.

[skeleton] remove no-break space from skeleton setup.py

[server/test] drop unused import

[doc] update 3.21 release notes

[devtools] Sort _cw_fields in fake_form Try to generate a predictable string, to help testing with PYTHONHASHSEED=random.

[entity] sort fetchattrs when generating rql Try to generate a predictable rql query, to help testing with PYTHONHASHSEED=random.

Update hgignore for temporary test databases

[js] removed unused and broken SuggestForm Widget (closes #5213487)

[test] Fix unittest_i18n to run properly with "pytest -t" (closes #5576169) When running "pytest -t", many tests modules will be imported and run within a single python process. unittest_i18n fails when run after tests such as unittest_webtest.py or unittest_httptest.py. The main reason is that unittest_i18n does various tricks to add a test cube to the search path, which badly fails when a previous test module has already loaded "stuff" (for lack of a better word). Instead, let's just call the 'i18ncube' command with subprocess. The change from logilab.common.testlib to unittest is not directly needed, but comes as a free cleanup with the above changes.

[server/test] fix random error in unittest_security When entering a new Connection, we cache the user and its 'login' attribute (with no permissions checking). This test makes 'CWUser.login' unreadable by guests, and then proceeds to make sure the 'anon' user can actually not read any 'login' attribute. However, due to the above cnx initialization, anon's login is actually cached, hence readable. This happens to make the test fail sometimes depending on the order in which CWUser entities are returned, because one of them has .complete() called, which as a side effect sets the attribute cache to None for unreadable attributes. Call .complete() on both entities to reset the login cache. While this is still highly debatable, at least it's consistent.

[entity] fix typo in comment

[doc] Add a tutorial and extend documentation for ExtEntityImporter Related to #5414753.

[dataimport] introduce the importer and extentity classes This introduces the ``ExtEntity`` class which is a transitional state between data at external source and the actual CubicWeb entities. ``ExtEntitiesImporter`` is then in charge to turn a bunch of ext entities into CW entities in repository, using a given store. This changeset also introduces ``SimpleImportLog`` and ``HTMLImportLog`` which implement the CW DataImportLog interface in order to show log messages in UI using simple text and HTML formats respectively, instead of storing these messages in database. Both have mostly been backported from cubes.skos.dataimport. Closes #5414753.

[test] Use store.prepare_insert_relation instead of deprecated relate method Follow-up for 1f5026e7d848. Related to #5040344.

[dataimport] since we have foreign key constraints in the database, 'entities' table should be updated first

[dataimport] Move stores to new API. Here is the final store API: * ``prepare_insert_entity(<entity type>, **kwargs) -> eid``: given an entity type, attributes and inlined relations, return the eid of the entity to be inserted, *with no guarantee that anything has been inserted in database*, * ``prepare_update_entity(<entity type>, eid, **kwargs) -> None``: given an entity type and eid, promise for update given attributes and inlined relations *with no guarantee that anything has been inserted in database*, * ``prepare_insert_relation(eid_from, rtype, eid_to) -> None``: indicate that a relation ``rtype`` should be added between entities with eids ``eid_from`` and ``eid_to``. Similarly to ``prepare_insert_entity()``, *there is no guarantee that the relation will be inserted in database*, * ``flush() -> None``: flush any temporary data to database. May be called several times during an import, * ``finish() -> None``: additional stuff to do after import is terminated. **Warning:** ``prepare_update_entity()`` still needs to be implemented for NoHookRQLObjectStore. Related to #5040344

[js] Using appropriate http method when calling a remote function to respect REST principles. closes #5355952

[server/test] Add a test for db-statement-timeout option Sadly I don't think there's a way to test that functionality without adding sleeps which make the test suite longer :/ Related to #2547026

[datafeed] give an error message if a source is missing a parser id Closes #3484231

[storage] use mkstemp to create files in bfss Avoids race condition, at the cost of slightly less predictable file names.

[test] Skip spa2rql tests when fyzz is not available Closes #5456750.

[web/staticcontrollers] fix docstring spelling

[server/rql2sql] fix spelling in comment Not that I understand what that sentence means.

[doc/book] fix rql syntax examples "Any", not "ANY". "EXISTS" is a function.

[migration] add sql constraints on upgrade Related to #5154406.

[schema] neuter check method for constraints we implement in sql Closes #5154406.

add IUserFriendlyError adapter for violation of check constraints This way we get back the same error messages we get from the python check. Related to #5154406

[hooks/syncschema] drop/add check constraints as appropriate Related to #5154406

[schema2sql] support NOW and TODAY in check constraints Related to #5154406.

[schema2sql] insert some constraints into the backend BoundaryConstraint, IntervalBoundConstraint and StaticVocabularyConstraint can be implemented in SQL with CHECK constraints. Next steps, not part of this changeset: - migrations - removing no longer necessary pre-insertion work on the python side - translating backend exception due to constraint violation into ValidationError Note: this means that these constraints can no longer be ignored by disabling integrity hooks. Related to #5154406

[test] run unittest_syncschema with postgresql Lets us test more things, since we can actually do ALTER TABLE.

[test] run unittest_migractions on postgresql sqlite's lack of support for a bunch of alter table functionality means some thing just don't get tested otherwise.

[devtools] Put pg sockets in /tmp Makes for shorter paths (limited by sockaddr_un). Also use the default port, since we're not listening on TCP sockets.

[devtools] change the way we start/stop postgresql Instead of having the test db handler start a cluster on demand, use the test module's setUp/tearDown callbacks to do it. This allows to have one data directory (and thus cluster) per test module, allowing different test modules to run in parallel, each using its own database cluster whose path is based on the test module.

[devtools] make test db name for postgresql process-specific In case we run two tests concurrently on the same cluster, better not use the same db name for both.

[devtools] extract functions to start/stop a postgresql cluster

[test] don't leave NULL columns around when making an attribute required It doesn't matter on sqlite (it doesn't do ALTER COLUMN), but when running this test on postgresql it fails to add the 'NOT NULL' constraint otherwise.

[server/session] Roll back in connection exit This was changed in commit 8b35a898b334 "[server] remove cnxset tracking, it is now unneeded". When exiting a CubicWeb Connection, the database connection is put back into the common pool. At that point anything done during the lifetime of the CubicWeb Connection needs to be committed or rolled back. Do a proper rollback to finish the transaction, as was happening before the above-mentionned commit. This showed up on testing with postgresql, when a DB schema modification was blocked by an old connection that was still "in transaction".

[web/views/formrenderer] do not use `cubicweb:target` attribute on form (closes #5534074) The `cubicweb:target` is a flag indicating that the form data (with file input) shall be posted inside an iframe - this is a well known "ajax-like" workaround to post files with browsers that do not support `FormData` (a.k.a IE<10). The `cubicweb:target` was introduced when CW used the "xhtml strict" doctype. Now that CW uses the "html5" doctype, this namespaced attribute is no longer necessary and the iframe can be generated directly. Before this patch, CW inserts the `cubicweb:target` attribute in the form DOM element (server side) and `setFormsTarget()` updates the DOM with a new `<iframe>` element (client side). Now, CW inserts the `<iframe>` DOM element directly (server side), making `setFormsTarget` useless.

[js] lint : make it explicit that typeof is an operator, not a function. Also, use === / !== except for null / undefined comparison

[devtools] qunit: show helpful data when an assertion fails Related to #5533333.

[web/test] Use shorter $() instead of jQuery() Consistency FTW. Related to #5533333.

[devtools] qunit: use new async testing APIs http://qunitjs.com/cookbook/#asynchronous-callbacks QUnit keeps track of all the assert.async() objects created inside the test functions and expects all done() functions to be called. Failure to do so will result in the test being failed. Unlike .start and .stop which were internal APIs, assert.async() is stricter and fails tests if assert methods are used *after* all done() functions are called (see "test callback execution order"). Related to #5533333.

[devtools] qunit: use new assert APIs test functions now accept an "assert" argument which exposes QUnit's assert methods: assert.{expect,equal,deepEqual,ok} Related to #5533333.

[devtools] qunit: stop using global variables test -> QUnit.test module -> QUnit.module Related to #5533333.

[devtools] qunit: update to 1.18.0 (closes #5533333) * the logging API has changed (register callbacks instead of setting attributes on the QUnit object) * the logging callbacks all have a single "details" object argument * the "main" div was renamed to "qunit-fixture" (this change is responsible for all the churn in our own JS test files) * QUnit can now build the DOM elements it needs for reporting from a single "qunit" div

[devtools] qunit: stop using deprecated assert functions equals -> equal same -> deepEqual Related to #5533333.

[devtools] qunit: remove useless window.close() Modern browsers don't allow it anymore. Related to #5533333.

[devtools] qunit: refactor functions into QUnitView Related to #5533333.

[devtools] qunit: only start firefox once per test The separate profile initialization step doesn't seem necessary anymore. This was due to the system-wide installation of the "Sage" Firefox extensions which breaks -url on a fresh profile. Removing this obnoxious extension allows us to launch Firefox in a standard way. Related to #5533333.

[devtools] qunit: decrease test timeout Related to #5533333.

[devtools] qunit: increase resolution and depth of Xvfb screen Related to #5533333.

[web/test] Always call QUnit.ok with a message Related to #5533333.

[web/test] Fix remaining JS tests (closes #5533303)

[web/test] Stop testing that exceptions raised in Deferred callbacks call errback They no longer do as of changeset 7f1f7f710b16. Related to #5533303.

[web/test] stop calling loadxhtml with a form.callback It was deprecated in 3.9 (changeset 7b9553a9db65) and removed in 3.18. Related to #5533303.

[web/test] Fix reset of loaded JS/CSS files Name changed in commit ede740bd7077. Related to #5533303.

[web/test] Use proper JSON syntax Related to #5533303.

[web/test] Update mock ajax reply from "render" ajaxfunc CubicWeb transforms add_js() calls in AJAX context to <cubicweb:script> tags since ede740bd7077 and cd5738fc440f. Related to #5533303.

[devtools] Serve JS tests over HTTP (closes #5533285) Instead of mixing file:// URLs with XHR.

[web/data] Support Deferred objects outside XHR JS tests actually use this. Related to #5533303.

merge 3.20.8 in 3.21

[views] Revert previous changeset 8f05aaabf355 Breaks cubicweb/web/test/test_views.py as some views don't have a 'vid' kwarg.

Added tag 3.20.8, debian/3.20.8-1, centos/3.20.8-1 for changeset ec284980ed9e

[pkg] Prepare 3.20.8

[js] simplify cw.log implementation There's no need to rebuild and array and join its content, just call console.log with ``arguments``. The only trick is that console.log must be called on a console instance.

[web/views] don't force http redirect on logout We no longer forbid anonymous browsing on https, so we can keep the same scheme when logged out. Closes #5507479.

[views/schema] extract a hardcoded css class attribute

[formwidgets] move TextArea constants to the class level This will make it easy to override them.

[server/test] fix broken test The first bad revision is: changeset: 10396:82071f767cb8 user: Sylvain Thénault <sylvain.thenault@logilab.fr> date: Thu Jun 11 17:35:11 2015 +0200 summary: [schemas] cwuri should be read-only This changed the number of RQLExpressions to be more than the number of CWRelations in the test's schema. Related to #5457566.

[test] typo in entities test comment

[cwconfig] typo in comment

[serverctl] allow changing the log level for source-sync command When calling this from cron we only want to see output if something fails.

[web, urlrewrite] remove unused `cachekey` argument from `rgx_action` (closes #5355967) this argument prevent from generating the resulting rset descripition

[web] Fix the path to the `static directory` (closes #5355965)

[web/json] Controller.publish must return encoded data We were mixing binary from super() and text from our params, thus wrongly returning text. Closes #5507573.

[entities] fix dc_title output for bool(value) == False Closes #5194853.

[hooks] manhandle the parsed rql query, not its string representation ... and add a groupby clause to make it actually work (the selection might well use an aggregate function, so we need to tell the backend to group on our target entity). Closes #5362574

merge 3.19.12 in 3.20

Added tag 3.19.12, debian/3.19.12-1, centos/3.19.12-1 for changeset 5932de3d50bf

[pkg] prepare 3.19.12

[web/views/formrenderer] remove `cubicweb:target` attribute on form once the target <iframe> is inserted For some unknown reason, IE9 does not want to submit the form inside the <iframe> when the ``cubicweb:target`` attribute exists. (closes #5377672)

[views] implement a custom json error view If the client asked for ``(e)jsonexport`` view, answer him with a json response rather than an error formatted in a big HTML blob. add a custom _requested_vid predicate. In cw 3.21, it will be deprecated in favor of ``match_form_params(vid=('jsonexport', 'ejsonexport'))``

don't override req.form['vid'] in the publisher's error_handler closes #5359234

when some inlined relation is set using cw_edited, its security shouldn't be checked. It's currently checked anyway because upon modification of `cw_edited`, `entity.skip_security` is updated to avoid security checking, but this is only considered for attributes, not relations. Closes #5477315

[test] stop depending on the email cube for hooks test Backport necessary part of its schema.

[schemas] cwuri should be read-only Closes #5457566

fix EditController._ordered_formparams method in case some mandatory relation isn't specified If you have an HTML form which doesn't carry information about some mandatory relation (maybe because this relation is set by a hook), you ends up with a KeyError when the form is posted. Using a get instead of subscription protocol fixes the problem. Closes #5481617.

[views] take into account subvid form param in SameETypeListView closes #5098473

[doc] fix errors in test_visibility_propagation example * forget to use the access instructions * wrong use of the function entity_from_eid * some indentation problems

i18n typo

[predicates] Use proper deprecation mechanisms

[doc] loadRemote does a POST by default

[doc] how to change logo using css

[doc] escape | operator so it will be visible in the docs

[doc] add missing import for FourOhFour class

[doc] the instance is sytweb_instance and sytweb is the cube

[server] Do not use progress bar when stdout is not a TTY Closes #5460552.

[wf] Fix Workflow.replace_state() Closes #5359131

[tags] add properly formated tags for 3.19 and 3.20 releases

[reledit] also check for cardinality before displaying the delete relation action. Closes #5334750

[schema] add a unique index on cwuri Closes #4985752

[web] move AbstractAuthManager near its immediate concrete subclass Related to #1381328.

[web] move abstract session manager to web/sessions Related to #1381328.

[repository] rename session -> cnx Related to #3933480.

[server/test] delete commented-out test Related to #3933480.

[connection] remove the `mode` attribute It is now unused. While removing mode, we also drop some sqlserver-specific reconnection logic snippets. These had several downsides: * untested * partial coverage * done at the wrong level Related to #2919309.

[connection] remove ensure_cnx_set context manager uses It has been deprecated in a previous changeset and is now a no-op. Related to #2919309.

[server] remove cnxset tracking, it is now unneeded Indeed, all cnxsets will now be acquired using the following API:: with session.new_cnx() as cnx: cnx.execute(...) # do stuff cnx.commit() These well-scoped blocks are the only place a cnxset can be acquired. The old use-case for un-delimited cnxsets (pyro/zmqpickle protocols) have been removed. The "mode" of connection objects becomes "write" forever (it will be removed in a couple of changesets). Related to #2919309.

[devtools] remove the remaining bw compat for old-style tests It is just: * dangerously untested right now * on the path of further bw compat removal (cnxset handling) webtest drops an apparently pointless .login call Related to #3933480.

[session] remove session-related bw compat code Related to #3933480.

[session/transaction] removed unused transaction class Related to #3933480.

[transactions] cleanup after the dbapi removal * end req vs cnx dichotomy * Transaction takes a cnx at __init__ time * a couple of super calls are fixed Related to #3933480.

[session/cnx] remove `session_handled` logic Related to #3837233 .

[web/sessions] reduce repoapi usage where possible Using session.new_cnx() should be the new preferred way. Related to #3837233.

[entity,storage] remove hackish code from the bfss path The ClientConnection/Connection duality brought some painful hacks that are no longer needed. Hence we can send entity._cw_update_attr_cache and ._cw_dont_cache_attribute to the grave. Related to #3837233

[connection] eliminate ClientConnection leftovers * in the documentation * in the docstrings * in code notations (e.g. clt_cnx -> cnx) Related to #3837233. From now, there should be just one ClientConnection left in repoapi.py.

[session] all cnx._session become cnx.session The _session belonged to ClientConnection, which is just gone. Related to #3837233.

[repoapi] fold ClientConnection into Connection Connection replaces ClientConnection everywhere. Some notes: * testlib: .client_cnx and .repo_cnx become aliases of .cnx (we might not want to tell people to update their tests again for just no real benefit, so we'll live with these aliases for a while) * entity.as_rset must not be cached because we risk caching result sets having a Connection object as .req (helps unittest_breadcrumbs) * entity._cw_dont_cache_attributes loses its repo/request special paths and only keeps its storage/bfss user (this helps unittest_wfobjs) * moreover, entity.cw_instantiate and .cw_set stop overriding the attributes cache *after* the before_*_entity hooks have run, because there is no need to (it is now actually harmful to do it and unittest_hooks.test_html_tidy* tests remain green because of this) * rset._build_entity sticks its .req onto the entity just fetched from the cache, because otherwise it might carry a _cw that is a Connection object where a Request is expected (helps unittest_views_actions) * we get overall better cache usages (entity caches were split over Request + ClientConnection and Connection), hence the changes unittest_entity and unittest_wfobjs * void the ecache when providing the cnx to a request object Having the entity cache pre-filled when we bind it to the request object hurts because these entities are bound to Connection objects, that lack e.g. `.form` or `.add_js` and crash the views subsystem. Thus, the unittest_testlib.test_error_raised test will are kept green. Closes #3837233

[migractions] remove any session related leftovers Also, the migration handler really wants a "repo" session anyway. This highlights a bug in unittest_wfobjs.test_duplicated_transition where an old connection is reused. It probably used to work because it was made of session + set_cnxset magic dust. Related to #3933480.

[session] provide missing apis to Connection From this point, Connection provides a superset of ClientConnection. Related to #3837233.

[connection] replace .running_dbapi_query with .hooks_in_progress The thing was badly named. It tries to help distinguish between queries issued directly by the programmer (e.g in the views: cnx.execute(...)) from queries issued from the hooks, operations ... or even the repository or the native source objects. It worked heuristically being associated with the security being disabled. We provide a better name and an implementation distinct from the security management methods. Related to #3933480.

[dataimport] Turn the module into a package Just `hg mv` dataimport.py and test/unittest_dataimport.py.

[dataimport] Properly escape strings sent to COPY FROM (closes #5278743) See http://www.postgresql.org/docs/9.1/static/sql-copy.html#AEN64296 for escaping codes.

[http] fix cache-control syntax in EtagHTTPCacheManager use , to separate cache response directive closes #5307539

[connection] provide some missing documentation bits The previous changesets remove information about session data and transaction data. We provide the necessary entries in the docstring of the Connection class. Related to #3933480.

[repoapi,session] remove all session-as-cnx backward compat The `dbapi` being gone, we now can drop the session object bw-compatibility layer. This will allow further simplifications, such as folding ClientConnection and Connection (without too much pain), and then having persistent sessions. Related to #3933480.

[server] drop repo.internal_session and InternalSession

Added tag cubicweb-version-3.20.7, cubicweb-debian-version-3.20.7-1, cubicweb-centos-version-3.20.7-1 for changeset 359d68bc1260

[pkg] 3.20.7

[dbapi] retire repo.[gs]et_shared_data which were used by the dbapi Related to #3933480

[dbapi] retire repository transaction methods which were used by the dbapi Related to #3933480.

[dbapi] retire repo.call_service which was used by the dbapi Related to #3933480.

[dbapi] retire repo.commit/rollback which was used by the dbapi Related to #3933480.

[dbapi] retire repo.user_info which was used by the dbapi Related to #3933480.

[dbapi] retire repo.entity_metas which was used by the dbapi Related to #3933480.

[dbapi] retire repo.execute, which was used by the dbapi We will retire or reform pieces, mostly of the Repository class, that were related to the old dbapi way of life. Related to #3933480.

[devtools/testlib] remove dbapi backward-compat Related to #3933480.

[test] fix copy-pasta in doc string

[book] remove dbapi documentation Move some still-relevant doc related to the RQL connection API to the 'sessions' section. It might make sense to move this back outside of the "Repository development" chapter at some point since this is relevant in other contexts.

[uilib] `uilib.js` helper now honors explicit JSString (closes #4959538) Previously, if a JSString was inside a nested container (i.e. list or dict), it was handled as a string, not a verbatim js chunk (see docstring and test). So the following code:: >>> str(js.cw.pouet(1, {'callback': JSString('cw.mycallback')}) resulted in:: "cw.pouet(1, {'callback': 'cw.mycallback')})" while we expect (note the removed quotes):: "cw.pouet(1, {'callback': cw.mycallback)})" We use ``cubiweb.utils.js_dumps`` instead of ``cubicweb.utils.json_dumps`` which is aware of JSString.

[dbapi] remove the dbapi module and its immediate remaining users We suppress toolsutils.config_connect, which has currently only one known user (the email cube), which is being patched. It can be replaced with utils.admincnx function for a local access. Next will come a series to: * remove the session backward compatibility * fold ClientConnection into Connection Closes #3933480.

[web/views] remove dbapi usage from 'gc' view

[sqlite] Fix integrity error parsing Up to sqlite 3.7, integrity error message may be in the singular form if only one column is in the unique index. Closes #5224632

[postgres] fix limit_size regexp to properly remove one char tags like <p>. Closes #5221847

[postgres] _regproc.postgres.sql should be reloadable

[mail] include Date header in generated mails (closes #5271058) Mails without Date gets flagged as spam. Plus it's wrong anyway.

[doc] mention datadir-url configuration option in release notes Related to #5204550

[web] add test for datadir-url option (related to #5204550)

[web] Add datadir-url configuration option (closes #5204550) Makes it possible to use a different host for static data instead of ${base_url}/data

[web] kill https-deny-anonymous option (closes #5193687) It doesn't make much sense to prevent people from using https.

[cwctl] don't filter DeprecationWarnings

[exceptions] avoid calling UniqueExceptionError.rtypes after closing the session

[web/test] Fix broken tests after commit b81adb5e32f9 Add missing n3rdf view to RDF view list.

[web/test] don't delete uicache directory Other tests use it, so the rmtree was a bit too much. Set CACHEDIR to a temporary subdirectory, and only delete that.

[web/views] add text/n3 rdf view Less shit than xml.

[schemas] make CWEType.final default to False (closes #5049201)

[repository] replace session with cnx in a few places

Don't pass an encoding to rqlst.as_string()

[rset] Deprecate the 'encoded' argument to ResultSet.printable_rql() It's hardly used at all, and when used inside CubicWeb, it's only set as false. Better make sure that this function always returns a proper unicode string and let the caller handle conversion on its own.

[gcdebug] ignore _NeedAuthAccessMock instances When looping over gc.get_objects(), we'd crash in _NeedAuthAccessMock.__getattribute__.

[views] Fix 'gc' view to use 'repo_gc_stats' service

[services] Fix 'repo_gc_stats' to return a list of unreachable objects' repr

[services] Prevent 'repo_stats' service from aggregating information It's up to the consumer of the service to decide whether it wants aggregated data or not... Ensure the /siteinfo page rendering is not modified.

[hooks] add a looping task that dumps the stats regularly in a file

[web/facet] Use an Exists node in HasRelationFacet The generated RQL now includes an EXISTS clause. For instance, one gets: :: DISTINCT Any WHERE X is CWUser, EXISTS(X in_group A) instead of: :: DISTINCT Any WHERE X is CWUser, X in_group A for a HasRelationFacet with ``rtype = 'in_group'``, which is more appropriate for testing the existence of the relation. Add a test for this facet along the way.

[debian] Add a Breaks for cubicweb-folder which stills use GMTOFFSET Related to #2154655.

[web] kill GMTOFFSET (closes #2154655) Its last user, set_cookie(expires=...), can just as well interpret its argument as UTC. Naïve datetime objects with an implicit non-UTC timezone are just awful.

[web/request] don't play tricks with utc offset We need an UTC datetime from a time.time()-type value. It turns out datetime provides a method to do just that. Related to #2154655.

[server/test] Drop dependency on folder cube Just copy schema bits when needed and drop a view (not possible anymore) in unittest_repository. Closes #5168939.

[web/test] Drop folder cube dependency Copy useful schema bits, define IBreadCrumbs adapter for Folder/fild_under (previously relying on ITree adapter) and adjust unittest_viewselector tests to drop a view that does not exist anymore. Related to #5168939.

merge 3.20.6 into 3.21

[web/views] Allow changing CSVView's separator (closes #5227442)

[web/view/treeview] add a 'selected' class on <li> if the entity's url is the currently displayed url An alternative would be to refactor the corresponding view so that controlling css classes wouldn't require overriding a 5O-lines method. Notice by default there is no associated class in the CSS so there is no rendering difference unless one explicitly add rule in his CSS file.

Added tag cubicweb-version-3.20.6, cubicweb-debian-version-3.20.6-1, cubicweb-centos-version-3.20.6-1 for changeset 7f64859dcbcd

[pkg] 3.20.6

[notification] don't block while sending mails Regression from #7c17659c9eae, closes #5190001.

[dataimport] make MetaDataGenerator / NoHookObjectStore usable from a datafeed parser ie make it possible to create entities with external source as cw_source and proper external id. Closes #4891552

[dataimport] don't generate metadata which are explicitly specified Related to #4891552

[debian] prepare 3.20.5-2

merge 3.19.11-2 into 3.20

Added tag cubicweb-debian-version-3.19.11-2 for changeset 6d265ea7d56f

[debian] Update watch file

[debian] Fix cubicweb-dev dependency

Added tag cubicweb-version-3.20.5, cubicweb-debian-version-3.20.5-1, cubicweb-centos-version-3.20.5-1 for changeset 51aa56e7d507

[pkg] 3.20.5

[dataimport] don't insert created_by / owned_by relations when user is the internal manager (eid = -1). Closes #5162943

[repository] don't attempt to delete computed relation, they have no table in the database. Closes #5162935

[migration] sync_schema_props_perms should skip computed relations. Closes #5147796

[schema] typo

[web/test] fix test_views reusing a request for two different view calls leads to badness

[documentation] add note concerning Attribute, TODAY and NOW in yams documentation

[i18n] properly translate error messages related to violated unicity constraint. Closes #5100373

[web/data] Add missing mode='swap' in loadxhtml calls (closes #5087432) Let's avoid putting divs inside divs, dawg.

[web/views] Fix PropertyKeyField widget (closes #5087413) CWProperties don't have unittests, this will likely break again.

[web/views] Fix exception in CWProperty edit form (closes #5087412) Also add missing help message. The overall display is ugly, but at least the message is shown.

[web/request] Restore attribute setting for search_state This was dropped in 1f84295bfe95 when `search_state` attribute was turned into a property, but without a setter. So adding a setter here restoring previous behaviour, as some tests and cubes use it. Related to #4875761.

[web/request] simplify search_state loading, fix comment

[webrequest] Lazy-load search_state - The update_search_state function will hit the session data only if __mode is present in the form. - set_search_state keep the same semantics - Direct access to search_state in the session data is changed by an access to the new property. Closes #4875761

[config] disable fckeditor by default (closes #1368900) FCKEditor is no longer maintained, and has a history of XSS issues. Let's use plain text as default text format.

merge 3.19.11 into 3.20

Added tag cubicweb-version-3.19.11, cubicweb-debian-version-3.19.11-1, cubicweb-centos-version-3.19.11-1 for changeset 1ae64186af94

[migration] reorder system tables / meta-schema migration into bootstrap_migration Also fix a bug in the backwards-compat eid_type_source method where it returned an extid (possibly null) instead of a source name. With these changes, migrating a database from cw 3.12 is now possible. Closes #4962681

[dataimport] add missing import (closes #4985916)

[debian] cubicweb-dev needs cubicweb-etwist Closes #4965410

[pkg] 3.19.11

Binary initializer should accept bytearrays (closes #5084075) Recent versions of pyodbc return bytearray objects for binary data.

[native source] log the used connection in doexecmany as in doexec

[server] use unicode wherever appropriate in transaction/undo related methods Silences lgdb/sqlite warnings about using str instead of unicode.

[web] typo

[entities] return the entity itself from notification_references if the view uses a random message-id Useful for entity update notifications to reference the entity creation message.

[notifications] use non-fixed message-id for EntityUpdatedNotificationView More than one of those can be sent for a given entity, so they shouldn't all have the same message-id. Closes #4962712.

[server/test] kill remaining 3.19 deprecation warnings in undo tests

[devtools] accept str objects in CubicWebTC.new_access and .create_user

[utils] use the "predictable" argument to js_dumps Fixes doctest under python -R. Related to #4959402.

[server/test] don't rely on a sorted output from .object_relations() Fixes test under python -R. Related to #4959402.

[req] make rebuild_url predictable Fixes test with python -R. Related to #4959402.

[web/test] don't depend on iteration order on a set Fixes test under python -R ('pending_insert' in session data is a set, so we can't rely on any particular ordering). Related to #4959402.

[web/test] fix cubicweb 3.19 deprecation warning

[web/test] use unicode instead of str for user logins makes lgdb/sqlite quieter.

[test] Don't depend on dict iteration ordering Fixes test with python -R. Related to #4959402.

[entities] make cw_fti_index_rql_queries predictable Don't depend on the hash function, fixing test with python -R. Related to #4959402.

[hooks/syncschema] use a list instead of a set for UpdateFTIndexOp The order in which etypes get indexed matters. Related to #4959402.

[rqlrewrite] sort possible types when turning is_instance_of into is Helps get predictable output even with python -R. Related to #4959402.

[server/test] use unicode strings for user logins

[web/test] use unicode object as user login (py.warnings) WARNING: /home/jcr/src/cw/cubicweb/devtools/testlib.py:206: DeprecationWarning: Sanitizing an input dictionary with str values, please check your data (key = 'login')

[server/querier] stop passing a rqlst to empty_rset (py.warnings) WARNING: /home/jcr/src/cw/cubicweb/server/querier.py:46: DeprecationWarning: [3.20] rqlst parameter is deprecated return ResultSet([], rql, args, rqlst=rqlst)

[web/views] Use predictable ordering for attributes in entity XML view Fixes test under python -R. Related to #4959402.

[web/views] set explicit order for ManageSourcesAction Gives predictable ordering wrt ManageUsersAction under python -R. Related to #4959402.

[web/test] don't depend on dict iteration ordering Fixes tests under python -R. Related to #4959402.

[web/test] pass login as unicode string silences lgdb/sqlite warning.

[web/test] fix 3.19 deprecation warning

[sobjects/test] Fix dependency on predictable hashing Fix test under python -R by ignoring url query parameter ordering. Related to #4959402.

[test] don't rely on a stable ordering for appobject_path Fixes test under python -R. Related to #4959402.

[server] remove "standalone RQL server" from possible configuration choices Related to #2919297.

Remove leftover bits from start-repository command, and document its demise Related to #2919297

Remove the remote repository-access-through-zmq support Modern methods such as the rqlcontroller cube + the cwclientlib library are the way forward. Closes #2919297.

Remove remote repository-access-through-pyro support Modern methods such as the rqlcontroller cube + the cwclientlib library are the way forward. Closes #2919309.

[web/views/rdf] Take the second element of XY equivalent for non final relation When given a Yams snippet of the form `<etype> <rtype>` ``xy.xeq()`` will return the XML snippet equivalent to this relation in the form of a tuple `(subject, rtype, object)` so ``item`` will always be a tuple here. In fact, the correct code appears just a few lines above (around line 88) for final relations so I guess this is a copy-paste mistake. Closes #4745929.

[server] Make sure 'zmq-address-sub' config is always iterable

[web] split timeline in an individual cube Closes #1279544

[devtools] don't hide warnings and errors during tests The markdown module in jessie turns all warnings into logging calls, so this allows me to still see deprecation warnings when running tests.

[test/i18n] remove unneeded hack

merge 3.19.10 into 3.20

[server/session] Fix typo "meta vs. metas" in describe()

Added tag cubicweb-version-3.19.10, cubicweb-debian-version-3.19.10-1, cubicweb-centos-version-3.19.10-1 for changeset 3bab0b9b0ee7

[pkg] 3.19.10

[i18n] Update translations Related to #5012581.

[config] Lowercase the FQDN we get from the OS (closes #5040345) Browsers convert the host name part of URLs to lowercase. This has the unfortunate effect of triggering various CORS error messages in CubicWeb (because the Origin: header will contain lowercase letters). As per RFC 4343, host name comparison should be case-insensitive. So let's put it in lowercase when we grab it from the host system. If admins put uppercase letters in "host" or "base-url" in their all-in-one.conf, then it's their fault.

[web/views] Honor 'action' attribute on AutomaticEntityForm, closes #4943392 On AutomaticEntityForms, get_action used to bypass the logic from FieldsForm.form_action() which handles the 'action' attribute itself, preventing callers from overriding it at selection time. This capability is used by the inlinedit cube. In a nutshell, we let the deprecated AutomaticEntityForm.get_action() raise an AttributeError, which allows FieldsForm.form_action() to handle it properly.

[docstrings] fix project-wide English language mistake Even though 'u' is a wovel, it may be pronounced 'yu' which is a consonant sound. In such cases precede it with 'a' instead of 'an'.

[doc] Don't pretend we know cubicweb runs on mysql As far as I know it hasn't been tested in years.

[test/i18n] Update reference po file following 75cd7df5c32e This changeset added a (missing) message "add a <etype>", so reference po file needed an update.

[devtools] add a method on CubicWebTC to fake data posted by standard cw forms. Closes #4985805

[web/wdoc] fix links They're relative to ${base-url}/doc/, not ${base-url}. Closes #3161099.

[i18n generation] add missing 'add a EntityType' message to generated catalogs. Closes #5012581

[server/test] fix test_zmq We were only asserting in a thread, thus ignoring the results.

[views,js] remove user_callback feature Related to #3975260.

[newcube] Make the generated code PEP-8 compliant Closes #4925396

Remove unused imports

[ext] Remove use of cStringIO We create a StringIO never to read anything from it. Docutils seems to support setting the 'warning_stream' to False to ignore warnings. So let's just do that.

[view] Use io.BytesIO as stream for binary views

[undo/test] check that the entity was restored even though the fiche relation wasn't

Use a moved_entities table to record external entities moved to the system source Instead of using a negative eid in the entities table, move the record to a new table so we don't have an interval with a missing eid in entities. Related to #4846892

Add unique index on entities.extid

[undo/test] if we try to delete an entity referenced by an inline relation, we fail

[undo] when we can't undo an inlined relation change, point at None, not a broken eid This whole thing is "a bit" weird.

Ask sqlite to check foreign keys Related to #4846892

Add reference from etype table's eid column to the entities table Related to #4846892.

[server] change order of entities table modification vs entity creation/deletion In order to have a foreign key referencing the entities table, the insertion into entities must happen first, and the deletion must happen last. For the deletion case, this means we need to: 1. delete all relations (cascade) 2. delete the entities themselves 3. delete the corresponding lines from the entities table This means the _delete_info{,_multi} methods can't keep doing 1 and 3. Thankfully the "public" delete_info method appears to be unused, so drop it. Related to #4846892.

Use foreign keys for relations tables Closes #4846892

Add foreign key for inline relations Related to #4846892

Use our version of schema2sql Related to #4846892

Steal schema2sql module from yams It has no user inside yams, and we need it here if we want to be able to reference the (cubicweb-only) entities table from other tables. Related to #4846892

[dataimport] fix pylint detected error in SQLGenObjectStore

[restore/portable] properly restore the numrange generator If the dump format version is 1.1, read and restore numranges. Closes #4870338

[backup/portable] properly write the eid numrange generator (the eid is no more a sequence) Bump format version to 1.1. Related to #4870338

Added tag cubicweb-version-3.20.4, cubicweb-debian-version-3.20.4-1, cubicweb-centos-version-3.20.4-1 for changeset 49831fdc84dc

[pkg] 3.20.4

[schema] set permissions that do not allow edition on computed relation. Closes #4903918 Also, drop 'del schema' from test which clutters the code for no benefit.

[computed attribute] ensure attribute's formula apply only to the correct type. Closes #4901163

[schemaserial] fix detection of CWComputedRType table, closes #4901150

[dataimport] source.add_info doesn't take anymore a 'complete' argument Closes #4891550

[dataimport] massive renaming of session to cnx and make NoHookObjectStore working with the new connection handling. Closes #4891547

[web/data] use the right selector for hiding jquery treeview placeholders Closes #4903927

merge 3.19.9 into 3.20 branch

Added tag cubicweb-version-3.19.9, cubicweb-debian-version-3.19.9-1, cubicweb-centos-version-3.19.9-1 for changeset b7c373d74754

[pkg] 3.19.9

[hooks] don't insert use_email relation if the entity on either side is going away Closes #4912946

[undo] fix deletion of cw_source relation Closes #4912882

[hooks] don't insert an owned_by relation for deleted entities If, in the same transaction, a composed entity is created and deleted, we'll have scheduled a SyncOwnersOp which would end up creating an owned_by relation for the deleted entity. Closes #4846883

[etwist/service] adjust for a signature change Closes #4878570.

[rset] fix crash on displaying rset where some symmetric relation is used. Closes #4739253 Avoid a KeyError.

Fix handling of limit within the primary view for related elements. Closes #4723579 * respect limit set by uicfg (like the 'autolimited' view itself does) * set limit in rql query to one more than the specified one so the view knows there are more things to display

[native] Change SQL errors log level. - The SQL error itself is logged as 'info'. - The 'transaction has been rolled back' message is logged as 'debug'. These log entries should not be critical: - If an error occur at the sql level, it will be transmitted as a CW error. The exception is raised and handled in a higher level function, which will decide if it is critical or not. - Admitting that rolling-back automatically on errors is wanted in some cases, it cannot be considered 'critical'. The caller will receive the original exception anyway, it is its job to decide if it is critical or not. Closes #4801120

Added tag cubicweb-version-3.20.3, cubicweb-debian-version-3.20.3-1, cubicweb-centos-version-3.20.3-1 for changeset 7d3a583ed539

[debian] build-depend on markdown for documentation build

[devtools] Make PostgresTestDataBaseHandler multi-use friendly Dont init the pgdb twice on the same datadir - If two test case with the same apphome use a Postgres configuration, the handler is initialised twice, so it has to check if the pgdb directory has already been initialized. - Work with the realpath of the pgdb In some cases, the self.config.apphome will resolve symbolic links, but not always. It can result in an attempt to start twice the pg server for the same directory, in the same test run... resulting in failure. Closes #4875827

[devtools] restore i18n of string removed in 4001cfe2f44d Removing that string from the po files means that after running i18ncube against newer cubicweb, we lose the translation when running on older cubicweb's, which is probably not a good idea.

[pkg] fix setup.py install One day I'll get this thing right. Maybe.

[pkg] 3.20.3

[server] Increase the stacklevel (due to @_open_only decorator)

[pkg] install jquery-treeview

[migration/3.20] sync rtypes before etypes We need to make sure the rtypes are inlined before adding unique indices.

[doc] Fix class name for ComputedRelation; fix code block about computed attributes

[web] Allow propagating 'klass' parameter on tree views recursively Closes #4881302

[utils] Fix define_var in ajax context (closes #4881301) Use namespaced <cubicweb:script> instead of just <script>

[web/data] Never hide JS errors in our Deferred implementation Modern browsers have good debugging tools, let exceptions go all the way up. This only affects exceptions raised in the js code, if there's an issue on the server/python side, they'll keep going through our error callbacks. Closes #4881300

[web/data] Fix userCallbackThenUpdateUI to completely replace old DOM elements Without the 'swap' parameter, userCallbackThenUpdateUI() would replace the _children_ of the component with a new version of the componenent, thus creating nested divs/spans/buttons/etc. Closes #4881299

[web/views] Stop mishandling the fieldset name in the default form renderer While not really documented, it is generally assumed that: * the fieldset is a user-visible string (much like a title), * it will be translated by whichever piece of code uses it. So: * fix AutomaticEntityForm to generate untranslated fieldset names, * fix FormRenderer to properly translate the fieldset name before rendering it, * stop using the field set (which may contain whitespace) as a CSS class name (the feature was unusable, so just drop it), * properly handle the colon as part of the translated fieldset name Closes #4881298

Added tag cubicweb-version-3.20.2, cubicweb-debian-version-3.20.2-1, cubicweb-centos-version-3.20.2-1 for changeset 138464fc1c33

merge 3.19.8 into 3.20 branch

[pkg] fix mistake in c7b218125c25 We ended up with the wrong filenames

Added tag cubicweb-version-3.19.8, cubicweb-debian-version-3.19.8-1, cubicweb-centos-version-3.19.8-1 for changeset efc8645ece43

[server/test] convert new test to 3.19 API

[pkg] 3.19.9

merge 3.18.8 into 3.19 branch

Added tag cubicweb-version-3.18.8, cubicweb-debian-version-3.18.8-1, cubicweb-centos-version-3.18.8-1 for changeset 231094063d62

[pkg] 3.18.8

[security] Test case and fix for an INSERT security hole 7099bbd685aa introduced a untested corner case in which an Entity with no attribute specified could be created whatever the permissions. Report and test case by Christophe de Vienne, fix by Aurelien Campeas. Thanks! Closes #4854359

[pkg] 3.20.2

[debian] cubicweb 3.20 breaks cubicweb-bootstrap << 0.6.6 Closes #4842316

[pkg] don't include extra files in tarballs Blind wildcards for dirs that may contain generated or temporary files can lead to trouble and unreproducible results.

[pkg] exclude generated js_api documentation from tarballs

[doc] remove old and unused argouml files

[web/data] remove unused xcf files

[doc] remove nonexistent css from theme

[doc] Drop autoclass directive for IDownloadableLineView This class has been dropped in 3.20.

[pkg] include missing files from doc/book/ in release tarballs Closes #4832700

[datafeed] Set headers parameter to an empty dict in DataFeedParser.retrieve_url This parameter is then passed to urllib2.Request, which expects a dict and won't check for None. Closes #4842333.

[views/forms] Fix EntityFieldsForm.link_to when parent entity is being created When the parent entity is being created and the inlined form is displayed, the parent entity has no eid yet and `peid` is usually 'A'. So int() raises a ValueError. Closes #4627739.

[entities] Optimize CWUser.properties. This property gets called each time a user connects to a repository, which means a lot. This optimisation cuts its execution time by 5 according to my tests

[schema] Typo and indentation fix in RQLVocabularyConstraint doc string

Added tag cubicweb-version-3.20.1, cubicweb-debian-version-3.20.1-1, cubicweb-centos-version-3.20.1-1 for changeset 43eef610ef11

[pkg] 3.20.1

i18n update (closes #4826490)

merge 3.19.7 into 3.20

merge two default heads

Added tag cubicweb-version-3.19.7, cubicweb-debian-version-3.19.7-1, cubicweb-centos-version-3.19.7-1 for changeset ac4f5f615597

[pkg] 3.19.7

Added tag cubicweb-version-3.20.0, cubicweb-debian-version-3.20.0-1, cubicweb-centos-version-3.20.0-1 for changeset 7e6b7739afe6

[pkg] prepare 3.20 release

[skeleton] simplify debian rules file

[migration] don't crash when deleting a non-existing rtype When asked to delete a relation type, instead of checking if it was computed or not, just delete both the CWRType and CWComputedRType.

[debian] include "metapackage" in the cubicweb package's description Lets lintian know it's ok for it to be empty.

[debian] ensure all packages use ${misc:Depends} and ${python:Depends}

[debian] update cubicweb-dev's lintian override Switching from python-support to dh-python changed the installation path.

[server] add a db-namespace option in source definition (closes #1631339) "namespace" is preferred to the postgresql term "schema" to try and avoid confusion with cubicweb's schema. cubicweb now depends on logilab-database >= 1.13.0

[serverctl] add db_transaction context manager

[server] refactor cleanup() code used by "cubicweb-ctl delete" Try to make each logical step (i.e. "delete database" and "delete user") a bit more visible and explicit. This is a preparatory work to easily plug the "delete namespace" feature.

[views] visual refactorings (closes #3803685) - pageContent: - remove the shadow - add a light border - slightly round the corners - left column: - ensure width make it aligned with the logo (header) - ensure the top border of the first box is aligned with the pageContent div - do not use serif font for box headers

[web] everything old is new again Kill the "new" cubicweb.css, and move cubicweb.old.css back in its place. Closes #4763360.

[pkg] bump version to 3.20.0

[views] use icons for user menu and login (header) (closes #3803684) - use icons from the embedded pictograms font for: - authenticated user menu (userActionsBox) - login (CookieLoginComponent) - do not display "anonymous" when not logged in

[utils] provide a function to return an admin connection from an appid Closes #4723580.

[debian] Install wsgi directory in cubicweb-web package Closes #4752375.

[test] fixes to make tests pass with recent lxml lxml (on jessie at least, ie. 3.4.0) does not remove endlines anymore.

[server] Remove useless 'is None' checks 'pb' is never none since 3386fd89c914 when support for the 'APYCOT_ROOT' env var was removed.

[security] check attributes: dispatch on the "add" action if entity was just created cw_set on a just-created entity (i.e. created in the same transaction) should behave the same as setting the attribute directly on creation: check the 'add' permissions, not 'update'. Closes #4740310.

[doc] Add BigInt to the list of built-in types

[devtools] call turn_repo_off in tearDown (closes #4673491) lets it catch leaked sessions even when running a single test

[book] typos / clarifications in create-instance section Clarify that cubicweb-ctl upgrade needs to be run whenever cubicweb or a cube is upgraded. Related to #2632425.

[repository] don't mangle the stack trace on exception

[pkg] Make twisted recommended only It's not necessary when using wsgi or pyramid. Closes #4639508

[devtools] Use the pause_trace context manager instead of pause_tracing/resume_tracing The later were deprecated in logilab-common 0.63.1

[serverctl] rename remove_cube to drop_cube (closes #4545093) For the sake of consistency, since commands are currently named: - add_{cube,entity,attribute,relation} - drop_{entity,attribute,relation} - remove_cube

[wsgi] add tornado http server

[wsgi] add message about the active interface Similarly to what done in "stdlib".

[cwctl] rewrite wsgi method choices This will help devs to add new choices by adding their callback to WSGI_CHOICES. Note: ``options`` becomes a property because ``wsgichoices()`` should be evaluated at the last-minute.

[skeleton/debian] use dh_python2 instead of dh_pysupport for new cubes The former is deprecated.

[web] Add source for jquery.qtip.js Downloaded from https://github.com/Craga89/qTip1/raw/master/1.0.0-rc3/jquery.qtip-1.0.0-rc3.js Related to #3851121.

[web] Replace minified copy of jquery.flot.js 0.6 with non-minified version Related to #3851121

[web] Update jquery-treeview to the latest version Files copied from commit 61f230828ace1b54c2cc54bc549ece261b11842e of https://github.com/jzaefferer/jquery-treeview

[web/css] move jquery.treeview.css override to a separate file Patched embedded code copies are a maintenance disaster (even more so than plain embedded code copies). Let's not do that.

[web] Stop patching jquery.treeview.js This logically reverts part of e9b7cd2e9012 "allow treeview to work correctly in a tab #345293". The treeview and ajax code have suffered a number of changes since then, this change doesn't seem to be necessary anymore. It is most likely unneeded since f65208c9dbdc "[javascript] use jQuery.one('ajax-loaded') instead of jQuery.bind() in add_onload to avoid multiple callback executions".

[web] Beautify jquery.treeview.js Fully mechanical, just whitespace changes.

[entities/wfobjs] add missing `DBG_SEC` debugging informations and a new `transition` capability

[warnings] put an end to warnings in the sqlite driver over `str` being sent instead of unicode strings

[web/request] clearly mark user_callback-related methods as deprecated The deprecated decorator not only emits a warning at run-time but above all makes the deprecation immediately apparent in the source. closes #3567793.

[book] document __unique_together__, remove bad RQLUniqueConstraint example RQLUniqueConstraint should be avoided whenever possible. Related to #3753250.

[schema] stop using RQLUniqueConstraint (closes #3753250) The last uses are replaced with unique together constraints.

[dataimport] Have ucsvreader's API match that of csv.reader (closes #3705701) 'separator' and 'quote' are replaced with 'delimiter' and 'quotechar'.

[utils] Add a '_cwtracehtml' GET parameter to trace self._cw.w() calls (closes #4601327) The core of this patch is in UStringIO.write(). When tracing is enabled, write() doesn't just append the 'value' argument to the underlying list. Instead, a stack trace is recorded and a special HTML "source" is formatted. The output with tracing enabled is an HTML page, with the original HTML escaped, and made clickable to show the stack trace when the write() call was done. This allows answering the recurring question: "who wrote this tag here?!"

[datafeed] Add a raise_on_error parameter to DataFeedSource.extid2entity And pass the option from various `process_` methods in existing parsers. This makes debugging in tests easier. Closes #4601191.

[test/view forms] use the official, undeprecated API

[rset] kill the rset._rqlst cache Right now it "works" for the standard, internal uses. However when we will fold ClientConnection and Connection, it will hurt, because suddenly we get more cache hits, and the following situation would become commonplace: * there is an un-annotated _rqlst given by the querier * some view (e.g. facets) requests the .syntax_tree, which takes a copy of _rqlst * the view actually expects the rql syntax tree to be annotated, but it was not, hence we crash. Related to #3837233.

[source/native] session -> cnx Also swap process_results arguments order to make cnx come earlier.

[hookstests] change got/expected order

[config] kill a getattr

[tests/web] switch previous commit (136b5f995f8) to the new test api Related to #3670209.

[web] remove unused argument from treeview._init_headers

[doc/3.20] mention CWEP-002

[web] stop accepting the magic __message form parameter This has been deprecated for a while, and replaced by _cwmsgid, which doesn't allow arbitrary content.

[web/request] add security_enabled method Just forward to the underlying repo connection. This can be useful in the ORM when dealing with a "code-only" attribute, i.e. something that shouldn't be directly user-visible but must be accessible from a web request.

[dataimport] Fix use of _create_copyfrom_buffer() (related to #3845572) This is used indirectly by SQLGenObjectStore when running on a PostgreSQL database. Regression introduced by commit 70056633085c.

[c-c configure] make it possible to specify the section for sources configuration (closes #3477678)

[server] fix 'cnx' variable confusion in DBG_SQL exception case The rollback handling expects 'cnx' to be the cubicweb Connection, but the DBG_SQL block was replacing it with an sql connection, leading to lulz down the line. Also remove obsolete getattr (the sqlite wrapping is now done at the cnxset level, so cnx.cnxset.cnx should be the right thing already).

Provide sufficient context to check 'delete' permission in AjaxEditRelationCtxComponent Call rdef.check only when both fromeid and toeid are available. Though only call it once (for the first encountered related entity). Factorize a bit to keep handling of CSS/JS addition the same. Closes #3670209.

merge 3.19.6 into 3.20 branch

Added tag cubicweb-version-3.19.6, cubicweb-debian-version-3.19.6-1, cubicweb-centos-version-3.19.6-1 for changeset 934341b848a6

[test] missing unittest.main() call in unittest_http_headers.py

[test] make unittest_webconfig independant of CWD if apphome is unset in ApptestConfiguration constructor, this later looks for a 'data' directory in CWD, making this test runnable only from web/test (and pytest do chdir before executing tests, so we did not noticed this problem).

[webctl] do not ask questions if verbosity is at 0 Closes #4641960.

[web] Slight tweak on jQueryUI's colors on links (related to #4564046) Follow up to commit 16f550b48d57 which was somewhat brutal. This change only overrides jQueryUI's CSS rules inside tab panes and leaves actual tab buttons alone.

[view] Remove unused imports

Remove obsolete __future__ imports Generators have been available since 2.3, "with" statements since 2.6.

[devtools] Remove unused local function Unused since cw's first commit.

[uihelper] Fix a possible NameError in meta_formconfig could occur when using FormConfig with a custom uicfg object. Closes #3802298.

[devtools] Avoid db_cache collisions and mis-loading This is done by stopping to share the db_cache between TestDataBaseHandler instances It may seem overkill, but is the only way I could get it to work properly. I think the whole db caching code needs some rework to better work with different db handlers. Closes #4601328

[devtools][pg] Remove a wrong assertion in _backup_database When using pre_setup_database, a repo gets initialised before calling backup_database. Hence self._repo is NOT None it that case.

[pkg] prepare 3.19.6

[merge] backport 3.18 fixes

Accept '==' operator in cubes dependencies '==' is the operator used in the python dependencies specifications. See PEP 440 http://legacy.python.org/dev/peps/pep-0440/#version-specifiers Using the same operator in cubes dependencies allow to make setuptools handle the cubes dependencies while keeping cubicweb happy about the dependencies. Closes #4375144

[session] don't silently ignore commits commits while in the precommit or postcommit phase are silently ignored. This is OK in precommit since there'll be a commit afterwards anyway; OTOH in postcommit it may surprise the developer who didn't read the documentation for postcommit operations carefully enough. Related to #4383922.

[migration/bootstrap] add explicit index removal for sqlserver, be tolerant to migration replay Closes #4618944.

Added tag cubicweb-version-3.18.7, cubicweb-debian-version-3.18.7-1, cubicweb-centos-version-3.18.7-1 for changeset cb96f4403cf2

[pkg] prepare 3.18.7

[migration/3.18] speed up the migration and be a little bit more informative Closes #4619277.

[schemaserial] be robust against duplicated CWUniqueTogetherConstrain entities arising from bug in the pre 3.18 era Closes #4619278.

[server] Handle unique constraint violations under recent sqlite The error message changed from "columns foo, bar, baz are not unique" to "UNIQUE constraint failed: table.foo, table.bar, table.baz". Closes #3564510

[merge] backport 3.17 fixes

Added tag cubicweb-version-3.17.18, cubicweb-debian-version-3.17.18-1, cubicweb-centos-version-3.17.18-1 for changeset cda4b066933f

[pkg] prepare 3.17.18

[source/native] handle newer Integrity error messages from sqlite > 3.7 Similar to a0cf2993b6d3 that was done for 3.18.

[source/native/backup_restore] have a unique tunable blocksize for the dump phase Thus we make memory errors much less likely. Closes #4618949.

[urlpublishing] handle sub-types in RestPathEvaluator Since commit 31a1813d53f3 "[entity/url publishing] fetch_rqlst should use 'is_instance_of' instead of 'is'", the RestPathEvaluator's input rset can have more than one etype, so it can't always use sameetypelist. Related to #3825488

[views] fix ecsvexport selector introduced by e48e5a597ccc Fix the view and make web/test/test_views.py pass again.

Add custom checker for Password values We override the default converter to pass Binary values through, but don't do anything about the checker. This worked previously because yams allowed StringIO instances, although its converter didn't do the right thing for them. Fixing this in yams requires that we properly register a checker.

[component] give Links a __repr__

[debian] move markdown dependency from cubicweb-web to cubicweb-common Pointed out by David Douard.

[web/auth] stop playing games with locals() We only want to know if any retriever found something.

[entity] rather than crashing with an IndexError on entity creation, raise a meaningful exception

[web] partial backout of #8391bf718485 to restore RelatedObjectsVComponent Apparently the intended deprecation did not work, and this component is used in at least the tracker cube.

[server] typo fix

[devtools][pg] Activate the non-durability settings of Postgresql. It speed up the PG tests by a factor of 3. See http://www.postgresql.org/docs/current/static/non-durability.html.

[server] fix LoginPasswordAuthentifier.authenticate some kind of typo there.

[web] Override jQueryUI's colors on links (closes #4564046)

[hooks] Fix precommit event logging message

[service] allow repo_stats for users It's used by the siteinfo view, which is available to managers and users. This change prevents a crash in that view.

[views] Escape class attribute value in CWGroup incontext view

[devtools] use a specific test_db_id when disabling anon The anonymous user is created (or not) in postcreate, which for tests means when creating the cached empty database. The anonymous_allowed=False test classes should thus not share their template db with other test classes, otherwise either they end up with an unexpected anonymous user, or the others miss theirs and things fall apart.

Restrict yams version CW 3.19 is not compatible with yams >= 0.40

[facets] Honor 'start_unfolded' facet attribute (closes #4502799)

[facets] Correctly look for inputs of type "hidden" (closes #4502768) jQuery ':hidden' selector looks at CSS visual properties (eg, 'display', 'visibility'). The intent here was probably to look for inputs of type "hidden", which many facets use to store user selection data (eg, FacetRangeWidget). The problem is that regular text inputs (eg the "has_text" facet which has a '<input type="text"/>') will be picked up by this selector if they are inside a folded facet. Chaos and destruction ensue.

[web] Cache results from 'i18n' ajax controller (closes #4487856) This is a simple transient cache which will be forgotten as soon as the user leaves the current page.

[web] There is no global noop(), use jQuery's instead (closes #4487832)

[server] Add missing import of logilab.database Closes #4469407.

[migration] hackish black magic to bootstrap addition of formula attr during migration Turns out we can't add an attribute to CWAttribute before the attribute exists. add_attribute generates rql with a 'formula' relation, which CW doesn't know about yet, so things get unhappy. Once that is fixed, we need to make the CWAttribute addition operation deal with CWAttribute entities that don't have a formula yet. Finally, move the migration to bootstrapmigration_repository so it happens early on, and use add_entity_type rather than add_relation_type for CWComputedRType.

[server] commit after serializing schema Use separate transactions for schema serialization and postcreate scripts execution, so that the latter can rely on the schema being persistently stored. This matters e.g. if something in a postcreate violates a unique_together constraint, we need the constraint persisted in order to raise a meaningful error. See #4525069.

[book] fix docstrings to please sphinx example rst roles in docstring must be protected (in a literal).

[book] several fixes in the rst files - prevent duplicate markup definitions, - use existing Workflow markup instead of redefining one (in baseschema.rst) - include dataimport - fix non-existant directive autodocfunction - remove the empty 'embedding.rst' file

[book] activate the viewcode extension Automagically make link to class/modules/etc. it finds in the documents.

[book] new theme based on pyramid theme (closes #4291287)

Remove unused lgc.interface imports and leftovers

[repository] provide a .new_session entry point The current .connect only returns a `sessionid` which must be given to repo._get_session(...) to get the real session object. Related to #4151635.

[devtools/testlib] Use actual 'admin' user configuration values Previously ``db-user`` and ``db-password`` configuration values were used to fill ``admlogin`` and `admpassword`` attributes of CubicWebTC. The correct data for these should read from `[admin]` section. Closes #4065070.

[forms] closes #2437859 - Detect and prevent concurrent edition of the same entity. Add the timestamp of form generation to each entity's meta-information fields. On form validation, check that no concurrent change is overwritten and raises a ValidationError in case of concurrent change. A nicer handling with a message and a link to the new version of the entity would be a good thing...

[migration] make sure the repo knows about all constraint types Adding new constraint types is cumbersome. It's easy to forget to do it in a migration script. So sync the constraint types at the beginning of each migration. Closes #3724892

[migration] stop caching the mapping from constraint type name to eid It's not so frequent that a cache seems necessary, and we were not invalidating that cache when adding a new constraint type. Related to #3724892.

[web/json] an empty rset is just fine for ejsonexport The ejsonexport view can just return an empty list for an empty rset. Closes #4005518

[RichString] Add markdown support Supporting markdown requires the 'Markdown' python packages, and since it widely available on the target platforms adding it as a strong requirements will not be a big constraint. Closes #3814302

[ldapfeed] Reduce default value for user-attrs-map option (closes #3824889) This is needed because lgc.configuration.Configuration does not allow removing key/value pairs from the default (due to its use of dict.update() internally). Since CWUser.login is required, users of the add-source command will always be able to override it.

[serverctl] Ask for URL when adding a new source (closes #3824868)

[serverctl] Ask for parser type when adding a new source (closes #3484231) Remove 'quick_start' option to load all AppObjects, including feed parsers.

[doc/3.20] more details on removed code related to #3799117

remove 3.11 bw compat Closes #3799117.

remove most 3.10 bw compat Related to #3799117. The boxes and entityvcomponent objects cannot really be removed as they are still used throughout the code base (and possible cubes). This may be caused by a non-working deprecation and needs some more work.

remove leftover pre 3.9 deprecation warnings

[utils] Remove function-in-the-middle call

[web] set Vary response header to "Accept-Language" when using content negotiation This is slightly annoying because the response actually only varies based on the language we decide to send, which has much fewer possible values than Accept-Language, but that's not in the request, so we can't easily use it. Deployments using varnish or similar and controlling the set of available languages will likely want to override this to allow reasonable amounts of caching. Closes #2105812

[web/cors] don't overwrite other Vary headers Vary is a list of request headers, we shouldn't override others.

[web] add support for HttpOnly cookie flag And use it for session cookies. Closes #4142521.

merge 3.19.5 into 3.20 branch

Added tag cubicweb-version-3.19.5, cubicweb-debian-version-3.19.5-1, cubicweb-centos-version-3.19.5-1 for changeset 3ac86df519af

i18n update 4 strings have disappeared.

[pkg] 3.19.5

[server] hold connection to the db in tx_actions We can be called without a cnxset (e.g. from repoapi).

[wsgi] If multipart cannot parse the POST content, let it pass. multipart can only parse html form data. It the content_type is, for example, "application/json", get_posted_data should not fail but just stop trying to read the content. Closes #4421845

[devtools] Fix JS tests' HTML code

[devtools] "Keep" some temporary files/dirs around to help with debugging The whole QUnitTestCase runs with an @with_tempdir so it's redundant anyway.

[devtools] Fix Firefox launcher in QUnitTestCase (closes #4294727) The main changes are: - stop creating the profile, firefox will create it - point firefox to a profile directory instead of giving it a profile name (this has the added bonus of not polluting the user's profile list) - start firefox once and kill it 5 seconds later to let it finish its profile creation (along with system-wide extensions setup)

[devtools] allow cross-origin requests for qunit We have a mix of file:// html and http:// ajax calls. Which we should at some point fix to all be http, but. Related to #4294727.

merge 3.19.4 in 3.20 branch

[cors] Fix CORS headers generators The Access-Control-Allow-* and Access-Control-Expose-Headers are response headers, not request headers. Hence, we need generators for them. Closes #4412575.

[wsgi] Fix posted files filename reading The filenames are parsed by multipart.parse_form_data, which does the unicode decoding. Trying to re-decode the filename was leading to an error.

[pkg] Depend on Pillow instead of PIL The Pillow library is becoming the de-facto replacement for PIL. It also is way simpler to install with pip than PIL. Closes #4411354.

Added tag cubicweb-version-3.19.4, cubicweb-debian-version-3.19.4-1, cubicweb-centos-version-3.19.4-1 for changeset c4e740e50fc7

[pkg] 3.19.4

merge 3.18.6 into 3.19

Added tag cubicweb-version-3.18.6, cubicweb-debian-version-3.18.6-1, cubicweb-centos-version-3.18.6-1 for changeset d91501356742

[pkg] 3.18.6

[hooks/security] allow edition of attributes with permissive permissions If an attribute has more permissive security rules than the entity type itself, we should be green and not deny action because of an early global entity permission check (with the more restrictive rules). Only if one attribute with the entity-level permission rules is edited will the global check be performed. Note: * the "if action == 'delete'" check at the entry of check_entity_attributes is a guard for a condition currently not happening in cubicweb itself (but application hooks could conceivably call this function with a 'delete' action) Closes #3489895.

Almost backout afcd46716d6a which breaks _select_best raising an ambiguity exception in debug mode. The problem is, before afcd4, *tests* ran in debug mode and we want this (e.g. we want to show, rather than swallow, select ambigüities). We juste replace the bogus __init__(vreg.config) by __init__(True), which is practically equivalent and also much more clear.

[server] fix anonymous_user predicate in tests devtools' TestServerConfiguration overrides the anonymous_user method, but not the anonymous-user config option, so testing for the latter would give the wrong result. Closes #3996664.

[entities] cw_rest_attr_info() should only consider required attributes (closes #3766717) This prevents CW from choosing unique but non-required attributes. None/NULL is a poor choice for RESTful URIs.

[views] csvexport accept an empty rset (closes #4236928) When you tried to apply the 'csvexport' view on an empty rset, the view couldn't be selected and you got a HTTP 500 error. Also add two new test cases.

[views] Display attributes in entity creation form based on "add" permission Previously, the "update" permission was used. Hence in case the latter is more restrictive that the "add" permission, an user may not be able to set such an attribute, despite she may have "add" permission on it. As a result of the change of permissions action in `editable_attributes` method (add/update depending on whether the entity is being created or modified), the "eid" attribute would have shown up in the edition form. To avoid this, it is moved in the "hidden" section (where it should arguably belong anyways). Closes #4342844.

[datafeed] Commit after all deletions in datafeed parser This avoids misleading validation error because schema constraints could be temporarily broken depending on the deletion order. Closes #4372127.

[schema] CWComputedRType is a schema type Hide it from the default views.

[doc] proofreading CWEP002 section

[CWEP002] document computed relations and attributes Related to #3546717.

[CWEP002 migration] support sync_schema_props_perms for computed attribute

[CWEP002 migration] support drop_relation_type/drop_relation_definition/drop_attribute for computed attributes

[CWEP002 migration] support add_relation_type/add_attribute for computed attributes Related to #3546717.

[CWEP002] properly handle serialization of computed attributes Until this cset, only serialization of computed attribute's formula was handled (thanks to the relation definition's properties dict). We now: * properly serialize/deserialize attribute's formula * test first if the database has been migrated and contains related column or not Related to #3546717

[CWEP002] Add support for computed attribute synchronization Related to #3546717. Test and handle the behaviour with several formulas and identified use cases. To do so add a birth year and a computed attribute age to Person in the test schema.

[CWEP002] Add schema finalization checks for computed attributes ``finalize_computed_attributes`` essentially checks that computed attributes types match with the type declaration of the attribute. Related to #3546717.

[CWEP002] Account for attribute formula in schema bootstrap This isn't enough to have computed attribute support, it is only in order not to crash when yams 0.40 is used. Related to #3546717.

[CWEP002 migration] support sync_schema_props_perms for computed relations

[CWEP002 migration] support add_relation_type for computed relations Related to #3546717.

[CWEP002 migration] support drop_relation_type for computed relations Related to #3546717.

[CWEP002 migration] properly raise exception on (add|drop)_relation_definition for computed relation Related to #3546717

[hooks] do not abuse of inheritance for CWRType hooks Just copy the selector to ease readability.

[schema serial] add some comments

[schema] define full_rql on RQLExpression class without it, we may have an error while displaying error which occured during some expression initialization because this attribute is not yet defined while used in the __str__ method.

[schema] properly raise BadSchemaDefinition when some expression mains variables may not be guessed

[CWEP002] properly handle serialization of computed relations We now: * have CWComputedRelation in the bootstrap schema to store computed relations * properly serialize/deserialize it * test first if the database has been migrated and contains the related table Related to #3546717 [jcr: adjust unittest_querier to pass with the added entity type]

[CWEP002] Plug the computed relation rewriter in the querier Related to #3546717.

[CWEP002] refactor rql read security checking Split 'check_read_perms' into 'check_relations_perms' which checks relations 'read' permissions and 'get_local_checks' which build dictionary of local security checks (rql expression) for variables. This allows to check relations 'read' permissions earlier in the process and so to prepare insertion of the rql rewriter: we want to check permissions of the computed relation, not permissions of relations introduced by the associated rule, to conform to the CWEP. Related to #3546717

[CWEP002] introduce RQLRelationRewriter Refactor existing RQLRewriter for later reuse of rewriting relation as specified by CWEP002. Work is different because we simply want to replace a relation by another rql snippet and we don't have to bother with EXISTS, subqueries and all. This rewriter is not yet plugged into the querier. Depends on yams 0.40 API. Related to #3546717

[CWEP002] Add schema finalization checks for computed relations (rules) Related to #3546717

[schema] add utility function to build a CubicWebSchema from a namespace Will be useful for testing computed attributes and relations.

Fix test migration crash waiting to happen due to inferred relations Avoid crashing when a relation definition which is already known (due to being inferred) gets explicitly added. As explained in a comment, this would deserve more thinking, but at least this fixes a test failure with computed relations.

fix typo in syncschema hooks

[test] Make test_undo_api less random The order in which hooks are run is not predictable if they have the same 'order' attribute, which is the case for SetOwnershipHook and SetInitialStateHook. So don't assume in_state will be set before created_by.

merge 3.17.17 into 3.18 branch

[wsgi] Fix unicode decoding in POST The application/x-www-form-urlencoded sent by firefox or chrome is utf-8 encoded BEFORE being urlencoded. Hence, decoding must urldecode THEN decode the utf-8 string, and not the other way around.

[wsgi] add the --debug / -D option to the wsgi command

[wsgi] Fix multiple variables reading in params Closes #4306081

[wsgi] Set self.vreg Closes #4306049

[wsgi] Add missing import Closes #4305988

[wsgi] Fix https detection - 'HTTPS' is not part of wsgi, it is a CGI variable. The right wsgi variable is 'wsgi.url_scheme' - Force https if url starts with '/https/' (and remove the prefix from the path. Closes #4305985

[wsgi] Re-set the request content after calling the inherited constructor. Closes #4191813

[wsgi] Honor the 'CONTENT_TYPE' wsgi variable See http://legacy.python.org/dev/peps/pep-0333/#environ-variables Closes #4191812

Added tag cubicweb-version-3.17.17, cubicweb-debian-version-3.17.17-1, cubicweb-centos-version-3.17.17-1 for changeset 57e9d1c70512

[pkg] prepare 3.17.17

[web/headers] don't list a request header twice in "Vary"

[test] silence a few more deprecation warnings in unittest_hooks self.session → cnx.

[session] call rollback in Connection.__exit__ If we just free the cnxset and clear the Connection, we're missing the pending operation's rollback_event callbacks, which may be needed for cleanup.

[devtools] pre_setup_database takes a Connection, not a Session

[source/native] cPickle is available in all supported pythons

devtools: deprecate .req_from_url Which uses .request() And instead provide .admin_request_from_url, which is a context manager yielding the request.

[tests] Add a webtest based TestCase class Closes #4002134

[tests] Move 'anonymous_allowed' property from CubicWebServerTC to CubicWebTC The effective set of the property on the config is now done later that before, ie at the end of CubicWebTC.init_config instead of before. It is believed not to have an impact on the test suites that uses `anonymous_allowed`. Related to #4002134

[compat] Remove imports of "any" and "all" from lgc (closes #4306044) They're builtin since python 2.5.

[css] Remove the `div` tag specification of the pendingDelete css rule The HTML tag was changed by e2f96b16c3bd from a div to a span without changing the css rule accordingly. Closes #4285248.

[cwvreg] cleanup the event manager when reloading modules Closes #3848995 The event manager callbacks are not cleaned during reloading. So the callback defined in the reloaded module appears twice (old and new version). This may cause problem when the old version is called.

[test] Add missing attribute 'add' permission in test schema Silences yams warning.

[facet html] Add surrounding div with "facetBody" class to "has_text" facet. This makes its html structure consistent with the other facets and fixes both css and a javascript behaviour (hide the facet body when clicking on its title). Closes #3797501.

[cwctl] don't prompt for cube options in automatic mode Closes #3984223

[serverctl] fix default value for db-init config-level Otherwise cubicweb-ctl db-init --automatic complains that "--automatic and --config-level should not be used together" when they weren't. Closes #3984336

[web] Return useful error messages when exceptions arise in ajax controllers (closes #3932503) The call to super() in RemoteCallException passes 'reason' all the way to python's Exception.__init__() which is then stored in Exception.args. This way, CubicWebPublisher.ajax_error_handler() gets a useful error message when calling unicode(exc). The change to uilib is just to prepend the exception type name. Just a small improvement.

[test] Fix test breakage uncovered by previous changeset Now that Binary.__eq__ doesn't always return true, this test started failing. CWAttribute.defaultval is zpickled, and the description_format attribute is a String, meaning unicode, so adjust the expected test result.

[etwist] use more specific REQUEST_ENTITY_TOO_LARGE instead of BAD_REQUEST When a request body exceeds the configured limit, return 413 to the client instead of a generic 400.

[entities, view] delete dead code The auto_unwrap_bw_compat metaclass calls unwrap_adapter_compat, which was removed in changeset 697a8181ba30 "remove 3.9 bw compat". So this can't possibly work anymore, meaning we can get rid of it.

[book] stop talking about the hg `forest` extension

[book] update sections about dependencies This is probably incomplete, but it's a start.

[web/component] move unicode() call around Makes Aurélien happier.

[web/js] replace callAddOrDeleteThenReload with more generic callAjaxFuncThenReload That's going to be a more useful replacement for callUserCallbackThenReload. Closes #4178566.

[hooks/syncsession] try to remove cnx vs session confusion

Fix Binary.__eq__ (closes #4172701) Due to a typo we always returned true, which was unhelpful.

[dataimport] stop ignoring errors on flush That can only result in database corruption.

[dataimport] don't commit on flush Changeset 26cdfc6dd6f8 introduced this confusion for no apparent reason, and it doesn't make much sense.

[test] Add test for dataimport's RQLObjectStore

[dataimport] Stop swallowing errors from commit/flush Silent DB corruption is not OK. Also drop comment from ObjectStore.commit that I don't understand.

[dataimport] Update RQLObjectStore to Connection API Take a connection instead of a session. Don't play games with set_cnxset.

[dataimport] remove _rql heresy 1) ObjectStore knows nothing about rql, it just drops it on the floor 2) RQLObjectStore has a session, it can run rql that way 3) setting an object's "_rql" attribute is not a reasonable user-facing interface Also drop _commit attribute from base ObjectStore, it doesn't use it.

[dataimport] remove dead code The only caller of ObjectStore._put is ObjectStore.create_entity, which RQLObjectStore overrides (and doesn't call up).

[dataimport] do not use sys.exit() to raise missing argument error Related to #3845572

[dataimport] _create_copyfrom_buffer: add the tests Related to #3845572

[dataimport] _create_copyfrom_buffer: do not ignore columns if data is a list Related to #3845572

[dataimport] _create_copyfrom_buffer: fix datetime converter + add time converter Include second and microsecond in the output instead of dropping them on the floor. The time zone is not supported for now, though. Related to #3845572

[dataimport] _create_copyfrom_buffer: raise ValueError if conversion cannot be performed If the conversion fails, there is no reason to continue execution. One should notify the error. Continuing after a misconverted data could corrupt the database. Related to #3845572

[dataimport] _create_copyfrom_buffer: fix separator check in string converter The empty string is a valid replace_sep. Related to #3845572

[dataimport] _create_copyfrom_buffer: put converters into a list Cleans up the code to avoid a succession of ifs. Related to #3845572

merge 3.19.3 into 3.20 branch

Added tag cubicweb-version-3.19.3, cubicweb-debian-version-3.19.3-1, cubicweb-centos-version-3.19.3-1 for changeset 37f7c60f89f1

[pkg] prepare 3.19.3

merge from 3.18 branch

merge 3.17.16 into 3.18 branch

Remove uses of logilab.common.compat.{all,any} They're just aliases to the builtin ones on python 2.5+. If anyone needs convincing: >>> from logilab.common import compat >>> compat.any <built-in function any> >>> compat.all <built-in function all>

remove references to global environment variable APYCOT_ROOT

Add warning messages when enabling remote pickle-based repository access Warn when starting a pyro or zmq-only repo, or when one of these access methods is enabled. Closes #2919295

[web] Fix expiry of anonymous sessions (closes #4154479) Things like: try: foo except: bar else: baz doesn't work very well if you expect baz to run even in the exception case. Plus, session.cnx.check() is a dbapi remnant, so drop it and just look at session.mtime.

[hooks/syncschema] avoid altering a dropped table If the constraint's rdef is being removed, CWConstraintDelOp doesn't need to do anything. Otherwise it may try to alter a removed table/column. Closes #4154549

[js] fix name error

[devtools] improve error message when postgresql tools are missing By default in at least Debian, some pg tools are not present in the PATH. Or they may not be installed. But the tests tools expects them to be in the PATH, and give an unhelpful 'No such file or directory' backtrace if they're not found. To help devs using the pg tests we improve the error message.

[server] Replace non portable strftime formatter (closes #4132161)

[web/request] use a picklable Counter object for tab index counters Related to #1381328.

[web/auth] The auth retriever's authenticated() callback takes a session, not a cnx We don't have a cnx at that point.

[test] Fix the query-log-file test Testing this functionality in a CubicWebTC is awkward because the wrapping happens in handle_request, which tests basically bypass (they call core_handle directly, and do their own magic for linking the request with a cnx). The test method worked when ran on its own, but not together with the rest of the test class. So use a CubicWebServerTC instead, which goes through the whole stack.

[doc/book] spelling fixes in security tutorial

[doc/book] spelling fixes in "testing" chapter

[datafeed sources] finish the session -> cnx switch Related to #3933480.

[doc/book] update examples, using the new connection api

[test/entities] use the new connection api

[web] restore query logging functionality (closes #3972561) The query-log-file option stopped working when the web stack was moved from dbapi to repoapi.

[cwctl] make cubicweb-ctl versions lighter (closes #4002158) Set config.quick_start to avoid unnecessary appobjects and schema loading.

Added tag cubicweb-version-3.17.16, cubicweb-debian-version-3.17.16-1, cubicweb-centos-version-3.17.16-1 for changeset a979d1594af6

[forms] consider inline creation form information as linkto info. Closes #3161121

[pkg] prepare 3.17.16

[reledit] Fix reledit icons jumpiness (closes #4106867) Introduce an "invisible" class (copied verbatim from Bootstrap) that only sets the "visibility" property. "display: none" is the reason for the whole jumpiness as it actually detaches the element from the rendered layout. "visibility: hidden" only makes it transparent while keeping its original box properties (width, height, etc).

[views] Replace poorly named "invisible" class with "list-unstyled" "li.invisible" actually means "hide bullet". Use a reasonable name instead, such as bootstrap's "list-unstyled" class. This patch also changes the DOM element the class is applied to. "li.invisible" had to be enabled on every "li" tag, whereas the "list-unstyled" class only needs to be applied to the parent "ul" element.

[views] a 'div' with 'display: inline' screams 'span' (grafted from af6e3db801fcfbd6f562c4a7cfdc89f187042792)

[pkg] remove simplejson dependency We no longer use it, the stdlib's json module is fine.

Added tag cubicweb-version-3.19.2, cubicweb-debian-version-3.19.2-1, cubicweb-centos-version-3.19.2-1 for changeset 8ac2202866e7

[pkg] Fix version number Note to self: don't phase -p before coffee.

[pkg] 3.19.2

[webtests] finish to give all self.view(....) a req=req parameter

[tests/syncsession] use the new connection api

[webtests/application] remove unused imports

[webtests/application] .user(...) really wants a request object

[tests/datafeed] use the new connection api (a small leftover) Test assertions get indented because of self.session -> scoped cnx.

[entitiestests/base] don't store an entity on the test case Make do with an eid.

[entitiestests/base] use the new connection api

[testlib] use the new connection api

[exttests/rest] use the new connection api

[webtests/bookmarks] use the new connection api

[hookstests/integrity] use the new connection api

[hookstests/base] use connections instead of web requests

[hookstests/base] use the new connection api

[hookstests/syncschema] use the new connection api

[tests/rqlannotation,querier] use the new connection api

[webtests/web] use the new connection api

[webtests/navigation] use the new connection api

[webtests/basetemplates] use the new connection api

[webtests/basecontrollers] use the new connection api

[testlib] deprecate .remote_call and provide new connection api friendly .remote_calling

[webtests/propertysheet] kill an "import *"

[webtests/breadcrumbs] break long lines

[webtests/views_editforms] use the new connection api