[dataimport] Properly escape strings sent to COPY FROM (closes #5278743) See http://www.postgresql.org/docs/9.1/static/sql-copy.html#AEN64296 for escaping codes.

[http] fix cache-control syntax in EtagHTTPCacheManager use , to separate cache response directive closes #5307539

[connection] provide some missing documentation bits The previous changesets remove information about session data and transaction data. We provide the necessary entries in the docstring of the Connection class. Related to #3933480.

[repoapi,session] remove all session-as-cnx backward compat The `dbapi` being gone, we now can drop the session object bw-compatibility layer. This will allow further simplifications, such as folding ClientConnection and Connection (without too much pain), and then having persistent sessions. Related to #3933480.

[server] drop repo.internal_session and InternalSession

Added tag cubicweb-version-3.20.7, cubicweb-debian-version-3.20.7-1, cubicweb-centos-version-3.20.7-1 for changeset 359d68bc1260

[pkg] 3.20.7

[dbapi] retire repo.[gs]et_shared_data which were used by the dbapi Related to #3933480

[dbapi] retire repository transaction methods which were used by the dbapi Related to #3933480.

[dbapi] retire repo.call_service which was used by the dbapi Related to #3933480.

[dbapi] retire repo.commit/rollback which was used by the dbapi Related to #3933480.

[dbapi] retire repo.user_info which was used by the dbapi Related to #3933480.

[dbapi] retire repo.entity_metas which was used by the dbapi Related to #3933480.

[dbapi] retire repo.execute, which was used by the dbapi We will retire or reform pieces, mostly of the Repository class, that were related to the old dbapi way of life. Related to #3933480.

[devtools/testlib] remove dbapi backward-compat Related to #3933480.

[test] fix copy-pasta in doc string

[book] remove dbapi documentation Move some still-relevant doc related to the RQL connection API to the 'sessions' section. It might make sense to move this back outside of the "Repository development" chapter at some point since this is relevant in other contexts.

[uilib] `uilib.js` helper now honors explicit JSString (closes #4959538) Previously, if a JSString was inside a nested container (i.e. list or dict), it was handled as a string, not a verbatim js chunk (see docstring and test). So the following code:: >>> str(js.cw.pouet(1, {'callback': JSString('cw.mycallback')}) resulted in:: "cw.pouet(1, {'callback': 'cw.mycallback')})" while we expect (note the removed quotes):: "cw.pouet(1, {'callback': cw.mycallback)})" We use ``cubiweb.utils.js_dumps`` instead of ``cubicweb.utils.json_dumps`` which is aware of JSString.

[dbapi] remove the dbapi module and its immediate remaining users We suppress toolsutils.config_connect, which has currently only one known user (the email cube), which is being patched. It can be replaced with utils.admincnx function for a local access. Next will come a series to: * remove the session backward compatibility * fold ClientConnection into Connection Closes #3933480.

[web/views] remove dbapi usage from 'gc' view

[sqlite] Fix integrity error parsing Up to sqlite 3.7, integrity error message may be in the singular form if only one column is in the unique index. Closes #5224632

[postgres] fix limit_size regexp to properly remove one char tags like <p>. Closes #5221847

[postgres] _regproc.postgres.sql should be reloadable

[mail] include Date header in generated mails (closes #5271058) Mails without Date gets flagged as spam. Plus it's wrong anyway.

[doc] mention datadir-url configuration option in release notes Related to #5204550

[web] add test for datadir-url option (related to #5204550)

[web] Add datadir-url configuration option (closes #5204550) Makes it possible to use a different host for static data instead of ${base_url}/data

[web] kill https-deny-anonymous option (closes #5193687) It doesn't make much sense to prevent people from using https.

[cwctl] don't filter DeprecationWarnings

[exceptions] avoid calling UniqueExceptionError.rtypes after closing the session

[web/test] Fix broken tests after commit b81adb5e32f9 Add missing n3rdf view to RDF view list.

[web/test] don't delete uicache directory Other tests use it, so the rmtree was a bit too much. Set CACHEDIR to a temporary subdirectory, and only delete that.

[web/views] add text/n3 rdf view Less shit than xml.

[schemas] make CWEType.final default to False (closes #5049201)

[repository] replace session with cnx in a few places

Don't pass an encoding to rqlst.as_string()

[rset] Deprecate the 'encoded' argument to ResultSet.printable_rql() It's hardly used at all, and when used inside CubicWeb, it's only set as false. Better make sure that this function always returns a proper unicode string and let the caller handle conversion on its own.

[gcdebug] ignore _NeedAuthAccessMock instances When looping over gc.get_objects(), we'd crash in _NeedAuthAccessMock.__getattribute__.

[views] Fix 'gc' view to use 'repo_gc_stats' service

[services] Fix 'repo_gc_stats' to return a list of unreachable objects' repr

[services] Prevent 'repo_stats' service from aggregating information It's up to the consumer of the service to decide whether it wants aggregated data or not... Ensure the /siteinfo page rendering is not modified.

[hooks] add a looping task that dumps the stats regularly in a file

[web/facet] Use an Exists node in HasRelationFacet The generated RQL now includes an EXISTS clause. For instance, one gets: :: DISTINCT Any WHERE X is CWUser, EXISTS(X in_group A) instead of: :: DISTINCT Any WHERE X is CWUser, X in_group A for a HasRelationFacet with ``rtype = 'in_group'``, which is more appropriate for testing the existence of the relation. Add a test for this facet along the way.

[debian] Add a Breaks for cubicweb-folder which stills use GMTOFFSET Related to #2154655.

[web] kill GMTOFFSET (closes #2154655) Its last user, set_cookie(expires=...), can just as well interpret its argument as UTC. Naïve datetime objects with an implicit non-UTC timezone are just awful.

[web/request] don't play tricks with utc offset We need an UTC datetime from a time.time()-type value. It turns out datetime provides a method to do just that. Related to #2154655.

[server/test] Drop dependency on folder cube Just copy schema bits when needed and drop a view (not possible anymore) in unittest_repository. Closes #5168939.

[web/test] Drop folder cube dependency Copy useful schema bits, define IBreadCrumbs adapter for Folder/fild_under (previously relying on ITree adapter) and adjust unittest_viewselector tests to drop a view that does not exist anymore. Related to #5168939.

merge 3.20.6 into 3.21

[web/views] Allow changing CSVView's separator (closes #5227442)

[web/view/treeview] add a 'selected' class on <li> if the entity's url is the currently displayed url An alternative would be to refactor the corresponding view so that controlling css classes wouldn't require overriding a 5O-lines method. Notice by default there is no associated class in the CSS so there is no rendering difference unless one explicitly add rule in his CSS file.

Added tag cubicweb-version-3.20.6, cubicweb-debian-version-3.20.6-1, cubicweb-centos-version-3.20.6-1 for changeset 7f64859dcbcd

[pkg] 3.20.6

[notification] don't block while sending mails Regression from #7c17659c9eae, closes #5190001.

[dataimport] make MetaDataGenerator / NoHookObjectStore usable from a datafeed parser ie make it possible to create entities with external source as cw_source and proper external id. Closes #4891552

[dataimport] don't generate metadata which are explicitly specified Related to #4891552

[debian] prepare 3.20.5-2

merge 3.19.11-2 into 3.20

Added tag cubicweb-debian-version-3.19.11-2 for changeset 6d265ea7d56f

[debian] Update watch file

[debian] Fix cubicweb-dev dependency

Added tag cubicweb-version-3.20.5, cubicweb-debian-version-3.20.5-1, cubicweb-centos-version-3.20.5-1 for changeset 51aa56e7d507

[pkg] 3.20.5

[dataimport] don't insert created_by / owned_by relations when user is the internal manager (eid = -1). Closes #5162943

[repository] don't attempt to delete computed relation, they have no table in the database. Closes #5162935

[migration] sync_schema_props_perms should skip computed relations. Closes #5147796

[schema] typo

[web/test] fix test_views reusing a request for two different view calls leads to badness

[documentation] add note concerning Attribute, TODAY and NOW in yams documentation

[i18n] properly translate error messages related to violated unicity constraint. Closes #5100373

[web/data] Add missing mode='swap' in loadxhtml calls (closes #5087432) Let's avoid putting divs inside divs, dawg.

[web/views] Fix PropertyKeyField widget (closes #5087413) CWProperties don't have unittests, this will likely break again.

[web/views] Fix exception in CWProperty edit form (closes #5087412) Also add missing help message. The overall display is ugly, but at least the message is shown.

[web/request] Restore attribute setting for search_state This was dropped in 1f84295bfe95 when `search_state` attribute was turned into a property, but without a setter. So adding a setter here restoring previous behaviour, as some tests and cubes use it. Related to #4875761.

[web/request] simplify search_state loading, fix comment

[webrequest] Lazy-load search_state - The update_search_state function will hit the session data only if __mode is present in the form. - set_search_state keep the same semantics - Direct access to search_state in the session data is changed by an access to the new property. Closes #4875761

[config] disable fckeditor by default (closes #1368900) FCKEditor is no longer maintained, and has a history of XSS issues. Let's use plain text as default text format.

merge 3.19.11 into 3.20

Added tag cubicweb-version-3.19.11, cubicweb-debian-version-3.19.11-1, cubicweb-centos-version-3.19.11-1 for changeset 1ae64186af94

[migration] reorder system tables / meta-schema migration into bootstrap_migration Also fix a bug in the backwards-compat eid_type_source method where it returned an extid (possibly null) instead of a source name. With these changes, migrating a database from cw 3.12 is now possible. Closes #4962681

[dataimport] add missing import (closes #4985916)

[debian] cubicweb-dev needs cubicweb-etwist Closes #4965410

[pkg] 3.19.11

Binary initializer should accept bytearrays (closes #5084075) Recent versions of pyodbc return bytearray objects for binary data.

[native source] log the used connection in doexecmany as in doexec

[server] use unicode wherever appropriate in transaction/undo related methods Silences lgdb/sqlite warnings about using str instead of unicode.

[web] typo

[entities] return the entity itself from notification_references if the view uses a random message-id Useful for entity update notifications to reference the entity creation message.

[notifications] use non-fixed message-id for EntityUpdatedNotificationView More than one of those can be sent for a given entity, so they shouldn't all have the same message-id. Closes #4962712.

[server/test] kill remaining 3.19 deprecation warnings in undo tests

[devtools] accept str objects in CubicWebTC.new_access and .create_user

[utils] use the "predictable" argument to js_dumps Fixes doctest under python -R. Related to #4959402.

[server/test] don't rely on a sorted output from .object_relations() Fixes test under python -R. Related to #4959402.

[req] make rebuild_url predictable Fixes test with python -R. Related to #4959402.

[web/test] don't depend on iteration order on a set Fixes test under python -R ('pending_insert' in session data is a set, so we can't rely on any particular ordering). Related to #4959402.

[web/test] fix cubicweb 3.19 deprecation warning

[web/test] use unicode instead of str for user logins makes lgdb/sqlite quieter.

[test] Don't depend on dict iteration ordering Fixes test with python -R. Related to #4959402.

[entities] make cw_fti_index_rql_queries predictable Don't depend on the hash function, fixing test with python -R. Related to #4959402.

[hooks/syncschema] use a list instead of a set for UpdateFTIndexOp The order in which etypes get indexed matters. Related to #4959402.

[rqlrewrite] sort possible types when turning is_instance_of into is Helps get predictable output even with python -R. Related to #4959402.

[server/test] use unicode strings for user logins

[web/test] use unicode object as user login (py.warnings) WARNING: /home/jcr/src/cw/cubicweb/devtools/testlib.py:206: DeprecationWarning: Sanitizing an input dictionary with str values, please check your data (key = 'login')

[server/querier] stop passing a rqlst to empty_rset (py.warnings) WARNING: /home/jcr/src/cw/cubicweb/server/querier.py:46: DeprecationWarning: [3.20] rqlst parameter is deprecated return ResultSet([], rql, args, rqlst=rqlst)

[web/views] Use predictable ordering for attributes in entity XML view Fixes test under python -R. Related to #4959402.

[web/views] set explicit order for ManageSourcesAction Gives predictable ordering wrt ManageUsersAction under python -R. Related to #4959402.

[web/test] don't depend on dict iteration ordering Fixes tests under python -R. Related to #4959402.

[web/test] pass login as unicode string silences lgdb/sqlite warning.

[web/test] fix 3.19 deprecation warning

[sobjects/test] Fix dependency on predictable hashing Fix test under python -R by ignoring url query parameter ordering. Related to #4959402.

[test] don't rely on a stable ordering for appobject_path Fixes test under python -R. Related to #4959402.

[server] remove "standalone RQL server" from possible configuration choices Related to #2919297.

Remove leftover bits from start-repository command, and document its demise Related to #2919297

Remove the remote repository-access-through-zmq support Modern methods such as the rqlcontroller cube + the cwclientlib library are the way forward. Closes #2919297.

Remove remote repository-access-through-pyro support Modern methods such as the rqlcontroller cube + the cwclientlib library are the way forward. Closes #2919309.

[web/views/rdf] Take the second element of XY equivalent for non final relation When given a Yams snippet of the form `<etype> <rtype>` ``xy.xeq()`` will return the XML snippet equivalent to this relation in the form of a tuple `(subject, rtype, object)` so ``item`` will always be a tuple here. In fact, the correct code appears just a few lines above (around line 88) for final relations so I guess this is a copy-paste mistake. Closes #4745929.

[server] Make sure 'zmq-address-sub' config is always iterable

[web] split timeline in an individual cube Closes #1279544

[devtools] don't hide warnings and errors during tests The markdown module in jessie turns all warnings into logging calls, so this allows me to still see deprecation warnings when running tests.

[test/i18n] remove unneeded hack

merge 3.19.10 into 3.20

[server/session] Fix typo "meta vs. metas" in describe()

Added tag cubicweb-version-3.19.10, cubicweb-debian-version-3.19.10-1, cubicweb-centos-version-3.19.10-1 for changeset 3bab0b9b0ee7

[pkg] 3.19.10

[i18n] Update translations Related to #5012581.

[config] Lowercase the FQDN we get from the OS (closes #5040345) Browsers convert the host name part of URLs to lowercase. This has the unfortunate effect of triggering various CORS error messages in CubicWeb (because the Origin: header will contain lowercase letters). As per RFC 4343, host name comparison should be case-insensitive. So let's put it in lowercase when we grab it from the host system. If admins put uppercase letters in "host" or "base-url" in their all-in-one.conf, then it's their fault.

[web/views] Honor 'action' attribute on AutomaticEntityForm, closes #4943392 On AutomaticEntityForms, get_action used to bypass the logic from FieldsForm.form_action() which handles the 'action' attribute itself, preventing callers from overriding it at selection time. This capability is used by the inlinedit cube. In a nutshell, we let the deprecated AutomaticEntityForm.get_action() raise an AttributeError, which allows FieldsForm.form_action() to handle it properly.

[docstrings] fix project-wide English language mistake Even though 'u' is a wovel, it may be pronounced 'yu' which is a consonant sound. In such cases precede it with 'a' instead of 'an'.

[doc] Don't pretend we know cubicweb runs on mysql As far as I know it hasn't been tested in years.

[test/i18n] Update reference po file following 75cd7df5c32e This changeset added a (missing) message "add a <etype>", so reference po file needed an update.

[devtools] add a method on CubicWebTC to fake data posted by standard cw forms. Closes #4985805

[web/wdoc] fix links They're relative to ${base-url}/doc/, not ${base-url}. Closes #3161099.

[i18n generation] add missing 'add a EntityType' message to generated catalogs. Closes #5012581

[server/test] fix test_zmq We were only asserting in a thread, thus ignoring the results.

[views,js] remove user_callback feature Related to #3975260.

[newcube] Make the generated code PEP-8 compliant Closes #4925396

Remove unused imports

[ext] Remove use of cStringIO We create a StringIO never to read anything from it. Docutils seems to support setting the 'warning_stream' to False to ignore warnings. So let's just do that.

[view] Use io.BytesIO as stream for binary views

[undo/test] check that the entity was restored even though the fiche relation wasn't

Use a moved_entities table to record external entities moved to the system source Instead of using a negative eid in the entities table, move the record to a new table so we don't have an interval with a missing eid in entities. Related to #4846892

Add unique index on entities.extid

[undo/test] if we try to delete an entity referenced by an inline relation, we fail

[undo] when we can't undo an inlined relation change, point at None, not a broken eid This whole thing is "a bit" weird.

Ask sqlite to check foreign keys Related to #4846892

Add reference from etype table's eid column to the entities table Related to #4846892.

[server] change order of entities table modification vs entity creation/deletion In order to have a foreign key referencing the entities table, the insertion into entities must happen first, and the deletion must happen last. For the deletion case, this means we need to: 1. delete all relations (cascade) 2. delete the entities themselves 3. delete the corresponding lines from the entities table This means the _delete_info{,_multi} methods can't keep doing 1 and 3. Thankfully the "public" delete_info method appears to be unused, so drop it. Related to #4846892.

Use foreign keys for relations tables Closes #4846892

Add foreign key for inline relations Related to #4846892

Use our version of schema2sql Related to #4846892

Steal schema2sql module from yams It has no user inside yams, and we need it here if we want to be able to reference the (cubicweb-only) entities table from other tables. Related to #4846892

[dataimport] fix pylint detected error in SQLGenObjectStore

[restore/portable] properly restore the numrange generator If the dump format version is 1.1, read and restore numranges. Closes #4870338

[backup/portable] properly write the eid numrange generator (the eid is no more a sequence) Bump format version to 1.1. Related to #4870338

Added tag cubicweb-version-3.20.4, cubicweb-debian-version-3.20.4-1, cubicweb-centos-version-3.20.4-1 for changeset 49831fdc84dc

[pkg] 3.20.4

[schema] set permissions that do not allow edition on computed relation. Closes #4903918 Also, drop 'del schema' from test which clutters the code for no benefit.

[computed attribute] ensure attribute's formula apply only to the correct type. Closes #4901163

[schemaserial] fix detection of CWComputedRType table, closes #4901150

[dataimport] source.add_info doesn't take anymore a 'complete' argument Closes #4891550

[dataimport] massive renaming of session to cnx and make NoHookObjectStore working with the new connection handling. Closes #4891547

[web/data] use the right selector for hiding jquery treeview placeholders Closes #4903927

merge 3.19.9 into 3.20 branch

Added tag cubicweb-version-3.19.9, cubicweb-debian-version-3.19.9-1, cubicweb-centos-version-3.19.9-1 for changeset b7c373d74754

[pkg] 3.19.9

[hooks] don't insert use_email relation if the entity on either side is going away Closes #4912946

[undo] fix deletion of cw_source relation Closes #4912882

[hooks] don't insert an owned_by relation for deleted entities If, in the same transaction, a composed entity is created and deleted, we'll have scheduled a SyncOwnersOp which would end up creating an owned_by relation for the deleted entity. Closes #4846883

[etwist/service] adjust for a signature change Closes #4878570.

[rset] fix crash on displaying rset where some symmetric relation is used. Closes #4739253 Avoid a KeyError.

Fix handling of limit within the primary view for related elements. Closes #4723579 * respect limit set by uicfg (like the 'autolimited' view itself does) * set limit in rql query to one more than the specified one so the view knows there are more things to display

[native] Change SQL errors log level. - The SQL error itself is logged as 'info'. - The 'transaction has been rolled back' message is logged as 'debug'. These log entries should not be critical: - If an error occur at the sql level, it will be transmitted as a CW error. The exception is raised and handled in a higher level function, which will decide if it is critical or not. - Admitting that rolling-back automatically on errors is wanted in some cases, it cannot be considered 'critical'. The caller will receive the original exception anyway, it is its job to decide if it is critical or not. Closes #4801120

Added tag cubicweb-version-3.20.3, cubicweb-debian-version-3.20.3-1, cubicweb-centos-version-3.20.3-1 for changeset 7d3a583ed539

[debian] build-depend on markdown for documentation build

[devtools] Make PostgresTestDataBaseHandler multi-use friendly Dont init the pgdb twice on the same datadir - If two test case with the same apphome use a Postgres configuration, the handler is initialised twice, so it has to check if the pgdb directory has already been initialized. - Work with the realpath of the pgdb In some cases, the self.config.apphome will resolve symbolic links, but not always. It can result in an attempt to start twice the pg server for the same directory, in the same test run... resulting in failure. Closes #4875827

[devtools] restore i18n of string removed in 4001cfe2f44d Removing that string from the po files means that after running i18ncube against newer cubicweb, we lose the translation when running on older cubicweb's, which is probably not a good idea.

[pkg] fix setup.py install One day I'll get this thing right. Maybe.

[pkg] 3.20.3

[server] Increase the stacklevel (due to @_open_only decorator)

[pkg] install jquery-treeview

[migration/3.20] sync rtypes before etypes We need to make sure the rtypes are inlined before adding unique indices.

[doc] Fix class name for ComputedRelation; fix code block about computed attributes

[web] Allow propagating 'klass' parameter on tree views recursively Closes #4881302

[utils] Fix define_var in ajax context (closes #4881301) Use namespaced <cubicweb:script> instead of just <script>

[web/data] Never hide JS errors in our Deferred implementation Modern browsers have good debugging tools, let exceptions go all the way up. This only affects exceptions raised in the js code, if there's an issue on the server/python side, they'll keep going through our error callbacks. Closes #4881300

[web/data] Fix userCallbackThenUpdateUI to completely replace old DOM elements Without the 'swap' parameter, userCallbackThenUpdateUI() would replace the _children_ of the component with a new version of the componenent, thus creating nested divs/spans/buttons/etc. Closes #4881299

[web/views] Stop mishandling the fieldset name in the default form renderer While not really documented, it is generally assumed that: * the fieldset is a user-visible string (much like a title), * it will be translated by whichever piece of code uses it. So: * fix AutomaticEntityForm to generate untranslated fieldset names, * fix FormRenderer to properly translate the fieldset name before rendering it, * stop using the field set (which may contain whitespace) as a CSS class name (the feature was unusable, so just drop it), * properly handle the colon as part of the translated fieldset name Closes #4881298

Added tag cubicweb-version-3.20.2, cubicweb-debian-version-3.20.2-1, cubicweb-centos-version-3.20.2-1 for changeset 138464fc1c33

merge 3.19.8 into 3.20 branch

[pkg] fix mistake in c7b218125c25 We ended up with the wrong filenames

Added tag cubicweb-version-3.19.8, cubicweb-debian-version-3.19.8-1, cubicweb-centos-version-3.19.8-1 for changeset efc8645ece43

[server/test] convert new test to 3.19 API

[pkg] 3.19.9

merge 3.18.8 into 3.19 branch

Added tag cubicweb-version-3.18.8, cubicweb-debian-version-3.18.8-1, cubicweb-centos-version-3.18.8-1 for changeset 231094063d62

[pkg] 3.18.8

[security] Test case and fix for an INSERT security hole 7099bbd685aa introduced a untested corner case in which an Entity with no attribute specified could be created whatever the permissions. Report and test case by Christophe de Vienne, fix by Aurelien Campeas. Thanks! Closes #4854359

[pkg] 3.20.2

[debian] cubicweb 3.20 breaks cubicweb-bootstrap << 0.6.6 Closes #4842316

[pkg] don't include extra files in tarballs Blind wildcards for dirs that may contain generated or temporary files can lead to trouble and unreproducible results.

[pkg] exclude generated js_api documentation from tarballs

[doc] remove old and unused argouml files

[web/data] remove unused xcf files

[doc] remove nonexistent css from theme

[doc] Drop autoclass directive for IDownloadableLineView This class has been dropped in 3.20.

[pkg] include missing files from doc/book/ in release tarballs Closes #4832700

[datafeed] Set headers parameter to an empty dict in DataFeedParser.retrieve_url This parameter is then passed to urllib2.Request, which expects a dict and won't check for None. Closes #4842333.

[views/forms] Fix EntityFieldsForm.link_to when parent entity is being created When the parent entity is being created and the inlined form is displayed, the parent entity has no eid yet and `peid` is usually 'A'. So int() raises a ValueError. Closes #4627739.

[entities] Optimize CWUser.properties. This property gets called each time a user connects to a repository, which means a lot. This optimisation cuts its execution time by 5 according to my tests

[schema] Typo and indentation fix in RQLVocabularyConstraint doc string

Added tag cubicweb-version-3.20.1, cubicweb-debian-version-3.20.1-1, cubicweb-centos-version-3.20.1-1 for changeset 43eef610ef11

[pkg] 3.20.1

i18n update (closes #4826490)

merge 3.19.7 into 3.20

merge two default heads

Added tag cubicweb-version-3.19.7, cubicweb-debian-version-3.19.7-1, cubicweb-centos-version-3.19.7-1 for changeset ac4f5f615597

[pkg] 3.19.7

Added tag cubicweb-version-3.20.0, cubicweb-debian-version-3.20.0-1, cubicweb-centos-version-3.20.0-1 for changeset 7e6b7739afe6

[pkg] prepare 3.20 release

[skeleton] simplify debian rules file

[migration] don't crash when deleting a non-existing rtype When asked to delete a relation type, instead of checking if it was computed or not, just delete both the CWRType and CWComputedRType.

[debian] include "metapackage" in the cubicweb package's description Lets lintian know it's ok for it to be empty.

[debian] ensure all packages use ${misc:Depends} and ${python:Depends}

[debian] update cubicweb-dev's lintian override Switching from python-support to dh-python changed the installation path.

[server] add a db-namespace option in source definition (closes #1631339) "namespace" is preferred to the postgresql term "schema" to try and avoid confusion with cubicweb's schema. cubicweb now depends on logilab-database >= 1.13.0

[serverctl] add db_transaction context manager

[server] refactor cleanup() code used by "cubicweb-ctl delete" Try to make each logical step (i.e. "delete database" and "delete user") a bit more visible and explicit. This is a preparatory work to easily plug the "delete namespace" feature.

[views] visual refactorings (closes #3803685) - pageContent: - remove the shadow - add a light border - slightly round the corners - left column: - ensure width make it aligned with the logo (header) - ensure the top border of the first box is aligned with the pageContent div - do not use serif font for box headers

[web] everything old is new again Kill the "new" cubicweb.css, and move cubicweb.old.css back in its place. Closes #4763360.

[pkg] bump version to 3.20.0

[views] use icons for user menu and login (header) (closes #3803684) - use icons from the embedded pictograms font for: - authenticated user menu (userActionsBox) - login (CookieLoginComponent) - do not display "anonymous" when not logged in

[utils] provide a function to return an admin connection from an appid Closes #4723580.

[debian] Install wsgi directory in cubicweb-web package Closes #4752375.

[test] fixes to make tests pass with recent lxml lxml (on jessie at least, ie. 3.4.0) does not remove endlines anymore.

[server] Remove useless 'is None' checks 'pb' is never none since 3386fd89c914 when support for the 'APYCOT_ROOT' env var was removed.

[security] check attributes: dispatch on the "add" action if entity was just created cw_set on a just-created entity (i.e. created in the same transaction) should behave the same as setting the attribute directly on creation: check the 'add' permissions, not 'update'. Closes #4740310.

[doc] Add BigInt to the list of built-in types

[devtools] call turn_repo_off in tearDown (closes #4673491) lets it catch leaked sessions even when running a single test

[book] typos / clarifications in create-instance section Clarify that cubicweb-ctl upgrade needs to be run whenever cubicweb or a cube is upgraded. Related to #2632425.

[repository] don't mangle the stack trace on exception

[pkg] Make twisted recommended only It's not necessary when using wsgi or pyramid. Closes #4639508

[devtools] Use the pause_trace context manager instead of pause_tracing/resume_tracing The later were deprecated in logilab-common 0.63.1

[serverctl] rename remove_cube to drop_cube (closes #4545093) For the sake of consistency, since commands are currently named: - add_{cube,entity,attribute,relation} - drop_{entity,attribute,relation} - remove_cube

[wsgi] add tornado http server

[wsgi] add message about the active interface Similarly to what done in "stdlib".

[cwctl] rewrite wsgi method choices This will help devs to add new choices by adding their callback to WSGI_CHOICES. Note: ``options`` becomes a property because ``wsgichoices()`` should be evaluated at the last-minute.

[skeleton/debian] use dh_python2 instead of dh_pysupport for new cubes The former is deprecated.

[web] Add source for jquery.qtip.js Downloaded from https://github.com/Craga89/qTip1/raw/master/1.0.0-rc3/jquery.qtip-1.0.0-rc3.js Related to #3851121.

[web] Replace minified copy of jquery.flot.js 0.6 with non-minified version Related to #3851121

[web] Update jquery-treeview to the latest version Files copied from commit 61f230828ace1b54c2cc54bc549ece261b11842e of https://github.com/jzaefferer/jquery-treeview

[web/css] move jquery.treeview.css override to a separate file Patched embedded code copies are a maintenance disaster (even more so than plain embedded code copies). Let's not do that.

[web] Stop patching jquery.treeview.js This logically reverts part of e9b7cd2e9012 "allow treeview to work correctly in a tab #345293". The treeview and ajax code have suffered a number of changes since then, this change doesn't seem to be necessary anymore. It is most likely unneeded since f65208c9dbdc "[javascript] use jQuery.one('ajax-loaded') instead of jQuery.bind() in add_onload to avoid multiple callback executions".

[web] Beautify jquery.treeview.js Fully mechanical, just whitespace changes.

[entities/wfobjs] add missing `DBG_SEC` debugging informations and a new `transition` capability

[warnings] put an end to warnings in the sqlite driver over `str` being sent instead of unicode strings

[web/request] clearly mark user_callback-related methods as deprecated The deprecated decorator not only emits a warning at run-time but above all makes the deprecation immediately apparent in the source. closes #3567793.

[book] document __unique_together__, remove bad RQLUniqueConstraint example RQLUniqueConstraint should be avoided whenever possible. Related to #3753250.

[schema] stop using RQLUniqueConstraint (closes #3753250) The last uses are replaced with unique together constraints.

[dataimport] Have ucsvreader's API match that of csv.reader (closes #3705701) 'separator' and 'quote' are replaced with 'delimiter' and 'quotechar'.

[utils] Add a '_cwtracehtml' GET parameter to trace self._cw.w() calls (closes #4601327) The core of this patch is in UStringIO.write(). When tracing is enabled, write() doesn't just append the 'value' argument to the underlying list. Instead, a stack trace is recorded and a special HTML "source" is formatted. The output with tracing enabled is an HTML page, with the original HTML escaped, and made clickable to show the stack trace when the write() call was done. This allows answering the recurring question: "who wrote this tag here?!"

[datafeed] Add a raise_on_error parameter to DataFeedSource.extid2entity And pass the option from various `process_` methods in existing parsers. This makes debugging in tests easier. Closes #4601191.

[test/view forms] use the official, undeprecated API

[rset] kill the rset._rqlst cache Right now it "works" for the standard, internal uses. However when we will fold ClientConnection and Connection, it will hurt, because suddenly we get more cache hits, and the following situation would become commonplace: * there is an un-annotated _rqlst given by the querier * some view (e.g. facets) requests the .syntax_tree, which takes a copy of _rqlst * the view actually expects the rql syntax tree to be annotated, but it was not, hence we crash. Related to #3837233.

[source/native] session -> cnx Also swap process_results arguments order to make cnx come earlier.

[hookstests] change got/expected order

[config] kill a getattr

[tests/web] switch previous commit (136b5f995f8) to the new test api Related to #3670209.

[web] remove unused argument from treeview._init_headers

[doc/3.20] mention CWEP-002

[web] stop accepting the magic __message form parameter This has been deprecated for a while, and replaced by _cwmsgid, which doesn't allow arbitrary content.

[web/request] add security_enabled method Just forward to the underlying repo connection. This can be useful in the ORM when dealing with a "code-only" attribute, i.e. something that shouldn't be directly user-visible but must be accessible from a web request.

[dataimport] Fix use of _create_copyfrom_buffer() (related to #3845572) This is used indirectly by SQLGenObjectStore when running on a PostgreSQL database. Regression introduced by commit 70056633085c.

[c-c configure] make it possible to specify the section for sources configuration (closes #3477678)

[server] fix 'cnx' variable confusion in DBG_SQL exception case The rollback handling expects 'cnx' to be the cubicweb Connection, but the DBG_SQL block was replacing it with an sql connection, leading to lulz down the line. Also remove obsolete getattr (the sqlite wrapping is now done at the cnxset level, so cnx.cnxset.cnx should be the right thing already).

Provide sufficient context to check 'delete' permission in AjaxEditRelationCtxComponent Call rdef.check only when both fromeid and toeid are available. Though only call it once (for the first encountered related entity). Factorize a bit to keep handling of CSS/JS addition the same. Closes #3670209.

merge 3.19.6 into 3.20 branch

Added tag cubicweb-version-3.19.6, cubicweb-debian-version-3.19.6-1, cubicweb-centos-version-3.19.6-1 for changeset 934341b848a6

[test] missing unittest.main() call in unittest_http_headers.py

[test] make unittest_webconfig independant of CWD if apphome is unset in ApptestConfiguration constructor, this later looks for a 'data' directory in CWD, making this test runnable only from web/test (and pytest do chdir before executing tests, so we did not noticed this problem).

[webctl] do not ask questions if verbosity is at 0 Closes #4641960.

[web] Slight tweak on jQueryUI's colors on links (related to #4564046) Follow up to commit 16f550b48d57 which was somewhat brutal. This change only overrides jQueryUI's CSS rules inside tab panes and leaves actual tab buttons alone.

[view] Remove unused imports

Remove obsolete __future__ imports Generators have been available since 2.3, "with" statements since 2.6.

[devtools] Remove unused local function Unused since cw's first commit.

[uihelper] Fix a possible NameError in meta_formconfig could occur when using FormConfig with a custom uicfg object. Closes #3802298.

[devtools] Avoid db_cache collisions and mis-loading This is done by stopping to share the db_cache between TestDataBaseHandler instances It may seem overkill, but is the only way I could get it to work properly. I think the whole db caching code needs some rework to better work with different db handlers. Closes #4601328

[devtools][pg] Remove a wrong assertion in _backup_database When using pre_setup_database, a repo gets initialised before calling backup_database. Hence self._repo is NOT None it that case.

[pkg] prepare 3.19.6

[merge] backport 3.18 fixes

Accept '==' operator in cubes dependencies '==' is the operator used in the python dependencies specifications. See PEP 440 http://legacy.python.org/dev/peps/pep-0440/#version-specifiers Using the same operator in cubes dependencies allow to make setuptools handle the cubes dependencies while keeping cubicweb happy about the dependencies. Closes #4375144

[session] don't silently ignore commits commits while in the precommit or postcommit phase are silently ignored. This is OK in precommit since there'll be a commit afterwards anyway; OTOH in postcommit it may surprise the developer who didn't read the documentation for postcommit operations carefully enough. Related to #4383922.

[migration/bootstrap] add explicit index removal for sqlserver, be tolerant to migration replay Closes #4618944.

Added tag cubicweb-version-3.18.7, cubicweb-debian-version-3.18.7-1, cubicweb-centos-version-3.18.7-1 for changeset cb96f4403cf2

[pkg] prepare 3.18.7

[migration/3.18] speed up the migration and be a little bit more informative Closes #4619277.

[schemaserial] be robust against duplicated CWUniqueTogetherConstrain entities arising from bug in the pre 3.18 era Closes #4619278.

[server] Handle unique constraint violations under recent sqlite The error message changed from "columns foo, bar, baz are not unique" to "UNIQUE constraint failed: table.foo, table.bar, table.baz". Closes #3564510

[merge] backport 3.17 fixes

Added tag cubicweb-version-3.17.18, cubicweb-debian-version-3.17.18-1, cubicweb-centos-version-3.17.18-1 for changeset cda4b066933f

[pkg] prepare 3.17.18

[source/native] handle newer Integrity error messages from sqlite > 3.7 Similar to a0cf2993b6d3 that was done for 3.18.

[source/native/backup_restore] have a unique tunable blocksize for the dump phase Thus we make memory errors much less likely. Closes #4618949.

[urlpublishing] handle sub-types in RestPathEvaluator Since commit 31a1813d53f3 "[entity/url publishing] fetch_rqlst should use 'is_instance_of' instead of 'is'", the RestPathEvaluator's input rset can have more than one etype, so it can't always use sameetypelist. Related to #3825488

[views] fix ecsvexport selector introduced by e48e5a597ccc Fix the view and make web/test/test_views.py pass again.

Add custom checker for Password values We override the default converter to pass Binary values through, but don't do anything about the checker. This worked previously because yams allowed StringIO instances, although its converter didn't do the right thing for them. Fixing this in yams requires that we properly register a checker.

[component] give Links a __repr__

[debian] move markdown dependency from cubicweb-web to cubicweb-common Pointed out by David Douard.

[web/auth] stop playing games with locals() We only want to know if any retriever found something.

[entity] rather than crashing with an IndexError on entity creation, raise a meaningful exception

[web] partial backout of #8391bf718485 to restore RelatedObjectsVComponent Apparently the intended deprecation did not work, and this component is used in at least the tracker cube.

[server] typo fix

[devtools][pg] Activate the non-durability settings of Postgresql. It speed up the PG tests by a factor of 3. See http://www.postgresql.org/docs/current/static/non-durability.html.

[server] fix LoginPasswordAuthentifier.authenticate some kind of typo there.

[web] Override jQueryUI's colors on links (closes #4564046)

[hooks] Fix precommit event logging message

[service] allow repo_stats for users It's used by the siteinfo view, which is available to managers and users. This change prevents a crash in that view.

[views] Escape class attribute value in CWGroup incontext view

[devtools] use a specific test_db_id when disabling anon The anonymous user is created (or not) in postcreate, which for tests means when creating the cached empty database. The anonymous_allowed=False test classes should thus not share their template db with other test classes, otherwise either they end up with an unexpected anonymous user, or the others miss theirs and things fall apart.

Restrict yams version CW 3.19 is not compatible with yams >= 0.40

[facets] Honor 'start_unfolded' facet attribute (closes #4502799)

[facets] Correctly look for inputs of type "hidden" (closes #4502768) jQuery ':hidden' selector looks at CSS visual properties (eg, 'display', 'visibility'). The intent here was probably to look for inputs of type "hidden", which many facets use to store user selection data (eg, FacetRangeWidget). The problem is that regular text inputs (eg the "has_text" facet which has a '<input type="text"/>') will be picked up by this selector if they are inside a folded facet. Chaos and destruction ensue.

[web] Cache results from 'i18n' ajax controller (closes #4487856) This is a simple transient cache which will be forgotten as soon as the user leaves the current page.

[web] There is no global noop(), use jQuery's instead (closes #4487832)

[server] Add missing import of logilab.database Closes #4469407.

[migration] hackish black magic to bootstrap addition of formula attr during migration Turns out we can't add an attribute to CWAttribute before the attribute exists. add_attribute generates rql with a 'formula' relation, which CW doesn't know about yet, so things get unhappy. Once that is fixed, we need to make the CWAttribute addition operation deal with CWAttribute entities that don't have a formula yet. Finally, move the migration to bootstrapmigration_repository so it happens early on, and use add_entity_type rather than add_relation_type for CWComputedRType.

[server] commit after serializing schema Use separate transactions for schema serialization and postcreate scripts execution, so that the latter can rely on the schema being persistently stored. This matters e.g. if something in a postcreate violates a unique_together constraint, we need the constraint persisted in order to raise a meaningful error. See #4525069.

[book] fix docstrings to please sphinx example rst roles in docstring must be protected (in a literal).

[book] several fixes in the rst files - prevent duplicate markup definitions, - use existing Workflow markup instead of redefining one (in baseschema.rst) - include dataimport - fix non-existant directive autodocfunction - remove the empty 'embedding.rst' file

[book] activate the viewcode extension Automagically make link to class/modules/etc. it finds in the documents.

[book] new theme based on pyramid theme (closes #4291287)

Remove unused lgc.interface imports and leftovers

[repository] provide a .new_session entry point The current .connect only returns a `sessionid` which must be given to repo._get_session(...) to get the real session object. Related to #4151635.

[devtools/testlib] Use actual 'admin' user configuration values Previously ``db-user`` and ``db-password`` configuration values were used to fill ``admlogin`` and `admpassword`` attributes of CubicWebTC. The correct data for these should read from `[admin]` section. Closes #4065070.

[forms] closes #2437859 - Detect and prevent concurrent edition of the same entity. Add the timestamp of form generation to each entity's meta-information fields. On form validation, check that no concurrent change is overwritten and raises a ValidationError in case of concurrent change. A nicer handling with a message and a link to the new version of the entity would be a good thing...

[migration] make sure the repo knows about all constraint types Adding new constraint types is cumbersome. It's easy to forget to do it in a migration script. So sync the constraint types at the beginning of each migration. Closes #3724892

[migration] stop caching the mapping from constraint type name to eid It's not so frequent that a cache seems necessary, and we were not invalidating that cache when adding a new constraint type. Related to #3724892.

[web/json] an empty rset is just fine for ejsonexport The ejsonexport view can just return an empty list for an empty rset. Closes #4005518

[RichString] Add markdown support Supporting markdown requires the 'Markdown' python packages, and since it widely available on the target platforms adding it as a strong requirements will not be a big constraint. Closes #3814302

[ldapfeed] Reduce default value for user-attrs-map option (closes #3824889) This is needed because lgc.configuration.Configuration does not allow removing key/value pairs from the default (due to its use of dict.update() internally). Since CWUser.login is required, users of the add-source command will always be able to override it.

[serverctl] Ask for URL when adding a new source (closes #3824868)

[serverctl] Ask for parser type when adding a new source (closes #3484231) Remove 'quick_start' option to load all AppObjects, including feed parsers.

[doc/3.20] more details on removed code related to #3799117

remove 3.11 bw compat Closes #3799117.

remove most 3.10 bw compat Related to #3799117. The boxes and entityvcomponent objects cannot really be removed as they are still used throughout the code base (and possible cubes). This may be caused by a non-working deprecation and needs some more work.

remove leftover pre 3.9 deprecation warnings

[utils] Remove function-in-the-middle call

[web] set Vary response header to "Accept-Language" when using content negotiation This is slightly annoying because the response actually only varies based on the language we decide to send, which has much fewer possible values than Accept-Language, but that's not in the request, so we can't easily use it. Deployments using varnish or similar and controlling the set of available languages will likely want to override this to allow reasonable amounts of caching. Closes #2105812

[web/cors] don't overwrite other Vary headers Vary is a list of request headers, we shouldn't override others.

[web] add support for HttpOnly cookie flag And use it for session cookies. Closes #4142521.

merge 3.19.5 into 3.20 branch

Added tag cubicweb-version-3.19.5, cubicweb-debian-version-3.19.5-1, cubicweb-centos-version-3.19.5-1 for changeset 3ac86df519af

i18n update 4 strings have disappeared.

[pkg] 3.19.5

[server] hold connection to the db in tx_actions We can be called without a cnxset (e.g. from repoapi).

[wsgi] If multipart cannot parse the POST content, let it pass. multipart can only parse html form data. It the content_type is, for example, "application/json", get_posted_data should not fail but just stop trying to read the content. Closes #4421845

[devtools] Fix JS tests' HTML code

[devtools] "Keep" some temporary files/dirs around to help with debugging The whole QUnitTestCase runs with an @with_tempdir so it's redundant anyway.

[devtools] Fix Firefox launcher in QUnitTestCase (closes #4294727) The main changes are: - stop creating the profile, firefox will create it - point firefox to a profile directory instead of giving it a profile name (this has the added bonus of not polluting the user's profile list) - start firefox once and kill it 5 seconds later to let it finish its profile creation (along with system-wide extensions setup)

[devtools] allow cross-origin requests for qunit We have a mix of file:// html and http:// ajax calls. Which we should at some point fix to all be http, but. Related to #4294727.

merge 3.19.4 in 3.20 branch

[cors] Fix CORS headers generators The Access-Control-Allow-* and Access-Control-Expose-Headers are response headers, not request headers. Hence, we need generators for them. Closes #4412575.

[wsgi] Fix posted files filename reading The filenames are parsed by multipart.parse_form_data, which does the unicode decoding. Trying to re-decode the filename was leading to an error.

[pkg] Depend on Pillow instead of PIL The Pillow library is becoming the de-facto replacement for PIL. It also is way simpler to install with pip than PIL. Closes #4411354.

Added tag cubicweb-version-3.19.4, cubicweb-debian-version-3.19.4-1, cubicweb-centos-version-3.19.4-1 for changeset c4e740e50fc7

[pkg] 3.19.4

merge 3.18.6 into 3.19

Added tag cubicweb-version-3.18.6, cubicweb-debian-version-3.18.6-1, cubicweb-centos-version-3.18.6-1 for changeset d91501356742

[pkg] 3.18.6

[hooks/security] allow edition of attributes with permissive permissions If an attribute has more permissive security rules than the entity type itself, we should be green and not deny action because of an early global entity permission check (with the more restrictive rules). Only if one attribute with the entity-level permission rules is edited will the global check be performed. Note: * the "if action == 'delete'" check at the entry of check_entity_attributes is a guard for a condition currently not happening in cubicweb itself (but application hooks could conceivably call this function with a 'delete' action) Closes #3489895.

Almost backout afcd46716d6a which breaks _select_best raising an ambiguity exception in debug mode. The problem is, before afcd4, *tests* ran in debug mode and we want this (e.g. we want to show, rather than swallow, select ambigüities). We juste replace the bogus __init__(vreg.config) by __init__(True), which is practically equivalent and also much more clear.

[server] fix anonymous_user predicate in tests devtools' TestServerConfiguration overrides the anonymous_user method, but not the anonymous-user config option, so testing for the latter would give the wrong result. Closes #3996664.

[entities] cw_rest_attr_info() should only consider required attributes (closes #3766717) This prevents CW from choosing unique but non-required attributes. None/NULL is a poor choice for RESTful URIs.

[views] csvexport accept an empty rset (closes #4236928) When you tried to apply the 'csvexport' view on an empty rset, the view couldn't be selected and you got a HTTP 500 error. Also add two new test cases.

[views] Display attributes in entity creation form based on "add" permission Previously, the "update" permission was used. Hence in case the latter is more restrictive that the "add" permission, an user may not be able to set such an attribute, despite she may have "add" permission on it. As a result of the change of permissions action in `editable_attributes` method (add/update depending on whether the entity is being created or modified), the "eid" attribute would have shown up in the edition form. To avoid this, it is moved in the "hidden" section (where it should arguably belong anyways). Closes #4342844.

[datafeed] Commit after all deletions in datafeed parser This avoids misleading validation error because schema constraints could be temporarily broken depending on the deletion order. Closes #4372127.

[schema] CWComputedRType is a schema type Hide it from the default views.

[doc] proofreading CWEP002 section

[CWEP002] document computed relations and attributes Related to #3546717.

[CWEP002 migration] support sync_schema_props_perms for computed attribute

[CWEP002 migration] support drop_relation_type/drop_relation_definition/drop_attribute for computed attributes

[CWEP002 migration] support add_relation_type/add_attribute for computed attributes Related to #3546717.

[CWEP002] properly handle serialization of computed attributes Until this cset, only serialization of computed attribute's formula was handled (thanks to the relation definition's properties dict). We now: * properly serialize/deserialize attribute's formula * test first if the database has been migrated and contains related column or not Related to #3546717

[CWEP002] Add support for computed attribute synchronization Related to #3546717. Test and handle the behaviour with several formulas and identified use cases. To do so add a birth year and a computed attribute age to Person in the test schema.

[CWEP002] Add schema finalization checks for computed attributes ``finalize_computed_attributes`` essentially checks that computed attributes types match with the type declaration of the attribute. Related to #3546717.