[server/test] delete commented-out test Related to #3933480.

[connection] remove the `mode` attribute It is now unused. While removing mode, we also drop some sqlserver-specific reconnection logic snippets. These had several downsides: * untested * partial coverage * done at the wrong level Related to #2919309.

[connection] remove ensure_cnx_set context manager uses It has been deprecated in a previous changeset and is now a no-op. Related to #2919309.

[server] remove cnxset tracking, it is now unneeded Indeed, all cnxsets will now be acquired using the following API:: with session.new_cnx() as cnx: cnx.execute(...) # do stuff cnx.commit() These well-scoped blocks are the only place a cnxset can be acquired. The old use-case for un-delimited cnxsets (pyro/zmqpickle protocols) have been removed. The "mode" of connection objects becomes "write" forever (it will be removed in a couple of changesets). Related to #2919309.

[devtools] remove the remaining bw compat for old-style tests It is just: * dangerously untested right now * on the path of further bw compat removal (cnxset handling) webtest drops an apparently pointless .login call Related to #3933480.

[session] remove session-related bw compat code Related to #3933480.

[session/transaction] removed unused transaction class Related to #3933480.

[transactions] cleanup after the dbapi removal * end req vs cnx dichotomy * Transaction takes a cnx at __init__ time * a couple of super calls are fixed Related to #3933480.

[session/cnx] remove `session_handled` logic Related to #3837233 .

[web/sessions] reduce repoapi usage where possible Using session.new_cnx() should be the new preferred way. Related to #3837233.

[entity,storage] remove hackish code from the bfss path The ClientConnection/Connection duality brought some painful hacks that are no longer needed. Hence we can send entity._cw_update_attr_cache and ._cw_dont_cache_attribute to the grave. Related to #3837233

[connection] eliminate ClientConnection leftovers * in the documentation * in the docstrings * in code notations (e.g. clt_cnx -> cnx) Related to #3837233. From now, there should be just one ClientConnection left in repoapi.py.

[session] all cnx._session become cnx.session The _session belonged to ClientConnection, which is just gone. Related to #3837233.

[repoapi] fold ClientConnection into Connection Connection replaces ClientConnection everywhere. Some notes: * testlib: .client_cnx and .repo_cnx become aliases of .cnx (we might not want to tell people to update their tests again for just no real benefit, so we'll live with these aliases for a while) * entity.as_rset must not be cached because we risk caching result sets having a Connection object as .req (helps unittest_breadcrumbs) * entity._cw_dont_cache_attributes loses its repo/request special paths and only keeps its storage/bfss user (this helps unittest_wfobjs) * moreover, entity.cw_instantiate and .cw_set stop overriding the attributes cache *after* the before_*_entity hooks have run, because there is no need to (it is now actually harmful to do it and unittest_hooks.test_html_tidy* tests remain green because of this) * rset._build_entity sticks its .req onto the entity just fetched from the cache, because otherwise it might carry a _cw that is a Connection object where a Request is expected (helps unittest_views_actions) * we get overall better cache usages (entity caches were split over Request + ClientConnection and Connection), hence the changes unittest_entity and unittest_wfobjs * void the ecache when providing the cnx to a request object Having the entity cache pre-filled when we bind it to the request object hurts because these entities are bound to Connection objects, that lack e.g. `.form` or `.add_js` and crash the views subsystem. Thus, the unittest_testlib.test_error_raised test will are kept green. Closes #3837233

[migractions] remove any session related leftovers Also, the migration handler really wants a "repo" session anyway. This highlights a bug in unittest_wfobjs.test_duplicated_transition where an old connection is reused. It probably used to work because it was made of session + set_cnxset magic dust. Related to #3933480.

[session] provide missing apis to Connection From this point, Connection provides a superset of ClientConnection. Related to #3837233.

[connection] replace .running_dbapi_query with .hooks_in_progress The thing was badly named. It tries to help distinguish between queries issued directly by the programmer (e.g in the views: cnx.execute(...)) from queries issued from the hooks, operations ... or even the repository or the native source objects. It worked heuristically being associated with the security being disabled. We provide a better name and an implementation distinct from the security management methods. Related to #3933480.

[dataimport] Turn the module into a package Just `hg mv` dataimport.py and test/unittest_dataimport.py.

[dataimport] Properly escape strings sent to COPY FROM (closes #5278743) See http://www.postgresql.org/docs/9.1/static/sql-copy.html#AEN64296 for escaping codes.

[http] fix cache-control syntax in EtagHTTPCacheManager use , to separate cache response directive closes #5307539

[connection] provide some missing documentation bits The previous changesets remove information about session data and transaction data. We provide the necessary entries in the docstring of the Connection class. Related to #3933480.

[repoapi,session] remove all session-as-cnx backward compat The `dbapi` being gone, we now can drop the session object bw-compatibility layer. This will allow further simplifications, such as folding ClientConnection and Connection (without too much pain), and then having persistent sessions. Related to #3933480.

[server] drop repo.internal_session and InternalSession

Added tag cubicweb-version-3.20.7, cubicweb-debian-version-3.20.7-1, cubicweb-centos-version-3.20.7-1 for changeset 359d68bc1260

[pkg] 3.20.7

[dbapi] retire repo.[gs]et_shared_data which were used by the dbapi Related to #3933480

[dbapi] retire repository transaction methods which were used by the dbapi Related to #3933480.

[dbapi] retire repo.call_service which was used by the dbapi Related to #3933480.

[dbapi] retire repo.commit/rollback which was used by the dbapi Related to #3933480.

[dbapi] retire repo.user_info which was used by the dbapi Related to #3933480.

[dbapi] retire repo.entity_metas which was used by the dbapi Related to #3933480.

[dbapi] retire repo.execute, which was used by the dbapi We will retire or reform pieces, mostly of the Repository class, that were related to the old dbapi way of life. Related to #3933480.

[devtools/testlib] remove dbapi backward-compat Related to #3933480.

[test] fix copy-pasta in doc string

[book] remove dbapi documentation Move some still-relevant doc related to the RQL connection API to the 'sessions' section. It might make sense to move this back outside of the "Repository development" chapter at some point since this is relevant in other contexts.

[uilib] `uilib.js` helper now honors explicit JSString (closes #4959538) Previously, if a JSString was inside a nested container (i.e. list or dict), it was handled as a string, not a verbatim js chunk (see docstring and test). So the following code:: >>> str(js.cw.pouet(1, {'callback': JSString('cw.mycallback')}) resulted in:: "cw.pouet(1, {'callback': 'cw.mycallback')})" while we expect (note the removed quotes):: "cw.pouet(1, {'callback': cw.mycallback)})" We use ``cubiweb.utils.js_dumps`` instead of ``cubicweb.utils.json_dumps`` which is aware of JSString.

[dbapi] remove the dbapi module and its immediate remaining users We suppress toolsutils.config_connect, which has currently only one known user (the email cube), which is being patched. It can be replaced with utils.admincnx function for a local access. Next will come a series to: * remove the session backward compatibility * fold ClientConnection into Connection Closes #3933480.

[web/views] remove dbapi usage from 'gc' view

[sqlite] Fix integrity error parsing Up to sqlite 3.7, integrity error message may be in the singular form if only one column is in the unique index. Closes #5224632

[postgres] fix limit_size regexp to properly remove one char tags like <p>. Closes #5221847

[postgres] _regproc.postgres.sql should be reloadable

[mail] include Date header in generated mails (closes #5271058) Mails without Date gets flagged as spam. Plus it's wrong anyway.

[doc] mention datadir-url configuration option in release notes Related to #5204550

[web] add test for datadir-url option (related to #5204550)

[web] Add datadir-url configuration option (closes #5204550) Makes it possible to use a different host for static data instead of ${base_url}/data

[web] kill https-deny-anonymous option (closes #5193687) It doesn't make much sense to prevent people from using https.

[cwctl] don't filter DeprecationWarnings

[exceptions] avoid calling UniqueExceptionError.rtypes after closing the session

[web/test] Fix broken tests after commit b81adb5e32f9 Add missing n3rdf view to RDF view list.

[web/test] don't delete uicache directory Other tests use it, so the rmtree was a bit too much. Set CACHEDIR to a temporary subdirectory, and only delete that.

[web/views] add text/n3 rdf view Less shit than xml.

[schemas] make CWEType.final default to False (closes #5049201)

[repository] replace session with cnx in a few places

Don't pass an encoding to rqlst.as_string()

[rset] Deprecate the 'encoded' argument to ResultSet.printable_rql() It's hardly used at all, and when used inside CubicWeb, it's only set as false. Better make sure that this function always returns a proper unicode string and let the caller handle conversion on its own.

[gcdebug] ignore _NeedAuthAccessMock instances When looping over gc.get_objects(), we'd crash in _NeedAuthAccessMock.__getattribute__.

[views] Fix 'gc' view to use 'repo_gc_stats' service

[services] Fix 'repo_gc_stats' to return a list of unreachable objects' repr

[services] Prevent 'repo_stats' service from aggregating information It's up to the consumer of the service to decide whether it wants aggregated data or not... Ensure the /siteinfo page rendering is not modified.

[hooks] add a looping task that dumps the stats regularly in a file

[web/facet] Use an Exists node in HasRelationFacet The generated RQL now includes an EXISTS clause. For instance, one gets: :: DISTINCT Any WHERE X is CWUser, EXISTS(X in_group A) instead of: :: DISTINCT Any WHERE X is CWUser, X in_group A for a HasRelationFacet with ``rtype = 'in_group'``, which is more appropriate for testing the existence of the relation. Add a test for this facet along the way.

[debian] Add a Breaks for cubicweb-folder which stills use GMTOFFSET Related to #2154655.

[web] kill GMTOFFSET (closes #2154655) Its last user, set_cookie(expires=...), can just as well interpret its argument as UTC. Naïve datetime objects with an implicit non-UTC timezone are just awful.

[web/request] don't play tricks with utc offset We need an UTC datetime from a time.time()-type value. It turns out datetime provides a method to do just that. Related to #2154655.

[server/test] Drop dependency on folder cube Just copy schema bits when needed and drop a view (not possible anymore) in unittest_repository. Closes #5168939.

[web/test] Drop folder cube dependency Copy useful schema bits, define IBreadCrumbs adapter for Folder/fild_under (previously relying on ITree adapter) and adjust unittest_viewselector tests to drop a view that does not exist anymore. Related to #5168939.

merge 3.20.6 into 3.21

[web/views] Allow changing CSVView's separator (closes #5227442)

[web/view/treeview] add a 'selected' class on <li> if the entity's url is the currently displayed url An alternative would be to refactor the corresponding view so that controlling css classes wouldn't require overriding a 5O-lines method. Notice by default there is no associated class in the CSS so there is no rendering difference unless one explicitly add rule in his CSS file.

Added tag cubicweb-version-3.20.6, cubicweb-debian-version-3.20.6-1, cubicweb-centos-version-3.20.6-1 for changeset 7f64859dcbcd

[pkg] 3.20.6

[notification] don't block while sending mails Regression from #7c17659c9eae, closes #5190001.

[dataimport] make MetaDataGenerator / NoHookObjectStore usable from a datafeed parser ie make it possible to create entities with external source as cw_source and proper external id. Closes #4891552

[dataimport] don't generate metadata which are explicitly specified Related to #4891552

[debian] prepare 3.20.5-2

merge 3.19.11-2 into 3.20

Added tag cubicweb-debian-version-3.19.11-2 for changeset 6d265ea7d56f

[debian] Update watch file

[debian] Fix cubicweb-dev dependency

Added tag cubicweb-version-3.20.5, cubicweb-debian-version-3.20.5-1, cubicweb-centos-version-3.20.5-1 for changeset 51aa56e7d507

[pkg] 3.20.5

[dataimport] don't insert created_by / owned_by relations when user is the internal manager (eid = -1). Closes #5162943

[repository] don't attempt to delete computed relation, they have no table in the database. Closes #5162935

[migration] sync_schema_props_perms should skip computed relations. Closes #5147796

[schema] typo

[web/test] fix test_views reusing a request for two different view calls leads to badness

[documentation] add note concerning Attribute, TODAY and NOW in yams documentation

[i18n] properly translate error messages related to violated unicity constraint. Closes #5100373

[web/data] Add missing mode='swap' in loadxhtml calls (closes #5087432) Let's avoid putting divs inside divs, dawg.

[web/views] Fix PropertyKeyField widget (closes #5087413) CWProperties don't have unittests, this will likely break again.

[web/views] Fix exception in CWProperty edit form (closes #5087412) Also add missing help message. The overall display is ugly, but at least the message is shown.

[web/request] Restore attribute setting for search_state This was dropped in 1f84295bfe95 when `search_state` attribute was turned into a property, but without a setter. So adding a setter here restoring previous behaviour, as some tests and cubes use it. Related to #4875761.

[web/request] simplify search_state loading, fix comment

[webrequest] Lazy-load search_state - The update_search_state function will hit the session data only if __mode is present in the form. - set_search_state keep the same semantics - Direct access to search_state in the session data is changed by an access to the new property. Closes #4875761

[config] disable fckeditor by default (closes #1368900) FCKEditor is no longer maintained, and has a history of XSS issues. Let's use plain text as default text format.

merge 3.19.11 into 3.20

Added tag cubicweb-version-3.19.11, cubicweb-debian-version-3.19.11-1, cubicweb-centos-version-3.19.11-1 for changeset 1ae64186af94

[migration] reorder system tables / meta-schema migration into bootstrap_migration Also fix a bug in the backwards-compat eid_type_source method where it returned an extid (possibly null) instead of a source name. With these changes, migrating a database from cw 3.12 is now possible. Closes #4962681

[dataimport] add missing import (closes #4985916)

[debian] cubicweb-dev needs cubicweb-etwist Closes #4965410

[pkg] 3.19.11

Binary initializer should accept bytearrays (closes #5084075) Recent versions of pyodbc return bytearray objects for binary data.

[native source] log the used connection in doexecmany as in doexec

[server] use unicode wherever appropriate in transaction/undo related methods Silences lgdb/sqlite warnings about using str instead of unicode.

[web] typo

[entities] return the entity itself from notification_references if the view uses a random message-id Useful for entity update notifications to reference the entity creation message.

[notifications] use non-fixed message-id for EntityUpdatedNotificationView More than one of those can be sent for a given entity, so they shouldn't all have the same message-id. Closes #4962712.

[server/test] kill remaining 3.19 deprecation warnings in undo tests

[devtools] accept str objects in CubicWebTC.new_access and .create_user

[utils] use the "predictable" argument to js_dumps Fixes doctest under python -R. Related to #4959402.

[server/test] don't rely on a sorted output from .object_relations() Fixes test under python -R. Related to #4959402.

[req] make rebuild_url predictable Fixes test with python -R. Related to #4959402.

[web/test] don't depend on iteration order on a set Fixes test under python -R ('pending_insert' in session data is a set, so we can't rely on any particular ordering). Related to #4959402.

[web/test] fix cubicweb 3.19 deprecation warning

[web/test] use unicode instead of str for user logins makes lgdb/sqlite quieter.

[test] Don't depend on dict iteration ordering Fixes test with python -R. Related to #4959402.

[entities] make cw_fti_index_rql_queries predictable Don't depend on the hash function, fixing test with python -R. Related to #4959402.

[hooks/syncschema] use a list instead of a set for UpdateFTIndexOp The order in which etypes get indexed matters. Related to #4959402.

[rqlrewrite] sort possible types when turning is_instance_of into is Helps get predictable output even with python -R. Related to #4959402.

[server/test] use unicode strings for user logins

[web/test] use unicode object as user login (py.warnings) WARNING: /home/jcr/src/cw/cubicweb/devtools/testlib.py:206: DeprecationWarning: Sanitizing an input dictionary with str values, please check your data (key = 'login')

[server/querier] stop passing a rqlst to empty_rset (py.warnings) WARNING: /home/jcr/src/cw/cubicweb/server/querier.py:46: DeprecationWarning: [3.20] rqlst parameter is deprecated return ResultSet([], rql, args, rqlst=rqlst)

[web/views] Use predictable ordering for attributes in entity XML view Fixes test under python -R. Related to #4959402.

[web/views] set explicit order for ManageSourcesAction Gives predictable ordering wrt ManageUsersAction under python -R. Related to #4959402.

[web/test] don't depend on dict iteration ordering Fixes tests under python -R. Related to #4959402.

[web/test] pass login as unicode string silences lgdb/sqlite warning.

[web/test] fix 3.19 deprecation warning

[sobjects/test] Fix dependency on predictable hashing Fix test under python -R by ignoring url query parameter ordering. Related to #4959402.

[test] don't rely on a stable ordering for appobject_path Fixes test under python -R. Related to #4959402.

[server] remove "standalone RQL server" from possible configuration choices Related to #2919297.

Remove leftover bits from start-repository command, and document its demise Related to #2919297

Remove the remote repository-access-through-zmq support Modern methods such as the rqlcontroller cube + the cwclientlib library are the way forward. Closes #2919297.

Remove remote repository-access-through-pyro support Modern methods such as the rqlcontroller cube + the cwclientlib library are the way forward. Closes #2919309.

[web/views/rdf] Take the second element of XY equivalent for non final relation When given a Yams snippet of the form `<etype> <rtype>` ``xy.xeq()`` will return the XML snippet equivalent to this relation in the form of a tuple `(subject, rtype, object)` so ``item`` will always be a tuple here. In fact, the correct code appears just a few lines above (around line 88) for final relations so I guess this is a copy-paste mistake. Closes #4745929.

[server] Make sure 'zmq-address-sub' config is always iterable

[web] split timeline in an individual cube Closes #1279544

[devtools] don't hide warnings and errors during tests The markdown module in jessie turns all warnings into logging calls, so this allows me to still see deprecation warnings when running tests.

[test/i18n] remove unneeded hack

merge 3.19.10 into 3.20

[server/session] Fix typo "meta vs. metas" in describe()

Added tag cubicweb-version-3.19.10, cubicweb-debian-version-3.19.10-1, cubicweb-centos-version-3.19.10-1 for changeset 3bab0b9b0ee7

[pkg] 3.19.10

[i18n] Update translations Related to #5012581.

[config] Lowercase the FQDN we get from the OS (closes #5040345) Browsers convert the host name part of URLs to lowercase. This has the unfortunate effect of triggering various CORS error messages in CubicWeb (because the Origin: header will contain lowercase letters). As per RFC 4343, host name comparison should be case-insensitive. So let's put it in lowercase when we grab it from the host system. If admins put uppercase letters in "host" or "base-url" in their all-in-one.conf, then it's their fault.

[web/views] Honor 'action' attribute on AutomaticEntityForm, closes #4943392 On AutomaticEntityForms, get_action used to bypass the logic from FieldsForm.form_action() which handles the 'action' attribute itself, preventing callers from overriding it at selection time. This capability is used by the inlinedit cube. In a nutshell, we let the deprecated AutomaticEntityForm.get_action() raise an AttributeError, which allows FieldsForm.form_action() to handle it properly.

[docstrings] fix project-wide English language mistake Even though 'u' is a wovel, it may be pronounced 'yu' which is a consonant sound. In such cases precede it with 'a' instead of 'an'.

[doc] Don't pretend we know cubicweb runs on mysql As far as I know it hasn't been tested in years.

[test/i18n] Update reference po file following 75cd7df5c32e This changeset added a (missing) message "add a <etype>", so reference po file needed an update.

[devtools] add a method on CubicWebTC to fake data posted by standard cw forms. Closes #4985805

[web/wdoc] fix links They're relative to ${base-url}/doc/, not ${base-url}. Closes #3161099.

[i18n generation] add missing 'add a EntityType' message to generated catalogs. Closes #5012581

[server/test] fix test_zmq We were only asserting in a thread, thus ignoring the results.

[views,js] remove user_callback feature Related to #3975260.

[newcube] Make the generated code PEP-8 compliant Closes #4925396

Remove unused imports

[ext] Remove use of cStringIO We create a StringIO never to read anything from it. Docutils seems to support setting the 'warning_stream' to False to ignore warnings. So let's just do that.

[view] Use io.BytesIO as stream for binary views

[undo/test] check that the entity was restored even though the fiche relation wasn't

Use a moved_entities table to record external entities moved to the system source Instead of using a negative eid in the entities table, move the record to a new table so we don't have an interval with a missing eid in entities. Related to #4846892

Add unique index on entities.extid

[undo/test] if we try to delete an entity referenced by an inline relation, we fail

[undo] when we can't undo an inlined relation change, point at None, not a broken eid This whole thing is "a bit" weird.

Ask sqlite to check foreign keys Related to #4846892

Add reference from etype table's eid column to the entities table Related to #4846892.

[server] change order of entities table modification vs entity creation/deletion In order to have a foreign key referencing the entities table, the insertion into entities must happen first, and the deletion must happen last. For the deletion case, this means we need to: 1. delete all relations (cascade) 2. delete the entities themselves 3. delete the corresponding lines from the entities table This means the _delete_info{,_multi} methods can't keep doing 1 and 3. Thankfully the "public" delete_info method appears to be unused, so drop it. Related to #4846892.

Use foreign keys for relations tables Closes #4846892

Add foreign key for inline relations Related to #4846892

Use our version of schema2sql Related to #4846892

Steal schema2sql module from yams It has no user inside yams, and we need it here if we want to be able to reference the (cubicweb-only) entities table from other tables. Related to #4846892

[dataimport] fix pylint detected error in SQLGenObjectStore

[restore/portable] properly restore the numrange generator If the dump format version is 1.1, read and restore numranges. Closes #4870338

[backup/portable] properly write the eid numrange generator (the eid is no more a sequence) Bump format version to 1.1. Related to #4870338

Added tag cubicweb-version-3.20.4, cubicweb-debian-version-3.20.4-1, cubicweb-centos-version-3.20.4-1 for changeset 49831fdc84dc

[pkg] 3.20.4

[schema] set permissions that do not allow edition on computed relation. Closes #4903918 Also, drop 'del schema' from test which clutters the code for no benefit.

[computed attribute] ensure attribute's formula apply only to the correct type. Closes #4901163

[schemaserial] fix detection of CWComputedRType table, closes #4901150

[dataimport] source.add_info doesn't take anymore a 'complete' argument Closes #4891550

[dataimport] massive renaming of session to cnx and make NoHookObjectStore working with the new connection handling. Closes #4891547

[web/data] use the right selector for hiding jquery treeview placeholders Closes #4903927

merge 3.19.9 into 3.20 branch

Added tag cubicweb-version-3.19.9, cubicweb-debian-version-3.19.9-1, cubicweb-centos-version-3.19.9-1 for changeset b7c373d74754

[pkg] 3.19.9

[hooks] don't insert use_email relation if the entity on either side is going away Closes #4912946

[undo] fix deletion of cw_source relation Closes #4912882

[hooks] don't insert an owned_by relation for deleted entities If, in the same transaction, a composed entity is created and deleted, we'll have scheduled a SyncOwnersOp which would end up creating an owned_by relation for the deleted entity. Closes #4846883

[etwist/service] adjust for a signature change Closes #4878570.

[rset] fix crash on displaying rset where some symmetric relation is used. Closes #4739253 Avoid a KeyError.

Fix handling of limit within the primary view for related elements. Closes #4723579 * respect limit set by uicfg (like the 'autolimited' view itself does) * set limit in rql query to one more than the specified one so the view knows there are more things to display

[native] Change SQL errors log level. - The SQL error itself is logged as 'info'. - The 'transaction has been rolled back' message is logged as 'debug'. These log entries should not be critical: - If an error occur at the sql level, it will be transmitted as a CW error. The exception is raised and handled in a higher level function, which will decide if it is critical or not. - Admitting that rolling-back automatically on errors is wanted in some cases, it cannot be considered 'critical'. The caller will receive the original exception anyway, it is its job to decide if it is critical or not. Closes #4801120

Added tag cubicweb-version-3.20.3, cubicweb-debian-version-3.20.3-1, cubicweb-centos-version-3.20.3-1 for changeset 7d3a583ed539

[debian] build-depend on markdown for documentation build