[i18n test] simplify test cube a bit by only providing a views module, no package

[web] consider inlined relations in has_editable_relation. Closes #3049970 In some cases where the user can only add/edit inlined relations (though actually edit perms of the inlined relation doesn't seem checked), the "modify" action doesn't appear while it should.

[web test] silent 3.13 warning

[editcontrollers] Account for role in the ordering of entities (Complements #3031719) Only role='object' was correctly accounted for by 570208f74a84

[migration.sync_schema_props_perms] ensure all participants to a unique together constraint are there before adding CWUniqueConstraint. Closes #3038345

[session commit] save back exception context to avoid potentiel cluttering if some revert operation raise an exception

[pkg] python-central has been removed from Debian

[pkg] prepare 3.17.4

[pkg] merge centos 3.17.1-2 in stable (prepare 3.17.4)

Added tag cubicweb-centos-version-3.17.1-2 for changeset 965f894b63cb

[rqlrewrite] fix rqlrewrite unpredictability vs relation sharing. Closes #3036144 The relations index used to determine if relation may be shared only considered a single relation node per relation type. So when the same relation type occurs several time, dict order give unpredictable result. We shall properly consider all relations instead. Tentative test case included (but bug reproduction is by definition unpredictable...)

[rqlrewrite] rewrite doesn't need a solutions argument, always use the Select'ones Also, do a copy systematically else it may lead to empty solutions depending on the rewrite path. Test case tentative included in the following changeset.

[rql rewrite] equivalent but much simpler flow

[pkg] Prepare a new RPM release

[pkg] Add log directory to rpm

[repo] normalize ValidationError on edited entity (closes #2509729) In CubicWeb, ValidationError.entity MUST be the eid of the involved entity, not the entity iteself (as done by yams).

[doc] one must now manipulate the req.session.data dict (closes #2842345) instead of deprecated get/set/del_session_data methods.

[datafeed] add a timeout config option (closes #2745677) So a HTTP GET do not hang forever in the datafeed looping task.

[dataimport] ucsvreader should skip empty lines unless specified otherwise. Closes #3035944

[querier] Add timings to debug prints (DBG_RQL) this may ease to spot some problematic queries

[editcontrollers] Ensure entities are created in an order satisfying schema constraints. Closes #3031719 changes below are also necessary to make the whole thing works: * stop considering InlinedFormFile as eidparam field since they don't hold any value * rework 'pendingfields' handling to have separate processing of inlined fields whose subject entity is created during the edition

[editcontroller] extract RQLQuery.set_attribute/set_inlined methods from handle_field/handle_inlined_relation

[editcontroller] req=self._cw makes things easier to read

[facets,js] fix bogus checkbox icon appearing after the first interaction (closes #2790332) This is because of the hardcoding of the data/ url.

fix typos in docstring, doc and comments

[debian] remove workaround for broken sphinx/jquery combination sphinx in squeeze was updated: http://packages.qa.debian.org/s/sphinx/news/20130112T154737Z.html

[schema] mark CWDataImport as an internal type. Closes #3025536

[pkg] remove deprecated dependency on pysqlite2

Fix two crashes in db-check (closes #3024964) Also improves various debugging messages.

[rql rewrite] may_be_shared_with should consider relation's scope (closes #3024730) When a relation's scope is not the current statement, it must not be shared (instead of comparing the statement).

[rql rewrite] fix may_be_shared_with method so that it actually considers all variable infos and not only the first one

[schema/security] add __hash__ to rql expression. Closes #3013535 This is required so that when some rql expression participate to a dictionary key, only the expression is considered (consistent with comparison). This behaviour is expected in security at least, see the related bug for instance. This scramble msplanner unit tests a bit.

[rql rewrite] move some code from querier to rqlrewrite where it makes more sense. Also, make some minor cleanup/refactoring on the way and try to enhance docstrings. Relates to #3013535

Do not compute actions list in TableLayout view when display_actions attribute is unset Closes #3007281

Use the list of cubes from the filesystem when reading the schema from the filesystem Closes #3007259

[migration] when adding a cube, skip infered relations (closes #3005576) Skipped infered relations are recalculated later.

[3.17 migration] when some cube is missing, add_cube raise ConfigurationError, not ImportError. Closes #2981477

[migraction] rename_entity_type simply warn if old entity type isn't in the schema. Closes #3004069

[deprecation] fix uihelper deprecation warning, use the rtag name, not the class, so it may be copy/pasted

[test] makes fti test order-agnostic (closes #3005633)

[web/views/staticcontrollers] Make ConcatFilesHandler write to a tempfile In order to be more resistant to errors such as ENOSPC, and against different threads accessing the same static data, we: - concatenate data files to a temporary file, and rename it once we're done - delete the tempfile in case of errors - protect the creation of the cache_concat file with a lock Closes #3005547

[test] make unittest_schema.py compatible with cubicweb-file 0.14

merge 3.16.6 branch in 3.17

Added tag cubicweb-centos-version-3.16.6-1, cubicweb-debian-version-3.16.6-1, cubicweb-version-3.16.6 for changeset b4ccaf13081d

Prepare 3.16.6

[devctl] properly generate i18n messsages for cubes that uses uicfg instances (closes #2811282)

[test/devctl] add a test case for i18ncube command

Protect against crash in the `relation_possible` predicate with ambiguous relations. It was still possible to get a KeyError when using relations that are ambiguous at both ends, like (Executable, version_of, Program) & (Version, version_of, Project). Closes #3010148.

[test] give a non-ambiguous order to sync_schema assertion (closes #3001959) Sorting by ordernum alone is unstable since several values have the same ordernum. ordernum + name should be stable.

Added tag cubicweb-version-3.17.3, cubicweb-debian-version-3.17.3-1, cubicweb-centos-version-3.17.3-1 for changeset 32b4d5314fd9

[test] typo

[schema,server] add a security debugging aid (closes #2920304) - Add a DGB_SEC debugging flag (to be used with set_debug/debugged). - Add a context manager (tunesecurity) to filter security assertions. Note: this does not address all read-security mecanisms.

[etwist] fix handling of multiple files per field html5 permits multiple files uploads, which can be expressed as:: <input type='file' multiple='multiple' /> This changeset avoids previous crash. Nothing is changed when a single file is uploaded (backward compat is thus preserved). When multiple files are uploaded for a single html input tag, the corresponding web request form key receives a list of tuples like [('filename-1', IStream1), ('filename-2', IStream2), ...]. closes #2847207.

[merge] start 3.18 development

[pkg] prepare 3.17.3

[pkg] Remove obsolete ubuntu hardy packaging It's been EOL for a while.

[book] fix sphinx documentation generation (closes #2991997) changeset 17994bf95d6a ([doc] update Session documentation) added cubicweb.server.session.Session autodoc to the book. This caused errors from sphinx due to a section title not being allowed in the class's docstring.

[test/ldap] fix ldap tests - make sure the url is properly updated on database setup (when the test database already exists and the ldap URI has changed) - fix the ldapuser test setup process.

[test] fix unittest_schemaserial.py A spurious add permission has been added in expected result by d988eec2d5d3

[test] make unittest_schemaserial.py runnable with python used to run fine only when launched using pytest

[facet] use facet name as input name for text widget (eg has_text) we need to ba able to distinguish between text-inputs so we can write a widget for a single facet that uses more than one text input. This fix consists in the diff in cubicweb.facts.js; modifications in facet.py result from this former.

[web doctype] don't give through reset_xmldecl to avoid double deprecation warning

[web doctype test] don't give reset_xmldecl to avoid deprecation warning

[devtool] randomise available ports search in http test This lowers the chance of parallel tests to race for the same port.

[facet] don't crash if no title specified on a facet and filtered rset is empty. Closes #2587883

[css, html] add a css_class attribute on Button, allowing to change easily default CSS class for buttons (think orbui integration)

[view] add 2 missing spaces before the previous link

[server/repository] Go through the repo to close pyro sessions Turns out session.close() doesn't DTRT.

fix migration from pre-3.13.1 versions (closes #2846978) Need to check the existence of the asource column before the first call to eid_type_source.

[constraint] more robust unicity constraint failures reporting for end-users Postgres or Sqlserver have limits on the index names (around resp. 64 and 128 characters). Because `logilab.database` encodes the `unique together` constraint rtypes in the index names, we sometimes get truncated index names, from which it is impossible to retrieve all rtypes. In the long run, the way such index are named should be changed. In the short term, we try to reduce the end-user confusion resulting from this design flaw: * in source/native, the regex filtering ``IntegrityError`` message does not impose an `_idx` suffix, which indeed may be absent (the result being an UI message that resembles a catastrophic failure), * also we avoid including a trailing " (double quote) from the error message * in entities/adapters, the well-named ``IUserFriendly`` adapter is made a bit smarter about how to handle missing rtypes. * the adapter also always produces a global message explaining the issue (and the fact that sometimes, the user is not shown all the relevant info) * i18n is updated Closes #2793789

[hooks/security] Streamline attributes default permission check. The current default permission on attributes delegates the check to the entity permission update policy. Since this is already checked it can be skipped. The equality comparison will work, even with a deserialized schema, because the default update perm is:: ('managers', ERQLExpression(Any X WHERE U has_update_permission X, X eid %(x)s, U eid %(u)s)) which will always be deserialized in this order (groups first). However this is a slight semantic change: entity type level 'update' permissions can now be effectively used to encode update-time rules if the default attribute permissions are used (before this change, the 'update' rules at entity type level were fired at creation time). Closes #2930861.

[test/schemaserial] swap got/expected to get nicer unittest2 diagnostics Prepares #2965518.

[repository] drop safe attribute on ``internal_cnx`` People that need to disable hook can do it explicitly anyway.

[dbapi] deprecated the dbapi

[repoapi] deprecate dbapi compat method

[testlib] deprecated the older api to access the repo.

[connection] deprecated free_cnset and set_cnxset

[session] deprecate all Connection related method on session

[session] drop dead _current_cnx_id Not used anymore for a few commit.

[session] privatise get_cnx and close_cnx The only user, repoapi now use ``new_cnx``

[doc] add documentation for the new API in test

[testlib] add an default testcase.adminaccess (and use it for default session) This adminaccess is the new offical way to get connection, and request on a repo.

[testlib] introduce a RepoAccess class to easily create connection and request Each RepoAccess hold a session for a user and three helper function to help create Connection, ClientConnection of WebRequest related to this session. related to #2920299

[repoapi] make ClientConnection.__enter__ return self This allow the standard idiom:: with repoapi.connect(repo, login='babar', passwork='elephant') as cnx: cnx.execute(…)

[documentation] describe repoapi and web side change. Short version explaining what object replace what and that BC existes for a few version.

[testlib] use internal_cnx instead of internal_session internal_session is deprecated.

[repository] add an ``internal_cnx`` method to replace ``internal_session`` Accessing the repo through a Session is deprecated. We need an easy replacement for ``internal_session``. This API change was a good occasion to stop disabling integrity hook by default. This is huge source of bug in user-code. related to #2503918

[connection] enforce that a connection must be open to be used The same than for ClientConnection, we ensure the connection is used inside a its context. .. note:: We may rely on that for ClientConnection and remove de dedicated code in Client Connection but I prefer the current explicite and duplicated version for now.

use standalone Connection to Client Connection

[session] add a new_cnx factory Having user-code importing cubicweb.server.session.Connection is inconvenient. We add a simple factory fonction on the session.

[connection] allow simple instantiation of standalone Connection Such connection will automatically pick a connection id. Note, They are not automatically closed on session close. But they will fails to grab new cnxset once the session is closed.

[connection] handle and explicitly life cycle on Connection Like ClientConnection, Connection object need to be explicitly started and stop. They aims to be used as context manager.

[sesion] distinction between Connection handled by the session and other. Not mixing the new and backward compat approach seems a good idea. Let's enforce it.

[session] replace _clear_thread_storage with close_cnx There is not good reason to keep two distinct method.

[session] explicitly take Connection object in close_cnx Now that ClientConnection explicitly reference and use the Connection object we do not need to use connectionid here. We can safely change this signature, ClientConnection is the only use of close_cnx for now.

[session] drop the Session._clear_cnx_storage method It is just cnx.clear() now.

[session] wrap too long line Too long line is too long. (Confucius 503 BC)

[session] gather close_cnx with get_cnx and set_cnx They do the same kind of operation and deserve to be grouped together.

[client-connection] remove the _srv_cnx usage It does not do anything special now that we use explicite Connection object with automatic cnx_set handling.

[client-connection] explicitly check that the client-connection is open The check is also perform by the _srv_cnx property. But we do not need those property anymore.

[client-connection] handle the lack of connection id while not open the connection id is computed at opening and forgotten when closing. We can't rely on it in various messages … like the "connection closed" exception.

[connection] transparent cnx_set handling Connection object while take cares of there cnxset themself (as dbapi connection does). The ``set_cnxset`` and ``free_cnxset`` operation are still available for backward compatibility purpose. The ``_auto_free_cnx_set`` is introduced to handle mixed usage. A new context manager ``connection.ensure_cnx_set`` is added for code that access ``cnx.cnxset`` directly and are not wrapped in any specific ``Connection`` method. A ``_with_cnx_set`` decorator is used on all Connection method that need a cnxset.

[client-connection] explicit the client part in __repr__ Now that we have real server side connection we need to remove ambiguity.

[ClientConnection] directly use the Connection object to access the database Now that Connection are a full featured standalone object we can directly reference and use it in the ClientConnection instead of using the session. The session object is kept around for a while to perform various utility role.

[connection] invert __init__ parameter Takes session first. At some point, the connection_id will become optional for Connection created explicitly.

[service] enforce that Service argument and return are json-serialisable The call_service API need to be able to run through RPC. So we ensure front start that it is possible serialise both input and output.

[connection] move call_service on Connection This is the last step toward standalone transaction.

[connection] move the commit method on Connection object One step closer of standalone Connection!

[connection] move the rollback method on Connection object One step closer of standalon Connection.

[connection] pass a Connection object to RQLRewriter RQLRewriter can now directly use a Connection object. No need for specific handling session side.

[connection] Connection.execute touch Session timestamp This allow to keep Session alive while using Connection object only for database access.

[connection] Connection now call _set_user to CWUser object linked to itseld Session.user is linked to the session.

[server/session] better faking of user for InternalManager

[connection] move execute All necessary method are now available on the Connection object. So the Connection can handle RQL execution itself.

[connection] mark Connection as "not a request" The ORM use this information. Having a wrong value will lead to crash.

[server/session] ensure appobject obtained from the session are linked to the session

[connection] have rql_rewriter accession on Connection Code expecting a session search the RQLRewriter on this attribute.

[connection] move local_perm_cache management on Connection It belongs here anyway.

[session] use a rich object for timestamp This allows the Transaction to keep a reference to it

[migraction] do not access session.data directly use set_shared_data and get_shared_data instead

[connection] move relation management method

[connection] give access to shared data method A reference to session data are kept

[connection] move relation cache method to Connection needed for execute (and need execute themself...)

[connection] give access to session execute from connection This allows multiple function that both requires an execute method and are required by the execute method.

[connection] move last part of undo logic in connection Now that Connection have a reference to the repo it can handle all of it itself

[connection] initialize connection.user and connection.lang A RequestSessionBase need a user and lang. For now we use the session ones. Later the Connection will have it's own.

[Connection] inherit from RequestSessionBase This contains a lot method. We need them to move more code from Session to connection.

[multi-sources] drop multi-sources related test. another branch is removing the multi-source itself. We do not want to bother fixing those test.

Use new repoapi for the web stack The publisher now link repoapi.ClientConnection to request. and explicitly control there scope. Web side, appobject._cw.cnx is now a repoapi.ClientConnection. This actually kill webonly possibility until the repoapi is able to use some RPC. The change in the authentication stack is very hasty and need cleanup

PARTIAL: Using the repoapi in test Test now use the repoapi.ClientConnection to access the repo. This is very big change. The current method to access the repo are kept for backward compatibility. The new methods will be introduced much later. The TestCase keep a ClientConnection To an admin session for the whole test. This changeset does not makes all tests pass without the next one that change set htt without the next one that change the http stack

[web/test] drop proff of concept Facebook login in test The authentication stack will change. The change in API will requires rework of user cube anyway.

[test/dbapi] do not rely on the Testcase provided cnx We plan to use the repoapi in the test. So we need the DBAPI test to explicitly create a DBAPI connection.

[dbapi] explicitly use the DBAPI version of CubicwebRequestBase As we plan to use a ConnectionCubicwebRequestBase by default we better have the dbapi explicitly asking for a DBAPI implementation.

[web/application] drop unused import unused since 3c85e734ce00

[web/test] properly reset the request connection related attribute A new request does not have those attribute to None. There have special value that we now properly reinstall.

[web/test] drop two undocumented suspicious regression test Those two test are not documented and use the API in the wrong way. They do not raise exception by chance. the current request signature is:: def request(self, rollbackfirst=False, url=None, headers={}, **kwargs): So the RQL have nothing to do here.

[webrequest] introduce an alternative implementation using the repoapi DBAPI specific parts of ``CubicWebRequestBase`` are extracted in a ``DBAPICubicWebRequestBase`` decicated class and a new ``ConnectionCubicWebRequestBase`` is introduced that provide the same services but using a repoapi.ClientConnection as data source. For now the ``DBAPICubicWebRequestBase`` is still used by default.

[repoapi] add an anonymous_cnx function This function return an anonymous ClientConnection. It aims to replace ``dbapi.anonymous_session``.

[repoapi] add a connect function This function takes a repo and authentication information to open a new Session and return a ClientConnection associated to it.

[client-connection] add an auto-close property for ClientConnection The next commit introduce a connect function that open a new Session and return an associated ClientConnection. The Session should not be used by anything else than the created ClientConnection, so we want to close it at the same time than the ClientConnection. The implementation in this changeset is simplistic. We probably want to move this notion in the session itself. This be improved once we have server side Connection

[cwuser] make CWUser callable, returning self for dbapi compatibility In the dbapi, Connection and Session have a ``user`` method to generated a user for a request In the repo api, Connection and Session have a user attribute inherited from SessionRequestBase prototype. This ugly hack allows to not break user of the user method. To ease transition from dbapi to repoapi we need the ClientConnection to be as compatible as possible with the dbapi. Adding this method goes in this direction. It'll get deprecated in the deprecation wave that will conclude the repoapi refactoring. related to #2503918

[client-connect] drop rqlst on rset returned client side The DBAPI used to drop the RQL syntax tree on rset to save bandwidth. We could stop doing it for In-memory client connection. However keeping the syntax tree leads to multiple tests failures. So we keep dropping it as the DBAPI always did. related to #2503918

[client-connection] add a repo property for dbapi compatibility To ease transition from dbapi to repoapi we need the ClientConnection to be as compatible as possible with the dbapi. Adding this method goes in this direction. It'll get deprecated in the deprecation wave that will conclude the repoapi refactoring. related to #2503918

[client-connection] add a connection property for dbapi compatibility To ease transition from dbapi to repoapi we need the ClientConnection to be as compatible as possible with the dbapi. Adding this method goes in this direction. It'll get deprecated in the deprecation wave that will conclude the repoapi refactoring. related to #2503918

[client-connection] add a sessionid property for dbapi compatibility To ease transition from dbapi to repoapi we need the ClientConnection to be as compatible as possible with the dbapi. Adding this method goes in this direction. It'll get deprecated in the deprecation wave that will conclude the repoapi refactoring. related to #2503918

[client-connection] add a cursor() method for dbapi compatibility To ease transition from dbapi to repoapi we need the ClientConnection to be as compatible as possible with the dbapi. Adding this method goes in this direction. It'll get deprecated in the deprecation wave that will conclude the repoapi refactoring. related to #2503918

[client-connection] add a request() method for dbapi compatibility To ease transition from dbapi to repoapi we need the ClientConnection to be as compatible as possible with the dbapi. Adding this method goes in this direction. It'll get deprecated in the deprecation wave that will conclude the repoapi refactoring. related to #2503918

[repoapi] introduce a basic ClientConnection class This is the new official way to access the repo from client side. It still access Session object directly as the server side connection is not up yet (and it's not up because it would have no user). Multiple follow up commit will install compatibility with the DBAPI. This will ease the migration from dbapi to repoapi. ClientConnection has no user yet but later commit will use it in the whole Web stack. related to #2503918

[req] add a _set_user method RequestSessionBase usally need to recreate a new CWuser appobject linked to themself. We add it to the base class to avoid multiple redifinition.

[server/session] add a login property session.login is a DBAPISession attribute. Having it on server side session will helps the rework of the API to access repository. The new schema drop the concept of DBAPISession and use server side session for the same purpose. related to #2503918

[server/session] allow access to session id using sessionid session.sessionid is a DBAPISession attribute. Having it on server side session will helps the rework of the API to access repository. The new schema drop the concept of DBAPISession and use server side session for the same purpose. related to #2503918

[server/session] Implement anonymous_session Now we have a simple rule to compute if a session is anonymous we can implement the property for server session too. Having it on server side session will helps the rework of the API to access repository. The new schema drop the concept of DBAPISession and use server side session for the same purpose. Related to #2953943 Related to #2503918

[server/session] do not clear session.cnx if unrelated to cnx actually cleared The cnx parameter may be different from the one actually loaded for the current thread. This will be possible in future commit when you close a client connection for example.

[dbapi] move ProgrammingError into cubicweb module A new ``repoapi`` will be introduced as a replacement for the dbapi. It will need ProgrammingError too. Related to #2503918

[repoapi] move get_repository function into a new repoapi module This new module aims to host the function of the new API replacing the old DBAPI. `get_repository` is still needed hence moved to the new module. Related to #2503918

[testlib] gather all repository access logic in one place Refactoring of the repository access API in test is imminent. We plan to move from the "old" dbapi to the new repoapi. Gathering all impacted method in one place help to understand how all those method interact and help readability for both patch and resulting code. No code change is done at all in this changeset. The refactoring will code later.

[testlib] move repo and related attribute back on Instance instead of Class The repo and cnx was hold by the TestCase class to work as a cache and avoid recreation of the repo from scratch of each test. However since bad26a22fe29 the caching is done by the DatabaseHandler object and it is not necessary have a second layer of cache on the TestCase itself. We move the repo and cnx attribute back on the TestCase instance itself and make several class methods instance methods again. This will helps to change dbapi access in test from dbapi to repoapi. related to #2503918

[dbapi] makes anonymous_connection a computed property The current implementation is a boolean flag set manually by client code after connection creation. This led to different way to decide a anonymous_connection should be True (eg. different in the test than in the actual application code). It should not be client responsibility to set this flag. ``cnx.anonymous_connection`` is now a purely computed property. Connection with user in the "guests" group are anonymous, the other ain't. ``Session.anonymous_session`` is computed from ``cnx.anonymous_connection`` and get the updated behavior transparently. Closes #2953943

[webrequest] simplify set_session code Thanks to the previous changeset we are assured that session handed to set_session is full featured one. This allows a simpler code for this method. related to #2503918

[webrequest] set DBAPISession without cnx at initialisation time Such session are necessary for minimal use of a Request. Setting on by default allow simplification later linking with a full featured DBAPISession or equivalent. This was not possible before as all session was tracked by session manager. They are not tracked anymore since aa709bc6b6c1 and we can safely create them by default. related to #2503918

[testlib] rework request building in init_authentication The previous code was building a full authenticated request and tried to undo the authentication afterward. The new code just create a bare Request with no authenticated session linked yet. This is much closer of what the actual authentication process does.

[request] drop the user argument for set_session I see no code nor test that use this optional argument. removing it help to clean the session code. The set_session function will soon be deprecated anyway (at the same time than the dbapi) related to #2503918

[web-request] handle default language earlier We now read language negotiation header at initialisation time (previously done during ``set_session``). This simplify the set_session code for later reworking. There is no change in behavior, the user selected language continue to overwrite the one picked from the header/default when applicable. related to #2503918

[etwist] pass the repository to the root resource This is another step toward a cleaner instantiation scheme for the repo. The http test now pass an already created object to the etwist server instead of relying on the config cache. Related to #2249513

[repo] move repo.gc_stats to Service API (closes #2951068) We currently have a method on the repo that anyone can call. `call_service` gives a similar thing, modulo: * can be restricted access to manager only (thanks to selection on services), * less unrelated code on the repository class, * no need to fetch a reference to a repo object from the client side * (`req._cnx.repo` is not an API and config.repository() is on its way out). The old way to access this information (repo.gc_stats()) is deprecated.

[repo] move repo stats to Service (closes #2951067) We currently add a new method on repo that anyone can call. Call service allows to reach the same result with: * Restricted access to manager only (thanks to selection on services), * Less unrelated code on the repo. * No need to fetch a reference to a repo object from client side. `req._cnx.repo` is not an API and config.repository() is on its way out. The old way to access this information (repo.stats()) is deprecated.

[service] drop the asynchronous execution possibility Call_service was able of both sync and sync execution making the API confusing. There is not user of the async case. We drop the async argument in favor of synchronous execution only. This makes call_service the official API to call server side code from the client side. This is a remplacement for the usual monkey patching of the repo object. The zmq notification bus is a solid alternative for codes that needs to start an async execution.

[auth] pass `repo` instead of `vreg` to SessionManager and AuthenticationManager Those object actually need a repo object to connect and create new session object. They used to retrieve the repo object using ``vreg.config.repository()``. I'm trying to sanitise the initialisation stack and get ride of ``config.repository()``, a function that mix factory and cache. Retrieving the vreg from the repo is trivial (``repo.vreg``) so we now pass the repo object instead of the vreg.

web/application: instantiate the repository outside of CubicWebPublisher This improves decoupling and allows ``CubicWebPublisher`` user to choose which argument are used during ``Repository`` instantiation. In particular is allows caller to provide a ``TaskManager``. The `vreg` argument of publisher is made mandatory. See CubicWebPublisher docstring for details. Related to #2249513

remove vreg argument to CWPublisher The passing a vreg to CWPublisher was used by test to allow reused of an existing vreg. We dropped this feature two commit ago and can safely removes it. Removing this argument allows later cleanup and improvement on the CWPublisher and Repository front. closes #2944813

[devtools] vreg becomes a class ppty After multiple devtools refactoring, the vreg used for test is now *always* got from the repository. Making this explicit is a step toward simplification of the vreg handling during various initialisation phases. Related to #2944813

[devtools] drop unused `reset_schema` and `reset_vreg` class attribute They have no effect and no user... Related to #2944813

[connection] move security control logic on Connection The _security_enabled context manager now operate on connection. We have to keep a specific entry point in Session to ensure Connection object are properly cleaned up when using session to "manage" Connection live cycle. Related to #2503918

[connection] move hook control logic on Connection The _hook_control context manager now operate on connection. We have to keep a specific entry point in Session to ensure Connection object are properly cleaned up when using session to "manage" Connection live cycle. Related to #2503918

[session] update _hook_control docstring

[connection] reinstall cnx.data as cnx.transaction_data Too much code expects self._cw.data to be session data. backout 9b2f68916474 Related to #2503918 Related to #2912807

[connection] give access to is_internal_session boolean Needed if we are to use this object like we currently do with session. Related to #2503918

[req] drop from_controller on non WebRequest object (Closes #2901079) The `controller` concept is purely a web things. It does not belong to generic `req` object. The `build_url` code is slightly changed to handle that. This probably requires extended cleanup later. This is a barely used and very internal API. No deprecation period

[connection] add logging method on connection If we are to use this object like we currently do with session, we need to have the logging method as a lot of code use them.

rename server.session.transaction into server.session.connection The ongoing rework of the API to access the database include the splitting of Session in two objects: The ``Session`` object will only hold credential (user) and global session data. A new object will publicly emerge to handle a database access. This object is named ``Connection`` since that the way database accessors with the same property are named in other system. So we renamed the ``Transaction`` object into ``Connection``. The ``Transaction`` object have already grown in the direction of something directly usable by generic code, but the name ``Transaction`` is ill suited as such object can be used for multiple database transaction in a row. Related to #2503918

[application] call req.set_session in application.main_handle_request The Session handling chain is no more responsible for calling req.set_session. It just returns a valid session and lets the caller link it to the Request. This opens the way to explicitly creating and closing a connection/transaction in ``application.main_handle_request``. Related to #2503918

[session-handler] use session directly to update last usage We don't really need the WebRequest for that. Not using the WebRequest to access the cubicweb repository here will allow a delayed set_session. Related to #2503918

[application/connect] simplify connection logic ``application.connect`` now either sets a full featured ``DBAPISession`` to the ``WebRequest`` object or raises ``AuthenticationError``. The creation and usage of a fake DBAPISession is now handled by ``main_handle_request`` when needed. This means that fake DBAPISession are no longer tracked by the session manager and that user are not given anyway to retrieve them for a later request. This fake DBAPISession is still passed to ``core_handle`` because multiple cubes like registration or forgotten password need this behavior. We would like to get ride of it in the future. This clarification of the connection API greatly simplifies ``DBAPISession`` retrieval//creation process opening the way to improvements in this area. Related to #2503918

Drop hijack user (closes #2901093) Deprecated since 3.17.0. Dropping this function help the session cleanup business.

[sources] drop support for ldapuser source (closes #2936496) The ldapfeed source is a replacement for ldapuser. Use it instead.

merge stable back into default

3.17 is the new stable

Added tag cubicweb-version-3.16.5, cubicweb-debiann-version-3.16.5-1, cubicweb-centos-version-3.16.5-1 for changeset 810a05fba1a4

[pkg] prepare 3.16.5

Added tag cubicweb-version-3.17.2, cubicweb-debian-version-3.17.2-1 for changeset 195e519fe97c

[pkg] prepare 3.17.2

repository: make tests pass again The tests use mono-threaded pyro, which breaks assumptions made in e27337dfec8c. To fix that, remove sessions from the _pyro_sessions dict when they're closed, and force the test pyro client to actually disconnect when it's done.

repository: monkey patch pyro connection handling to detect clients going away When a pyro client goes away we need to close their session (to release their cnxset). Add a dict to the repository mapping pyro client threads to session object. Closes #2932058

notification: use viewargs for notif_entity_updated instead of transaction_data Now that notifications use separate sessions, they can't rely on the original transaction_data being around, so pass the data through view arguments instead, so the notification view knows what changed on updated entities. Closes #2936180

notification: properly handle cnx and lifetime of the hijacked session When we create a session for the notification rendering, we need to 1) give it a cnxset 2) commit and close it after we're done 3) restore the original session for subsequent notifications This changes the ordering of actual mail sending, since there are several different commits involved, but I don't see a way to fix that short of restoring hijack_user. Related to #2934523

notification: fix session creation (closes #2934523) Session() takes a user and repo, not repo and user.

Added tag cubicweb-version-3.17.1, cubicweb-debian-version-3.17.1-1, cubicweb-centos-version-3.17.1-1 for changeset f98d1c46ed9f

[cwctl] add configure command to cw-ctl (closes #2709702) usage: cw-ctl configure <appid> --param key=value,key2=value2 --param key3=value3

[cwctl] rename option no-post-create to no-db-create

[pkg] version 3.17.1 for rpm packages

[session] upgrade session closed error from Exception to SessionClosedError Exception is far too wide and we already have a SessionClosedError anyway. Closes #2897696

[pkg] prepare 3.17.1

[web/application] add some minimal documentation Some documentation to the Application main entry point.

[session] raise proper exception with get_tx is called on a closed session Prevent attribute error on _txs Closes #2897836

[view] typo in a docstring

[cwctl] pass missing inputlevel argument to postcreate() (closes #2901037)

[server] fix documentation of ZMQ options harder The syntax for zmq addresses is tcp://host:port, no zmqpickle involved.

merge with 3.16.x fixes

[server] fix documentation of ZMQ options

[zmq] set order for ZMQStartHook so other hooks don't need to If an application wants to subscribe to some messages on the zmq bus, that needs to happen after ZMQStartHook.

import merge_options directly from logilab.common

[server] fix error messages in write_sources_file()

[doc] a couple almost-typos

[schemaserial] mitigate critical message during migration We can't fetch extra_props because it is not yet there.

[doc] typo

[cw-ctl] fix help message for schemadiff In global help (closes #2888538) The leading space make it appears on multiple line. The help content is a bit improved in the process.

[dbapi] fix connect backwards compat harder Passing None as group and host should result in "pyro:///cubicweb.appid", not "pyro://None/None.appid". Followup for 2091d275fe5c. Closes #2882666.

[zmq] make publish address optional It should be possible to use the zmq communication bus in a read-only manner. Allow setting zmq-address-sub without zmq-address-pub. Closes #2897178.

Fix documentation for zmq pub/sub hooks and configuration Closes #2897177

[entity] improve deprecation messages

[devtools] use self._parse so AssertionError is properly raised instead of lxml error (test failure introduced in 6711f78c18be)

[testlib] fix page validator selection. Closes #2869456 * use a validator that checks XML well-formness for the html5 doctype (as a convenient side-effect, it does not choke on e.g. a canvas tag like the HTMLParser does) * use a DTD validator if there is an xml declaration * else use the HTMLValidator

[testlib] introduce a validator that check xml-well formness This validator simply check XML is well formed and accept any entities (think HTML defines much more entities than bare XML)

[view] return HTML5 doctype. Closes #2869426 We dropped xhtml support in 3.17 but we kept providing documents using the XHTML strict doctype, while our content wasn't conform to the DTD since we dropped proper namespaces definition as well as declaration of cubicweb DTD extensions.

[testlib] repreprocess content so contextual display has a chance to show the proper line. Closes #2869481 the context is calculated using position information relative to a preprocessed content, not the original source

[web request] drop no more necessary request.document_surrounding_div method This is awful and no more necessary since we dropped xhtml support

[testlib] unspaghettify Validator / PageInfo api

[testlib] update htmlparsers.VALMAP: stop using SaxOnlyValidator and add an entry for html

[testlib] SaxOnlyValidator is deprecated, use XMLValidator instead

[dataimport] minor typo in error handling

[views] attempt to clarify the inline help system documentation

[sobject] fix notification operation regression introduced in bf4003760e02 Operation use self. session. not self._cw. I'm a bit surprised that its not tested. closes ##2870077

Added tag cubicweb-debian-version-3.17.0-2 for changeset 09a0c7ea6c3c

[pkg/debian] prepare release 3.17.0-2

[pkg/debian] update yams runtime depends Only the built depends were updated to 0.37.0.

[pkg/debian] expand all depends and co on multiple line This is more readable and simplifies patching.

Added tag cubicweb-debian-version-3.17.0-1 for changeset 22be40c492e9

Added tag cubicweb-version-3.17.0 for changeset cc1a0aad580c

[test/view] exclude some Blog entity from automatic view testing Some Blog view requires the sioc cubes that we extracted in 3.17 we do not really want to test cubes view here. So we exclude CWEtype that really requires the sioc cubes.

[doc] fix: cubicweb.gmap.js has moved to the geocoding cube

[test/ldap] do not assume order Do no assume a specific order is stable when testing group members

Added tags for version 3.16.4 on changeset 041804bc48e9

[notification] ensure official API is test By using the official API in the hook we ensure it is tested.

fix typo in notification causing NameError introduced by: f5b40b66d36e

[test/notification] merge related test classe They have no reason to be distinct.

merge with stable

[server] add a looping task to free unused cnxsets capture by dead pyro When a pyro client disconnects, we're not necessarily notified so they can hang on to their session/cnxset forever. Don't let them do that. The hacky looping task is only installed when pyro is enabled. This optional activation save this very hacky patch from being rejected. closes #2825380

[pkg/debian] update changelog to 3.17.0

[devtools] Use XMLValidator instead deprecated SaxOnlyValidator The renaming was introduced in 198fdadafed6.o This prevent a meta class conflict introduced by logilab-common 0fcd2e050621.

[pkg] update changelog

[pkg] prepare 3.17.0

[pkg] prepare 3.16.4

[repo] straightforward bootstrap sequence. Closes #2841188 * kill fill_schema method which was doing to much different things, in favor of a deserialize_method which is simply reading schema from the db and returning it * set_schema doesn't bother with cubes This allows to easilly follow what's going on and so to cleanup init_cnxset_pool to get the following boostrap order: 1. load cubes (implies loading site_cubicweb) 2. load schema 3. init data sources 4. init cnx sets

[repo] minor cleanups to bootstrap sequence * fix some docstring/comments * move initialization of HooksManager to the __init__.py, this has nothing to do with pool initialization / boostraping

[schemaserial] serialize additional yams parameter for customs type Yams have a new custom type feature. We add support for extra parameter on CWAttribute in schema serialization handling. Closes #2847151

[sql] preprocess_entity uses lgdb helper's SQL converters. The preprocess_entity does not yield anymore the conversion logic for different base types, but relies on a new SQL_CONVERTERS dictionary of the db helper. This will allow later inclusion of new base types.

[repo] kill rebuildinfered feature from Repository.set_schema This clutters thing and is useless in most cases: when schema just have been readen from the database or from the file-system, infered relations have just be calculated. The only use-case where it should be rebuilt is on migration (ie schema modification by editing the meta-model using rql), so move this feature there.

[test] stop deepcopying for nothing

drop xhtml content-type support (closes #2065651) * HTMLStream does not care about xml any more * reqquest.demote_to_html and .xhtml_browser are deprecated * web config: drop force-html-content-type option * adjust tests

[htmlparser] exclude <script> tag from html source lxml are confused by them :-(

[htmlparser] store unaltered source in pageinfo.source Do not store the parser to preserve the source aspect. Store the initial input in the source argument instead. This is very useful when parsing html. In such case we need to drop ``<script>`` tag.

[htmlparser] rename SaxOnlyValidator to XMLValidator

[htmlparser] add missing deprecation message

[form] remove hard coded id html id should be unique. It is clearly not achieved here.

[server/test] use server config in migration This allows a lighter initialisation (no rtag needed). Execution of those test move from 300 secondes to 220 seconds here. A 25% speedup

[session] deprecate `hijack_user` method The semantic of this method is wicked and lead to very hard violation of sanity. (multiple transaction using the very same cursors). We deprecated the API to be able to drop it as soon as possible. The method was added long ago for some notification trick. It is not needed since we sent notification on ``postcommit_event``. (closes #2781782)

[mail] drop hijack_user usage The transaction is now commited before notification. We can create a new session directly and drop `hijack_user` usage. This prepare #2781782

[notification] merge BaseNotificationView and NotificationView They are in the same module now.

[notification] move notification view in ``sobject.notification`` It has no user outside this module. This enforce serversideness of notification and allow future cleanup. No backward compat is set up to prevent circular import. The class has no other user anyway. (closes 2845144)

remove unused import pyflakes say they are unused. Tests suite confirms.

one more merge

[notification] introduce an official `notify_on_commit` function It provides a proper notification api instead of replacing it by yet another clumsy one. Closes #2837251

[cubicweb/doc] Replace dc_type() by cw_etype dc_type() is may be translated. Leading to terrible terrible damage.

[dataimport] backout 6947201033be (related to #2788402) (and add a try: except to cache the intended error) The problem actually comes from the ``MassiveObjectStore`` in the ``dataio`` cube, so it should be corrected there. Here, we only protect it with a ``RuntimeWarning`` so that the user can see the problem. ``value`` is set to ``None`` (whence to ``NULL`` from a database standpoint), so that the data can be nevertheless inserted in the database. However, only the keys present in ``row`` are actually non-'``NULL``'. The real solution is to work out the issue in ``MassiveObjectStore`` directly. The current try/except should only be a temporary hack.

merge default heads

[c-c serverctl] import locally to speed up c-c

[facet js] minor refactoring and cleanups * extract facetCheckBoxSelect and facetCheckBoxUnselect * use $ as prefix for jQuery variable (facet -> $facet) * other minor rephrasing

[ldapfeed] Add support for LDAP groups (closes #2528116) Groups from the LDAP server are imported as CWGourp entities, and in_group relationships are created between existing CWUsers and CWGroups Unit tests have been refactored a bit, especially ti add helper methods to manage LDAP database.

[test/ldap] small improvement to ldapfeed unit tests better group testing and further testing of imported user life cycle.

[ldap] prepare import of CWGroup All CWUser specific code is put in dedicated sections.

[ldap] handle modification date We are now able to import modification date from ldap.

[ldap] refactor attributes mapping handling * use explicit attributes lists when calling ldap search * direct and reverse attributes maps contains now the same elements.

[ldapfeed] add support for multiple email addresses from ldap Multiple emails case was ignored before this change. email test

merge 3.16.x fix in 3.17.x branch

[session] fix hooks_control backward compat Session object have no ``hooks_control`` control method. it has ``allow_all_hooks_but`` and ``deny_all_hooks_but`` ones. Regression introduced in bd5b5759c9b3.

[facet js] fix reordering of facet check boxes. Closes #2732947 Before this patch, when one select an element, it's moved to the top of the select content. Fine. But when it's later deselected, it stays there instead of moving back to its original location. This patch fixes that by introducing a facetCheckBoxReorder function which properly reorder the whole facet, instead of buggy attempt to locally reorder.

[vreg] (re)build entity classes cache after registry initialization. This fix potential memory fragmentation issue. Closes #2719113

[task] allow negative intervals for add_looping_task (closes #2818280) Instead of rejecting values < 0 with a ValueError, we ignore tasks with a negative interval. This introduce a simple way to disable task from configuration.

[task] allow interval=0 for looping tasks There is valid usecase for perpetual running task.

[portable dump] skip virtual relations, they have no table associated. Closes #2841199

[test] fix skipIf import, currently failing if unittest2 is not there

[test] fix bad merge

merge with other default head

[ldap] an empty 'user-base-dn' disable the user importation process, The same semantic will be used for group. We start by using it for user for the sake of consistency.

drop commented print

[ldap] Major test refactoring Mostly fix tests (previous refactoring), but also refactor a bit the base class to have better/simpler interaction with the slapd server.

[test/ldap] have RQL2LDAPFilterTC use its own database I do not know what RQL2LDAPFilterTC is about and why it does this strange reuse of LDAPUserSourceTC pre_setup_database but it is certainly a good idea to have it use a dedicated database-cache.

[ldap] quieter tests Capture slapadd and slapd output if everything goes well.

[test/ldap source] better source naming, some cleanups (prepares #2528116) ldap uri becomes "ldap" in all cases prepare separation of CONFIGs by source type

use cw_etype instead of __regid__ We have a clean public API now.

[entity] add an official ``cw_etype`` property for entity This property holds the CWEtype id of the entity. This information was previously available through __regid__ but this is an ugly public API. So we have a new clean public API. The ``dc_type()`` method could not be used as it is translated. (closes #2793792)

[notification] use new style operation for notification This results in far less operations created. Operation creation is expensive. We keep some compatibility with the previous API because some cubes have been reported to use it.

[cleanup] drop code deprecated since 3.6 The 3.6 API have been dropped some time ago. Those three lines survived the purge. One of them is even deprecated since 3.5.

[notification] fix documentation The doc say precommit, the code says postcommit. The code is right. We also clarify the documentation of another related operation looking suspiciously wrong but actually correct.

[ldapuser] Deprecated the source We have ldapfeed for some time. maintaining both is cumbersome. This old style source will be dropped in 3.18. closes #2567712

[rql2sql] Remove an XXX by explaining why Int non-substitution is desired (in the current state of the API)

[rql2sql] Boolean values should be substitued like others. Closes #2829481 Furthermore lgdb 1.10 deprecate boolean_value method and this simplify testing.

[toward-py3k] rewrite __cmp__ (closes #2715115) In Python 3k `__cmp__` special method is no longer supported. So we split it in `__lt__` and `__eq__`. details on Python3k behavior here: http://docs.python.org/3.0/whatsnew/3.0.html#ordering-comparisons

skeleton: add RPM spec template (closes #2800884)

[clean_session] revert itervalues usage from 0bb18407c053 self.close(session) changes the self._sessions content. We need to create a full list of value before starting closing them.

merge 3.16.x fix

[ldap] Use correct API for logging message The good practice cat prefer logging to do the substitution himself.

Fix obsolete docstring for add_looping_task The docstring was telling lies since changeset cae198371548 "add_looping_task can be used any time"

[devtools] prevent Xvfb resets Seems like firefox opens more than one connection to the X server in some cases, and things get utterly confused.

[LDAP] rename unittest ldapuser to ldapsource (prepares #2528116) the tests involved are not only for ldapuser module but also for the more general ldapfeed Necessary to prepare LDAP patch for CWgroup

[devtools] protect xvfb-run.sh against clean_up being called twice

Added tag cubicweb-version-3.16.3, cubicweb-debian-version-3.16.3-1, cubicweb-centos-version-3.16.3-1 for changeset ee860c51f56b

3.16.3

[service] make sure synchronous call_service has a cnxset Fixes regression in 3.16.2 (closes #2807720)

Merge testfunc testfunc resurrection Closes #2535377

[web] move qunit test back into web and fix the hanging qunit test can not work outside web. a lot of try/finally clause have been added to prevent qunit to hang on error. A not a test a indeed broken. But we at least put in them a running state again..

[test] drop forsaken windmill test (Closes 2535376) * not run since multiple years * windmill itself is in poor shape not even packaged in Debian. The peace which other seek they find; The heaviest storms not longet last; Heaven grants even to the guiltiest mind An amnesty for what is past; When will my sentence be reversed? I only pray to know the worst; And wish as if my heart would burst.

Added tag cubicweb-debian-version-3.16.2-1 for changeset b3c1ad0cbf00

Added tag cubicweb-version-3.16.2 for changeset a2b4f245aa57

3.16.2

[server/test] remove dependency on a pyro name server (closes #2801737) Use a direct pyroloc:// url instead of going through a NS.

[packaging] add spec file to build RPMs (closes #2800850)

[predicates] keep making `yes` predicate importable from cubicweb (closes #2790319) The `yes` predicate was moved out in CubicWeb 3.15 but it should have remained importable from the cubicweb.predicates module, for bw compat reasons and also because logilab.common.registry is an implementation detail.

[views/tables|css] backout rev 895e34d9ae0d This changeset introduced a GUI behaviour change that has usability side-effects (eg. shows a non-undoable massive action as a first-class action). The idea is good, but it requires a better implementation; tabs are not the proper GUI solution to the original usability issue.

[webconfig] cleanly handles ``fckeditor_installed`` when no uiprops The previous version of the code relied on a default value with a dummy ``FCKEDITOR_PATH`` entry. The new code check if uiprops is initialized or not.

[webconfig] fallback to default configs if https ones are not defined When cubicweb serves a request with an https scheme, it tries to access ``https_uiprops`` and ``https_datadir_urls``. However if those are not defined we should fallback to unsecure url instead of crashing with a traceback. There are very valid usecases for undefined https url. For example if your ``base_url``, ``datadir_url`` and uiprops content are already https urls. Closes #2786643

[webconfig] explicit some webconfig attribute Explicit declaration of attribute are much clearer. Just do it.

merge 3.16.x fix in 3.17.x branch

merge with 3.15.x fix

Added tag cubicweb-debian-version-3.15.11-1 for changeset 09d65bc1f025

Added tag cubicweb-version-3.15.11 for changeset 38c6a3ea8252

[querier] improve 'Password selection not allowed' error message to report the involved variable name whose type is Password, it usually help understanding the problem.

[predicates] use select_or_none rather than select+try/except, expected to be more efficient

[pkg] prepare version 3.15.11 ?

merge 3.14.x fix into 3.15.x

[facet] ensure a facet DOM id is a valid jQuery identifier. Closes #2789089

[predicates] fix 'adaptable' implementation, used to return 0 on a rset with individualy adaptable entities of different types. Closes #2709663

[cmd] fix exlog documentation (closes #2715913) The table documentation was out of sync since 8a075bc94563.

Added tag cubicweb-debian-version-3.14.10-1 for changeset 197bcd087c87

Added tag cubicweb-version-3.14.10 for changeset 0ff798f80138

[pkg] prepare 3.14.10 release

[querier] fix eid relations handling in SET queries (closes #2797052) _extract_eid_consts() implementation must consider arithmetic operators when trying to associate rql variables to specific eids.

[ext/rest] add directive bookmark to rest (closes #2545595)

[c-c create] make post-create step optional (closes #2712041) In this context the "post-create" step refers to the database's creation and initialization. We want to be able to create a new instance but skip the database creation (`db-create` and `db-init` commands) in automatic setup.

[cwconfig] Add optional configuration param to load a UI cube before other cubes Needed later on to load UI cubes such as legacyui, and one day squareui.

[web/views] Move massmailing to its own cube (closes #2788086)

merge with stable fix

[req] explicit `user` attribute The `RequestSessionBase` object needs a `user` attribute. It is initialized by sub class, but having it explicitly declared is much cleaner.

[test] use the right order in some assertEqual The unittest2 function use ``(expected, got)`` order.

[session/transaction] move resource accessors on Transaction

[session/transaction] move multiple utility on Transaction

[session/transaction] handle cnxset repository logic in transaction Interacting with the repo is now handled by the Transaction code directly.

[transaction] keep a reference to the repo object

[session] make security_enabled API private The doc already says "internal API". The old entry point is now officially deprecated and the context manager have been made private.

[session] make hook_control API private The doc already says "internal API". The old entry point is now officially deprecated and the context manager have been made private.

[service] split session retrieval and service execution Internal session are not tracker by the repository and can not be retrieved using _getsession. Moreover their session id is not unique… Closes #2735700

[service] handle cnxset in the async case only The service may be called by a session, we really do not want the call to service to mess with the transaction state. This does not apply to async were thread magic ensure a dedicated transaction.

[service] extract session retrieval from "task" closure The session was retrieved there because `set_cnxset` must be called in the same thread as the code using it. We now retrieve the session beforehand and explicitly call `set_cnxset`.

[cubicweb/doc] Add tutorial on data import in CubicWeb. This involves creating the "tutorials/dataimport" directory structure under "cubicweb/doc" and, inside the "dataimport" directory, putting several files: - a ResT file containing the tutorial *per se*; this tutorial addresses the following issues: * creating a CubicWeb schema for representing a given data set (here, the Diseasome RDF data, for illustration purposes); * parsing the data; * importing the data, by using several stores: + the ``RQLObjectStore``, ``NoHookRQLObjectStore`` and ``SQLGenObjectStore`` from the ``dataimport`` module in CubicWeb; + the ``MassiveObjectStore`` from the ``dataimport`` module in the ``dataio`` cube. The tutorial also provides timing benchmarks of the various stores. - a set of Python files illustrating the data import, in the context of Diseasome RDF data parsing: * a Diseasome RDF data parse module, * a Diseasome data import module, * a CubicWeb schema for representing Diseasome data.

[dataimport] Slight message modification in exception handling code.

[dataimport] Handle various data formats when creating buffers from data. :note: On the long run, this should be addressed before, when these data are created.

[dataimport] Add a ``flush`` method for all stores. For ``ObjectStore``, ``RQLObjectStore`` and ``NoHookRQLObjectStore`` the method tries to call the ``commit`` method. For ``SQLGenObjectStore``, the ``flush`` method is basically unchanged with respect to previous changesets.

[dataimport] Uniformize the API across the different stores. This is achieved by modifying the ``relate`` method so that it takes an extra ``**kwargs``. More specifically, ``SQLGenObjectStore``'s ``relate`` method needs the the type of the subject entity which is passed through ``**kwargs`` as the ``subjtype`` keyword argument. Actually, it is the ``add_relation`` method of the ``SQLGenObjectStore`` who needs this argument. However, as this method is not called directly (but via the ``relate`` method), the ``subjtype`` argument is passed to ``add_relation`` via ``relate``. The other stores' ``relate`` methods do not need this extra argument, hence for the other stores ``**kwargs`` is empty. In this manner, the API is unified across the different stores.

[data] Add a CSS for undo UI rendering The nesting of ordered list and unordered list was badly handled. This has gone unoticed becaused it was not used. But the undo feature use this markup to render the changesets. That's why this minor change to ths CSS is introduced.

[web/views] extract cube geocoding (closes #353000)

[c-c list] add an optional argument to the list command (closes #2709703) This optional argument allows to restrict listing to only a given type of items (cubes, instances, configurations).

default base-url now uses fdqn instead of hostname only (closes #2542815)

[editcontroller] a small debugging help (closes #2518980)

[doc] small doc improvement

[doc] fix Pyro url

[doc] fix Narval project url

[tutorial] minor documentation fixes

merge 3.16.x fix in 3.17.x

merge 3.15.x into 3.16.x

server/source/native: fix wrong usage of .lstrip that produce garbled error messages (closes #2777641) Fixed an .rstrip(...) for the general case (untested). Fixed an .lstrip(...) for the sqlite path (tested).

[doc] update 3.17 release note

[migration] fix bug in `CWAttributeAddOp.revertprecommit_event` The `rdefdef` attribute is set during the operation execution. In some the operation crash before this assignment. In such case `revertprecommit` raise an attribute error crashing the whole process and shadowing the original error. This changeset detect and Avoid this situation.

[session] security context directly use transaction object when applicable Most of the security logic have been moved on Transaction anyway.

[session] move context manager logic in the security context manager The `init_security` and `reset_security` method have no other user that the context manager itself. Having dedicated public function is not useful.

[session] hook control context directly use transaction object when applicable Most of the hooks control logic have been moved on Transaction anyway.

[session] move context manager logic in the hook controls context manager The `init_hooks_mode_categories` and `reset_hooks_mode_categories` have no other user that the context manager itself. Having dedicated public function is not useful.

[session] drop the useless set_hooks_mode method It has not external user and trivial code

[session/transaction] move most undo support into transaction A small part that need repo access has not been moved into transaction yet.

[session] simplify mode property We can delegate very thing to transaction utility

[session/transaction] pass the session object to the Transaction __init__ This simplify access to variables. As we are moving more and more logic and states in the Transaction it will greatly helps in future commit.

[session] use a dedicated class to track cnxset We introduce a new CnxSetTracker to track `cnxset` used by Transaction and allows to wait for them. This new class does not use Thread ID not thread joining to work. This allows to use multiple transaction per thread and a transaction in multiple thread. The class itself is totally threadsafe by the Transaction is still not thread safe. The old _threads_in_transaction attribute is dropped in favor of a new logic based on this object. The registration of cnxset used is not done by the Transaction itself. tx.cnset is a property handling the Consistency of its value with the CnxSetTracker instance. Note: The CnxSetTracker instance only track transaction id, not transaction itself, So not reference cycle are created.

[session] cleanup Transaction.cnxset The attribute should always be there. It is None when not cnxset exists.

[session] drop useless getter and setter for security Those function have very few user and does not anything fancy. Internal user now access tx.read/write_security directly. External user have been blessed with a writable property. They are only interested in the write part of they did not used the return of the removed function. They access it through the session since I prefer the external world to not be aware of Transaction yet.

[session/transaction] move read_security subtlety in transaction This is transaction data, itshould be handled by the transaction itself.

[transaction] initialize dbapi_request in __init__ There is not reason to have it implicitly declared. The attribute is renamed to match its public name.

[whatsnew] notify drop of deprecated code

[web/views] extract cube embed (closes #1916015)

[skeleton] Update debian packaging to support only python 2.6 and up

Drop iprogress code (closes #2777628)

[web/views] extract cube sioc (closes #1916018)

[doc] Add 3.17 release notes

[migration] mark all version prior 3.7 as non-migrable API possible used by such old migration script are not supported in recent Cubiweb. Closes #2772958

[dataimport] drop checkpoint backward compat API Deprecated since 3.7. See #2772958 for details.

[dataimport] drop index support deprecated since 3.7 Dropping of pre-3.7 API decided in #2772958

[utils] drop pre-3.7 compat in util Dropping of API deprecated since 3.7 decided in #2772958

[migration] drop pre-3.7 utility Dropping of API deprecated since 3.7 decided in #2772958

[entity] drop pre-3.7 api on session Dropping of API deprecated since 3.7 decided in #2772958

drop dubious comment

[req] drop pre-3.7 api on request Dropping of API deprecated since 3.7 decided in #2772958

[session] drop pre-3.7 api on session Dropping of API deprecated since 3.7 decided in #2772958

[transaction] use set operation in the hook control code We use direct operator on set instead of looping by hand, This simplify the codes.

[session/transaction] move hook management in transaction itself This is transaction level code.

[session] allow writable tx_attr and use it for commit_state This keep clarifying the code.

[session/transaction] move add operation in session code

[session/transaction] move entity life utility on transaction This access transaction related data and therefor should live in transaction.

[transaction] small simplification in ecache code A single assignation is simpler.

[session/transaction] move entity cache management on session A new utility method is added for that. It is similar to the one used for attribute access. code is copy pasted as his. Function documentation is added.

[session] have a nice helper function to forward access to session This allows a lighter and clearer code. This will also ease the addition of deprecation warning in the future.

[session] drop dead code session._tx is never None. New transaction are created if necessary.

[session] use tx.data directly in session code.

[transaction] rename transaction_data to data Transaction.transaction_data is redundant. A transaction_data property is left for backward compatibility. Deprecation mechanism will come in later changeset

[session] document Session._tx

[session] split session creation from default session assignation This is the first step for more independence for Transaction.

[session] make session lock reentrant We are going to use it for other session related business. It is renamed from _closed_lock to _lock in the same move.

[session] document set_tx

[session] rename txstore variable to tx This is a more accurate name

[session] rename `_threaddata` to `_tx` The returned object is a Transaction object.

[session] rename self.__threaddata.txdata to self.__threaddata.tx This is Transaction object now.

[session] rename _tx_data into _txs The object inside are now Transaction object. Not TransactionData object.

[transaction] relocate method building transaction context manager It is in the middle of methods going to be refactored.

[transaction] move RQLRewriter in Transaction This remove the last ``AttributeError`` magic. The ``RQLRewriter`` needs a Session. Its reset is handled in ``Session`` code to handle that

[transaction] initialize security control attribute in Transaction There is no reason to not initialize them in __init__ time. This simplify the Session code.

[session] move security constant out of the class There is no reason for it to not be global. This will ease it use by ``Transaction object``.

[transaction] initialize hook control attribute in Transaction There is no reason to not initialize them in __init__ time. this simplify the transaction code.

[session] Move hook control constants out of the class There is no reason for them to not be global. This will ease they use by Transaction object.

[transaction] initialize transaction data and state related attribute. There is not reason to not explicitly declare and initialize attribute in Transaction.__init__. This change requires the introduction of a Transaction.clear() method to get ride of some wicked ``AttributeError`` based logic.

[transaction] handle ``mode`` default value in Transaction The transaction mode is now explicitly passed at creation time and always read from the Transaction object. Note that there is a slight behavior change. The transaction mode is now set at the creation of the transaction. Changes made to the default value have no longer any effect on existing transaction.

[transaction] explicit Transaction cnxset attribute The Transaction object have a lot more attribute that those previously declared in __init__. They are just dynamically assigned in Session code. This changeset explicit the ``cnxset`` and ``mode`` attribute declaration.

[server] rename TransactionData to Transaction Transaction will become a full featured object.

[doc] update Session documentation A major refactoring of Session/Transaction handling is commit. This is a good occasion to improves the documention.

merge 3.16 fix in default

[devtools] fix a couple issues with xvfb-run xvfb-run didn't quite clean up correctly after itself, because: - dash doesn't run EXIT traps on signals - if we don't run the command in the background, a TERM handler doesn't run until the command exits

[devtools/qunit] don't open-code subprocess.Popen.terminate

[devtools/qunit] get rid of unused variable

[server] *init_repository* lookup the database instead of the schema to drop tables (closes #810743) So, tables are dropped even if the schema was changed. :note: if mssql, drop views before tables. :note: because of table dependencies we try a second drop sequence

[book] remove XXX to an old discussion about appengine

[web publish] in case of error, ensure proper http status is set and Content-disposition header is reseted. Closes #2553066

Add CubicWebRequestBase.content (closes #2742453) This is the body of the HTTP request (stream)

add the match_http_method predicate (closes #2559932) This aim at creating views which are selected on other HTTP methods (PUT, DELETE, ...)

[pkg] the previous commit requires new feature in yams.

add a command to compare db schema and file system schema (closes #464991)

drop typed_eid() in favour of int() (closes #2742462) typed_eid was introduced to abstract the eid type when running on Google AppEngine. It is not used anymore and can be removed. Let's use int() instead.

merge with stable

3.15 is the new old stable

[web] don't link to None in author box (when author has been deleted) (closes #2409855)

dbapi: try to restore compatibility of the connect api with 3.15 A few issues here: - urlparse('foo').scheme is '', not None - cnxprops is an optional argument Regression from 62213a34726e ("[db-api/configuration] simplify db-api and configuration so that all the connection information is in the repository url, closes #2521848") Closes #2754322

3.16 is the new stable After discussion with David Douard I'm merging 3.16.x branches in stable and starting 3.17 feature on default.

Added tag cubicweb-debian-version-3.15.10-1 for changeset feca12e4a618

Added tag cubicweb-version-3.15.10 for changeset 89bdb5444cd2

prepare 3.15.10

Added tag cubicweb-debian-version-3.16.1-1 for changeset 84fbcdc8021c

Added tag cubicweb-version-3.16.1 for changeset d95cbb7349f0

prepare 3.16.1

[views/primary] some inner sections should use the `limit` by default to avoid a denial of service (closes #2719110) Today, it is possible to call .related and get a huge unlimited database-dos-inducing resultset that will be nevertheless limited a bit further in pure python in the `autolimited` view. While we cannot completely avoid potential denial of services such as these we mitigate the problem with the default ui settings: if the inner vid is `autolimited`, then the relation result sets is computed using the user-defined limit. This change respects the semantics of the `autolimited` view and shouldn't break anything.

[entity] ensure the .related(entities=False) parameter is honored all the way down (closes #2755994) As of today, such a call will always fill the relation cache by calling .entities() on every single related rset entry. As a consequence, the `limit` parameter handling also had to be fixed. It was bogus in the following ways: * not used in the related_rql, hence potentially huge database requests, but also actually * foolishly used in the .entities()-calling cache routine we now bypass (this changeset ticket's main topic) Now: * we set a limit on the rql expression, and * forbid caching if given a non-None limit (as we don't want to make the cache handling code more complicated than it is already) With this, entity.unrelated gets a better limit implementation (so the code in related/unrelated is nice and symmetric) Risk: * _cw_relation_cache disappears completely, which is good, but this is Python, so you never know ...

[test/web] fix invisibly bogus test (prepares #2755994) The test was wrong but that was cancelled out by a cache effect and fuzzy naming. Wiping the entity caches restores sanity: the choices list are the same before and after the SET. Also field.choices uses entity.unrelated but always returns related + unrelated elements. Hence `choice` replaces `unrelated` where it makes sense. AssertIn is used in place of AssertTrue.

[cwconfig] Fix exception handling when building the cube dependency graph UnorderableGraph exceptions do not have a 'cycles' property. A simple cast to str does the job.

[merge] backport stable fixes

[test/sobject] fix test regression Was complaining: - * updated comment #EID (#EID) ? ^^^^ + * updated comment #EID (duh?) ? ^^^^ in sobjects/test/unittest_supervising.

[web/request] Prune extraneous 'pageid' from generated ajax URL parameters (closes #2758130) If 'pageid' is given through extraparams, it is sent twice to the browser. On the JS side, the final URL loadxhtml() will end up using will have 'pageid' set twice which CubicWeb will readily accept as a list. Pruning this parameter makes sure it is exactly once.

[web/component] Use global variable to point to ajax controller (part of #2758254)

[web] Use the new '/ajax' URL path to access the AjaxController (closes #2758254) 5a81fa526b30 took care of all the JS code, this patch cleans up the remaining python bits.

[devtools/httptest] fix syntax error introduced by ce5ae7b80d2c

[component] Fix URL generation for navigation component (closes #2464832)

merge with another default heads

Merge stable into default (NON-TRIVIAL) - dropping __future__ statement added by d2472948da9c - added migration from 407acc41beb5 to migration file for 3.16.1

maintain python2.5 compatibility

[schema/workflow] one more typo fix Also add proper migration.

[devtools] add http_publish to CubicWebTC (closes #2565882) Fakes a http request, without overriding the normal error handling. Returns the request object so the caller can check for errors.

[testlib] url_publish should give url to the request and the rset returned by the path evaluator to ctrl_publish. Closes #2557468 Also, url_publish now accepts a `data` argument to simulate POST of values.