[pkg] fix cubicweb.spec syntax

[cwconfig] Use imp.load_source instead of exec() to load site_cubicweb.py from apphome Lets us return a module object and go back to the previous code in CubicWebConfiguration._load_site_cubicweb.

[cwconfig] Stop using lgc.modutils to load *config.py and *ctl.py files

[cwconfig] Stop using lgc.modutils to load site_cubicweb.py files config.load_site_cubicweb function now takes a list of cubes (instead of paths) and default to all known cubes.

[test] Use a Popen in i18n tests Instead of check_call, so that we clearly see the failure. A bit of flake8 along the way.

[forms] Fix HTableFormRenderer _render_fields method behavior when display_label is False. Indeed, now the extra <tr><th>...</th></tr> are not inserted anymore unless display_label is True. Closes #3811325.

[connection] remove a dead counter Related to #1381328.

[session] drop .is_repo_in_memory This is deader than a dead dog's bone buried down a blind alley off a dead-end street in a ghost town. Related to #1381328.

[session] drop session-as-a-context-manager It made little sense in the 3.19 - session-as-cnx-bw-compat era, maybe. Related to #1381328.

[devtools/repotest] use the RepoAccess object Quite better than playing tricks with the Repo._sessions. Related to #1381328.

[devtools] adapt stresstester to current APIs

[session] retire session.vreg Related to #1381328.

[session] fix super call in Connection.__init__ The vile hacks in test fixtures are fixed.

[session,repo] remove last trace of "cnxprops" Related to #1381328.

[pkg] Use Twisted < 16.0.0 We have some test failures with 16.0.0, such as: - test_put_content (unittest_server.ETwistHTTPTC) - test_base_url (unittest_httptest.TwistedCWAnonTC)

[web/views] stop using cwaction='cancel' It's broken as of https://hg.logilab.org/master/cubicweb/rev/cb217b2b3

[test] fix test_printable_value_bytes with current pygments An empty span was added in https://bitbucket.org/birkenfeld/pygments-main/commits/164574c13533195a555181a2b9c685fea2470403

[session, repository] deprecate repo.connect and move .close reponsibility to session object Repository.new_session just returns a plain session object, and `.connect` (which returns a sessionid) is deprecated. For .close:: session.close() # done ! The session will bear the responsibility to call the "session_close" event but that's better than the previous idiom:: repo.close(session.sessionid) which involves both objects. Related to #1381328. Related to #2919309.

[web/test] fix typo in doc string

[web/httpcache] send weak ETags There's no way we can provide a strong validator in general (especially considering how ETags are generated by the current EtagHTTPCacheManager and EntityHTTPCacheManager). So mark our entity-tags as weak.

[server] print backend exception if we fail to execute an sql script It's more useful to see could not open extension control file "/usr/share/postgresql/9.4/extension/postgis.control": No such file or directory rather than merely -> ERROR, skipping /usr/share/cubicweb/cubes/postgis/schema/_ext.postgres.sql

[cwctl] drop unnecessary admincnx call from 'shell' The connection is done by the migration helper.

[serverctl] allow schema-diff when db and fs are not in sync

[cwctl] remove networked remote shell This was a dbapi leftover. Related to #3933480.

[web/calendar] fix regression in previous changeset "Enables css customization in CalendarView" was untested, and introduced a crash: TypeError: not all arguments converted during string formatting

Enables css customization in CalendarView now a custom css class can be set on the calendar div (closes #3654043).

refactor a bit breadcrumbs to ease overriding of the root's display

Add migration helper function to delete entity types faster Bypasses all hooks and (for now) ignores any composite relations.

[testlib] Import nocoverage from logilab.common.testlib This depends on a change in logilab-common 1.2.0

[web/test] replace unicode with six.text_type Some uses of unicode reappeared when merging changes from 3.19.14.

[devtools/testlib] Issue a RuntimeError when a generative tests is detected Otherwise, generative tests *à la logilab-common* in CubicWebTC test cases would silently not run (at least using logilab.common.pytest runner).

[devtools] stop using lgc.testlib.TestCase It's become incompatible with unittest(2).TestCase due to generative tests. Just keep the datadir/datapath and assertCountEqual bits in our BaseTestCase.

[test] Replace remaining generative tests by subtests

[tox] Enable all applicable py34 environments All except etwist. Comes with a few tricks in tox.ini: * add ``-e.`` to force installing of cubicweb in development because dependency cubes would otherwise install cubicweb 3.22 (the latest release), which is not python3 installable. * "install" cubes/__init__.py because the file is not installed in develop mode; maybe this file should be managed by cubes themselves...

[migration/3.10] fix AttributeError 'Connection' object has no attribute 'HOOKS_DENY_ALL'

[server/sources] Fix authenticate on pre-3.10 database When migrating from a pre-3.10 database, the CWSource entity type and cw_source relation type are not known, but authenticate should still work to be able to add them.

[test] Add a test ensuring new cube built from skeleton is flake8-compliant And make skeleton files actually flake8-compliant henceforth.

[skeleton] Ship a tox.ini file upon new cube

[web] fix headers on 304 responses (closes #11466875) The response MUST include the following header fields: [...] - ETag and/or Content-Location, if the header would have been sent in a 200 response to the same request https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.5 Unfortunately we weren't copying headers from what would have been the 200 response, but from the request.

[web/httpcache] GMTOFFSET no longer exists

[web] fix headers on 304 responses (closes #11466875) The response MUST include the following header fields: [...] - ETag and/or Content-Location, if the header would have been sent in a 200 response to the same request https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.5 Unfortunately we weren't copying headers from what would have been the 200 response, but from the request.

[rset] fix infinite recursion introduced in c1eb5a676c80 The mechanism to avoid loops in c1eb5a676c80 "[rset] Always complete attribute/relation caches in ResultSet.get_entity" breaks down: we would loop forever if two entities that were already in the cache were linked by a relation with ? or 1 cardinality in both directions. To avoid that, keep a set of already-considered columns. Related to #9942503.

pep8 unittest_application and unittest_formfields

[rset] Always complete attribute/relation caches in ResultSet.get_entity RQL queries are often designed to fill up the ORM's caches when fetching entities out of the result set. Until now, if an entry already existed in the entity cache, ResultSet.get_entity would return it unchanged, not using the new ResultSet's contents to update the attribute cache, breaking expectations (if the attributes are needed, they'd then be fetched later one at a time, one entity at a time), resulting in loads of DB accesses. So we change ResultSet.get_entity so that: * if the entity is already cached and has been instantiated from the same rset, it is returned as-is (to avoid loops) * if the entity is not yet cached, it is instantiated * if the entity is cached via another rset, its attribute/relation caches are completed Closes #9942503

[rset] refactoring of _build_entity method define a method for entity instantiation and storage in cache (related to #9942503)

[entity] comparing two entities returns NotImplemented Fixes a test when using some versions of unittest2, where assertItemsEqual tries a sort and expects TypeError, which it doesn't get if we directly raise NotImplementedError. The NotImplemented value is treated specially by the interpreter: https://docs.python.org/3.4/library/constants.html#NotImplemented

[test] replace one use of lgc.testlib.unittest_main with unittest.main

[test] stop using TestCase.set_description It's a lgc-ism.

[test] convert a lgc-style generative test into unittest2-style

[web/application] fix query log handling on python3 Writing bytes to a file opened in text mode doesn't work.

[devtools/test] duplicate CubicWebServerTC tests for CubicWebWsgiTC

[devtools/test] fix bytes vs text in test_base_url Needed for python3 with the next changeset

[devtools] skip CubicWebServerTC on python3 People should migrate to wsgi if they're going to use CubicWeb on python3.

[web/test] switch unittest_web from twisted to wsgi And make it pass on python3 while at it.

[devtools] add CubicWebWsgiTC Similar to CubicWebServerTC, except using wsgiref instead of twisted as an http server.

[web/test] return the hash of uploaded files in FileUploadTC instead of their contents json must be unicode, which doesn't go well with arbitrary file contents. Compute the file hash instead, as we know the hexdigest is (ascii) text.

[web/test] open() as a context manager

[web/test] json.loads expects text, not bytes

[wsgi] fix handling of multiple posted files We were only keeping the last received file.

[web/test] Reword doc string for fileupload ajaxfunc

[skeleton] do not import the __pkginfo__.py module to make cubes installable via setuptools When using a simple "import" statement, running 'python setup.py install' for a cube fails in a fresh virtualenv due to the fact the install of the dependencies (here cubicweb iteself) is done by the python process doing the cube's installation, so the __pkginfo__ module from the cube is already loaded (thus used) when installing cubicweb... So we apply the third technique from: https://packaging.python.org/en/latest/single_source_version.html

merge changes from 3.22.2

Added tag 3.22.2, debian/3.22.2-1, centos/3.22.2-1 for changeset 1b93ff37755b

[pkg] 3.22.2

[pkg] fix centos install

[datafeed] do NOT call commit in DataFeedXMLParser.process() (closes #10928592) and so, remove more useless code. Doing a commit in DataFeedXMLParser.process() is a bad idea because this method is called for every imported/updated entity, before handling deletions. So if we commit here, odds are quite high that we're violating some integrity constraint and end up with a ValidationError.

[schema] restore constraint checking when running on old sqlite Old sqlite3 doesn't provide CHECK constraint names in error messages, preventing us from translating a backend integrity error into a ValidationError. This was added in 2012, but the sqlite3 version in RHEL6 is older; so if we run on old sqlite, keep checking the constraints in python rather than only in SQL. Closes #10927494

[web/autoform] fix regression introduced by #5758ba784ebd The changes from cset 5758ba784ebd "[autoform] when building inlined-form view for a relation, consider related entities intead of other views" lead to an unexpected 'add new' link on creation of an entity with inlined form for a relation of cardinality 1. To avoid this, we should consider formviews and not existing relations when the entity is being created. Closes #10860258

[tox] Generate test environments for Python 2.7 and 3.4 Test commands for each environment are written down explicitly since I could not find a way to extract the "package" name (e.g. "hooks") from the environment name (e.g. "py34-hooks"). For Python 3.4 interpreter, only environments (subpackages) that do not depend on cubes for their tests are listed since those test dependency cubes are not yet installable with Python 3.x. etwist is also not included since the Python 3 port is not complete at the moment. From local testing, py34 tests pass for the following subpackages: - dataimport - entities - ext - hooks - wsgi

[notification] avoid using InternalManager when we've got a CWUser This will avoid potential information leak due to missed security restriction and is more compatible with legacy code (pb encountered in some client code). Closes #10294761

[dataimport] rename .time to .now attribute of meta generator

[pkg] Set version to 3.22.2.dev0 So that cubes used in test dependencies do not install a released CubicWeb.

[dataimport] cleanup a bit nohook store, mostly protecting some attributes

[dataimport] add a filter function to not fail if some extentity has several values for an attribute of final relation the function will simply record a warning in the import log and keep a value randomly. Notice the function has to be explicitly inserted in the transformation stream.

[sources] synchronize source asynchronously when started from the UI and redirect to the forthcoming import log, for a better user experience (there is still a lot more to do there though). Closes #10468967

[test] fix unittest_devctl on python3.4

[test] Make sure that newcube command test produces expected files

[server] remove some pre 3.10 compat Let's assume nobody will try upgrading an instance from pre-3.10 cubicweb to 3.23+. If they do, they're setting themselves up for a lot of pain anyway.

[pkg] use https URLs to pypi.{python.org,debian.net}

[pkg] use https URL for www.cubicweb.org

README: use https URLs for docs.cubicweb.org

pep8 bits

[i18n] enhance text formats french translation

Merge with 3.22 branch The merge was clean, just dropped cubicweb/web/data/cubicweb.goa.js.

[dataimport] update example to use standard importer

merge changes from 3.21.6

flake8 serverctl

c-c source-sync now actually force synchronization as advertized by its help message. This was because "force" argument wasn't considered by acquire_synchronization_lock().

Added tag 3.21.6, debian/3.21.6-1, centos/3.21.6-1 for changeset b3cbbb7690b6

[pkg] 3.21.6

merge changes from 3.20.13

Added tag 3.20.13, debian/3.20.13-1, centos/3.20.13-1 for changeset 8c5dabbcd4d9

[pkg] 3.20.13

[pkg] Yams >= 0.42 breaks CubicWeb < 3.21 Closes #9720390.

merge changes from 3.19.14

Added tag 3.19.14, debian/3.19.14-1, centos/3.19.14-1 for changeset 72a0f70879ac

[pkg] 3.19.14

[sources/native] remove extra space before colon This isn't French.

[web/views] fix editcontroller regression unittest_views_basecontrollers.py's EditControllerTC.test_edit_mandatory_inlined3_object regressed with changeset dcbb64d3a1d9 because it relied on POSTing nonexisting eids. This seems like a silly idea, but let's keep the behaviour unchanged.

Added tag 3.22.1, debian/3.22.1-1, centos/3.22.1-1 for changeset d0d86803a804

fix test broken by #012e6c7d02ef

[pkg] Attempt to fix rhel6 build I don't know what I'm doing, but this seems to work.

[dataimport/test] fix python2.6 syntax error Don't use braces for set literals.

Remove unused import in entity.py

[migration] don't ask confirm for those sql queries

quick fix which closes #0673348 - see comment for more info

[migration] test if entity type has been in the deleted in the transaction should protect both clauses else we may attempt to execute the UPDATE query on a table which has been deleted. Closes #10692374

[pkg] 3.22.1

[cwctl] remove support for startorder file in /etc/cubicweb.d (closes #5618947) It's now useless.

[exceptions] fix string conversion for Unauthorized exception changeset 3cb87b61b067 changed CubicWebException to define __unicode__ as the main entry point for string conversion, but left Unauthorized's __str__ definition unchanged, leading to a mismatch. Closes #10646396.

Fix Binary on python < 2.7.4 (closes #10593811) Before the fix for http://bugs.python.org/issue10211 we can't pass a buffer object to BytesIO, so keep using StringIO in that case. And because StringIO is an old-style class, we can't use super().

[migration] fix crash on 3.22 upgrade if entity type doesn't exist in fsschema (pending removal)

[autoform] when building inlined-form view for a relation, consider related entities intead of other views by considering previous form-views, we may miss that there exists related entities but which are not editable. Closes #10495602

Add a __main__ to cubicweb This avoids calling cubicweb-ctl relatively in tests and could be useful in virtualenv. This is just a copy of bin/cubicweb-ctl file, which should probably be generated from a setuptools entry points.

merge 3.19 heads Getting an unexpected test error though: ERROR: test_edit_mandatory_inlined3_object (unittest_views_basecontrollers.EditControllerTC)

[cwvreg] Fix propertyvalues hot updates When CWProperty instances are touched, the corresponding value gets updated in vreg['propertyvalues'], but their type was ignored.

[pkg] remove useless python >= 3.4 dependency python is python2 is never >= 3.4.

[dataimport/test] fix test failures since 35b29f1eb37a The index name changed, so we need to follow suit.

[pkg] do not include toplevel directories in rpm (closes #10362119) Our spec file conflicts with the filesystem package in centos7: we set permissions to 0755 for /usr/bin and /usr/lib whereas filesystem sets them to 0555. So to be compatible with both centos7 and others we simply ignore these directories.

[devtools] fix config.repository signature Its argument is called 'vreg', not 'x'. Closes #10515113.

[predicates] fix paginated_rset when the class has no page_size_property Fixes test failure introduced by 70733b2dfd2a, which assumed paginated_rset would only be called on a class with a page_size_property attribute.

[repository] drop usage of no more necessary eschema_eid function since the previous cset, we are guaranteed that repository's entity schema will have their eid attribute properly set, so we don't have anymore to check everytime if some entity schema's .eid attributes is set or not (and retrieve it if not). Related to #10450092

[repository] set .eid on eschema when schema is loaded from the filesystem enforcing the contract that a repository's schema should have .eid attribute of entity schema set to the eid of the entity used to serialize them in the db. Before this cset, this was not true during tests or for some c-c commands where 'quick_start' is set (eg db-restore, i18ncube...). The change in server __init__ makes this assumption true during instance creation: the serializing code was actually setting eid on schema object, but a reference to a previously built schema was given instead of the one for the latest created repository. Closes #10450092

[dataimport] simplify SQLGenSourceWrapper.add_info At this point 'CWSource' is a thing (we're not in the middle of a migration), and source.eid won't be None (we read it from the db)

ensure .tox is blacklisted when recursing on the fs to get some files among others, avoid crash when using 'c-c i18ncube' on a cube using tox. Closes #10429317

drop a note about usage of sqls_create_multicol_unique_index and avoid a lgdb warning

[devtools] don't auto-populate virtual relations

[skeleton] Restore verbose output in setup.py This part of commit fdadf59be612 missing the integration window.

[hooks] Skip modification_date attribute in "notifentityupdated" hook Depending on the call order of hooks, "modification_date" may come in entity.cw_edited or not (from "metaattrsupdate" hook usually). It should however not be considered in the notification context. Closes #8842064.

[test] Gather tests about mail module From sobjects/test/unittest_notification.py to test/unittest_mail.py where other tests for mail.py are already. Some style fixes along the way.

[web/component] guard against invalid page_size form parameter Same as dff28956f37e, except the almost same code exists in two places (yay us).

[predicates] fix paginated_rset for components that change page_size_property The predicate and component had better agree on the page size.

[devtools] fix get_bounds Say an attribute has two IntervalBoundConstraints, one with minvalue=0 and the other with maxvalue=10 (not literally, or that could be a single constraint, but e.g. with references to other attributes). We don't want to override the minvalue of the former with None just because the latter happens last in the constraints list. This doesn't fix the case where two constraints both have a min or a max value. Meh.

[devtools] "fix" boundary condition for autofill maxvalue in an IntervalBoundConstraint is inclusive. If the min and max values happened to coincide, we would call random.choice on an empty list, which it didn't like. This is still broken in the case of BoundaryConstraint('>', minvalue), as we assume the minvalue is in the range, but that's something for another time.

[test] Use stdlib doctest module instead of logilab-common class The former just works fine now but the later does not play well with py.test.

[test] Re-implement generative tests with subtests

[devtools] Re-implement generative tests using subtests Generative tests as implemented in logilab.common.testib are not compatible with tests runner other than lgc.pytest and this implementation differs from the standard library, which has support for subtests_ since Python 3.4. Use unittest2 to bridge the gap. Maybe it'd be good to implement this on logilab.common.testlib side at some point. Let's see how this gets received here first. .. _subtests: https://docs.python.org/3/library/unittest.html#subtests

cleanups

[pkg] python-tz does not exist in centos should be pytz (closes #10290056)

[test] Skip LDAP tests if any of the underlying programs is not found Some programs are in ldap-utils or ldapscripts, which may not be installed despite slapd may. Follow-up on 282880f81311.

Handle virtualenv development mode Looking for VIRTUAL_ENV environment variable as a sign of virtualenv activation. This at least works for developing CubicWeb itself (not sure yet about cubes). And it makes `tox --develop` work as well. Add a test for cwconfig._find_prefix within a virtualenv. In tox.ini, use `develop option`_ accordingly. This makes tox running significantly faster since it avoids building and installing the sdist. Aside, the "develop" install is now performed *after* all test requirements are processed, which means that the cubicweb package should always be the one from source -- instead of a released version pulled by a cube through test dependencies, which may occur in "recreate" step. .. _`develop option`: <http://tox.readthedocs.org/en/latest/config.html#confval-usedevelop=BOOL

[skeleton] Use os.walk in setup.py for Python3 compatibility

[skeleton] Use unittest.main directly Instead of lgc.testlib.unittest_main as they are identical.

[devtools/testlib] Disable pause trace in setUp Closes #9856028.

[test] Remove dependency on person cube in devtools package

[pkg] add missing python-tz dependency

[edit controller] Fix composite entity reparenting when inlined This implies handling the reparenting case sooner, otherwise the child entity modifications are cancelled (see 113e9da47afc). Add a new test to ensure non-regression in all previous tests. Closes #10291977.

[forms] Do not raise errors on composite required relation being deleted The entity included in the composite will be deleted in such a case, so the field is not required even when the cardinality is '1' or '+'. Related to #8529868

[edit controller] Cancel RQL queries to be performed on entities to be deleted Composite relation removal via the EditController trigger an entity removal. Changes on such a to-be-removed entity must be cancelled as they may trigger integrity errors before the deletion occurs. Closes #8529868.

[edit controller] fix handling of removal of subentities from an edit form In CubicWeb 3.18 the semantics of dropping a composite relation were changed. It had before the effect of deleting the subentities. Now the edit controller must handle this by itself (it was not adapted then by mistake). So if it detects the removal of a composite relation (on the browser side, the user clicked on the "remove" action), the subentity is now explicitly deleted. It was chosen not to change the form generation but the EditController: - to ensure pre-3.18 forms backward compat - to avoid the introduction of a specific entity-deletion API (RQL is probably our best API for complex DB operations) Related to #8529868.

remove unused import

[editcontroller] a slight code reorganisation, in preparation for a fix * `entity` defined at method entry * a combined `continue` and a dedent of the next big block * some vertical white space Related to #8529868.

[sources/native] ask permission to continue restoring even if metadata do not match (closes #7771864) Moreover we try to clarify the message (Expected/Found was never too clear for users).

[views] fix ProcessInformationView: SESSION_MANAGER can be None (closes #5753280) eg. with pyramid. (grafted from 6a07c2dc3d85f4de0cf48221607e02c4473e11e4)

[web] Move "display all"/ "back to pagination" link generation into the navigation component This makes the html generation customizable. Closes #9219057

Reorganize source tree to have a "cubicweb" top-level package Basically: mkdir cubicweb hg mv *.py -X setup.py cubicweb hg mv dataimport devtools entities etwist ext hooks i18n misc schemas server skeleton sobjects test web wsgi cubicweb Other changes: * adjust path to cubicweb-ctl in devtools tests * update setup.py to avoid importing __pkginfo__ (exec it instead), replace os.path.walk by os.walk and prepend `modname` here and there * update tox.ini to account for new test locations * update doc/conf.py so that it still finds __pkginfo__.py and CWDIR in doc/Makefile

clean some remaining dust from goa time

[datafeed] io.BytesIO requires a buffer, not a unicode (closes #9783743) The URLLibResponseAdapter should probably be refactored (or killed?)

[pkg] rql 0.34 and logilab-database 1.15 break cubicweb 3.21

[pkg] rql 0.34 and logilab-database 1.15 break cubicweb 3.21

Added tag 3.22.0, debian/3.22.0-1, centos/3.22.0-1 for changeset de472896fc0a

[pkg] 3.22.0

merge

[formwidgets] a bit of pep8

[views] improve JQueryDatePicker so that a date field can be defined as the min/max of an other one closes #5169039

[entity] don't look at fetch order recursively If fetch_attrs includes relations, stop including the entities on the other side in our sorting. Fixes regression from 73ea636a5562 where we would sort on the target entity before the attribute we wanted if the relation's name sorted before the attribute's. This showed up in the forge/tracker cubes with Version entities being sorted by their State's name in preference to their version number.

[pkg] fix paths in debian doc-base control file

[ldapfeed] extid and cwuri aren't the same thing Historically, we used a ldap://... URI as cwuri attribute, not just the dn. The extid field, OTOH, was always the dn. So when initializing our dict of known entities from an ldap source, look at existing extids, not cwuris, to avoid importing already-existing users (and violated unicity constraints).

[entities] add eid to ISerializableAdapter Life without an eid is made of sadness.

[doc] 3.22 release notes update

[datafeed] use tz-aware datetime objects Avoid TypeError when comparing tz-naive and tz-aware objects in fresh() method.

[pkg] fix syntax error in debian/control

[pkg] remove no-longer-used addresses from debian control file

[schema] rebuild computed attributes on schema reload If we have new computed attributes we need to set their formula_select attribute or things get sad.

[hooks] fix extra_props handling on python3 The json module expects str, not bytes, so we need to do the decoding.

[py3k/ldapparser] avoid confusion between unicode and bytes - ldap3 dicts contain unicode strings - extid must be bytes

[pkg] use correct lgdb version for tz rework Related to #4848923.

pep8 bits

[server] improve TZDatetime support by depending on logilab-database 1.15.0 including https://www.logilab.org/ticket/1485893, we can now keep the tzinfo attribute on datetime objects for TZDatetime attributes, so one knows that this is a tz-aware datetime. To easily make it work with backends that have no tz support, we keep converting tz-aware datetime objects into utc naive datetime objects before sending them to the database.

[schema] Use TZDatetime for creation_date and modification_date No work has been done for form fields/widgets since creation_date / modification_date are by default not editable through the default UI. One may want to add such feature at some point. Time will be displayed as UTC, let the end-application decide otherwise by customizing the tzdatetime printer if desired. Closes #4848923

[server] fix SQLite patch to ensure TZDatetime are properly typed NB: at this point, the TZ info is completely ignored. It will be properly handled in later commits.

[server] fix name and add docstring to connection initialization functions

[server] always monkeypatch QuerierHelper to handle dates and times on SQLite The patch prevents problems due to a SQLite bug causing some values to be returned as strings instead of properly typed objects. This should probably be improved to use SQLite connection hooks. But the major improvement is that the monkeypatch is now done all the time, and not just for tests.

[testlib] put postgres test database in test/data/database directory with other (sqlite) test databases. Closes #9240765 Simplify .hgignore in passing.

[dataimport] make eids_seq_range as massive store instance attribute again I changed it to a class attribute recently, but on further thought (and prompting from Adrien) it seems to make more sense as an instance attribute.

[dataimport] stop supporting None as eids_seq_range It couldn't work as-is because entities_id_seq is not a sequence object.

[dataimport] remove eids_seq_start attribute from massive store Instead provide a utility function to reset the eid sequence.

[dataimport] massive store in slave mode shouldn't flush metadata The metadata insertion is to be done only by the master-mode store.

[dataimport] check that MassiveObjectStore restores the db schema properly Ideally we'd use "pg_dump --schema-only" to compare before/after, but when restoring a constraint such as: cstr48f9ceae537d68859b62c601681c8d3e CHECK (((cw_type)::text = ANY ((ARRAY['normal'::character varying, 'auto'::character varying])::text[]))) postgres will convert the text cast on array to a cast on each cell of array cstr48f9ceae537d68859b62c601681c8d3e CHECK (cw_type::text = ANY (ARRAY['normal'::character varying::text, 'auto'::character varying::text]))

[dataimport] split out listing indices/constraints from getting the corresponding query

[dataimport] massive_store: do not drop constraints multiple times It is now done in __init__.

[dataimport] massive_store: do not drop constraints with `cascade` since it breaks things We want to be able to restore constraints at the end of the import. If they are silently dropped due to a CASCADE, we can't do that.

[dataimport] massive_store: drop all constraints in __init__ and restore all in finish This should avoid dependency issues when dropping specific constraints (e.g. when dropping a primary constraint referenced by a foreign key constraint). We only do this in master mode, slave mode stores are supposed to work with an initialized db already.

[dataimport] massive_store: ignore primary key when requesting indexes on a table Primary keys show up in both lists of constraints and indexes. Pick a side and stick to it.

[dataimport] massive_store: reformat sql query

[devtools] Use super() in TestServerConfiguration.__init__() While cleaning up the use of init_config() in WSGIAppTC, I wondered why the method was trying to set the 'https_uiprops' and 'https_datadir_url' attributes on the config, when WebConfiguration's __init__ should be taking care of setting them. WSGIAppTC uses the default configuration - one that derives from ApptestConfiguration (noted as G below). The issue is that the class hierarchy of ApptestConfiguration includes TestServerConfiguration (C in the graph below) which calls ServerConfiguration's __init__ directly (B below). As B does not have an __init__ method, CubicWebConfiguration's __init__ is called directly (A below). This has the unintended consequence of preventing WebConfiguration's __init__ from being called. A / \ B D | | C E \ / F | G | ... A: cwconfig.CubicWebConfiguration B: server.serverconfig.ServerConfiguration C: devtools.TestServerConfiguration D: web.webconfig.WebConfiguration E: etwist.twconfig.WebConfigurationBase F: devtools.BaseApptestConfiguration G: devtools.ApptestConfiguration NB: unittest_wsgi wasn't really broken, as it's the only test in wsgi/test/, but I figured I might as well clean it up. Related to #9297611.

[devtools] Fix database creation issues for CubicWebTestTC tests To reproduce (with a purged repo): * python web/test/test_jscript.py * python web/test/unittest_views_xmlrss.py The '__default_empty_db__' will contain the wrong 'base_url' configuration, thus breaking URLs in the xmlrss tests. Closes #9297611.

[devtools] Fix database creation issues for CubicWebServerTC tests To reproduce (with a purged repo): * python web/test/unittest_web.py MiscOptionsTC * python web/test/unittest_views_xmlrss.py The core issue is that CubicWebServerConfig.default_base_url() is called inside init_config(), which is called during the equivalent of 'db-create'. The '__default_empty_db__' will contain the wrong 'base_url' configuration, thus breaking URLs in the xmlrss tests. Related to #9297611.

[devtools] Move attribute docstring to correct location

[web] Make tests pass if tal isn't available

[py3k] fix cubicweb.mail.parse_message_id base64.b64decode returns bytes. Fixes sobjects/test/unittest_notification.py.

[test/py3k] fix test_recipients_finder AssertionError: <zip object at 0x7f1dfb0230c8> != [('abcd@logilab.fr', 'en'), ('efgh@logilab.fr', 'en')]

[dataimport] fix SQLGenObjectStore's add_info for python3

[dataimport] py3k doesn't like + on dict_items objects

[repository] remove unused check_session method and its test Related to #1381328.

[migration/test] avoid test interdependency by removing fakecustomtype cube There is still a failing test afterward but it doesn't seem related to changes in this series.

[migration/test] use numeric(precision, scale) instead of geometry The latter requires the postgis extension which is overkill for this test.

[server, hooks] allow callable in dbh.TYPE_MAPPING This is necessary for cubes introducing new custom final types whose generated sql may depend on some extra configuration, avoiding ugly-buggy hack as done currently by the postgis cube. Closes #7569998

[migration] consider extra_props in rdef created during attribute creation hook else we may miss important data such as type parametrization (see postgis.Geometry for instance). Related to #7569998

[migration] only define variable if needed

[migration] handle non-existing attribute type when adding an entity type The final type needs to be added to the schema first. Related to #7569998.

[migration] handle the case where new final entity type is not yet in the schema This may occur on new cube addition. Related to #7569998.

[migration] config.add_cubes should ensure site_cubicweb of new cubes is called Related to #7569998

[migration] test addition of a cube providing a custom final type This does not work yet. Related to #7569998

[server] add test demonstrating cache update pb since cw > 3.21 Fixed by the previous changeset.

Bring back the separate web-side entity cache Prior to changeset 635cfac73d28 "[repoapi] fold ClientConnection into Connection", we had two entity caches: one on the client/dbapi/web side, and one on the server/repo side. This is a waste, but it is actually needed as long as we have the magic _cw attribute on entities which must sometimes be a request and sometimes a cnx. Removing the duplication caused weird problems with entity._cw alternating between both types of objects, which is unexpected by both the repo and the web sides. We add an entity cache on ConnectionCubicWebRequestBase, separate from the Connection's, and bring back the _cw_update_attr_cache/dont-cache-attrs mechanism, to let the server/edition code let caches know which attributes have been modified Entity.as_rset can be cached again, as ResultSet no longer modifies the entity when fetching it from a cache. Contrary to the pre-3.21 code, _cw_update_attr_cache now handles web requests and connections in the same way (otherwise the cache ends up with wrong values if a hook modifies attributes), but dont-cache-attrs is never set for (inlined) relations. Closes #6863543

Backout "[entity,storage] remove hackish code from the bfss path The merge of the repo-side and web-side entity caches doesn't actually work, because entities have a _cw attribute which is sometimes a connection and sometimes a web request, and whose nature changes not-really-predictably whenever ResultSet.get_entity() is called and brings it out of the cache. This reverts commit 59a79300f213.

merge with 3.21.5-2

[doc] add documentation about drop_cube

Added tag debian/3.21.5-2 for changeset 228b6d2777e4

[pkg] don't install dataimport/test in both cubicweb-server and cubicweb-dev Closes #9239572

merge with 3.21.5

Added tag 3.21.5, debian/3.21.5-1, centos/3.21.5-1 for changeset e0572a786e6b

[pkg] 3.21.5