[dataimport] introduce usage of MetadataGenerator into the massive store This allows more flexibility and code sharing w/ nohook store. Closes #10338621

[dataimport] rename massive store's metagen_push_relation method to _insert_meta_relation.

[dataimport] meta_data -> metadata

[dataimport] use sql shortcut

[dataimport] move cwmassive_constraint temporary table handling to the PGHelper class with consistent renaming and some rationalization to avoid doing things several times and have code slightly easier to grasp.

[dataimport] inline some methods of the massive store

[dataimport] move everything related to the "URI handling" feature of the massive store together

[dataimport] test for a value is in a set and insertion in a set should live together also inline some methods on the way, in the hope to make the public api clearer at some point, and only put valuable values in those sets so we don't have to endlessly test for consistency.

[dataimport] use a single _initialized set instead of _entities / _rtypes and rework a bit methods using them so that 'in' test and set insertion leave in the same method.

[dataimport] drop massive store etype to eid index and fix potential problem with is_instance_of relation insertion in case where the class of inserted entity has some (yams) parent.

[dataimport] add a .schema shortcut attribute on the massive store also rephrase a bit a method along the way

[dataimport] rework massive store's __init__ * _cnx is set by the parent class * count_cwuri is not used * group related things together

[dataimport] rework PGHelper class

[dataimport] flake8

[dataimport] enhance a bit sql queries readability

[dataimport] use IF EXISTS when possible it's better than try/except and avoid potential concurrency problems.

[dataimport] PGHelper should be responsible to retrieve the database schema

[dataimport] introduce a new MetadataGenerator to replace former MetaGenerator with a similar-but-different API that will allow its usage by the massive store (no more entity object manipulation), while making it a bit more flexible for customization by subclasses (experience taken from client code). Related to #10338621

[dataimport] Prefer now(pytz.utc) to utcnow

[dataimport] rename nohook store .source attribute so it's clearly not the samething as metagenerator.source. Also add a bit of doc along the way.

[dataimport] make rql store test easier to read and test a few more thing, then reuse it to test nohook start which had no tests at all

[dataimport] remove useless assignment in massive store

[dataimport] move eid attribution from the metagenerator back to the nohook store

[dataimport] rephrase minor bits of the meta generator

[pkg] Depends on logilab-common >= 1.2.2

Merge with 3.22.3

[pkg] Depends on yams >= 0.43 RPM spec file and debian/control got updated previously.

Added tag 3.22.3, debian/3.22.3-1, centos/3.22.3-1 for changeset b1e7de000536

[pkg] Version 3.22.3

[pkg] Depends on yams < 0.43

Define BASE_URL and DATA_URL js variables in the login template. Closes #13385185 Those are expected to be defined in almost every js files. They were not before this patch because login form doesn't use the regular main template.

[web] Remove some unused imports in editforms

Fix cancel action on deletion confirmation form Currently crashed with RequestError: invalid action '__action_cancel'. It's broken as of https://hg.logilab.org/master/cubicweb/rev/cb217b2b3 This fix is similar as the one done in https://hg.logilab.org/master/cubicweb/rev/9f1c89e7426d Closes #13521493

[rql2sql] Test and fix some case of sql generation when an unstable variable (ie whose type change from one solution to another) is used in an EXISTS node but belongs to an outer scope. Closes #13518646.

[test] Add missing os import in unittest_migractions.py Dropped in ^ merge.

backport 3.22 changes

[entity] Make it possible to specify sorting terms in cw_related_rqlst method Closes #12306543.

[sql gen] Properly create UNIQUE index on non string column. Closes #13728492 Unique was previously detected in a loop triggered only for String columns.

[server] some pep8 changes

pep8 schema2sql

[migration] Drop custom "readfunc" for code.interact in shell As the referenced bug http://bugs.python.org/issue1288615 got fixed in Python 2.6, this is no longer needed. Furthermore the implementation is not Python 3 compatible (raw_input, unicode).

[tox] Put development dependencies into a dev-requirements.txt file The latter being sourced in tox.ini. For now this contains references to hg repositories for some dependencies but under normal circumstances it should only contain pytest. This is useful when developping cubicweb itself without using tox but working with a virtualenv so that one can simply do: :: $ virtualenv --system-site-packages venv $ . venv/bin/activate $ pip install -r dev-requirements.txt $ python setup.py develop

[py3k] Fix config-level comparison in "cubicweb-ctl create" command dict.get -> None, and None cannot compare to an int in Python 3.

[pkg] PEP8 fixes in __pkginfo__.py

[pkg] Drop "support" of unused subpackage_of option in setup.py __pkginfo__ does not declare subpackage_of.

Add missing import, eaten by previous merge

Only access once to _cw.message in the login form else the first access may pop the message and the second won't get anything to display. This works on default message implementation but doesn't on the pyramid's one, leading to no message displayed on erroneous authentication attempt, which is not very user friendly.

backport 3.22 changes

backport changes from 3.21

[server,bfss] fix bfss migration test for python 3 (related #13519541)

[pkg] Use io.open with utf-8 encoding to read long description from README This forces decoding so that this is consistent between Python 2 and 3 as, in Python 3, decoding which would occur anyways and may fail depending on the environment locale settings.

[dataimport] order of ExtEntities should be irrelevant (closes #13117472)

Test and fix "pickleability" of Binary objects which has been recently broken while some cubes rely on this (eg fastimport). Do some licensing/pep8 cleanup along the way. Closes #13385274

[server,bfss] fix storage migration when Bytes attributes is None (closes #13519541) When a Bytes attributes is None before the BFSS storage migration, the migration crashes. A Bytes attribute can be None, when the attribute is not required and there is no default value. In this case, the attributes should remained None after migration and no file is stored on the disk.

Let hg Ignore py.test's .cache directory And drop rule for .svn along the way.

[entity] Add a cw_related_rqlst method returning the RQL select node of cw_related method This is useful if one wants to modify the query before execution. A bit a PEP8 style fix in tests along the way. Related to #12306543.

[devtools] set .config in setUpClass instead of using a cache in a classproperty This is much clearer and avoid problem with py.test introspection that triggered creation and caching of the configuration before actual test execution. This caused pb in unittest_postgres.PostgresStatementTimeoutTC where configuration was built before module's setup, hence source configuration was missing information about where to find the pg cluster.

[devtools] Simplify test configuration's init Specify module's __file__ instead of apphome. __file__ is saved on the class as it will be used later (eg to find the proper pg cluster without relying on shared dictionnary modifications). This will require some tests update, but it should be less common in cubes than in cubicweb itself.

[web] Have the LoginController return 403 on failed auth (closes #11569950)

[devtools] Drop old init_test_database backward compat

[devtools] show DeprecationWarning during test execution They are not shown by default since python 2.7

[devtools] deprecate usage of sources file for test This is old support which clutters the code and should no more be used. As this is rarely used (if it is) and test only, it feels ok to drop support at once without a deprecation phase.

[devtools] Set dumb value for postgres's db-host to avoid unexpected connection to a local server Currently, if something goes wrong (eg config built before call to startpgcluster) while a pg server is running on localhost, it may connect to this server and run the test in there unexepectedly.

[ldapfeed] ldap3 do not raise an exception in case of failure of cnx.bind() but return 'False' instead.

[test] add a test to ensure we get an AuthenticationError with ldapfeed when using (correct login, wrong password) credentials

[entity] Exclude computed relations from Entity.copy_relations Closes #12481591.

[pkg] Set version to 3.22.2.dev0 So that cubes used in test dependencies do not install a released CubicWeb.

[devtools] pep8 a bit package's __init__

[test/server] Only use configcls with statement timeout when needed Other tests in PostgresFTITC class apparently do not make use of this particular configuration.

[server/test] Add an assertion in test_statement_timeout

[datafeed] minor cleanups to cwxml parser

[datafeed] don't allow arbitrary kwargs on DatafeedParser initializer else they will be handled nicely by appobject base class, while we rather want an error in the case of a source parser (arguable).

[datafeed] gives information about moved entities to the parser Also, don't support anymore arbitrary parameters at parser initialization. Related to #5414754

[datafeed] simplify SQL query used to retrieve information about entities from an external source no need to join on cw_source

[datafeed] rename parser.sourceuris to source_uris Its more readable. Also, uniformize related method and local identifier names.

[datafeed] move process_urls to the parser

[tox] Use py.test to run tests In unittest_rqlrewrite.py and repotest.py, rename utility functions so that they are not considered as "test functions" by py.test.

[test] Fix devtools unittest_i18n when ran without pytest There is currently some weird error, requiring both to hack cubicweb.__path__ and to stop using load_module_from_name to load the registry. Introduce a dependency on to-be-released version of logilab common integrating fix for https://www.logilab.org/ticket/6085650 Do some extra import cleanups along the way.

[server] explicitly set postgres LC_MESSAGES variable to C on connection We parse some error messages so we need them to be in English. It's not clear to me exactly when that's not the case, but this fixes running unittest_postgres' "test_constraint_validationerror" with py.test for Sylvain.

[web] cleanup xmlrss view Don't trigger exceptions unnecessarily, and add line breaks to the output.

[migration] add rollback when replace_eid_sequence_with_eid_numrange fails If we get here, the current transaction is aborted, so if we are to assume everything is fine and continue we need to rollback first.

[migration] move CWComputedRType later The ordering in this file is a mess, and things don't actually work this way (when upgrading from 3.17).

docstring fix

[cwctl] make cwctl not depends on cubicweb.wsgi (closes #12210063) cubicweb.wsgi may not be available at all on Debian.

[hooks/workflow] pass eids, not entities, from hooks to operations Passing entity objects around runs the risk of getting out of date data from caches.

[web/httpcache] fix Last-Modified generation time.mktime takes a time tuple in *local* time. We have an UTC datetime coming from the last_modified method, which we were interpreting as local time, leading to hilarity. Use calendar.timegm instead. Add tests for Last-Modified generation of the 'download' and 'manage' views.

[forms] Fix edition of TZDatetime attributes Parse the submitted web form values into a timezone-aware datetime (by default, UTC) when such an attribute is edited. This is done using an automatically-selected (by `web.formfield.guess_field`) field named `TZDatetimeField`. Closes #12666598.

[doc] fix abstract and example of data import

[doc] fix abstract of data import

[repository] drop remanescence of old multi-sources code we don't use anymore this inputmap/varmap thing.

Added tag 3.20.15, debian/3.20.15-1, centos/3.20.15-1 for changeset 636a83e65870

[pkg] 3.20.15

[views] do not eat the self._cw.message in LogFormView (closes #12421177) accessing self._cw.message now(?) eat the message, so don't call "self._cw.message" twice.

[server] make InternalManager.name() return an empty string (closes #12418708) Otherwise the notification emails sent via an internal_session never honor the EMAIL.sender-name configuration option.

[test/migractions] work around drop_cube not deleting final etypes Closes #10096736

[hooks] deleting a final entity type shouldn't try dropping an sql table

[tox] Use a revision on proper branch for Yams "default" is not good when some bug fix release suddenly comes out...

[webconfig] disable generate-staticdir by default Its side effects mean it can be inconvenient, and using it requires manual (web server) config changes anyway, so there's no reason it needs to be enabled by default.

[test] Skip test_deserialization_base (unittest_repository.SchemaDeserialTC) See ticket #10096736.

[devtools] drop unused DumbOrderedDict2 class

[devtools] fix sorting in ExecutionPlan._check_permissions on python3 Sort lists instead of dict_items objects to get predictable results. With this change, unittest_querier.UtilsTC.test_preprocess_security passes every time instead of once in a blue moon. Closes #11759133.

[test] assertItemsEqual → assertCountEqual in unittest_repository.py

merge 3.20.14 in 3.21 branch

Added tag 3.20.14, debian/3.20.14-1, centos/3.20.14-1 for changeset f66a4895759e

[pkg] 3.20.14

[tox] use master URL for yams repository No need to clone from review. This should help get consistent results.

[pylint] Avoid invalid-name on schema relation class names

[pylint] more work on the pylint support * uniformize usage of ClassDef and simplify related import * don't insert in the ast a python function but an astroid representation of that function * ignore __pkginfo__ file * drop attr-rgx, variable-rgx and argument-rgx, put the name in good-names instead * drop additional-builtins * add cell_call to the list of methods for which we don't want 'abstract-method'

[pylint] remove abstract-method warning This kind of message is targeted: "Method 'entity_call' is abstract in class 'EntityView' but is not overridden (abstract-method)" For now, only "entity_call" and "render_body" are dealt with, the list can be extended.

[pylint] upgrade pylintext plugin

update and enhance pylintrc

[schema] use json to serialize constraints Require yams 0.43: constraints are serialized to json, which means we need to recreate the actual checks in the database on upgrade. Temporary deps in tox.ini to pull respective changes in yams.

[pkg] fix cubicweb.spec syntax (grafted from 3309ddb970594cd54529f5c205d1ea5b5da4131e)

[pkg] Use Twisted < 16.0.0 We have some test failures with 16.0.0, such as: - test_put_content (unittest_server.ETwistHTTPTC) - test_base_url (unittest_httptest.TwistedCWAnonTC) (grafted from 10c12f31d0fe38c2e8814176fc42bf2ea210efdf)

[pkg] fix cubicweb.spec syntax

[cwconfig] Use imp.load_source instead of exec() to load site_cubicweb.py from apphome Lets us return a module object and go back to the previous code in CubicWebConfiguration._load_site_cubicweb.

[cwconfig] Stop using lgc.modutils to load *config.py and *ctl.py files

[cwconfig] Stop using lgc.modutils to load site_cubicweb.py files config.load_site_cubicweb function now takes a list of cubes (instead of paths) and default to all known cubes.

[test] Use a Popen in i18n tests Instead of check_call, so that we clearly see the failure. A bit of flake8 along the way.

[forms] Fix HTableFormRenderer _render_fields method behavior when display_label is False. Indeed, now the extra <tr><th>...</th></tr> are not inserted anymore unless display_label is True. Closes #3811325.

[connection] remove a dead counter Related to #1381328.

[session] drop .is_repo_in_memory This is deader than a dead dog's bone buried down a blind alley off a dead-end street in a ghost town. Related to #1381328.

[session] drop session-as-a-context-manager It made little sense in the 3.19 - session-as-cnx-bw-compat era, maybe. Related to #1381328.

[devtools/repotest] use the RepoAccess object Quite better than playing tricks with the Repo._sessions. Related to #1381328.

[devtools] adapt stresstester to current APIs

[session] retire session.vreg Related to #1381328.

[session] fix super call in Connection.__init__ The vile hacks in test fixtures are fixed.

[session,repo] remove last trace of "cnxprops" Related to #1381328.

[pkg] Use Twisted < 16.0.0 We have some test failures with 16.0.0, such as: - test_put_content (unittest_server.ETwistHTTPTC) - test_base_url (unittest_httptest.TwistedCWAnonTC)

[web/views] stop using cwaction='cancel' It's broken as of https://hg.logilab.org/master/cubicweb/rev/cb217b2b3

[test] fix test_printable_value_bytes with current pygments An empty span was added in https://bitbucket.org/birkenfeld/pygments-main/commits/164574c13533195a555181a2b9c685fea2470403

[session, repository] deprecate repo.connect and move .close reponsibility to session object Repository.new_session just returns a plain session object, and `.connect` (which returns a sessionid) is deprecated. For .close:: session.close() # done ! The session will bear the responsibility to call the "session_close" event but that's better than the previous idiom:: repo.close(session.sessionid) which involves both objects. Related to #1381328. Related to #2919309.

[web/test] fix typo in doc string

[web/httpcache] send weak ETags There's no way we can provide a strong validator in general (especially considering how ETags are generated by the current EtagHTTPCacheManager and EntityHTTPCacheManager). So mark our entity-tags as weak.

[server] print backend exception if we fail to execute an sql script It's more useful to see could not open extension control file "/usr/share/postgresql/9.4/extension/postgis.control": No such file or directory rather than merely -> ERROR, skipping /usr/share/cubicweb/cubes/postgis/schema/_ext.postgres.sql

[cwctl] drop unnecessary admincnx call from 'shell' The connection is done by the migration helper.

[serverctl] allow schema-diff when db and fs are not in sync

[cwctl] remove networked remote shell This was a dbapi leftover. Related to #3933480.

[web/calendar] fix regression in previous changeset "Enables css customization in CalendarView" was untested, and introduced a crash: TypeError: not all arguments converted during string formatting

Enables css customization in CalendarView now a custom css class can be set on the calendar div (closes #3654043).

refactor a bit breadcrumbs to ease overriding of the root's display

Add migration helper function to delete entity types faster Bypasses all hooks and (for now) ignores any composite relations.

[testlib] Import nocoverage from logilab.common.testlib This depends on a change in logilab-common 1.2.0

[web/test] replace unicode with six.text_type Some uses of unicode reappeared when merging changes from 3.19.14.

[devtools/testlib] Issue a RuntimeError when a generative tests is detected Otherwise, generative tests *à la logilab-common* in CubicWebTC test cases would silently not run (at least using logilab.common.pytest runner).

[devtools] stop using lgc.testlib.TestCase It's become incompatible with unittest(2).TestCase due to generative tests. Just keep the datadir/datapath and assertCountEqual bits in our BaseTestCase.

[test] Replace remaining generative tests by subtests

[tox] Enable all applicable py34 environments All except etwist. Comes with a few tricks in tox.ini: * add ``-e.`` to force installing of cubicweb in development because dependency cubes would otherwise install cubicweb 3.22 (the latest release), which is not python3 installable. * "install" cubes/__init__.py because the file is not installed in develop mode; maybe this file should be managed by cubes themselves...

[migration/3.10] fix AttributeError 'Connection' object has no attribute 'HOOKS_DENY_ALL'

[server/sources] Fix authenticate on pre-3.10 database When migrating from a pre-3.10 database, the CWSource entity type and cw_source relation type are not known, but authenticate should still work to be able to add them.

[test] Add a test ensuring new cube built from skeleton is flake8-compliant And make skeleton files actually flake8-compliant henceforth.

[skeleton] Ship a tox.ini file upon new cube

[web] fix headers on 304 responses (closes #11466875) The response MUST include the following header fields: [...] - ETag and/or Content-Location, if the header would have been sent in a 200 response to the same request https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.5 Unfortunately we weren't copying headers from what would have been the 200 response, but from the request.

[web/httpcache] GMTOFFSET no longer exists

[web] fix headers on 304 responses (closes #11466875) The response MUST include the following header fields: [...] - ETag and/or Content-Location, if the header would have been sent in a 200 response to the same request https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.5 Unfortunately we weren't copying headers from what would have been the 200 response, but from the request.

[rset] fix infinite recursion introduced in c1eb5a676c80 The mechanism to avoid loops in c1eb5a676c80 "[rset] Always complete attribute/relation caches in ResultSet.get_entity" breaks down: we would loop forever if two entities that were already in the cache were linked by a relation with ? or 1 cardinality in both directions. To avoid that, keep a set of already-considered columns. Related to #9942503.

pep8 unittest_application and unittest_formfields

[rset] Always complete attribute/relation caches in ResultSet.get_entity RQL queries are often designed to fill up the ORM's caches when fetching entities out of the result set. Until now, if an entry already existed in the entity cache, ResultSet.get_entity would return it unchanged, not using the new ResultSet's contents to update the attribute cache, breaking expectations (if the attributes are needed, they'd then be fetched later one at a time, one entity at a time), resulting in loads of DB accesses. So we change ResultSet.get_entity so that: * if the entity is already cached and has been instantiated from the same rset, it is returned as-is (to avoid loops) * if the entity is not yet cached, it is instantiated * if the entity is cached via another rset, its attribute/relation caches are completed Closes #9942503

[rset] refactoring of _build_entity method define a method for entity instantiation and storage in cache (related to #9942503)

[entity] comparing two entities returns NotImplemented Fixes a test when using some versions of unittest2, where assertItemsEqual tries a sort and expects TypeError, which it doesn't get if we directly raise NotImplementedError. The NotImplemented value is treated specially by the interpreter: https://docs.python.org/3.4/library/constants.html#NotImplemented

[test] replace one use of lgc.testlib.unittest_main with unittest.main

[test] stop using TestCase.set_description It's a lgc-ism.

[test] convert a lgc-style generative test into unittest2-style

[web/application] fix query log handling on python3 Writing bytes to a file opened in text mode doesn't work.

[devtools/test] duplicate CubicWebServerTC tests for CubicWebWsgiTC

[devtools/test] fix bytes vs text in test_base_url Needed for python3 with the next changeset

[devtools] skip CubicWebServerTC on python3 People should migrate to wsgi if they're going to use CubicWeb on python3.

[web/test] switch unittest_web from twisted to wsgi And make it pass on python3 while at it.

[devtools] add CubicWebWsgiTC Similar to CubicWebServerTC, except using wsgiref instead of twisted as an http server.

[web/test] return the hash of uploaded files in FileUploadTC instead of their contents json must be unicode, which doesn't go well with arbitrary file contents. Compute the file hash instead, as we know the hexdigest is (ascii) text.

[web/test] open() as a context manager

[web/test] json.loads expects text, not bytes

[wsgi] fix handling of multiple posted files We were only keeping the last received file.

[web/test] Reword doc string for fileupload ajaxfunc

[skeleton] do not import the __pkginfo__.py module to make cubes installable via setuptools When using a simple "import" statement, running 'python setup.py install' for a cube fails in a fresh virtualenv due to the fact the install of the dependencies (here cubicweb iteself) is done by the python process doing the cube's installation, so the __pkginfo__ module from the cube is already loaded (thus used) when installing cubicweb... So we apply the third technique from: https://packaging.python.org/en/latest/single_source_version.html

merge changes from 3.22.2

Added tag 3.22.2, debian/3.22.2-1, centos/3.22.2-1 for changeset 1b93ff37755b

[pkg] 3.22.2

[pkg] fix centos install

[datafeed] do NOT call commit in DataFeedXMLParser.process() (closes #10928592) and so, remove more useless code. Doing a commit in DataFeedXMLParser.process() is a bad idea because this method is called for every imported/updated entity, before handling deletions. So if we commit here, odds are quite high that we're violating some integrity constraint and end up with a ValidationError.

[schema] restore constraint checking when running on old sqlite Old sqlite3 doesn't provide CHECK constraint names in error messages, preventing us from translating a backend integrity error into a ValidationError. This was added in 2012, but the sqlite3 version in RHEL6 is older; so if we run on old sqlite, keep checking the constraints in python rather than only in SQL. Closes #10927494

[web/autoform] fix regression introduced by #5758ba784ebd The changes from cset 5758ba784ebd "[autoform] when building inlined-form view for a relation, consider related entities intead of other views" lead to an unexpected 'add new' link on creation of an entity with inlined form for a relation of cardinality 1. To avoid this, we should consider formviews and not existing relations when the entity is being created. Closes #10860258

[tox] Generate test environments for Python 2.7 and 3.4 Test commands for each environment are written down explicitly since I could not find a way to extract the "package" name (e.g. "hooks") from the environment name (e.g. "py34-hooks"). For Python 3.4 interpreter, only environments (subpackages) that do not depend on cubes for their tests are listed since those test dependency cubes are not yet installable with Python 3.x. etwist is also not included since the Python 3 port is not complete at the moment. From local testing, py34 tests pass for the following subpackages: - dataimport - entities - ext - hooks - wsgi

[notification] avoid using InternalManager when we've got a CWUser This will avoid potential information leak due to missed security restriction and is more compatible with legacy code (pb encountered in some client code). Closes #10294761

[dataimport] rename .time to .now attribute of meta generator

[pkg] Set version to 3.22.2.dev0 So that cubes used in test dependencies do not install a released CubicWeb.

[dataimport] cleanup a bit nohook store, mostly protecting some attributes

[dataimport] add a filter function to not fail if some extentity has several values for an attribute of final relation the function will simply record a warning in the import log and keep a value randomly. Notice the function has to be explicitly inserted in the transformation stream.

[sources] synchronize source asynchronously when started from the UI and redirect to the forthcoming import log, for a better user experience (there is still a lot more to do there though). Closes #10468967

[test] fix unittest_devctl on python3.4

[test] Make sure that newcube command test produces expected files

[server] remove some pre 3.10 compat Let's assume nobody will try upgrading an instance from pre-3.10 cubicweb to 3.23+. If they do, they're setting themselves up for a lot of pain anyway.