[test] Avoid usage of lgc.testlib TestCase in unittest_cwconfig.py

[test] Move setup of spa2rql tests into setUpClass method Thus avoiding cryptic errors upon unittest discovery.

[schema] Set CubicWebSchemaLoader's extrapath using config eponymous property

[doc] Document change of instances location in virtualenv installation Closes #14789440.

[cwconfig] Do not override user-defined CW_MODE in virtualenv setup And thus consider `_forced_mode` first. Related to #14789440.

Added tag 3.22.4, debian/3.22.4-1, centos/3.22.4-1 for changeset 92db0bb8e26d

[pkg] remove .dev0 from version

[pkg] 3.22.4

[session] Ensure access to rql varmaker always mark the session as dirty When one accesses the rql_varmaker, that's usually to use it. The pb was that when the varmaker was already in page's data (which is stored as session data), session storage such as redis won't see that the session data is dirty and has to be stored back at the end of the request. To fix this, systematically call set_page_data. (grafted from 3432f0e2540d)

[test] fix test_printable_value_bytes with current pygments An empty span was added in https://bitbucket.org/birkenfeld/pygments-main/commits/164574c13533195a555181a2b9c685fea2470403 (grafted from 74b04a88d28a)

[webctl] set uid of file produced by gen-static-datadir (closes #11298794)

[webconfig] ensure uicache content has correct uid (related to #11298794)

[cwconfig] add the CubicWebConfiguration.ensure_uid_directory() method (related to #11298794) that recursively enforce uid on files.

[cwconfig] extract uid file setting code from check_writeable_uid_directory (related to #11298794) into an ensure_uid() method

[web] Fix bug with usage of os.rename under windows environment In 7c386161ebd6 we removed cache handling from property sheet and introduced usage of a tempfile + os.rename to get atomic generation of files. The pb is that this is not portable, since under windows os.rename will raise an exception if the file already exists (because there is no way to write a file atomatically in such case). This kind of thing should be out of the CW scope anyway, so implements a quick & dirty fix in the mean time. Closes #14214794

[migration] More resilient migration script Query the information schema to retrieve existing constraint (primary key and unique) and drop them. Thus don't use anymore DROP CONSTRAINT IF EXISTS (not implemented by e.g. old versions of sqlserver). To do so a missing DISTINCT was necessary to avoid duplicates.

[session] Ensure access to rql varmaker always mark the session as dirty When one accesses the rql_varmaker, that's usually to use it. The pb was that when the varmaker was already in page's data (which is stored as session data), session storage such as redis won't see that the session data is dirty and has to be stored back at the end of the request. To fix this, systematically call set_page_data.

typo

[native] Use cnx.system_sql instead of doexec method in syntax_tree_search The latter handles connection issues and tries to reconnect. Method syntax_tree_search appears to be called from quite a few place where it seems useful to be resilient to connection issues (authentication for instance). This reconnection logic appears to have disappeared at some point when working on https://www.cubicweb.org/2919309. Could not find the exact reason though...

Remove usage of cnx.ensure_cnx_set context manager It has been deprecated since 3.21, no reason to keep using it internally.

[hooks/test/synschema] remove an unneeded skipTest. Related to #5557633.

[test] Check retcode of subprocesses in devctl tests And display decoded stdout in case of failure.

merge 3.23

[pkg] Bump version to 3.24.0.dev0

[pkg] Drop NO_SETUPTOOLS option setuptools is the defacto standard nowadays. The only place where NO_SETUPTOOLS option was used is during packaging (RPM, Debian) and even in this context, it seems that setuptools is the way to go (see https://wiki.debian.org/Python/LibraryStyleGuide).

[debian] update cubicweb-dev.lintian-overrides cubicweb/skeleton/debian/rules.tmpl is now cubicweb/skeleton/debian/rules prevent lintian from complaining

[pkg] Properly export data files in setup.py and adjust "newcube" test With the new package layout (everything under "cubicweb" package), the custom install_lib rule which makes use of include_dirs defined in __pkginfo__.py did not prepend the package name to source directories to be copied. Fixing this. Also, in setup.py's export() function, the destination directories' path to be created during source tree walk was wrong. All this makes cubicweb/skeleton directory (which is not a package) properly installed by setup.py. The test in cubicweb/devtools/test/unittest_devctl.py wasn't properly implemented because it used an installation of cubicweb in "develop" mode which shadows such packaging issues. Also it used "python -m cubicweb" instead of directly "cubicweb-ctl" and the former appears to fall back to using the cubicweb package *from sources* instead of the installed one. Now that this test runs against the installed version of cubicweb, fix MANIFEST.in to include tox.ini files (cubicweb's and skeleton's) as this is expected from the test. Closes #14127941.

[pkg] Handle verbose option in setup.py's export function Make debugging easier.

[pkg] Drop unused EMPTY_FILE global variable

[tox] Do not install package in "develop" This might shadow so packaging issues. And user can still pass --develop to tox.

[pkg] Bump version to 3.23.2

pep8 bits

[schema] Add a method on yams constraints to compute its unique name used to identify it in the DB backend.

[migration] Test and fix case of addition of an attribute with some serialized constraint It was crashing on attempt to remove an unexisting constraint. This code may be removed safely as it's expected to be handled by the constraint removal hook. Notice that adding a vocabulary on shortpara makes yams change the varchar max size. This is arguable but not the point of this test, so simply adapt the max size defined in the schema coherently so we don't have to bother with that at this point.

[test] Add a migration test for update of a static vocabulary constraint This is a constraint serialized to be checked by the DB backend, hence ensure the old constraint is dropped and the new one is created.

[schema2sql] Avoid "parsing" SQL statements for database initialization A big SQL string was generated, then splitted. This caused bug if some value in the schema (eg vocabulary, default) contained the separator (';'). To properly fix this, yield each individual statement instead of generating a string. Closes #14050899

[schema2sql] Drop most of the DB DROP related code This code is dead for a while, it has been superseded by `sql_drop_all_user_tables`. Related to #14050899

[hooks] Use already imported module alias

[schema2sql] Rename as_sql to constraint_value_as_sql which is easier to grasp independantly of the context.

[schema2sql] Drop unused indent argument on aschema2sql Related to #14050899

[schema2sql] More docstrings and minor API improvements Along the way, renamed local `column` variable to `attr` where we're NOT manipulating a DB column name but a schema attribute. Related to #14050899

[schema2sql] inline eschema_attrs method * not a public API * doesn't makes things easier to grasp -> useless Related to #14050899

[schema2sql] Give a rdef to check_constraint This simplifies the API. Related to #14050899

merge 3.20.16 in 3.22

merge 3.20.16 in 3.21

Added tag 3.20.16, debian/3.20.16-1, centos/3.20.16-1 for changeset e60a8e5d29ef

[debian] add a lintian override for source-is-missing about jquery.flot.js

[pkg] 3.20.16

[dataimport] fix count_lines function (closes #14407650) ensure i allways exists whatever f content is

[captcha] improve a bit the redability of generated captcha images (closes #13500378)

3.20 branch

merge 3.23 changes

Added tag 3.23.1, centos/3.23.1-1, debian/3.23.1-1 for changeset 8f035d5d5953

prepare 3.23.1

[debian] Adjust symlink to cubicweb package so that documention builds fine Broken since source tree re-organization.

[dataimport] Restore handling of Binary in ExtEntity's values Broken since 62a7100d774b. Closes #13973741.

[tox] Ensure virtualenv's python is used to build the documentation

[doc] Reference 3.23 page in changelog

[tox] Ensure virtualenv's python is used to build the documentation

Backed out changeset b288debc6736 Only superusers can set lc_messages variable. This prevents connection in many cases. Closes #14010345.

Merge with 3.23

[pkg] Bump version to 3.22.4.dev0

Added tag 3.23.0, centos/3.23.0-1, debian/3.23.0-1 for changeset 4b07f2a79021

[doc] Fix symlinks of sphinx theme images

Added tag 3.23.0, centos/3.23.0-1, debian/3.23.0-1 for changeset 93b0f836cb07

[debian] Move python-unittest2 >= 0.7.0 as runtime dependency It seems to me that this is not used during Debian package build (tests are not run), but should be needed at runtime with the cubicweb-dev package. Also, at the moment, we have no pbuilder setting with this package available and it blocks the release.

Symlink __pkginfo__.py at root Maybe this will make apycot work again...

[3.23] More release notes

i18n update

[doc/changes] CubicWebTC.set_description no longer available

Add some release notes

[pkg] Version 3.23.0

[pkg] Bump yams version in cubicweb-common Debian package

[tox] Drop yams from dev-requirements.txt Yams 0.44.0 got released.

Merge with 3.22

Merge with 3.21

backport 3.21 changes

backport 3.20 changes

[sobjects] Drop custom unique constraint checks in user registration service This constraints (on unique CWUser's login and EmailAddress's login) are handled by the backend. This follows "[hooks/integrity] kill CheckUniqueHook". Extend test for the ValidationError along the way.

[test] Add a test for user registration unicity constraint on email

[test] Clarify user registration service tests * split permissions test appart * avoid usage of internal_cnx as this does not seem need

[test/hooks] pep8 and unittest.main instead in unittest_hooks.py

[hooks/integrity] kill CheckUniqueHook It's the backend's job. The noticeable consequence (as can be seen in tests updated here) is that error message are a bit less user-friendly.

[migration] drop identity_relation if it exists It has been generated erroneously for a while, then fixed but not properly deleted from old databases. Don't use IF EXISTS for sqlserver compat.

[migration] Reserialize all constraints since yams introduce new JSON serializing format So we may drop support for older format in yams at some point.

[migration] Add IF EXISTS on DROP CONSTRAINT problem encountered on migration of a legacy database

[schema sync] Rename index when an entity type is renamed So we may still find them later using their expected name, and avoid collisions. Closes #13822045

[c-c] Add a command to check database index List indices expected by the schema but not in the database and vice-versa Closes #13822027

[sql gen] Control size of index name using an md5 hash When using concatenation of table and column, we may hit the backend size limit on index names, which results on truncated name (64 for postgres). Then we may have conflict or be unable to find expected index on later synchronization attempt. This is done by monkeypatching lgdb for now to avoid dependancy problems. Unique and primary key indices of entity types handling is left to the backend, as they are not explicitly named. See https://www.logilab.org/ticket/4789189 Closes #13821949

[sql gen] Explicitly name unique index This patch hotfix logilab database for https://www.logilab.org/6662663 and fix database creation to stop using UNIQUE in the table schema but rather create the index later on, so its name is under control. Also consider this change in schema synchronization hooks. This is preliminary work related to #13822045

Update to yams 0.44 API

[test] Fix randomness in tests

[test] Drop unused import

[serverctl] allow to sync multiple and all sources in source-sync command Breaking change: by default source are not synced anymore if it is fresh (regards to synchronization-interval). To get the previous behavior an option --force is added. Closes #13886467

[tox] Split migractions tests from server environment These are the longest tests in this environment, so running them apart (in parallel of others) would hopefully speed up the whole suite.

[dataimport] Fix rdef loopup in importer

[py3k] left-over unicode() in authobjs.py

[server] Improve massive deletion of entities sqlserver is limited on the "IN" array size and produce an error "The query processor ran out of internal resources and could not produce a query plan". The exact limitation is unclear but can occur with > 10000 item in the array. See https://support.microsoft.com/en-us/kb/288095 Now we process the deletion by batch of 10000, some quick benchmarks show that there is no performance impact: delete_in(50000): 46.765182972 delete_in_batch(50000): 46.3242430687 delete_in(100000): 62.2969429493 delete_in_batch(100000): 61.8402349949

[hooks] Add a bunch of tests for session synchronization And change unicode into six.text_type in authobjs.py spotted by new tests.

[entities] Drop very old backward compat from erudi time (even the standard datetime type didn't exist at that time)

pep8 + docstrings and comments improvments

[server] some pep8 in rql2sql

[dataimport] Deprecate SQLGenObjectStore Closes #10338709.

[datafeed] attempt to acquire synchronization lock even when force is given instead of the implementation in e717da3dc164, raise an error if the lock is already grabbed and catch this error in the caller. See discussion on https://www.cubicweb.org/revision/10790765 Closes #10451635

[schema] Cache RQL expressions on schema loading Add a mapping rql string to rql statement Deserializing schema is ~36% faster with a lot (> 4000) expressions like those generated by cubicweb-container.

[schema] Avoid parsing RQL expressions twice on schema loading Kill full_rql (unnecessary) Use snippet_rql rather than rqlst when possible Don't access to rqlst cachedproperty from __init__ Deserializing schema is ~40% faster with a lot (> 4000) expressions like those generated by cubicweb-container.

[dataimport] Handle Bytes data in ExtEntity and convert them to Binary Closes #13887276.

[test] Avoid assertCountEqual in unittest_syncschema.py It's not clear why this happens, but it seems that assertCountEqual does strange things with unittest2 on Python 2 for equality comparison (trying to sort constraint objects, even though BaseConstraint raise NotImplemented in __lt__ from Yams 0.43). So just avoid using it to get this test passing.

[packaging] Bump version number

[web/session] make sure not to modify an inner value of session.data (closes #11217958) When we modify a mutable value stored in the session's data (dict-like object) we prevent an alternative session data handler (eg. pyramid-redis-session) from detecting this update and synchronizing it in its backend (like redis).

[book] improve a bit the documentation of the database setup also remove the MySQL section.

[doc] add documentation for specializing translation of other cubes in an application cube

[doc] fix some URLs (cs to test jenkins autobuild )

[uicfg] Fix typo in set_fields_order

[forms] We don't want cwjs-edition-cancel class on reledit's cancel button Else it's bound to a js function that collides with onclick provided by reledit code. Closes #13120795

[test] Prefer unittest.main

[test] py.test related fix We ought not import a concrete test case from a sibling module, else it ends up partially initialized at some point and later raise assertion error in setUpClass.

[dataimport] add a bit of extra-documentation on the massive store about prepare_insert_relation limitation on inlined relation and lack of full-text indexation.

[dataimport] no need to call .keys()

[dataimport] drop massive store's flush_metadata method This method handle a temporary table which sounds useless, and even buggy in some cases (metadata for a single entity types will be only flushed once even if flush is called several time). Instead, simply call the method-doing-the job after entities have been inserted (and make it private along the way).

[dataimport] test and fix external source support for the massive store

[dataimport] add new test for nohook store importing in a custom source

[dataimport] introduce usage of MetadataGenerator into the massive store This allows more flexibility and code sharing w/ nohook store. Closes #10338621

[dataimport] rename massive store's metagen_push_relation method to _insert_meta_relation.

[dataimport] meta_data -> metadata

[dataimport] use sql shortcut

[dataimport] move cwmassive_constraint temporary table handling to the PGHelper class with consistent renaming and some rationalization to avoid doing things several times and have code slightly easier to grasp.

[dataimport] inline some methods of the massive store

[dataimport] move everything related to the "URI handling" feature of the massive store together

[dataimport] test for a value is in a set and insertion in a set should live together also inline some methods on the way, in the hope to make the public api clearer at some point, and only put valuable values in those sets so we don't have to endlessly test for consistency.

[dataimport] use a single _initialized set instead of _entities / _rtypes and rework a bit methods using them so that 'in' test and set insertion leave in the same method.

[dataimport] drop massive store etype to eid index and fix potential problem with is_instance_of relation insertion in case where the class of inserted entity has some (yams) parent.

[dataimport] add a .schema shortcut attribute on the massive store also rephrase a bit a method along the way

[dataimport] rework massive store's __init__ * _cnx is set by the parent class * count_cwuri is not used * group related things together

[dataimport] rework PGHelper class

[dataimport] flake8

[dataimport] enhance a bit sql queries readability

[dataimport] use IF EXISTS when possible it's better than try/except and avoid potential concurrency problems.

[dataimport] PGHelper should be responsible to retrieve the database schema

[dataimport] introduce a new MetadataGenerator to replace former MetaGenerator with a similar-but-different API that will allow its usage by the massive store (no more entity object manipulation), while making it a bit more flexible for customization by subclasses (experience taken from client code). Related to #10338621

[dataimport] Prefer now(pytz.utc) to utcnow

[dataimport] rename nohook store .source attribute so it's clearly not the samething as metagenerator.source. Also add a bit of doc along the way.

[dataimport] make rql store test easier to read and test a few more thing, then reuse it to test nohook start which had no tests at all

[dataimport] remove useless assignment in massive store

[dataimport] move eid attribution from the metagenerator back to the nohook store

[dataimport] rephrase minor bits of the meta generator

[pkg] Depends on logilab-common >= 1.2.2

Merge with 3.22.3

[pkg] Depends on yams >= 0.43 RPM spec file and debian/control got updated previously.

Added tag 3.22.3, debian/3.22.3-1, centos/3.22.3-1 for changeset b1e7de000536

[pkg] Version 3.22.3

[pkg] Depends on yams < 0.43

Define BASE_URL and DATA_URL js variables in the login template. Closes #13385185 Those are expected to be defined in almost every js files. They were not before this patch because login form doesn't use the regular main template.

[web] Remove some unused imports in editforms

Fix cancel action on deletion confirmation form Currently crashed with RequestError: invalid action '__action_cancel'. It's broken as of https://hg.logilab.org/master/cubicweb/rev/cb217b2b3 This fix is similar as the one done in https://hg.logilab.org/master/cubicweb/rev/9f1c89e7426d Closes #13521493

[rql2sql] Test and fix some case of sql generation when an unstable variable (ie whose type change from one solution to another) is used in an EXISTS node but belongs to an outer scope. Closes #13518646.

[test] Add missing os import in unittest_migractions.py Dropped in ^ merge.

backport 3.22 changes

[entity] Make it possible to specify sorting terms in cw_related_rqlst method Closes #12306543.

[sql gen] Properly create UNIQUE index on non string column. Closes #13728492 Unique was previously detected in a loop triggered only for String columns.

[server] some pep8 changes

pep8 schema2sql

[migration] Drop custom "readfunc" for code.interact in shell As the referenced bug http://bugs.python.org/issue1288615 got fixed in Python 2.6, this is no longer needed. Furthermore the implementation is not Python 3 compatible (raw_input, unicode).

[tox] Put development dependencies into a dev-requirements.txt file The latter being sourced in tox.ini. For now this contains references to hg repositories for some dependencies but under normal circumstances it should only contain pytest. This is useful when developping cubicweb itself without using tox but working with a virtualenv so that one can simply do: :: $ virtualenv --system-site-packages venv $ . venv/bin/activate $ pip install -r dev-requirements.txt $ python setup.py develop

[py3k] Fix config-level comparison in "cubicweb-ctl create" command dict.get -> None, and None cannot compare to an int in Python 3.

[pkg] PEP8 fixes in __pkginfo__.py

[pkg] Drop "support" of unused subpackage_of option in setup.py __pkginfo__ does not declare subpackage_of.

Add missing import, eaten by previous merge

Only access once to _cw.message in the login form else the first access may pop the message and the second won't get anything to display. This works on default message implementation but doesn't on the pyramid's one, leading to no message displayed on erroneous authentication attempt, which is not very user friendly.

backport 3.22 changes

backport changes from 3.21

[server,bfss] fix bfss migration test for python 3 (related #13519541)

[pkg] Use io.open with utf-8 encoding to read long description from README This forces decoding so that this is consistent between Python 2 and 3 as, in Python 3, decoding which would occur anyways and may fail depending on the environment locale settings.

[dataimport] order of ExtEntities should be irrelevant (closes #13117472)

Test and fix "pickleability" of Binary objects which has been recently broken while some cubes rely on this (eg fastimport). Do some licensing/pep8 cleanup along the way. Closes #13385274

[server,bfss] fix storage migration when Bytes attributes is None (closes #13519541) When a Bytes attributes is None before the BFSS storage migration, the migration crashes. A Bytes attribute can be None, when the attribute is not required and there is no default value. In this case, the attributes should remained None after migration and no file is stored on the disk.

Let hg Ignore py.test's .cache directory And drop rule for .svn along the way.

[entity] Add a cw_related_rqlst method returning the RQL select node of cw_related method This is useful if one wants to modify the query before execution. A bit a PEP8 style fix in tests along the way. Related to #12306543.

[devtools] set .config in setUpClass instead of using a cache in a classproperty This is much clearer and avoid problem with py.test introspection that triggered creation and caching of the configuration before actual test execution. This caused pb in unittest_postgres.PostgresStatementTimeoutTC where configuration was built before module's setup, hence source configuration was missing information about where to find the pg cluster.

[devtools] Simplify test configuration's init Specify module's __file__ instead of apphome. __file__ is saved on the class as it will be used later (eg to find the proper pg cluster without relying on shared dictionnary modifications). This will require some tests update, but it should be less common in cubes than in cubicweb itself.

[web] Have the LoginController return 403 on failed auth (closes #11569950)

[devtools] Drop old init_test_database backward compat

[devtools] show DeprecationWarning during test execution They are not shown by default since python 2.7

[devtools] deprecate usage of sources file for test This is old support which clutters the code and should no more be used. As this is rarely used (if it is) and test only, it feels ok to drop support at once without a deprecation phase.

[devtools] Set dumb value for postgres's db-host to avoid unexpected connection to a local server Currently, if something goes wrong (eg config built before call to startpgcluster) while a pg server is running on localhost, it may connect to this server and run the test in there unexepectedly.

[ldapfeed] ldap3 do not raise an exception in case of failure of cnx.bind() but return 'False' instead.

[test] add a test to ensure we get an AuthenticationError with ldapfeed when using (correct login, wrong password) credentials

[entity] Exclude computed relations from Entity.copy_relations Closes #12481591.

[pkg] Set version to 3.22.2.dev0 So that cubes used in test dependencies do not install a released CubicWeb.

[devtools] pep8 a bit package's __init__

[test/server] Only use configcls with statement timeout when needed Other tests in PostgresFTITC class apparently do not make use of this particular configuration.

[server/test] Add an assertion in test_statement_timeout

[datafeed] minor cleanups to cwxml parser

[datafeed] don't allow arbitrary kwargs on DatafeedParser initializer else they will be handled nicely by appobject base class, while we rather want an error in the case of a source parser (arguable).

[datafeed] gives information about moved entities to the parser Also, don't support anymore arbitrary parameters at parser initialization. Related to #5414754

[datafeed] simplify SQL query used to retrieve information about entities from an external source no need to join on cw_source

[datafeed] rename parser.sourceuris to source_uris Its more readable. Also, uniformize related method and local identifier names.

[datafeed] move process_urls to the parser

[tox] Use py.test to run tests In unittest_rqlrewrite.py and repotest.py, rename utility functions so that they are not considered as "test functions" by py.test.

[test] Fix devtools unittest_i18n when ran without pytest There is currently some weird error, requiring both to hack cubicweb.__path__ and to stop using load_module_from_name to load the registry. Introduce a dependency on to-be-released version of logilab common integrating fix for https://www.logilab.org/ticket/6085650 Do some extra import cleanups along the way.

[server] explicitly set postgres LC_MESSAGES variable to C on connection We parse some error messages so we need them to be in English. It's not clear to me exactly when that's not the case, but this fixes running unittest_postgres' "test_constraint_validationerror" with py.test for Sylvain.

[web] cleanup xmlrss view Don't trigger exceptions unnecessarily, and add line breaks to the output.

[migration] add rollback when replace_eid_sequence_with_eid_numrange fails If we get here, the current transaction is aborted, so if we are to assume everything is fine and continue we need to rollback first.

[migration] move CWComputedRType later The ordering in this file is a mess, and things don't actually work this way (when upgrading from 3.17).

docstring fix

[cwctl] make cwctl not depends on cubicweb.wsgi (closes #12210063) cubicweb.wsgi may not be available at all on Debian.

[hooks/workflow] pass eids, not entities, from hooks to operations Passing entity objects around runs the risk of getting out of date data from caches.

[web/httpcache] fix Last-Modified generation time.mktime takes a time tuple in *local* time. We have an UTC datetime coming from the last_modified method, which we were interpreting as local time, leading to hilarity. Use calendar.timegm instead. Add tests for Last-Modified generation of the 'download' and 'manage' views.

[forms] Fix edition of TZDatetime attributes Parse the submitted web form values into a timezone-aware datetime (by default, UTC) when such an attribute is edited. This is done using an automatically-selected (by `web.formfield.guess_field`) field named `TZDatetimeField`. Closes #12666598.

[doc] fix abstract and example of data import

[doc] fix abstract of data import

[repository] drop remanescence of old multi-sources code we don't use anymore this inputmap/varmap thing.

Added tag 3.20.15, debian/3.20.15-1, centos/3.20.15-1 for changeset 636a83e65870

[pkg] 3.20.15

[views] do not eat the self._cw.message in LogFormView (closes #12421177) accessing self._cw.message now(?) eat the message, so don't call "self._cw.message" twice.

[server] make InternalManager.name() return an empty string (closes #12418708) Otherwise the notification emails sent via an internal_session never honor the EMAIL.sender-name configuration option.

[test/migractions] work around drop_cube not deleting final etypes Closes #10096736

[hooks] deleting a final entity type shouldn't try dropping an sql table

[tox] Use a revision on proper branch for Yams "default" is not good when some bug fix release suddenly comes out...

[webconfig] disable generate-staticdir by default Its side effects mean it can be inconvenient, and using it requires manual (web server) config changes anyway, so there's no reason it needs to be enabled by default.

[test] Skip test_deserialization_base (unittest_repository.SchemaDeserialTC) See ticket #10096736.

[devtools] drop unused DumbOrderedDict2 class

[devtools] fix sorting in ExecutionPlan._check_permissions on python3 Sort lists instead of dict_items objects to get predictable results. With this change, unittest_querier.UtilsTC.test_preprocess_security passes every time instead of once in a blue moon. Closes #11759133.

[test] assertItemsEqual → assertCountEqual in unittest_repository.py

merge 3.20.14 in 3.21 branch

Added tag 3.20.14, debian/3.20.14-1, centos/3.20.14-1 for changeset f66a4895759e

[pkg] 3.20.14

[tox] use master URL for yams repository No need to clone from review. This should help get consistent results.

[pylint] Avoid invalid-name on schema relation class names

[pylint] more work on the pylint support * uniformize usage of ClassDef and simplify related import * don't insert in the ast a python function but an astroid representation of that function * ignore __pkginfo__ file * drop attr-rgx, variable-rgx and argument-rgx, put the name in good-names instead * drop additional-builtins * add cell_call to the list of methods for which we don't want 'abstract-method'

[pylint] remove abstract-method warning This kind of message is targeted: "Method 'entity_call' is abstract in class 'EntityView' but is not overridden (abstract-method)" For now, only "entity_call" and "render_body" are dealt with, the list can be extended.

[pylint] upgrade pylintext plugin

update and enhance pylintrc

[schema] use json to serialize constraints Require yams 0.43: constraints are serialized to json, which means we need to recreate the actual checks in the database on upgrade. Temporary deps in tox.ini to pull respective changes in yams.

[pkg] fix cubicweb.spec syntax (grafted from 3309ddb970594cd54529f5c205d1ea5b5da4131e)

[pkg] Use Twisted < 16.0.0 We have some test failures with 16.0.0, such as: - test_put_content (unittest_server.ETwistHTTPTC) - test_base_url (unittest_httptest.TwistedCWAnonTC) (grafted from 10c12f31d0fe38c2e8814176fc42bf2ea210efdf)

[pkg] fix cubicweb.spec syntax

[cwconfig] Use imp.load_source instead of exec() to load site_cubicweb.py from apphome Lets us return a module object and go back to the previous code in CubicWebConfiguration._load_site_cubicweb.

[cwconfig] Stop using lgc.modutils to load *config.py and *ctl.py files

[cwconfig] Stop using lgc.modutils to load site_cubicweb.py files config.load_site_cubicweb function now takes a list of cubes (instead of paths) and default to all known cubes.

[test] Use a Popen in i18n tests Instead of check_call, so that we clearly see the failure. A bit of flake8 along the way.

[forms] Fix HTableFormRenderer _render_fields method behavior when display_label is False. Indeed, now the extra <tr><th>...</th></tr> are not inserted anymore unless display_label is True. Closes #3811325.

[connection] remove a dead counter Related to #1381328.

[session] drop .is_repo_in_memory This is deader than a dead dog's bone buried down a blind alley off a dead-end street in a ghost town. Related to #1381328.

[session] drop session-as-a-context-manager It made little sense in the 3.19 - session-as-cnx-bw-compat era, maybe. Related to #1381328.

[devtools/repotest] use the RepoAccess object Quite better than playing tricks with the Repo._sessions. Related to #1381328.

[devtools] adapt stresstester to current APIs

[session] retire session.vreg Related to #1381328.

[session] fix super call in Connection.__init__ The vile hacks in test fixtures are fixed.

[session,repo] remove last trace of "cnxprops" Related to #1381328.

[pkg] Use Twisted < 16.0.0 We have some test failures with 16.0.0, such as: - test_put_content (unittest_server.ETwistHTTPTC) - test_base_url (unittest_httptest.TwistedCWAnonTC)

[web/views] stop using cwaction='cancel' It's broken as of https://hg.logilab.org/master/cubicweb/rev/cb217b2b3

[test] fix test_printable_value_bytes with current pygments An empty span was added in https://bitbucket.org/birkenfeld/pygments-main/commits/164574c13533195a555181a2b9c685fea2470403

[session, repository] deprecate repo.connect and move .close reponsibility to session object Repository.new_session just returns a plain session object, and `.connect` (which returns a sessionid) is deprecated. For .close:: session.close() # done ! The session will bear the responsibility to call the "session_close" event but that's better than the previous idiom:: repo.close(session.sessionid) which involves both objects. Related to #1381328. Related to #2919309.

[web/test] fix typo in doc string

[web/httpcache] send weak ETags There's no way we can provide a strong validator in general (especially considering how ETags are generated by the current EtagHTTPCacheManager and EntityHTTPCacheManager). So mark our entity-tags as weak.

[server] print backend exception if we fail to execute an sql script It's more useful to see could not open extension control file "/usr/share/postgresql/9.4/extension/postgis.control": No such file or directory rather than merely -> ERROR, skipping /usr/share/cubicweb/cubes/postgis/schema/_ext.postgres.sql

[cwctl] drop unnecessary admincnx call from 'shell' The connection is done by the migration helper.

[serverctl] allow schema-diff when db and fs are not in sync

[cwctl] remove networked remote shell This was a dbapi leftover. Related to #3933480.

[web/calendar] fix regression in previous changeset "Enables css customization in CalendarView" was untested, and introduced a crash: TypeError: not all arguments converted during string formatting

Enables css customization in CalendarView now a custom css class can be set on the calendar div (closes #3654043).

refactor a bit breadcrumbs to ease overriding of the root's display

Add migration helper function to delete entity types faster Bypasses all hooks and (for now) ignores any composite relations.

[testlib] Import nocoverage from logilab.common.testlib This depends on a change in logilab-common 1.2.0

[web/test] replace unicode with six.text_type Some uses of unicode reappeared when merging changes from 3.19.14.

[devtools/testlib] Issue a RuntimeError when a generative tests is detected Otherwise, generative tests *à la logilab-common* in CubicWebTC test cases would silently not run (at least using logilab.common.pytest runner).

[devtools] stop using lgc.testlib.TestCase It's become incompatible with unittest(2).TestCase due to generative tests. Just keep the datadir/datapath and assertCountEqual bits in our BaseTestCase.

[test] Replace remaining generative tests by subtests

[tox] Enable all applicable py34 environments All except etwist. Comes with a few tricks in tox.ini: * add ``-e.`` to force installing of cubicweb in development because dependency cubes would otherwise install cubicweb 3.22 (the latest release), which is not python3 installable. * "install" cubes/__init__.py because the file is not installed in develop mode; maybe this file should be managed by cubes themselves...

[migration/3.10] fix AttributeError 'Connection' object has no attribute 'HOOKS_DENY_ALL'

[server/sources] Fix authenticate on pre-3.10 database When migrating from a pre-3.10 database, the CWSource entity type and cw_source relation type are not known, but authenticate should still work to be able to add them.

[test] Add a test ensuring new cube built from skeleton is flake8-compliant And make skeleton files actually flake8-compliant henceforth.

[skeleton] Ship a tox.ini file upon new cube

[web] fix headers on 304 responses (closes #11466875) The response MUST include the following header fields: [...] - ETag and/or Content-Location, if the header would have been sent in a 200 response to the same request https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.5 Unfortunately we weren't copying headers from what would have been the 200 response, but from the request.

[web/httpcache] GMTOFFSET no longer exists

[web] fix headers on 304 responses (closes #11466875) The response MUST include the following header fields: [...] - ETag and/or Content-Location, if the header would have been sent in a 200 response to the same request https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.5 Unfortunately we weren't copying headers from what would have been the 200 response, but from the request.

[rset] fix infinite recursion introduced in c1eb5a676c80 The mechanism to avoid loops in c1eb5a676c80 "[rset] Always complete attribute/relation caches in ResultSet.get_entity" breaks down: we would loop forever if two entities that were already in the cache were linked by a relation with ? or 1 cardinality in both directions. To avoid that, keep a set of already-considered columns. Related to #9942503.

pep8 unittest_application and unittest_formfields

[rset] Always complete attribute/relation caches in ResultSet.get_entity RQL queries are often designed to fill up the ORM's caches when fetching entities out of the result set. Until now, if an entry already existed in the entity cache, ResultSet.get_entity would return it unchanged, not using the new ResultSet's contents to update the attribute cache, breaking expectations (if the attributes are needed, they'd then be fetched later one at a time, one entity at a time), resulting in loads of DB accesses. So we change ResultSet.get_entity so that: * if the entity is already cached and has been instantiated from the same rset, it is returned as-is (to avoid loops) * if the entity is not yet cached, it is instantiated * if the entity is cached via another rset, its attribute/relation caches are completed Closes #9942503

[rset] refactoring of _build_entity method define a method for entity instantiation and storage in cache (related to #9942503)

[entity] comparing two entities returns NotImplemented Fixes a test when using some versions of unittest2, where assertItemsEqual tries a sort and expects TypeError, which it doesn't get if we directly raise NotImplementedError. The NotImplemented value is treated specially by the interpreter: https://docs.python.org/3.4/library/constants.html#NotImplemented

[test] replace one use of lgc.testlib.unittest_main with unittest.main

[test] stop using TestCase.set_description It's a lgc-ism.

[test] convert a lgc-style generative test into unittest2-style

[web/application] fix query log handling on python3 Writing bytes to a file opened in text mode doesn't work.

[devtools/test] duplicate CubicWebServerTC tests for CubicWebWsgiTC

[devtools/test] fix bytes vs text in test_base_url Needed for python3 with the next changeset

[devtools] skip CubicWebServerTC on python3 People should migrate to wsgi if they're going to use CubicWeb on python3.

[web/test] switch unittest_web from twisted to wsgi And make it pass on python3 while at it.

[devtools] add CubicWebWsgiTC Similar to CubicWebServerTC, except using wsgiref instead of twisted as an http server.

[web/test] return the hash of uploaded files in FileUploadTC instead of their contents json must be unicode, which doesn't go well with arbitrary file contents. Compute the file hash instead, as we know the hexdigest is (ascii) text.

[web/test] open() as a context manager

[web/test] json.loads expects text, not bytes

[wsgi] fix handling of multiple posted files We were only keeping the last received file.

[web/test] Reword doc string for fileupload ajaxfunc

[skeleton] do not import the __pkginfo__.py module to make cubes installable via setuptools When using a simple "import" statement, running 'python setup.py install' for a cube fails in a fresh virtualenv due to the fact the install of the dependencies (here cubicweb iteself) is done by the python process doing the cube's installation, so the __pkginfo__ module from the cube is already loaded (thus used) when installing cubicweb... So we apply the third technique from: https://packaging.python.org/en/latest/single_source_version.html

merge changes from 3.22.2

Added tag 3.22.2, debian/3.22.2-1, centos/3.22.2-1 for changeset 1b93ff37755b

[pkg] 3.22.2

[pkg] fix centos install

[datafeed] do NOT call commit in DataFeedXMLParser.process() (closes #10928592) and so, remove more useless code. Doing a commit in DataFeedXMLParser.process() is a bad idea because this method is called for every imported/updated entity, before handling deletions. So if we commit here, odds are quite high that we're violating some integrity constraint and end up with a ValidationError.

[schema] restore constraint checking when running on old sqlite Old sqlite3 doesn't provide CHECK constraint names in error messages, preventing us from translating a backend integrity error into a ValidationError. This was added in 2012, but the sqlite3 version in RHEL6 is older; so if we run on old sqlite, keep checking the constraints in python rather than only in SQL. Closes #10927494

[web/autoform] fix regression introduced by #5758ba784ebd The changes from cset 5758ba784ebd "[autoform] when building inlined-form view for a relation, consider related entities intead of other views" lead to an unexpected 'add new' link on creation of an entity with inlined form for a relation of cardinality 1. To avoid this, we should consider formviews and not existing relations when the entity is being created. Closes #10860258

[tox] Generate test environments for Python 2.7 and 3.4 Test commands for each environment are written down explicitly since I could not find a way to extract the "package" name (e.g. "hooks") from the environment name (e.g. "py34-hooks"). For Python 3.4 interpreter, only environments (subpackages) that do not depend on cubes for their tests are listed since those test dependency cubes are not yet installable with Python 3.x. etwist is also not included since the Python 3 port is not complete at the moment. From local testing, py34 tests pass for the following subpackages: - dataimport - entities - ext - hooks - wsgi

[notification] avoid using InternalManager when we've got a CWUser This will avoid potential information leak due to missed security restriction and is more compatible with legacy code (pb encountered in some client code). Closes #10294761

[dataimport] rename .time to .now attribute of meta generator

[pkg] Set version to 3.22.2.dev0 So that cubes used in test dependencies do not install a released CubicWeb.

[dataimport] cleanup a bit nohook store, mostly protecting some attributes

[dataimport] add a filter function to not fail if some extentity has several values for an attribute of final relation the function will simply record a warning in the import log and keep a value randomly. Notice the function has to be explicitly inserted in the transformation stream.

[sources] synchronize source asynchronously when started from the UI and redirect to the forthcoming import log, for a better user experience (there is still a lot more to do there though). Closes #10468967

[test] fix unittest_devctl on python3.4

[test] Make sure that newcube command test produces expected files

[server] remove some pre 3.10 compat Let's assume nobody will try upgrading an instance from pre-3.10 cubicweb to 3.23+. If they do, they're setting themselves up for a lot of pain anyway.

[pkg] use https URLs to pypi.{python.org,debian.net}

[pkg] use https URL for www.cubicweb.org

README: use https URLs for docs.cubicweb.org

pep8 bits

[i18n] enhance text formats french translation

Merge with 3.22 branch The merge was clean, just dropped cubicweb/web/data/cubicweb.goa.js.

[dataimport] update example to use standard importer

merge changes from 3.21.6

flake8 serverctl

c-c source-sync now actually force synchronization as advertized by its help message. This was because "force" argument wasn't considered by acquire_synchronization_lock().

Added tag 3.21.6, debian/3.21.6-1, centos/3.21.6-1 for changeset b3cbbb7690b6

[pkg] 3.21.6

merge changes from 3.20.13

Added tag 3.20.13, debian/3.20.13-1, centos/3.20.13-1 for changeset 8c5dabbcd4d9

[pkg] 3.20.13

[pkg] Yams >= 0.42 breaks CubicWeb < 3.21 Closes #9720390.

merge changes from 3.19.14

Added tag 3.19.14, debian/3.19.14-1, centos/3.19.14-1 for changeset 72a0f70879ac

[pkg] 3.19.14

[sources/native] remove extra space before colon This isn't French.

[web/views] fix editcontroller regression unittest_views_basecontrollers.py's EditControllerTC.test_edit_mandatory_inlined3_object regressed with changeset dcbb64d3a1d9 because it relied on POSTing nonexisting eids. This seems like a silly idea, but let's keep the behaviour unchanged.

Added tag 3.22.1, debian/3.22.1-1, centos/3.22.1-1 for changeset d0d86803a804

fix test broken by #012e6c7d02ef

[pkg] Attempt to fix rhel6 build I don't know what I'm doing, but this seems to work.

[dataimport/test] fix python2.6 syntax error Don't use braces for set literals.

Remove unused import in entity.py

[migration] don't ask confirm for those sql queries

quick fix which closes #0673348 - see comment for more info

[migration] test if entity type has been in the deleted in the transaction should protect both clauses else we may attempt to execute the UPDATE query on a table which has been deleted. Closes #10692374

[pkg] 3.22.1

[cwctl] remove support for startorder file in /etc/cubicweb.d (closes #5618947) It's now useless.

[exceptions] fix string conversion for Unauthorized exception changeset 3cb87b61b067 changed CubicWebException to define __unicode__ as the main entry point for string conversion, but left Unauthorized's __str__ definition unchanged, leading to a mismatch. Closes #10646396.

Fix Binary on python < 2.7.4 (closes #10593811) Before the fix for http://bugs.python.org/issue10211 we can't pass a buffer object to BytesIO, so keep using StringIO in that case. And because StringIO is an old-style class, we can't use super().

[migration] fix crash on 3.22 upgrade if entity type doesn't exist in fsschema (pending removal)

[autoform] when building inlined-form view for a relation, consider related entities intead of other views by considering previous form-views, we may miss that there exists related entities but which are not editable. Closes #10495602

Add a __main__ to cubicweb This avoids calling cubicweb-ctl relatively in tests and could be useful in virtualenv. This is just a copy of bin/cubicweb-ctl file, which should probably be generated from a setuptools entry points.

merge 3.19 heads Getting an unexpected test error though: ERROR: test_edit_mandatory_inlined3_object (unittest_views_basecontrollers.EditControllerTC)

[cwvreg] Fix propertyvalues hot updates When CWProperty instances are touched, the corresponding value gets updated in vreg['propertyvalues'], but their type was ignored.

[pkg] remove useless python >= 3.4 dependency python is python2 is never >= 3.4.

[dataimport/test] fix test failures since 35b29f1eb37a The index name changed, so we need to follow suit.

[pkg] do not include toplevel directories in rpm (closes #10362119) Our spec file conflicts with the filesystem package in centos7: we set permissions to 0755 for /usr/bin and /usr/lib whereas filesystem sets them to 0555. So to be compatible with both centos7 and others we simply ignore these directories.

[devtools] fix config.repository signature Its argument is called 'vreg', not 'x'. Closes #10515113.

[predicates] fix paginated_rset when the class has no page_size_property Fixes test failure introduced by 70733b2dfd2a, which assumed paginated_rset would only be called on a class with a page_size_property attribute.

[repository] drop usage of no more necessary eschema_eid function since the previous cset, we are guaranteed that repository's entity schema will have their eid attribute properly set, so we don't have anymore to check everytime if some entity schema's .eid attributes is set or not (and retrieve it if not). Related to #10450092

[repository] set .eid on eschema when schema is loaded from the filesystem enforcing the contract that a repository's schema should have .eid attribute of entity schema set to the eid of the entity used to serialize them in the db. Before this cset, this was not true during tests or for some c-c commands where 'quick_start' is set (eg db-restore, i18ncube...). The change in server __init__ makes this assumption true during instance creation: the serializing code was actually setting eid on schema object, but a reference to a previously built schema was given instead of the one for the latest created repository. Closes #10450092

[dataimport] simplify SQLGenSourceWrapper.add_info At this point 'CWSource' is a thing (we're not in the middle of a migration), and source.eid won't be None (we read it from the db)

ensure .tox is blacklisted when recursing on the fs to get some files among others, avoid crash when using 'c-c i18ncube' on a cube using tox. Closes #10429317

drop a note about usage of sqls_create_multicol_unique_index and avoid a lgdb warning

[devtools] don't auto-populate virtual relations

[skeleton] Restore verbose output in setup.py This part of commit fdadf59be612 missing the integration window.

[hooks] Skip modification_date attribute in "notifentityupdated" hook Depending on the call order of hooks, "modification_date" may come in entity.cw_edited or not (from "metaattrsupdate" hook usually). It should however not be considered in the notification context. Closes #8842064.

[test] Gather tests about mail module From sobjects/test/unittest_notification.py to test/unittest_mail.py where other tests for mail.py are already. Some style fixes along the way.

[web/component] guard against invalid page_size form parameter Same as dff28956f37e, except the almost same code exists in two places (yay us).

[predicates] fix paginated_rset for components that change page_size_property The predicate and component had better agree on the page size.

[devtools] fix get_bounds Say an attribute has two IntervalBoundConstraints, one with minvalue=0 and the other with maxvalue=10 (not literally, or that could be a single constraint, but e.g. with references to other attributes). We don't want to override the minvalue of the former with None just because the latter happens last in the constraints list. This doesn't fix the case where two constraints both have a min or a max value. Meh.

[devtools] "fix" boundary condition for autofill maxvalue in an IntervalBoundConstraint is inclusive. If the min and max values happened to coincide, we would call random.choice on an empty list, which it didn't like. This is still broken in the case of BoundaryConstraint('>', minvalue), as we assume the minvalue is in the range, but that's something for another time.

[test] Use stdlib doctest module instead of logilab-common class The former just works fine now but the later does not play well with py.test.

[test] Re-implement generative tests with subtests

[devtools] Re-implement generative tests using subtests Generative tests as implemented in logilab.common.testib are not compatible with tests runner other than lgc.pytest and this implementation differs from the standard library, which has support for subtests_ since Python 3.4. Use unittest2 to bridge the gap. Maybe it'd be good to implement this on logilab.common.testlib side at some point. Let's see how this gets received here first. .. _subtests: https://docs.python.org/3/library/unittest.html#subtests

cleanups

[pkg] python-tz does not exist in centos should be pytz (closes #10290056)

[test] Skip LDAP tests if any of the underlying programs is not found Some programs are in ldap-utils or ldapscripts, which may not be installed despite slapd may. Follow-up on 282880f81311.

Handle virtualenv development mode Looking for VIRTUAL_ENV environment variable as a sign of virtualenv activation. This at least works for developing CubicWeb itself (not sure yet about cubes). And it makes `tox --develop` work as well. Add a test for cwconfig._find_prefix within a virtualenv. In tox.ini, use `develop option`_ accordingly. This makes tox running significantly faster since it avoids building and installing the sdist. Aside, the "develop" install is now performed *after* all test requirements are processed, which means that the cubicweb package should always be the one from source -- instead of a released version pulled by a cube through test dependencies, which may occur in "recreate" step. .. _`develop option`: <http://tox.readthedocs.org/en/latest/config.html#confval-usedevelop=BOOL

[skeleton] Use os.walk in setup.py for Python3 compatibility

[skeleton] Use unittest.main directly Instead of lgc.testlib.unittest_main as they are identical.

[devtools/testlib] Disable pause trace in setUp Closes #9856028.

[test] Remove dependency on person cube in devtools package

[pkg] add missing python-tz dependency

[edit controller] Fix composite entity reparenting when inlined This implies handling the reparenting case sooner, otherwise the child entity modifications are cancelled (see 113e9da47afc). Add a new test to ensure non-regression in all previous tests. Closes #10291977.

[forms] Do not raise errors on composite required relation being deleted The entity included in the composite will be deleted in such a case, so the field is not required even when the cardinality is '1' or '+'. Related to #8529868

[edit controller] Cancel RQL queries to be performed on entities to be deleted Composite relation removal via the EditController trigger an entity removal. Changes on such a to-be-removed entity must be cancelled as they may trigger integrity errors before the deletion occurs. Closes #8529868.

[edit controller] fix handling of removal of subentities from an edit form In CubicWeb 3.18 the semantics of dropping a composite relation were changed. It had before the effect of deleting the subentities. Now the edit controller must handle this by itself (it was not adapted then by mistake). So if it detects the removal of a composite relation (on the browser side, the user clicked on the "remove" action), the subentity is now explicitly deleted. It was chosen not to change the form generation but the EditController: - to ensure pre-3.18 forms backward compat - to avoid the introduction of a specific entity-deletion API (RQL is probably our best API for complex DB operations) Related to #8529868.

remove unused import

[editcontroller] a slight code reorganisation, in preparation for a fix * `entity` defined at method entry * a combined `continue` and a dedent of the next big block * some vertical white space Related to #8529868.

[sources/native] ask permission to continue restoring even if metadata do not match (closes #7771864) Moreover we try to clarify the message (Expected/Found was never too clear for users).

[views] fix ProcessInformationView: SESSION_MANAGER can be None (closes #5753280) eg. with pyramid. (grafted from 6a07c2dc3d85f4de0cf48221607e02c4473e11e4)

[web] Move "display all"/ "back to pagination" link generation into the navigation component This makes the html generation customizable. Closes #9219057

Reorganize source tree to have a "cubicweb" top-level package Basically: mkdir cubicweb hg mv *.py -X setup.py cubicweb hg mv dataimport devtools entities etwist ext hooks i18n misc schemas server skeleton sobjects test web wsgi cubicweb Other changes: * adjust path to cubicweb-ctl in devtools tests * update setup.py to avoid importing __pkginfo__ (exec it instead), replace os.path.walk by os.walk and prepend `modname` here and there * update tox.ini to account for new test locations * update doc/conf.py so that it still finds __pkginfo__.py and CWDIR in doc/Makefile

clean some remaining dust from goa time

[datafeed] io.BytesIO requires a buffer, not a unicode (closes #9783743) The URLLibResponseAdapter should probably be refactored (or killed?)

[pkg] rql 0.34 and logilab-database 1.15 break cubicweb 3.21

[pkg] rql 0.34 and logilab-database 1.15 break cubicweb 3.21

Added tag 3.22.0, debian/3.22.0-1, centos/3.22.0-1 for changeset de472896fc0a

[pkg] 3.22.0

merge

[formwidgets] a bit of pep8

[views] improve JQueryDatePicker so that a date field can be defined as the min/max of an other one closes #5169039

[entity] don't look at fetch order recursively If fetch_attrs includes relations, stop including the entities on the other side in our sorting. Fixes regression from 73ea636a5562 where we would sort on the target entity before the attribute we wanted if the relation's name sorted before the attribute's. This showed up in the forge/tracker cubes with Version entities being sorted by their State's name in preference to their version number.

[pkg] fix paths in debian doc-base control file

[ldapfeed] extid and cwuri aren't the same thing Historically, we used a ldap://... URI as cwuri attribute, not just the dn. The extid field, OTOH, was always the dn. So when initializing our dict of known entities from an ldap source, look at existing extids, not cwuris, to avoid importing already-existing users (and violated unicity constraints).

[entities] add eid to ISerializableAdapter Life without an eid is made of sadness.

[doc] 3.22 release notes update

[datafeed] use tz-aware datetime objects Avoid TypeError when comparing tz-naive and tz-aware objects in fresh() method.

[pkg] fix syntax error in debian/control

[pkg] remove no-longer-used addresses from debian control file

[schema] rebuild computed attributes on schema reload If we have new computed attributes we need to set their formula_select attribute or things get sad.

[hooks] fix extra_props handling on python3 The json module expects str, not bytes, so we need to do the decoding.

[py3k/ldapparser] avoid confusion between unicode and bytes - ldap3 dicts contain unicode strings - extid must be bytes

[pkg] use correct lgdb version for tz rework Related to #4848923.

pep8 bits

[server] improve TZDatetime support by depending on logilab-database 1.15.0 including https://www.logilab.org/ticket/1485893, we can now keep the tzinfo attribute on datetime objects for TZDatetime attributes, so one knows that this is a tz-aware datetime. To easily make it work with backends that have no tz support, we keep converting tz-aware datetime objects into utc naive datetime objects before sending them to the database.

[schema] Use TZDatetime for creation_date and modification_date No work has been done for form fields/widgets since creation_date / modification_date are by default not editable through the default UI. One may want to add such feature at some point. Time will be displayed as UTC, let the end-application decide otherwise by customizing the tzdatetime printer if desired. Closes #4848923

[server] fix SQLite patch to ensure TZDatetime are properly typed NB: at this point, the TZ info is completely ignored. It will be properly handled in later commits.

[server] fix name and add docstring to connection initialization functions

[server] always monkeypatch QuerierHelper to handle dates and times on SQLite The patch prevents problems due to a SQLite bug causing some values to be returned as strings instead of properly typed objects. This should probably be improved to use SQLite connection hooks. But the major improvement is that the monkeypatch is now done all the time, and not just for tests.

[testlib] put postgres test database in test/data/database directory with other (sqlite) test databases. Closes #9240765 Simplify .hgignore in passing.

[dataimport] make eids_seq_range as massive store instance attribute again I changed it to a class attribute recently, but on further thought (and prompting from Adrien) it seems to make more sense as an instance attribute.

[dataimport] stop supporting None as eids_seq_range It couldn't work as-is because entities_id_seq is not a sequence object.

[dataimport] remove eids_seq_start attribute from massive store Instead provide a utility function to reset the eid sequence.

[dataimport] massive store in slave mode shouldn't flush metadata The metadata insertion is to be done only by the master-mode store.

[dataimport] check that MassiveObjectStore restores the db schema properly Ideally we'd use "pg_dump --schema-only" to compare before/after, but when restoring a constraint such as: cstr48f9ceae537d68859b62c601681c8d3e CHECK (((cw_type)::text = ANY ((ARRAY['normal'::character varying, 'auto'::character varying])::text[]))) postgres will convert the text cast on array to a cast on each cell of array cstr48f9ceae537d68859b62c601681c8d3e CHECK (cw_type::text = ANY (ARRAY['normal'::character varying::text, 'auto'::character varying::text]))

[dataimport] split out listing indices/constraints from getting the corresponding query

[dataimport] massive_store: do not drop constraints multiple times It is now done in __init__.

[dataimport] massive_store: do not drop constraints with `cascade` since it breaks things We want to be able to restore constraints at the end of the import. If they are silently dropped due to a CASCADE, we can't do that.

[dataimport] massive_store: drop all constraints in __init__ and restore all in finish This should avoid dependency issues when dropping specific constraints (e.g. when dropping a primary constraint referenced by a foreign key constraint). We only do this in master mode, slave mode stores are supposed to work with an initialized db already.

[dataimport] massive_store: ignore primary key when requesting indexes on a table Primary keys show up in both lists of constraints and indexes. Pick a side and stick to it.

[dataimport] massive_store: reformat sql query

[devtools] Use super() in TestServerConfiguration.__init__() While cleaning up the use of init_config() in WSGIAppTC, I wondered why the method was trying to set the 'https_uiprops' and 'https_datadir_url' attributes on the config, when WebConfiguration's __init__ should be taking care of setting them. WSGIAppTC uses the default configuration - one that derives from ApptestConfiguration (noted as G below). The issue is that the class hierarchy of ApptestConfiguration includes TestServerConfiguration (C in the graph below) which calls ServerConfiguration's __init__ directly (B below). As B does not have an __init__ method, CubicWebConfiguration's __init__ is called directly (A below). This has the unintended consequence of preventing WebConfiguration's __init__ from being called. A / \ B D | | C E \ / F | G | ... A: cwconfig.CubicWebConfiguration B: server.serverconfig.ServerConfiguration C: devtools.TestServerConfiguration D: web.webconfig.WebConfiguration E: etwist.twconfig.WebConfigurationBase F: devtools.BaseApptestConfiguration G: devtools.ApptestConfiguration NB: unittest_wsgi wasn't really broken, as it's the only test in wsgi/test/, but I figured I might as well clean it up. Related to #9297611.

[devtools] Fix database creation issues for CubicWebTestTC tests To reproduce (with a purged repo): * python web/test/test_jscript.py * python web/test/unittest_views_xmlrss.py The '__default_empty_db__' will contain the wrong 'base_url' configuration, thus breaking URLs in the xmlrss tests. Closes #9297611.

[devtools] Fix database creation issues for CubicWebServerTC tests To reproduce (with a purged repo): * python web/test/unittest_web.py MiscOptionsTC * python web/test/unittest_views_xmlrss.py The core issue is that CubicWebServerConfig.default_base_url() is called inside init_config(), which is called during the equivalent of 'db-create'. The '__default_empty_db__' will contain the wrong 'base_url' configuration, thus breaking URLs in the xmlrss tests. Related to #9297611.

[devtools] Move attribute docstring to correct location

[web] Make tests pass if tal isn't available

[py3k] fix cubicweb.mail.parse_message_id base64.b64decode returns bytes. Fixes sobjects/test/unittest_notification.py.

[test/py3k] fix test_recipients_finder AssertionError: <zip object at 0x7f1dfb0230c8> != [('abcd@logilab.fr', 'en'), ('efgh@logilab.fr', 'en')]

[dataimport] fix SQLGenObjectStore's add_info for python3

[dataimport] py3k doesn't like + on dict_items objects

[repository] remove unused check_session method and its test Related to #1381328.

[migration/test] avoid test interdependency by removing fakecustomtype cube There is still a failing test afterward but it doesn't seem related to changes in this series.

[migration/test] use numeric(precision, scale) instead of geometry The latter requires the postgis extension which is overkill for this test.

[server, hooks] allow callable in dbh.TYPE_MAPPING This is necessary for cubes introducing new custom final types whose generated sql may depend on some extra configuration, avoiding ugly-buggy hack as done currently by the postgis cube. Closes #7569998

[migration] consider extra_props in rdef created during attribute creation hook else we may miss important data such as type parametrization (see postgis.Geometry for instance). Related to #7569998

[migration] only define variable if needed

[migration] handle non-existing attribute type when adding an entity type The final type needs to be added to the schema first. Related to #7569998.

[migration] handle the case where new final entity type is not yet in the schema This may occur on new cube addition. Related to #7569998.

[migration] config.add_cubes should ensure site_cubicweb of new cubes is called Related to #7569998

[migration] test addition of a cube providing a custom final type This does not work yet. Related to #7569998

[server] add test demonstrating cache update pb since cw > 3.21 Fixed by the previous changeset.

Bring back the separate web-side entity cache Prior to changeset 635cfac73d28 "[repoapi] fold ClientConnection into Connection", we had two entity caches: one on the client/dbapi/web side, and one on the server/repo side. This is a waste, but it is actually needed as long as we have the magic _cw attribute on entities which must sometimes be a request and sometimes a cnx. Removing the duplication caused weird problems with entity._cw alternating between both types of objects, which is unexpected by both the repo and the web sides. We add an entity cache on ConnectionCubicWebRequestBase, separate from the Connection's, and bring back the _cw_update_attr_cache/dont-cache-attrs mechanism, to let the server/edition code let caches know which attributes have been modified Entity.as_rset can be cached again, as ResultSet no longer modifies the entity when fetching it from a cache. Contrary to the pre-3.21 code, _cw_update_attr_cache now handles web requests and connections in the same way (otherwise the cache ends up with wrong values if a hook modifies attributes), but dont-cache-attrs is never set for (inlined) relations. Closes #6863543

Backout "[entity,storage] remove hackish code from the bfss path The merge of the repo-side and web-side entity caches doesn't actually work, because entities have a _cw attribute which is sometimes a connection and sometimes a web request, and whose nature changes not-really-predictably whenever ResultSet.get_entity() is called and brings it out of the cache. This reverts commit 59a79300f213.

merge with 3.21.5-2

[doc] add documentation about drop_cube

Added tag debian/3.21.5-2 for changeset 228b6d2777e4

[pkg] don't install dataimport/test in both cubicweb-server and cubicweb-dev Closes #9239572

merge with 3.21.5

Added tag 3.21.5, debian/3.21.5-1, centos/3.21.5-1 for changeset e0572a786e6b

[pkg] 3.21.5

[js] Fix a ReferenceError in cubicweb.preferences.js Rename function using proper English along the way. Closes #8864731.

[dataimport] add tests for SQLGenObjectStore (closes #9177565) Stolen from the massive import store, with the irrelevant bits removed. Remove redundant tests from test_pgstore.py.

[dataimport] separate entities table from other metadata in SQLGenObjectStore 'entities' needs to be updated first as every other table has references to its rows. Remove use of thread-local storage in the process. related to #9177565.

[dataimport] remove threading support from SQLGenObjectStore It's untested and does not actually work at the moment with nb_threads > 1. related to #9177565.

[dataimport] prepare_insert_entity should not crash with SQLGenObjectStore grafted from 1079d68130e1e780791ceb2050e8a8b7a910f48d (3.22) related to #9177565.

[dataimport] Fix method signature Since d4bd28d5fca8, _storage_handler expect 4 args. grafted from b089858c49a2806b6ea8a7ee66773436701e947c (3.22) related to #9177565.

[dataimport] add missing import 'eschema_eid' grafted from e3ec46b5c710067a0cecca53cc593a0dd4bd927e (3.22) related to #9177565

merge with 3.21.4

Added tag 3.21.4, debian/3.21.4-1, centos/3.21.4-1 for changeset d3b92d3a7db0

[pkg] 3.21.4

merge with 3.20.12

Added tag 3.20.12, debian/3.20.12-1, centos/3.20.12-1 for changeset 03e8fc9f79a6

[tox] Ensure pytest gets run with virtualenv's python

[tox] Use generating environments Drop the "env" option, which is unknown, and set a generative_ "envlist". Then use conditional dependencies based on factors_ to avoid repeating environment sections. The only singular one is the "cubicweb" environment because of its "commands". At some point, it'd be nice to find a way to have python version prefixes (py27-hooks or py34-web) for environment names but I could not find a way to extract the package name from such names. .. _generative: http://tox.readthedocs.org/en/latest/config.html#generative-envlist .. _factors: http://tox.readthedocs.org/en/latest/config.html#factors-and-factor-conditional-settings

a bit of pep8

[adapters] Use an adapter for serializing entities to a bare python structure which may be used for e.g. JSON export or other and allow to have different serializations for each entity types. Closes #6378035

[pkg] 3.20.12

[dataimport] Fallback from COPY to executemany (closes #9135780) If we can't build a suitable COPY command (e.g. because data has unsupported type), use executemany instead. Regression introduced in 3.20.0.

[web/views] unlink svg temp file after we're done Otherwise we fill up the tempdir. Closes #9133268.

Use cubicweb._

[web/views] Controllers should always return bytes Previous code breaks only on py3k.

[devtools] Fix 'newcube' on py3k

[toolsutils] Small fixes to fill_templated_file() * Use context managers to close _both_ files * Use io.open to enforce text mode on both python 2 and 3

[skeleton] Fix broken spec file template Broken in commit 73c1c9cf6bda.

[devtools] Fix fd leak in tests

[devtools] stop using datetime.now() in cwctl newcube and i18ncube

[hooks] fix timestamp confusion in DataImportsCleanupStartupHook CWDataImport.start_timestamp is inserted as datetime.utcnow(), in server/sources/datafeed.py:DataFeedSourceinit_import_log. Don't compare it to datetime.now().

[hooks] fix timestamp confusion in TransactionsCleanupStartupHook transactions.tx_time is inserted as datetime.utcnow(), in server/sources/native.py:NativeSQLSource.start_undoable_transaction. Don't compare it to datetime.now().

merge with 3.21.3 The change in unittest_serverctl.py is needed because of daef7ce08fea (from 3.20.11) and 3914388b2d0f (from the 3.22 branch). Due to both changes, CubicWebTC.config.repository no longer creates a new repository (and thus, a new connection). Since both DBDumpCommand and CubicWebTC.tearDown try to shutdown the repo, tearDown breaks apart. The solution is to temporarily disable ServerConfiguration's config cache. By forcing DBDumpCommand to get a new configuration object, it then gets its own Repo object, allowing tearDown and DBDumpCommand to work independently.

Added tag 3.21.3, debian/3.21.3-1, centos/3.21.3-1 for changeset 9edfe9429209

[pkg] 3.21.3

merge with 3.20.11

update repo sources on update of source's entities .url attribute

Added tag 3.20.11, debian/3.20.11-1, centos/3.20.11-1 for changeset c44930ac9579

[pkg] 3.20.11

[rest] a bit of pep8

[rest] avoid field on two lines if name is above some size limit. Closes #8961835

action, view and service so managers can start source synchronization from the web ui Closes #5474286

[test] Skip ldap source tests if slapd is not available

[doc] Minor changes to the blog tutorial

[skel] remove (what looks like) a spurious copy/paste

Merge default heads published "concurrently"

[qunit] move from CubicWebServerTC to CubicWebTestTC This test now uses wsgi instead of twisted, and thus passes under python3.

[qunit] no need to enable CORS anymore We serve everything needed for the tests through cubicweb.

[web/test] use GET, not POST, to load html files POST comes with a body. There isn't one.

[dataimport] prepare_insert_entity should not crash with SQLGenObjectStore

[dataimport] Fix method signature Since d4bd28d5fca8, _storage_handler expect 4 args.

[dataimport] add missing import 'eschema_eid'

[test] Remove unused import in unittest_qunit.py

[dataimport] Format strings with % instead of .format() For consistency with CubicWeb code base and to avoid Python 2.6 compatibility traps.

[test] Use datapath() in massive store tests

[devtools] Disable anonymous login right before executing tests CubicWebTC.init_config is called many times, especially when creating the default database. This database is then reused by all tests in the same directory. The issue is that, if a test has 'anonymous_allowed' set to False, the anonymous user won't be created by postcreate.py in the default database. All other tests will fail miserably if they expected anonymous login to work. By actually disabling anonymous login after _init_repo, the default database (and configuration) will be initialized with a working anonymous user. Closes #7485127.

[hooks] build EnsureSymmetricRelations{Add,Delete} dynamically Instead of setting the __select__ attribute at registration time, which confuses the filterable_selectors cache when running different tests (with different schemas) in the same process, just make entirely new classes. Closes #8785708.

[qunit] stop dealing with filesystem paths qunit tests need a few things served by cubicweb: - qunit itself, which is now handled by CWDevtoolsStaticController (serving files in cubicweb/devtools/data) - standard cubicweb or cubes data files, handled by the DataController - the tests themselves and their dependencies. These can live in <apphome>/data or <apphome>/static and be served by one of the STATIC_CONTROLLERS This avoids having to guess in CWSoftwareRootStaticController where to serve things from (some files may be installed, others are in the source tree), and should hopefully make it possible to have these tests pass when using tox, and to write qunit tests for cubes, outside of cubicweb itself. This requires modifying the tests to only declare URL paths instead of filesystem paths, and moving support files below test/data/static.

[devtools] fix QUnitResultController It needs to return something.

[utils] Drop now-unused SizeConstrainedList It's last usage in CubicWeb got removed from changeset '[views] remove breadcrumbs usage to retrieve last visited page' Related to #5456850.

[views] remove breadcrumbs usage to retrieve last visited page Use JavaScript history.back to go to the last visited page. Rename _return_to_lastpage to _redirect as it becomes misnamed. Drop `test_redirect_default` test in `unittest_views_basecontrollers` which essentially tested this (now removed) breadcrumbs "feature". Closes #5456850.

[cwconfig] Explicitly close generated config file

[doc] Add 3.21 release date

[dataimport] fix db-namespace default value handling In a fresh new instance, the "db-namespace" option is set as empty by default in the "sources" file. This seems correct but empty string is not a valid pg_schema and actually means that we should use "public"

[server] unbreak db-restore

[doc] add 3.22 changelog

[rset] remove stray (commented-out) print Left from debugging in af266f27c4d5

Un-hide jshintrc dot-files belong in ~, not anywhere else.

[views] Use 'pyvaltable' instead of hand-crafted <table> tags

[web] Stop using config.repository() Since commit 3914388b2d0f, this method creates a new Repository object and thus reloads all the cubes, taking for ever. Closes #8634417.

[web] Stop using webconfig.vc_config() The main issue is that this creates an entirely new Repository object (since commit 3914388b2d0f) and thus reloads all the cubes, taking for ever. Related to #8634417.

[migration] don't handle data deletion anymore on schema changes In most cases when we want to drop some entity/relation type, we don't care whether hooks are called on their deletion. There is even low chances that some hooks still exists, based on an old version of the schema. Last but not least, this is horribly inefficient. So this should be clearly documented and handled by application's programmer if desired. This patch removes unnecessary deletion (because table or column will be later dropped) and reimplements the case of partial deletion (only one relation definition among several, hence the database structure isn't modified) using sql. Only one test regarding deletion of inlined relation def is added as other cases seem to be covered by existing tests. Closes #7023315

[hooks/syncschema] drop unused import

[hooks/syncschema] indentation fix

[server/test] cosmetic whitespace fix

[hooks/syncschema] don't set NOT NULL on inlined relation columns The NOT NULL constraint was being added on inlined relation cardinality change while it shouldn't. We don't want this, because this forbids to add an entity without the relation set at once (it may be added later in the transaction). This patch fix synchronization hook to stop doing this. Closes #6211101

[hooks/syncschema] Turn DropColumnOp into a data operation While writing a test for a migration issue where "NOT NULL" was added to an inlined relation column, I ran into a case where the same column was deleted twice, which of course results in an error. This is because the "ecrit_par" relation used by the test is ambiguous. I've hence turned the DropColumnOp into a data operation to ensure we'll delete the column only once in such case. Related to #6211101

[server/sources] fix name error in eid_type_source_pre_131 i.e. the eid_type_source implementation used when some pre 3.13.1 database is detected. Regression from 5fbdbbe51867.

[repository] deprecate the extid2eid based multi-sources API This API is cumbersome and lead to obfuscated code because of the callback mecanism implied when some entity has to be created. Since we dropped the "true" multi-source, this mecanism is not needed anymore and one should prefer to use the dataimport API inside its parser instead. Notice the cwxml parser will trigger the deprecation warning, it should not be used anymore in favor of ad-hoc parsers.

[ldapfeed] use source's extid2eid, the repo's one will be deprecated in a near future.

[ldapfeed / dataimport] port ldapfeed parser to dataimport API This makes the code easier to understand and will allow to deprecate the old multi-sources api, based on creation callback through repo.extid2eid. Currently with this patch, modification dates are not checked, hence entities will be systematically updated. We run the import with no hooks, because RQLObjectStore can only add entities to the system source.

[server/test] document expected packages to run ldap tests

[ldapfeed] simplify ldap2attrs * tdict argument is never given * separate value retrieval from setting output (will ease a latter cset)

[ldapfeed] simplify deactivation of unencountered users there is no need for the by_etype intermediary dictionnary since we only consider CWUsers.

[server/test] display rset content on failure

Remove trailing empty lines Makes pep8 happier

[ldapparser] simplify extid splitting

[test/ldap] a bit of pep8

[js] add configuration file for jshint (closes #5593987) copied from https://gist.github.com/haschek/2595796

spelling: *aly → *ally

[web] Add a <label> around choices on CheckBox and Radio form widgets Allows clicking directly on the text rather than only on the widget. Closes #7672774.

[web] xml_escape CheckBox and Radio labels (closes #7672764)

[test/tox] Enforce installation of modified yapps 580a6821713d is not enough for tox apparently... The thing is that it will first install dependencies (the `deps` option from tox.ini), actually pulling yapps from Logilab github but then, it will install the project (that is, CubicWeb) from the sdist *along with its dependencies*, which will pull yapps again and override the previously installed version for some mysterious reason. So add a `pip install` command in tox.ini, so that the modified yapps is the last one installed.

[web] delete unused PropertySheet._sourcefile_mtime

[etwist] remove no-longer-necessary uicache cleaning It was done before we were actually up, which means it could be too eager when combined with #6893185. Closes #6893247.

[web] remove PropertySheet._cache Instead of remembering that a css file has been processed, we re-process it at every request. This prevents our cache getting out of date with the file system. Closes #6893185 While we're at it, write the file atomically so that we don't risk serving partial data.

[test] Add missing u'' Test actually works in py3k, but not in python 2... That has to be a first!

[data] Use correct syntax to unset CSS attributes (closes #8602446) 'default' is actually a recognized CSS cursor name (the regular pointer). Using this value doesn't revert to whatever cursor DOM elements have by default ('pointer' on links, 'text' on text, etc). The proper way to unset a CSS attribute on DOM elements is to set it to an empty string. http://stackoverflow.com/questions/2027935/how-to-remove-css-property-using-javascript

[server] install custom sql scripts before creating tables for the schema A cube might need something like 'CREATE EXTENSION postgis' before creating tables for its entities, to add new final types.

[hooks] update internal URL in doc string

[doc] Fix restructured tables + improve english a bit

[debian] Install dataimport package Closes #8173104.

[web/test] Remove use of old test API to silence warning

[web/views] Fix sqlite unicode warning

[devtools] Fix sqlite unicode warning

[tests] Don't import QUnitTestCase into tests' global namespace Unittest's test loader will try to execute QUnitTestCase's test_javascripts method. However this class is intended to be subclassed, not to be executed directly. This change has the following consequences: * slightly faster test execution * almost all rtags warnings are gone

[devtools/qunit] Fail quickly if firefox is not installed

[devtools/qunit] Remove unused code VerboseCalledProcessError is unused since 3905682ece11.

[py3k] import range using six.moves Also, use create_entity instead of hand-crafting RQL queries. Fixes regression introduced in commit 768d332de17b.

[dataimport] remove size constraint handling We do not want to silently truncate data. If that is wanted, the application-specific importer should handle that itself. [v2: Keep the get_size_constraints function so applications don't have to reimplement it if they need to keep the truncate behaviour]

[dataimport] fix doc string to stop referring to the old API

[dataimport] drop duplicate init of MassiveObjectStore.sql

[dataimport] remove implicit commits from MassiveObjectStore Let the user decide when to commit the transaction.

[dataimport] drop extra indirection through MassiveObjectStore._initialized dict We already have self.__dict__, no need to add another one for a static set of keys.

[dataimport] remove remaining references to dataio from MassiveObjectStore In particular, rename metadata tables to live in the ^cw.* namespace.

[dataimport] rename logger from dataio.massiveimport to dataimport.massive_store

[dataimport] remove commit_at_flush parameter from MassiveObjectStore WARNING: the default value was True, but flush() now never commits.

[dataimport] remove autoflush_metadata from MassiveObjectStore parameters Hopefully not needed.

[dataimport] remove uri_param_name from MassiveObjectStore parameters It doesn't belong here.

[dataimport] turn iid_maxsize into a class attribute

[dataimport] turn eids_seq_{start,range} into class attributes Only the master-mode store will actually use eids_seq_start to reset the eid sequence.

[dataimport] remove replace_sep parameter from massive store It is unused since changeset efbbf1e93a04 "[dataimport] Properly escape strings sent to COPY FROM"

[dataimport] remove pg_schema parameter from massive store It can be recovered from the system source's configuration.

[dataimport] remove drop_index parameter from massive store "drop_index=False" also implied not dropping any constraints, in particular foreign keys, which meant any attempt to import entities would fail, because we only add metadata (the entities table) after the entity's insertion.

[dataimport/test] update call to ucsvreader Silences warning: [3.20] 'separator' kwarg is deprecated, use 'delimiter' instead

[dataimport] remove deprecated methods from massive store This code is new in cubicweb, no reason for it to carry deprecated stuff from the start.

[dataimport/test] use store.finish instead of deprecated store.cleanup

[dataimport] drop no more used parameter on massive store

[dataimport] methods that modify in-place shouldn't return value Also mark those methods private and drop some useless comments in passing. Related to #5414760

[dataimport] implement new store API on massive store and deprecate the old one. Related to #5414760

[dataimport] .finish method was missing from (NoHook)RQLObjectStore

[massive store] identity_relation should not be considered, is_relation seems missing The "identity_relation" table should not even exist.

[dataimport/test] use the right assert methods instead of assertTrue with a comparison

[dataimport] import range from six.moves

[dataimport/test] feed binary data to ucsvreader

[dataimport/test] fix broken comparison We want to compare two ints, not an int and a db row

[dataimport] dict.iteritems() → dict.items()

[dataimport] use next builtin instead of next method on iterators

[dataimport] import StringIO from io

[dataimport] backport massive store from dataio cube Some tweaks have been made to work with 3.21 foreign key constraints. Also test are not executed in our own pg cluster instead of polluting the system one. Closes #5414760

[web] set proper Cache-Control header for static files - set Cache-Control max-age to match Expires - use the max_age method (defaults to 1 week) instead of hardcoded 6 months

[web] use a redirect for /data/foo requests that don't use the instance md5 hash The actual file contents can be cached for a long time. If the request doesn't use the hash, that's wrong. Using a redirect lets keep the caching for the actual file contents while not breaking requests missing the hash.

[web] fix typo in doc string

[web] stop adding an Expires header with all responses As per RFC 1945 (HTTP 1.0): If the date given is equal to or earlier than the value of the Date header, the recipient must not cache the enclosed entity. As per RFC 7234 (HTTP 1.1 Caching): If a response includes a Cache-Control field with the max-age directive (Section 5.2.2.8), a recipient MUST ignore the Expires field. Bottom line, the Expires header is now handled wherever the Cache-Control is handled: inside cache managers.

[web] stop looking at a request's Cache-Control header As per RFC 7234 (HTTP 1.1 Caching): The "Cache-Control" header field is used to specify directives for caches along the request/response chain. CubicWeb is not a cache.

[fti] add cw_fti_index_rql_limit method Improves on and deprecate cw_fti_index_rql_queries: execute the rql directly, so that we don't need to compute the set of eids up-front, but can just keep track of the last seen entity. Use the new method in reindex_entities. Keep calling cw_fti_index_rql_queries if it has been redefined (e.g in cubes). Related to #3621392.

[fti] backout 166c6f7b1be4 * breaks tests * see next commit for a better solution

[py3k] Fix unicode added in commit 2350424585f8

[server/ldap] deal with unicode or str coming out of ldap It seems we can get either, I've no idea in which exact circumstances, so this should let us work with both.

[test] pull yapps from Logilab's github Fixes a couple of failures, pull request is almost a year old (https://github.com/smurfix/yapps/pull/2).

[pkg] bump yams requirement to 0.41.1 Needed by changeset c9321aedfbf9.

[entities] import range from six.moves

merge head published by accident

[fti] Improve big table reindexation * Slice the reindexing in batches of 1000 entities. * Make the output more verbose. Closes #3621392

[tox] Reference requirements.txt for hooks environment The file existed but was not referenced.

[devtools/testlib] Remove assertions about direct usage of CubicWebTC/AutomaticWebTest The point is to be able to use a test runner other than pytest (e.g. nose2, nosetests or py.test) to run CubicWeb test suite. The issue is that most other test loaders inspects members of all base classes of test case classes to determine test methods and it hits this `config` property (actually executing it), which breaks the assertion. So just return None instead of failing. Closes #7817147.

Add ldap3 to server test requirements

[pkg] bump version number so tox won't use cw from pypi

[schema] add CubicWebRelationDefinitionSchema Stop monkeypatching yams, inherit from RelationDefinitionSchema instead.

[schema] Let RQLConstraint inherit from BaseRQLConstraint instead of RQLVocabularyConstraint There is apparently no reason for RQLConstraint to inherit from RQLVocabularyConstraint as it gets its `repo_check` method from `RepoEnforcedRQLConstraintMixIn` and RQLVocabularyConstraint (which inherits from BaseRQLConstraint) actually only overrides that method. So simplify inheritance by removing a useless level of indirection. Adjust cw_unrelated_rql accordingly, where it was that RQLConstraint inherited from RQLVocabularyConstraint.

[server] unormalize source logger's name to avoid encoding errors

[entities] parameters should be given to fire_transition, else they are simply ignored

[entity] set _cw_adapters_cache in __init__ rather than on first use

fix typos and indentation in docstrings/docs

session.id → session.sessionid

[test] test page_size problem (related to #8123231)

[predicates] guard against non-integer page_size form parameter Closes #8123231

[server/test] add test for db-dump command

[server] fix db-dump with sqlite 'gzip -c dbname > backupfile' needs to go through the shell.

[serverctl] fix db-dump Since changeset af47954c1015 we would die with: AttributeError: 'ServerMigrationHelper' object has no attribute 'repo'

[server/test] sorting vs dict vs python3

[devtools] more sorted() vs dict vs python3

[web/application] return bytes from redirect_handler and validation_error_handler

[devtools] fix ResourceWarning: unclosed file tmpdb-__default_empty_db__.config

[serverconfig] fix ResourceWarning: unclosed file bootstrap_cubes

Use cubicweb._

[syncschema] hashlib.md5() wants bytes

[server/test] use next() builtin instead of next() method

[server/test] dict.iterkeys

[syncschema] don't arbitrarily remove constraints from the in-memory schema For some constraint types, it may make sense to have several constraints of that type simultaneously. In such a case, removing one of them from the in-memory schema is wrong. Closes #7446353

[server/test] don't leak timeout from one test class to the next Reset SQL_CONNECT_HOOKS after we're done with this test class so the next one doesn't inherit a timeout it doesn't want.

[server/test] CWAttribute.extra_props is Bytes

[dataimport] give unicode objects to psycopg2 copy_from

[dataimport] extid must be a bytes object

[dataimport] import StringIO from io

[dataimport] fix ucsvreader for python3 The csv module in python3 wants str, so use a StreamReader to feed csv.reader and don't decode the output

[server/test] import StringIO from io on python3 Don't change the import for python2, as we print a mix of unicode and str.

[server/datafeed] drop URLLibResponseAdapter.info I don't know where it's used

[devtools] pass a key to sort() method python3 dicts are not comparable

[postcreate] properly set is_instance_of for bootstrap entities Entities created before CWEType itself need the is/is_instance_of relations created later. We were actually only inserting "is". Closes #7656606.

[migration] re-try without print_function when compiling a script fails As part of the python3 porting we added the __future__.print_function import to cubicweb.migration. That started affecting all migration scripts it runs, which may not be prepared. So if compiling a script results in SyntaxError, try again without the print_function flag. Closes #7935030

[test] do not overwrite user's CW_CUBES_PATH in unittest_migration Currently when you rely on your CW_CUBES_PATH environment variable to locate cubes necessary for cubicweb's tests, the migration test crashes because it overrides the env var.

[server/test] dict.itervalues() → iter(dict.values())

make sync_schema_props_perms(<computed rtype>) work as expected It currently ends up with an ExecutionError while we want to synchronize permissions. This makes 3.21.1 migration of any application using computed relation crashing. Also, remove pre 3.6 compat code in hooks that prevent living schema update on computed relation's permission changes.

[web/http_headers] use classmethod decorator

[schema] use classmethod decorator

[devtools] delete postgresql socket directory after killing the server ... or when failing to start it.

[skel] stop deleting cubes egg-info It doesn't hurt and provides some metadata.

[dataimport] Update doc to reflect new API relate() looks deprecated.

[server] fix generate_source_config regression on python2 lgc outputs "str" objects on python2 and python3, so we can't just use io.StringIO (which expects unicode objects). Fixes: c36cda9074c5

[tests] Fix config.repository() in tests The config.repository was overriden too late. Closes #5778294

[schema] RQL queries are always unicode objects in python 3 fixes commit 42c5bd7286b7.

[server/test] use relative import Fixes py3k breakage

[server/schemaserial] fix extra_props on py3k json.dumps returns unicode, and json.load eats unicode, so add explicit encoding/decoding.

[server/test] unicode → six.text_type

[web] fix Field.__str__ on py3k

[web] unicode → six.text_type

[web/test] CubicWebTC.view returns bytes

[uilib] fix traceback on python3

[web] integer division

[web/test] Binary eats bytes

[devtools] make httptest module importable under py3k

[web/test] CubicWebPublisher.handle_request returns bytes

[web] base64 works on bytes not str (py3k)

[web] accept unicode in parsed Authorization header

[web] CubicWebPublisher.ajax_error_handler must return bytes

[py3k] fix i18n

Fix md5crypt and crypt_password test for python3 Make sure to always work with bytes for hashing.

[server/test] use bytes for extid

[devtools/test] firstnames.txt is in latin1

[web/test] unicode → cubicweb._

[web/formwidgets] fix for py3k no idea why we encode that key...

[sobjects] unicode → six.text_type

[server] fix unittest_security for py3k - use source.binary_to_str instead of str(), helps when the backend returns memoryviews - fixup str vs bytes

[py3k] make ldapfeed tests pass

[server] import filter from six.moves

[ldapfeed] port to ldap3 python-ldap has no python3 support. XXX I only checked that unittest_ldapfeed passes.

[server/test] str vs bytes

[server/test] list() around map and zip

[server/test] to fold, unicode → six.text_type

[server/test] delete test that doesn't make sense on python3 XXX if PY2?

[server/test] unicode → six.text_type

[server] unicode → six.text_type

[py3k] fix building rset description for INSERT queries

[datafeed] use explicit encode/decode instead of casting to str

[py3k] io.BytesIO

[py3k] get urllib foo from six.moves

[devtools] CubicWebTC._check_html may have to deal with unicode or bytes If called with template=None, CubicWebTC.view will not go through main-template, and return the unicode data from the view directly instead of encoded data.

[ext/test] fix test relying on repr(text)

[req] fix url_parse_qsl on py3 should be folded with previous conversion

[server/session] unicode → six.text_type

[web/test] str vs bytes

[web/ajax] fix AjaxController on python3

[web/test] str → bytes

[wsgi/test] use bytes instead of str for POST body

[devtools] 'dict_values' object does not support indexing

[multipart] decode form data before calling parse_qs in python 3 parse_qs eats and returns 'str' on both python versions.

[multipart] unicode → six.text_type

[web] assert that CubicWebPublisher.handle_request returns bytes

[py3k] fix validateform controller We need to return bytes not str.

[web] unicode → six.text_type

[multipart] make sure form keys are not bytes

[web] unicode → six.text_type

[wsgi] don't convert form values This is done earlier in multipart.py

[server] import input from six.moves

[server] don't convert user.login to bytes to generate sessionid

[cwctl] stop playing tricks with std{out,err} buffering Python3 doesn't allow unbuffered text I/O, because it doesn't make sense.

[py3k] CubicWebTC.app_handle_request returns bytes