Added tag 3.22.4, debian/3.22.4-1, centos/3.22.4-1 for changeset 92db0bb8e26d

[pkg] remove .dev0 from version

[pkg] 3.22.4

[session] Ensure access to rql varmaker always mark the session as dirty When one accesses the rql_varmaker, that's usually to use it. The pb was that when the varmaker was already in page's data (which is stored as session data), session storage such as redis won't see that the session data is dirty and has to be stored back at the end of the request. To fix this, systematically call set_page_data. (grafted from 3432f0e2540d)

[test] fix test_printable_value_bytes with current pygments An empty span was added in https://bitbucket.org/birkenfeld/pygments-main/commits/164574c13533195a555181a2b9c685fea2470403 (grafted from 74b04a88d28a)

[webctl] set uid of file produced by gen-static-datadir (closes #11298794)

[webconfig] ensure uicache content has correct uid (related to #11298794)

[cwconfig] add the CubicWebConfiguration.ensure_uid_directory() method (related to #11298794) that recursively enforce uid on files.

[cwconfig] extract uid file setting code from check_writeable_uid_directory (related to #11298794) into an ensure_uid() method

[web] Fix bug with usage of os.rename under windows environment In 7c386161ebd6 we removed cache handling from property sheet and introduced usage of a tempfile + os.rename to get atomic generation of files. The pb is that this is not portable, since under windows os.rename will raise an exception if the file already exists (because there is no way to write a file atomatically in such case). This kind of thing should be out of the CW scope anyway, so implements a quick & dirty fix in the mean time. Closes #14214794

[migration] More resilient migration script Query the information schema to retrieve existing constraint (primary key and unique) and drop them. Thus don't use anymore DROP CONSTRAINT IF EXISTS (not implemented by e.g. old versions of sqlserver). To do so a missing DISTINCT was necessary to avoid duplicates.

[session] Ensure access to rql varmaker always mark the session as dirty When one accesses the rql_varmaker, that's usually to use it. The pb was that when the varmaker was already in page's data (which is stored as session data), session storage such as redis won't see that the session data is dirty and has to be stored back at the end of the request. To fix this, systematically call set_page_data.

typo

[native] Use cnx.system_sql instead of doexec method in syntax_tree_search The latter handles connection issues and tries to reconnect. Method syntax_tree_search appears to be called from quite a few place where it seems useful to be resilient to connection issues (authentication for instance). This reconnection logic appears to have disappeared at some point when working on https://www.cubicweb.org/2919309. Could not find the exact reason though...

Remove usage of cnx.ensure_cnx_set context manager It has been deprecated since 3.21, no reason to keep using it internally.

[hooks/test/synschema] remove an unneeded skipTest. Related to #5557633.

[test] Check retcode of subprocesses in devctl tests And display decoded stdout in case of failure.

merge 3.23

[pkg] Bump version to 3.24.0.dev0

[pkg] Drop NO_SETUPTOOLS option setuptools is the defacto standard nowadays. The only place where NO_SETUPTOOLS option was used is during packaging (RPM, Debian) and even in this context, it seems that setuptools is the way to go (see https://wiki.debian.org/Python/LibraryStyleGuide).

[debian] update cubicweb-dev.lintian-overrides cubicweb/skeleton/debian/rules.tmpl is now cubicweb/skeleton/debian/rules prevent lintian from complaining

[pkg] Properly export data files in setup.py and adjust "newcube" test With the new package layout (everything under "cubicweb" package), the custom install_lib rule which makes use of include_dirs defined in __pkginfo__.py did not prepend the package name to source directories to be copied. Fixing this. Also, in setup.py's export() function, the destination directories' path to be created during source tree walk was wrong. All this makes cubicweb/skeleton directory (which is not a package) properly installed by setup.py. The test in cubicweb/devtools/test/unittest_devctl.py wasn't properly implemented because it used an installation of cubicweb in "develop" mode which shadows such packaging issues. Also it used "python -m cubicweb" instead of directly "cubicweb-ctl" and the former appears to fall back to using the cubicweb package *from sources* instead of the installed one. Now that this test runs against the installed version of cubicweb, fix MANIFEST.in to include tox.ini files (cubicweb's and skeleton's) as this is expected from the test. Closes #14127941.

[pkg] Handle verbose option in setup.py's export function Make debugging easier.

[pkg] Drop unused EMPTY_FILE global variable

[tox] Do not install package in "develop" This might shadow so packaging issues. And user can still pass --develop to tox.

[pkg] Bump version to 3.23.2

pep8 bits

[schema] Add a method on yams constraints to compute its unique name used to identify it in the DB backend.

[migration] Test and fix case of addition of an attribute with some serialized constraint It was crashing on attempt to remove an unexisting constraint. This code may be removed safely as it's expected to be handled by the constraint removal hook. Notice that adding a vocabulary on shortpara makes yams change the varchar max size. This is arguable but not the point of this test, so simply adapt the max size defined in the schema coherently so we don't have to bother with that at this point.

[test] Add a migration test for update of a static vocabulary constraint This is a constraint serialized to be checked by the DB backend, hence ensure the old constraint is dropped and the new one is created.

[schema2sql] Avoid "parsing" SQL statements for database initialization A big SQL string was generated, then splitted. This caused bug if some value in the schema (eg vocabulary, default) contained the separator (';'). To properly fix this, yield each individual statement instead of generating a string. Closes #14050899

[schema2sql] Drop most of the DB DROP related code This code is dead for a while, it has been superseded by `sql_drop_all_user_tables`. Related to #14050899

[hooks] Use already imported module alias

[schema2sql] Rename as_sql to constraint_value_as_sql which is easier to grasp independantly of the context.

[schema2sql] Drop unused indent argument on aschema2sql Related to #14050899

[schema2sql] More docstrings and minor API improvements Along the way, renamed local `column` variable to `attr` where we're NOT manipulating a DB column name but a schema attribute. Related to #14050899

[schema2sql] inline eschema_attrs method * not a public API * doesn't makes things easier to grasp -> useless Related to #14050899

[schema2sql] Give a rdef to check_constraint This simplifies the API. Related to #14050899

merge 3.20.16 in 3.22

merge 3.20.16 in 3.21

Added tag 3.20.16, debian/3.20.16-1, centos/3.20.16-1 for changeset e60a8e5d29ef

[debian] add a lintian override for source-is-missing about jquery.flot.js

[pkg] 3.20.16

[dataimport] fix count_lines function (closes #14407650) ensure i allways exists whatever f content is

[captcha] improve a bit the redability of generated captcha images (closes #13500378)

3.20 branch

merge 3.23 changes

Added tag 3.23.1, centos/3.23.1-1, debian/3.23.1-1 for changeset 8f035d5d5953

prepare 3.23.1

[debian] Adjust symlink to cubicweb package so that documention builds fine Broken since source tree re-organization.

[dataimport] Restore handling of Binary in ExtEntity's values Broken since 62a7100d774b. Closes #13973741.

[tox] Ensure virtualenv's python is used to build the documentation

[doc] Reference 3.23 page in changelog

[tox] Ensure virtualenv's python is used to build the documentation

Backed out changeset b288debc6736 Only superusers can set lc_messages variable. This prevents connection in many cases. Closes #14010345.

Merge with 3.23

[pkg] Bump version to 3.22.4.dev0

Added tag 3.23.0, centos/3.23.0-1, debian/3.23.0-1 for changeset 4b07f2a79021

[doc] Fix symlinks of sphinx theme images

Added tag 3.23.0, centos/3.23.0-1, debian/3.23.0-1 for changeset 93b0f836cb07

[debian] Move python-unittest2 >= 0.7.0 as runtime dependency It seems to me that this is not used during Debian package build (tests are not run), but should be needed at runtime with the cubicweb-dev package. Also, at the moment, we have no pbuilder setting with this package available and it blocks the release.

Symlink __pkginfo__.py at root Maybe this will make apycot work again...

[3.23] More release notes

i18n update

[doc/changes] CubicWebTC.set_description no longer available

Add some release notes

[pkg] Version 3.23.0

[pkg] Bump yams version in cubicweb-common Debian package

[tox] Drop yams from dev-requirements.txt Yams 0.44.0 got released.

Merge with 3.22

Merge with 3.21

backport 3.21 changes

backport 3.20 changes

[sobjects] Drop custom unique constraint checks in user registration service This constraints (on unique CWUser's login and EmailAddress's login) are handled by the backend. This follows "[hooks/integrity] kill CheckUniqueHook". Extend test for the ValidationError along the way.

[test] Add a test for user registration unicity constraint on email

[test] Clarify user registration service tests * split permissions test appart * avoid usage of internal_cnx as this does not seem need

[test/hooks] pep8 and unittest.main instead in unittest_hooks.py

[hooks/integrity] kill CheckUniqueHook It's the backend's job. The noticeable consequence (as can be seen in tests updated here) is that error message are a bit less user-friendly.

[migration] drop identity_relation if it exists It has been generated erroneously for a while, then fixed but not properly deleted from old databases. Don't use IF EXISTS for sqlserver compat.

[migration] Reserialize all constraints since yams introduce new JSON serializing format So we may drop support for older format in yams at some point.

[migration] Add IF EXISTS on DROP CONSTRAINT problem encountered on migration of a legacy database

[schema sync] Rename index when an entity type is renamed So we may still find them later using their expected name, and avoid collisions. Closes #13822045

[c-c] Add a command to check database index List indices expected by the schema but not in the database and vice-versa Closes #13822027

[sql gen] Control size of index name using an md5 hash When using concatenation of table and column, we may hit the backend size limit on index names, which results on truncated name (64 for postgres). Then we may have conflict or be unable to find expected index on later synchronization attempt. This is done by monkeypatching lgdb for now to avoid dependancy problems. Unique and primary key indices of entity types handling is left to the backend, as they are not explicitly named. See https://www.logilab.org/ticket/4789189 Closes #13821949

[sql gen] Explicitly name unique index This patch hotfix logilab database for https://www.logilab.org/6662663 and fix database creation to stop using UNIQUE in the table schema but rather create the index later on, so its name is under control. Also consider this change in schema synchronization hooks. This is preliminary work related to #13822045

Update to yams 0.44 API

[test] Fix randomness in tests

[test] Drop unused import

[serverctl] allow to sync multiple and all sources in source-sync command Breaking change: by default source are not synced anymore if it is fresh (regards to synchronization-interval). To get the previous behavior an option --force is added. Closes #13886467

[tox] Split migractions tests from server environment These are the longest tests in this environment, so running them apart (in parallel of others) would hopefully speed up the whole suite.

[dataimport] Fix rdef loopup in importer

[py3k] left-over unicode() in authobjs.py

[server] Improve massive deletion of entities sqlserver is limited on the "IN" array size and produce an error "The query processor ran out of internal resources and could not produce a query plan". The exact limitation is unclear but can occur with > 10000 item in the array. See https://support.microsoft.com/en-us/kb/288095 Now we process the deletion by batch of 10000, some quick benchmarks show that there is no performance impact: delete_in(50000): 46.765182972 delete_in_batch(50000): 46.3242430687 delete_in(100000): 62.2969429493 delete_in_batch(100000): 61.8402349949

[hooks] Add a bunch of tests for session synchronization And change unicode into six.text_type in authobjs.py spotted by new tests.

[entities] Drop very old backward compat from erudi time (even the standard datetime type didn't exist at that time)

pep8 + docstrings and comments improvments

[server] some pep8 in rql2sql

[dataimport] Deprecate SQLGenObjectStore Closes #10338709.

[datafeed] attempt to acquire synchronization lock even when force is given instead of the implementation in e717da3dc164, raise an error if the lock is already grabbed and catch this error in the caller. See discussion on https://www.cubicweb.org/revision/10790765 Closes #10451635

[schema] Cache RQL expressions on schema loading Add a mapping rql string to rql statement Deserializing schema is ~36% faster with a lot (> 4000) expressions like those generated by cubicweb-container.

[schema] Avoid parsing RQL expressions twice on schema loading Kill full_rql (unnecessary) Use snippet_rql rather than rqlst when possible Don't access to rqlst cachedproperty from __init__ Deserializing schema is ~40% faster with a lot (> 4000) expressions like those generated by cubicweb-container.

[dataimport] Handle Bytes data in ExtEntity and convert them to Binary Closes #13887276.

[test] Avoid assertCountEqual in unittest_syncschema.py It's not clear why this happens, but it seems that assertCountEqual does strange things with unittest2 on Python 2 for equality comparison (trying to sort constraint objects, even though BaseConstraint raise NotImplemented in __lt__ from Yams 0.43). So just avoid using it to get this test passing.

[packaging] Bump version number

[web/session] make sure not to modify an inner value of session.data (closes #11217958) When we modify a mutable value stored in the session's data (dict-like object) we prevent an alternative session data handler (eg. pyramid-redis-session) from detecting this update and synchronizing it in its backend (like redis).

[book] improve a bit the documentation of the database setup also remove the MySQL section.

[doc] add documentation for specializing translation of other cubes in an application cube

[doc] fix some URLs (cs to test jenkins autobuild )

[uicfg] Fix typo in set_fields_order

[forms] We don't want cwjs-edition-cancel class on reledit's cancel button Else it's bound to a js function that collides with onclick provided by reledit code. Closes #13120795

[test] Prefer unittest.main

[test] py.test related fix We ought not import a concrete test case from a sibling module, else it ends up partially initialized at some point and later raise assertion error in setUpClass.

[dataimport] add a bit of extra-documentation on the massive store about prepare_insert_relation limitation on inlined relation and lack of full-text indexation.

[dataimport] no need to call .keys()

[dataimport] drop massive store's flush_metadata method This method handle a temporary table which sounds useless, and even buggy in some cases (metadata for a single entity types will be only flushed once even if flush is called several time). Instead, simply call the method-doing-the job after entities have been inserted (and make it private along the way).

[dataimport] test and fix external source support for the massive store

[dataimport] add new test for nohook store importing in a custom source

[dataimport] introduce usage of MetadataGenerator into the massive store This allows more flexibility and code sharing w/ nohook store. Closes #10338621

[dataimport] rename massive store's metagen_push_relation method to _insert_meta_relation.

[dataimport] meta_data -> metadata

[dataimport] use sql shortcut

[dataimport] move cwmassive_constraint temporary table handling to the PGHelper class with consistent renaming and some rationalization to avoid doing things several times and have code slightly easier to grasp.

[dataimport] inline some methods of the massive store

[dataimport] move everything related to the "URI handling" feature of the massive store together

[dataimport] test for a value is in a set and insertion in a set should live together also inline some methods on the way, in the hope to make the public api clearer at some point, and only put valuable values in those sets so we don't have to endlessly test for consistency.

[dataimport] use a single _initialized set instead of _entities / _rtypes and rework a bit methods using them so that 'in' test and set insertion leave in the same method.

[dataimport] drop massive store etype to eid index and fix potential problem with is_instance_of relation insertion in case where the class of inserted entity has some (yams) parent.

[dataimport] add a .schema shortcut attribute on the massive store also rephrase a bit a method along the way

[dataimport] rework massive store's __init__ * _cnx is set by the parent class * count_cwuri is not used * group related things together

[dataimport] rework PGHelper class

[dataimport] flake8

[dataimport] enhance a bit sql queries readability

[dataimport] use IF EXISTS when possible it's better than try/except and avoid potential concurrency problems.

[dataimport] PGHelper should be responsible to retrieve the database schema

[dataimport] introduce a new MetadataGenerator to replace former MetaGenerator with a similar-but-different API that will allow its usage by the massive store (no more entity object manipulation), while making it a bit more flexible for customization by subclasses (experience taken from client code). Related to #10338621

[dataimport] Prefer now(pytz.utc) to utcnow

[dataimport] rename nohook store .source attribute so it's clearly not the samething as metagenerator.source. Also add a bit of doc along the way.

[dataimport] make rql store test easier to read and test a few more thing, then reuse it to test nohook start which had no tests at all

[dataimport] remove useless assignment in massive store

[dataimport] move eid attribution from the metagenerator back to the nohook store

[dataimport] rephrase minor bits of the meta generator

[pkg] Depends on logilab-common >= 1.2.2

Merge with 3.22.3

[pkg] Depends on yams >= 0.43 RPM spec file and debian/control got updated previously.

Added tag 3.22.3, debian/3.22.3-1, centos/3.22.3-1 for changeset b1e7de000536

[pkg] Version 3.22.3

[pkg] Depends on yams < 0.43

Define BASE_URL and DATA_URL js variables in the login template. Closes #13385185 Those are expected to be defined in almost every js files. They were not before this patch because login form doesn't use the regular main template.

[web] Remove some unused imports in editforms

Fix cancel action on deletion confirmation form Currently crashed with RequestError: invalid action '__action_cancel'. It's broken as of https://hg.logilab.org/master/cubicweb/rev/cb217b2b3 This fix is similar as the one done in https://hg.logilab.org/master/cubicweb/rev/9f1c89e7426d Closes #13521493

[rql2sql] Test and fix some case of sql generation when an unstable variable (ie whose type change from one solution to another) is used in an EXISTS node but belongs to an outer scope. Closes #13518646.

[test] Add missing os import in unittest_migractions.py Dropped in ^ merge.

backport 3.22 changes

[entity] Make it possible to specify sorting terms in cw_related_rqlst method Closes #12306543.

[sql gen] Properly create UNIQUE index on non string column. Closes #13728492 Unique was previously detected in a loop triggered only for String columns.

[server] some pep8 changes

pep8 schema2sql

[migration] Drop custom "readfunc" for code.interact in shell As the referenced bug http://bugs.python.org/issue1288615 got fixed in Python 2.6, this is no longer needed. Furthermore the implementation is not Python 3 compatible (raw_input, unicode).

[tox] Put development dependencies into a dev-requirements.txt file The latter being sourced in tox.ini. For now this contains references to hg repositories for some dependencies but under normal circumstances it should only contain pytest. This is useful when developping cubicweb itself without using tox but working with a virtualenv so that one can simply do: :: $ virtualenv --system-site-packages venv $ . venv/bin/activate $ pip install -r dev-requirements.txt $ python setup.py develop

[py3k] Fix config-level comparison in "cubicweb-ctl create" command dict.get -> None, and None cannot compare to an int in Python 3.

[pkg] PEP8 fixes in __pkginfo__.py

[pkg] Drop "support" of unused subpackage_of option in setup.py __pkginfo__ does not declare subpackage_of.

Add missing import, eaten by previous merge

Only access once to _cw.message in the login form else the first access may pop the message and the second won't get anything to display. This works on default message implementation but doesn't on the pyramid's one, leading to no message displayed on erroneous authentication attempt, which is not very user friendly.

backport 3.22 changes

backport changes from 3.21

[server,bfss] fix bfss migration test for python 3 (related #13519541)

[pkg] Use io.open with utf-8 encoding to read long description from README This forces decoding so that this is consistent between Python 2 and 3 as, in Python 3, decoding which would occur anyways and may fail depending on the environment locale settings.

[dataimport] order of ExtEntities should be irrelevant (closes #13117472)

Test and fix "pickleability" of Binary objects which has been recently broken while some cubes rely on this (eg fastimport). Do some licensing/pep8 cleanup along the way. Closes #13385274

[server,bfss] fix storage migration when Bytes attributes is None (closes #13519541) When a Bytes attributes is None before the BFSS storage migration, the migration crashes. A Bytes attribute can be None, when the attribute is not required and there is no default value. In this case, the attributes should remained None after migration and no file is stored on the disk.

Let hg Ignore py.test's .cache directory And drop rule for .svn along the way.

[entity] Add a cw_related_rqlst method returning the RQL select node of cw_related method This is useful if one wants to modify the query before execution. A bit a PEP8 style fix in tests along the way. Related to #12306543.

[devtools] set .config in setUpClass instead of using a cache in a classproperty This is much clearer and avoid problem with py.test introspection that triggered creation and caching of the configuration before actual test execution. This caused pb in unittest_postgres.PostgresStatementTimeoutTC where configuration was built before module's setup, hence source configuration was missing information about where to find the pg cluster.

[devtools] Simplify test configuration's init Specify module's __file__ instead of apphome. __file__ is saved on the class as it will be used later (eg to find the proper pg cluster without relying on shared dictionnary modifications). This will require some tests update, but it should be less common in cubes than in cubicweb itself.

[web] Have the LoginController return 403 on failed auth (closes #11569950)

[devtools] Drop old init_test_database backward compat

[devtools] show DeprecationWarning during test execution They are not shown by default since python 2.7

[devtools] deprecate usage of sources file for test This is old support which clutters the code and should no more be used. As this is rarely used (if it is) and test only, it feels ok to drop support at once without a deprecation phase.

[devtools] Set dumb value for postgres's db-host to avoid unexpected connection to a local server Currently, if something goes wrong (eg config built before call to startpgcluster) while a pg server is running on localhost, it may connect to this server and run the test in there unexepectedly.

[ldapfeed] ldap3 do not raise an exception in case of failure of cnx.bind() but return 'False' instead.

[test] add a test to ensure we get an AuthenticationError with ldapfeed when using (correct login, wrong password) credentials

[entity] Exclude computed relations from Entity.copy_relations Closes #12481591.

[pkg] Set version to 3.22.2.dev0 So that cubes used in test dependencies do not install a released CubicWeb.

[devtools] pep8 a bit package's __init__

[test/server] Only use configcls with statement timeout when needed Other tests in PostgresFTITC class apparently do not make use of this particular configuration.

[server/test] Add an assertion in test_statement_timeout

[datafeed] minor cleanups to cwxml parser

[datafeed] don't allow arbitrary kwargs on DatafeedParser initializer else they will be handled nicely by appobject base class, while we rather want an error in the case of a source parser (arguable).

[datafeed] gives information about moved entities to the parser Also, don't support anymore arbitrary parameters at parser initialization. Related to #5414754

[datafeed] simplify SQL query used to retrieve information about entities from an external source no need to join on cw_source

[datafeed] rename parser.sourceuris to source_uris Its more readable. Also, uniformize related method and local identifier names.