Denis Laxalde <denis.laxalde@logilab.fr> [Thu, 23 Jun 2016 11:05:31 +0200] rev 11367
[test/hooks] pep8 and unittest.main instead in unittest_hooks.py
Julien Cristau <julien.cristau@logilab.fr> [Mon, 13 Apr 2015 14:34:11 +0200] rev 11366
[hooks/integrity] kill CheckUniqueHook
It's the backend's job.
The noticeable consequence (as can be seen in tests updated here) is that
error message are a bit less user-friendly.
Sylvain Thénault <sylvain.thenault@logilab.fr> [Mon, 20 Jun 2016 15:10:24 +0200] rev 11365
[migration] drop identity_relation if it exists
It has been generated erroneously for a while, then fixed but not properly deleted from old
databases.
Don't use IF EXISTS for sqlserver compat.
Sylvain Thénault <sylvain.thenault@logilab.fr> [Wed, 22 Jun 2016 14:35:43 +0200] rev 11364
[migration] Reserialize all constraints since yams introduce new JSON serializing format
So we may drop support for older format in yams at some point.
Sylvain Thénault <sylvain.thenault@logilab.fr> [Thu, 23 Jun 2016 15:45:30 +0200] rev 11363
[migration] Add IF EXISTS on DROP CONSTRAINT
problem encountered on migration of a legacy database
Sylvain Thénault <sylvain.thenault@logilab.fr> [Mon, 20 Jun 2016 18:00:00 +0200] rev 11362
[schema sync] Rename index when an entity type is renamed
So we may still find them later using their expected name, and avoid collisions.
Closes #13822045
Sylvain Thénault <sylvain.thenault@logilab.fr> [Fri, 17 Jun 2016 13:26:13 +0200] rev 11361
[c-c] Add a command to check database index
List indices expected by the schema but not in the database and vice-versa
Closes #13822027
Sylvain Thénault <sylvain.thenault@logilab.fr> [Wed, 22 Jun 2016 07:57:13 +0200] rev 11360
[sql gen] Control size of index name using an md5 hash
When using concatenation of table and column, we may hit the backend size limit
on index names, which results on truncated name (64 for postgres). Then we may
have conflict or be unable to find expected index on later synchronization
attempt.
This is done by monkeypatching lgdb for now to avoid dependancy problems.
Unique and primary key indices of entity types handling is left to the backend,
as they are not explicitly named.
See https://www.logilab.org/ticket/4789189
Closes #13821949
Sylvain Thénault <sylvain.thenault@logilab.fr> [Mon, 20 Jun 2016 17:59:43 +0200] rev 11359
[sql gen] Explicitly name unique index
This patch hotfix logilab database for https://www.logilab.org/6662663 and fix
database creation to stop using UNIQUE in the table schema but rather create the
index later on, so its name is under control.
Also consider this change in schema synchronization hooks.
This is preliminary work related to #13822045
Sylvain Thénault <sylvain.thenault@logilab.fr> [Wed, 22 Jun 2016 17:13:10 +0200] rev 11358
Update to yams 0.44 API
Sylvain Thénault <sylvain.thenault@logilab.fr> [Mon, 20 Jun 2016 17:56:49 +0200] rev 11357
[test] Fix randomness in tests
Sylvain Thénault <sylvain.thenault@logilab.fr> [Wed, 22 Jun 2016 07:56:51 +0200] rev 11356
[test] Drop unused import
Philippe Pepiot <philippe.pepiot@logilab.fr> [Wed, 22 Jun 2016 11:38:53 +0200] rev 11355
[serverctl] allow to sync multiple and all sources in source-sync command
Breaking change: by default source are not synced anymore if it is fresh
(regards to synchronization-interval). To get the previous behavior an option
--force is added.
Closes #13886467
Denis Laxalde <denis.laxalde@logilab.fr> [Wed, 22 Jun 2016 15:57:17 +0200] rev 11354
[tox] Split migractions tests from server environment
These are the longest tests in this environment, so running them apart (in
parallel of others) would hopefully speed up the whole suite.
Sylvain Thénault <sylvain.thenault@logilab.fr> [Thu, 23 Jun 2016 10:24:11 +0200] rev 11353
[dataimport] Fix rdef loopup in importer
Denis Laxalde <denis.laxalde@logilab.fr> [Wed, 22 Jun 2016 16:23:11 +0200] rev 11352
[py3k] left-over unicode() in authobjs.py
Philippe Pepiot <philippe.pepiot@logilab.fr> [Tue, 10 May 2016 16:09:34 +0200] rev 11351
[server] Improve massive deletion of entities
sqlserver is limited on the "IN" array size and produce an error "The query
processor ran out of internal resources and could not produce a query plan".
The exact limitation is unclear but can occur with > 10000 item in the array.
See https://support.microsoft.com/en-us/kb/288095
Now we process the deletion by batch of 10000, some quick benchmarks show that
there is no performance impact:
delete_in(50000): 46.765182972
delete_in_batch(50000): 46.3242430687
delete_in(100000): 62.2969429493
delete_in_batch(100000): 61.8402349949
Sylvain Thénault <sylvain.thenault@logilab.fr> [Mon, 06 Jun 2016 15:18:12 +0200] rev 11350
[hooks] Add a bunch of tests for session synchronization
And change unicode into six.text_type in authobjs.py spotted by new tests.
Sylvain Thénault <sylvain.thenault@logilab.fr> [Mon, 06 Jun 2016 15:17:23 +0200] rev 11349
[entities] Drop very old backward compat
from erudi time (even the standard datetime type didn't exist at that time)
Sylvain Thénault <sylvain.thenault@logilab.fr> [Mon, 06 Jun 2016 15:28:35 +0200] rev 11348
pep8 + docstrings and comments improvments
Sylvain Thénault <sylvain.thenault@logilab.fr> [Mon, 06 Jun 2016 21:17:33 +0200] rev 11347
[server] some pep8 in rql2sql
Denis Laxalde <denis.laxalde@logilab.fr> [Tue, 21 Jun 2016 17:51:11 +0200] rev 11346
[dataimport] Deprecate SQLGenObjectStore
Closes #10338709.
Sylvain Thénault <sylvain.thenault@logilab.fr> [Thu, 18 Feb 2016 17:51:12 +0100] rev 11345
[datafeed] attempt to acquire synchronization lock even when force is given
instead of the implementation in e717da3dc164, raise an error if the lock is
already grabbed and catch this error in the caller.
See discussion on https://www.cubicweb.org/revision/10790765
Closes #10451635
Philippe Pepiot <philippe.pepiot@logilab.fr> [Mon, 30 May 2016 17:41:12 +0200] rev 11344
[schema] Cache RQL expressions on schema loading
Add a mapping rql string to rql statement
Deserializing schema is ~36% faster with a lot (> 4000) expressions like those
generated by cubicweb-container.
Philippe Pepiot <philippe.pepiot@logilab.fr> [Tue, 07 Jun 2016 12:04:21 +0200] rev 11343
[schema] Avoid parsing RQL expressions twice on schema loading
Kill full_rql (unnecessary)
Use snippet_rql rather than rqlst when possible
Don't access to rqlst cachedproperty from __init__
Deserializing schema is ~40% faster with a lot (> 4000) expressions like those
generated by cubicweb-container.
Denis Laxalde <denis.laxalde@logilab.fr> [Wed, 22 Jun 2016 10:22:37 +0200] rev 11342
[dataimport] Handle Bytes data in ExtEntity and convert them to Binary
Closes #13887276.
Denis Laxalde <denis.laxalde@logilab.fr> [Tue, 21 Jun 2016 18:11:40 +0200] rev 11341
[test] Avoid assertCountEqual in unittest_syncschema.py
It's not clear why this happens, but it seems that assertCountEqual does
strange things with unittest2 on Python 2 for equality comparison (trying to
sort constraint objects, even though BaseConstraint raise NotImplemented in
__lt__ from Yams 0.43).
So just avoid using it to get this test passing.
Sylvain Thénault <sylvain.thenault@logilab.fr> [Wed, 22 Jun 2016 11:01:16 +0200] rev 11340
[packaging] Bump version number
David Douard <david.douard@logilab.fr> [Mon, 29 Feb 2016 17:07:56 +0100] rev 11339
[web/session] make sure not to modify an inner value of session.data (closes #11217958)
When we modify a mutable value stored in the session's data (dict-like object)
we prevent an alternative session data handler (eg. pyramid-redis-session)
from detecting this update and synchronizing it in its backend (like redis).
David Douard <david.douard@logilab.fr> [Fri, 19 Feb 2016 12:11:31 +0100] rev 11338
[book] improve a bit the documentation of the database setup
also remove the MySQL section.
Alexandre Richardson <alexandre.richardson@logilab.fr> [Tue, 22 Dec 2015 11:26:49 +0100] rev 11337
[doc] add documentation for specializing translation of other cubes in an application cube
Arthur Lutz <arthur.lutz@logilab.fr> [Fri, 21 Aug 2015 11:25:50 +0200] rev 11336
[doc] fix some URLs (cs to test jenkins autobuild )
Sylvain Thénault <sylvain.thenault@logilab.fr> [Thu, 26 May 2016 15:37:28 +0200] rev 11335
[uicfg] Fix typo in set_fields_order
Sylvain Thénault <sylvain.thenault@logilab.fr> [Mon, 23 May 2016 17:40:31 +0200] rev 11334
[forms] We don't want cwjs-edition-cancel class on reledit's cancel button
Else it's bound to a js function that collides with onclick provided by reledit
code.
Closes #13120795
Sylvain Thénault <sylvain.thenault@logilab.fr> [Tue, 21 Jun 2016 13:49:32 +0200] rev 11333
[test] Prefer unittest.main
Sylvain Thénault <sylvain.thenault@logilab.fr> [Tue, 21 Jun 2016 13:49:15 +0200] rev 11332
[test] py.test related fix
We ought not import a concrete test case from a sibling module, else it ends up
partially initialized at some point and later raise assertion error in
setUpClass.
Sylvain Thénault <sylvain.thenault@logilab.fr> [Tue, 02 Feb 2016 12:08:49 +0100] rev 11331
[dataimport] add a bit of extra-documentation on the massive store
about prepare_insert_relation limitation on inlined relation and lack of
full-text indexation.
Sylvain Thénault <sylvain.thenault@logilab.fr> [Fri, 29 Jan 2016 18:57:30 +0100] rev 11330
[dataimport] no need to call .keys()
Sylvain Thénault <sylvain.thenault@logilab.fr> [Mon, 01 Feb 2016 17:50:05 +0100] rev 11329
[dataimport] drop massive store's flush_metadata method
This method handle a temporary table which sounds useless, and even buggy in
some cases (metadata for a single entity types will be only flushed once even if
flush is called several time). Instead, simply call the method-doing-the job
after entities have been inserted (and make it private along the way).
Sylvain Thénault <sylvain.thenault@logilab.fr> [Tue, 02 Feb 2016 12:10:42 +0100] rev 11328
[dataimport] test and fix external source support for the massive store
Sylvain Thénault <sylvain.thenault@logilab.fr> [Tue, 21 Jun 2016 13:51:19 +0200] rev 11327
[dataimport] add new test for nohook store importing in a custom source
Sylvain Thénault <sylvain.thenault@logilab.fr> [Mon, 01 Feb 2016 17:29:59 +0100] rev 11326
[dataimport] introduce usage of MetadataGenerator into the massive store
This allows more flexibility and code sharing w/ nohook store.
Closes #10338621
Sylvain Thénault <sylvain.thenault@logilab.fr> [Thu, 28 Jan 2016 18:14:27 +0100] rev 11325
[dataimport] rename massive store's metagen_push_relation method
to _insert_meta_relation.
Sylvain Thénault <sylvain.thenault@logilab.fr> [Thu, 28 Jan 2016 18:14:55 +0100] rev 11324
[dataimport] meta_data -> metadata
Sylvain Thénault <sylvain.thenault@logilab.fr> [Thu, 28 Jan 2016 16:08:08 +0100] rev 11323
[dataimport] use sql shortcut
Sylvain Thénault <sylvain.thenault@logilab.fr> [Thu, 28 Jan 2016 18:16:47 +0100] rev 11322
[dataimport] move cwmassive_constraint temporary table handling to the PGHelper class
with consistent renaming and some rationalization to avoid doing things several
times and have code slightly easier to grasp.
Sylvain Thénault <sylvain.thenault@logilab.fr> [Thu, 28 Jan 2016 18:17:08 +0100] rev 11321
[dataimport] inline some methods of the massive store
Sylvain Thénault <sylvain.thenault@logilab.fr> [Thu, 28 Jan 2016 15:25:32 +0100] rev 11320
[dataimport] move everything related to the "URI handling" feature of the massive store together
Sylvain Thénault <sylvain.thenault@logilab.fr> [Thu, 28 Jan 2016 15:13:39 +0100] rev 11319
[dataimport] test for a value is in a set and insertion in a set should live together
also inline some methods on the way, in the hope to make the public api clearer
at some point, and only put valuable values in those sets so we don't have to
endlessly test for consistency.
Sylvain Thénault <sylvain.thenault@logilab.fr> [Thu, 28 Jan 2016 15:08:16 +0100] rev 11318
[dataimport] use a single _initialized set instead of _entities / _rtypes
and rework a bit methods using them so that 'in' test and set insertion leave in
the same method.
Sylvain Thénault <sylvain.thenault@logilab.fr> [Thu, 28 Jan 2016 14:34:47 +0100] rev 11317
[dataimport] drop massive store etype to eid index
and fix potential problem with is_instance_of relation insertion in case where
the class of inserted entity has some (yams) parent.
Sylvain Thénault <sylvain.thenault@logilab.fr> [Thu, 28 Jan 2016 14:00:50 +0100] rev 11316
[dataimport] add a .schema shortcut attribute on the massive store
also rephrase a bit a method along the way
Sylvain Thénault <sylvain.thenault@logilab.fr> [Thu, 28 Jan 2016 13:41:39 +0100] rev 11315
[dataimport] rework massive store's __init__
* _cnx is set by the parent class
* count_cwuri is not used
* group related things together
Sylvain Thénault <sylvain.thenault@logilab.fr> [Thu, 28 Jan 2016 18:18:20 +0100] rev 11314
[dataimport] rework PGHelper class
Sylvain Thénault <sylvain.thenault@logilab.fr> [Thu, 28 Jan 2016 18:18:44 +0100] rev 11313
[dataimport] flake8
Sylvain Thénault <sylvain.thenault@logilab.fr> [Thu, 28 Jan 2016 14:02:31 +0100] rev 11312
[dataimport] enhance a bit sql queries readability
Sylvain Thénault <sylvain.thenault@logilab.fr> [Thu, 28 Jan 2016 14:14:57 +0100] rev 11311
[dataimport] use IF EXISTS when possible
it's better than try/except and avoid potential concurrency problems.
Sylvain Thénault <sylvain.thenault@logilab.fr> [Thu, 28 Jan 2016 11:32:12 +0100] rev 11310
[dataimport] PGHelper should be responsible to retrieve the database schema
Sylvain Thénault <sylvain.thenault@logilab.fr> [Mon, 01 Feb 2016 17:31:35 +0100] rev 11309
[dataimport] introduce a new MetadataGenerator to replace former MetaGenerator
with a similar-but-different API that will allow its usage by the massive store
(no more entity object manipulation), while making it a bit more flexible for
customization by subclasses (experience taken from client code).
Related to #10338621
Sylvain Thénault <sylvain.thenault@logilab.fr> [Tue, 21 Jun 2016 16:37:17 +0200] rev 11308
[dataimport] Prefer now(pytz.utc) to utcnow