Sylvain Thénault <sylvain.thenault@logilab.fr> [Tue, 11 Oct 2016 10:24:13 +0200] rev 11788
[massive store] Drop useless check for empty buffer
If something went wrong, an error will be raised.
Sylvain Thénault <sylvain.thenault@logilab.fr> [Tue, 11 Oct 2016 10:23:44 +0200] rev 11787
[massive store] Docstring / comment cleanups
Sylvain Thénault <sylvain.thenault@logilab.fr> [Tue, 11 Oct 2016 10:23:06 +0200] rev 11786
[massive store] Turn a runtime error into an assertion
geting there is a bug.
Sylvain Thénault <sylvain.thenault@logilab.fr> [Wed, 05 Oct 2016 09:27:25 +0200] rev 11785
[massive store] Delay metadata attributes and default values handling to flush_entities
This may be done once for all there, instead of for each entity in
prepare_insert_entity.
Sylvain Thénault <sylvain.thenault@logilab.fr> [Wed, 05 Oct 2016 09:38:05 +0200] rev 11784
[massive store] Use a slave specific table for relation insertion in the massive store
* attribute a random id to the store
* add a column containing this id in the cwmassive_initialized table
* separate generic `master_init` which create the cwmassive_initialized table
from rtype specific initialization which must be done in each slave now
* delay removal of table indexes to `finish`
* get back relations from each table on `finish`
Similar work still has to be done for entities insertion - which will allow more
cleanup to the constraints handling which is still rough for now.
Related to #15538303
Sylvain Thénault <sylvain.thenault@logilab.fr> [Tue, 04 Oct 2016 13:14:35 +0200] rev 11783
[massive store] docstring and __init__ cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr> [Tue, 04 Oct 2016 13:14:18 +0200] rev 11782
[massive store] Drop dead code
This function is never used
Sylvain Thénault <sylvain.thenault@logilab.fr> [Fri, 30 Sep 2016 17:34:59 +0200] rev 11781
[massive store] Reintroduce methods that are necessary to properly handle master/slave configuration
Related to #15538303
Sylvain Thénault <sylvain.thenault@logilab.fr> [Wed, 28 Sep 2016 09:02:14 +0200] rev 11780
[massive store] Follow configuration of the metadata generator
Don't drop constraints and indexes for tables that are ignored by the metadata
generator given to the store. One may now easily disable insertion of e.g.
created_by / owned_by by removing them from the MetadataGenerator.META_RELATIONS
set, in which case indexes for associated table won't be removed by the massive
store.
Adrien Di Mascio <Adrien.DiMascio@logilab.fr> [Mon, 17 Oct 2016 16:53:28 +0200] rev 11779
[dataimport] make MetadataGenerator.META_RELATIONS customizable
This should be done on the instance rather than on the class
Sylvain Thénault <sylvain.thenault@logilab.fr> [Wed, 28 Sep 2016 08:57:48 +0200] rev 11778
[massive store] Rework constraint/index handling
The basic idea is to keep the primary constraint on entities.eid since it's
heavily used in metadata insertions. Other option would have been to drop /
recreate but its very costly on big database, and the index is used for
insertion into the entities table itself, so it's not worth droping it at a
first glance.
Also, keeping it avoids to systematically drop all constraints which depends on
it. We may thus now lazily drop constraints, only on insertion of some
etype/rtype for the related table.
Related to #15538359
Sylvain Thénault <sylvain.thenault@logilab.fr> [Tue, 27 Sep 2016 12:02:07 +0200] rev 11777
[massive store] Lazy removal of constraints and metadata indexes
They should not be removed during store's init, because we may want to query the
database with its index between store creation and call to prepare_insert* (e.g.
to build the extid2eid map).
Along the way:
* rename drop_metadata_constraints into drop_metadata_indexes, because that's
what it does
* rework a bit impacted tests
Closes #15538359
Sylvain Thénault <sylvain.thenault@logilab.fr> [Mon, 17 Oct 2016 14:50:47 +0200] rev 11776
[repository] Drop the index on entities.type
It should be now only used in maintenance queries in checkintegrity. Those are
not worth maintaining it.
Closes #15538317
Sylvain Thénault <sylvain.thenault@logilab.fr> [Thu, 06 Oct 2016 12:11:23 +0200] rev 11775
[source] Drop source mapping handling
It was only used by the cwxmlparser which has been deleted. This is too complex
for litle benefit, we don't want to maintain that in cubicweb.
Sylvain Thénault <sylvain.thenault@logilab.fr> [Wed, 05 Oct 2016 15:30:10 +0200] rev 11774
[repository] Drop the entities.extid column and associated cache
This was not necessary anymore with promoted usage of the new data import API.
Turn repository's _type_extid_cache to _type_cache with only the entity's type
as key.
This introduces an backward incompatible change: entity_metas dict doesn't
contains anymore the extid key, but it doesn't seem used at all anywhere, so
this sounds acceptable.
Closes #15538317
Sylvain Thénault <sylvain.thenault@logilab.fr> [Fri, 30 Sep 2016 17:38:12 +0200] rev 11773
[repository] Drop the entities.asource column
It not used anymore thanks to previous patches.
Closes #15538288
Sylvain Thénault <sylvain.thenault@logilab.fr> [Thu, 29 Sep 2016 22:54:26 +0200] rev 11772
[integrity check] Stop using entities.asource column in integrity checks
Simply check if entities have a cw_source, and link it to the system source when not found.
Related to #15538288
Sylvain Thénault <sylvain.thenault@logilab.fr> [Mon, 17 Oct 2016 13:42:30 +0200] rev 11771
[rql2sql/test] Ensure test results predictability
Without this, test were randomly failing within tox due to different order of solutions.
Sylvain Thénault <sylvain.thenault@logilab.fr> [Thu, 06 Oct 2016 21:17:01 +0200] rev 11770
[rql2sql] Stop generating SQL query from RQL using entities.type
This was necessary in the "true multi-sources" time, it's not anymore, while
maintaining this index is costly.
Related to #15538317
Sylvain Thénault <sylvain.thenault@logilab.fr> [Tue, 11 Oct 2016 10:30:59 +0200] rev 11769
[test] Remove some usages of lgc.testlib
Sylvain Thénault <sylvain.thenault@logilab.fr> [Thu, 06 Oct 2016 21:14:49 +0200] rev 11768
[test] Avoid pytest discovery warnings
Having TestServerConfiguration and alike in the test module namespace causes
pytest discovery errors, because it thinks it's a test class.
Sylvain Thénault <sylvain.thenault@logilab.fr> [Fri, 30 Sep 2016 18:25:08 +0200] rev 11767
flake8 and all
* update some copyright
* drop most __docformat__ declaration
* fix some flake8 warnings / errors
Sylvain Thénault <sylvain.thenault@logilab.fr> [Wed, 05 Oct 2016 15:28:43 +0200] rev 11766
[schema2sql] Drop deprecated comment
Sylvain Thénault <sylvain.thenault@logilab.fr> [Sat, 01 Oct 2016 11:56:27 +0200] rev 11765
[cnx] Use entity_type instead of entity_metas()['type']
The latter is deprecated in favor of the former which has been introduced on cnx
to replace it.
Sylvain Thénault <sylvain.thenault@logilab.fr> [Thu, 06 Oct 2016 12:15:50 +0200] rev 11764
[entity] Reimplement and deprecate entity.cw_metainformation
It shouldn't rely anymore on cnx.entity_metas which is itself deprecated, and
the cw_metainformation API is similarly not needed anymore, access directly to
cwuri / cw_source if needed.
Related to #15538288
Sylvain Thénault <sylvain.thenault@logilab.fr> [Fri, 30 Sep 2016 17:39:17 +0200] rev 11763
[repository] Drop type_and_source_from_eid and rename related cache
We don't want to handle anymore the entities.asource column so we shouldn't use
those anymore. Also rename repository's _type_source_cache into
_type_extid_cache as this is what it's containing now.
Do similar renaming to the system source API.
Related to #15538288
Sylvain Thénault <sylvain.thenault@logilab.fr> [Fri, 30 Sep 2016 17:34:11 +0200] rev 11762
[entity] Stop returning source information in cw_metainformation()
We should not need that. It has been introduced to handle things we should not
do and probably don't do anymore (e.g. attempting to grasp recursive import of
several sites).
Related to #15538288.
Sylvain Thénault <sylvain.thenault@logilab.fr> [Fri, 30 Sep 2016 17:04:42 +0200] rev 11761
[json] Stop serializing cw_source into default json representation of an entity
Related to #15538288
Sylvain Thénault <sylvain.thenault@logilab.fr> [Thu, 29 Sep 2016 23:11:38 +0200] rev 11760
Drop deprecated LDAP related script
both are relying on the 'entities.source' column which has been dropped in 3.19.
They have been written with the old ldapsource in mind, which has been dropped at
that time.
Sylvain Thénault <sylvain.thenault@logilab.fr> [Fri, 30 Sep 2016 17:18:15 +0200] rev 11759
[entity] Stop linking to external site for external entities
This behaviour was ok when we had "true" multi-sources but its rather painful
with datafeed sources. Also, it makes absolute_url() costlier than what it
should. Besides, it relies on cw_metainformation()['source'] that is pending for
removal.
Instead, add a link to the original object in the metadata view (the one that
displays eid and source at the bottom right corner of the primary view).
Related to #15538288.
Sylvain Thénault <sylvain.thenault@logilab.fr> [Thu, 29 Sep 2016 22:23:16 +0200] rev 11758
[datafeed] Drop entity deletion handling in the default source / parser
This should be handled by specific parser to avoid using "entities.asource".
Reimplement it in the ldap parser.
Related to #15538288
Sylvain Thénault <sylvain.thenault@logilab.fr> [Thu, 29 Sep 2016 11:44:19 +0200] rev 11757
[sources] Drop 'moved_entities' table handling
This was clumsy and could be handled using exturi / cwsource, as the ldap source
now does.
Sylvain Thénault <sylvain.thenault@logilab.fr> [Fri, 30 Sep 2016 17:36:40 +0200] rev 11756
[repository] Drop deprecated extid2eid API and friends
This will break cwxmlparser based sources. They should be rewritten using a
specific parser, based on xml representation or on rqlio. This is harsh but
allows a so big cleanup of the code base.
Furthermore, it's necessary for asource/extid handling in the entities table
which is costly for most app that don't care at all about that...
In this cset, delete:
* all extid2eid methods
* repo._extid_cache handling
* [before/after]_entity_insertion source callback
* the cwxmlparser
and update related tests, notably unittest_datafeed where 'repull' testing has
been removed, since it's now handled by the dataimport API and should not be
retested there.
Related to #15538288
Closes #15538383
Sylvain Thénault <sylvain.thenault@logilab.fr> [Fri, 30 Sep 2016 17:36:02 +0200] rev 11755
[ldap] Stop using entities table in ldap source authentication and parser
We may used cwuri for the same purpose, and do one more step towards deletion of
entities.extid column.
Related to #15538288
Denis Laxalde <denis.laxalde@logilab.fr> [Wed, 26 Oct 2016 11:24:57 +0200] rev 11754
[test] Use stdlib unittest in unittest_webconfig.py
Denis Laxalde <denis.laxalde@logilab.fr> [Thu, 03 Nov 2016 12:15:42 +0100] rev 11753
[tox] Require ldap3 < 2 for server tests
Apparently we are using some constants that got dropped/moved in ldap3 >=
2.0.0 (which came out last week). So stay with version 1.4 until this gets
sorted out.
David Douard <david.douard@logilab.fr> [Wed, 02 Nov 2016 11:11:01 +0100] rev 11752
[setup] fix CWDevelop to keep compat with python 2.7
in Python, 2.7, setuptools.command.develop.develop is an old style class, thus
it cannot be called with super.
Denis Laxalde <denis.laxalde@logilab.fr> [Wed, 26 Oct 2016 08:34:34 +0200] rev 11751
[devtools] Make dependency on backports.tempfile (Python2) optional
The package may not be available in all systems (e.g. no Debian package exist
at the moment), and we should not crash with ImportError when importing
testlib from client code. Follow up on a6dc650bc230 where the dependency was
introduced.
Sylvain Thénault <sylvain.thenault@logilab.fr> [Wed, 28 Sep 2016 22:17:36 +0200] rev 11750
[sources] Drop source support_entity / support_relation source API
this was used in the pre-datafeed area, it's not worth it anymore.
Its only valid usage was in authentication to detect if the source was
supporting CWUser, hence we now call it systematically and catch
NotImplementedError.
Sylvain Thénault <sylvain.thenault@logilab.fr> [Fri, 30 Sep 2016 18:24:17 +0200] rev 11749
[dataimport] Add a new NullStore
This is useful to do some import tests where you only want to test steps prior
to the database insertion, as well as "dry-run" commands.
Use it as a documenting base class.
Sylvain Thénault <sylvain.thenault@logilab.fr> [Fri, 30 Sep 2016 17:03:33 +0200] rev 11748
[devtools] Properly reraise the exception
with previous implementation, the original exception wasn't visible in python 2.
Denis Laxalde <denis.laxalde@logilab.fr> [Mon, 24 Oct 2016 14:41:36 +0200] rev 11747
[pkg] Do not install sql scripts in <prefix>/share/cubicweb/schemas
These are already site-packages along with cubicweb package.
Denis Laxalde <denis.laxalde@logilab.fr> [Mon, 24 Oct 2016 10:08:36 +0200] rev 11746
[devtools/qunit] Retain firefox process stdout to eventually print it
The bare RuntimeError that gets emitted upon timeout does not help much to
diagnose the problem. Let's see if we can get more information from stdout by
not dropping it.
Denis Laxalde <denis.laxalde@logilab.fr> [Mon, 24 Oct 2016 09:45:38 +0200] rev 11745
[devtools/qunit] Turn FirefoxHelper into a context manager
So that stop() method always gets called.
Also ensure temporary directory created on __init__ gets deleted as well upon
exit.
Denis Laxalde <denis.laxalde@logilab.fr> [Mon, 24 Oct 2016 09:31:55 +0200] rev 11744
[test] Replace logilab-common's with_tempdir by tempfile.TemporaryDirectory
logilab-common's with_tempdir decorator, which is considered harmful (see
https://www.logilab.org/ticket/8267966).
Rely on backports.tempfile to provide TemporaryDirectory on Python 2.
Re-export it in testlib module for convenience.
Denis Laxalde <denis.laxalde@logilab.fr> [Mon, 24 Oct 2016 09:29:08 +0200] rev 11743
[test] Use `with open` syntax in cwconfig "prefix" tests
Denis Laxalde <denis.laxalde@logilab.fr> [Mon, 24 Oct 2016 09:02:53 +0200] rev 11742
[test] Use stdlib unittest.main() in qunit
Denis Laxalde <denis.laxalde@logilab.fr> [Tue, 16 Aug 2016 16:00:32 +0200] rev 11741
[web] Clear "pending_others" session key after all relations got processed
The RelationField appends data to "pending_others" key while processing its
posted information, but this key may already have been removed by the edit
controller. So remove the key from session data only when all "others"
relations have been processed using a while loop.
A test case would be nice, but I could not build one easily unfortunately.
Closes #4354551.
Denis Laxalde <denis.laxalde@logilab.fr> [Wed, 28 Sep 2016 11:06:28 +0200] rev 11740
[datafeed] Complete the import log even if parser could not be found
It happens that if the parser could not be found, _pull_data() would just
return an empty dict without taking care to close the import log which it just
opened. This leads to misleading information in the user interface where
CWDataImport entities kept accumulating in the "imports" tab of CWSource
primary view without anything else happening.
So:
* log an error message when parser cannot be found
* always close (write logs and set "end_timestamp" attribute) import log when
leaving _pull_data().
Closes #15505460.
Denis Laxalde <denis.laxalde@logilab.fr> [Fri, 21 Oct 2016 18:03:06 +0200] rev 11739
[twconfig] Set default value for "interface" to 0.0.0.0
This value means "all IP addresses on this host", which seems to fit with
"default to everywhere" that's documented.
Closes #15135610.
Denis Laxalde <denis.laxalde@logilab.fr> [Wed, 12 Oct 2016 17:03:02 +0200] rev 11738
[session] Allow self._open to be False in __enter__
I do not understand why this assertion was there in the first place. And
currently, it makes running any migration impossible. Hopefully, someone may
be able to explain.
See also commit b48020a80dc3 which mentions this assertion and suggests to
remove it.
Denis Laxalde <denis.laxalde@logilab.fr> [Fri, 21 Oct 2016 18:35:46 +0200] rev 11737
[pkg] Warn about "develop" command not installing legacy cubes directory
Not sure we can do better than a warning, which is better than nothing
anyways.
Closes #15785635.
Denis Laxalde <denis.laxalde@logilab.fr> [Fri, 21 Oct 2016 18:43:13 +0200] rev 11736
[pkg] Cleanup MANIFEST.in from non-matching entries
This cleans up output of setup.py executions.
Adrien Di Mascio <Adrien.DiMascio@logilab.fr> [Fri, 21 Oct 2016 18:10:15 +0200] rev 11735
[devtools] make i18ncube customizable in a cube
closes #15613724
Sylvain Thénault <sylvain.thenault@logilab.fr> [Tue, 27 Sep 2016 12:28:39 +0200] rev 11734
[rql2sql] Drop deprecated __docformat__
it causes flake8 error on some of its versions.
Sylvain Thénault <sylvain.thenault@logilab.fr> [Thu, 22 Sep 2016 16:21:51 +0200] rev 11733
[rql2sql] Handle comparison of eid on variables from an subquery
We don't want to force using 'identity' in this case which may be easily
handled by detecting variable is a `ColumnAlias` and relation is 'eid'.
Closes #15393583