[massive store] Docstring / comment cleanups

[massive store] Turn a runtime error into an assertion geting there is a bug.

[massive store] Delay metadata attributes and default values handling to flush_entities This may be done once for all there, instead of for each entity in prepare_insert_entity.

[massive store] Use a slave specific table for relation insertion in the massive store * attribute a random id to the store * add a column containing this id in the cwmassive_initialized table * separate generic `master_init` which create the cwmassive_initialized table from rtype specific initialization which must be done in each slave now * delay removal of table indexes to `finish` * get back relations from each table on `finish` Similar work still has to be done for entities insertion - which will allow more cleanup to the constraints handling which is still rough for now. Related to #15538303

[massive store] docstring and __init__ cleanup

[massive store] Drop dead code This function is never used

[massive store] Reintroduce methods that are necessary to properly handle master/slave configuration Related to #15538303

[massive store] Follow configuration of the metadata generator Don't drop constraints and indexes for tables that are ignored by the metadata generator given to the store. One may now easily disable insertion of e.g. created_by / owned_by by removing them from the MetadataGenerator.META_RELATIONS set, in which case indexes for associated table won't be removed by the massive store.

[dataimport] make MetadataGenerator.META_RELATIONS customizable This should be done on the instance rather than on the class

[massive store] Rework constraint/index handling The basic idea is to keep the primary constraint on entities.eid since it's heavily used in metadata insertions. Other option would have been to drop / recreate but its very costly on big database, and the index is used for insertion into the entities table itself, so it's not worth droping it at a first glance. Also, keeping it avoids to systematically drop all constraints which depends on it. We may thus now lazily drop constraints, only on insertion of some etype/rtype for the related table. Related to #15538359

[massive store] Lazy removal of constraints and metadata indexes They should not be removed during store's init, because we may want to query the database with its index between store creation and call to prepare_insert* (e.g. to build the extid2eid map). Along the way: * rename drop_metadata_constraints into drop_metadata_indexes, because that's what it does * rework a bit impacted tests Closes #15538359

[repository] Drop the index on entities.type It should be now only used in maintenance queries in checkintegrity. Those are not worth maintaining it. Closes #15538317

[source] Drop source mapping handling It was only used by the cwxmlparser which has been deleted. This is too complex for litle benefit, we don't want to maintain that in cubicweb.

[repository] Drop the entities.extid column and associated cache This was not necessary anymore with promoted usage of the new data import API. Turn repository's _type_extid_cache to _type_cache with only the entity's type as key. This introduces an backward incompatible change: entity_metas dict doesn't contains anymore the extid key, but it doesn't seem used at all anywhere, so this sounds acceptable. Closes #15538317

[repository] Drop the entities.asource column It not used anymore thanks to previous patches. Closes #15538288

[integrity check] Stop using entities.asource column in integrity checks Simply check if entities have a cw_source, and link it to the system source when not found. Related to #15538288

[rql2sql/test] Ensure test results predictability Without this, test were randomly failing within tox due to different order of solutions.

[rql2sql] Stop generating SQL query from RQL using entities.type This was necessary in the "true multi-sources" time, it's not anymore, while maintaining this index is costly. Related to #15538317

[test] Remove some usages of lgc.testlib

[test] Avoid pytest discovery warnings Having TestServerConfiguration and alike in the test module namespace causes pytest discovery errors, because it thinks it's a test class.

flake8 and all * update some copyright * drop most __docformat__ declaration * fix some flake8 warnings / errors

[schema2sql] Drop deprecated comment

[cnx] Use entity_type instead of entity_metas()['type'] The latter is deprecated in favor of the former which has been introduced on cnx to replace it.

[entity] Reimplement and deprecate entity.cw_metainformation It shouldn't rely anymore on cnx.entity_metas which is itself deprecated, and the cw_metainformation API is similarly not needed anymore, access directly to cwuri / cw_source if needed. Related to #15538288

[repository] Drop type_and_source_from_eid and rename related cache We don't want to handle anymore the entities.asource column so we shouldn't use those anymore. Also rename repository's _type_source_cache into _type_extid_cache as this is what it's containing now. Do similar renaming to the system source API. Related to #15538288

[entity] Stop returning source information in cw_metainformation() We should not need that. It has been introduced to handle things we should not do and probably don't do anymore (e.g. attempting to grasp recursive import of several sites). Related to #15538288.

[json] Stop serializing cw_source into default json representation of an entity Related to #15538288

Drop deprecated LDAP related script both are relying on the 'entities.source' column which has been dropped in 3.19. They have been written with the old ldapsource in mind, which has been dropped at that time.

[entity] Stop linking to external site for external entities This behaviour was ok when we had "true" multi-sources but its rather painful with datafeed sources. Also, it makes absolute_url() costlier than what it should. Besides, it relies on cw_metainformation()['source'] that is pending for removal. Instead, add a link to the original object in the metadata view (the one that displays eid and source at the bottom right corner of the primary view). Related to #15538288.

[datafeed] Drop entity deletion handling in the default source / parser This should be handled by specific parser to avoid using "entities.asource". Reimplement it in the ldap parser. Related to #15538288

[sources] Drop 'moved_entities' table handling This was clumsy and could be handled using exturi / cwsource, as the ldap source now does.

[repository] Drop deprecated extid2eid API and friends This will break cwxmlparser based sources. They should be rewritten using a specific parser, based on xml representation or on rqlio. This is harsh but allows a so big cleanup of the code base. Furthermore, it's necessary for asource/extid handling in the entities table which is costly for most app that don't care at all about that... In this cset, delete: * all extid2eid methods * repo._extid_cache handling * [before/after]_entity_insertion source callback * the cwxmlparser and update related tests, notably unittest_datafeed where 'repull' testing has been removed, since it's now handled by the dataimport API and should not be retested there. Related to #15538288 Closes #15538383

[ldap] Stop using entities table in ldap source authentication and parser We may used cwuri for the same purpose, and do one more step towards deletion of entities.extid column. Related to #15538288

[test] Use stdlib unittest in unittest_webconfig.py

[tox] Require ldap3 < 2 for server tests Apparently we are using some constants that got dropped/moved in ldap3 >= 2.0.0 (which came out last week). So stay with version 1.4 until this gets sorted out.

[setup] fix CWDevelop to keep compat with python 2.7 in Python, 2.7, setuptools.command.develop.develop is an old style class, thus it cannot be called with super.

[devtools] Make dependency on backports.tempfile (Python2) optional The package may not be available in all systems (e.g. no Debian package exist at the moment), and we should not crash with ImportError when importing testlib from client code. Follow up on a6dc650bc230 where the dependency was introduced.

[sources] Drop source support_entity / support_relation source API this was used in the pre-datafeed area, it's not worth it anymore. Its only valid usage was in authentication to detect if the source was supporting CWUser, hence we now call it systematically and catch NotImplementedError.

[dataimport] Add a new NullStore This is useful to do some import tests where you only want to test steps prior to the database insertion, as well as "dry-run" commands. Use it as a documenting base class.

[devtools] Properly reraise the exception with previous implementation, the original exception wasn't visible in python 2.

[pkg] Do not install sql scripts in <prefix>/share/cubicweb/schemas These are already site-packages along with cubicweb package.

[devtools/qunit] Retain firefox process stdout to eventually print it The bare RuntimeError that gets emitted upon timeout does not help much to diagnose the problem. Let's see if we can get more information from stdout by not dropping it.

[devtools/qunit] Turn FirefoxHelper into a context manager So that stop() method always gets called. Also ensure temporary directory created on __init__ gets deleted as well upon exit.

[test] Replace logilab-common's with_tempdir by tempfile.TemporaryDirectory logilab-common's with_tempdir decorator, which is considered harmful (see https://www.logilab.org/ticket/8267966). Rely on backports.tempfile to provide TemporaryDirectory on Python 2. Re-export it in testlib module for convenience.

[test] Use `with open` syntax in cwconfig "prefix" tests

[test] Use stdlib unittest.main() in qunit

[web] Clear "pending_others" session key after all relations got processed The RelationField appends data to "pending_others" key while processing its posted information, but this key may already have been removed by the edit controller. So remove the key from session data only when all "others" relations have been processed using a while loop. A test case would be nice, but I could not build one easily unfortunately. Closes #4354551.

[datafeed] Complete the import log even if parser could not be found It happens that if the parser could not be found, _pull_data() would just return an empty dict without taking care to close the import log which it just opened. This leads to misleading information in the user interface where CWDataImport entities kept accumulating in the "imports" tab of CWSource primary view without anything else happening. So: * log an error message when parser cannot be found * always close (write logs and set "end_timestamp" attribute) import log when leaving _pull_data(). Closes #15505460.

[twconfig] Set default value for "interface" to 0.0.0.0 This value means "all IP addresses on this host", which seems to fit with "default to everywhere" that's documented. Closes #15135610.

[session] Allow self._open to be False in __enter__ I do not understand why this assertion was there in the first place. And currently, it makes running any migration impossible. Hopefully, someone may be able to explain. See also commit b48020a80dc3 which mentions this assertion and suggests to remove it.

[pkg] Warn about "develop" command not installing legacy cubes directory Not sure we can do better than a warning, which is better than nothing anyways. Closes #15785635.

[pkg] Cleanup MANIFEST.in from non-matching entries This cleans up output of setup.py executions.

[devtools] make i18ncube customizable in a cube closes #15613724

[rql2sql] Drop deprecated __docformat__ it causes flake8 error on some of its versions.

[rql2sql] Handle comparison of eid on variables from an subquery We don't want to force using 'identity' in this case which may be easily handled by detecting variable is a `ColumnAlias` and relation is 'eid'. Closes #15393583

Take care that cnx may not be set in error view

[autoform] Ease overriding of inlined form renderer. Closes #15755515

[ldapfeed] Use new MetadataGenerator api instead of deprecated MetaGenerator

Fix typos

[devctl] make sure i18ncube always delete its tempdir closes #15613724

i18ncube should ignore node_modules directory by default related to #15613724

refactor i18n messages extraction This refactoring will ease later implementation of i18n cube customization. related to #15613724

[web/application] remove `path` argument from CubicwebPublisher methods Path can actually be accessed from `req` object. This allows to avoid duplicating this information. This cset prepares the next ones that aim at adding the language as a prefix of the relative path.

Merge 3.23 branch

[pkg] Fix recursive-include missing a "*" in MANIFEST.in

Added tag 3.23.2, centos/3.23.2-1, debian/3.23.2-1 for changeset 8b287ad7b95b

Release 3.23.2

[test] Add test method required by d41832121b29

[ldapfeed] Fix crash when user's password has to be generated Binary isn't hashable hence can't be added to a set. Test added.

[test/ldapfeed] Drop unused constant

[migration] Test and fix case of addition of an attribute with some serialized constraint It was crashing on attempt to remove an unexisting constraint. This code may be removed safely as it's expected to be handled by the constraint removal hook. Notice that adding a vocabulary on shortpara makes yams change the varchar max size. This is arguable but not the point of this test, so simply adapt the max size defined in the schema coherently so we don't have to bother with that at this point. Grafted from f87da59faea1 in default branch.

[skeleton] Properly format debian/changelog using current datetime and author information Reformat "context" dict to have it PEP8 compliant along the way (only white space changes).

[utils] Encode Binary value in base64 for JSON export In unittest_utils.py, use unittest2's TestCase instead of lgc.testlib one to benefit from subTest API. Closes #15409885.

[test] Make cubicweb/test/unittest_utils.py flake8-clean

[skeleton] Set Debian distribution to "UNRELEASED" in changelog So to as to make it clear that changelog needs to be edited before the Debian package may be published.

Fix log level parameter not taken into account in cwsource log table An error in the jquery selector of the html widget controlling the log level was the cause of this bug. Closes #15772634

[config] Ignore cubicweb-pyramid in available_cubes() So that, in particular, its ccplugin.py does not get loaded (which would hide the "pyramid" command in cubicweb). Related to #14023058.

[config] Exclude "pyramid" cube when initializing application cubes Related to #14023058.

[migration] Drop cubicweb-pyramid CWProperty in 3.24.0 migration This does a partial "drop_cube('pyramid')" equivalent but only considering CWProperty. We cannot just drop the cube since we have introduced some mechanism to exclude pyramid cube from config's cubes list and that would make MigrationHelper.cmd_drop_cube() call not return "pyramid" cube. Related to #14023058.

[dataimport/test] Use minimal schema for massive import tests Drop unused stuff.

[massive store] Drop deprecated code This is not a desired part of the store API. Mapping between extid and eid should be done by the importer. This had been backported from the dataio cube, but not one seems to rely on it yet, so don't provide any backward compat.

[flake8] cubicweb/dataimport/test/test_massive_store.py is now flake8 friendly

[massive store] Drop buggy log call PGHelper has no logger, the store has.

[massive store] Drop duplicated log

[flake8] cubicweb/server/rqlannotation.py is now flake8 friendly

[pyramid] Override cubicweb.hooks.syncsession.get_user_sessions() for Pyramid Closes #13436818

[pyramid] Retrieve user's groups using a custom RQL instead of user's groups property With cubicweb 3.24, user.groups is set lazily and so retrieving it at this point will cause a traceback because the connection is not yet entered. Also, user doesn't accept anymore groups and properties arguments.

[hooks] Change get_user_sessions prototype, necessary for hijacking from other session implementations (e.g. pyramid)

Store user groups and properties as session data * stop retrieving them systematically, only when need, * reimplement session synchronization hooks with some cleanups along the way, * cleanup call to set language: not needed from the base request nor from the server side, only for the web request (on the server side, language is necessary only for notification and such code should set it explicitly). There is still a XXX remaining about one can only "enter" a connection once and this is a problem in some cases. IMO, this restriction could be removed. Closes #13500113.

[skeleton] Remove override_dh_python2 We do not ship files as "data files" anymore. Related to #13001466.

[skeleton] Have setuptools install data files Just add `include_package_data=True` as recommended by `setuptools documentation`_ as this appears to be the simplest way to handle this. Related to #13001466. .. _`setuptools documentation`: \ http://setuptools.readthedocs.io/en/latest/setuptools.html#including-data-files

[skeleton] Update MANIFEST.in following new "package" layout Related to #13001466.

[skeleton] Add python-setuptools to Debian Build-Depends Related to #13001466.

[skeleton] Add dh-python to Debian Build-Depends Related to #13001466.

[skeleton] Copy files mode bits (debian/rules in particular)

[skeleton] Do not exclude setup.py from flake8 check in tox.ini The file is flake8 compliant, actually.

Merge public heads

[doc] Document configuration override by environment variables Related to #13889793.

[doc] Update book sections about cubes being Python packages Related to #1300146006.

[doc] Start filling changes/3.24.rst with recent changes

[doc] Drop commented mention of "live-server" command Follow-up for d83676aaea21.

[pyramid] Drop guard of old cubicweb version

[pkg] Declare extras (optional) dependencies This is primary to extract "pyramid" dependencies out of install_requires as they are actually optional. Along the way, resurrect all things in __pkginfo__.__recommends__ which were not used at all and convert them into extra_requires. It appears that __depends__ and __recommends__ in __pkginfo__.py are not useful so drop them and inline dependencies as install_requires/extra_requires in setup function call. For pyramid tests to continue working, add respective dependencies to test-misc.txt requirements file.

[tox] Exclude all files in symlinked directories from check-manifest call

[tox] Get rid of "touch test-results.xml" for flake8 and check-manifest environments CI server does not rely on this anymore.

[debian] cubicweb-pyramid conflicts and replaces pyramid-cubicweb Related to #14023058.

Merge cubicweb-pyramid cube Only keep the CWSession schema definition and the ctl command, now in cubicweb/pyramid/pyramidctl.py Related to #14023058.

[ccplugin] flake8

[ccplugin] docstring clarification

[doc] improve a bit the README to explain how to allow login in dev mode

Added tag 0.4.0, debian/0.4.0-1, centos/0.4.0-1 for changeset c0c98e5ba55e

[pkg] typo

[pkg] 0.4.0

[doc] add a bit more infos in the README

[cfg] remove the pyramid-{auth,session}-secret config options (closes #11027189) these should now be set in the pyramid.ini config file.

[cc] add a dbglevel cmdline option to specify the DBG_XXX flags to set also display the list of valid loglevel values in the help message.

Added tag centos/0.3.2-1 for changeset c9daf54738fd

[pkg] add spec file

Added tag 0.3.2, debian/0.3.2-1 for changeset ab4c5509407f

[pkg] fixup debian control metadata add Homepage field, replace deprecated XS-Python-Version with X-Python-Version.

[pkg] add debian watch file

[pkg] don't remove egg_info from debian package

[pkg] 0.3.2

[ccplugin] get the cw repo from the wsgi app Otherwise with cubicweb 3.22 we end up re-creating a new repo from scratch, which ends up in tears.

[ccplugin] print_function

Move tag debian/0.3.1-1 on changeset ab68206f765b

[debian] typo in debian/changelog

Added tag 0.3.1, debian/0.3.1-1 for changeset 9d82848e82a7

[pkg] 0.3.1

[ccplugin] fix a bug introduced in e95725d7ce90 (closes #5731783)

Added tag 0.3.0, debian/0.3.0-1 for changeset 7fbeb77fe690

[pkg] 0.3.0

[debian] add missing dependency on pyramid-cubicweb and python-wsgicors

[pkg] depends on waitress (closes #5576180)

[pkg] update to dh9 and dh_python2

Allow to override config file options by cmdline arguments (closes #5724484) For the sake of consistency with cw, the 'profile' option's short name has been removed, so we can use the lower 'p' for this purpose (for consistency with cubicweb). Requires CubicWeb 3.21 to work.

Added tag cubicweb-pyramid-version-0.2.0, cubicweb-pyramid-debian-version-0.2.0-1 for changeset 4c16305d5825

set debian version 0.2.0-1

Set version 0.2.0 And requires pyramid-cubicweb 0.2.0

Add profiling options

Monitor more configuration files Closes #4811352

Set version 0.1.1

--debug now activates all debug options Added a --debug-mode so the debug toolbar can be activated without switching reload on. Closes #4783342

Move the cors middleware initialisation to pyramid-cubicweb to reduce code duplication

Added tag cubicweb-pyramid-version-0.1.0, cubicweb-pyramid-debian-version-0.1.0-1 for changeset 3defbb0f147a

Prepare release

Add a --no-daemon option

Use 'wsgicors' for CORS handling. The CW CORS handling (in web/cors.py) is only able to work on cubicweb requests. When a request is not handled by bwcompat, we need a proper solution. The `wsgicors` library provides what we need as a wsgi middleware.

Watch for i18n files changes for auto-reload

Auto-reload now survives failed reload When the server stops within the reloader and let a list of files that were monitored, the reloader waits for any of these files to change and re-attempt to start the server. This work well when a syntax error is saved to a file: there is no need to restart manually the server anymore.

Implements auto-reload and daemon mode. Heavily inspired by pyramid pserve, with pieces of code taken from it. auto-reload Start the server in a subprocess that auto-stops when a file is modified, and exit with a specific code. daemon mode Uses some code from pserve, but use the cw configuratione so the command is compatible with 'status' and 'stop' commands.

Add the 'pyramid-auth-secret' option to all-in-one.conf

Add the 'pyramid-session-secret' option to all-in-one.conf This value is used by pyramid_cubicweb to initialize the cookie factory.

Add a CWSession entity. It is a subset of the CWSession that will be added to cubicweb in a future version. Closes #4317363

Implements the 'pyramid' command. It load an instance config, and load in the pyramid scaffolding provided by pyramid_cubicweb. The application is then served by a waitress server. Closes #4317312

Project structure

Merge with pyramid-cubicweb The following tasks have been done: - merge packaging files - merge documentation - move pyramid_cubicweb package at cubicweb/pyramid and update imports accordingly - rename tests directory into test - move pyramid-cubicweb README.rst into README.pyramid.rst until better idea - add a test dependency on unreleased cubicweb-pyramid to have both py27 and py34 tests pass Closes #14023058.

Port to Python3 (closes #14159555) Add py34 environments to tox configuration (only for CubicWeb >= 3.23). And depend on hg version of cubicweb-pyramid since it is not currently Python3-compatible.

Use opened connections as much as possible instead of opening a new internal connection to set the session cookie or retrieve the session data, which may exhaust the connection pools.

Test and fix status code and content of the login form on authentication error The status code was 200 and is now fixed to 403, but the content part of the test will fail until cset 02328f8cbd5c is integrated in cubicweb.

Added tag 0.7.0, debian/0.7.0-1, centos/0.7.0-1 for changeset 0cf2972d2011

[pkg] Version 0.7.0

retrieve session data in a single query instead of two (one for the session object, the other for its cwsessiondata attribute).

[tox] Update CubicWeb version targets * In -release, no need to specify cubicweb dependency, its pulled by setup.py. * Use 3.22 as -compat. * Point to "default" branch for -dev. Also remove test directory from posargs.

wsgi: clearer exception when CW_INSTANCE is missing instance_name is mandatory for cwcfg.config_for() Raise with a KeyError: 'CW_INSTANCE' instead of hard to read exception.

typo

[doc] update a bit the documentation Also convert the README in rst (so it can be properly displayed on cubicweb.org).

split collecting setting before using them, so the function can be reused when inserting WSGI middlewares

[bwcompat] log execption even when cubicweb.bwcompat.errorhandler = True (closes #13421901)

[pkg] add spec file

Added tag 0.6.1, debian/0.6.1-1 for changeset 2621daafa10c

[pkg] 0.6.1

[bwcompat] send 403 on authentication errors (closes #12219849) 200 is just wrong.

[bwcompat] set response headers on AuthenticationError (closes #12219860) Turns out sending a response without a Content-Type header (among others) is suboptimal. 5b36399b6b21 was not good enough.

Added tag 0.6.0, debian/0.6.0-1 for changeset 94d1a024b3bf

[pkg] 0.6.0

[tests] add a __main__ handler the relative import in test_rest_api.py needs to be modified to prevent a ValueError: Attempted relative import in non-package

[pkg] add a requirements-test.txt file for test dependencies on WebTest and cubicweb-pyramid

[login] fix the redirect url after login (closes #11689118) redirecting to '/' by default after login doesn't work properly when a prefix is used, whether we're sitting behind a PrefixMiddleware or not. To fix this, rely on cubicweb's build_url to turn any relative path into an absolute url.

keep track of all traceback in error handling, not just the exception message (closes #11689093)

[bwcompat] also set response headers in error cases I had this sitting around in my local copy, I don't remember why I needed this but it seems correct.

[config] move config of the secret used to encrypt session's data ID in pyramid.ini (closes #11689082) Introduce a new config entry (in pyramid.ini) for this (cubicweb.session.secret) to replace the (now deprecated) pyramid-session-secret (in all-in-one.conf). So we have now 3 secrets to configure: - cubicweb.session.secret: to encrypt session's data ID stored in a cookie, - cubicweb.auth.authtkt.session.secret: to encrypt auth cookie - cubicweb.auth.authtkt.persistent.secret: to encrypt persistent session auth cookie

Document how to configure the "secure" flag for authentication policies Closes #11376233.

[tox] Let posargs override py.test args

[tox] Use py.test Rename test view so that py.test does not consider it as a test function.

Add a tox configuration

Register predicates from the predicates module

[refactoring] Move MatchIsETypePredicate to a separate module

[refactoring] Move EntityResource and ETypeResource to a separate module

use CubicWeb request to execute RQL rset should be retrieved with cw_request, as it's then bound to it and propagate to all entities created from this rset (._cw). From there it may reach code expecting a request, not a connection (view, selector, etc).

[views] Rename entities to rest_api

[routes] Add a 'cwentities' route with traversal The route uses a factory that produces ETypeResource and EntityResource as a context. A 'delete' view serve as a test and demonstration. The module being experimental, it has to be explicitely included.