[dataimport] turn eids_seq_{start,range} into class attributes Only the master-mode store will actually use eids_seq_start to reset the eid sequence.

[dataimport] remove replace_sep parameter from massive store It is unused since changeset efbbf1e93a04 "[dataimport] Properly escape strings sent to COPY FROM"

[dataimport] remove pg_schema parameter from massive store It can be recovered from the system source's configuration.

[dataimport] remove drop_index parameter from massive store "drop_index=False" also implied not dropping any constraints, in particular foreign keys, which meant any attempt to import entities would fail, because we only add metadata (the entities table) after the entity's insertion.

[dataimport/test] update call to ucsvreader Silences warning: [3.20] 'separator' kwarg is deprecated, use 'delimiter' instead

[dataimport] remove deprecated methods from massive store This code is new in cubicweb, no reason for it to carry deprecated stuff from the start.

[dataimport/test] use store.finish instead of deprecated store.cleanup

[dataimport] drop no more used parameter on massive store

[dataimport] methods that modify in-place shouldn't return value Also mark those methods private and drop some useless comments in passing. Related to #5414760

[dataimport] implement new store API on massive store and deprecate the old one. Related to #5414760

[dataimport] .finish method was missing from (NoHook)RQLObjectStore

[massive store] identity_relation should not be considered, is_relation seems missing The "identity_relation" table should not even exist.

[dataimport/test] use the right assert methods instead of assertTrue with a comparison

[dataimport] import range from six.moves

[dataimport/test] feed binary data to ucsvreader

[dataimport/test] fix broken comparison We want to compare two ints, not an int and a db row

[dataimport] dict.iteritems() → dict.items()

[dataimport] use next builtin instead of next method on iterators

[dataimport] import StringIO from io

[dataimport] backport massive store from dataio cube Some tweaks have been made to work with 3.21 foreign key constraints. Also test are not executed in our own pg cluster instead of polluting the system one. Closes #5414760

[web] set proper Cache-Control header for static files - set Cache-Control max-age to match Expires - use the max_age method (defaults to 1 week) instead of hardcoded 6 months

[web] use a redirect for /data/foo requests that don't use the instance md5 hash The actual file contents can be cached for a long time. If the request doesn't use the hash, that's wrong. Using a redirect lets keep the caching for the actual file contents while not breaking requests missing the hash.

[web] fix typo in doc string

[web] stop adding an Expires header with all responses As per RFC 1945 (HTTP 1.0): If the date given is equal to or earlier than the value of the Date header, the recipient must not cache the enclosed entity. As per RFC 7234 (HTTP 1.1 Caching): If a response includes a Cache-Control field with the max-age directive (Section 5.2.2.8), a recipient MUST ignore the Expires field. Bottom line, the Expires header is now handled wherever the Cache-Control is handled: inside cache managers.

[web] stop looking at a request's Cache-Control header As per RFC 7234 (HTTP 1.1 Caching): The "Cache-Control" header field is used to specify directives for caches along the request/response chain. CubicWeb is not a cache.

[fti] add cw_fti_index_rql_limit method Improves on and deprecate cw_fti_index_rql_queries: execute the rql directly, so that we don't need to compute the set of eids up-front, but can just keep track of the last seen entity. Use the new method in reindex_entities. Keep calling cw_fti_index_rql_queries if it has been redefined (e.g in cubes). Related to #3621392.

[fti] backout 166c6f7b1be4 * breaks tests * see next commit for a better solution

[py3k] Fix unicode added in commit 2350424585f8

[server/ldap] deal with unicode or str coming out of ldap It seems we can get either, I've no idea in which exact circumstances, so this should let us work with both.

[test] pull yapps from Logilab's github Fixes a couple of failures, pull request is almost a year old (https://github.com/smurfix/yapps/pull/2).

[pkg] bump yams requirement to 0.41.1 Needed by changeset c9321aedfbf9.

[entities] import range from six.moves

merge head published by accident

[fti] Improve big table reindexation * Slice the reindexing in batches of 1000 entities. * Make the output more verbose. Closes #3621392

[tox] Reference requirements.txt for hooks environment The file existed but was not referenced.

[devtools/testlib] Remove assertions about direct usage of CubicWebTC/AutomaticWebTest The point is to be able to use a test runner other than pytest (e.g. nose2, nosetests or py.test) to run CubicWeb test suite. The issue is that most other test loaders inspects members of all base classes of test case classes to determine test methods and it hits this `config` property (actually executing it), which breaks the assertion. So just return None instead of failing. Closes #7817147.

Add ldap3 to server test requirements

[pkg] bump version number so tox won't use cw from pypi

[schema] add CubicWebRelationDefinitionSchema Stop monkeypatching yams, inherit from RelationDefinitionSchema instead.

[schema] Let RQLConstraint inherit from BaseRQLConstraint instead of RQLVocabularyConstraint There is apparently no reason for RQLConstraint to inherit from RQLVocabularyConstraint as it gets its `repo_check` method from `RepoEnforcedRQLConstraintMixIn` and RQLVocabularyConstraint (which inherits from BaseRQLConstraint) actually only overrides that method. So simplify inheritance by removing a useless level of indirection. Adjust cw_unrelated_rql accordingly, where it was that RQLConstraint inherited from RQLVocabularyConstraint.

[server] unormalize source logger's name to avoid encoding errors

[entities] parameters should be given to fire_transition, else they are simply ignored

[entity] set _cw_adapters_cache in __init__ rather than on first use

fix typos and indentation in docstrings/docs

session.id → session.sessionid

[test] test page_size problem (related to #8123231)

[predicates] guard against non-integer page_size form parameter Closes #8123231

[server/test] add test for db-dump command

[server] fix db-dump with sqlite 'gzip -c dbname > backupfile' needs to go through the shell.

[serverctl] fix db-dump Since changeset af47954c1015 we would die with: AttributeError: 'ServerMigrationHelper' object has no attribute 'repo'

[server/test] sorting vs dict vs python3

[devtools] more sorted() vs dict vs python3

[web/application] return bytes from redirect_handler and validation_error_handler

[devtools] fix ResourceWarning: unclosed file tmpdb-__default_empty_db__.config

[serverconfig] fix ResourceWarning: unclosed file bootstrap_cubes

Use cubicweb._

[syncschema] hashlib.md5() wants bytes

[server/test] use next() builtin instead of next() method

[server/test] dict.iterkeys

[syncschema] don't arbitrarily remove constraints from the in-memory schema For some constraint types, it may make sense to have several constraints of that type simultaneously. In such a case, removing one of them from the in-memory schema is wrong. Closes #7446353

[server/test] don't leak timeout from one test class to the next Reset SQL_CONNECT_HOOKS after we're done with this test class so the next one doesn't inherit a timeout it doesn't want.

[server/test] CWAttribute.extra_props is Bytes

[dataimport] give unicode objects to psycopg2 copy_from

[dataimport] extid must be a bytes object

[dataimport] import StringIO from io

[dataimport] fix ucsvreader for python3 The csv module in python3 wants str, so use a StreamReader to feed csv.reader and don't decode the output

[server/test] import StringIO from io on python3 Don't change the import for python2, as we print a mix of unicode and str.

[server/datafeed] drop URLLibResponseAdapter.info I don't know where it's used

[devtools] pass a key to sort() method python3 dicts are not comparable

[postcreate] properly set is_instance_of for bootstrap entities Entities created before CWEType itself need the is/is_instance_of relations created later. We were actually only inserting "is". Closes #7656606.

[migration] re-try without print_function when compiling a script fails As part of the python3 porting we added the __future__.print_function import to cubicweb.migration. That started affecting all migration scripts it runs, which may not be prepared. So if compiling a script results in SyntaxError, try again without the print_function flag. Closes #7935030

[test] do not overwrite user's CW_CUBES_PATH in unittest_migration Currently when you rely on your CW_CUBES_PATH environment variable to locate cubes necessary for cubicweb's tests, the migration test crashes because it overrides the env var.

[server/test] dict.itervalues() → iter(dict.values())

make sync_schema_props_perms(<computed rtype>) work as expected It currently ends up with an ExecutionError while we want to synchronize permissions. This makes 3.21.1 migration of any application using computed relation crashing. Also, remove pre 3.6 compat code in hooks that prevent living schema update on computed relation's permission changes.

[web/http_headers] use classmethod decorator

[schema] use classmethod decorator

[devtools] delete postgresql socket directory after killing the server ... or when failing to start it.

[skel] stop deleting cubes egg-info It doesn't hurt and provides some metadata.

[dataimport] Update doc to reflect new API relate() looks deprecated.

[server] fix generate_source_config regression on python2 lgc outputs "str" objects on python2 and python3, so we can't just use io.StringIO (which expects unicode objects). Fixes: c36cda9074c5

[tests] Fix config.repository() in tests The config.repository was overriden too late. Closes #5778294

[schema] RQL queries are always unicode objects in python 3 fixes commit 42c5bd7286b7.

[server/test] use relative import Fixes py3k breakage

[server/schemaserial] fix extra_props on py3k json.dumps returns unicode, and json.load eats unicode, so add explicit encoding/decoding.

[server/test] unicode → six.text_type

[web] fix Field.__str__ on py3k

[web] unicode → six.text_type

[web/test] CubicWebTC.view returns bytes

[uilib] fix traceback on python3

[web] integer division

[web/test] Binary eats bytes

[devtools] make httptest module importable under py3k

[web/test] CubicWebPublisher.handle_request returns bytes

[web] base64 works on bytes not str (py3k)

[web] accept unicode in parsed Authorization header

[web] CubicWebPublisher.ajax_error_handler must return bytes

[py3k] fix i18n

Fix md5crypt and crypt_password test for python3 Make sure to always work with bytes for hashing.

[server/test] use bytes for extid

[devtools/test] firstnames.txt is in latin1

[web/test] unicode → cubicweb._

[web/formwidgets] fix for py3k no idea why we encode that key...

[sobjects] unicode → six.text_type

[server] fix unittest_security for py3k - use source.binary_to_str instead of str(), helps when the backend returns memoryviews - fixup str vs bytes

[py3k] make ldapfeed tests pass

[server] import filter from six.moves

[ldapfeed] port to ldap3 python-ldap has no python3 support. XXX I only checked that unittest_ldapfeed passes.

[server/test] str vs bytes

[server/test] list() around map and zip

[server/test] to fold, unicode → six.text_type

[server/test] delete test that doesn't make sense on python3 XXX if PY2?

[server/test] unicode → six.text_type

[server] unicode → six.text_type

[py3k] fix building rset description for INSERT queries

[datafeed] use explicit encode/decode instead of casting to str

[py3k] io.BytesIO

[py3k] get urllib foo from six.moves

[devtools] CubicWebTC._check_html may have to deal with unicode or bytes If called with template=None, CubicWebTC.view will not go through main-template, and return the unicode data from the view directly instead of encoded data.

[ext/test] fix test relying on repr(text)

[req] fix url_parse_qsl on py3 should be folded with previous conversion

[server/session] unicode → six.text_type

[web/test] str vs bytes

[web/ajax] fix AjaxController on python3

[web/test] str → bytes

[wsgi/test] use bytes instead of str for POST body

[devtools] 'dict_values' object does not support indexing

[multipart] decode form data before calling parse_qs in python 3 parse_qs eats and returns 'str' on both python versions.

[multipart] unicode → six.text_type

[web] assert that CubicWebPublisher.handle_request returns bytes

[py3k] fix validateform controller We need to return bytes not str.

[web] unicode → six.text_type

[multipart] make sure form keys are not bytes

[web] unicode → six.text_type

[wsgi] don't convert form values This is done earlier in multipart.py

[server] import input from six.moves

[server] don't convert user.login to bytes to generate sessionid

[cwctl] stop playing tricks with std{out,err} buffering Python3 doesn't allow unbuffered text I/O, because it doesn't make sense.

[py3k] CubicWebTC.app_handle_request returns bytes

[py3k] unicode → six.text_type

[py3k] http headers should be ascii-only unicode objects

[py3k] Clarify IDownloadable expected types

[py3k] IDownloadable.download_data() returns bytes

[py3k] unicode → six.text_type

[py3k] unicode → six.text_type

[py3k] unicode → six.text_type

[py3k] unicode → six.text_type

[py3k] PageInfo.source contains bytes

[py3k] unicode → six.text_type

[py3k] unicode → six.text_type

[py3k] Don't encode URLs Python 3's urllib works with unicode objects.

[tests] Stop counting pickle bytes in ResultSet tests Between python 2's pickle and cPickle implementations, the various protocol versions added in python 3, the length of the pickled string is completely unpredictable. Instead, do a simple round trip and test a few attributes.

[tests] Port unittest_cwctl to py3k On python 2, sys.stdout takes bytes-like objects whereas python 3's takes unicode-like objects and handles the encoding on its own.

[migration] NoneType is no longer comparable in py3k

Rename gettext to cwgettext, use stdlib gettext cubicweb/gettext.py was a copy of Python 2.6's gettext.py with the context patch from bug https://bugs.python.org/issue2504 Instead of keeping that full copy, just inherit from gettext.GNUTranslations and add the context-related stuff.

[py3k] MethodType no longer takes a class argument

[py3k] unicode → six.text_type

[py3k] im_self → __self__

[py3k] Don't sort Action objects In python 3, arbitrary objects are no longer sortable.

[py3k] use inspect module to get a function's arguments

[py3k] __unicode__ and __str__

[test] call rql.parse with unicode objects Allows to get same results from rqlst.as_string() in python 2 and 3

[py3k] replace cmp with key in sorted() And stop sorting dicts in snippet_key()

[tests] remove useless sort Useless in python 2, actively harmful in python 3 where arbitrary objects are no longer sortable.

[py3k] dict methods no longer return indexable objects

[py3k] unicode → six.text_type

[py3k] unicode → six.text_type

[server] Port BFSS to py3k The BFSS API changes in python 3: * 'defaultdir' MUST be a unicode object * 'fsencoding' MUST NOT be set In python 2, fsencoding handles both the encoding of file paths on the file system (utf-8 by default, but the system may actually be using something else) and the encoding of file paths that will be stored in the database. So in python 3, we wipe the slate clean: * rely on sys.getfilesystemencoding() to convert unicode objects to bytes * always encode paths to utf-8 for storage in the database Caveat emptor / here be dragons: * sys.getfilesystemencoding() depends on the current locale, which therefore MUST be set properly * when migrating an existing instance from py2 to py3, one MAY need to reencode file paths stored in the database

[py3k] Binary.getvalue() returns bytes

[py3k] ensure Binary objects are initialized with bytes

[py3k] __unicode__ and __str__

[py3k] unicode → six.text_type

[mail] port to py3k

[web/views] port JSON views to py3k

[py3k] unicode → six.text_type

[web/controller] req.form is supposed to contain unicode-data

[py3k] unicode → six.text_type

[py3k] unicode vs str vs bytes vs the world

[web/views] Port csvexport views to py3k There are a few hoops to jump through since python 3's csv module _is_ unicode aware.

[py3k] unicode vs str vs bytes vs the world

[web/request] don't convert request parameters if py3k

[py3k] unicode → six.text_type

[py3k] __getslice__ → __getitem__ with slice support

[py3k] __nonzero__ → __bool__

[py3k] unicode vs str vs bytes vs the world

[py3k] use six.add_metaclass

[py3k] unicode → six.text_type

[py3k] add list() around dict methods

[py3k] unicode → six.text_type

[py3k] unicode → six.text_type

[py3k] unicode → six.text_type

[py3k] unicode → six.text_type

[server/utils] crypt_password should always return bytes

[py3k] str → six.binary_type

[py3k] unicode → six.text_type

[server/utils] convert to py3k using six

[py3k] six.PY2

[py3k] use text/binary types from six

[py3k] unicode → six.text_type

[py3k] bytes vs str

[py3k] io.BytesIO

[py3k] unicode → six.text_type

[py3k] bytes required for hashlib.{md5,sha1}()

[wsgi/test] add __main__ section

[py3k] use next builtin instead of next method

[web/views] use next builtin instead of next method

[py3k] declare meta classes using six.add_metaclass

[py3k] replace '_ = unicode' in global scope (closes #7589459)

[py3k] introduce cubicweb._ (related to #7589459) The end goal is to stop relying on '_' being in the builtin scope. In the mean time, deprecate the builtin '_' so that consumers are warned.

[py3k] dict.iterkeys → dict

[py3k] dict.itervalues → dict.values

[py3k] dict.iteritems → dict.items

implement __ne__ whenever we implement __eq__

[test] use unicode for rql queries (7/7) Closes #6694446

[test] use unicode for rql queries (6/7) Related to #6694446

[test] use unicode for rql queries (5/7) Related to #6694446

[test] use unicode for rql queries (4/7) Related to #6694446

[test] use unicode for rql queries (3/7) Related to #6694446

[test] use unicode for rql queries (2/7) Related to #6694446

[test] use unicode for rql queries (1/7) Related to #6694446

[schema] improve normalization of RQLExpressions Parse and print back the expression instead of manipulating the string. Among other benefits, it means we don't mangle embedded string constants that contain commas or multiple spaces. Closes #6694426

[server/rql2sql] use VariableRef.is_equivalent explicitly instead of relying on __eq__ What we want to compare is the variable referenced by the VariableRef, we should make that explicit. Closes #6694406

merge with 3.21.2

[statsd] fix the statsd logger (closes #7558147) socket.inet_pton's second argument is the ip, not a couple (ip, port)

Added tag 3.21.2, debian/3.21.2-1, centos/3.21.2-1 for changeset a5428e1ab364

[pkg] 3.21.2

[pkg] Remove leftover dep on pyro (closes #7479155)

merge with 3.20.10

[migration] fix change_attribute_type to update the live schema Closes #7170830

[devtools] add has_cache for postgres (closes #5739624) devtools stores info about existing dbs in the db handler, but in the case of postgresql that doesn't take into account the path to the cluster's datadir. Which means if we run two test modules (in the same test run), we'll create a "__default_empty_db__" for the first one, cache its existence, and then when moving on to the other module, believe the template already exists (but since the datadir depends on the test module's path, it does not). This patch is a bit of a kludge, and it would be better to make the cache key include enough data to not need this, but I'm not sure how to do that.

fix bad-caching of datetime with tz info at sql generation time There is a special handling for datetime with tzinfo, where value was stored in the query cache. The implementation of merge_args was simply overwriting parameters of the query with those in the query cache, expecting no collision. To fix this: * handle replacement of tzinfo in merge_args, not at sql generation time * add an assertion to ensure we've actually no collision Closes #6978316

Added tag 3.20.10, debian/3.20.10-1, centos/3.20.10-1 for changeset 8f82e9523962

[pkg] 3.20.10

merge with 3.19.13

[devtools] Fix CubicWebTC.temporary_permissions If the context manager is exited via an exception, the original permissions should be restored.

[devtools] Use TestCase.assertGreater

[autoform] fix appearance of link to add inlined creation form On entity creation, if there are some local permissions on the relation, we have no way of checking them since neither the subject nor the object of the relation exists yet. In such a case, we should add the link by default, for consistency (see other places where we use `may_have_permission`). Closes #6711900

[predicates] Simplify specified_etype_implements

[web/views] Fix `has_editable_relation` predicate wrt inlined relations Prevents the "modify" action from being shown when there is nothing to edit.

[web/views] avoid propagation of NoSelectableObject in some case of inlined relations / permissions When selecting an inlined creation form, we should catch the NoSelectable exception that will be raised if the user cannot add entities of the target type (this is not and cannot be verified earlier) or if some other custom selector prevents the form from being selected. Closes #6510921

[server/test] bit of PEP8

[migration] don't attempt to carry over values when renaming a computed relation Without the fix in migractions.py, the introduced test crashes with an operational error because of an attempt to write to a non-existing relation table (computed relations are not materialized). Closes #6304946

consider .do_fti flag in index_entity method as this method may be used from outsite the class (eg in UpdateFTIHook), this is the proper way to consider it anyway. Closes #6206636

Added tag 3.19.13, debian/3.19.13-1, centos/3.19.13-1 for changeset f933a38d7ab5

[pkg] 3.19.13

[migration/3.21] add support for sqlserver Closes #7173702

moved_entities.extid may not be NULL

[server] use sqls_create_multicol_unique_index to index entities.extid SQL Server's UNIQUE is not standards compliant, it doesn't accept two NULL values. Related to #7173702.

[server] fix checkintegrity with pyodbc "msg % row" dies with "not enough arguments for format string" if row is a pyodbc.Row.

[schema] remove unique constraint on unlimited String attribute SQL Server doesn't allow indexing (and thus unique constraints) on nvarchar(max). Related to #7173702.

[sources/native] fix potential NameError (closes #7364249)

[py3k] unicode → six.text_type

[devtools/testlib] Port CubicWebTC.view() to py3k testlib.TestCase.set_description's argument will eventually be printed out to sys.stdout/stderr. On python 2, this object takes bytes-like objects (eg, str) whereas python 3's takes unicode-like objects and handles the encoding on its own. Thus the 'str' type can used in both versions to eventually display the description on screen.

[devtools/testlib] port 'fake_form' test to io.BytesIO

[dataimport] port ucsvreader tests to io.BytesIO

[py3k] io.BytesIO

[py3k] io.StringIO

Port cw.Binary to io.BytesIO And add plenty of tests too!

[py3k] execfile → exec

[py3k] file → open

[py3k] six.integer_types

[py3k] basestring → six.string_types

[py3k] import CookieJar using six.moves

[py3k] import urllib2 from six.moves

[py3k] import range using six.moves

[py3k] import queue using six.moves

[py3k] import filter using six.moves

[py3k] import zip using six.moves

[py3k] import SimpleCookie using six.moves

[py3k] import HTTP client constants and exceptions using six.moves

[py3k] import URL mangling functions using six.moves

[py3k] import 'pickle' using six.moves

[pkg] Add dependency on six 1.4.0

[tests] Replace use of deprecated TestCase.assert_

[predicates] Simplify range(len()) iteration

[dataimport] Use pickle.dump instead of dumps

[web/facet] Simplify reversed/sorted calls

[web/test] Use TestCase.setUpClass instead of setUpModule

Use isinstance() instead of type comparison

Unused imports

[py3k] tuple unpacking in function args

[py3k] ur'' is gone

[py3k] octals and long

[py3k] Fix raise with embedded traceback python 3 removed the raise syntax with 2 and 3 expressions.

[py3k] print function

[py3k] except as

[py3k] backtick to repr

[hooks/syncschema] make sure CWUniqueTogetherConstraintDelOp happens before CWConstraintDelOp SQLServer refuses to index an unlimited text column, so we should drop unique_together constraints (which imply an index) before we drop size constraints. Closes #5560601.

[hooks/syncschema] only call "ALTER TABLE" once when changing a size constraint Until now we would: - remove the old size constraint from the in-memory schema - call update_rdef_column which removes the size restriction from the column's type - add the new constraint object - call update_rdef_column which adds the size restriction back This breaks on SQL Server when the column is involved in an index (e.g. as part of a multi-column unique constraint), because in the intermediate stage the column's type is "nvarchar(max)", which is not indexable. Of course we must still detect the case where a size constraint is really dropped and update the db schema accordingly. Closes #5557633.

[syncschema] only add to the `synchronized` set if all possible updates have been done Otherwise a simple sync_schema(syncprops=False) turns the subsequent sync_schema() calls into a no-op. Closes #6510918.

[qunit] remove unused 'data_files' parameter Its use was removed when we stoppped pointing firefox at file:// web/test/jstests/ajax_url2.html has never actually been used, it's only been ever mentioned in a QUnit comment.

[urlpublish] RESTPathEvaluator now use vid_from_rset This avoid cases where vid may be unexpectedly overwritten. For instance when you define a custom vid for a mime type in VID_BY_MIMETYPE, it will currently be overriden by the URL publisher when you access to /<etype> with the proper Accept header. To do so, a new 'check_table' argument is introduced, which may cause bw compatibility problems if the function has been monkey-patched. Also pep8 a bit the tests. Closes #5705835

[server/sources] use decode_extid in datafeed

[server/sources] add decode_extid method avoids having the same dance in both eid_type_source variants

[server/repo] kill repo.pinfo() session.cnxset is dead, this can't work.

[web/sessions] simplify session cleanup session.cnx.check() is no longer a thing. It dates back to dbapi.

[server/repo] fix typo

[server/sources] AbstractSource.__ne__ Seems confusing to have __eq__ but not __ne__.

[server/sources] fix docstring for eid_type_source

[views] fix ProcessInformationView: SESSION_MANAGER can be None (closes #5753280) eg. with pyramid.

[server/migractions] drop mock_object It doesn't seem to have ever been used.

[test] kill unused lgc.testlib imports

[server/migractions] raising SystemExit sounds like a bad idea Simply skip the action (rename_relation_type) with an early return.

[server/storage] rename 'session' variables to 'cnx' We no longer deal with Session objects here.

[migration] remove repo_connect and use config.repository() instead One less more redundant path to the repo. Related to #3933480.

[repoapi, cwconfig] give a convenience method to get a repository from a config This is a very common need. The private repoapi._get_inmemory_repo becomes config.repository(...)

[cwconfig] remove unused method Not to be confused with serverconfig.migration_handler.

[captcha] Port to io.BytesIO

[web/sessions] the session managers are definitely not components Component (or AppObjects) take a _cw at __init__ time. The session managers want a repository. Related to #1381328.

[web] the AuthenticationManager is no more an appobject It is now explictly instanciated by the session manager. Related to #1381328.

[hooksmanager] remove very old unused module

[devtools] fix CubicWebTC.assertDocTestFile (closes #5930028)

Added tag 3.21.1, debian/3.21.1-1, centos/3.21.1-1 for changeset a8a0de0298a5

[pkg] Let dh_installdocs deal with doc-base

[pkg] various debian packaging fixes - stop using *.install.in files, just use wildcards in *.install - switch to X-Python-Version (preferred for dh_python2) - also call dh_python2 for /usr/share/cubicweb, not just public modules - drop long obsolete ginco and erudi migration code from cubicweb-ctl.postinst - fix cubicweb-ctl cleanup: purge code belongs in postrm, not prerm - don't compress changelog.html

[pkg] 3.21.1

[web] fix AbstractSessionManager.clean_sessions (closes #5757240) sessions were not closed if "last_usage_time = session.cnx.check()" actually raises an AtributeError.

[migration/3.21] ask fewer questions Several questions per table is just unreasonable.

[server] typo fix

[test] check that we reject an rqlexpression in computed relation perms

Fix (de)serialization of ComputedRelation read permissions For normal relation types, permissions don't need to be stored since they're just default values for the relation definitions. However, computed relations are serialized (as CWComputedRType), while their relation definitions are added at schema finalization time, and are only in memory. So add the 'read_permission' relation to CWComputedRType, and the appropriate hooks to save and restore those permissions. To avoid having to touch yams, we drop the 'add' and 'delete' permissions from the default computed relation permissions; this should probably be backported there. The actual permissions (set on the relation definitions) are hardcoded in finalize_computed_relations anyway. In deserialize_schema, the CWComputedRType handling needs to be delayed a little bit, until after we've called deserialize_ertype_permissions. The rql2sql test is adjusted because CWComputedRType has a 'name' attribute and the 'read_permission' relation, which generates ambiguity vs CWEType. We add an explicit CubicWebRelationSchema.check_permission_definitions, since we need to check both that computed and non-computed rtypes are defined properly. Based on report and initial patch from Christophe de Vienne (thanks!). Closes #5706307

Copy read permissions on ComputedRelation rdefs Closes #5694110

[datafeed] fix typo in DataFeedXMLParser.is_deleted (closes #5729755)

[server/sources] make sure entity._cw is a Connection before calling Storages The storage callbacks don't get an explicit Connection objects, so they go through entity._cw to access the repo and schedule operations. Since that entity object comes out of the entity cache, its _cw may be a web request instead of a Connection. So fix it up around the storage callbacks. Closes #5753543

[hooks] base64-encode extid before inserting it into moved_entities If we insert the wrong value we're never going to find these entities again... Closes #5741105.

[hook] remove assumption about entity cache vs cw_edited We were assuming that if an entity is in the connection's cache, it has a cw_edited attribute. This doesn't seem to always be the case. At least in cubicweb 3.21, merging the web and repo-side caches makes this more likely, but Denis says he's also seen this on 3.20.

[hook] don't open-code Connection.entity_cache

[tox] Add missing dependency on webtest for devtools tests

[tox] Add missing dependency on webtest for wsgi tests

[tox] Remove set of PATH environment variable The : concatenation does not work actually so the resulting PATH was just wrong (it did not contain the previous PATH). Anyways, the PATH environment variable is passed by default by tox so it's probably better to let the caller set it (thus avoiding to hard-code the specific postgres version and path). http://tox.readthedocs.org/en/latest/example/basic.html#passing-down-environment-variables

[doc] generalize info about debian/ubuntu repositories That section was not kept up to date with recent distro releases; stop trying.

[doc] fix link to setup info in README ... and fix URL to the repository's gpg key.

[server/session] remove unused cnx_meth function

[server] don't issue a warning for something expected Creating an instance happens all the time in tests, there's no reason to warn every time.

[server] stick a _cw on internal connnection's user Entity._fetch_restrictions goes through user._cw.vreg :( Closes #5705769

[migration/3.21] fix stupid error in migration script str.join doesn't like a list of ints. Closes #5706359.

Added tag 3.21.0, debian/3.21.0-1, centos/3.21.0-1 for changeset 887c6eef8077

[pkg] various fixes - add missing files to MANIFEST - exclude doc/changes from dh_installchangelogs (it's a directory, which dh_installchangelogs doesn't expect) - fix doc build in debian package

[doc] fix broken symlinks

[pkg] update 3.21.0 release date

[doc/3.21] mention the postgresql-only state of the upgrade script

[datafeed] also catch EnvironmentError when trying to load the cwclientlib config file means the config file has not been found, so fall back to the old implementation instead of crashing.

[schema] Make syntax compatible with print_function yams seems to exec() us sometimes (instead of __import__()), and yams.reader uses print_function nowadays.

[pkg] 3.21.0

[test] use FakeFile instead of File in test_fulltextindex File.data is no longer fulltextindexed.

i18n update

[serverctl] fix schema-diff command It hadn't been ported away from dbapi.

[serverct] call server_{maintenance,shutdown} hooks around source-sync command

[serverctl] call server_{maintenance,shutdown} hooks around add-source

Update hgignore

delete unmaintained clone_deps script Closes #2517580.

merge 3.20.9 into 3.21 Fix conflict in unittest_postgres.py by changing sources configuration in PostgresTimeoutConfiguration.__init__ so it happens *after* setUpModule().

Added tag 3.20.9, debian/3.20.9-1, centos/3.20.9-1 for changeset d477e6447582

[pkg] 3.20.9

[forms] don't display 'remove' link of inlined form for the first entity related using a '+' cardinality to its parent entity Closes #5227394

[tox] Update doc testenv to account for documentation restructuring * Build doc from `doc` directory. * Use `sphinx-build` directly instead of `make` as this now works and allows to build within .tox temporary environment (instead of doc/_build), preferable for a "test" action. Related to #5447161.

[schema] drop old backward compat code preventing yams to be itself fully cleaned

[datafeed] allow to use cwclientlib for datafeed's queries (closes #5456849) HTTP based URLs will be handled by a CWProxy if: - cwclientlib is available and - the URL matches a cwclientlibrc config entry Otherwise, fallback to previous implementation. See https://www.cubicweb.org/project/cwclientlib :Warning: This comes with an API modification of DataFeedParser.retrieve_url: it used to accept data and headers arguments to build GET or POST (if data was given) queries, but this was not used by any known code and imply more complicated code.

[cwxml parser] refactor url handling code to use urlparse and urlencode (related to #5456849)

add use_extid_as_cwuri ext entity transform, that will be often necessary and not so easy to write at once

[dataimport] dispatch and deprecate old code previously, the whole dataimport.py content has been copied in dataimport/__init__.py, then dataimport/importer.py API has been introduced. Following these changes, this cset: * dispatch __init__ code into sub-modules * dispatch unittest_dataimport accordingly * deprecates functions and classes that are deprecated by the new API Related to #5414753.

Add tox.ini file * Factor out `commands` for most environments using {envname} substitution to build pytest commande. * Positional arguments are passed on to `pytest` (others may also be passed after a `--`). * Set PATH env variable for tests requiring PostgreSQL. * Also added a `doc` target to build the book. Also update .hgignore exclude .tox directory (where virtualenvs will be created and cubicweb.egg-info created by sdist command). Closes #5447161.

Add requirements.txt files in test directories These files, located in each test directory (when needed), list test dependencies supplementary to install requirements. Some requirements.txt files include psycopg2 because it won't get installed by logilab-database apparently. Related to #5447161.

Spelling fixes in comments and docstrings

[web/views] Clean up confusing backwards compatibility code Dead code since fbb370195199, related to #4943392.

[server/test] avoid lgdb warnings from str to unicode conversion

[web/test] Modernize unittest_propertysheet.py - remove call to os.makedirs in global scope - fold CACHEDIR as an instance attribute - bonus: use unittest directly instead of lgc.testlib

[web/webconfig] Drop pre-3.9 backward compatibility method "has_resource"

[web] Stop using negative indices with os.stat results The python doc says: More items may be added at the end by some implementations. .st_mtime is also more readable than [-2].

[doc] small rst fixes and cleanups in changelog files Related to #4832808

[doc] make the changelog chapter a one html page Related to #4832808

[doc] reformat changes files headers to remove the "what's new" phrasing and add the publication date instead. Related to #4832808

[doc] remove mentions of pyro and reposity instance type from the concepts chapter Related to #4832808

[doc] put the tutorials between the intro and the `Repository development` toc of the book Related to #4832808

[doc] move the documentations from docstring directly to the book. and improve a bit API documentations. Related to #4832808

[predicates] remove deprecated sentences in docstrings

[predicates] objectify the anonymous_user predicate for the sake of consistency.

[doc] more fixes of warnings/errors in doc build - move 3.21.rst changelog description at it's proper place - include changelogs in the main index - few typos - add :noindex: for autoxxx directives not in api/ Related to #4832808

[doc] fix warnings/errors in doc build - fix links to images - fix a couple of typos - re-add IDownloadableOneLineView doc - rename documenting.rst back to .txt, it's intended as a doc of how to write rst, not part of the rst doc Related to #4832808

[doc] Start documenting the API Related to #4832808

[doc] Remove useless indexes Related to #4832808

[doc] Insert the dev docs in the toc Related to #4832808

[doc] Restructure the documentation * Create a new index file * Move the sphinx configuration files do the documentation root * Move book/README to dev/documenting.rst * Move book/mode_plan.py to tools/ * Move book/en/images to images * Move book/en/* to book/ * Move changelogs to changes/* * Adapt the Makefile * Add a title to the javascript api index Related to #4832808

[migration/3.21.0] performance and reliability fixes - use EXCEPT instead of NOT IN, seems to improve performance dramatically on postgresql - delete rows from cw_<etype> tables whose eid is unknown - deal with missing/dangling required inline relations (by deleting the subject entities) instead of asserting - fix/improve print statements - don't ask for confirmation before running pure select queries

[serverctl] fix source-sync command Closes #5491700

[server] missing import Fixes NameError in an unlikely error case.

[server/test] stop using email cube - make unittest_migractions use a separate datadir, and add a fake email cube there (we use it to test {add,drop}_cube, so need something that looks like a cube) - for the other test modules, just copy what we need from cubicweb-email's schema to our test schema

[web/views] Remove unused variable

[server/session] bring anonymous_connection back Fixes cubicweb-comment.

[migration/3.21] idempotency fixes Drop constraints before (re)creating them This is postgresql-specific syntax, sqlserver needs work.

[migration/3.21] Make index and table creation idempotent

[migration/3.21] Add missing foreign keys when upgrading Try to delete dangling relations, since there's no point in keeping them. Related to #4846892.

[schema2sql] properly consider skip_relations parameters. This is the same bug as yams #286912. Also, introducing a unique function telling if the relation should have a table or not unhide some other problems related to computed relation, or missing skipped relations information.

[hooks/test] don't rely on RelationDefinition implicitly Its presence in globals is deprecated and going away.

[web] write css_class to attrs on rendering and not on button creation Because css_class attribute may be overriden by bootstrap after button creation and this modification has to be taken into account. closes #5457548

[statsd_logger] handle ipv6 addresses

implement a simple statsd logger (closes #5488711) and add a few timing probes on: - etwist.server.CubicWebRootResource.render_request() - server.querier.QuerierHelper.execute() - server.sources.native.NativeSQLSource: - sqlexec() - compile_rql() - authenticate() - doexec() - doexecmany() as well as a pair of counters on server.querier.QuerierHelper cache (hit/miss).

[predicates] ExpectedValuePredicate now accepts a dict parameter ``match_form_params`` and ``match_kwargs`` benefit from that. For instance, the following statements are valid:: __select__ = match_form_params('vid', 'subvid') __select__ = match_form_params(vid='list', subvid='incontext') __select__ = match_form_params(vid=('list', 'tsearch')) In the latter cases, not only the parameters (``vid``/``subvid``) must be in the request form but their corresponding value must also match the expected values (or one of possible choices if `values` is a sequence). closes #5484070

[rql2sql] Fix iter_exists_sols() excessive cleaning of _state.tables Closes #5503548

[cwctl] allow overriding config settings from the command line (closes #5557656) eg. cubicweb-ctl start -p port:8082 myapp to overload the port parameter defined in the all-in-one.conf file.

avoid sanitizing warnings

[devtools] avoid warning about source['global-db-name'] sqlite and postgres db handlers store the db template name there.

[skeleton] remove no-break space from skeleton setup.py

[server/test] drop unused import

[doc] update 3.21 release notes

[devtools] Sort _cw_fields in fake_form Try to generate a predictable string, to help testing with PYTHONHASHSEED=random.

[entity] sort fetchattrs when generating rql Try to generate a predictable rql query, to help testing with PYTHONHASHSEED=random.

Update hgignore for temporary test databases

[js] removed unused and broken SuggestForm Widget (closes #5213487)

[test] Fix unittest_i18n to run properly with "pytest -t" (closes #5576169) When running "pytest -t", many tests modules will be imported and run within a single python process. unittest_i18n fails when run after tests such as unittest_webtest.py or unittest_httptest.py. The main reason is that unittest_i18n does various tricks to add a test cube to the search path, which badly fails when a previous test module has already loaded "stuff" (for lack of a better word). Instead, let's just call the 'i18ncube' command with subprocess. The change from logilab.common.testlib to unittest is not directly needed, but comes as a free cleanup with the above changes.

[server/test] fix random error in unittest_security When entering a new Connection, we cache the user and its 'login' attribute (with no permissions checking). This test makes 'CWUser.login' unreadable by guests, and then proceeds to make sure the 'anon' user can actually not read any 'login' attribute. However, due to the above cnx initialization, anon's login is actually cached, hence readable. This happens to make the test fail sometimes depending on the order in which CWUser entities are returned, because one of them has .complete() called, which as a side effect sets the attribute cache to None for unreadable attributes. Call .complete() on both entities to reset the login cache. While this is still highly debatable, at least it's consistent.

[entity] fix typo in comment

[doc] Add a tutorial and extend documentation for ExtEntityImporter Related to #5414753.

[dataimport] introduce the importer and extentity classes This introduces the ``ExtEntity`` class which is a transitional state between data at external source and the actual CubicWeb entities. ``ExtEntitiesImporter`` is then in charge to turn a bunch of ext entities into CW entities in repository, using a given store. This changeset also introduces ``SimpleImportLog`` and ``HTMLImportLog`` which implement the CW DataImportLog interface in order to show log messages in UI using simple text and HTML formats respectively, instead of storing these messages in database. Both have mostly been backported from cubes.skos.dataimport. Closes #5414753.

[test] Use store.prepare_insert_relation instead of deprecated relate method Follow-up for 1f5026e7d848. Related to #5040344.

[dataimport] since we have foreign key constraints in the database, 'entities' table should be updated first

[dataimport] Move stores to new API. Here is the final store API: * ``prepare_insert_entity(<entity type>, **kwargs) -> eid``: given an entity type, attributes and inlined relations, return the eid of the entity to be inserted, *with no guarantee that anything has been inserted in database*, * ``prepare_update_entity(<entity type>, eid, **kwargs) -> None``: given an entity type and eid, promise for update given attributes and inlined relations *with no guarantee that anything has been inserted in database*, * ``prepare_insert_relation(eid_from, rtype, eid_to) -> None``: indicate that a relation ``rtype`` should be added between entities with eids ``eid_from`` and ``eid_to``. Similarly to ``prepare_insert_entity()``, *there is no guarantee that the relation will be inserted in database*, * ``flush() -> None``: flush any temporary data to database. May be called several times during an import, * ``finish() -> None``: additional stuff to do after import is terminated. **Warning:** ``prepare_update_entity()`` still needs to be implemented for NoHookRQLObjectStore. Related to #5040344

[js] Using appropriate http method when calling a remote function to respect REST principles. closes #5355952

[server/test] Add a test for db-statement-timeout option Sadly I don't think there's a way to test that functionality without adding sleeps which make the test suite longer :/ Related to #2547026

[datafeed] give an error message if a source is missing a parser id Closes #3484231

[storage] use mkstemp to create files in bfss Avoids race condition, at the cost of slightly less predictable file names.

[test] Skip spa2rql tests when fyzz is not available Closes #5456750.

[web/staticcontrollers] fix docstring spelling

[server/rql2sql] fix spelling in comment Not that I understand what that sentence means.

[doc/book] fix rql syntax examples "Any", not "ANY". "EXISTS" is a function.

[migration] add sql constraints on upgrade Related to #5154406.

[schema] neuter check method for constraints we implement in sql Closes #5154406.

add IUserFriendlyError adapter for violation of check constraints This way we get back the same error messages we get from the python check. Related to #5154406

[hooks/syncschema] drop/add check constraints as appropriate Related to #5154406

[schema2sql] support NOW and TODAY in check constraints Related to #5154406.

[schema2sql] insert some constraints into the backend BoundaryConstraint, IntervalBoundConstraint and StaticVocabularyConstraint can be implemented in SQL with CHECK constraints. Next steps, not part of this changeset: - migrations - removing no longer necessary pre-insertion work on the python side - translating backend exception due to constraint violation into ValidationError Note: this means that these constraints can no longer be ignored by disabling integrity hooks. Related to #5154406

[test] run unittest_syncschema with postgresql Lets us test more things, since we can actually do ALTER TABLE.

[test] run unittest_migractions on postgresql sqlite's lack of support for a bunch of alter table functionality means some thing just don't get tested otherwise.

[devtools] Put pg sockets in /tmp Makes for shorter paths (limited by sockaddr_un). Also use the default port, since we're not listening on TCP sockets.

[devtools] change the way we start/stop postgresql Instead of having the test db handler start a cluster on demand, use the test module's setUp/tearDown callbacks to do it. This allows to have one data directory (and thus cluster) per test module, allowing different test modules to run in parallel, each using its own database cluster whose path is based on the test module.

[devtools] make test db name for postgresql process-specific In case we run two tests concurrently on the same cluster, better not use the same db name for both.

[devtools] extract functions to start/stop a postgresql cluster

[test] don't leave NULL columns around when making an attribute required It doesn't matter on sqlite (it doesn't do ALTER COLUMN), but when running this test on postgresql it fails to add the 'NOT NULL' constraint otherwise.

[server/session] Roll back in connection exit This was changed in commit 8b35a898b334 "[server] remove cnxset tracking, it is now unneeded". When exiting a CubicWeb Connection, the database connection is put back into the common pool. At that point anything done during the lifetime of the CubicWeb Connection needs to be committed or rolled back. Do a proper rollback to finish the transaction, as was happening before the above-mentionned commit. This showed up on testing with postgresql, when a DB schema modification was blocked by an old connection that was still "in transaction".

[web/views/formrenderer] do not use `cubicweb:target` attribute on form (closes #5534074) The `cubicweb:target` is a flag indicating that the form data (with file input) shall be posted inside an iframe - this is a well known "ajax-like" workaround to post files with browsers that do not support `FormData` (a.k.a IE<10). The `cubicweb:target` was introduced when CW used the "xhtml strict" doctype. Now that CW uses the "html5" doctype, this namespaced attribute is no longer necessary and the iframe can be generated directly. Before this patch, CW inserts the `cubicweb:target` attribute in the form DOM element (server side) and `setFormsTarget()` updates the DOM with a new `<iframe>` element (client side). Now, CW inserts the `<iframe>` DOM element directly (server side), making `setFormsTarget` useless.

[js] lint : make it explicit that typeof is an operator, not a function. Also, use === / !== except for null / undefined comparison

[devtools] qunit: show helpful data when an assertion fails Related to #5533333.

[web/test] Use shorter $() instead of jQuery() Consistency FTW. Related to #5533333.

[devtools] qunit: use new async testing APIs http://qunitjs.com/cookbook/#asynchronous-callbacks QUnit keeps track of all the assert.async() objects created inside the test functions and expects all done() functions to be called. Failure to do so will result in the test being failed. Unlike .start and .stop which were internal APIs, assert.async() is stricter and fails tests if assert methods are used *after* all done() functions are called (see "test callback execution order"). Related to #5533333.

[devtools] qunit: use new assert APIs test functions now accept an "assert" argument which exposes QUnit's assert methods: assert.{expect,equal,deepEqual,ok} Related to #5533333.

[devtools] qunit: stop using global variables test -> QUnit.test module -> QUnit.module Related to #5533333.

[devtools] qunit: update to 1.18.0 (closes #5533333) * the logging API has changed (register callbacks instead of setting attributes on the QUnit object) * the logging callbacks all have a single "details" object argument * the "main" div was renamed to "qunit-fixture" (this change is responsible for all the churn in our own JS test files) * QUnit can now build the DOM elements it needs for reporting from a single "qunit" div

[devtools] qunit: stop using deprecated assert functions equals -> equal same -> deepEqual Related to #5533333.

[devtools] qunit: remove useless window.close() Modern browsers don't allow it anymore. Related to #5533333.

[devtools] qunit: refactor functions into QUnitView Related to #5533333.

[devtools] qunit: only start firefox once per test The separate profile initialization step doesn't seem necessary anymore. This was due to the system-wide installation of the "Sage" Firefox extensions which breaks -url on a fresh profile. Removing this obnoxious extension allows us to launch Firefox in a standard way. Related to #5533333.

[devtools] qunit: decrease test timeout Related to #5533333.

[devtools] qunit: increase resolution and depth of Xvfb screen Related to #5533333.

[web/test] Always call QUnit.ok with a message Related to #5533333.

[web/test] Fix remaining JS tests (closes #5533303)

[web/test] Stop testing that exceptions raised in Deferred callbacks call errback They no longer do as of changeset 7f1f7f710b16. Related to #5533303.

[web/test] stop calling loadxhtml with a form.callback It was deprecated in 3.9 (changeset 7b9553a9db65) and removed in 3.18. Related to #5533303.

[web/test] Fix reset of loaded JS/CSS files Name changed in commit ede740bd7077. Related to #5533303.

[web/test] Use proper JSON syntax Related to #5533303.

[web/test] Update mock ajax reply from "render" ajaxfunc CubicWeb transforms add_js() calls in AJAX context to <cubicweb:script> tags since ede740bd7077 and cd5738fc440f. Related to #5533303.

[devtools] Serve JS tests over HTTP (closes #5533285) Instead of mixing file:// URLs with XHR.

[web/data] Support Deferred objects outside XHR JS tests actually use this. Related to #5533303.

merge 3.20.8 in 3.21

[views] Revert previous changeset 8f05aaabf355 Breaks cubicweb/web/test/test_views.py as some views don't have a 'vid' kwarg.

Added tag 3.20.8, debian/3.20.8-1, centos/3.20.8-1 for changeset ec284980ed9e

[pkg] Prepare 3.20.8

[js] simplify cw.log implementation There's no need to rebuild and array and join its content, just call console.log with ``arguments``. The only trick is that console.log must be called on a console instance.

[web/views] don't force http redirect on logout We no longer forbid anonymous browsing on https, so we can keep the same scheme when logged out. Closes #5507479.

[views/schema] extract a hardcoded css class attribute

[formwidgets] move TextArea constants to the class level This will make it easy to override them.

[server/test] fix broken test The first bad revision is: changeset: 10396:82071f767cb8 user: Sylvain Thénault <sylvain.thenault@logilab.fr> date: Thu Jun 11 17:35:11 2015 +0200 summary: [schemas] cwuri should be read-only This changed the number of RQLExpressions to be more than the number of CWRelations in the test's schema. Related to #5457566.

[test] typo in entities test comment

[cwconfig] typo in comment

[serverctl] allow changing the log level for source-sync command When calling this from cron we only want to see output if something fails.

[web, urlrewrite] remove unused `cachekey` argument from `rgx_action` (closes #5355967) this argument prevent from generating the resulting rset descripition

[web] Fix the path to the `static directory` (closes #5355965)

[web/json] Controller.publish must return encoded data We were mixing binary from super() and text from our params, thus wrongly returning text. Closes #5507573.

[entities] fix dc_title output for bool(value) == False Closes #5194853.

[hooks] manhandle the parsed rql query, not its string representation ... and add a groupby clause to make it actually work (the selection might well use an aggregate function, so we need to tell the backend to group on our target entity). Closes #5362574

merge 3.19.12 in 3.20

Added tag 3.19.12, debian/3.19.12-1, centos/3.19.12-1 for changeset 5932de3d50bf