[dataimport] remove replace_sep parameter from massive store It is unused since changeset efbbf1e93a04 "[dataimport] Properly escape strings sent to COPY FROM"

[dataimport] remove pg_schema parameter from massive store It can be recovered from the system source's configuration.

[dataimport] remove drop_index parameter from massive store "drop_index=False" also implied not dropping any constraints, in particular foreign keys, which meant any attempt to import entities would fail, because we only add metadata (the entities table) after the entity's insertion.

[dataimport/test] update call to ucsvreader Silences warning: [3.20] 'separator' kwarg is deprecated, use 'delimiter' instead

[dataimport] remove deprecated methods from massive store This code is new in cubicweb, no reason for it to carry deprecated stuff from the start.

[dataimport/test] use store.finish instead of deprecated store.cleanup

[dataimport] drop no more used parameter on massive store

[dataimport] methods that modify in-place shouldn't return value Also mark those methods private and drop some useless comments in passing. Related to #5414760

[dataimport] implement new store API on massive store and deprecate the old one. Related to #5414760

[dataimport] .finish method was missing from (NoHook)RQLObjectStore

[massive store] identity_relation should not be considered, is_relation seems missing The "identity_relation" table should not even exist.

[dataimport/test] use the right assert methods instead of assertTrue with a comparison

[dataimport] import range from six.moves

[dataimport/test] feed binary data to ucsvreader

[dataimport/test] fix broken comparison We want to compare two ints, not an int and a db row

[dataimport] dict.iteritems() → dict.items()

[dataimport] use next builtin instead of next method on iterators

[dataimport] import StringIO from io

[dataimport] backport massive store from dataio cube Some tweaks have been made to work with 3.21 foreign key constraints. Also test are not executed in our own pg cluster instead of polluting the system one. Closes #5414760

[web] set proper Cache-Control header for static files - set Cache-Control max-age to match Expires - use the max_age method (defaults to 1 week) instead of hardcoded 6 months

[web] use a redirect for /data/foo requests that don't use the instance md5 hash The actual file contents can be cached for a long time. If the request doesn't use the hash, that's wrong. Using a redirect lets keep the caching for the actual file contents while not breaking requests missing the hash.

[web] fix typo in doc string

[web] stop adding an Expires header with all responses As per RFC 1945 (HTTP 1.0): If the date given is equal to or earlier than the value of the Date header, the recipient must not cache the enclosed entity. As per RFC 7234 (HTTP 1.1 Caching): If a response includes a Cache-Control field with the max-age directive (Section 5.2.2.8), a recipient MUST ignore the Expires field. Bottom line, the Expires header is now handled wherever the Cache-Control is handled: inside cache managers.

[web] stop looking at a request's Cache-Control header As per RFC 7234 (HTTP 1.1 Caching): The "Cache-Control" header field is used to specify directives for caches along the request/response chain. CubicWeb is not a cache.

[fti] add cw_fti_index_rql_limit method Improves on and deprecate cw_fti_index_rql_queries: execute the rql directly, so that we don't need to compute the set of eids up-front, but can just keep track of the last seen entity. Use the new method in reindex_entities. Keep calling cw_fti_index_rql_queries if it has been redefined (e.g in cubes). Related to #3621392.

[fti] backout 166c6f7b1be4 * breaks tests * see next commit for a better solution

[py3k] Fix unicode added in commit 2350424585f8

[server/ldap] deal with unicode or str coming out of ldap It seems we can get either, I've no idea in which exact circumstances, so this should let us work with both.

[test] pull yapps from Logilab's github Fixes a couple of failures, pull request is almost a year old (https://github.com/smurfix/yapps/pull/2).

[pkg] bump yams requirement to 0.41.1 Needed by changeset c9321aedfbf9.

[entities] import range from six.moves

merge head published by accident

[fti] Improve big table reindexation * Slice the reindexing in batches of 1000 entities. * Make the output more verbose. Closes #3621392

[tox] Reference requirements.txt for hooks environment The file existed but was not referenced.

[devtools/testlib] Remove assertions about direct usage of CubicWebTC/AutomaticWebTest The point is to be able to use a test runner other than pytest (e.g. nose2, nosetests or py.test) to run CubicWeb test suite. The issue is that most other test loaders inspects members of all base classes of test case classes to determine test methods and it hits this `config` property (actually executing it), which breaks the assertion. So just return None instead of failing. Closes #7817147.

Add ldap3 to server test requirements

[pkg] bump version number so tox won't use cw from pypi

[schema] add CubicWebRelationDefinitionSchema Stop monkeypatching yams, inherit from RelationDefinitionSchema instead.

[schema] Let RQLConstraint inherit from BaseRQLConstraint instead of RQLVocabularyConstraint There is apparently no reason for RQLConstraint to inherit from RQLVocabularyConstraint as it gets its `repo_check` method from `RepoEnforcedRQLConstraintMixIn` and RQLVocabularyConstraint (which inherits from BaseRQLConstraint) actually only overrides that method. So simplify inheritance by removing a useless level of indirection. Adjust cw_unrelated_rql accordingly, where it was that RQLConstraint inherited from RQLVocabularyConstraint.

[server] unormalize source logger's name to avoid encoding errors

[entities] parameters should be given to fire_transition, else they are simply ignored

[entity] set _cw_adapters_cache in __init__ rather than on first use

fix typos and indentation in docstrings/docs

session.id → session.sessionid

[test] test page_size problem (related to #8123231)

[predicates] guard against non-integer page_size form parameter Closes #8123231

[server/test] add test for db-dump command

[server] fix db-dump with sqlite 'gzip -c dbname > backupfile' needs to go through the shell.

[serverctl] fix db-dump Since changeset af47954c1015 we would die with: AttributeError: 'ServerMigrationHelper' object has no attribute 'repo'

[server/test] sorting vs dict vs python3

[devtools] more sorted() vs dict vs python3

[web/application] return bytes from redirect_handler and validation_error_handler

[devtools] fix ResourceWarning: unclosed file tmpdb-__default_empty_db__.config

[serverconfig] fix ResourceWarning: unclosed file bootstrap_cubes

Use cubicweb._

[syncschema] hashlib.md5() wants bytes

[server/test] use next() builtin instead of next() method

[server/test] dict.iterkeys

[syncschema] don't arbitrarily remove constraints from the in-memory schema For some constraint types, it may make sense to have several constraints of that type simultaneously. In such a case, removing one of them from the in-memory schema is wrong. Closes #7446353

[server/test] don't leak timeout from one test class to the next Reset SQL_CONNECT_HOOKS after we're done with this test class so the next one doesn't inherit a timeout it doesn't want.

[server/test] CWAttribute.extra_props is Bytes

[dataimport] give unicode objects to psycopg2 copy_from

[dataimport] extid must be a bytes object

[dataimport] import StringIO from io

[dataimport] fix ucsvreader for python3 The csv module in python3 wants str, so use a StreamReader to feed csv.reader and don't decode the output

[server/test] import StringIO from io on python3 Don't change the import for python2, as we print a mix of unicode and str.

[server/datafeed] drop URLLibResponseAdapter.info I don't know where it's used

[devtools] pass a key to sort() method python3 dicts are not comparable

[postcreate] properly set is_instance_of for bootstrap entities Entities created before CWEType itself need the is/is_instance_of relations created later. We were actually only inserting "is". Closes #7656606.

[migration] re-try without print_function when compiling a script fails As part of the python3 porting we added the __future__.print_function import to cubicweb.migration. That started affecting all migration scripts it runs, which may not be prepared. So if compiling a script results in SyntaxError, try again without the print_function flag. Closes #7935030

[test] do not overwrite user's CW_CUBES_PATH in unittest_migration Currently when you rely on your CW_CUBES_PATH environment variable to locate cubes necessary for cubicweb's tests, the migration test crashes because it overrides the env var.

[server/test] dict.itervalues() → iter(dict.values())

make sync_schema_props_perms(<computed rtype>) work as expected It currently ends up with an ExecutionError while we want to synchronize permissions. This makes 3.21.1 migration of any application using computed relation crashing. Also, remove pre 3.6 compat code in hooks that prevent living schema update on computed relation's permission changes.

[web/http_headers] use classmethod decorator

[schema] use classmethod decorator

[devtools] delete postgresql socket directory after killing the server ... or when failing to start it.

[skel] stop deleting cubes egg-info It doesn't hurt and provides some metadata.

[dataimport] Update doc to reflect new API relate() looks deprecated.

[server] fix generate_source_config regression on python2 lgc outputs "str" objects on python2 and python3, so we can't just use io.StringIO (which expects unicode objects). Fixes: c36cda9074c5

[tests] Fix config.repository() in tests The config.repository was overriden too late. Closes #5778294

[schema] RQL queries are always unicode objects in python 3 fixes commit 42c5bd7286b7.

[server/test] use relative import Fixes py3k breakage

[server/schemaserial] fix extra_props on py3k json.dumps returns unicode, and json.load eats unicode, so add explicit encoding/decoding.

[server/test] unicode → six.text_type

[web] fix Field.__str__ on py3k

[web] unicode → six.text_type

[web/test] CubicWebTC.view returns bytes

[uilib] fix traceback on python3

[web] integer division

[web/test] Binary eats bytes

[devtools] make httptest module importable under py3k

[web/test] CubicWebPublisher.handle_request returns bytes

[web] base64 works on bytes not str (py3k)

[web] accept unicode in parsed Authorization header

[web] CubicWebPublisher.ajax_error_handler must return bytes

[py3k] fix i18n

Fix md5crypt and crypt_password test for python3 Make sure to always work with bytes for hashing.

[server/test] use bytes for extid

[devtools/test] firstnames.txt is in latin1

[web/test] unicode → cubicweb._

[web/formwidgets] fix for py3k no idea why we encode that key...

[sobjects] unicode → six.text_type

[server] fix unittest_security for py3k - use source.binary_to_str instead of str(), helps when the backend returns memoryviews - fixup str vs bytes

[py3k] make ldapfeed tests pass

[server] import filter from six.moves

[ldapfeed] port to ldap3 python-ldap has no python3 support. XXX I only checked that unittest_ldapfeed passes.

[server/test] str vs bytes

[server/test] list() around map and zip

[server/test] to fold, unicode → six.text_type

[server/test] delete test that doesn't make sense on python3 XXX if PY2?

[server/test] unicode → six.text_type

[server] unicode → six.text_type

[py3k] fix building rset description for INSERT queries

[datafeed] use explicit encode/decode instead of casting to str

[py3k] io.BytesIO

[py3k] get urllib foo from six.moves

[devtools] CubicWebTC._check_html may have to deal with unicode or bytes If called with template=None, CubicWebTC.view will not go through main-template, and return the unicode data from the view directly instead of encoded data.

[ext/test] fix test relying on repr(text)

[req] fix url_parse_qsl on py3 should be folded with previous conversion

[server/session] unicode → six.text_type

[web/test] str vs bytes

[web/ajax] fix AjaxController on python3

[web/test] str → bytes

[wsgi/test] use bytes instead of str for POST body

[devtools] 'dict_values' object does not support indexing

[multipart] decode form data before calling parse_qs in python 3 parse_qs eats and returns 'str' on both python versions.

[multipart] unicode → six.text_type

[web] assert that CubicWebPublisher.handle_request returns bytes

[py3k] fix validateform controller We need to return bytes not str.

[web] unicode → six.text_type

[multipart] make sure form keys are not bytes

[web] unicode → six.text_type

[wsgi] don't convert form values This is done earlier in multipart.py

[server] import input from six.moves

[server] don't convert user.login to bytes to generate sessionid

[cwctl] stop playing tricks with std{out,err} buffering Python3 doesn't allow unbuffered text I/O, because it doesn't make sense.

[py3k] CubicWebTC.app_handle_request returns bytes

[py3k] unicode → six.text_type

[py3k] http headers should be ascii-only unicode objects

[py3k] Clarify IDownloadable expected types

[py3k] IDownloadable.download_data() returns bytes

[py3k] unicode → six.text_type

[py3k] unicode → six.text_type

[py3k] unicode → six.text_type

[py3k] unicode → six.text_type

[py3k] PageInfo.source contains bytes

[py3k] unicode → six.text_type

[py3k] unicode → six.text_type

[py3k] Don't encode URLs Python 3's urllib works with unicode objects.

[tests] Stop counting pickle bytes in ResultSet tests Between python 2's pickle and cPickle implementations, the various protocol versions added in python 3, the length of the pickled string is completely unpredictable. Instead, do a simple round trip and test a few attributes.

[tests] Port unittest_cwctl to py3k On python 2, sys.stdout takes bytes-like objects whereas python 3's takes unicode-like objects and handles the encoding on its own.

[migration] NoneType is no longer comparable in py3k

Rename gettext to cwgettext, use stdlib gettext cubicweb/gettext.py was a copy of Python 2.6's gettext.py with the context patch from bug https://bugs.python.org/issue2504 Instead of keeping that full copy, just inherit from gettext.GNUTranslations and add the context-related stuff.

[py3k] MethodType no longer takes a class argument

[py3k] unicode → six.text_type

[py3k] im_self → __self__

[py3k] Don't sort Action objects In python 3, arbitrary objects are no longer sortable.

[py3k] use inspect module to get a function's arguments

[py3k] __unicode__ and __str__

[test] call rql.parse with unicode objects Allows to get same results from rqlst.as_string() in python 2 and 3

[py3k] replace cmp with key in sorted() And stop sorting dicts in snippet_key()

[tests] remove useless sort Useless in python 2, actively harmful in python 3 where arbitrary objects are no longer sortable.

[py3k] dict methods no longer return indexable objects

[py3k] unicode → six.text_type

[py3k] unicode → six.text_type

[server] Port BFSS to py3k The BFSS API changes in python 3: * 'defaultdir' MUST be a unicode object * 'fsencoding' MUST NOT be set In python 2, fsencoding handles both the encoding of file paths on the file system (utf-8 by default, but the system may actually be using something else) and the encoding of file paths that will be stored in the database. So in python 3, we wipe the slate clean: * rely on sys.getfilesystemencoding() to convert unicode objects to bytes * always encode paths to utf-8 for storage in the database Caveat emptor / here be dragons: * sys.getfilesystemencoding() depends on the current locale, which therefore MUST be set properly * when migrating an existing instance from py2 to py3, one MAY need to reencode file paths stored in the database

[py3k] Binary.getvalue() returns bytes

[py3k] ensure Binary objects are initialized with bytes

[py3k] __unicode__ and __str__

[py3k] unicode → six.text_type

[mail] port to py3k

[web/views] port JSON views to py3k

[py3k] unicode → six.text_type

[web/controller] req.form is supposed to contain unicode-data

[py3k] unicode → six.text_type

[py3k] unicode vs str vs bytes vs the world

[web/views] Port csvexport views to py3k There are a few hoops to jump through since python 3's csv module _is_ unicode aware.

[py3k] unicode vs str vs bytes vs the world

[web/request] don't convert request parameters if py3k

[py3k] unicode → six.text_type

[py3k] __getslice__ → __getitem__ with slice support

[py3k] __nonzero__ → __bool__

[py3k] unicode vs str vs bytes vs the world

[py3k] use six.add_metaclass

[py3k] unicode → six.text_type

[py3k] add list() around dict methods

[py3k] unicode → six.text_type

[py3k] unicode → six.text_type

[py3k] unicode → six.text_type

[py3k] unicode → six.text_type

[server/utils] crypt_password should always return bytes

[py3k] str → six.binary_type

[py3k] unicode → six.text_type

[server/utils] convert to py3k using six

[py3k] six.PY2

[py3k] use text/binary types from six

[py3k] unicode → six.text_type

[py3k] bytes vs str

[py3k] io.BytesIO

[py3k] unicode → six.text_type

[py3k] bytes required for hashlib.{md5,sha1}()

[wsgi/test] add __main__ section

[py3k] use next builtin instead of next method

[web/views] use next builtin instead of next method

[py3k] declare meta classes using six.add_metaclass

[py3k] replace '_ = unicode' in global scope (closes #7589459)

[py3k] introduce cubicweb._ (related to #7589459) The end goal is to stop relying on '_' being in the builtin scope. In the mean time, deprecate the builtin '_' so that consumers are warned.

[py3k] dict.iterkeys → dict

[py3k] dict.itervalues → dict.values

[py3k] dict.iteritems → dict.items

implement __ne__ whenever we implement __eq__

[test] use unicode for rql queries (7/7) Closes #6694446

[test] use unicode for rql queries (6/7) Related to #6694446

[test] use unicode for rql queries (5/7) Related to #6694446

[test] use unicode for rql queries (4/7) Related to #6694446

[test] use unicode for rql queries (3/7) Related to #6694446

[test] use unicode for rql queries (2/7) Related to #6694446

[test] use unicode for rql queries (1/7) Related to #6694446

[schema] improve normalization of RQLExpressions Parse and print back the expression instead of manipulating the string. Among other benefits, it means we don't mangle embedded string constants that contain commas or multiple spaces. Closes #6694426

[server/rql2sql] use VariableRef.is_equivalent explicitly instead of relying on __eq__ What we want to compare is the variable referenced by the VariableRef, we should make that explicit. Closes #6694406

merge with 3.21.2

[statsd] fix the statsd logger (closes #7558147) socket.inet_pton's second argument is the ip, not a couple (ip, port)

Added tag 3.21.2, debian/3.21.2-1, centos/3.21.2-1 for changeset a5428e1ab364

[pkg] 3.21.2

[pkg] Remove leftover dep on pyro (closes #7479155)

merge with 3.20.10

[migration] fix change_attribute_type to update the live schema Closes #7170830

[devtools] add has_cache for postgres (closes #5739624) devtools stores info about existing dbs in the db handler, but in the case of postgresql that doesn't take into account the path to the cluster's datadir. Which means if we run two test modules (in the same test run), we'll create a "__default_empty_db__" for the first one, cache its existence, and then when moving on to the other module, believe the template already exists (but since the datadir depends on the test module's path, it does not). This patch is a bit of a kludge, and it would be better to make the cache key include enough data to not need this, but I'm not sure how to do that.

fix bad-caching of datetime with tz info at sql generation time There is a special handling for datetime with tzinfo, where value was stored in the query cache. The implementation of merge_args was simply overwriting parameters of the query with those in the query cache, expecting no collision. To fix this: * handle replacement of tzinfo in merge_args, not at sql generation time * add an assertion to ensure we've actually no collision Closes #6978316

Added tag 3.20.10, debian/3.20.10-1, centos/3.20.10-1 for changeset 8f82e9523962

[pkg] 3.20.10

merge with 3.19.13

[devtools] Fix CubicWebTC.temporary_permissions If the context manager is exited via an exception, the original permissions should be restored.

[devtools] Use TestCase.assertGreater

[autoform] fix appearance of link to add inlined creation form On entity creation, if there are some local permissions on the relation, we have no way of checking them since neither the subject nor the object of the relation exists yet. In such a case, we should add the link by default, for consistency (see other places where we use `may_have_permission`). Closes #6711900

[predicates] Simplify specified_etype_implements

[web/views] Fix `has_editable_relation` predicate wrt inlined relations Prevents the "modify" action from being shown when there is nothing to edit.

[web/views] avoid propagation of NoSelectableObject in some case of inlined relations / permissions When selecting an inlined creation form, we should catch the NoSelectable exception that will be raised if the user cannot add entities of the target type (this is not and cannot be verified earlier) or if some other custom selector prevents the form from being selected. Closes #6510921

[server/test] bit of PEP8

[migration] don't attempt to carry over values when renaming a computed relation Without the fix in migractions.py, the introduced test crashes with an operational error because of an attempt to write to a non-existing relation table (computed relations are not materialized). Closes #6304946