[dataimport] remove size constraint handling We do not want to silently truncate data. If that is wanted, the application-specific importer should handle that itself. [v2: Keep the get_size_constraints function so applications don't have to reimplement it if they need to keep the truncate behaviour]

[dataimport] fix doc string to stop referring to the old API

[dataimport] drop duplicate init of MassiveObjectStore.sql

[dataimport] remove implicit commits from MassiveObjectStore Let the user decide when to commit the transaction.

[dataimport] drop extra indirection through MassiveObjectStore._initialized dict We already have self.__dict__, no need to add another one for a static set of keys.

[dataimport] remove remaining references to dataio from MassiveObjectStore In particular, rename metadata tables to live in the ^cw.* namespace.

[dataimport] rename logger from dataio.massiveimport to dataimport.massive_store

[dataimport] remove commit_at_flush parameter from MassiveObjectStore WARNING: the default value was True, but flush() now never commits.

[dataimport] remove autoflush_metadata from MassiveObjectStore parameters Hopefully not needed.

[dataimport] remove uri_param_name from MassiveObjectStore parameters It doesn't belong here.

[dataimport] turn iid_maxsize into a class attribute

[dataimport] turn eids_seq_{start,range} into class attributes Only the master-mode store will actually use eids_seq_start to reset the eid sequence.

[dataimport] remove replace_sep parameter from massive store It is unused since changeset efbbf1e93a04 "[dataimport] Properly escape strings sent to COPY FROM"

[dataimport] remove pg_schema parameter from massive store It can be recovered from the system source's configuration.

[dataimport] remove drop_index parameter from massive store "drop_index=False" also implied not dropping any constraints, in particular foreign keys, which meant any attempt to import entities would fail, because we only add metadata (the entities table) after the entity's insertion.

[dataimport/test] update call to ucsvreader Silences warning: [3.20] 'separator' kwarg is deprecated, use 'delimiter' instead

[dataimport] remove deprecated methods from massive store This code is new in cubicweb, no reason for it to carry deprecated stuff from the start.

[dataimport/test] use store.finish instead of deprecated store.cleanup

[dataimport] drop no more used parameter on massive store

[dataimport] methods that modify in-place shouldn't return value Also mark those methods private and drop some useless comments in passing. Related to #5414760

[dataimport] implement new store API on massive store and deprecate the old one. Related to #5414760

[dataimport] .finish method was missing from (NoHook)RQLObjectStore

[massive store] identity_relation should not be considered, is_relation seems missing The "identity_relation" table should not even exist.

[dataimport/test] use the right assert methods instead of assertTrue with a comparison

[dataimport] import range from six.moves

[dataimport/test] feed binary data to ucsvreader

[dataimport/test] fix broken comparison We want to compare two ints, not an int and a db row

[dataimport] dict.iteritems() → dict.items()

[dataimport] use next builtin instead of next method on iterators

[dataimport] import StringIO from io

[dataimport] backport massive store from dataio cube Some tweaks have been made to work with 3.21 foreign key constraints. Also test are not executed in our own pg cluster instead of polluting the system one. Closes #5414760

[web] set proper Cache-Control header for static files - set Cache-Control max-age to match Expires - use the max_age method (defaults to 1 week) instead of hardcoded 6 months

[web] use a redirect for /data/foo requests that don't use the instance md5 hash The actual file contents can be cached for a long time. If the request doesn't use the hash, that's wrong. Using a redirect lets keep the caching for the actual file contents while not breaking requests missing the hash.

[web] fix typo in doc string

[web] stop adding an Expires header with all responses As per RFC 1945 (HTTP 1.0): If the date given is equal to or earlier than the value of the Date header, the recipient must not cache the enclosed entity. As per RFC 7234 (HTTP 1.1 Caching): If a response includes a Cache-Control field with the max-age directive (Section 5.2.2.8), a recipient MUST ignore the Expires field. Bottom line, the Expires header is now handled wherever the Cache-Control is handled: inside cache managers.

[web] stop looking at a request's Cache-Control header As per RFC 7234 (HTTP 1.1 Caching): The "Cache-Control" header field is used to specify directives for caches along the request/response chain. CubicWeb is not a cache.

[fti] add cw_fti_index_rql_limit method Improves on and deprecate cw_fti_index_rql_queries: execute the rql directly, so that we don't need to compute the set of eids up-front, but can just keep track of the last seen entity. Use the new method in reindex_entities. Keep calling cw_fti_index_rql_queries if it has been redefined (e.g in cubes). Related to #3621392.

[fti] backout 166c6f7b1be4 * breaks tests * see next commit for a better solution

[py3k] Fix unicode added in commit 2350424585f8

[server/ldap] deal with unicode or str coming out of ldap It seems we can get either, I've no idea in which exact circumstances, so this should let us work with both.

[test] pull yapps from Logilab's github Fixes a couple of failures, pull request is almost a year old (https://github.com/smurfix/yapps/pull/2).

[pkg] bump yams requirement to 0.41.1 Needed by changeset c9321aedfbf9.

[entities] import range from six.moves

merge head published by accident

[fti] Improve big table reindexation * Slice the reindexing in batches of 1000 entities. * Make the output more verbose. Closes #3621392

[tox] Reference requirements.txt for hooks environment The file existed but was not referenced.

[devtools/testlib] Remove assertions about direct usage of CubicWebTC/AutomaticWebTest The point is to be able to use a test runner other than pytest (e.g. nose2, nosetests or py.test) to run CubicWeb test suite. The issue is that most other test loaders inspects members of all base classes of test case classes to determine test methods and it hits this `config` property (actually executing it), which breaks the assertion. So just return None instead of failing. Closes #7817147.

Add ldap3 to server test requirements

[pkg] bump version number so tox won't use cw from pypi

[schema] add CubicWebRelationDefinitionSchema Stop monkeypatching yams, inherit from RelationDefinitionSchema instead.

[schema] Let RQLConstraint inherit from BaseRQLConstraint instead of RQLVocabularyConstraint There is apparently no reason for RQLConstraint to inherit from RQLVocabularyConstraint as it gets its `repo_check` method from `RepoEnforcedRQLConstraintMixIn` and RQLVocabularyConstraint (which inherits from BaseRQLConstraint) actually only overrides that method. So simplify inheritance by removing a useless level of indirection. Adjust cw_unrelated_rql accordingly, where it was that RQLConstraint inherited from RQLVocabularyConstraint.

[server] unormalize source logger's name to avoid encoding errors

[entities] parameters should be given to fire_transition, else they are simply ignored

[entity] set _cw_adapters_cache in __init__ rather than on first use

fix typos and indentation in docstrings/docs

session.id → session.sessionid

[test] test page_size problem (related to #8123231)

[predicates] guard against non-integer page_size form parameter Closes #8123231

[server/test] add test for db-dump command

[server] fix db-dump with sqlite 'gzip -c dbname > backupfile' needs to go through the shell.

[serverctl] fix db-dump Since changeset af47954c1015 we would die with: AttributeError: 'ServerMigrationHelper' object has no attribute 'repo'

[server/test] sorting vs dict vs python3

[devtools] more sorted() vs dict vs python3

[web/application] return bytes from redirect_handler and validation_error_handler

[devtools] fix ResourceWarning: unclosed file tmpdb-__default_empty_db__.config

[serverconfig] fix ResourceWarning: unclosed file bootstrap_cubes

Use cubicweb._

[syncschema] hashlib.md5() wants bytes

[server/test] use next() builtin instead of next() method

[server/test] dict.iterkeys

[syncschema] don't arbitrarily remove constraints from the in-memory schema For some constraint types, it may make sense to have several constraints of that type simultaneously. In such a case, removing one of them from the in-memory schema is wrong. Closes #7446353

[server/test] don't leak timeout from one test class to the next Reset SQL_CONNECT_HOOKS after we're done with this test class so the next one doesn't inherit a timeout it doesn't want.

[server/test] CWAttribute.extra_props is Bytes

[dataimport] give unicode objects to psycopg2 copy_from

[dataimport] extid must be a bytes object

[dataimport] import StringIO from io

[dataimport] fix ucsvreader for python3 The csv module in python3 wants str, so use a StreamReader to feed csv.reader and don't decode the output

[server/test] import StringIO from io on python3 Don't change the import for python2, as we print a mix of unicode and str.

[server/datafeed] drop URLLibResponseAdapter.info I don't know where it's used

[devtools] pass a key to sort() method python3 dicts are not comparable

[postcreate] properly set is_instance_of for bootstrap entities Entities created before CWEType itself need the is/is_instance_of relations created later. We were actually only inserting "is". Closes #7656606.

[migration] re-try without print_function when compiling a script fails As part of the python3 porting we added the __future__.print_function import to cubicweb.migration. That started affecting all migration scripts it runs, which may not be prepared. So if compiling a script results in SyntaxError, try again without the print_function flag. Closes #7935030

[test] do not overwrite user's CW_CUBES_PATH in unittest_migration Currently when you rely on your CW_CUBES_PATH environment variable to locate cubes necessary for cubicweb's tests, the migration test crashes because it overrides the env var.

[server/test] dict.itervalues() → iter(dict.values())

make sync_schema_props_perms(<computed rtype>) work as expected It currently ends up with an ExecutionError while we want to synchronize permissions. This makes 3.21.1 migration of any application using computed relation crashing. Also, remove pre 3.6 compat code in hooks that prevent living schema update on computed relation's permission changes.

[web/http_headers] use classmethod decorator

[schema] use classmethod decorator

[devtools] delete postgresql socket directory after killing the server ... or when failing to start it.

[skel] stop deleting cubes egg-info It doesn't hurt and provides some metadata.

[dataimport] Update doc to reflect new API relate() looks deprecated.

[server] fix generate_source_config regression on python2 lgc outputs "str" objects on python2 and python3, so we can't just use io.StringIO (which expects unicode objects). Fixes: c36cda9074c5

[tests] Fix config.repository() in tests The config.repository was overriden too late. Closes #5778294

[schema] RQL queries are always unicode objects in python 3 fixes commit 42c5bd7286b7.

[server/test] use relative import Fixes py3k breakage

[server/schemaserial] fix extra_props on py3k json.dumps returns unicode, and json.load eats unicode, so add explicit encoding/decoding.

[server/test] unicode → six.text_type

[web] fix Field.__str__ on py3k

[web] unicode → six.text_type

[web/test] CubicWebTC.view returns bytes

[uilib] fix traceback on python3

[web] integer division

[web/test] Binary eats bytes

[devtools] make httptest module importable under py3k

[web/test] CubicWebPublisher.handle_request returns bytes

[web] base64 works on bytes not str (py3k)

[web] accept unicode in parsed Authorization header

[web] CubicWebPublisher.ajax_error_handler must return bytes

[py3k] fix i18n

Fix md5crypt and crypt_password test for python3 Make sure to always work with bytes for hashing.

[server/test] use bytes for extid

[devtools/test] firstnames.txt is in latin1

[web/test] unicode → cubicweb._

[web/formwidgets] fix for py3k no idea why we encode that key...

[sobjects] unicode → six.text_type

[server] fix unittest_security for py3k - use source.binary_to_str instead of str(), helps when the backend returns memoryviews - fixup str vs bytes

[py3k] make ldapfeed tests pass

[server] import filter from six.moves

[ldapfeed] port to ldap3 python-ldap has no python3 support. XXX I only checked that unittest_ldapfeed passes.

[server/test] str vs bytes

[server/test] list() around map and zip

[server/test] to fold, unicode → six.text_type

[server/test] delete test that doesn't make sense on python3 XXX if PY2?

[server/test] unicode → six.text_type

[server] unicode → six.text_type

[py3k] fix building rset description for INSERT queries

[datafeed] use explicit encode/decode instead of casting to str

[py3k] io.BytesIO

[py3k] get urllib foo from six.moves

[devtools] CubicWebTC._check_html may have to deal with unicode or bytes If called with template=None, CubicWebTC.view will not go through main-template, and return the unicode data from the view directly instead of encoded data.

[ext/test] fix test relying on repr(text)

[req] fix url_parse_qsl on py3 should be folded with previous conversion

[server/session] unicode → six.text_type

[web/test] str vs bytes

[web/ajax] fix AjaxController on python3

[web/test] str → bytes

[wsgi/test] use bytes instead of str for POST body

[devtools] 'dict_values' object does not support indexing

[multipart] decode form data before calling parse_qs in python 3 parse_qs eats and returns 'str' on both python versions.

[multipart] unicode → six.text_type

[web] assert that CubicWebPublisher.handle_request returns bytes

[py3k] fix validateform controller We need to return bytes not str.

[web] unicode → six.text_type

[multipart] make sure form keys are not bytes

[web] unicode → six.text_type

[wsgi] don't convert form values This is done earlier in multipart.py

[server] import input from six.moves

[server] don't convert user.login to bytes to generate sessionid

[cwctl] stop playing tricks with std{out,err} buffering Python3 doesn't allow unbuffered text I/O, because it doesn't make sense.

[py3k] CubicWebTC.app_handle_request returns bytes

[py3k] unicode → six.text_type

[py3k] http headers should be ascii-only unicode objects

[py3k] Clarify IDownloadable expected types

[py3k] IDownloadable.download_data() returns bytes

[py3k] unicode → six.text_type

[py3k] unicode → six.text_type

[py3k] unicode → six.text_type

[py3k] unicode → six.text_type

[py3k] PageInfo.source contains bytes

[py3k] unicode → six.text_type

[py3k] unicode → six.text_type

[py3k] Don't encode URLs Python 3's urllib works with unicode objects.

[tests] Stop counting pickle bytes in ResultSet tests Between python 2's pickle and cPickle implementations, the various protocol versions added in python 3, the length of the pickled string is completely unpredictable. Instead, do a simple round trip and test a few attributes.

[tests] Port unittest_cwctl to py3k On python 2, sys.stdout takes bytes-like objects whereas python 3's takes unicode-like objects and handles the encoding on its own.

[migration] NoneType is no longer comparable in py3k

Rename gettext to cwgettext, use stdlib gettext cubicweb/gettext.py was a copy of Python 2.6's gettext.py with the context patch from bug https://bugs.python.org/issue2504 Instead of keeping that full copy, just inherit from gettext.GNUTranslations and add the context-related stuff.

[py3k] MethodType no longer takes a class argument

[py3k] unicode → six.text_type

[py3k] im_self → __self__

[py3k] Don't sort Action objects In python 3, arbitrary objects are no longer sortable.

[py3k] use inspect module to get a function's arguments

[py3k] __unicode__ and __str__

[test] call rql.parse with unicode objects Allows to get same results from rqlst.as_string() in python 2 and 3

[py3k] replace cmp with key in sorted() And stop sorting dicts in snippet_key()

[tests] remove useless sort Useless in python 2, actively harmful in python 3 where arbitrary objects are no longer sortable.

[py3k] dict methods no longer return indexable objects

[py3k] unicode → six.text_type

[py3k] unicode → six.text_type

[server] Port BFSS to py3k The BFSS API changes in python 3: * 'defaultdir' MUST be a unicode object * 'fsencoding' MUST NOT be set In python 2, fsencoding handles both the encoding of file paths on the file system (utf-8 by default, but the system may actually be using something else) and the encoding of file paths that will be stored in the database. So in python 3, we wipe the slate clean: * rely on sys.getfilesystemencoding() to convert unicode objects to bytes * always encode paths to utf-8 for storage in the database Caveat emptor / here be dragons: * sys.getfilesystemencoding() depends on the current locale, which therefore MUST be set properly * when migrating an existing instance from py2 to py3, one MAY need to reencode file paths stored in the database

[py3k] Binary.getvalue() returns bytes

[py3k] ensure Binary objects are initialized with bytes

[py3k] __unicode__ and __str__

[py3k] unicode → six.text_type

[mail] port to py3k

[web/views] port JSON views to py3k

[py3k] unicode → six.text_type

[web/controller] req.form is supposed to contain unicode-data

[py3k] unicode → six.text_type

[py3k] unicode vs str vs bytes vs the world

[web/views] Port csvexport views to py3k There are a few hoops to jump through since python 3's csv module _is_ unicode aware.

[py3k] unicode vs str vs bytes vs the world

[web/request] don't convert request parameters if py3k

[py3k] unicode → six.text_type