[server/test] list() around map and zip

[server/test] to fold, unicode → six.text_type

[server/test] delete test that doesn't make sense on python3 XXX if PY2?

[server/test] unicode → six.text_type

[server] unicode → six.text_type

[py3k] fix building rset description for INSERT queries

[datafeed] use explicit encode/decode instead of casting to str

[py3k] io.BytesIO

[py3k] get urllib foo from six.moves

[devtools] CubicWebTC._check_html may have to deal with unicode or bytes If called with template=None, CubicWebTC.view will not go through main-template, and return the unicode data from the view directly instead of encoded data.

[ext/test] fix test relying on repr(text)

[req] fix url_parse_qsl on py3 should be folded with previous conversion

[server/session] unicode → six.text_type

[web/test] str vs bytes

[web/ajax] fix AjaxController on python3

[web/test] str → bytes

[wsgi/test] use bytes instead of str for POST body

[devtools] 'dict_values' object does not support indexing

[multipart] decode form data before calling parse_qs in python 3 parse_qs eats and returns 'str' on both python versions.

[multipart] unicode → six.text_type

[web] assert that CubicWebPublisher.handle_request returns bytes

[py3k] fix validateform controller We need to return bytes not str.

[web] unicode → six.text_type

[multipart] make sure form keys are not bytes

[web] unicode → six.text_type

[wsgi] don't convert form values This is done earlier in multipart.py

[server] import input from six.moves

[server] don't convert user.login to bytes to generate sessionid

[cwctl] stop playing tricks with std{out,err} buffering Python3 doesn't allow unbuffered text I/O, because it doesn't make sense.

[py3k] CubicWebTC.app_handle_request returns bytes

[py3k] unicode → six.text_type

[py3k] http headers should be ascii-only unicode objects

[py3k] Clarify IDownloadable expected types

[py3k] IDownloadable.download_data() returns bytes

[py3k] unicode → six.text_type

[py3k] unicode → six.text_type

[py3k] unicode → six.text_type

[py3k] unicode → six.text_type

[py3k] PageInfo.source contains bytes

[py3k] unicode → six.text_type

[py3k] unicode → six.text_type

[py3k] Don't encode URLs Python 3's urllib works with unicode objects.

[tests] Stop counting pickle bytes in ResultSet tests Between python 2's pickle and cPickle implementations, the various protocol versions added in python 3, the length of the pickled string is completely unpredictable. Instead, do a simple round trip and test a few attributes.

[tests] Port unittest_cwctl to py3k On python 2, sys.stdout takes bytes-like objects whereas python 3's takes unicode-like objects and handles the encoding on its own.

[migration] NoneType is no longer comparable in py3k

Rename gettext to cwgettext, use stdlib gettext cubicweb/gettext.py was a copy of Python 2.6's gettext.py with the context patch from bug https://bugs.python.org/issue2504 Instead of keeping that full copy, just inherit from gettext.GNUTranslations and add the context-related stuff.

[py3k] MethodType no longer takes a class argument

[py3k] unicode → six.text_type

[py3k] im_self → __self__

[py3k] Don't sort Action objects In python 3, arbitrary objects are no longer sortable.

[py3k] use inspect module to get a function's arguments

[py3k] __unicode__ and __str__

[test] call rql.parse with unicode objects Allows to get same results from rqlst.as_string() in python 2 and 3

[py3k] replace cmp with key in sorted() And stop sorting dicts in snippet_key()

[tests] remove useless sort Useless in python 2, actively harmful in python 3 where arbitrary objects are no longer sortable.

[py3k] dict methods no longer return indexable objects

[py3k] unicode → six.text_type

[py3k] unicode → six.text_type

[server] Port BFSS to py3k The BFSS API changes in python 3: * 'defaultdir' MUST be a unicode object * 'fsencoding' MUST NOT be set In python 2, fsencoding handles both the encoding of file paths on the file system (utf-8 by default, but the system may actually be using something else) and the encoding of file paths that will be stored in the database. So in python 3, we wipe the slate clean: * rely on sys.getfilesystemencoding() to convert unicode objects to bytes * always encode paths to utf-8 for storage in the database Caveat emptor / here be dragons: * sys.getfilesystemencoding() depends on the current locale, which therefore MUST be set properly * when migrating an existing instance from py2 to py3, one MAY need to reencode file paths stored in the database

[py3k] Binary.getvalue() returns bytes

[py3k] ensure Binary objects are initialized with bytes

[py3k] __unicode__ and __str__

[py3k] unicode → six.text_type

[mail] port to py3k

[web/views] port JSON views to py3k

[py3k] unicode → six.text_type

[web/controller] req.form is supposed to contain unicode-data

[py3k] unicode → six.text_type

[py3k] unicode vs str vs bytes vs the world

[web/views] Port csvexport views to py3k There are a few hoops to jump through since python 3's csv module _is_ unicode aware.

[py3k] unicode vs str vs bytes vs the world

[web/request] don't convert request parameters if py3k

[py3k] unicode → six.text_type

[py3k] __getslice__ → __getitem__ with slice support

[py3k] __nonzero__ → __bool__

[py3k] unicode vs str vs bytes vs the world

[py3k] use six.add_metaclass

[py3k] unicode → six.text_type

[py3k] add list() around dict methods

[py3k] unicode → six.text_type

[py3k] unicode → six.text_type

[py3k] unicode → six.text_type

[py3k] unicode → six.text_type

[server/utils] crypt_password should always return bytes

[py3k] str → six.binary_type

[py3k] unicode → six.text_type

[server/utils] convert to py3k using six

[py3k] six.PY2

[py3k] use text/binary types from six

[py3k] unicode → six.text_type

[py3k] bytes vs str

[py3k] io.BytesIO

[py3k] unicode → six.text_type

[py3k] bytes required for hashlib.{md5,sha1}()

[wsgi/test] add __main__ section

[py3k] use next builtin instead of next method

[web/views] use next builtin instead of next method

[py3k] declare meta classes using six.add_metaclass

[py3k] replace '_ = unicode' in global scope (closes #7589459)

[py3k] introduce cubicweb._ (related to #7589459) The end goal is to stop relying on '_' being in the builtin scope. In the mean time, deprecate the builtin '_' so that consumers are warned.

[py3k] dict.iterkeys → dict

[py3k] dict.itervalues → dict.values

[py3k] dict.iteritems → dict.items

implement __ne__ whenever we implement __eq__

[test] use unicode for rql queries (7/7) Closes #6694446

[test] use unicode for rql queries (6/7) Related to #6694446

[test] use unicode for rql queries (5/7) Related to #6694446

[test] use unicode for rql queries (4/7) Related to #6694446

[test] use unicode for rql queries (3/7) Related to #6694446

[test] use unicode for rql queries (2/7) Related to #6694446

[test] use unicode for rql queries (1/7) Related to #6694446

[schema] improve normalization of RQLExpressions Parse and print back the expression instead of manipulating the string. Among other benefits, it means we don't mangle embedded string constants that contain commas or multiple spaces. Closes #6694426

[server/rql2sql] use VariableRef.is_equivalent explicitly instead of relying on __eq__ What we want to compare is the variable referenced by the VariableRef, we should make that explicit. Closes #6694406

merge with 3.21.2

[statsd] fix the statsd logger (closes #7558147) socket.inet_pton's second argument is the ip, not a couple (ip, port)

Added tag 3.21.2, debian/3.21.2-1, centos/3.21.2-1 for changeset a5428e1ab364

[pkg] 3.21.2

[pkg] Remove leftover dep on pyro (closes #7479155)

merge with 3.20.10

[migration] fix change_attribute_type to update the live schema Closes #7170830

[devtools] add has_cache for postgres (closes #5739624) devtools stores info about existing dbs in the db handler, but in the case of postgresql that doesn't take into account the path to the cluster's datadir. Which means if we run two test modules (in the same test run), we'll create a "__default_empty_db__" for the first one, cache its existence, and then when moving on to the other module, believe the template already exists (but since the datadir depends on the test module's path, it does not). This patch is a bit of a kludge, and it would be better to make the cache key include enough data to not need this, but I'm not sure how to do that.

fix bad-caching of datetime with tz info at sql generation time There is a special handling for datetime with tzinfo, where value was stored in the query cache. The implementation of merge_args was simply overwriting parameters of the query with those in the query cache, expecting no collision. To fix this: * handle replacement of tzinfo in merge_args, not at sql generation time * add an assertion to ensure we've actually no collision Closes #6978316

Added tag 3.20.10, debian/3.20.10-1, centos/3.20.10-1 for changeset 8f82e9523962

[pkg] 3.20.10

merge with 3.19.13

[devtools] Fix CubicWebTC.temporary_permissions If the context manager is exited via an exception, the original permissions should be restored.

[devtools] Use TestCase.assertGreater

[autoform] fix appearance of link to add inlined creation form On entity creation, if there are some local permissions on the relation, we have no way of checking them since neither the subject nor the object of the relation exists yet. In such a case, we should add the link by default, for consistency (see other places where we use `may_have_permission`). Closes #6711900

[predicates] Simplify specified_etype_implements

[web/views] Fix `has_editable_relation` predicate wrt inlined relations Prevents the "modify" action from being shown when there is nothing to edit.

[web/views] avoid propagation of NoSelectableObject in some case of inlined relations / permissions When selecting an inlined creation form, we should catch the NoSelectable exception that will be raised if the user cannot add entities of the target type (this is not and cannot be verified earlier) or if some other custom selector prevents the form from being selected. Closes #6510921

[server/test] bit of PEP8

[migration] don't attempt to carry over values when renaming a computed relation Without the fix in migractions.py, the introduced test crashes with an operational error because of an attempt to write to a non-existing relation table (computed relations are not materialized). Closes #6304946

consider .do_fti flag in index_entity method as this method may be used from outsite the class (eg in UpdateFTIHook), this is the proper way to consider it anyway. Closes #6206636

Added tag 3.19.13, debian/3.19.13-1, centos/3.19.13-1 for changeset f933a38d7ab5

[pkg] 3.19.13

[migration/3.21] add support for sqlserver Closes #7173702

moved_entities.extid may not be NULL

[server] use sqls_create_multicol_unique_index to index entities.extid SQL Server's UNIQUE is not standards compliant, it doesn't accept two NULL values. Related to #7173702.

[server] fix checkintegrity with pyodbc "msg % row" dies with "not enough arguments for format string" if row is a pyodbc.Row.

[schema] remove unique constraint on unlimited String attribute SQL Server doesn't allow indexing (and thus unique constraints) on nvarchar(max). Related to #7173702.

[sources/native] fix potential NameError (closes #7364249)

[py3k] unicode → six.text_type

[devtools/testlib] Port CubicWebTC.view() to py3k testlib.TestCase.set_description's argument will eventually be printed out to sys.stdout/stderr. On python 2, this object takes bytes-like objects (eg, str) whereas python 3's takes unicode-like objects and handles the encoding on its own. Thus the 'str' type can used in both versions to eventually display the description on screen.

[devtools/testlib] port 'fake_form' test to io.BytesIO

[dataimport] port ucsvreader tests to io.BytesIO

[py3k] io.BytesIO

[py3k] io.StringIO

Port cw.Binary to io.BytesIO And add plenty of tests too!

[py3k] execfile → exec

[py3k] file → open

[py3k] six.integer_types

[py3k] basestring → six.string_types

[py3k] import CookieJar using six.moves

[py3k] import urllib2 from six.moves

[py3k] import range using six.moves

[py3k] import queue using six.moves

[py3k] import filter using six.moves

[py3k] import zip using six.moves

[py3k] import SimpleCookie using six.moves

[py3k] import HTTP client constants and exceptions using six.moves

[py3k] import URL mangling functions using six.moves

[py3k] import 'pickle' using six.moves

[pkg] Add dependency on six 1.4.0

[tests] Replace use of deprecated TestCase.assert_

[predicates] Simplify range(len()) iteration

[dataimport] Use pickle.dump instead of dumps

[web/facet] Simplify reversed/sorted calls

[web/test] Use TestCase.setUpClass instead of setUpModule

Use isinstance() instead of type comparison

Unused imports

[py3k] tuple unpacking in function args

[py3k] ur'' is gone

[py3k] octals and long

[py3k] Fix raise with embedded traceback python 3 removed the raise syntax with 2 and 3 expressions.

[py3k] print function

[py3k] except as

[py3k] backtick to repr

[hooks/syncschema] make sure CWUniqueTogetherConstraintDelOp happens before CWConstraintDelOp SQLServer refuses to index an unlimited text column, so we should drop unique_together constraints (which imply an index) before we drop size constraints. Closes #5560601.

[hooks/syncschema] only call "ALTER TABLE" once when changing a size constraint Until now we would: - remove the old size constraint from the in-memory schema - call update_rdef_column which removes the size restriction from the column's type - add the new constraint object - call update_rdef_column which adds the size restriction back This breaks on SQL Server when the column is involved in an index (e.g. as part of a multi-column unique constraint), because in the intermediate stage the column's type is "nvarchar(max)", which is not indexable. Of course we must still detect the case where a size constraint is really dropped and update the db schema accordingly. Closes #5557633.

[syncschema] only add to the `synchronized` set if all possible updates have been done Otherwise a simple sync_schema(syncprops=False) turns the subsequent sync_schema() calls into a no-op. Closes #6510918.

[qunit] remove unused 'data_files' parameter Its use was removed when we stoppped pointing firefox at file:// web/test/jstests/ajax_url2.html has never actually been used, it's only been ever mentioned in a QUnit comment.

[urlpublish] RESTPathEvaluator now use vid_from_rset This avoid cases where vid may be unexpectedly overwritten. For instance when you define a custom vid for a mime type in VID_BY_MIMETYPE, it will currently be overriden by the URL publisher when you access to /<etype> with the proper Accept header. To do so, a new 'check_table' argument is introduced, which may cause bw compatibility problems if the function has been monkey-patched. Also pep8 a bit the tests. Closes #5705835

[server/sources] use decode_extid in datafeed

[server/sources] add decode_extid method avoids having the same dance in both eid_type_source variants

[server/repo] kill repo.pinfo() session.cnxset is dead, this can't work.

[web/sessions] simplify session cleanup session.cnx.check() is no longer a thing. It dates back to dbapi.

[server/repo] fix typo

[server/sources] AbstractSource.__ne__ Seems confusing to have __eq__ but not __ne__.

[server/sources] fix docstring for eid_type_source

[views] fix ProcessInformationView: SESSION_MANAGER can be None (closes #5753280) eg. with pyramid.

[server/migractions] drop mock_object It doesn't seem to have ever been used.

[test] kill unused lgc.testlib imports

[server/migractions] raising SystemExit sounds like a bad idea Simply skip the action (rename_relation_type) with an early return.

[server/storage] rename 'session' variables to 'cnx' We no longer deal with Session objects here.

[migration] remove repo_connect and use config.repository() instead One less more redundant path to the repo. Related to #3933480.

[repoapi, cwconfig] give a convenience method to get a repository from a config This is a very common need. The private repoapi._get_inmemory_repo becomes config.repository(...)

[cwconfig] remove unused method Not to be confused with serverconfig.migration_handler.

[captcha] Port to io.BytesIO

[web/sessions] the session managers are definitely not components Component (or AppObjects) take a _cw at __init__ time. The session managers want a repository. Related to #1381328.

[web] the AuthenticationManager is no more an appobject It is now explictly instanciated by the session manager. Related to #1381328.

[hooksmanager] remove very old unused module

[devtools] fix CubicWebTC.assertDocTestFile (closes #5930028)

Added tag 3.21.1, debian/3.21.1-1, centos/3.21.1-1 for changeset a8a0de0298a5

[pkg] Let dh_installdocs deal with doc-base

[pkg] various debian packaging fixes - stop using *.install.in files, just use wildcards in *.install - switch to X-Python-Version (preferred for dh_python2) - also call dh_python2 for /usr/share/cubicweb, not just public modules - drop long obsolete ginco and erudi migration code from cubicweb-ctl.postinst - fix cubicweb-ctl cleanup: purge code belongs in postrm, not prerm - don't compress changelog.html

[pkg] 3.21.1

[web] fix AbstractSessionManager.clean_sessions (closes #5757240) sessions were not closed if "last_usage_time = session.cnx.check()" actually raises an AtributeError.

[migration/3.21] ask fewer questions Several questions per table is just unreasonable.

[server] typo fix

[test] check that we reject an rqlexpression in computed relation perms

Fix (de)serialization of ComputedRelation read permissions For normal relation types, permissions don't need to be stored since they're just default values for the relation definitions. However, computed relations are serialized (as CWComputedRType), while their relation definitions are added at schema finalization time, and are only in memory. So add the 'read_permission' relation to CWComputedRType, and the appropriate hooks to save and restore those permissions. To avoid having to touch yams, we drop the 'add' and 'delete' permissions from the default computed relation permissions; this should probably be backported there. The actual permissions (set on the relation definitions) are hardcoded in finalize_computed_relations anyway. In deserialize_schema, the CWComputedRType handling needs to be delayed a little bit, until after we've called deserialize_ertype_permissions. The rql2sql test is adjusted because CWComputedRType has a 'name' attribute and the 'read_permission' relation, which generates ambiguity vs CWEType. We add an explicit CubicWebRelationSchema.check_permission_definitions, since we need to check both that computed and non-computed rtypes are defined properly. Based on report and initial patch from Christophe de Vienne (thanks!). Closes #5706307

Copy read permissions on ComputedRelation rdefs Closes #5694110

[datafeed] fix typo in DataFeedXMLParser.is_deleted (closes #5729755)

[server/sources] make sure entity._cw is a Connection before calling Storages The storage callbacks don't get an explicit Connection objects, so they go through entity._cw to access the repo and schedule operations. Since that entity object comes out of the entity cache, its _cw may be a web request instead of a Connection. So fix it up around the storage callbacks. Closes #5753543

[hooks] base64-encode extid before inserting it into moved_entities If we insert the wrong value we're never going to find these entities again... Closes #5741105.

[hook] remove assumption about entity cache vs cw_edited We were assuming that if an entity is in the connection's cache, it has a cw_edited attribute. This doesn't seem to always be the case. At least in cubicweb 3.21, merging the web and repo-side caches makes this more likely, but Denis says he's also seen this on 3.20.

[hook] don't open-code Connection.entity_cache

[tox] Add missing dependency on webtest for devtools tests

[tox] Add missing dependency on webtest for wsgi tests

[tox] Remove set of PATH environment variable The : concatenation does not work actually so the resulting PATH was just wrong (it did not contain the previous PATH). Anyways, the PATH environment variable is passed by default by tox so it's probably better to let the caller set it (thus avoiding to hard-code the specific postgres version and path). http://tox.readthedocs.org/en/latest/example/basic.html#passing-down-environment-variables

[doc] generalize info about debian/ubuntu repositories That section was not kept up to date with recent distro releases; stop trying.

[doc] fix link to setup info in README ... and fix URL to the repository's gpg key.

[server/session] remove unused cnx_meth function