[server] Make connection pooler configurable and set better default values Drop the configuration connections-pool-size and add new configurations options: * connections-pool-min-size. Set to 0 by default so we open connections only when needed. This avoid opening min-size*processes connections at startup, which is, it think, a good default. * connections-pool-max-size. Set to 0 (unlimited) by default, so we move the bottleneck to postgresql. * connections-idle-timeout. Set to 10 minutes. I don't have arguments about this except that this is the default in pgbouncer.

[server] Enhance connections-pooler-enabled documentation

[server] move connection pooler initialization logic to get_cnxset() This avoid complex logic in Repository initialization.

[server] dynamically close idle database connections When pool hasn't been empty for `idle_timeout` time, start closing connections.

[server] implement dynamic database pooler Opening too much database connection has a cost at startup and also PostgreSQL as a maximum number of connection (100 by default). This get worse when starting multiple wsgi processes, since each process has its own database pool. Instead of opening `connections-pool-size` connections to the database at startup, just open one and open more only when needed.

[server] use a LifoQueue in _CnxSetPool In postgresql, some cache is attached to the connection. Using a LifoQueue (last-in, first-out) makes a few connections to get the most load which give best performance.

[server] extract creating a new cnxset in a _new_cnxset() helper So we can move logic specific to _CnxSetPool here.

[server] avoid a possible race condition on _CnxSetPool.close() The pool could become empty between time to check and time to use.

[server] extract "no pooler" CnxSet class to a _BaseCnxSet class So we get rid of "if self._queue is None" in each method of _CnxSetPool Also add helper get_cnxset(source, size) to instantiate the correct pooler class.

[server] prevent returning closed cursor to the database pool In since c8c6ad8 init_repository use repo.internal_cnx() instead of repo.system_source.get_connection() so it use the pool and we should not close cursors from the pool before returning it back. Otherwise we may have "connection already closed" error. This bug only trigger when connection-pool-size = 1. Since we are moving to use a dynamic pooler we need to get this fixed. This does not occur with sqlite since the connection wrapper instantiate new cursor everytime, but this occur with other databases.

[rql] Store selected variables for RQL select queries in ResultSet (#17218476) By storing the name of the selected variables for RQL select queries in the ResultSet (within the "variables" attribute), the information can be passed down to specific protocols, e.g. rqlio that may wish to pass is down further to clients. In turn, clients can then choose to present the results of RQL select queries as symbolic bindings using the names used in the query's projection, instead of ordinal arrays.

[debugtoolbar] Make SQL and RQL tables sortable It's usefull to sort query by duration for instance.

[debugtoolbar] Do not render CW controller panel when no controller got collected This can happen on pure pyramid applications for instance.

[doc] Fix bugs in code examples

[doc] Remove zone cube which causes to crash on install The zone cube is not compatible with python3.

[doc] Precise which component to modify in gallery tutorial

[doc] Fix example about Community view customization

[doc] Add command pip install of new cube

[doc] update cubicweb version

[doc, blog tutorial] Remove cubicweb-dev from pip dependencies This pip package does not exist

[doc] Add requirement for gettext and graphviz When running the tutorial from scratch, I had errors because these two packages were not installed.

Added tag 3.27.3, debian/3.27.3-1 for changeset 85284f028266

[pkg] version 3.27.3

[pkg] require python >= 3.6 in debian packaging

[pkg] merge 3.27 Require python >= 3.6 since recent typing notations require >= 3.6

[pkg] add python3 only pypi specifiers This is only for pypi statistics purposes. Also add 'Programming Language :: Python', looks obvious but seems to be sane to have this as well.

[pkg] the wheel is no longer universal Since we're only py3 compatible. In practice this changes nothing since we don't upload cubicweb wheels to pypi.

[pkg] require python >= 3.4 This avoid pip pulling a version that does not run on python2 when using a python2 environment. Since we already released some 3.27 releases in pypi, I think we should release 3.27.3 and remove releases 3.27.2, 3.27.1 and 3.27.0 from pypi.

merge public heads

[web.views] add Link alternate in HTTP response header in HTML view Team: famarger, schabot, nchauvat, fferry, ethieblin

[pyramid] add routes /{eid} and /{etype}/{eid} to return RDF when rdf mimetype in Accept HTTP headers * simplify pyramid/resources.py by making the classmethod that returns a closure a simple function and removing the EntityResource and ETYpeResource classes that are barely used * replace predicate MatchIsETypePredicate with MatchIsETypeAndEIDPredicate Team: famarger, schabot, nchauvat, fferry, ethieblin

[entities] simplify rdf generation and add a generic rdf adapter Team: famarger, schabot, nchauvat, fferry, ethieblin

Added tag 3.26.17, debian/3.26.17-1 for changeset 5c6d242069b6

[pkg] version 3.26.17

[fix, calendar] Ensure that the buffer is a string 'unicode' is not defined in python3

Added tag 3.26.16, debian/3.26.16-1 for changeset fb6aecd654a3

[pkg] version 3.26.16

[ldapfeed] add option to enable starttls on ldap servers Modern ldap servers connection often use, or even require starttls.

[ldapfeed] re-enable ldap tests for python 3.7 This was disabled because ldap3<2 wasn't compatible with python 3.7 While I'm here, drop unseless newline.

[ldapfeed] make code compatible with ldap3>=2 * Some constants have been renamed. * Directly bind when data-cnx-dn/data-cnx-password are provided, some servers, including ours require this. * Use raise_exceptions=True to avoid ignored ldap errors * raise in case of failed anonymous bind * do not search for "dn" attribute because this raise an "invalid attribute" with new ldap3 versions * Password is now returned as bytes, so no longer need to encode them before crypt. * modification_date is now returned as a datetime object Co-Authored-By: Philippe Pepiot <philippe.pepiot@logilab.fr> Closes #16073071

[ldapfeed] use LDAP_SCOPES to find user-scope We already have LDAP_SCOPES dict defining possible scopes, this avoid possible code injection from config... Thus we can drop global variables already defined in LDAP_SCOPES since they are not used directly anymore.

Merge 3.27

Added tag 3.27.2, debian/3.27.2-1 for changeset e731c31eaed0

[pkg] version 3.27.2

[ldapfeed] add option to enable starttls on ldap servers Modern ldap servers connection often use, or even require starttls.

[ldapfeed] re-enable ldap tests for python 3.7 This was disabled because ldap3<2 wasn't compatible with python 3.7 While I'm here, drop unseless newline.

[ldapfeed] make code compatible with ldap3>=2 * Some constants have been renamed. * Directly bind when data-cnx-dn/data-cnx-password are provided, some servers, including ours require this. * Use raise_exceptions=True to avoid ignored ldap errors * raise in case of failed anonymous bind * do not search for "dn" attribute because this raise an "invalid attribute" with new ldap3 versions * Password is now returned as bytes, so no longer need to encode them before crypt. * modification_date is now returned as a datetime object Co-Authored-By: Philippe Pepiot <philippe.pepiot@logilab.fr> Closes #16073071

[ldapfeed] use LDAP_SCOPES to find user-scope We already have LDAP_SCOPES dict defining possible scopes, this avoid possible code injection from config... Thus we can drop global variables already defined in LDAP_SCOPES since they are not used directly anymore.

[ldapfeed] drop now useless import from __future__

[rdf] Add a basic foaf adapter for CWUser By the way, use RDFLib namespaces

[rdf] add functions and tools to generate rdf graph

[pyramid, predicate] Add a predicate that matches if the pattern is an eid in the DB

[server.querier] construct ExecutionPlan from schema since querier is not needed

[server.querier] remove useless attribute Querier._planner

[server.querier] remove useless indirection Querier.rqlannotator

[server.querier] remove useless indirections ExecutionPlan.{syssource,rqlhelper,schema} Simpler is better.

[server.rqlannotation] rename SQLGenAnnotator to RQLAnnotator This class is in charge of annotating the RQL syntax tree with various bits of information, like (in)variance, use of full-text-index, etc. It is a needed step before the generation of SQL, but does not touch SQL directly. Hence RQLAnnotator seems a better name.

[server] drop system_source_cnx for a simpler internal API system_source_cnx() can be replaced with source_cnx(). If system_source_cnx() is called with dbms_system_base=False, it is strictly equivalent to calling source_cnx() with the same source, privileges and interactivity. If system_source_cnx() is called with dbms_system_base=True, it is equivalent to calling source_cnx() with the system database and the same source, privileges and interactivity. No use of the default value for 'special_privs' was found in the code.

[server/sources/native] refactor to remove useless injection

[cwvreg] rename CWRegistryStore.solutions to compute_var_types solutions() seems an obscure name to me. Since this method computes the types of the variables of the RQL query string, let us rename it to compute_var_types.

[doc] replace python with python3 in shell commands

[adapters] move class EntityAdapter from cubicweb.view to cubicweb.entity The adapter mechanism is not specific to the use of views. Moving the class the cubicweb.entity makes this clearer.

[doc] replace tabs with spaces (and make linter happy)

[setup] add rdflib-jsonld to dependencies for cubicweb[rdf]