[doc/book] add q/a on i18n customisation, misc tweaks

Switched from TwistedWeb2 to TwistedWeb - added HTTPResponse class in etwist/http.py (could be then abstracted in cubicweb/web) - added twisted.web2 http_headers.py file in cubicweb/web to handle HTTP headers conversion between raw headers and python object - deleted caching for base views (except for startup views). A better solution would be using weak entity tags (but they don't seem to be implemented in twisted.web). - added forbidden access message when browsing static local directories - tested with TwistedWeb 8, 9 and 10 TODO: ===== - Handle file uploading in forms. twisted.web seems to keep very little information (only file content) about uploaded files in twisted_request.args['input_field_name']. But it doesn't seem to keep track of filenames. Possible solutions : - use web2 code to parse raw request content still stored and available in twisted_request.content - find a magic function in twisted.web API to get the filenames - More tests.

merge

[ldap] fix email synchronization code: bad rql query + avoid integrity error on creating the address + nicer use_email / primary_email handling

[doc/book] adjust a bit

[doc/book] expand the js chapter with an ajax story

[doc] remove useless directories

temporary move

[web components] refactor as book recommends :$

more / cleaner / in code documentation of vreg, selectors and appobject

add note about running repository / client code

added information about the naming conventions in schema.py also: * some typo fix * note about ObjectRelation soon being deprecated * clarified a few hazy points

[doc/book] a new chapter on how to use the ORM

improved doc on selectors an vregistry

fix rest formatting

typo fix

merge

catch and log errors caused by bad json data

list allowed values for ldap protocol setting in help, so that the generated file includes this in a comment

[doc/book] enhance i18n section #656191

[doc] more (doc oriented) feature in list

[doc] enhanced setup section: fix sections'level, more postgres tips, merge environment doc with cwconfig.py module docstring using custom autodocstring directive

[doc] enhanced concepts section

remove ref to no more existing file

[cleaning] simpler implementation of cmd_create_entity

[storages] fix fs_importing side-effect on entity.data When creating a new File object, if fs_importing is set, we want entity.data to be the file content instead of the filepath for the rest of the transaction. (see test_bfss_fs_importing_transparency) for test implementation To make this possible, the storage hooks (entity_added / entity_updated) must return the correct value to set in the entity dict.

nicer warning

[doc] started full features list

[python] take care to this detail of @contextmanager: if an unhandled exception occurs in the block, it is reraised inside the generator at the point where the yield occurred

[entity] explain why the assert failed

[test] fix expected error

fix bad merge

fix #822890, missing argument on abstract source backup method make backup failed on multi-sources instances

merge

[schema] ensure we don't remove has_text relation type unfortunatly

merge

Added tag cubicweb-debian-version-3.7.3-1 for changeset 44c7bf90df71

Added tag cubicweb-version-3.7.3 for changeset 768beb8e15f1

[distrib] fix pkginfo

3.7.3

[hooks] better message on bad .event class attribute

[repository] forbid usage of set_attributes() in before_add_entity set_attributes() will generate a RQL query that will silently fail. An explicit error is better. You can still use the ``entity[attr] = value`` notation that won't generate a SQL query but still update the inner state of the entity and its edited_attributes attribute.

merge

[cleanup] tb already printed by self.exception; add note

[facet] fix 'bookmark this search': * vid wasn't included in the link generated by the facet box * proper url quoting in both js / box code (try an has_text search with '&' inside...)

[widget] more wide textarea on bookmark edition

[distribution] the web/data/images directory has to be handled in pkginfo

[test] use the new temporary_appojects CM to simplfy unittest_storage

[session] try to reconnect on unexpected error in system_sql. We need this for proper reconnection of pool used by hooks or looping task doing sql queries (such as cw_cleanup_transaction)

[source] log attempt to reconnect using warning level. Also, don't relog failed sql (already logged)

[test] avoid critical log by informing the source we consider Personne as a multi-sources type

[test] use new temporary_appobjects context manager

[test] that doesn't smell good but fix the test...

[vreg] we do want the opposite...

[testlib] simple helper method to register temporary appobjects (hooks for instance) with a context manager

[undo] fix transactions table name in cleanup operation

Add context management methode to dbapi.Connection the connection is automaticly closed when exiting the context.

i18n update

[i18n] use named substitution to avoid gettext warnings

[dataimport] print transaction id when we get one

[cleanup] add some notes

[optimisation] _select_best is used very intensivly. Peanuts optimization may count :)

[optimization] traced_selection is used very intensively in debug mode optimizing it is worth the pain. Basically : don't do stuff unnecessary when we don't trace anything.

[optimization] improve massive write performance by optimizing hooks selection profiling on some massive deletion showed up that 2/3 of the time was spent in hooks selection. Those changes make it much more acceptable (through selection is still not negligeable): * use one registry for each event, so we've much less hooks to check when emiting an event as well as no more need for the match_event selector. This required ability to put one appobject into several registries, using a __registries__ class attribute. * check for deprecated .enabled at registry initialization time instead of at selection time A very simple HooksManager class has been reintroduce to choose the right registry on call_hooks. Those optimisations leads to a ~x3 factor of time necessary to delete 16000 entities.

[upgrade] after upgrade, fork command to start instance avoid grave uicfg pb (explicit rules all removed...)

merge

[repo creation] removing existing entities of 'single' cardinality relatino should be considered as 'activeintegrity' hook. Also don't disable that category during repo creation to avoid pb such as two default workflows for one entity types

merge

handle Service Close request correctly (closes #736039) The windows event self._stop_event was not used, and twisted was not getting a chance of terminating cleanly by calling its shutdown event handlers (which would eventually call pyro_unregister). Fixed by calling reactor.stop()

[js form validation] fix case where we've None/null keys in the errors dictionnary

[cleanup] exists imported, use it

[form] CompoundField should not return itself in actual_fields, this avoid error on iliane with 3.7.2 where it's marked as required (from the schema) while value are provided by subfields

[repo] we want a hooks manager and hooks initialized event on quick start (we trigger events such as server_backup / server_restore for instance)

[distribution] sql extensions files now have proper extension, fix manifest file

backport stable

[events manager] onevent decorator may be given extra arguments to give to em.bind

[cleanup] some notes

[reload] on registry reloading, we should change class of users of opened session

[appobject imports] don't include __init__ in package's name. See comment for more info

[rset] cubes such as iliane hi-jack rset description by poping some lines from it. Handle this on RepeatList

[undo] basic support for undoing of entity creation / relation addition

[undo] init entity cache when undoing an entity deletion

[undo] consistent is/is_instance_of processing

[repo] don't need rset description on those queries

[cleanup] style fixes, add nodes, 0.2 cents refactorings

[undo] during undoing, call hooks in the [active]integrity and undo categories

[repo] refactor fti operation to use set_operation and a single operation whatever the number of entities to ft index

[querier] introduce RepeatList class, used to optimize size of data returned for result set description When rql query has no ambiguity, we used to return the same description * N where N is the size of the result set. Returning RepeatList class avoid that multiplication. According to quick benchmark this improve performance for result set whose size is ~ > 50, has very small penalty for rset < 50, and in any case improve the size of data to be transfered over the network through pyro connection.

[repo] on add entity, set cache as soon as possible + fill type/source cache

[repo] more efficient eid cache operations handling based on set_operation; refactor * split CleanupEidTypeCache operation into one operation for deleted eids and another for added eids, and put them in cw.server.hook * for consistency, move fti unindexing handling from the repository to the system source * handling of deleted eids is done in plan execution (ssplanner), no needs to redo it in *delete_info -> no more need for _prepare_delete_info method, killed

[repo] move eschema_eid function from hooks.metadata to server.utils

[primary view] skip virtual rtypes anyway

[test] CubicWebTC.create_user expect an actual query, and we don't need to give session here

[dataimport] stop disabling undo ; commit return transaction id

[logging] demote static file served to debug

[cleanup]

[repo] optimize massive insertion/deletion by using the new set_operation function Idea is that on massive insertion, cost of handling the list of operation become non negligeable, so we should minimize the number of operations in that list. The set_operation function ease usage of operation associated to data in session.transaction_data, and we only add the operation when data set isn't initialized yet, else we simply add data to the set. The operation then simply process accumulated data.

[repo] make etype which should go in deleted_entities configurable: we only need this for types imported from other multi-sources instances

[repo] don't execute child step twice in EntitiesDeleteStep

[form] consider autoform_field_kwargs for meta fields; allow required customization

[workflow] support for automatic transition starting from the initial state

[testlib] create_user support extra kwargs to give to create_entity(CWUser)

fix dataimport for 3.7.2

use lgc.testlib.TestCase

backport stable

Added tag cubicweb-debian-version-3.7.2-1 for changeset 8fda29a6c219

Added tag cubicweb-version-3.7.2 for changeset d010f749c21d

take care to empty string in cubes search path

backport stable

[ms] fix name error introduced by rql 0.26 api update

add note

3.7.2

[selectors] new is_in_state select to avoid common error when writing state based selector

[repo config] cleanup read_instance_schema / bootstrap_schema / creating mess

fix #726167: i18ninstance use data in bootstrab_cubes instead of the database one

[cleanup] remove 3.7 deprecation warning

[form controller] clear caches in case some attribute participating to the rest path has been modified, avoid redirection to a no more existant page (fix #753567)

i18nupdate

[form] fix validation error handling * type when possible ValidationError raised by the edit controller * don't rely on repository to check required field * turn css class used for field error message to errorMsg to avoid confusion with the error class added to input * fix css when errors are added by the form renderer * fix form renderer to add the error message

[captcha] handle captcha validation properly in the captcha widget also, avoid error if pil isn't installed (only a recommendation)

[testlib] __errorurl may be a relative path, check we're actually removing base url first

[i18ncube] clear_rtag_objects imported when generating cube's .po file

[maintainance] don't crash if we've no in-memory repository

[db dump/restore] fix '#615840: db-dump must be always runnable' by setting repairing on config. Also, avoid to read the whole schema from the database on dump/restore: the bootstrap schema is enough.

cleanup

__missing__ main

[form] put qualified name on validation error, should fix #784299

[test] fixes when executed under apycot environment (also added ldap1 to test host /etc/hosts)

...

[config] move ConfigurationProblem to migration + refactor it to benefit from config methods

[config] more cleanup

deprecate __use__ and __recommend__

add egg support with dependencies auto-installation

fix dumb name error

fix _cubes dir

handle nicely typical installation other than debian package / mercurial forest * new _find_prefix method to detect installation prefix (maybe specified using CW_INSTALL_PREFIX environment variable) * set various resource accessors according to this, cleanup on the way * remove no more necessary APYCOT_ROOT hacks

When a TC failed to initiliase the database, the following test are SKIPPED When a TC failed to initiliase the database, the other test of the same TestCase are skipped with an explicite message instead of failing later for with an unrelated message.

[c-c] new server_maintenance hook, called on c-c shell / upgrade

[appobject] add version number to deprecation warning

update rql dependency

backport stable

fix migration script

[cleanup] useless try except

[source storage] refactor source sql generation and results handling to allow repository side callbacks for instance with the BytesFileSystemStorage, before this change: * fspath, _fsopen function were stored procedures executed on the database -> files had to be available both on the repository *and* the database host * we needed implementation for each handled database Now, those function are python callbacks executed when necessary on the repository side, on data comming from the database. The litle cons are: * you can't do anymore restriction on mapped attributes * you can't write queries which will return in the same rset column some mapped attributes (or not mapped the same way) / some not This seems much acceptable since: * it's much more easy to handle when you start having the db on another host than the repo * BFSS works seemlessly on any backend now * you don't bother that much about the cons (at least in the bfss case): you usually don't do any restriction on Bytes... Bonus points: BFSS is more efficient (no queries under the cover as it was done in the registered procedure) and we have a much nicer/efficient fspath implementation. IMO, that rocks :D

[bfss] fix name error

[source storage] on deletion, entity has no edited_attributes

[cleanup] rename dbms_helper to dbhelper for consistency

[entity] we also have to override dict.pop to properly maintain edited_attributes consistency

[bfss test] some minor refactoring. Test deletion + creation/update/deletion rollback

don't put hooks deleting orphan composites into the 'integrity' category, we usually want it when integrity is deactivated

[facet] allow to specify a target type on relation facets for case such as Keyword/CodeKeyword where we only targets Keyword

don't rewrite config's url during migration

le patch rql26 a été importé

backport stable

[D] book: update backup procedures in additional tips

[cleanup] remove accidentaly commited print

[test] don't try to reset session manager during test, this leads to weird failures when running multiple tests

don't update dontcheck until everything went fine: see usage in after_update_entity, where if we got an Unauthorized at hook time, we will retry and commit time.

[test] having two eid relations is now forbiden by rql

[web test] a basic test, to be completed

[test] don't clear rtags during tests, may cause breakage with manually import appobjects modules (see unittest_tracker for instance)

[web] move workflow related uicfg stuff to the workflow module

dont get eid from entity's dict

[notification] don't create 'changes' in transaction data until actually needed

backport pending_operations on hi-jacked session, see comment

[entity] fix dumb name error on .set_relations(rel=None)

[security] don't add attribute into skip_security if already in edited_attributes, else we may accidentaly skip a desired security check

[repo entity] we have to provide a __delitem__ implementation as well, see example in the docstring

[yams] on creation, specify relations anyway so we don't get spurious warning for accessing eid through the dict api on yams validation

[repo] fix warning message and implementation of bw compat unsafe_execute

stable is now 3.7

oldstable is now 3.6

backport stable

merge

Added tag cubicweb-debian-version-3.6.3-1 for changeset f9fce56d6a0c

Added tag cubicweb-version-3.6.3 for changeset 9c342fa4f1b7

Added tag cubicweb-debian-version-3.7.1-1 for changeset 9194740f070e

Added tag cubicweb-version-3.7.1 for changeset 6b0832bbd1da

backport debian control changes done in 3.7

3.6.3

ticket #636927, add ajaxhtmlhead in controller js_component

[widget] allow kwargs passed to Select widget

take care to empty breadcrumbs

fix bug with damned transmutation functionality, occuring when using an INSERT query which also add relation to the entity

update skeleton for __pkginfo__.py

don't use optional on final relation

fix security issue introduced by 4967:04543ed0bbdc: attributes explicitly set by hooks should not be checked by security hooks

[form js] apply vgodard patch fixing #750056: reorderTabindex js function crashes when form dom id is not 'entityForm'

missing unittest_main

[server] add unit tests for storages

[registry] trigger reload events on set_schema()

[source] only consider edited_attributes in source.preprocess_entity()

[source] implement storages right in the source rather than in hooks The problem is that Storage objects will most probably change entity's dictionary so that values are correctly set before the source's corresponding method (e.g. entity_added()) is called. For instance, the BFSFileStorage will change the original binary data and replace it with the destination file path in order to store the file path in the database. This change must be local to the source in order not to impact other hooks or attribute access during the transaction, the whole idea being that the same application code should work exactly the same whether or not a BFSStorage is used or not.

i18n update

3.7.1

backport stable

[etwist] cleanup, use cProfile instead of hotshot

[repo] start looping task on repo only config (though I bet more work is needed to get it actually working

[repo] call server_startup event in start_looping_task the repo is fully started (registered in pyro for instance)

[security] take care uidrels may contains a relation targetting something else than a constant node (eg IN function)

[ms] this test is already done before putting a relation in uidrels

cleanup

[security] experimental rqlst cache for read security: to activate using a 'security-rqlst-cache' flag in transaction data when you'll issue a lot of identic queries with only kwargs varying

[read security] minor optimizations

[web] get rid of spurious '[Errno 2] No such file or directory' on load This was caused by can_do_pdf_conversion() which tries to use '/usr/bin/fop'. When fop was not available, an OSError was raised by suprocess.Popen. This changeset first checks for /usr/bin/fop presence, then tries to run fop. If an error occurs, log it instead of printing it. NOTE for later: why do we test explicitly for "/usr/bin/fop" rather than simply "fop" ?

backport stable

[schema sync] fix another potential name error on synchronizing rdefs

[schema sync] don't try to synchronize infered relation defs, fixing a name error on the way

cleanup

don't update constraints when it's not necessary

[repo] don't call server_shutdown hooks when creating/repairing instance to be consistence with server_startup hooks

[migration] new build_descr argument to rqlexec on the migration helper

fix migration when upgrading from cw < 3.6 to cw 3.7

disable security when undoing

merge

[sql] fix FSPATH implementation plpy.execute() returns a list of dictionaries with column names as keys and corresponding values as values.

[web deletion] must use set_message

fix limited_rql w/ UNION query

backport stable

fix import

Fixed some of the documentation warnings when building the book with sphinx.

move pdf icon component with it's template so we don't see it when the template isn't available due to missing dependancies

backport stable

add missing warn imports