server/hook.py
author Rémi Cardona <remi.cardona@logilab.fr>
Tue, 15 Sep 2015 18:23:35 +0200
changeset 10682 7e111b606005
parent 10669 155c29e0ed1c
child 10903 da30851f9706
permissions -rw-r--r--
[py3k] unicode → six.text_type

# copyright 2003-2014 LOGILAB S.A. (Paris, FRANCE), all rights reserved.
# contact http://www.logilab.fr/ -- mailto:contact@logilab.fr
#
# This file is part of CubicWeb.
#
# CubicWeb is free software: you can redistribute it and/or modify it under the
# terms of the GNU Lesser General Public License as published by the Free
# Software Foundation, either version 2.1 of the License, or (at your option)
# any later version.
#
# CubicWeb is distributed in the hope that it will be useful, but WITHOUT
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
# FOR A PARTICULAR PURPOSE.  See the GNU Lesser General Public License for more
# details.
#
# You should have received a copy of the GNU Lesser General Public License along
# with CubicWeb.  If not, see <http://www.gnu.org/licenses/>.
"""
Generalities
------------

Paraphrasing the `emacs`_ documentation, let us say that hooks are an important
mechanism for customizing an application. A hook is basically a list of
functions to be called on some well-defined occasion (this is called `running
the hook`).

.. _`emacs`: http://www.gnu.org/software/emacs/manual/html_node/emacs/Hooks.html

Hooks
~~~~~

In |cubicweb|, hooks are subclasses of the :class:`~cubicweb.server.hook.Hook`
class. They are selected over a set of pre-defined `events` (and possibly more
conditions, hooks being selectable appobjects like views and components).  They
should implement a :meth:`~cubicweb.server.hook.Hook.__call__` method that will
be called when the hook is triggered.

There are two families of events: data events (before / after any individual
update of an entity / or a relation in the repository) and server events (such
as server startup or shutdown).  In a typical application, most of the hooks are
defined over data events.

Also, some :class:`~cubicweb.server.hook.Operation` may be registered by hooks,
which will be fired when the transaction is commited or rolled back.

The purpose of data event hooks is usually to complement the data model as
defined in the schema, which is static by nature and only provide a restricted
builtin set of dynamic constraints, with dynamic or value driven behaviours.
For instance they can serve the following purposes:

* enforcing constraints that the static schema cannot express (spanning several
  entities/relations, exotic value ranges and cardinalities, etc.)

* implement computed attributes

It is functionally equivalent to a `database trigger`_, except that database
triggers definition languages are not standardized, hence not portable (for
instance, PL/SQL works with Oracle and PostgreSQL but not SqlServer nor Sqlite).

.. _`database trigger`: http://en.wikipedia.org/wiki/Database_trigger


.. hint::

   It is a good practice to write unit tests for each hook. See an example in
   :ref:`hook_test`

Operations
~~~~~~~~~~

Operations are subclasses of the :class:`~cubicweb.server.hook.Operation` class
that may be created by hooks and scheduled to happen on `precommit`,
`postcommit` or `rollback` event (i.e. respectivly before/after a commit or
before a rollback of a transaction).

Hooks are being fired immediately on data operations, and it is sometime
necessary to delay the actual work down to a time where we can expect all
information to be there, or when all other hooks have run (though take case
since operations may themselves trigger hooks). Also while the order of
execution of hooks is data dependant (and thus hard to predict), it is possible
to force an order on operations.

So, for such case where you may miss some information that may be set later in
the transaction, you should instantiate an operation in the hook.

Operations may be used to:

* implements a validation check which needs that all relations be already set on
  an entity

* process various side effects associated with a transaction such as filesystem
  udpates, mail notifications, etc.


Events
------

Hooks are mostly defined and used to handle `dataflow`_ operations. It
means as data gets in (entities added, updated, relations set or
unset), specific events are issued and the Hooks matching these events
are called.

You can get the event that triggered a hook by accessing its `event`
attribute.

.. _`dataflow`: http://en.wikipedia.org/wiki/Dataflow


Entity modification related events
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

When called for one of these events, hook will have an `entity` attribute
containing the entity instance.

- `before_add_entity`, `before_update_entity`:

  On those events, you can access the modified attributes of the entity using
  the `entity.cw_edited` dictionary. The values can be modified and the old
  values can be retrieved.

  If you modify the `entity.cw_edited` dictionary in the hook, that is before
  the database operations take place, you will avoid the need to process a whole
  new rql query and the underlying backend query (eg usually sql) will contain
  the modified data. For example:

  .. sourcecode:: python

     self.entity.cw_edited['age'] = 42

  will modify the age before it is written to the backend storage.

  Similarly, removing an attribute from `cw_edited` will cancel its
  modification:

  .. sourcecode:: python

     del self.entity.cw_edited['age']

  On a `before_update_entity` event, you can access the old and new values:

  .. sourcecode:: python

     old, new = entity.cw_edited.oldnewvalue('age')

- `after_add_entity`, `after_update_entity`

  On those events, you can get the list of attributes that were modified using
  the `entity.cw_edited` dictionary, but you can not modify it or get the old
  value of an attribute.

- `before_delete_entity`, `after_delete_entity`

  On those events, the entity has no `cw_edited` dictionary.

.. note:: `self.entity.cw_set(age=42)` will set the `age` attribute to
  42. But to do so, it will generate a rql query that will have to be processed,
  hence may trigger some hooks, etc. This could lead to infinitely looping hooks.

Relation modification related events
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

When called for one of these events, hook will have `eidfrom`, `rtype`, `eidto`
attributes containing respectively the eid of the subject entity, the relation
type and the eid of the object entity.

* `before_add_relation`, `before_delete_relation`

  On those events, you can still get the original relation by issuing a rql query.

* `after_add_relation`, `after_delete_relation`

Specific selectors are shipped for these kinds of events, see in particular
:class:`~cubicweb.server.hook.match_rtype`.

Also note that relations can be added or deleted, but not updated.

Non data events
~~~~~~~~~~~~~~~

Hooks called on server start/maintenance/stop event (e.g.
`server_startup`, `server_maintenance`, `before_server_shutdown`,
`server_shutdown`) have a `repo` attribute, but *their `_cw` attribute
is None*.  The `server_startup` is called on regular startup, while
`server_maintenance` is called on cubicweb-ctl upgrade or shell
commands. `server_shutdown` is called anyway but connections to the
native source is impossible; `before_server_shutdown` handles that.

Hooks called on backup/restore event (eg `server_backup`,
`server_restore`) have a `repo` and a `timestamp` attributes, but
*their `_cw` attribute is None*.

Hooks called on session event (eg `session_open`, `session_close`) have no
special attribute.


API
---

Hooks control
~~~~~~~~~~~~~

It is sometimes convenient to explicitly enable or disable some hooks. For
instance if you want to disable some integrity checking hook. This can be
controlled more finely through the `category` class attribute, which is a string
giving a category name.  One can then uses the
:meth:`~cubicweb.server.session.Connection.deny_all_hooks_but` and
:meth:`~cubicweb.server.session.Connection.allow_all_hooks_but` context managers to
explicitly enable or disable some categories.

The existing categories are:

* ``security``, security checking hooks

* ``worfklow``, workflow handling hooks

* ``metadata``, hooks setting meta-data on newly created entities

* ``notification``, email notification hooks

* ``integrity``, data integrity checking hooks

* ``activeintegrity``, data integrity consistency hooks, that you should **never**
  want to disable

* ``syncsession``, hooks synchronizing existing sessions

* ``syncschema``, hooks synchronizing instance schema (including the physical database)

* ``email``, email address handling hooks

* ``bookmark``, bookmark entities handling hooks


Nothing precludes one to invent new categories and use existing mechanisms to
filter them in or out.


Hooks specific predicates
~~~~~~~~~~~~~~~~~~~~~~~~~
.. autoclass:: cubicweb.server.hook.match_rtype
.. autoclass:: cubicweb.server.hook.match_rtype_sets


Hooks and operations classes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. autoclass:: cubicweb.server.hook.Hook
.. autoclass:: cubicweb.server.hook.Operation
.. autoclass:: cubicweb.server.hook.LateOperation
.. autoclass:: cubicweb.server.hook.DataOperationMixIn
"""
from __future__ import print_function

__docformat__ = "restructuredtext en"

from warnings import warn
from logging import getLogger
from itertools import chain

from logilab.common.decorators import classproperty, cached
from logilab.common.deprecation import deprecated, class_renamed
from logilab.common.logging_ext import set_log_methods
from logilab.common.registry import (NotPredicate, OrPredicate,
                                     objectify_predicate)

from cubicweb import RegistryNotFound, server
from cubicweb.cwvreg import CWRegistry, CWRegistryStore
from cubicweb.predicates import ExpectedValuePredicate, is_instance
from cubicweb.appobject import AppObject

ENTITIES_HOOKS = set(('before_add_entity',    'after_add_entity',
                      'before_update_entity', 'after_update_entity',
                      'before_delete_entity', 'after_delete_entity'))
RELATIONS_HOOKS = set(('before_add_relation',   'after_add_relation' ,
                       'before_delete_relation','after_delete_relation'))
SYSTEM_HOOKS = set(('server_backup', 'server_restore',
                    'server_startup', 'server_maintenance',
                    'server_shutdown', 'before_server_shutdown',
                    'session_open', 'session_close'))
ALL_HOOKS = ENTITIES_HOOKS | RELATIONS_HOOKS | SYSTEM_HOOKS

def _iter_kwargs(entities, eids_from_to, kwargs):
    if not entities and not eids_from_to:
        yield kwargs
    elif entities:
        for entity in entities:
            kwargs['entity'] = entity
            yield kwargs
    else:
        for subject, object in eids_from_to:
            kwargs.update({'eidfrom': subject, 'eidto': object})
            yield kwargs


class HooksRegistry(CWRegistry):

    def register(self, obj, **kwargs):
        obj.check_events()
        super(HooksRegistry, self).register(obj, **kwargs)

    def call_hooks(self, event, cnx=None, **kwargs):
        """call `event` hooks for an entity or a list of entities (passed
        respectively as the `entity` or ``entities`` keyword argument).
        """
        kwargs['event'] = event
        if cnx is None: # True for events such as server_start
            for hook in sorted(self.possible_objects(cnx, **kwargs),
                               key=lambda x: x.order):
                hook()
        else:
            if 'entities' in kwargs:
                assert 'entity' not in kwargs, \
                       'can\'t pass "entities" and "entity" arguments simultaneously'
                assert 'eids_from_to' not in kwargs, \
                       'can\'t pass "entities" and "eids_from_to" arguments simultaneously'
                entities = kwargs.pop('entities')
                eids_from_to = []
            elif 'eids_from_to' in kwargs:
                entities = []
                eids_from_to = kwargs.pop('eids_from_to')
            else:
                entities = []
                eids_from_to = []
            pruned = self.get_pruned_hooks(cnx, event,
                                           entities, eids_from_to, kwargs)

            # by default, hooks are executed with security turned off
            with cnx.security_enabled(read=False):
                for _kwargs in _iter_kwargs(entities, eids_from_to, kwargs):
                    hooks = sorted(self.filtered_possible_objects(pruned, cnx, **_kwargs),
                                   key=lambda x: x.order)
                    debug = server.DEBUG & server.DBG_HOOKS
                    with cnx.security_enabled(write=False):
                        with cnx.running_hooks_ops():
                            for hook in hooks:
                                if debug:
                                    print(event, _kwargs, hook)
                                hook()

    def get_pruned_hooks(self, cnx, event, entities, eids_from_to, kwargs):
        """return a set of hooks that should not be considered by filtered_possible objects

        the idea is to make a first pass over all the hooks in the
        registry and to mark put some of them in a pruned list. The
        pruned hooks are the one which:

        * are disabled at the connection level

        * have a selector containing a :class:`match_rtype` or an
          :class:`is_instance` predicate which does not match the rtype / etype
          of the relations / entities for which we are calling the hooks. This
          works because the repository calls the hooks grouped by rtype or by
          etype when using the entities or eids_to_from keyword arguments

        Only hooks with a simple predicate or an AndPredicate of simple
        predicates are considered for disabling.

        """
        if 'entity' in kwargs:
            entities = [kwargs['entity']]
        if len(entities):
            look_for_selector = is_instance
            etype = entities[0].__regid__
        elif 'rtype' in kwargs:
            look_for_selector = match_rtype
            etype = None
        else: # nothing to prune, how did we get there ???
            return set()
        cache_key = (event, kwargs.get('rtype'), etype)
        pruned = cnx.pruned_hooks_cache.get(cache_key)
        if pruned is not None:
            return pruned
        pruned = set()
        cnx.pruned_hooks_cache[cache_key] = pruned
        if look_for_selector is not None:
            for id, hooks in self.items():
                for hook in hooks:
                    enabled_cat, main_filter = hook.filterable_selectors()
                    if enabled_cat is not None:
                        if not enabled_cat(hook, cnx):
                            pruned.add(hook)
                            continue
                    if main_filter is not None:
                        if isinstance(main_filter, match_rtype) and \
                           (main_filter.frometypes is not None  or \
                            main_filter.toetypes is not None):
                            continue
                        first_kwargs = next(_iter_kwargs(entities, eids_from_to, kwargs))
                        if not main_filter(hook, cnx, **first_kwargs):
                            pruned.add(hook)
        return pruned


    def filtered_possible_objects(self, pruned, *args, **kwargs):
        for appobjects in self.values():
            if pruned:
                filtered_objects = [obj for obj in appobjects if obj not in pruned]
                if not filtered_objects:
                    continue
            else:
                filtered_objects = appobjects
            obj = self._select_best(filtered_objects,
                                    *args, **kwargs)
            if obj is None:
                continue
            yield obj

class HooksManager(object):
    def __init__(self, vreg):
        self.vreg = vreg

    def call_hooks(self, event, cnx=None, **kwargs):
        try:
            registry = self.vreg['%s_hooks' % event]
        except RegistryNotFound:
            return # no hooks for this event
        registry.call_hooks(event, cnx, **kwargs)


for event in ALL_HOOKS:
    CWRegistryStore.REGISTRY_FACTORY['%s_hooks' % event] = HooksRegistry


# some hook specific predicates #################################################

@objectify_predicate
def enabled_category(cls, req, **kwargs):
    if req is None:
        return True # XXX how to deactivate server startup / shutdown event
    return req.is_hook_activated(cls)

@objectify_predicate
def issued_from_user_query(cls, req, **kwargs):
    return 0 if req.hooks_in_progress else 1

from_dbapi_query = class_renamed('from_dbapi_query',
                                 issued_from_user_query,
                                 message='[3.21] ')


class rechain(object):
    def __init__(self, *iterators):
        self.iterators = iterators
    def __iter__(self):
        return iter(chain(*self.iterators))


class match_rtype(ExpectedValuePredicate):
    """accept if the relation type is found in expected ones. Optional
    named parameters `frometypes` and `toetypes` can be used to restrict
    target subject and/or object entity types of the relation.

    :param \*expected: possible relation types
    :param frometypes: candidate entity types as subject of relation
    :param toetypes: candidate entity types as object of relation
    """
    def __init__(self, *expected, **more):
        self.expected = expected
        self.frometypes = more.pop('frometypes', None)
        self.toetypes = more.pop('toetypes', None)
        assert not more, "unexpected kwargs in match_rtype: %s" % more

    def __call__(self, cls, req, *args, **kwargs):
        if kwargs.get('rtype') not in self.expected:
            return 0
        if self.frometypes is not None and \
               req.entity_metas(kwargs['eidfrom'])['type'] not in self.frometypes:
            return 0
        if self.toetypes is not None and \
               req.entity_metas(kwargs['eidto'])['type'] not in self.toetypes:
            return 0
        return 1


class match_rtype_sets(ExpectedValuePredicate):
    """accept if the relation type is in one of the sets given as initializer
    argument. The goal of this predicate is that it keeps reference to original sets,
    so modification to thoses sets are considered by the predicate. For instance

    .. sourcecode:: python

      MYSET = set()

      class Hook1(Hook):
          __regid__ = 'hook1'
          __select__ = Hook.__select__ & match_rtype_sets(MYSET)
          ...

      class Hook2(Hook):
          __regid__ = 'hook2'
          __select__ = Hook.__select__ & match_rtype_sets(MYSET)

    Client code can now change `MYSET`, this will changes the selection criteria
    of :class:`Hook1` and :class:`Hook1`.
    """

    def __init__(self, *expected):
        self.expected = expected

    def __call__(self, cls, req, *args, **kwargs):
        for rel_set in self.expected:
            if kwargs.get('rtype') in rel_set:
                return 1
        return 0


# base class for hook ##########################################################

class Hook(AppObject):
    """Base class for hook.

    Hooks being appobjects like views, they have a `__regid__` and a `__select__`
    class attribute. Like all appobjects, hooks have the `self._cw` attribute which
    represents the current connection. In entity hooks, a `self.entity` attribute is
    also present.

    The `events` tuple is used by the base class selector to dispatch the hook
    on the right events. It is possible to dispatch on multiple events at once
    if needed (though take care as hook attribute may vary as described above).

    .. Note::

      Do not forget to extend the base class selectors as in:

      .. sourcecode:: python

          class MyHook(Hook):
            __regid__ = 'whatever'
            __select__ = Hook.__select__ & is_instance('Person')

      else your hooks will be called madly, whatever the event.
    """
    __select__ = enabled_category()
    # set this in derivated classes
    events = None
    category = None
    order = 0
    # stop pylint from complaining about missing attributes in Hooks classes
    eidfrom = eidto = entity = rtype = repo = None

    @classmethod
    @cached
    def filterable_selectors(cls):
        search = cls.__select__.search_selector
        if search((NotPredicate, OrPredicate)):
            return None, None
        enabled_cat = search(enabled_category)
        main_filter = search((is_instance, match_rtype))
        return enabled_cat, main_filter

    @classmethod
    def check_events(cls):
        try:
            for event in cls.events:
                if event not in ALL_HOOKS:
                    raise Exception('bad event %s on %s.%s' % (
                        event, cls.__module__, cls.__name__))
        except AttributeError:
            raise
        except TypeError:
            raise Exception('bad .events attribute %s on %s.%s' % (
                cls.events, cls.__module__, cls.__name__))

    @classmethod
    def __registered__(cls, reg):
        cls.check_events()

    @classproperty
    def __registries__(cls):
        if cls.events is None:
            return []
        return ['%s_hooks' % ev for ev in cls.events]

    known_args = set(('entity', 'rtype', 'eidfrom', 'eidto', 'repo', 'timestamp'))
    def __init__(self, req, event, **kwargs):
        for arg in self.known_args:
            if arg in kwargs:
                setattr(self, arg, kwargs.pop(arg))
        super(Hook, self).__init__(req, **kwargs)
        self.event = event

set_log_methods(Hook, getLogger('cubicweb.hook'))


# abtract hooks for relation propagation #######################################
# See example usage in hooks of the nosylist cube

class PropagateRelationHook(Hook):
    """propagate some `main_rtype` relation on entities linked as object of
    `subject_relations` or as subject of `object_relations` (the watched
    relations).

    This hook ensure that when one of the watched relation is added, the
    `main_rtype` relation is added to the target entity of the relation.
    Notice there are no default behaviour defined when a watched relation is
    deleted, you'll have to handle this by yourself.

    You usually want to use the :class:`match_rtype_sets` predicate on concrete
    classes.
    """
    events = ('after_add_relation',)

    # to set in concrete class
    main_rtype = None
    subject_relations = None
    object_relations = None

    def __call__(self):
        assert self.main_rtype
        for eid in (self.eidfrom, self.eidto):
            etype = self._cw.entity_metas(eid)['type']
            if self.main_rtype not in self._cw.vreg.schema.eschema(etype).subjrels:
                return
        if self.rtype in self.subject_relations:
            meid, seid = self.eidfrom, self.eidto
        else:
            assert self.rtype in self.object_relations
            meid, seid = self.eidto, self.eidfrom
        self._cw.execute(
            'SET E %s P WHERE X %s P, X eid %%(x)s, E eid %%(e)s, NOT E %s P'
            % (self.main_rtype, self.main_rtype, self.main_rtype),
            {'x': meid, 'e': seid})


class PropagateRelationAddHook(Hook):
    """Propagate to entities at the end of watched relations when a `main_rtype`
    relation is added.

    `subject_relations` and `object_relations` attributes should be specified on
    subclasses and are usually shared references with attributes of the same
    name on :class:`PropagateRelationHook`.

    Because of those shared references, you can use `skip_subject_relations` and
    `skip_object_relations` attributes when you don't want to propagate to
    entities linked through some particular relations.
    """
    events = ('after_add_relation',)

    # to set in concrete class (mandatory)
    subject_relations = None
    object_relations = None
    # to set in concrete class (optionaly)
    skip_subject_relations = ()
    skip_object_relations = ()

    def __call__(self):
        eschema = self._cw.vreg.schema.eschema(self._cw.entity_metas(self.eidfrom)['type'])
        execute = self._cw.execute
        for rel in self.subject_relations:
            if rel in eschema.subjrels and not rel in self.skip_subject_relations:
                execute('SET R %s P WHERE X eid %%(x)s, P eid %%(p)s, '
                        'X %s R, NOT R %s P' % (self.rtype, rel, self.rtype),
                        {'x': self.eidfrom, 'p': self.eidto})
        for rel in self.object_relations:
            if rel in eschema.objrels and not rel in self.skip_object_relations:
                execute('SET R %s P WHERE X eid %%(x)s, P eid %%(p)s, '
                        'R %s X, NOT R %s P' % (self.rtype, rel, self.rtype),
                        {'x': self.eidfrom, 'p': self.eidto})


class PropagateRelationDelHook(PropagateRelationAddHook):
    """Propagate to entities at the end of watched relations when a `main_rtype`
    relation is deleted.

    This is the opposite of the :class:`PropagateRelationAddHook`, see its
    documentation for how to use this class.
    """
    events = ('after_delete_relation',)

    def __call__(self):
        eschema = self._cw.vreg.schema.eschema(self._cw.entity_metas(self.eidfrom)['type'])
        execute = self._cw.execute
        for rel in self.subject_relations:
            if rel in eschema.subjrels and not rel in self.skip_subject_relations:
                execute('DELETE R %s P WHERE X eid %%(x)s, P eid %%(p)s, '
                        'X %s R' % (self.rtype, rel),
                        {'x': self.eidfrom, 'p': self.eidto})
        for rel in self.object_relations:
            if rel in eschema.objrels and not rel in self.skip_object_relations:
                execute('DELETE R %s P WHERE X eid %%(x)s, P eid %%(p)s, '
                        'R %s X' % (self.rtype, rel),
                        {'x': self.eidfrom, 'p': self.eidto})



# abstract classes for operation ###############################################

class Operation(object):
    """Base class for operations.

    Operation may be instantiated in the hooks' `__call__` method. It always
    takes a connection object as first argument (accessible as `.cnx` from the
    operation instance), and optionally all keyword arguments needed by the
    operation. These keyword arguments will be accessible as attributes from the
    operation instance.

    An operation is triggered on connections set events related to commit /
    rollback transations. Possible events are:

    * `precommit`:

      the transaction is being prepared for commit. You can freely do any heavy
      computation, raise an exception if the commit can't go. or even add some
      new operations during this phase. If you do anything which has to be
      reverted if the commit fails afterwards (eg altering the file system for
      instance), you'll have to support the 'revertprecommit' event to revert
      things by yourself

    * `revertprecommit`:

      if an operation failed while being pre-commited, this event is triggered
      for all operations which had their 'precommit' event already fired to let
      them revert things (including the operation which made the commit fail)

    * `rollback`:

      the transaction has been either rolled back either:

       * intentionaly
       * a 'precommit' event failed, in which case all operations are rolled back
         once 'revertprecommit'' has been called

    * `postcommit`:

      the transaction is over. All the ORM entities accessed by the earlier
      transaction are invalid. If you need to work on the database, you need to
      start a new transaction, for instance using a new internal connection,
      which you will need to commit.

    For an operation to support an event, one has to implement the `<event
    name>_event` method with no arguments.

    The order of operations may be important, and is controlled according to
    the insert_index's method output (whose implementation vary according to the
    base hook class used).
    """

    def __init__(self, cnx, **kwargs):
        self.cnx = cnx
        self.__dict__.update(kwargs)
        self.register(cnx)
        # execution information
        self.processed = None # 'precommit', 'commit'
        self.failed = False

    @property
    @deprecated('[3.19] Operation.session is deprecated, use Operation.cnx instead')
    def session(self):
        return self.cnx

    def register(self, cnx):
        cnx.add_operation(self, self.insert_index())

    def insert_index(self):
        """return the index of the latest instance which is not a
        LateOperation instance
        """
        # faster by inspecting operation in reverse order for heavy transactions
        i = None
        for i, op in enumerate(reversed(self.cnx.pending_operations)):
            if isinstance(op, (LateOperation, SingleLastOperation)):
                continue
            return -i or None
        if i is None:
            return None
        return -(i + 1)

    def handle_event(self, event):
        """delegate event handling to the opertaion"""
        getattr(self, event)()

    def precommit_event(self):
        """the observed connections set is preparing a commit"""

    def revertprecommit_event(self):
        """an error went when pre-commiting this operation or a later one

        should revert pre-commit's changes but take care, they may have not
        been all considered if it's this operation which failed
        """

    def rollback_event(self):
        """the observed connections set has been rolled back

        do nothing by default
        """

    def postcommit_event(self):
        """the observed connections set has committed"""

    # these are overridden by set_log_methods below
    # only defining here to prevent pylint from complaining
    info = warning = error = critical = exception = debug = lambda msg,*a,**kw: None

set_log_methods(Operation, getLogger('cubicweb.session'))

def _container_add(container, value):
    {set: set.add, list: list.append}[container.__class__](container, value)


class DataOperationMixIn(object):
    """Mix-in class to ease applying a single operation on a set of data,
    avoiding to create as many as operation as they are individual modification.
    The body of the operation must then iterate over the values that have been
    stored in a single operation instance.

    You should try to use this instead of creating on operation for each
    `value`, since handling operations becomes costly on massive data import.

    Usage looks like:

    .. sourcecode:: python

        class MyEntityHook(Hook):
            __regid__ = 'my.entity.hook'
            __select__ = Hook.__select__ & is_instance('MyEntity')
            events = ('after_add_entity',)

            def __call__(self):
                MyOperation.get_instance(self._cw).add_data(self.entity)


        class MyOperation(DataOperationMixIn, Operation):
            def precommit_event(self):
                for bucket in self.get_data():
                    process(bucket)

    You can modify the `containercls` class attribute, which defines the
    container class that should be instantiated to hold payloads. An instance is
    created on instantiation, and then the :meth:`add_data` method will add the
    given data to the existing container. Default to a `set`. Give `list` if you
    want to keep arrival ordering. You can also use another kind of container
    by redefining :meth:`_build_container` and :meth:`add_data`

    More optional parameters can be given to the `get_instance` operation, that
    will be given to the operation constructor (for obvious reasons those
    parameters should not vary accross different calls to this method for a
    given operation).

    .. Note::
        For sanity reason `get_data` will reset the operation, so that once
        the operation has started its treatment, if some hook want to push
        additional data to this same operation, a new instance will be created
        (else that data has a great chance to be never treated). This implies:

        * you should **always** call `get_data` when starting treatment

        * you should **never** call `get_data` for another reason.
    """
    containercls = set

    @classproperty
    def data_key(cls):
        return ('cw.dataops', cls.__name__)

    @classmethod
    def get_instance(cls, cnx, **kwargs):
        # no need to lock: transaction_data already comes from thread's local storage
        try:
            return cnx.transaction_data[cls.data_key]
        except KeyError:
            op = cnx.transaction_data[cls.data_key] = cls(cnx, **kwargs)
            return op

    def __init__(self, *args, **kwargs):
        super(DataOperationMixIn, self).__init__(*args, **kwargs)
        self._container = self._build_container()
        self._processed = False

    def __contains__(self, value):
        return value in self._container

    def _build_container(self):
        return self.containercls()

    def union(self, data):
        """only when container is a set"""
        assert not self._processed, """Trying to add data to a closed operation.
Iterating over operation data closed it and should be reserved to precommit /
postcommit method of the operation."""
        self._container |= data

    def add_data(self, data):
        assert not self._processed, """Trying to add data to a closed operation.
Iterating over operation data closed it and should be reserved to precommit /
postcommit method of the operation."""
        _container_add(self._container, data)

    def remove_data(self, data):
        assert not self._processed, """Trying to add data to a closed operation.
Iterating over operation data closed it and should be reserved to precommit /
postcommit method of the operation."""
        self._container.remove(data)

    def get_data(self):
        assert not self._processed, """Trying to get data from a closed operation.
Iterating over operation data closed it and should be reserved to precommit /
postcommit method of the operation."""
        self._processed = True
        op = self.cnx.transaction_data.pop(self.data_key)
        assert op is self, "Bad handling of operation data, found %s instead of %s for key %s" % (
            op, self, self.data_key)
        return self._container



class LateOperation(Operation):
    """special operation which should be called after all possible (ie non late)
    operations
    """
    def insert_index(self):
        """return the index of  the lastest instance which is not a
        SingleLastOperation instance
        """
        # faster by inspecting operation in reverse order for heavy transactions
        i = None
        for i, op in enumerate(reversed(self.cnx.pending_operations)):
            if isinstance(op, SingleLastOperation):
                continue
            return -i or None
        if i is None:
            return None
        return -(i + 1)



class SingleLastOperation(Operation):
    """special operation which should be called once and after all other
    operations
    """

    def register(self, cnx):
        """override register to handle cases where this operation has already
        been added
        """
        operations = cnx.pending_operations
        index = self.equivalent_index(operations)
        if index is not None:
            equivalent = operations.pop(index)
        else:
            equivalent = None
        cnx.add_operation(self, self.insert_index())
        return equivalent

    def equivalent_index(self, operations):
        """return the index of the equivalent operation if any"""
        for i, op in enumerate(reversed(operations)):
            if op.__class__ is self.__class__:
                return -(i+1)
        return None

    def insert_index(self):
        return None


class SendMailOp(SingleLastOperation):
    def __init__(self, cnx, msg=None, recipients=None, **kwargs):
        # may not specify msg yet, as
        # `cubicweb.sobjects.supervision.SupervisionMailOp`
        if msg is not None:
            assert recipients
            self.to_send = [(msg, recipients)]
        else:
            assert recipients is None
            self.to_send = []
        super(SendMailOp, self).__init__(cnx, **kwargs)

    def register(self, cnx):
        previous = super(SendMailOp, self).register(cnx)
        if previous:
            self.to_send = previous.to_send + self.to_send

    def postcommit_event(self):
        self.cnx.repo.threaded_task(self.sendmails)

    def sendmails(self):
        self.cnx.vreg.config.sendmails(self.to_send)


class RQLPrecommitOperation(Operation):
    # to be defined in concrete classes
    rqls = None

    def precommit_event(self):
        execute = self.cnx.execute
        for rql in self.rqls:
            execute(*rql)


class CleanupNewEidsCacheOp(DataOperationMixIn, SingleLastOperation):
    """on rollback of a insert query we have to remove from repository's
    type/source cache eids of entities added in that transaction.

    NOTE: querier's rqlst/solutions cache may have been polluted too with
    queries such as Any X WHERE X eid 32 if 32 has been rolled back however
    generated queries are unpredictable and analysing all the cache probably
    too expensive. Notice that there is no pb when using args to specify eids
    instead of giving them into the rql string.
    """
    data_key = 'neweids'

    def rollback_event(self):
        """the observed connections set has been rolled back,
        remove inserted eid from repository type/source cache
        """
        try:
            self.cnx.repo.clear_caches(self.get_data())
        except KeyError:
            pass

class CleanupDeletedEidsCacheOp(DataOperationMixIn, SingleLastOperation):
    """on commit of delete query, we have to remove from repository's
    type/source cache eids of entities deleted in that transaction.
    """
    data_key = 'pendingeids'
    def postcommit_event(self):
        """the observed connections set has been rolled back,
        remove inserted eid from repository type/source cache
        """
        try:
            eids = self.get_data()
            self.cnx.repo.clear_caches(eids)
            self.cnx.repo.app_instances_bus.publish(['delete'] + list(str(eid) for eid in eids))
        except KeyError:
            pass