cubicweb/server/repository.py
author Laurent Peuch <cortex@worlddomination.be>
Thu, 05 Dec 2019 18:37:27 +0100
changeset 12805 8cf02773a6b3
parent 12804 6de3af773073
child 12806 91e262556573
permissions -rw-r--r--
[refactoring] blank lines to make core more readable
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
11348
70337ad23145 pep8 + docstrings and comments improvments
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11204
diff changeset
     1
# copyright 2003-2016 LOGILAB S.A. (Paris, FRANCE), all rights reserved.
5421
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5376
diff changeset
     2
# contact http://www.logilab.fr/ -- mailto:contact@logilab.fr
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5376
diff changeset
     3
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5376
diff changeset
     4
# This file is part of CubicWeb.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5376
diff changeset
     5
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5376
diff changeset
     6
# CubicWeb is free software: you can redistribute it and/or modify it under the
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5376
diff changeset
     7
# terms of the GNU Lesser General Public License as published by the Free
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5376
diff changeset
     8
# Software Foundation, either version 2.1 of the License, or (at your option)
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5376
diff changeset
     9
# any later version.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5376
diff changeset
    10
#
5424
8ecbcbff9777 replace logilab-common by CubicWeb in disclaimer
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5421
diff changeset
    11
# CubicWeb is distributed in the hope that it will be useful, but WITHOUT
5421
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5376
diff changeset
    12
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5376
diff changeset
    13
# FOR A PARTICULAR PURPOSE.  See the GNU Lesser General Public License for more
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5376
diff changeset
    14
# details.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5376
diff changeset
    15
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5376
diff changeset
    16
# You should have received a copy of the GNU Lesser General Public License along
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5376
diff changeset
    17
# with CubicWeb.  If not, see <http://www.gnu.org/licenses/>.
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    18
"""Defines the central class for the CubicWeb RQL server: the repository.
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    19
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    20
The repository is an abstraction allowing execution of rql queries against
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    21
data sources. Most of the work is actually done in helper classes. The
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    22
repository mainly:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    23
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    24
* brings these classes all together to provide a single access
2476
1294a6bdf3bf application -> instance where it makes sense
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2473
diff changeset
    25
  point to a cubicweb instance.
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    26
* handles session management
5587
72679e450f6d [web] dont attempt to update last login time on ldap users, avoiding spurious tb in logs (closes #914464)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
    27
"""
11767
432f87a63057 flake8 and all
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11763
diff changeset
    28
6390
3766853656d7 [repo] add a find_users method for under-privileged accounts
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 6368
diff changeset
    29
from itertools import chain
9113
af6efc15fc90 [repository] add an ``internal_cnx`` method to replace ``internal_session``
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 9105
diff changeset
    30
from contextlib import contextmanager
11811
f09efeead7f9 Fix broken flake8 configuration
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11774
diff changeset
    31
from logging import getLogger
12567
26744ad37953 Drop python2 support
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12542
diff changeset
    32
import queue
10608
7fc548d9dd8e [py3k] import queue using six.moves
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10589
diff changeset
    33
6427
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6426
diff changeset
    34
from logilab.common.decorators import cached, clear_cache
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    35
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    36
from yams import BadSchemaDefinition
6390
3766853656d7 [repo] add a find_users method for under-privileged accounts
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 6368
diff changeset
    37
from rql.utils import rqlvar_maker
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    38
12054
d830ea048c97 [repo] Fix flake8 error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12051
diff changeset
    39
from cubicweb import (CW_MIGRATION_MAP,
2709
6ee816eb9f25 [hooksmanager] hooks should be reseted after vreg reload
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2708
diff changeset
    40
                      UnknownEid, AuthenticationError, ExecutionError,
12025
ba79f4a4f5d8 [session] Stop relying on _sessions to synchronize living session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12021
diff changeset
    41
                      UniqueTogetherError, ViolatedConstraint)
11811
f09efeead7f9 Fix broken flake8 configuration
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11774
diff changeset
    42
from cubicweb import set_log_methods
2839
6419af16faa0 imports cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2835
diff changeset
    43
from cubicweb import cwvreg, schema, server
12021
cc8135ecfbb8 [repo] Drop _get_session method
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12016
diff changeset
    44
from cubicweb.server import utils, hook, querier, sources
12043
b8d2e6b9f548 Stop using Session on the repository side
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12036
diff changeset
    45
from cubicweb.server.session import InternalManager, Connection
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    46
11348
70337ad23145 pep8 + docstrings and comments improvments
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11204
diff changeset
    47
70337ad23145 pep8 + docstrings and comments improvments
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11204
diff changeset
    48
NO_CACHE_RELATIONS = set([
70337ad23145 pep8 + docstrings and comments improvments
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11204
diff changeset
    49
    ('owned_by', 'object'),
70337ad23145 pep8 + docstrings and comments improvments
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11204
diff changeset
    50
    ('created_by', 'object'),
70337ad23145 pep8 + docstrings and comments improvments
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11204
diff changeset
    51
    ('cw_source', 'object'),
70337ad23145 pep8 + docstrings and comments improvments
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11204
diff changeset
    52
])
70337ad23145 pep8 + docstrings and comments improvments
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11204
diff changeset
    53
7134
01544b6d98fa allow CW to act as a Pyro Server without registering the server to a Pyro Nameserver (closes #1528533)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7085
diff changeset
    54
8456
c912d82f2166 [repository] drop unused argument
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8433
diff changeset
    55
def prefill_entity_caches(entity):
10368
1691be295517 [repository] rename session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 10365
diff changeset
    56
    cnx = entity._cw
6467
869813c2dd40 [entity, repo] move prefill_entity_cache function back to repository due to usage of del_existing_rel_if_needed function which is also defined there
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6466
diff changeset
    57
    # prefill entity relation caches
869813c2dd40 [entity, repo] move prefill_entity_cache function back to repository due to usage of del_existing_rel_if_needed function which is also defined there
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6466
diff changeset
    58
    for rschema in entity.e_schema.subject_relations():
869813c2dd40 [entity, repo] move prefill_entity_cache function back to repository due to usage of del_existing_rel_if_needed function which is also defined there
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6466
diff changeset
    59
        rtype = str(rschema)
7373
e5e6ef56cfb5 [repo cache] there are some relations we don't want to cache, they may cause memory leak
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7305
diff changeset
    60
        if rtype in schema.VIRTUAL_RTYPES or (rtype, 'subject') in NO_CACHE_RELATIONS:
6467
869813c2dd40 [entity, repo] move prefill_entity_cache function back to repository due to usage of del_existing_rel_if_needed function which is also defined there
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6466
diff changeset
    61
            continue
869813c2dd40 [entity, repo] move prefill_entity_cache function back to repository due to usage of del_existing_rel_if_needed function which is also defined there
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6466
diff changeset
    62
        if rschema.final:
869813c2dd40 [entity, repo] move prefill_entity_cache function back to repository due to usage of del_existing_rel_if_needed function which is also defined there
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6466
diff changeset
    63
            entity.cw_attr_cache.setdefault(rtype, None)
869813c2dd40 [entity, repo] move prefill_entity_cache function back to repository due to usage of del_existing_rel_if_needed function which is also defined there
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6466
diff changeset
    64
        else:
869813c2dd40 [entity, repo] move prefill_entity_cache function back to repository due to usage of del_existing_rel_if_needed function which is also defined there
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6466
diff changeset
    65
            entity.cw_set_relation_cache(rtype, 'subject',
10368
1691be295517 [repository] rename session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 10365
diff changeset
    66
                                         cnx.empty_rset())
6467
869813c2dd40 [entity, repo] move prefill_entity_cache function back to repository due to usage of del_existing_rel_if_needed function which is also defined there
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6466
diff changeset
    67
    for rschema in entity.e_schema.object_relations():
869813c2dd40 [entity, repo] move prefill_entity_cache function back to repository due to usage of del_existing_rel_if_needed function which is also defined there
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6466
diff changeset
    68
        rtype = str(rschema)
7373
e5e6ef56cfb5 [repo cache] there are some relations we don't want to cache, they may cause memory leak
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7305
diff changeset
    69
        if rtype in schema.VIRTUAL_RTYPES or (rtype, 'object') in NO_CACHE_RELATIONS:
6467
869813c2dd40 [entity, repo] move prefill_entity_cache function back to repository due to usage of del_existing_rel_if_needed function which is also defined there
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6466
diff changeset
    70
            continue
10368
1691be295517 [repository] rename session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 10365
diff changeset
    71
        entity.cw_set_relation_cache(rtype, 'object', cnx.empty_rset())
6467
869813c2dd40 [entity, repo] move prefill_entity_cache function back to repository due to usage of del_existing_rel_if_needed function which is also defined there
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6466
diff changeset
    72
11348
70337ad23145 pep8 + docstrings and comments improvments
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11204
diff changeset
    73
10368
1691be295517 [repository] rename session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 10365
diff changeset
    74
def del_existing_rel_if_needed(cnx, eidfrom, rtype, eidto):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    75
    """delete existing relation when adding a new one if card is 1 or ?
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    76
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    77
    have to be done once the new relation has been inserted to avoid having
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    78
    an entity without a relation for some time
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    79
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    80
    this kind of behaviour has to be done in the repository so we don't have
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    81
    hooks order hazardness
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    82
    """
7503
bc30c2faaadc [repository] refactor and optimize '?1' relation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7502
diff changeset
    83
    # skip that if integrity explicitly disabled
10368
1691be295517 [repository] rename session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 10365
diff changeset
    84
    if not cnx.is_hook_category_activated('activeintegrity'):
4687
082e66184f71 [integrity] skip auto-delete for internal sessions (but *not* for super session, see 4642:921737d2e3a8)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4686
diff changeset
    85
        return
10368
1691be295517 [repository] rename session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 10365
diff changeset
    86
    rdef = cnx.rtype_eids_rdef(rtype, eidfrom, eidto)
7502
e7190f7e850e [session] deprecates schema_rproperty in favor of more optimized rtype_eids_rdef which return the rdef (so reusable to gather other data)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7501
diff changeset
    87
    card = rdef.cardinality
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    88
    # one may be tented to check for neweids but this may cause more than one
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    89
    # relation even with '1?'  cardinality if thoses relations are added in the
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    90
    # same transaction where the entity is being created. This never occurs from
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    91
    # the web interface but may occurs during test or dbapi connection (though
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    92
    # not expected for this).  So: don't do it, we pretend to ensure repository
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    93
    # consistency.
4673
6f8b925a29f4 auto-delete relation with single cardinality using execute, not unsafe_execute, so permissions are correctly checked
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4643
diff changeset
    94
    #
6189
9fa02f352c8c cleanups
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6188
diff changeset
    95
    # notes:
9fa02f352c8c cleanups
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6188
diff changeset
    96
    # * inlined relations will be implicitly deleted for the subject entity
9fa02f352c8c cleanups
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6188
diff changeset
    97
    # * we don't want read permissions to be applied but we want delete
9fa02f352c8c cleanups
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6188
diff changeset
    98
    #   permission to be checked
7503
bc30c2faaadc [repository] refactor and optimize '?1' relation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7502
diff changeset
    99
    if card[0] in '1?':
10368
1691be295517 [repository] rename session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 10365
diff changeset
   100
        with cnx.security_enabled(read=False):
1691be295517 [repository] rename session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 10365
diff changeset
   101
            cnx.execute('DELETE X %s Y WHERE X eid %%(x)s, '
1691be295517 [repository] rename session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 10365
diff changeset
   102
                        'NOT Y eid %%(y)s' % rtype,
1691be295517 [repository] rename session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 10365
diff changeset
   103
                        {'x': eidfrom, 'y': eidto})
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   104
    if card[1] in '1?':
10368
1691be295517 [repository] rename session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 10365
diff changeset
   105
        with cnx.security_enabled(read=False):
1691be295517 [repository] rename session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 10365
diff changeset
   106
            cnx.execute('DELETE X %s Y WHERE Y eid %%(y)s, '
1691be295517 [repository] rename session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 10365
diff changeset
   107
                        'NOT X eid %%(x)s' % rtype,
1691be295517 [repository] rename session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 10365
diff changeset
   108
                        {'x': eidfrom, 'y': eidto})
1482
93c613913912 delete-trailing-whitespaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   109
3694
33dbb1da1db9 cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3689
diff changeset
   110
10368
1691be295517 [repository] rename session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 10365
diff changeset
   111
def preprocess_inlined_relations(cnx, entity):
8547
f23ac525ddd1 [datafeed] properly call hooks for inlined relations on entity creation. Closes #2481156
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8456
diff changeset
   112
    """when an entity is added, check if it has some inlined relation which
f23ac525ddd1 [datafeed] properly call hooks for inlined relations on entity creation. Closes #2481156
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8456
diff changeset
   113
    requires to be extrated for proper call hooks
f23ac525ddd1 [datafeed] properly call hooks for inlined relations on entity creation. Closes #2481156
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8456
diff changeset
   114
    """
f23ac525ddd1 [datafeed] properly call hooks for inlined relations on entity creation. Closes #2481156
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8456
diff changeset
   115
    relations = []
10368
1691be295517 [repository] rename session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 10365
diff changeset
   116
    activeintegrity = cnx.is_hook_category_activated('activeintegrity')
8547
f23ac525ddd1 [datafeed] properly call hooks for inlined relations on entity creation. Closes #2481156
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8456
diff changeset
   117
    eschema = entity.e_schema
8696
0bb18407c053 [toward py3k] rewrite dict.keys() and dict.values() (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8695
diff changeset
   118
    for attr in entity.cw_edited:
8547
f23ac525ddd1 [datafeed] properly call hooks for inlined relations on entity creation. Closes #2481156
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8456
diff changeset
   119
        rschema = eschema.subjrels[attr]
11348
70337ad23145 pep8 + docstrings and comments improvments
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11204
diff changeset
   120
        if not rschema.final:  # inlined relation
8547
f23ac525ddd1 [datafeed] properly call hooks for inlined relations on entity creation. Closes #2481156
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8456
diff changeset
   121
            value = entity.cw_edited[attr]
f23ac525ddd1 [datafeed] properly call hooks for inlined relations on entity creation. Closes #2481156
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8456
diff changeset
   122
            relations.append((attr, value))
10368
1691be295517 [repository] rename session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 10365
diff changeset
   123
            cnx.update_rel_cache_add(entity.eid, attr, value)
1691be295517 [repository] rename session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 10365
diff changeset
   124
            rdef = cnx.rtype_eids_rdef(attr, entity.eid, value)
8547
f23ac525ddd1 [datafeed] properly call hooks for inlined relations on entity creation. Closes #2481156
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8456
diff changeset
   125
            if rdef.cardinality[1] in '1?' and activeintegrity:
10368
1691be295517 [repository] rename session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 10365
diff changeset
   126
                with cnx.security_enabled(read=False):
1691be295517 [repository] rename session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 10365
diff changeset
   127
                    cnx.execute('DELETE X %s Y WHERE Y eid %%(y)s' % attr,
11348
70337ad23145 pep8 + docstrings and comments improvments
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11204
diff changeset
   128
                                {'x': entity.eid, 'y': value})
8547
f23ac525ddd1 [datafeed] properly call hooks for inlined relations on entity creation. Closes #2481156
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8456
diff changeset
   129
    return relations
f23ac525ddd1 [datafeed] properly call hooks for inlined relations on entity creation. Closes #2481156
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8456
diff changeset
   130
8320
cd2d332b3063 [repo looping task] move looping task logic in a dedicated object (progress #2204047)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8280
diff changeset
   131
8211
543e1579ba0d [repo] Add a publish/subscribe mechanism for inter-instance communication using zmq
Julien Cristau <julien.cristau@logilab.fr>, Quentin Roquefort <quentin@kpsule.me>
parents: 8190
diff changeset
   132
class NullEventBus(object):
8214
ce9556358dbd [event bus] event has been renamed to publish
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8211
diff changeset
   133
    def publish(self, msg):
8211
543e1579ba0d [repo] Add a publish/subscribe mechanism for inter-instance communication using zmq
Julien Cristau <julien.cristau@logilab.fr>, Quentin Roquefort <quentin@kpsule.me>
parents: 8190
diff changeset
   134
        pass
543e1579ba0d [repo] Add a publish/subscribe mechanism for inter-instance communication using zmq
Julien Cristau <julien.cristau@logilab.fr>, Quentin Roquefort <quentin@kpsule.me>
parents: 8190
diff changeset
   135
543e1579ba0d [repo] Add a publish/subscribe mechanism for inter-instance communication using zmq
Julien Cristau <julien.cristau@logilab.fr>, Quentin Roquefort <quentin@kpsule.me>
parents: 8190
diff changeset
   136
    def add_subscription(self, topic, callback):
543e1579ba0d [repo] Add a publish/subscribe mechanism for inter-instance communication using zmq
Julien Cristau <julien.cristau@logilab.fr>, Quentin Roquefort <quentin@kpsule.me>
parents: 8190
diff changeset
   137
        pass
543e1579ba0d [repo] Add a publish/subscribe mechanism for inter-instance communication using zmq
Julien Cristau <julien.cristau@logilab.fr>, Quentin Roquefort <quentin@kpsule.me>
parents: 8190
diff changeset
   138
543e1579ba0d [repo] Add a publish/subscribe mechanism for inter-instance communication using zmq
Julien Cristau <julien.cristau@logilab.fr>, Quentin Roquefort <quentin@kpsule.me>
parents: 8190
diff changeset
   139
    def start(self):
543e1579ba0d [repo] Add a publish/subscribe mechanism for inter-instance communication using zmq
Julien Cristau <julien.cristau@logilab.fr>, Quentin Roquefort <quentin@kpsule.me>
parents: 8190
diff changeset
   140
        pass
543e1579ba0d [repo] Add a publish/subscribe mechanism for inter-instance communication using zmq
Julien Cristau <julien.cristau@logilab.fr>, Quentin Roquefort <quentin@kpsule.me>
parents: 8190
diff changeset
   141
543e1579ba0d [repo] Add a publish/subscribe mechanism for inter-instance communication using zmq
Julien Cristau <julien.cristau@logilab.fr>, Quentin Roquefort <quentin@kpsule.me>
parents: 8190
diff changeset
   142
    def stop(self):
543e1579ba0d [repo] Add a publish/subscribe mechanism for inter-instance communication using zmq
Julien Cristau <julien.cristau@logilab.fr>, Quentin Roquefort <quentin@kpsule.me>
parents: 8190
diff changeset
   143
        pass
543e1579ba0d [repo] Add a publish/subscribe mechanism for inter-instance communication using zmq
Julien Cristau <julien.cristau@logilab.fr>, Quentin Roquefort <quentin@kpsule.me>
parents: 8190
diff changeset
   144
11934
ec69125c03a8 [server] Add a blank line before _CnxSetPool to fix PEP8 error
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11931
diff changeset
   145
12804
6de3af773073 [refactoring] not needed anymore in python3
Laurent Peuch <cortex@worlddomination.be>
parents: 12803
diff changeset
   146
class _CnxSetPool:
11931
ad7796dabeaa [repository] move cnxset pool handling to a helper class
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11908
diff changeset
   147
ad7796dabeaa [repository] move cnxset pool handling to a helper class
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11908
diff changeset
   148
    def __init__(self, source, size):
ad7796dabeaa [repository] move cnxset pool handling to a helper class
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11908
diff changeset
   149
        self._cnxsets = []
12805
8cf02773a6b3 [refactoring] blank lines to make core more readable
Laurent Peuch <cortex@worlddomination.be>
parents: 12804
diff changeset
   150
11950
68766861debe [repository] possibility to disable connections pooling
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11934
diff changeset
   151
        if size is not None:
68766861debe [repository] possibility to disable connections pooling
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11934
diff changeset
   152
            self._queue = queue.Queue()
12805
8cf02773a6b3 [refactoring] blank lines to make core more readable
Laurent Peuch <cortex@worlddomination.be>
parents: 12804
diff changeset
   153
11950
68766861debe [repository] possibility to disable connections pooling
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11934
diff changeset
   154
            for i in range(size):
68766861debe [repository] possibility to disable connections pooling
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11934
diff changeset
   155
                cnxset = source.wrapped_connection()
68766861debe [repository] possibility to disable connections pooling
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11934
diff changeset
   156
                self._cnxsets.append(cnxset)
68766861debe [repository] possibility to disable connections pooling
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11934
diff changeset
   157
                self._queue.put_nowait(cnxset)
12805
8cf02773a6b3 [refactoring] blank lines to make core more readable
Laurent Peuch <cortex@worlddomination.be>
parents: 12804
diff changeset
   158
11950
68766861debe [repository] possibility to disable connections pooling
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11934
diff changeset
   159
        else:
68766861debe [repository] possibility to disable connections pooling
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11934
diff changeset
   160
            self._queue = None
68766861debe [repository] possibility to disable connections pooling
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11934
diff changeset
   161
            self._source = source
11931
ad7796dabeaa [repository] move cnxset pool handling to a helper class
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11908
diff changeset
   162
ad7796dabeaa [repository] move cnxset pool handling to a helper class
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11908
diff changeset
   163
    def qsize(self):
12803
bf940d5416e5 [refactoring] inline useless variable
Laurent Peuch <cortex@worlddomination.be>
parents: 12749
diff changeset
   164
        if self._queue is None:
11950
68766861debe [repository] possibility to disable connections pooling
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11934
diff changeset
   165
            return None
12805
8cf02773a6b3 [refactoring] blank lines to make core more readable
Laurent Peuch <cortex@worlddomination.be>
parents: 12804
diff changeset
   166
12803
bf940d5416e5 [refactoring] inline useless variable
Laurent Peuch <cortex@worlddomination.be>
parents: 12749
diff changeset
   167
        return self._queue.qsize()
11931
ad7796dabeaa [repository] move cnxset pool handling to a helper class
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11908
diff changeset
   168
ad7796dabeaa [repository] move cnxset pool handling to a helper class
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11908
diff changeset
   169
    def get(self):
12803
bf940d5416e5 [refactoring] inline useless variable
Laurent Peuch <cortex@worlddomination.be>
parents: 12749
diff changeset
   170
        if self._queue is None:
11950
68766861debe [repository] possibility to disable connections pooling
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11934
diff changeset
   171
            return self._source.wrapped_connection()
12805
8cf02773a6b3 [refactoring] blank lines to make core more readable
Laurent Peuch <cortex@worlddomination.be>
parents: 12804
diff changeset
   172
11931
ad7796dabeaa [repository] move cnxset pool handling to a helper class
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11908
diff changeset
   173
        try:
ad7796dabeaa [repository] move cnxset pool handling to a helper class
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11908
diff changeset
   174
            return self._queue.get(True, timeout=5)
ad7796dabeaa [repository] move cnxset pool handling to a helper class
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11908
diff changeset
   175
        except queue.Empty:
ad7796dabeaa [repository] move cnxset pool handling to a helper class
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11908
diff changeset
   176
            raise Exception('no connections set available after 5 secs, probably either a '
ad7796dabeaa [repository] move cnxset pool handling to a helper class
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11908
diff changeset
   177
                            'bug in code (too many uncommited/rolled back '
ad7796dabeaa [repository] move cnxset pool handling to a helper class
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11908
diff changeset
   178
                            'connections) or too much load on the server (in '
ad7796dabeaa [repository] move cnxset pool handling to a helper class
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11908
diff changeset
   179
                            'which case you can try to set a bigger '
ad7796dabeaa [repository] move cnxset pool handling to a helper class
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11908
diff changeset
   180
                            'connections pool size)')
ad7796dabeaa [repository] move cnxset pool handling to a helper class
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11908
diff changeset
   181
ad7796dabeaa [repository] move cnxset pool handling to a helper class
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11908
diff changeset
   182
    def release(self, cnxset):
12803
bf940d5416e5 [refactoring] inline useless variable
Laurent Peuch <cortex@worlddomination.be>
parents: 12749
diff changeset
   183
        if self._queue is None:
11950
68766861debe [repository] possibility to disable connections pooling
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11934
diff changeset
   184
            cnxset.close(True)
68766861debe [repository] possibility to disable connections pooling
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11934
diff changeset
   185
        else:
68766861debe [repository] possibility to disable connections pooling
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11934
diff changeset
   186
            self._queue.put_nowait(cnxset)
11931
ad7796dabeaa [repository] move cnxset pool handling to a helper class
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11908
diff changeset
   187
ad7796dabeaa [repository] move cnxset pool handling to a helper class
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11908
diff changeset
   188
    def __iter__(self):
ad7796dabeaa [repository] move cnxset pool handling to a helper class
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11908
diff changeset
   189
        for cnxset in self._cnxsets:
ad7796dabeaa [repository] move cnxset pool handling to a helper class
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11908
diff changeset
   190
            yield cnxset
ad7796dabeaa [repository] move cnxset pool handling to a helper class
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11908
diff changeset
   191
ad7796dabeaa [repository] move cnxset pool handling to a helper class
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11908
diff changeset
   192
    def close(self):
12803
bf940d5416e5 [refactoring] inline useless variable
Laurent Peuch <cortex@worlddomination.be>
parents: 12749
diff changeset
   193
        if self._queue is not None:
bf940d5416e5 [refactoring] inline useless variable
Laurent Peuch <cortex@worlddomination.be>
parents: 12749
diff changeset
   194
            while not self._queue.empty():
bf940d5416e5 [refactoring] inline useless variable
Laurent Peuch <cortex@worlddomination.be>
parents: 12749
diff changeset
   195
                cnxset = self._queue.get_nowait()
12805
8cf02773a6b3 [refactoring] blank lines to make core more readable
Laurent Peuch <cortex@worlddomination.be>
parents: 12804
diff changeset
   196
11950
68766861debe [repository] possibility to disable connections pooling
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11934
diff changeset
   197
                try:
68766861debe [repository] possibility to disable connections pooling
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11934
diff changeset
   198
                    cnxset.close(True)
68766861debe [repository] possibility to disable connections pooling
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11934
diff changeset
   199
                except Exception:
68766861debe [repository] possibility to disable connections pooling
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11934
diff changeset
   200
                    self.exception('error while closing %s' % cnxset)
11931
ad7796dabeaa [repository] move cnxset pool handling to a helper class
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11908
diff changeset
   201
8211
543e1579ba0d [repo] Add a publish/subscribe mechanism for inter-instance communication using zmq
Julien Cristau <julien.cristau@logilab.fr>, Quentin Roquefort <quentin@kpsule.me>
parents: 8190
diff changeset
   202
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   203
class Repository(object):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   204
    """a repository provides access to a set of persistent storages for
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   205
    entities and relations
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   206
    """
1482
93c613913912 delete-trailing-whitespaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   207
12012
f7ff5217a02f [server] replace repository's tasks manager by a scheduler
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11995
diff changeset
   208
    def __init__(self, config, scheduler=None, vreg=None):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   209
        self.config = config
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   210
        if vreg is None:
8190
2a3c1b787688 [vreg] move base registry implementation to logilab.common. Closes #1916014
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8133
diff changeset
   211
            vreg = cwvreg.CWRegistryStore(config)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   212
        self.vreg = vreg
12012
f7ff5217a02f [server] replace repository's tasks manager by a scheduler
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11995
diff changeset
   213
        self._scheduler = scheduler
8320
cd2d332b3063 [repo looping task] move looping task logic in a dedicated object (progress #2204047)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8280
diff changeset
   214
8211
543e1579ba0d [repo] Add a publish/subscribe mechanism for inter-instance communication using zmq
Julien Cristau <julien.cristau@logilab.fr>, Quentin Roquefort <quentin@kpsule.me>
parents: 8190
diff changeset
   215
        self.app_instances_bus = NullEventBus()
8320
cd2d332b3063 [repo looping task] move looping task logic in a dedicated object (progress #2204047)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8280
diff changeset
   216
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   217
        # list of functions to be called at regular interval
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   218
        # list of running threads
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   219
        self._running_threads = []
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   220
        # initial schema, should be build or replaced latter
2839
6419af16faa0 imports cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2835
diff changeset
   221
        self.schema = schema.CubicWebSchema(config.appid)
11348
70337ad23145 pep8 + docstrings and comments improvments
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11204
diff changeset
   222
        self.vreg.schema = self.schema  # until actual schema is loaded...
5906
d40ced753291 [repository] fix so that when repository is shutting down, internal session in transaction are interrupted
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5813
diff changeset
   223
        # shutdown flag
11983
5de78b6fff2e [server] Separate repository bootstrap from initialization
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11981
diff changeset
   224
        self.shutting_down = None
6427
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6426
diff changeset
   225
        # sources (additional sources info in the system database)
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6426
diff changeset
   226
        self.system_source = self.get_source('native', 'system',
9460
a2a0bc984863 [config] cleanup/refactor server sources file values handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9458
diff changeset
   227
                                             config.system_source_config.copy())
6427
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6426
diff changeset
   228
        # querier helper, need to be created after sources initialization
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6426
diff changeset
   229
        self.querier = querier.QuerierHelper(self, self.schema)
11774
51c160677afe [repository] Drop the entities.extid column and associated cache
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11767
diff changeset
   230
        # cache eid -> type
51c160677afe [repository] Drop the entities.extid column and associated cache
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11767
diff changeset
   231
        self._type_cache = {}
8946
fae2f561b0f5 [repo] minor cleanups to bootstrap sequence
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8943
diff changeset
   232
        # the hooks manager
fae2f561b0f5 [repo] minor cleanups to bootstrap sequence
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8943
diff changeset
   233
        self.hm = hook.HooksManager(self.vreg)
11348
70337ad23145 pep8 + docstrings and comments improvments
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11204
diff changeset
   234
11981
56e99b59272f [server] Rename Repository's init_cnxset_pool method bootstrap
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11980
diff changeset
   235
    def bootstrap(self):
11977
6eb79532af12 [server] Move "starting repository..." message where this actually happens
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11975
diff changeset
   236
        self.info('starting repository from %s', self.config.apphome)
11983
5de78b6fff2e [server] Separate repository bootstrap from initialization
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11981
diff changeset
   237
        self.shutting_down = False
2959
daabb9bc5233 make db-restore command work even with no/corrupted database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2929
diff changeset
   238
        config = self.config
11908
7904fe436e82 [repository] Fix connection-pool-size not set to 1 with quick_start enabled
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11821
diff changeset
   239
        # copy pool size here since config.init_cube() and config.load_schema()
7904fe436e82 [repository] Fix connection-pool-size not set to 1 with quick_start enabled
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11821
diff changeset
   240
        # reload configuration from file and could reset a manually set pool
7904fe436e82 [repository] Fix connection-pool-size not set to 1 with quick_start enabled
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11821
diff changeset
   241
        # size.
11950
68766861debe [repository] possibility to disable connections pooling
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11934
diff changeset
   242
        if config['connections-pooler-enabled']:
68766861debe [repository] possibility to disable connections pooling
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11934
diff changeset
   243
            pool_size, min_pool_size = config['connections-pool-size'], 1
68766861debe [repository] possibility to disable connections pooling
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11934
diff changeset
   244
        else:
68766861debe [repository] possibility to disable connections pooling
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11934
diff changeset
   245
            pool_size = min_pool_size = None
68766861debe [repository] possibility to disable connections pooling
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11934
diff changeset
   246
        # 0. init a cnxset that will be used to fetch bootstrap information from
8947
3bbd416b09ec [repo] straightforward bootstrap sequence. Closes #2841188
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8946
diff changeset
   247
        #    the database
11950
68766861debe [repository] possibility to disable connections pooling
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11934
diff changeset
   248
        self.cnxsets = _CnxSetPool(self.system_source, min_pool_size)
8947
3bbd416b09ec [repo] straightforward bootstrap sequence. Closes #2841188
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8946
diff changeset
   249
        # 1. set used cubes
3bbd416b09ec [repo] straightforward bootstrap sequence. Closes #2841188
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8946
diff changeset
   250
        if config.creating or not config.read_instance_schema:
3bbd416b09ec [repo] straightforward bootstrap sequence. Closes #2841188
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8946
diff changeset
   251
            config.bootstrap_cubes()
3bbd416b09ec [repo] straightforward bootstrap sequence. Closes #2841188
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8946
diff changeset
   252
        else:
3bbd416b09ec [repo] straightforward bootstrap sequence. Closes #2841188
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8946
diff changeset
   253
            self.set_schema(self.config.load_bootstrap_schema(), resetvreg=False)
3bbd416b09ec [repo] straightforward bootstrap sequence. Closes #2841188
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8946
diff changeset
   254
            config.init_cubes(self.get_cubes())
3bbd416b09ec [repo] straightforward bootstrap sequence. Closes #2841188
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8946
diff changeset
   255
        # 2. load schema
5043
fe52dd3936cf [repo config] cleanup read_instance_schema / bootstrap_schema / creating mess
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5032
diff changeset
   256
        if config.quick_start:
8946
fae2f561b0f5 [repo] minor cleanups to bootstrap sequence
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8943
diff changeset
   257
            # quick start: only to get a minimal repository to get cubes
5084
d8f491cb046c [repo] we want a hooks manager and hooks initialized event on quick start (we trigger events such as server_backup / server_restore for instance)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5079
diff changeset
   258
            # information (eg dump/restore/...)
8947
3bbd416b09ec [repo] straightforward bootstrap sequence. Closes #2841188
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8946
diff changeset
   259
            #
3bbd416b09ec [repo] straightforward bootstrap sequence. Closes #2841188
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8946
diff changeset
   260
            # restrict appobject_path to only load hooks and entity classes in
3bbd416b09ec [repo] straightforward bootstrap sequence. Closes #2841188
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8946
diff changeset
   261
            # the registry
8537
e30d0a7f0087 [config] turn internal configuration methods building appobjects search path into normal method rather than class method
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8456
diff changeset
   262
            config.cube_appobject_path = set(('hooks', 'entities'))
e30d0a7f0087 [config] turn internal configuration methods building appobjects search path into normal method rather than class method
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8456
diff changeset
   263
            config.cubicweb_appobject_path = set(('hooks', 'entities'))
11950
68766861debe [repository] possibility to disable connections pooling
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11934
diff changeset
   264
            # limit connections pool size
68766861debe [repository] possibility to disable connections pooling
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11934
diff changeset
   265
            pool_size = min_pool_size
8947
3bbd416b09ec [repo] straightforward bootstrap sequence. Closes #2841188
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8946
diff changeset
   266
        if config.quick_start or config.creating or not config.read_instance_schema:
3bbd416b09ec [repo] straightforward bootstrap sequence. Closes #2841188
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8946
diff changeset
   267
            # load schema from the file system
8580
d753d6a6798f [repository] move modification of appobject_path to repository initialization code so we can restore it later to avoid side effect on the config. Fix regression introduced in d32ab8570e5d
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8573
diff changeset
   268
            if not config.creating:
11975
248039d0e6b1 [server] Fix a typo in log message in Repository.init_cnxset_pool()
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11962
diff changeset
   269
                self.info("set fs instance's schema")
9165
7ebf3b1cdb4c Use the list of cubes from the filesystem when reading the schema from the filesystem
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 9132
diff changeset
   270
            self.set_schema(config.load_schema(expand_cubes=True))
11090
b4b854c25de5 [repository] set .eid on eschema when schema is loaded from the filesystem
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11008
diff changeset
   271
            if not config.creating:
b4b854c25de5 [repository] set .eid on eschema when schema is loaded from the filesystem
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11008
diff changeset
   272
                # set eids on entities schema
b4b854c25de5 [repository] set .eid on eschema when schema is loaded from the filesystem
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11008
diff changeset
   273
                with self.internal_cnx() as cnx:
b4b854c25de5 [repository] set .eid on eschema when schema is loaded from the filesystem
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11008
diff changeset
   274
                    for etype, eid in cnx.execute('Any XN,X WHERE X is CWEType, X name XN'):
11477
3b4d41566de3 [repo] Don't crash on start when fs schema is missing some db schema entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11348
diff changeset
   275
                        try:
3b4d41566de3 [repo] Don't crash on start when fs schema is missing some db schema entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11348
diff changeset
   276
                            self.schema.eschema(etype).eid = eid
3b4d41566de3 [repo] Don't crash on start when fs schema is missing some db schema entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11348
diff changeset
   277
                        except KeyError:
3b4d41566de3 [repo] Don't crash on start when fs schema is missing some db schema entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11348
diff changeset
   278
                            # etype in the database doesn't exist in the fs schema, this may occur
3b4d41566de3 [repo] Don't crash on start when fs schema is missing some db schema entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11348
diff changeset
   279
                            # during dev and we shouldn't crash
3b4d41566de3 [repo] Don't crash on start when fs schema is missing some db schema entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11348
diff changeset
   280
                            self.warning('No %s entity type in the file system schema', etype)
8592
df16bd045cae [server] fix repository initialisation
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8580
diff changeset
   281
        else:
df16bd045cae [server] fix repository initialisation
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8580
diff changeset
   282
            # normal start: load the instance schema from the database
8947
3bbd416b09ec [repo] straightforward bootstrap sequence. Closes #2841188
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8946
diff changeset
   283
            self.info('loading schema from the repository')
3bbd416b09ec [repo] straightforward bootstrap sequence. Closes #2841188
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8946
diff changeset
   284
            self.set_schema(self.deserialize_schema())
3bbd416b09ec [repo] straightforward bootstrap sequence. Closes #2841188
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8946
diff changeset
   285
        # 3. initialize data sources
3bbd416b09ec [repo] straightforward bootstrap sequence. Closes #2841188
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8946
diff changeset
   286
        if config.creating:
6128
fbb8398f80dc cleanups
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6126
diff changeset
   287
            # call init_creating so that for instance native source can
fbb8398f80dc cleanups
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6126
diff changeset
   288
            # configurate tsearch according to postgres version
9456
a79e88aad555 [multi-sources-removal] Kill repo.sources
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9455
diff changeset
   289
            self.system_source.init_creating()
8947
3bbd416b09ec [repo] straightforward bootstrap sequence. Closes #2841188
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8946
diff changeset
   290
        else:
12135
e814ba7e51dc [server] Move "system" source initialization into its own method
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12130
diff changeset
   291
            self._init_system_source()
8947
3bbd416b09ec [repo] straightforward bootstrap sequence. Closes #2841188
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8946
diff changeset
   292
            if 'CWProperty' in self.schema:
3bbd416b09ec [repo] straightforward bootstrap sequence. Closes #2841188
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8946
diff changeset
   293
                self.vreg.init_properties(self.properties())
3bbd416b09ec [repo] straightforward bootstrap sequence. Closes #2841188
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8946
diff changeset
   294
        # 4. close initialization connection set and reopen fresh ones for
3bbd416b09ec [repo] straightforward bootstrap sequence. Closes #2841188
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8946
diff changeset
   295
        #    proper initialization
11931
ad7796dabeaa [repository] move cnxset pool handling to a helper class
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11908
diff changeset
   296
        self.cnxsets.close()
ad7796dabeaa [repository] move cnxset pool handling to a helper class
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11908
diff changeset
   297
        self.cnxsets = _CnxSetPool(self.system_source, pool_size)
11988
ffde65347797 [server] Call 'server_startup' hooks at the end of repository "bootstrap"
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11985
diff changeset
   298
        # 5. call instance level initialisation hooks
ffde65347797 [server] Call 'server_startup' hooks at the end of repository "bootstrap"
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11985
diff changeset
   299
        self.hm.call_hooks('server_startup', repo=self)
1482
93c613913912 delete-trailing-whitespaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   300
12146
d540defa0591 [server] Add source_by_eid and source_by_uri methods to repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12142
diff changeset
   301
    def source_by_uri(self, uri):
d540defa0591 [server] Add source_by_eid and source_by_uri methods to repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12142
diff changeset
   302
        with self.internal_cnx() as cnx:
d540defa0591 [server] Add source_by_eid and source_by_uri methods to repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12142
diff changeset
   303
            rset = cnx.find('CWSource', name=uri)
d540defa0591 [server] Add source_by_eid and source_by_uri methods to repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12142
diff changeset
   304
            if not rset:
d540defa0591 [server] Add source_by_eid and source_by_uri methods to repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12142
diff changeset
   305
                raise ValueError('no source with uri %s found' % uri)
d540defa0591 [server] Add source_by_eid and source_by_uri methods to repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12142
diff changeset
   306
            return self._source_from_cwsource(rset.one())
d540defa0591 [server] Add source_by_eid and source_by_uri methods to repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12142
diff changeset
   307
d540defa0591 [server] Add source_by_eid and source_by_uri methods to repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12142
diff changeset
   308
    def source_by_eid(self, eid):
d540defa0591 [server] Add source_by_eid and source_by_uri methods to repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12142
diff changeset
   309
        with self.internal_cnx() as cnx:
d540defa0591 [server] Add source_by_eid and source_by_uri methods to repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12142
diff changeset
   310
            rset = cnx.find('CWSource', eid=eid)
d540defa0591 [server] Add source_by_eid and source_by_uri methods to repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12142
diff changeset
   311
            if not rset:
d540defa0591 [server] Add source_by_eid and source_by_uri methods to repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12142
diff changeset
   312
                raise ValueError('no source with eid %d found' % eid)
d540defa0591 [server] Add source_by_eid and source_by_uri methods to repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12142
diff changeset
   313
            return self._source_from_cwsource(rset.one())
d540defa0591 [server] Add source_by_eid and source_by_uri methods to repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12142
diff changeset
   314
12142
db2fc87348ab [server] Make "sources_by_uri" and "sources_by_eid" properties of repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12135
diff changeset
   315
    @property
db2fc87348ab [server] Make "sources_by_uri" and "sources_by_eid" properties of repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12135
diff changeset
   316
    def sources_by_uri(self):
db2fc87348ab [server] Make "sources_by_uri" and "sources_by_eid" properties of repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12135
diff changeset
   317
        mapping = {'system': self.system_source}
db2fc87348ab [server] Make "sources_by_uri" and "sources_by_eid" properties of repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12135
diff changeset
   318
        mapping.update((sourceent.name, source)
db2fc87348ab [server] Make "sources_by_uri" and "sources_by_eid" properties of repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12135
diff changeset
   319
                       for sourceent, source in self._sources())
db2fc87348ab [server] Make "sources_by_uri" and "sources_by_eid" properties of repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12135
diff changeset
   320
        return mapping
db2fc87348ab [server] Make "sources_by_uri" and "sources_by_eid" properties of repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12135
diff changeset
   321
db2fc87348ab [server] Make "sources_by_uri" and "sources_by_eid" properties of repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12135
diff changeset
   322
    def _sources(self):
db2fc87348ab [server] Make "sources_by_uri" and "sources_by_eid" properties of repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12135
diff changeset
   323
        if self.config.quick_start:
db2fc87348ab [server] Make "sources_by_uri" and "sources_by_eid" properties of repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12135
diff changeset
   324
            return
db2fc87348ab [server] Make "sources_by_uri" and "sources_by_eid" properties of repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12135
diff changeset
   325
        with self.internal_cnx() as cnx:
db2fc87348ab [server] Make "sources_by_uri" and "sources_by_eid" properties of repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12135
diff changeset
   326
            for sourceent in cnx.execute(
db2fc87348ab [server] Make "sources_by_uri" and "sources_by_eid" properties of repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12135
diff changeset
   327
                    'Any S, SN, SA, SC WHERE S is_instance_of CWSource, '
db2fc87348ab [server] Make "sources_by_uri" and "sources_by_eid" properties of repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12135
diff changeset
   328
                    'S name SN, S type SA, S config SC, S name != "system"').entities():
12146
d540defa0591 [server] Add source_by_eid and source_by_uri methods to repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12142
diff changeset
   329
                source = self._source_from_cwsource(sourceent)
12142
db2fc87348ab [server] Make "sources_by_uri" and "sources_by_eid" properties of repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12135
diff changeset
   330
                yield sourceent, source
db2fc87348ab [server] Make "sources_by_uri" and "sources_by_eid" properties of repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12135
diff changeset
   331
        self._clear_source_defs_caches()
db2fc87348ab [server] Make "sources_by_uri" and "sources_by_eid" properties of repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12135
diff changeset
   332
12146
d540defa0591 [server] Add source_by_eid and source_by_uri methods to repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12142
diff changeset
   333
    def _source_from_cwsource(self, sourceent):
d540defa0591 [server] Add source_by_eid and source_by_uri methods to repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12142
diff changeset
   334
        source = self.get_source(sourceent.type, sourceent.name,
d540defa0591 [server] Add source_by_eid and source_by_uri methods to repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12142
diff changeset
   335
                                 sourceent.host_config, sourceent.eid)
d540defa0591 [server] Add source_by_eid and source_by_uri methods to repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12142
diff changeset
   336
        if self.config.source_enabled(source):
d540defa0591 [server] Add source_by_eid and source_by_uri methods to repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12142
diff changeset
   337
            # call source's init method to complete their initialisation if
d540defa0591 [server] Add source_by_eid and source_by_uri methods to repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12142
diff changeset
   338
            # needed (for instance looking for persistent configuration using an
d540defa0591 [server] Add source_by_eid and source_by_uri methods to repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12142
diff changeset
   339
            # internal session, which is not possible until connections sets have been
d540defa0591 [server] Add source_by_eid and source_by_uri methods to repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12142
diff changeset
   340
            # initialized)
12148
79160d54662e [sources] Simplify source's init method
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12147
diff changeset
   341
            source.init(sourceent)
12146
d540defa0591 [server] Add source_by_eid and source_by_uri methods to repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12142
diff changeset
   342
        return source
d540defa0591 [server] Add source_by_eid and source_by_uri methods to repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12142
diff changeset
   343
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   344
    # internals ###############################################################
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   345
12135
e814ba7e51dc [server] Move "system" source initialization into its own method
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12130
diff changeset
   346
    def _init_system_source(self):
e814ba7e51dc [server] Move "system" source initialization into its own method
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12130
diff changeset
   347
        if self.config.quick_start:
e814ba7e51dc [server] Move "system" source initialization into its own method
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12130
diff changeset
   348
            self.system_source.init_creating()
e814ba7e51dc [server] Move "system" source initialization into its own method
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12130
diff changeset
   349
            return
e814ba7e51dc [server] Move "system" source initialization into its own method
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12130
diff changeset
   350
        with self.internal_cnx() as cnx:
e814ba7e51dc [server] Move "system" source initialization into its own method
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12130
diff changeset
   351
            sourceent = cnx.execute(
e814ba7e51dc [server] Move "system" source initialization into its own method
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12130
diff changeset
   352
                'Any S, SA, SC WHERE S is_instance_of CWSource,'
e814ba7e51dc [server] Move "system" source initialization into its own method
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12130
diff changeset
   353
                ' S name "system", S type SA, S config SC'
e814ba7e51dc [server] Move "system" source initialization into its own method
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12130
diff changeset
   354
            ).one()
e814ba7e51dc [server] Move "system" source initialization into its own method
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12130
diff changeset
   355
            self.system_source.eid = sourceent.eid
12148
79160d54662e [sources] Simplify source's init method
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12147
diff changeset
   356
            self.system_source.init(sourceent)
12135
e814ba7e51dc [server] Move "system" source initialization into its own method
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12130
diff changeset
   357
6945
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6941
diff changeset
   358
    def get_source(self, type, uri, source_config, eid=None):
6427
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6426
diff changeset
   359
        # set uri and type in source config so it's available through
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6426
diff changeset
   360
        # source_defs()
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   361
        source_config['uri'] = uri
6427
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6426
diff changeset
   362
        source_config['type'] = type
6945
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6941
diff changeset
   363
        return sources.get_source(type, source_config, self, eid)
1482
93c613913912 delete-trailing-whitespaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   364
8943
58b3b2d9c965 [repo] kill rebuildinfered feature from Repository.set_schema
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8916
diff changeset
   365
    def set_schema(self, schema, resetvreg=True):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   366
        self.info('set schema %s %#x', schema.name, id(schema))
3240
8604a15995d1 refactor so that rql rewriter may be used outside the server. Enhance it to be usable for RRQLExpression as well
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3042
diff changeset
   367
        if resetvreg:
5447
f29dafba250a [vreg repo] shouldn't call init_registration in on hook registry bootstrap, already done before
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5442
diff changeset
   368
            # trigger full reload of all appobjects
3240
8604a15995d1 refactor so that rql rewriter may be used outside the server. Enhance it to be usable for RRQLExpression as well
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3042
diff changeset
   369
            self.vreg.set_schema(schema)
8604a15995d1 refactor so that rql rewriter may be used outside the server. Enhance it to be usable for RRQLExpression as well
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3042
diff changeset
   370
        else:
8604a15995d1 refactor so that rql rewriter may be used outside the server. Enhance it to be usable for RRQLExpression as well
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3042
diff changeset
   371
            self.vreg._set_schema(schema)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   372
        self.querier.set_schema(schema)
12142
db2fc87348ab [server] Make "sources_by_uri" and "sources_by_eid" properties of repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12135
diff changeset
   373
        self.system_source.set_schema(schema)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   374
        self.schema = schema
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   375
8947
3bbd416b09ec [repo] straightforward bootstrap sequence. Closes #2841188
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8946
diff changeset
   376
    def deserialize_schema(self):
3bbd416b09ec [repo] straightforward bootstrap sequence. Closes #2841188
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8946
diff changeset
   377
        """load schema from the database"""
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   378
        from cubicweb.server.schemaserial import deserialize_schema
2839
6419af16faa0 imports cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2835
diff changeset
   379
        appschema = schema.CubicWebSchema(self.config.appid)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   380
        self.debug('deserializing db schema into %s %#x', appschema.name, id(appschema))
9490
b3d2c4065e6a [server] use internal_cnx instead of internal_session in deserialize_schema
Julien Cristau <julien.cristau@logilab.fr>
parents: 9488
diff changeset
   381
        with self.internal_cnx() as cnx:
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   382
            try:
9490
b3d2c4065e6a [server] use internal_cnx instead of internal_session in deserialize_schema
Julien Cristau <julien.cristau@logilab.fr>
parents: 9488
diff changeset
   383
                deserialize_schema(appschema, cnx)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   384
            except BadSchemaDefinition:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   385
                raise
8695
358d8bed9626 [toward-py3k] rewrite to "except AnException as exc:" (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8694
diff changeset
   386
            except Exception as ex:
1398
5fe84a5f7035 rename internal entity types to have CW prefix instead of E
sylvain.thenault@logilab.fr
parents: 1372
diff changeset
   387
                import traceback
5fe84a5f7035 rename internal entity types to have CW prefix instead of E
sylvain.thenault@logilab.fr
parents: 1372
diff changeset
   388
                traceback.print_exc()
10110
f601d2fdeff7 [repository] don't mangle the stack trace on exception
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 10039
diff changeset
   389
                raise Exception('Is the database initialised ? (cause: %s)' % ex)
8947
3bbd416b09ec [repo] straightforward bootstrap sequence. Closes #2841188
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8946
diff changeset
   390
        return appschema
8321
b5d5a5630649 [repository] split repo initialization from starting looping task (closes #2204047)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8320
diff changeset
   391
12112
0a54f49314f6 [server] Introduce an `has_scheduler` method on Repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12111
diff changeset
   392
    def has_scheduler(self):
0a54f49314f6 [server] Introduce an `has_scheduler` method on Repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12111
diff changeset
   393
        """Return True if the repository has a scheduler attached and is able
0a54f49314f6 [server] Introduce an `has_scheduler` method on Repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12111
diff changeset
   394
        to register looping tasks.
0a54f49314f6 [server] Introduce an `has_scheduler` method on Repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12111
diff changeset
   395
        """
0a54f49314f6 [server] Introduce an `has_scheduler` method on Repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12111
diff changeset
   396
        return self._scheduler is not None
0a54f49314f6 [server] Introduce an `has_scheduler` method on Repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12111
diff changeset
   397
12012
f7ff5217a02f [server] replace repository's tasks manager by a scheduler
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11995
diff changeset
   398
    def run_scheduler(self):
f7ff5217a02f [server] replace repository's tasks manager by a scheduler
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11995
diff changeset
   399
        """Start repository scheduler after preparing the repository for that.
8321
b5d5a5630649 [repository] split repo initialization from starting looping task (closes #2204047)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8320
diff changeset
   400
b5d5a5630649 [repository] split repo initialization from starting looping task (closes #2204047)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8320
diff changeset
   401
        * trigger server startup hook,
12012
f7ff5217a02f [server] replace repository's tasks manager by a scheduler
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11995
diff changeset
   402
        * start the scheduler *and block*.
8321
b5d5a5630649 [repository] split repo initialization from starting looping task (closes #2204047)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8320
diff changeset
   403
b5d5a5630649 [repository] split repo initialization from starting looping task (closes #2204047)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8320
diff changeset
   404
        XXX Other startup related stuffs are done elsewhere. In Repository
b5d5a5630649 [repository] split repo initialization from starting looping task (closes #2204047)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8320
diff changeset
   405
        XXX __init__ or in external codes (various server managers).
b5d5a5630649 [repository] split repo initialization from starting looping task (closes #2204047)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8320
diff changeset
   406
        """
12112
0a54f49314f6 [server] Introduce an `has_scheduler` method on Repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12111
diff changeset
   407
        assert self.has_scheduler(), \
11348
70337ad23145 pep8 + docstrings and comments improvments
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11204
diff changeset
   408
            "This Repository is not intended to be used as a server"
12012
f7ff5217a02f [server] replace repository's tasks manager by a scheduler
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11995
diff changeset
   409
        self.info(
f7ff5217a02f [server] replace repository's tasks manager by a scheduler
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11995
diff changeset
   410
            'starting repository scheduler with tasks: %s',
f7ff5217a02f [server] replace repository's tasks manager by a scheduler
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11995
diff changeset
   411
            ', '.join(e.action.__name__ for e in self._scheduler.queue))
f7ff5217a02f [server] replace repository's tasks manager by a scheduler
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11995
diff changeset
   412
        self._scheduler.run()
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   413
2708
60d728bdcba5 allow to specify arbitrary argument when recording a looping task func
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2667
diff changeset
   414
    def looping_task(self, interval, func, *args):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   415
        """register a function to be called every `interval` seconds.
1482
93c613913912 delete-trailing-whitespaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   416
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   417
        looping tasks can only be registered during repository initialization,
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   418
        once done this method will fail.
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   419
        """
12110
a9266a72f3e0 [server] Exit quickly when a looping task is registered in maintenance mode
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12062
diff changeset
   420
        if self.config.repairing:
a9266a72f3e0 [server] Exit quickly when a looping task is registered in maintenance mode
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12062
diff changeset
   421
            return
12112
0a54f49314f6 [server] Introduce an `has_scheduler` method on Repository
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12111
diff changeset
   422
        if not self.has_scheduler():
12111
cd069068a5ef [server] Warn instead of failing when a looping task is registered and repo has no scheduler
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12110
diff changeset
   423
            self.warning(
cd069068a5ef [server] Warn instead of failing when a looping task is registered and repo has no scheduler
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12110
diff changeset
   424
                'looping task %s will not run in this process where repository '
cd069068a5ef [server] Warn instead of failing when a looping task is registered and repo has no scheduler
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12110
diff changeset
   425
                'has no scheduler; use "cubicweb-ctl scheduler <appid>" to '
cd069068a5ef [server] Warn instead of failing when a looping task is registered and repo has no scheduler
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12110
diff changeset
   426
                'have it running', func)
cd069068a5ef [server] Warn instead of failing when a looping task is registered and repo has no scheduler
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12110
diff changeset
   427
            return
12012
f7ff5217a02f [server] replace repository's tasks manager by a scheduler
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11995
diff changeset
   428
        event = utils.schedule_periodic_task(
f7ff5217a02f [server] replace repository's tasks manager by a scheduler
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11995
diff changeset
   429
            self._scheduler, interval, func, *args)
f7ff5217a02f [server] replace repository's tasks manager by a scheduler
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11995
diff changeset
   430
        self.info('scheduled periodic task %s (interval: %.2fs)',
f7ff5217a02f [server] replace repository's tasks manager by a scheduler
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11995
diff changeset
   431
                  event.action.__name__, interval)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   432
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   433
    def threaded_task(self, func):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   434
        """start function in a separated thread"""
8320
cd2d332b3063 [repo looping task] move looping task logic in a dedicated object (progress #2204047)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8280
diff changeset
   435
        utils.RepoThread(func, self._running_threads).start()
1482
93c613913912 delete-trailing-whitespaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   436
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   437
    def shutdown(self):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   438
        """called on server stop event to properly close opened sessions and
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   439
        connections
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   440
        """
5906
d40ced753291 [repository] fix so that when repository is shutting down, internal session in transaction are interrupted
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5813
diff changeset
   441
        assert not self.shutting_down, 'already shutting down'
8393
77c7158916c1 [events] add a new before_shutdown event (closes #2345728)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 8366
diff changeset
   442
        if not (self.config.creating or self.config.repairing
77c7158916c1 [events] add a new before_shutdown event (closes #2345728)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 8366
diff changeset
   443
                or self.config.quick_start):
77c7158916c1 [events] add a new before_shutdown event (closes #2345728)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 8366
diff changeset
   444
            # then, the system source is still available
77c7158916c1 [events] add a new before_shutdown event (closes #2345728)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 8366
diff changeset
   445
            self.hm.call_hooks('before_server_shutdown', repo=self)
5906
d40ced753291 [repository] fix so that when repository is shutting down, internal session in transaction are interrupted
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5813
diff changeset
   446
        self.shutting_down = True
12016
88ed82a25f8a [server] Add a "scheduler" command to run repository scheduler
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12012
diff changeset
   447
        self.info('shutting down repository')
5749
b4393b681f7a [repo] on repository shutdown, we've to close the new eid creation connection
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5747
diff changeset
   448
        self.system_source.shutdown()
8351
02f4f01375e8 [repository] fire 'server_shutdown' hooks before waiting for theads
David Douard <david.douard@logilab.fr>
parents: 8322
diff changeset
   449
        if not (self.config.creating or self.config.repairing
02f4f01375e8 [repository] fire 'server_shutdown' hooks before waiting for theads
David Douard <david.douard@logilab.fr>
parents: 8322
diff changeset
   450
                or self.config.quick_start):
02f4f01375e8 [repository] fire 'server_shutdown' hooks before waiting for theads
David Douard <david.douard@logilab.fr>
parents: 8322
diff changeset
   451
            self.hm.call_hooks('server_shutdown', repo=self)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   452
        for thread in self._running_threads:
5376
2c3f14bc2590 [python2.6] don't add a name property on Thread
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5292
diff changeset
   453
            self.info('waiting thread %s...', thread.getName())
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   454
            thread.join()
5376
2c3f14bc2590 [python2.6] don't add a name property on Thread
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5292
diff changeset
   455
            self.info('thread %s finished', thread.getName())
11931
ad7796dabeaa [repository] move cnxset pool handling to a helper class
Philippe Pepiot <philippe.pepiot@logilab.fr>
parents: 11908
diff changeset
   456
        self.cnxsets.close()
12060
0cdf5fafd234 [repo] Extract rql cache handling to a dedicated class
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12057
diff changeset
   457
        hits, misses = self.querier.rql_cache.cache_hit, self.querier.rql_cache.cache_miss
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   458
        try:
5747
d6ac0cd30fde [rset] do not filter rsets with __getstate__, ensure whatever flies with pyro has no .req attribute, also set the ._rqlst to None since it will be reconstructed later on demand
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 5711
diff changeset
   459
            self.info('rql st cache hit/miss: %s/%s (%s%% hits)', hits, misses,
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   460
                      (hits * 100) / (hits + misses))
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   461
            hits, misses = self.system_source.cache_hit, self.system_source.cache_miss
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   462
            self.info('sql cache hit/miss: %s/%s (%s%% hits)', hits, misses,
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   463
                      (hits * 100) / (hits + misses))
11348
70337ad23145 pep8 + docstrings and comments improvments
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11204
diff changeset
   464
            nocache = self.system_source.no_cache
70337ad23145 pep8 + docstrings and comments improvments
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11204
diff changeset
   465
            self.info('sql cache usage: %s/%s (%s%%)', hits + misses, nocache,
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   466
                      ((hits + misses) * 100) / (hits + misses + nocache))
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   467
        except ZeroDivisionError:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   468
            pass
1482
93c613913912 delete-trailing-whitespaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   469
9512
88dc96fc9fc1 [server] use a connection instead of a session for user authentication
Julien Cristau <julien.cristau@logilab.fr>
parents: 9510
diff changeset
   470
    def check_auth_info(self, cnx, login, authinfo):
6849
5a0c2cfc19bf [repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6822
diff changeset
   471
        """validate authentication, raise AuthenticationError on failure, return
5a0c2cfc19bf [repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6822
diff changeset
   472
        associated CWUser's eid on success.
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   473
        """
8229
b7bc631816f7 [ldapfeed] make authentication actually working
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8214
diff changeset
   474
        # iter on sources_by_uri then check enabled source since sources doesn't
b7bc631816f7 [ldapfeed] make authentication actually working
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8214
diff changeset
   475
        # contain copy based sources
10663
54b8a1f249fb [py3k] dict.itervalues → dict.values
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10662
diff changeset
   476
        for source in self.sources_by_uri.values():
11750
18e7b9829471 [sources] Drop source support_entity / support_relation source API
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11699
diff changeset
   477
            if self.config.source_enabled(source):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   478
                try:
10365
21461f80f348 [connection] remove ensure_cnx_set context manager uses
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 10354
diff changeset
   479
                    return source.authenticate(cnx, login, **authinfo)
11750
18e7b9829471 [sources] Drop source support_entity / support_relation source API
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11699
diff changeset
   480
                except (NotImplementedError, AuthenticationError):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   481
                    continue
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   482
        else:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   483
            raise AuthenticationError('authentication failed with all sources')
6849
5a0c2cfc19bf [repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6822
diff changeset
   484
9512
88dc96fc9fc1 [server] use a connection instead of a session for user authentication
Julien Cristau <julien.cristau@logilab.fr>
parents: 9510
diff changeset
   485
    def authenticate_user(self, cnx, login, **authinfo):
6849
5a0c2cfc19bf [repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6822
diff changeset
   486
        """validate login / password, raise AuthenticationError on failure
5a0c2cfc19bf [repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6822
diff changeset
   487
        return associated CWUser instance on success
5a0c2cfc19bf [repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6822
diff changeset
   488
        """
9512
88dc96fc9fc1 [server] use a connection instead of a session for user authentication
Julien Cristau <julien.cristau@logilab.fr>
parents: 9510
diff changeset
   489
        eid = self.check_auth_info(cnx, login, authinfo)
88dc96fc9fc1 [server] use a connection instead of a session for user authentication
Julien Cristau <julien.cristau@logilab.fr>
parents: 9510
diff changeset
   490
        cwuser = self._build_user(cnx, eid)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   491
        if self.config.consider_user_state and \
11348
70337ad23145 pep8 + docstrings and comments improvments
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11204
diff changeset
   492
           not cwuser.cw_adapt_to('IWorkflowable').state in cwuser.AUTHENTICABLE_STATES:
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   493
            raise AuthenticationError('user is not in authenticable state')
2268
2f336fd5e040 euser->cwuser
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2267
diff changeset
   494
        return cwuser
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   495
9512
88dc96fc9fc1 [server] use a connection instead of a session for user authentication
Julien Cristau <julien.cristau@logilab.fr>
parents: 9510
diff changeset
   496
    def _build_user(self, cnx, eid):
1398
5fe84a5f7035 rename internal entity types to have CW prefix instead of E
sylvain.thenault@logilab.fr
parents: 1372
diff changeset
   497
        """return a CWUser entity for user with the given eid"""
10365
21461f80f348 [connection] remove ensure_cnx_set context manager uses
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 10354
diff changeset
   498
        cls = self.vreg['etypes'].etype_class('CWUser')
21461f80f348 [connection] remove ensure_cnx_set context manager uses
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 10354
diff changeset
   499
        st = cls.fetch_rqlst(cnx.user, ordermethod=None)
21461f80f348 [connection] remove ensure_cnx_set context manager uses
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 10354
diff changeset
   500
        st.add_eid_restriction(st.get_variable('X'), 'x', 'Substitute')
21461f80f348 [connection] remove ensure_cnx_set context manager uses
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 10354
diff changeset
   501
        rset = cnx.execute(st.as_string(), {'x': eid})
21461f80f348 [connection] remove ensure_cnx_set context manager uses
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 10354
diff changeset
   502
        assert len(rset) == 1, rset
11699
b48020a80dc3 Store user groups and properties as session data
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11477
diff changeset
   503
        return rset.get_entity(0, 0)
1482
93c613913912 delete-trailing-whitespaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   504
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   505
    # public (dbapi) interface ################################################
1482
93c613913912 delete-trailing-whitespaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   506
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   507
    def get_schema(self):
6256
da77d3f95079 [repo] docstring cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6211
diff changeset
   508
        """Return the instance schema.
da77d3f95079 [repo] docstring cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6211
diff changeset
   509
da77d3f95079 [repo] docstring cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6211
diff changeset
   510
        This is a public method, not requiring a session id.
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   511
        """
7723
badfd5524ab6 [repo] Stop setting hashmode on schema
Julien Cristau <julien.cristau@logilab.fr>
parents: 7716
diff changeset
   512
        return self.schema
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   513
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   514
    def get_cubes(self):
6256
da77d3f95079 [repo] docstring cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6211
diff changeset
   515
        """Return the list of cubes used by this instance.
da77d3f95079 [repo] docstring cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6211
diff changeset
   516
da77d3f95079 [repo] docstring cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6211
diff changeset
   517
        This is a public method, not requiring a session id.
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   518
        """
2473
490f88fb99b6 new distinguish repairing/creating from regular start.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2268
diff changeset
   519
        versions = self.get_versions(not (self.config.creating
4689
4eb1f4490538 [test] skipping versions checking during test is enough, no need for monkey patch
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4687
diff changeset
   520
                                          or self.config.repairing
5043
fe52dd3936cf [repo config] cleanup read_instance_schema / bootstrap_schema / creating mess
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5032
diff changeset
   521
                                          or self.config.quick_start
4689
4eb1f4490538 [test] skipping versions checking during test is enough, no need for monkey patch
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4687
diff changeset
   522
                                          or self.config.mode == 'test'))
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   523
        cubes = list(versions)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   524
        cubes.remove('cubicweb')
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   525
        return cubes
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   526
12508
a8c1ea390400 Drop most of deprecated code
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12240
diff changeset
   527
    def get_option_value(self, option):
9458
e2dfdd313dfe [multi-sources-removal] Drop foreid argument of repo.get_option_value
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9456
diff changeset
   528
        """Return the value for `option` in the configuration.
6257
7eb5f1aed728 [repo] new method on repo+dbapi.Connection to get a value from repository's configuration
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6256
diff changeset
   529
7eb5f1aed728 [repo] new method on repo+dbapi.Connection to get a value from repository's configuration
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6256
diff changeset
   530
        This is a public method, not requiring a session id.
7eb5f1aed728 [repo] new method on repo+dbapi.Connection to get a value from repository's configuration
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6256
diff changeset
   531
        """
7eb5f1aed728 [repo] new method on repo+dbapi.Connection to get a value from repository's configuration
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6256
diff changeset
   532
        # XXX we may want to check we don't give sensible information
9458
e2dfdd313dfe [multi-sources-removal] Drop foreid argument of repo.get_option_value
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9456
diff changeset
   533
        return self.config[option]
6257
7eb5f1aed728 [repo] new method on repo+dbapi.Connection to get a value from repository's configuration
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6256
diff changeset
   534
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   535
    @cached
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   536
    def get_versions(self, checkversions=False):
6256
da77d3f95079 [repo] docstring cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6211
diff changeset
   537
        """Return the a dictionary containing cubes used by this instance
da77d3f95079 [repo] docstring cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6211
diff changeset
   538
        as key with their version as value, including cubicweb version.
da77d3f95079 [repo] docstring cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6211
diff changeset
   539
da77d3f95079 [repo] docstring cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6211
diff changeset
   540
        This is a public method, not requiring a session id.
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   541
        """
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   542
        from logilab.common.changelog import Version
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   543
        vcconf = {}
9505
53225f66e7ca [repo] Use a connection instead of a session for repo.get_versions()
Julien Cristau <julien.cristau@logilab.fr>
parents: 9496
diff changeset
   544
        with self.internal_cnx() as cnx:
53225f66e7ca [repo] Use a connection instead of a session for repo.get_versions()
Julien Cristau <julien.cristau@logilab.fr>
parents: 9496
diff changeset
   545
            for pk, version in cnx.execute(
11348
70337ad23145 pep8 + docstrings and comments improvments
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11204
diff changeset
   546
                    'Any K,V WHERE P is CWProperty, P value V, P pkey K, '
70337ad23145 pep8 + docstrings and comments improvments
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11204
diff changeset
   547
                    'P pkey ~="system.version.%"', build_descr=False):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   548
                cube = pk.split('.')[-1]
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   549
                # XXX cubicweb migration
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   550
                if cube in CW_MIGRATION_MAP:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   551
                    cube = CW_MIGRATION_MAP[cube]
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   552
                version = Version(version)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   553
                vcconf[cube] = version
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   554
                if checkversions:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   555
                    if cube != 'cubicweb':
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   556
                        fsversion = self.config.cube_version(cube)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   557
                    else:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   558
                        fsversion = self.config.cubicweb_version()
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   559
                    if version < fsversion:
2476
1294a6bdf3bf application -> instance where it makes sense
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2473
diff changeset
   560
                        msg = ('instance has %s version %s but %s '
12747
36d4a8ec1fb7 [ux] display instance name in suggested upgrade command
Laurent Peuch <cortex@worlddomination.be>
parents: 12240
diff changeset
   561
                               'is installed. Run "cubicweb-ctl upgrade %s".')
36d4a8ec1fb7 [ux] display instance name in suggested upgrade command
Laurent Peuch <cortex@worlddomination.be>
parents: 12240
diff changeset
   562
                        raise ExecutionError(msg % (cube, version, fsversion, self.config.appid))
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   563
        return vcconf
1482
93c613913912 delete-trailing-whitespaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   564
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   565
    @cached
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   566
    def source_defs(self):
6256
da77d3f95079 [repo] docstring cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6211
diff changeset
   567
        """Return the a dictionary containing source uris as value and a
da77d3f95079 [repo] docstring cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6211
diff changeset
   568
        dictionary describing each source as value.
da77d3f95079 [repo] docstring cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6211
diff changeset
   569
da77d3f95079 [repo] docstring cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6211
diff changeset
   570
        This is a public method, not requiring a session id.
da77d3f95079 [repo] docstring cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6211
diff changeset
   571
        """
6427
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6426
diff changeset
   572
        sources = {}
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   573
        # remove sensitive information
10662
10942ed172de [py3k] dict.iteritems → dict.items
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10609
diff changeset
   574
        for uri, source in self.sources_by_uri.items():
6722
3341521d857b [repo source] rename attribute so it's much easier to grasp its role
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6674
diff changeset
   575
            sources[uri] = source.public_config
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   576
        return sources
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   577
12057
90c40f229ab0 [repo] Move and rename repo._clear_planning_cache
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12054
diff changeset
   578
    def _clear_source_defs_caches(self):
90c40f229ab0 [repo] Move and rename repo._clear_planning_cache
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12054
diff changeset
   579
        clear_cache(self, 'source_defs')
90c40f229ab0 [repo] Move and rename repo._clear_planning_cache
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12054
diff changeset
   580
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   581
    def properties(self):
6256
da77d3f95079 [repo] docstring cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6211
diff changeset
   582
        """Return a result set containing system wide properties.
da77d3f95079 [repo] docstring cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6211
diff changeset
   583
da77d3f95079 [repo] docstring cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6211
diff changeset
   584
        This is a public method, not requiring a session id.
da77d3f95079 [repo] docstring cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6211
diff changeset
   585
        """
9491
e82370a30018 [repo] restore python 2.6 compatibility
Julien Cristau <julien.cristau@logilab.fr>
parents: 9490
diff changeset
   586
        with self.internal_cnx() as cnx:
9685
0033aa71e077 [server] make internal_cnx get and keep a connection to the db
Julien Cristau <julien.cristau@logilab.fr>
parents: 9683
diff changeset
   587
            # don't use cnx.execute, we don't want rset.req set
0033aa71e077 [server] make internal_cnx get and keep a connection to the db
Julien Cristau <julien.cristau@logilab.fr>
parents: 9683
diff changeset
   588
            return self.querier.execute(cnx, 'Any K,V WHERE P is CWProperty,'
5747
d6ac0cd30fde [rset] do not filter rsets with __getstate__, ensure whatever flies with pyro has no .req attribute, also set the ._rqlst to None since it will be reconstructed later on demand
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 5711
diff changeset
   589
                                        'P pkey K, P value V, NOT P for_user U',
d6ac0cd30fde [rset] do not filter rsets with __getstate__, ensure whatever flies with pyro has no .req attribute, also set the ._rqlst to None since it will be reconstructed later on demand
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 5711
diff changeset
   590
                                        build_descr=False)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   591
6390
3766853656d7 [repo] add a find_users method for under-privileged accounts
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 6368
diff changeset
   592
    def find_users(self, fetch_attrs, **query_attrs):
3766853656d7 [repo] add a find_users method for under-privileged accounts
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 6368
diff changeset
   593
        """yield user attributes for cwusers matching the given query_attrs
3766853656d7 [repo] add a find_users method for under-privileged accounts
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 6368
diff changeset
   594
        (the result set cannot survive this method call)
3766853656d7 [repo] add a find_users method for under-privileged accounts
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 6368
diff changeset
   595
3766853656d7 [repo] add a find_users method for under-privileged accounts
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 6368
diff changeset
   596
        This can be used by low-privileges account (anonymous comes to
3766853656d7 [repo] add a find_users method for under-privileged accounts
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 6368
diff changeset
   597
        mind).
3766853656d7 [repo] add a find_users method for under-privileged accounts
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 6368
diff changeset
   598
3766853656d7 [repo] add a find_users method for under-privileged accounts
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 6368
diff changeset
   599
        `fetch_attrs`: tuple of attributes to be fetched
3766853656d7 [repo] add a find_users method for under-privileged accounts
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 6368
diff changeset
   600
        `query_attrs`: dict of attr/values to restrict the query
3766853656d7 [repo] add a find_users method for under-privileged accounts
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 6368
diff changeset
   601
        """
3766853656d7 [repo] add a find_users method for under-privileged accounts
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 6368
diff changeset
   602
        assert query_attrs
3766853656d7 [repo] add a find_users method for under-privileged accounts
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 6368
diff changeset
   603
        if not hasattr(self, '_cwuser_attrs'):
3766853656d7 [repo] add a find_users method for under-privileged accounts
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 6368
diff changeset
   604
            cwuser = self.schema['CWUser']
3766853656d7 [repo] add a find_users method for under-privileged accounts
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 6368
diff changeset
   605
            self._cwuser_attrs = set(str(rschema)
3766853656d7 [repo] add a find_users method for under-privileged accounts
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 6368
diff changeset
   606
                                     for rschema, _eschema in cwuser.attribute_definitions()
3766853656d7 [repo] add a find_users method for under-privileged accounts
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 6368
diff changeset
   607
                                     if not rschema.meta)
3766853656d7 [repo] add a find_users method for under-privileged accounts
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 6368
diff changeset
   608
        cwuserattrs = self._cwuser_attrs
8696
0bb18407c053 [toward py3k] rewrite dict.keys() and dict.values() (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8695
diff changeset
   609
        for k in chain(fetch_attrs, query_attrs):
6390
3766853656d7 [repo] add a find_users method for under-privileged accounts
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 6368
diff changeset
   610
            if k not in cwuserattrs:
3766853656d7 [repo] add a find_users method for under-privileged accounts
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 6368
diff changeset
   611
                raise Exception('bad input for find_user')
10345
ef54ea75a642 [server] drop repo.internal_session and InternalSession
Julien Cristau <julien.cristau@logilab.fr>
parents: 10342
diff changeset
   612
        with self.internal_cnx() as cnx:
6390
3766853656d7 [repo] add a find_users method for under-privileged accounts
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 6368
diff changeset
   613
            varmaker = rqlvar_maker()
10669
155c29e0ed1c [py3k] use next builtin instead of next method
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10663
diff changeset
   614
            vars = [(attr, next(varmaker)) for attr in fetch_attrs]
6390
3766853656d7 [repo] add a find_users method for under-privileged accounts
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 6368
diff changeset
   615
            rql = 'Any %s WHERE X is CWUser, ' % ','.join(var[1] for var in vars)
3766853656d7 [repo] add a find_users method for under-privileged accounts
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 6368
diff changeset
   616
            rql += ','.join('X %s %s' % (var[0], var[1]) for var in vars) + ','
10345
ef54ea75a642 [server] drop repo.internal_session and InternalSession
Julien Cristau <julien.cristau@logilab.fr>
parents: 10342
diff changeset
   617
            rset = cnx.execute(rql + ','.join('X %s %%(%s)s' % (attr, attr)
ef54ea75a642 [server] drop repo.internal_session and InternalSession
Julien Cristau <julien.cristau@logilab.fr>
parents: 10342
diff changeset
   618
                                              for attr in query_attrs),
ef54ea75a642 [server] drop repo.internal_session and InternalSession
Julien Cristau <julien.cristau@logilab.fr>
parents: 10342
diff changeset
   619
                               query_attrs)
6390
3766853656d7 [repo] add a find_users method for under-privileged accounts
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 6368
diff changeset
   620
            return rset.rows
3766853656d7 [repo] add a find_users method for under-privileged accounts
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 6368
diff changeset
   621
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   622
    # session handling ########################################################
1482
93c613913912 delete-trailing-whitespaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   623
9113
af6efc15fc90 [repository] add an ``internal_cnx`` method to replace ``internal_session``
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 9105
diff changeset
   624
    @contextmanager
9127
aff75b69db92 [repository] drop safe attribute on ``internal_cnx``
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 9113
diff changeset
   625
    def internal_cnx(self):
9747
10108d9f502a [repo] make it clear in internal_cnx that security is disabled
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9687
diff changeset
   626
        """Context manager returning a Connection using internal user which have
10108d9f502a [repo] make it clear in internal_cnx that security is disabled
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9687
diff changeset
   627
        every access rights on the repository.
9113
af6efc15fc90 [repository] add an ``internal_cnx`` method to replace ``internal_session``
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 9105
diff changeset
   628
12034
e7e469d0539b [cleanup] Drop references to the old internal_session method
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12030
diff changeset
   629
        Internal connections have all hooks beside security enabled.
9113
af6efc15fc90 [repository] add an ``internal_cnx`` method to replace ``internal_session``
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 9105
diff changeset
   630
        """
12043
b8d2e6b9f548 Stop using Session on the repository side
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12036
diff changeset
   631
        with Connection(self, InternalManager()) as cnx:
11348
70337ad23145 pep8 + docstrings and comments improvments
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11204
diff changeset
   632
            cnx.user._cw = cnx  # XXX remove when "vreg = user._cw.vreg" hack in entity.py is gone
11204
71057a8bb19a [session] drop session-as-a-context-manager
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 11199
diff changeset
   633
            with cnx.security_enabled(read=False, write=False):
71057a8bb19a [session] drop session-as-a-context-manager
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 11199
diff changeset
   634
                yield cnx
9113
af6efc15fc90 [repository] add an ``internal_cnx`` method to replace ``internal_session``
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 9105
diff changeset
   635
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   636
    # data sources handling ###################################################
11763
39df042f4ab4 [repository] Drop type_and_source_from_eid and rename related cache
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11756
diff changeset
   637
    # * correspondance between eid and type
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   638
    # * correspondance between eid and local id (i.e. specific to a given source)
1482
93c613913912 delete-trailing-whitespaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   639
12061
94ae25593c38 [repo] Consistent API for cache clearing
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12060
diff changeset
   640
    def clear_caches(self, eids=None):
94ae25593c38 [repo] Consistent API for cache clearing
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12060
diff changeset
   641
        if eids is None:
94ae25593c38 [repo] Consistent API for cache clearing
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12060
diff changeset
   642
            self._type_cache = {}
94ae25593c38 [repo] Consistent API for cache clearing
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12060
diff changeset
   643
            etypes = None
94ae25593c38 [repo] Consistent API for cache clearing
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12060
diff changeset
   644
        else:
94ae25593c38 [repo] Consistent API for cache clearing
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12060
diff changeset
   645
            etypes = []
94ae25593c38 [repo] Consistent API for cache clearing
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12060
diff changeset
   646
            etcache = self._type_cache
94ae25593c38 [repo] Consistent API for cache clearing
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12060
diff changeset
   647
            for eid in eids:
94ae25593c38 [repo] Consistent API for cache clearing
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12060
diff changeset
   648
                try:
94ae25593c38 [repo] Consistent API for cache clearing
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12060
diff changeset
   649
                    etype = etcache.pop(int(eid))  # may be a string in some cases
94ae25593c38 [repo] Consistent API for cache clearing
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12060
diff changeset
   650
                except KeyError:
94ae25593c38 [repo] Consistent API for cache clearing
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12060
diff changeset
   651
                    etype = None
94ae25593c38 [repo] Consistent API for cache clearing
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12060
diff changeset
   652
                etypes.append(etype)
12062
601d65193619 [repo] Introduce a clear_caches method on the Querier class
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12061
diff changeset
   653
        self.querier.clear_caches(eids, etypes)
12061
94ae25593c38 [repo] Consistent API for cache clearing
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12060
diff changeset
   654
        self.system_source.clear_caches(eids, etypes)
1482
93c613913912 delete-trailing-whitespaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   655
11774
51c160677afe [repository] Drop the entities.extid column and associated cache
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11767
diff changeset
   656
    def type_from_eid(self, eid, cnx):
51c160677afe [repository] Drop the entities.extid column and associated cache
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11767
diff changeset
   657
        """Return the type of the entity with id `eid`"""
11763
39df042f4ab4 [repository] Drop type_and_source_from_eid and rename related cache
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11756
diff changeset
   658
        try:
39df042f4ab4 [repository] Drop type_and_source_from_eid and rename related cache
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11756
diff changeset
   659
            eid = int(eid)
39df042f4ab4 [repository] Drop type_and_source_from_eid and rename related cache
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11756
diff changeset
   660
        except ValueError:
39df042f4ab4 [repository] Drop type_and_source_from_eid and rename related cache
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11756
diff changeset
   661
            raise UnknownEid(eid)
39df042f4ab4 [repository] Drop type_and_source_from_eid and rename related cache
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11756
diff changeset
   662
        try:
11774
51c160677afe [repository] Drop the entities.extid column and associated cache
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11767
diff changeset
   663
            return self._type_cache[eid]
11763
39df042f4ab4 [repository] Drop type_and_source_from_eid and rename related cache
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11756
diff changeset
   664
        except KeyError:
11774
51c160677afe [repository] Drop the entities.extid column and associated cache
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11767
diff changeset
   665
            etype = self.system_source.eid_type(cnx, eid)
51c160677afe [repository] Drop the entities.extid column and associated cache
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11767
diff changeset
   666
            self._type_cache[eid] = etype
51c160677afe [repository] Drop the entities.extid column and associated cache
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11767
diff changeset
   667
            return etype
1482
93c613913912 delete-trailing-whitespaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   668
11774
51c160677afe [repository] Drop the entities.extid column and associated cache
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11767
diff changeset
   669
    def add_info(self, cnx, entity, source):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   670
        """add type and source info for an eid into the system table,
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   671
        and index the entity with the full text index
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   672
        """
11774
51c160677afe [repository] Drop the entities.extid column and associated cache
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11767
diff changeset
   673
        # begin by inserting eid/type/source into the entities table
10315
0f3489d622ba [repository] replace session with cnx in a few places
Julien Cristau <julien.cristau@logilab.fr>
parents: 10301
diff changeset
   674
        hook.CleanupNewEidsCacheOp.get_instance(cnx).add_data(entity.eid)
11774
51c160677afe [repository] Drop the entities.extid column and associated cache
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11767
diff changeset
   675
        self.system_source.add_info(cnx, entity, source)
1482
93c613913912 delete-trailing-whitespaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   676
10315
0f3489d622ba [repository] replace session with cnx in a few places
Julien Cristau <julien.cristau@logilab.fr>
parents: 10301
diff changeset
   677
    def _delete_cascade_multi(self, cnx, entities):
10203
1443fe643a38 [server] change order of entities table modification vs entity creation/deletion
Julien Cristau <julien.cristau@logilab.fr>
parents: 10110
diff changeset
   678
        """same as _delete_cascade but accepts a list of entities with
1443fe643a38 [server] change order of entities table modification vs entity creation/deletion
Julien Cristau <julien.cristau@logilab.fr>
parents: 10110
diff changeset
   679
        the same etype and belonging to the same source.
6889
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6801
diff changeset
   680
        """
10315
0f3489d622ba [repository] replace session with cnx in a few places
Julien Cristau <julien.cristau@logilab.fr>
parents: 10301
diff changeset
   681
        pendingrtypes = cnx.transaction_data.get('pendingrtypes', ())
6889
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6801
diff changeset
   682
        # delete remaining relations: if user can delete the entity, he can
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6801
diff changeset
   683
        # delete all its relations without security checking
10315
0f3489d622ba [repository] replace session with cnx in a few places
Julien Cristau <julien.cristau@logilab.fr>
parents: 10301
diff changeset
   684
        with cnx.security_enabled(read=False, write=False):
7895
0a967180794b cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7887
diff changeset
   685
            in_eids = ','.join([str(_e.eid) for _e in entities])
10351
91e63306e277 [connection] replace .running_dbapi_query with .hooks_in_progress
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 10346
diff changeset
   686
            with cnx.running_hooks_ops():
91e63306e277 [connection] replace .running_dbapi_query with .hooks_in_progress
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 10346
diff changeset
   687
                for rschema, _, role in entities[0].e_schema.relation_definitions():
91e63306e277 [connection] replace .running_dbapi_query with .hooks_in_progress
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 10346
diff changeset
   688
                    if rschema.rule:
11348
70337ad23145 pep8 + docstrings and comments improvments
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11204
diff changeset
   689
                        continue  # computed relation
10351
91e63306e277 [connection] replace .running_dbapi_query with .hooks_in_progress
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 10346
diff changeset
   690
                    rtype = rschema.type
91e63306e277 [connection] replace .running_dbapi_query with .hooks_in_progress
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 10346
diff changeset
   691
                    if rtype in schema.VIRTUAL_RTYPES or rtype in pendingrtypes:
91e63306e277 [connection] replace .running_dbapi_query with .hooks_in_progress
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 10346
diff changeset
   692
                        continue
91e63306e277 [connection] replace .running_dbapi_query with .hooks_in_progress
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 10346
diff changeset
   693
                    if role == 'subject':
91e63306e277 [connection] replace .running_dbapi_query with .hooks_in_progress
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 10346
diff changeset
   694
                        # don't skip inlined relation so they are regularly
91e63306e277 [connection] replace .running_dbapi_query with .hooks_in_progress
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 10346
diff changeset
   695
                        # deleted and so hooks are correctly called
91e63306e277 [connection] replace .running_dbapi_query with .hooks_in_progress
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 10346
diff changeset
   696
                        rql = 'DELETE X %s Y WHERE X eid IN (%s)' % (rtype, in_eids)
91e63306e277 [connection] replace .running_dbapi_query with .hooks_in_progress
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 10346
diff changeset
   697
                    else:
91e63306e277 [connection] replace .running_dbapi_query with .hooks_in_progress
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 10346
diff changeset
   698
                        rql = 'DELETE Y %s X WHERE X eid IN (%s)' % (rtype, in_eids)
11962
36851c8b6763 [server] Remove pseudo-handling of exceptions in Repository._delete_cascade_multi()
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11950
diff changeset
   699
                    cnx.execute(rql, build_descr=False)
6889
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6801
diff changeset
   700
9609
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   701
    def init_entity_caches(self, cnx, entity, source):
11774
51c160677afe [repository] Drop the entities.extid column and associated cache
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11767
diff changeset
   702
        """Add entity to connection entities cache and repo's cache."""
9609
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   703
        cnx.set_entity_cache(entity)
11774
51c160677afe [repository] Drop the entities.extid column and associated cache
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11767
diff changeset
   704
        self._type_cache[entity.eid] = entity.cw_etype
5068
10c3422d7419 [repo] on add entity, set cache as soon as possible + fill type/source cache
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5067
diff changeset
   705
9609
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   706
    def glob_add_entity(self, cnx, edited):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   707
        """add an entity to the repository
1482
93c613913912 delete-trailing-whitespaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   708
10039
1d1afe3b5081 [server] typo fix
Julien Cristau <julien.cristau@logilab.fr>
parents: 10018
diff changeset
   709
        the entity eid should originally be None and a unique eid is assigned to
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   710
        the entity instance
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   711
        """
6142
8bc6eac1fac1 [session] cleanup hook / operation / entity edition api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6139
diff changeset
   712
        entity = edited.entity
11348
70337ad23145 pep8 + docstrings and comments improvments
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11204
diff changeset
   713
        entity._cw_is_saved = False  # entity has an eid but is not yet saved
6142
8bc6eac1fac1 [session] cleanup hook / operation / entity edition api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6139
diff changeset
   714
        # init edited_attributes before calling before_add_entity hooks
8bc6eac1fac1 [session] cleanup hook / operation / entity edition api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6139
diff changeset
   715
        entity.cw_edited = edited
9454
a173f9cf9f26 [multi-sources-removal] drop source location search on glob_*[entity|relation] methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9452
diff changeset
   716
        source = self.system_source
5072
072ae171aeb0 [cleanup] style fixes, add nodes, 0.2 cents refactorings
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5068
diff changeset
   717
        # allocate an eid to the entity before calling hooks
9609
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   718
        entity.eid = self.system_source.create_eid(cnx)
5068
10c3422d7419 [repo] on add entity, set cache as soon as possible + fill type/source cache
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5067
diff changeset
   719
        # set caches asap
11774
51c160677afe [repository] Drop the entities.extid column and associated cache
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11767
diff changeset
   720
        self.init_entity_caches(cnx, entity, source)
2600
6cd6c5d11b45 [F repo debugging] log repo event on DBG_REPO debug level
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2596
diff changeset
   721
        if server.DEBUG & server.DBG_REPO:
10589
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 10579
diff changeset
   722
            print('ADD entity', self, entity.cw_etype, entity.eid, edited)
8456
c912d82f2166 [repository] drop unused argument
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8433
diff changeset
   723
        prefill_entity_caches(entity)
9609
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   724
        self.hm.call_hooks('before_add_entity', cnx, entity=entity)
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   725
        relations = preprocess_inlined_relations(cnx, entity)
6142
8bc6eac1fac1 [session] cleanup hook / operation / entity edition api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6139
diff changeset
   726
        edited.set_defaults()
9609
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   727
        if cnx.is_hook_category_activated('integrity'):
6142
8bc6eac1fac1 [session] cleanup hook / operation / entity edition api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6139
diff changeset
   728
            edited.check(creation=True)
11774
51c160677afe [repository] Drop the entities.extid column and associated cache
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11767
diff changeset
   729
        self.add_info(cnx, entity, source)
6211
e9d125fd1465 nicer error reporting for unique together constraints
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6189
diff changeset
   730
        try:
9609
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   731
            source.add_entity(cnx, entity)
10446
1e6655cff5ab add IUserFriendlyError adapter for violation of check constraints
Julien Cristau <julien.cristau@logilab.fr>
parents: 10368
diff changeset
   732
        except (UniqueTogetherError, ViolatedConstraint) as exc:
9609
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   733
            userhdlr = cnx.vreg['adapters'].select(
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   734
                'IUserFriendlyError', cnx, entity=entity, exc=exc)
6465
6401a9d0b5aa [architecture] introduce adapter to ease transformation of errors before display to the end user. Use it for UniqueTogetherError first
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6464
diff changeset
   735
            userhdlr.raise_user_exception()
6368
f907cc7f2875 [repo] properly mark object as saved once added to its source (code much probably gone during a merge...)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6333
diff changeset
   736
        edited.saved = entity._cw_is_saved = True
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   737
        # trigger after_add_entity after after_add_relation
9609
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   738
        self.hm.call_hooks('after_add_entity', cnx, entity=entity)
9454
a173f9cf9f26 [multi-sources-removal] drop source location search on glob_*[entity|relation] methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9452
diff changeset
   739
        # call hooks for inlined relations
a173f9cf9f26 [multi-sources-removal] drop source location search on glob_*[entity|relation] methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9452
diff changeset
   740
        for attr, value in relations:
9609
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   741
            self.hm.call_hooks('before_add_relation', cnx,
11348
70337ad23145 pep8 + docstrings and comments improvments
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11204
diff changeset
   742
                               eidfrom=entity.eid, rtype=attr, eidto=value)
9609
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   743
            self.hm.call_hooks('after_add_relation', cnx,
11348
70337ad23145 pep8 + docstrings and comments improvments
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11204
diff changeset
   744
                               eidfrom=entity.eid, rtype=attr, eidto=value)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   745
        return entity.eid
1482
93c613913912 delete-trailing-whitespaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   746
9609
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   747
    def glob_update_entity(self, cnx, edited):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   748
        """replace an entity in the repository
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   749
        the type and the eid of an entity must not be changed
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   750
        """
6142
8bc6eac1fac1 [session] cleanup hook / operation / entity edition api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6139
diff changeset
   751
        entity = edited.entity
2600
6cd6c5d11b45 [F repo debugging] log repo event on DBG_REPO debug level
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2596
diff changeset
   752
        if server.DEBUG & server.DBG_REPO:
10589
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 10579
diff changeset
   753
            print('UPDATE entity', entity.cw_etype, entity.eid,
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 10579
diff changeset
   754
                  entity.cw_attr_cache, edited)
5115
2e43ef618d14 [repository] forbid usage of set_attributes() in before_add_entity
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5093
diff changeset
   755
        hm = self.hm
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   756
        eschema = entity.e_schema
9609
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   757
        cnx.set_entity_cache(entity)
6142
8bc6eac1fac1 [session] cleanup hook / operation / entity edition api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6139
diff changeset
   758
        orig_edited = getattr(entity, 'cw_edited', None)
8bc6eac1fac1 [session] cleanup hook / operation / entity edition api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6139
diff changeset
   759
        entity.cw_edited = edited
9454
a173f9cf9f26 [multi-sources-removal] drop source location search on glob_*[entity|relation] methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9452
diff changeset
   760
        source = self.system_source
5115
2e43ef618d14 [repository] forbid usage of set_attributes() in before_add_entity
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5093
diff changeset
   761
        try:
2e43ef618d14 [repository] forbid usage of set_attributes() in before_add_entity
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5093
diff changeset
   762
            only_inline_rels, need_fti_update = True, False
2e43ef618d14 [repository] forbid usage of set_attributes() in before_add_entity
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5093
diff changeset
   763
            relations = []
6142
8bc6eac1fac1 [session] cleanup hook / operation / entity edition api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6139
diff changeset
   764
            for attr in list(edited):
5115
2e43ef618d14 [repository] forbid usage of set_attributes() in before_add_entity
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5093
diff changeset
   765
                if attr == 'eid':
2e43ef618d14 [repository] forbid usage of set_attributes() in before_add_entity
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5093
diff changeset
   766
                    continue
2e43ef618d14 [repository] forbid usage of set_attributes() in before_add_entity
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5093
diff changeset
   767
                rschema = eschema.subjrels[attr]
2e43ef618d14 [repository] forbid usage of set_attributes() in before_add_entity
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5093
diff changeset
   768
                if rschema.final:
2e43ef618d14 [repository] forbid usage of set_attributes() in before_add_entity
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5093
diff changeset
   769
                    if getattr(eschema.rdef(attr), 'fulltextindexed', False):
2e43ef618d14 [repository] forbid usage of set_attributes() in before_add_entity
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5093
diff changeset
   770
                        need_fti_update = True
2e43ef618d14 [repository] forbid usage of set_attributes() in before_add_entity
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5093
diff changeset
   771
                    only_inline_rels = False
2e43ef618d14 [repository] forbid usage of set_attributes() in before_add_entity
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5093
diff changeset
   772
                else:
2e43ef618d14 [repository] forbid usage of set_attributes() in before_add_entity
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5093
diff changeset
   773
                    # inlined relation
2e43ef618d14 [repository] forbid usage of set_attributes() in before_add_entity
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5093
diff changeset
   774
                    previous_value = entity.related(attr) or None
2e43ef618d14 [repository] forbid usage of set_attributes() in before_add_entity
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5093
diff changeset
   775
                    if previous_value is not None:
11348
70337ad23145 pep8 + docstrings and comments improvments
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11204
diff changeset
   776
                        previous_value = previous_value[0][0]  # got a result set
6142
8bc6eac1fac1 [session] cleanup hook / operation / entity edition api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6139
diff changeset
   777
                        if previous_value == entity.cw_attr_cache[attr]:
5115
2e43ef618d14 [repository] forbid usage of set_attributes() in before_add_entity
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5093
diff changeset
   778
                            previous_value = None
9454
a173f9cf9f26 [multi-sources-removal] drop source location search on glob_*[entity|relation] methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9452
diff changeset
   779
                        else:
9609
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   780
                            hm.call_hooks('before_delete_relation', cnx,
5115
2e43ef618d14 [repository] forbid usage of set_attributes() in before_add_entity
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5093
diff changeset
   781
                                          eidfrom=entity.eid, rtype=attr,
2e43ef618d14 [repository] forbid usage of set_attributes() in before_add_entity
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5093
diff changeset
   782
                                          eidto=previous_value)
6142
8bc6eac1fac1 [session] cleanup hook / operation / entity edition api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6139
diff changeset
   783
                    relations.append((attr, edited[attr], previous_value))
9454
a173f9cf9f26 [multi-sources-removal] drop source location search on glob_*[entity|relation] methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9452
diff changeset
   784
            # call hooks for inlined relations
a173f9cf9f26 [multi-sources-removal] drop source location search on glob_*[entity|relation] methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9452
diff changeset
   785
            for attr, value, _t in relations:
9609
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   786
                hm.call_hooks('before_add_relation', cnx,
9454
a173f9cf9f26 [multi-sources-removal] drop source location search on glob_*[entity|relation] methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9452
diff changeset
   787
                              eidfrom=entity.eid, rtype=attr, eidto=value)
a173f9cf9f26 [multi-sources-removal] drop source location search on glob_*[entity|relation] methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9452
diff changeset
   788
            if not only_inline_rels:
9609
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   789
                hm.call_hooks('before_update_entity', cnx, entity=entity)
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   790
            if cnx.is_hook_category_activated('integrity'):
6142
8bc6eac1fac1 [session] cleanup hook / operation / entity edition api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6139
diff changeset
   791
                edited.check()
6211
e9d125fd1465 nicer error reporting for unique together constraints
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6189
diff changeset
   792
            try:
9609
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   793
                source.update_entity(cnx, entity)
6225
a176e68b7d0d backport stable
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6142 6211
diff changeset
   794
                edited.saved = True
10446
1e6655cff5ab add IUserFriendlyError adapter for violation of check constraints
Julien Cristau <julien.cristau@logilab.fr>
parents: 10368
diff changeset
   795
            except (UniqueTogetherError, ViolatedConstraint) as exc:
9609
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   796
                userhdlr = cnx.vreg['adapters'].select(
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   797
                    'IUserFriendlyError', cnx, entity=entity, exc=exc)
9226
653f1d4a1101 [repository] properly use IUserFriendlyError when UniqueTogetherError is raised during entity update. Closes #3096638
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9184
diff changeset
   798
                userhdlr.raise_user_exception()
9609
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   799
            self.system_source.update_info(cnx, entity, need_fti_update)
9454
a173f9cf9f26 [multi-sources-removal] drop source location search on glob_*[entity|relation] methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9452
diff changeset
   800
            if not only_inline_rels:
9609
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   801
                hm.call_hooks('after_update_entity', cnx, entity=entity)
9454
a173f9cf9f26 [multi-sources-removal] drop source location search on glob_*[entity|relation] methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9452
diff changeset
   802
            for attr, value, prevvalue in relations:
a173f9cf9f26 [multi-sources-removal] drop source location search on glob_*[entity|relation] methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9452
diff changeset
   803
                # if the relation is already cached, update existant cache
a173f9cf9f26 [multi-sources-removal] drop source location search on glob_*[entity|relation] methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9452
diff changeset
   804
                relcache = entity.cw_relation_cached(attr, 'subject')
a173f9cf9f26 [multi-sources-removal] drop source location search on glob_*[entity|relation] methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9452
diff changeset
   805
                if prevvalue is not None:
9609
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   806
                    hm.call_hooks('after_delete_relation', cnx,
9454
a173f9cf9f26 [multi-sources-removal] drop source location search on glob_*[entity|relation] methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9452
diff changeset
   807
                                  eidfrom=entity.eid, rtype=attr, eidto=prevvalue)
5115
2e43ef618d14 [repository] forbid usage of set_attributes() in before_add_entity
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5093
diff changeset
   808
                    if relcache is not None:
9609
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   809
                        cnx.update_rel_cache_del(entity.eid, attr, prevvalue)
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   810
                del_existing_rel_if_needed(cnx, entity.eid, attr, value)
9770
112c884b2d8d merge 3.18.5 into 3.19 branch
Julien Cristau <julien.cristau@logilab.fr>
parents: 9757 9769
diff changeset
   811
                cnx.update_rel_cache_add(entity.eid, attr, value)
9609
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   812
                hm.call_hooks('after_add_relation', cnx,
9454
a173f9cf9f26 [multi-sources-removal] drop source location search on glob_*[entity|relation] methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9452
diff changeset
   813
                              eidfrom=entity.eid, rtype=attr, eidto=value)
5115
2e43ef618d14 [repository] forbid usage of set_attributes() in before_add_entity
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5093
diff changeset
   814
        finally:
6142
8bc6eac1fac1 [session] cleanup hook / operation / entity edition api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6139
diff changeset
   815
            if orig_edited is not None:
8bc6eac1fac1 [session] cleanup hook / operation / entity edition api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6139
diff changeset
   816
                entity.cw_edited = orig_edited
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   817
9609
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   818
    def glob_delete_entities(self, cnx, eids):
6889
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6801
diff changeset
   819
        """delete a list of  entities and all related entities from the repository"""
9609
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   820
        # mark eids as being deleted in cnx info and setup cache update
7501
2983dd24494a [repository] refactor/cleanup entity deletion methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7373
diff changeset
   821
        # operation (register pending eids before actual deletion to avoid
2983dd24494a [repository] refactor/cleanup entity deletion methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7373
diff changeset
   822
        # multiple call to glob_delete_entities)
9609
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   823
        op = hook.CleanupDeletedEidsCacheOp.get_instance(cnx)
7501
2983dd24494a [repository] refactor/cleanup entity deletion methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7373
diff changeset
   824
        eids = eids - op._container
2983dd24494a [repository] refactor/cleanup entity deletion methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7373
diff changeset
   825
        op._container |= eids
11348
70337ad23145 pep8 + docstrings and comments improvments
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11204
diff changeset
   826
        data_by_etype = {}  # values are [list of entities]
6889
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6801
diff changeset
   827
        #
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6801
diff changeset
   828
        # WARNING: the way this dictionary is populated is heavily optimized
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6801
diff changeset
   829
        # and does not use setdefault on purpose. Unless a new release
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6801
diff changeset
   830
        # of the Python interpreter advertises large perf improvements
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6801
diff changeset
   831
        # in setdefault, this should not be changed without profiling.
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6801
diff changeset
   832
        for eid in eids:
9609
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   833
            etype = self.type_from_eid(eid, cnx)
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   834
            entity = cnx.entity_from_eid(eid, etype)
7501
2983dd24494a [repository] refactor/cleanup entity deletion methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7373
diff changeset
   835
            try:
9454
a173f9cf9f26 [multi-sources-removal] drop source location search on glob_*[entity|relation] methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9452
diff changeset
   836
                data_by_etype[etype].append(entity)
7501
2983dd24494a [repository] refactor/cleanup entity deletion methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7373
diff changeset
   837
            except KeyError:
9454
a173f9cf9f26 [multi-sources-removal] drop source location search on glob_*[entity|relation] methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9452
diff changeset
   838
                data_by_etype[etype] = [entity]
a173f9cf9f26 [multi-sources-removal] drop source location search on glob_*[entity|relation] methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9452
diff changeset
   839
        source = self.system_source
10662
10942ed172de [py3k] dict.iteritems → dict.items
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10609
diff changeset
   840
        for etype, entities in data_by_etype.items():
6889
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6801
diff changeset
   841
            if server.DEBUG & server.DBG_REPO:
11348
70337ad23145 pep8 + docstrings and comments improvments
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11204
diff changeset
   842
                print('DELETE entities', etype, [e.eid for e in entities])
9609
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   843
            self.hm.call_hooks('before_delete_entity', cnx, entities=entities)
10203
1443fe643a38 [server] change order of entities table modification vs entity creation/deletion
Julien Cristau <julien.cristau@logilab.fr>
parents: 10110
diff changeset
   844
            self._delete_cascade_multi(cnx, entities)
9609
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   845
            source.delete_entities(cnx, entities)
10203
1443fe643a38 [server] change order of entities table modification vs entity creation/deletion
Julien Cristau <julien.cristau@logilab.fr>
parents: 10110
diff changeset
   846
            source.delete_info_multi(cnx, entities)
9609
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   847
            self.hm.call_hooks('after_delete_entity', cnx, entities=entities)
7501
2983dd24494a [repository] refactor/cleanup entity deletion methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7373
diff changeset
   848
        # don't clear cache here, it is done in a hook on commit
1482
93c613913912 delete-trailing-whitespaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   849
9609
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   850
    def glob_add_relation(self, cnx, subject, rtype, object):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   851
        """add a relation to the repository"""
9609
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   852
        self.glob_add_relations(cnx, {rtype: [(subject, object)]})
7237
9f619715665b [server] improve the speed of setting relations between entities (closes #1625257)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7134
diff changeset
   853
9609
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   854
    def glob_add_relations(self, cnx, relations):
7237
9f619715665b [server] improve the speed of setting relations between entities (closes #1625257)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7134
diff changeset
   855
        """add several relations to the repository
9f619715665b [server] improve the speed of setting relations between entities (closes #1625257)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7134
diff changeset
   856
9f619715665b [server] improve the speed of setting relations between entities (closes #1625257)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7134
diff changeset
   857
        relations is a dictionary rtype: [(subj_eid, obj_eid), ...]
9f619715665b [server] improve the speed of setting relations between entities (closes #1625257)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7134
diff changeset
   858
        """
9454
a173f9cf9f26 [multi-sources-removal] drop source location search on glob_*[entity|relation] methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9452
diff changeset
   859
        source = self.system_source
a173f9cf9f26 [multi-sources-removal] drop source location search on glob_*[entity|relation] methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9452
diff changeset
   860
        relations_by_rtype = {}
7513
8f4422391e5a [repo integrity] test and fix glob add relation where several entities are added at once for a relation of 1? cardinality
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7507
diff changeset
   861
        subjects_by_types = {}
8f4422391e5a [repo integrity] test and fix glob add relation where several entities are added at once for a relation of 1? cardinality
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7507
diff changeset
   862
        objects_by_types = {}
9609
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   863
        activintegrity = cnx.is_hook_category_activated('activeintegrity')
10662
10942ed172de [py3k] dict.iteritems → dict.items
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10609
diff changeset
   864
        for rtype, eids_subj_obj in relations.items():
7237
9f619715665b [server] improve the speed of setting relations between entities (closes #1625257)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7134
diff changeset
   865
            if server.DEBUG & server.DBG_REPO:
8548
eeabc752c32e [repo] fix debug code in DBG_REPO mode (closes #2469942)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 8547
diff changeset
   866
                for subjeid, objeid in eids_subj_obj:
10589
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 10579
diff changeset
   867
                    print('ADD relation', subjeid, rtype, objeid)
7513
8f4422391e5a [repo integrity] test and fix glob add relation where several entities are added at once for a relation of 1? cardinality
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7507
diff changeset
   868
            for subjeid, objeid in eids_subj_obj:
12240
8eb15566142f [server] Use dict.setdefault() instead of if/else in repo.glob_add_relations()
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12148
diff changeset
   869
                relations_by_rtype.setdefault(rtype, []).append((subjeid, objeid))
7513
8f4422391e5a [repo integrity] test and fix glob add relation where several entities are added at once for a relation of 1? cardinality
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7507
diff changeset
   870
                if not activintegrity:
8f4422391e5a [repo integrity] test and fix glob add relation where several entities are added at once for a relation of 1? cardinality
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7507
diff changeset
   871
                    continue
8f4422391e5a [repo integrity] test and fix glob add relation where several entities are added at once for a relation of 1? cardinality
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7507
diff changeset
   872
                # take care to relation of cardinality '?1', as all eids will
8f4422391e5a [repo integrity] test and fix glob add relation where several entities are added at once for a relation of 1? cardinality
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7507
diff changeset
   873
                # be inserted later, we've remove duplicated eids since they
10510
51321946da37 Spelling fixes in comments and docstrings
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10488
diff changeset
   874
                # won't be caught by `del_existing_rel_if_needed`
9609
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   875
                rdef = cnx.rtype_eids_rdef(rtype, subjeid, objeid)
7513
8f4422391e5a [repo integrity] test and fix glob add relation where several entities are added at once for a relation of 1? cardinality
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7507
diff changeset
   876
                card = rdef.cardinality
8f4422391e5a [repo integrity] test and fix glob add relation where several entities are added at once for a relation of 1? cardinality
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7507
diff changeset
   877
                if card[0] in '?1':
9609
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   878
                    with cnx.security_enabled(read=False):
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   879
                        cnx.execute('DELETE X %s Y WHERE X eid %%(x)s, '
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   880
                                    'NOT Y eid %%(y)s' % rtype,
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   881
                                    {'x': subjeid, 'y': objeid})
7513
8f4422391e5a [repo integrity] test and fix glob add relation where several entities are added at once for a relation of 1? cardinality
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7507
diff changeset
   882
                    subjects = subjects_by_types.setdefault(rdef, {})
8f4422391e5a [repo integrity] test and fix glob add relation where several entities are added at once for a relation of 1? cardinality
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7507
diff changeset
   883
                    if subjeid in subjects:
8f4422391e5a [repo integrity] test and fix glob add relation where several entities are added at once for a relation of 1? cardinality
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7507
diff changeset
   884
                        del relations_by_rtype[rtype][subjects[subjeid]]
8f4422391e5a [repo integrity] test and fix glob add relation where several entities are added at once for a relation of 1? cardinality
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7507
diff changeset
   885
                        subjects[subjeid] = len(relations_by_rtype[rtype]) - 1
8f4422391e5a [repo integrity] test and fix glob add relation where several entities are added at once for a relation of 1? cardinality
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7507
diff changeset
   886
                        continue
8f4422391e5a [repo integrity] test and fix glob add relation where several entities are added at once for a relation of 1? cardinality
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7507
diff changeset
   887
                    subjects[subjeid] = len(relations_by_rtype[rtype]) - 1
8f4422391e5a [repo integrity] test and fix glob add relation where several entities are added at once for a relation of 1? cardinality
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7507
diff changeset
   888
                if card[1] in '?1':
9609
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   889
                    with cnx.security_enabled(read=False):
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   890
                        cnx.execute('DELETE X %s Y WHERE Y eid %%(y)s, '
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   891
                                    'NOT X eid %%(x)s' % rtype,
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   892
                                    {'x': subjeid, 'y': objeid})
7513
8f4422391e5a [repo integrity] test and fix glob add relation where several entities are added at once for a relation of 1? cardinality
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7507
diff changeset
   893
                    objects = objects_by_types.setdefault(rdef, {})
8f4422391e5a [repo integrity] test and fix glob add relation where several entities are added at once for a relation of 1? cardinality
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7507
diff changeset
   894
                    if objeid in objects:
8f4422391e5a [repo integrity] test and fix glob add relation where several entities are added at once for a relation of 1? cardinality
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7507
diff changeset
   895
                        del relations_by_rtype[rtype][objects[objeid]]
8f4422391e5a [repo integrity] test and fix glob add relation where several entities are added at once for a relation of 1? cardinality
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7507
diff changeset
   896
                        objects[objeid] = len(relations_by_rtype[rtype])
8f4422391e5a [repo integrity] test and fix glob add relation where several entities are added at once for a relation of 1? cardinality
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7507
diff changeset
   897
                        continue
8f4422391e5a [repo integrity] test and fix glob add relation where several entities are added at once for a relation of 1? cardinality
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7507
diff changeset
   898
                    objects[objeid] = len(relations_by_rtype[rtype])
10662
10942ed172de [py3k] dict.iteritems → dict.items
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10609
diff changeset
   899
        for rtype, source_relations in relations_by_rtype.items():
9609
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   900
            self.hm.call_hooks('before_add_relation', cnx,
9454
a173f9cf9f26 [multi-sources-removal] drop source location search on glob_*[entity|relation] methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9452
diff changeset
   901
                               rtype=rtype, eids_from_to=source_relations)
10662
10942ed172de [py3k] dict.iteritems → dict.items
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10609
diff changeset
   902
        for rtype, source_relations in relations_by_rtype.items():
9609
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   903
            source.add_relations(cnx, rtype, source_relations)
9454
a173f9cf9f26 [multi-sources-removal] drop source location search on glob_*[entity|relation] methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9452
diff changeset
   904
            rschema = self.schema.rschema(rtype)
a173f9cf9f26 [multi-sources-removal] drop source location search on glob_*[entity|relation] methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9452
diff changeset
   905
            for subjeid, objeid in source_relations:
9609
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   906
                cnx.update_rel_cache_add(subjeid, rtype, objeid, rschema.symmetric)
10662
10942ed172de [py3k] dict.iteritems → dict.items
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10609
diff changeset
   907
        for rtype, source_relations in relations_by_rtype.items():
9609
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   908
            self.hm.call_hooks('after_add_relation', cnx,
9454
a173f9cf9f26 [multi-sources-removal] drop source location search on glob_*[entity|relation] methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9452
diff changeset
   909
                               rtype=rtype, eids_from_to=source_relations)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   910
9609
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   911
    def glob_delete_relation(self, cnx, subject, rtype, object):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   912
        """delete a relation from the repository"""
2600
6cd6c5d11b45 [F repo debugging] log repo event on DBG_REPO debug level
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2596
diff changeset
   913
        if server.DEBUG & server.DBG_REPO:
10589
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 10579
diff changeset
   914
            print('DELETE relation', subject, rtype, object)
9454
a173f9cf9f26 [multi-sources-removal] drop source location search on glob_*[entity|relation] methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9452
diff changeset
   915
        source = self.system_source
9609
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   916
        self.hm.call_hooks('before_delete_relation', cnx,
9454
a173f9cf9f26 [multi-sources-removal] drop source location search on glob_*[entity|relation] methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9452
diff changeset
   917
                           eidfrom=subject, rtype=rtype, eidto=object)
9609
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   918
        source.delete_relation(cnx, subject, rtype, object)
2647
b0a2e779845c enable server side entity caching, 25% speedup on codenaf insertion. ALL CW TESTS OK
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2641
diff changeset
   919
        rschema = self.schema.rschema(rtype)
9609
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   920
        cnx.update_rel_cache_del(subject, rtype, object, rschema.symmetric)
e7d38148799e [repository] operations get a connection instead of a session
Julien Cristau <julien.cristau@logilab.fr>
parents: 9607
diff changeset
   921
        self.hm.call_hooks('after_delete_relation', cnx,
9454
a173f9cf9f26 [multi-sources-removal] drop source location search on glob_*[entity|relation] methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9452
diff changeset
   922
                           eidfrom=subject, rtype=rtype, eidto=object)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   923
7083
b8e35cde46e9 help pylint by explicitely defining some attributes
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7064
diff changeset
   924
    # these are overridden by set_log_methods below
b8e35cde46e9 help pylint by explicitely defining some attributes
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7064
diff changeset
   925
    # only defining here to prevent pylint from complaining
9468
39b7a91a3f4c [repo] pylint cleanup, mainly of imports, with a bit of style
Julien Cristau <julien.cristau@logilab.fr>
parents: 9467
diff changeset
   926
    info = warning = error = critical = exception = debug = lambda msg, *a, **kw: None
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   927
11821
7534b32c45e3 Fix (new) flake8 errors
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11819
diff changeset
   928
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   929
set_log_methods(Repository, getLogger('cubicweb.repository'))