server/sources/native.py
author Alexandre Fayolle <alexandre.fayolle@logilab.fr>
Thu, 22 Sep 2011 17:29:40 +0200
branchstable
changeset 7839 daf46963f4fe
parent 7787 d9607ae447d2
child 7879 9aae456abab5
permissions -rw-r--r--
ensure the test for asource column won't crash on SQLServer (closes #1949621)
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
6957
ffda12be2e9f [repository] #1460066: backport datafeed cube as cubicweb source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6945
diff changeset
     1
# copyright 2003-2011 LOGILAB S.A. (Paris, FRANCE), all rights reserved.
5421
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5413
diff changeset
     2
# contact http://www.logilab.fr/ -- mailto:contact@logilab.fr
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5413
diff changeset
     3
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5413
diff changeset
     4
# This file is part of CubicWeb.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5413
diff changeset
     5
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5413
diff changeset
     6
# CubicWeb is free software: you can redistribute it and/or modify it under the
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5413
diff changeset
     7
# terms of the GNU Lesser General Public License as published by the Free
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5413
diff changeset
     8
# Software Foundation, either version 2.1 of the License, or (at your option)
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5413
diff changeset
     9
# any later version.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5413
diff changeset
    10
#
5424
8ecbcbff9777 replace logilab-common by CubicWeb in disclaimer
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5421
diff changeset
    11
# CubicWeb is distributed in the hope that it will be useful, but WITHOUT
5421
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5413
diff changeset
    12
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5413
diff changeset
    13
# FOR A PARTICULAR PURPOSE.  See the GNU Lesser General Public License for more
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5413
diff changeset
    14
# details.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5413
diff changeset
    15
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5413
diff changeset
    16
# You should have received a copy of the GNU Lesser General Public License along
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5413
diff changeset
    17
# with CubicWeb.  If not, see <http://www.gnu.org/licenses/>.
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    18
"""Adapters for native cubicweb sources.
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    19
1952
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1792
diff changeset
    20
Notes:
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1792
diff changeset
    21
* extid (aka external id, the primary key of an entity in the external source
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1792
diff changeset
    22
  from which it comes from) are stored in a varchar column encoded as a base64
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1792
diff changeset
    23
  string. This is because it should actually be Bytes but we want an index on
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1792
diff changeset
    24
  it for fast querying.
5824
de9b7e88660e cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5751
diff changeset
    25
"""
2056
57d287a2132a reminder
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1977
diff changeset
    26
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
    27
from __future__ import with_statement
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
    28
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    29
__docformat__ = "restructuredtext en"
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    30
7342
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
    31
try:
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
    32
    from cPickle import loads, dumps
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
    33
    import cPickle as pickle
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
    34
except ImportError:
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
    35
    from pickle import loads, dumps
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
    36
    import pickle
4900
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4893
diff changeset
    37
from threading import Lock
1016
26387b836099 use datetime instead of mx.DateTime
sylvain.thenault@logilab.fr
parents: 973
diff changeset
    38
from datetime import datetime
1952
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1792
diff changeset
    39
from base64 import b64decode, b64encode
4964
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
    40
from contextlib import contextmanager
7342
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
    41
from os.path import abspath, basename
6211
e9d125fd1465 nicer error reporting for unique together constraints
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6201
diff changeset
    42
import re
6889
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
    43
import itertools
7342
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
    44
import zipfile
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
    45
import logging
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
    46
import sys
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    47
4806
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
    48
from logilab.common.compat import any
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    49
from logilab.common.cache import Cache
4806
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
    50
from logilab.common.decorators import cached, clear_cache
3835
a191b3b9e455 more sensible default values to c-c "create" inputs
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3689
diff changeset
    51
from logilab.common.configuration import Method
a191b3b9e455 more sensible default values to c-c "create" inputs
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3689
diff changeset
    52
from logilab.common.shellutils import getlogin
7342
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
    53
from logilab.database import get_db_helper, sqlgen
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    54
5891
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
    55
from yams import schema2sql as y2sql
6945
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
    56
from yams.schema import role_name
5891
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
    57
6945
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
    58
from cubicweb import (UnknownEid, AuthenticationError, ValidationError, Binary,
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
    59
                      UniqueTogetherError)
5076
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
    60
from cubicweb import transaction as tx, server, neg_role
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
    61
from cubicweb.schema import VIRTUAL_RTYPES
3835
a191b3b9e455 more sensible default values to c-c "create" inputs
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3689
diff changeset
    62
from cubicweb.cwconfig import CubicWebNoAppConfiguration
4806
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
    63
from cubicweb.server import hook
5076
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
    64
from cubicweb.server.utils import crypt_password, eschema_eid
2759
23d7a75693f8 R refactor backup and use tar.gz to store all sources
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 2638
diff changeset
    65
from cubicweb.server.sqlutils import SQL_PREFIX, SQLAdapterMixIn
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    66
from cubicweb.server.rqlannotation import set_qdata
5076
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
    67
from cubicweb.server.hook import CleanupDeletedEidsCacheOp
4943
7f5b83578fec disable security when undoing
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4913
diff changeset
    68
from cubicweb.server.session import hooks_control, security_enabled
7118
e094b3d4eb95 [server] move EditedEntity class to its own module, to avoid cyclic dependency when needed from e.g. session.py
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6957
diff changeset
    69
from cubicweb.server.edition import EditedEntity
2625
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2620
diff changeset
    70
from cubicweb.server.sources import AbstractSource, dbg_st_search, dbg_results
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    71
from cubicweb.server.sources.rql2sql import SQLGenerator
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    72
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    73
2354
9b4bac626977 ability to map attributes to something else than usual cw mapping on sql generation
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2310
diff changeset
    74
ATTR_MAP = {}
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    75
NONSYSTEM_ETYPES = set()
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    76
NONSYSTEM_RELATIONS = set()
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    77
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    78
class LogCursor(object):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    79
    def __init__(self, cursor):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    80
        self.cu = cursor
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
    81
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    82
    def execute(self, query, args=None):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    83
        """Execute a query.
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    84
        it's a function just so that it shows up in profiling
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    85
        """
2593
16d9419a4a79 F: start to handle binary debug log level on the server side
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2589
diff changeset
    86
        if server.DEBUG & server.DBG_SQL:
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    87
            print 'exec', query, args
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    88
        try:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    89
            self.cu.execute(str(query), args)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    90
        except Exception, ex:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    91
            print "sql: %r\n args: %s\ndbms message: %r" % (
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    92
                query, args, ex.args[0])
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    93
            raise
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
    94
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    95
    def fetchall(self):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    96
        return self.cu.fetchall()
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
    97
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    98
    def fetchone(self):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    99
        return self.cu.fetchone()
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   100
2625
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2620
diff changeset
   101
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   102
def make_schema(selected, solution, table, typemap):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   103
    """return a sql schema to store RQL query result"""
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   104
    sql = []
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   105
    varmap = {}
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   106
    for i, term in enumerate(selected):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   107
        name = 'C%s' % i
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   108
        key = term.as_string()
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   109
        varmap[key] = '%s.%s' % (table, name)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   110
        ttype = term.get_type(solution)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   111
        try:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   112
            sql.append('%s %s' % (name, typemap[ttype]))
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   113
        except KeyError:
3689
deb13e88e037 follow yams 0.25 api changes to improve performance
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3497
diff changeset
   114
            # assert not schema(ttype).final
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   115
            sql.append('%s %s' % (name, typemap['Int']))
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   116
    return ','.join(sql), varmap
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   117
2625
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2620
diff changeset
   118
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   119
def _modified_sql(table, etypes):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   120
    # XXX protect against sql injection
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   121
    if len(etypes) > 1:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   122
        restr = 'type IN (%s)' % ','.join("'%s'" % etype for etype in etypes)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   123
    else:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   124
        restr = "type='%s'" % etypes[0]
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   125
    if table == 'entities':
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   126
        attr = 'mtime'
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   127
    else:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   128
        attr = 'dtime'
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   129
    return 'SELECT type, eid FROM %s WHERE %s AND %s > %%(time)s' % (
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   130
        table, restr, attr)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   131
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   132
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   133
def sql_or_clauses(sql, clauses):
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   134
    select, restr = sql.split(' WHERE ', 1)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   135
    restrclauses = restr.split(' AND ')
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   136
    for clause in clauses:
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   137
        restrclauses.remove(clause)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   138
    if restrclauses:
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   139
        restr = '%s AND (%s)' % (' AND '.join(restrclauses),
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   140
                                 ' OR '.join(clauses))
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   141
    else:
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   142
        restr = '(%s)' % ' OR '.join(clauses)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   143
    return '%s WHERE %s' % (select, restr)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   144
5891
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   145
def rdef_table_column(rdef):
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   146
    """return table and column used to store the given relation definition in
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   147
    the database
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   148
    """
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   149
    return (SQL_PREFIX + str(rdef.subject),
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   150
            SQL_PREFIX + str(rdef.rtype))
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   151
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   152
def rdef_physical_info(dbhelper, rdef):
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   153
    """return backend type and a boolean flag if NULL values should be allowed
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   154
    for a given relation definition
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   155
    """
6359
0bff5a05385c [sync schema] take care rdef may not be final, in which case we want to use type of eid attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6297
diff changeset
   156
    if rdef.object.final:
0bff5a05385c [sync schema] take care rdef may not be final, in which case we want to use type of eid attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6297
diff changeset
   157
        ttype = rdef.object
0bff5a05385c [sync schema] take care rdef may not be final, in which case we want to use type of eid attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6297
diff changeset
   158
    else:
0bff5a05385c [sync schema] take care rdef may not be final, in which case we want to use type of eid attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6297
diff changeset
   159
        ttype = 'Int' # eid type
0bff5a05385c [sync schema] take care rdef may not be final, in which case we want to use type of eid attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6297
diff changeset
   160
    coltype = y2sql.type_from_constraints(dbhelper, ttype,
5891
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   161
                                          rdef.constraints, creating=False)
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   162
    allownull = rdef.cardinality[0] != '1'
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   163
    return coltype, allownull
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   164
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   165
class UndoException(Exception):
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   166
    """something went wrong during undoing"""
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   167
7530
15178bf89fb6 [server] fix unicode conversion capability in UndoException
Julien Jehannet <julien.jehannet@logilab.fr>
parents: 7507
diff changeset
   168
    def __unicode__(self):
15178bf89fb6 [server] fix unicode conversion capability in UndoException
Julien Jehannet <julien.jehannet@logilab.fr>
parents: 7507
diff changeset
   169
        """Called by the unicode builtin; should return a Unicode object
15178bf89fb6 [server] fix unicode conversion capability in UndoException
Julien Jehannet <julien.jehannet@logilab.fr>
parents: 7507
diff changeset
   170
15178bf89fb6 [server] fix unicode conversion capability in UndoException
Julien Jehannet <julien.jehannet@logilab.fr>
parents: 7507
diff changeset
   171
        Type of UndoException message must be `unicode` by design in CubicWeb.
15178bf89fb6 [server] fix unicode conversion capability in UndoException
Julien Jehannet <julien.jehannet@logilab.fr>
parents: 7507
diff changeset
   172
15178bf89fb6 [server] fix unicode conversion capability in UndoException
Julien Jehannet <julien.jehannet@logilab.fr>
parents: 7507
diff changeset
   173
        .. warning::
15178bf89fb6 [server] fix unicode conversion capability in UndoException
Julien Jehannet <julien.jehannet@logilab.fr>
parents: 7507
diff changeset
   174
            This method is not available in python2.5"""
15178bf89fb6 [server] fix unicode conversion capability in UndoException
Julien Jehannet <julien.jehannet@logilab.fr>
parents: 7507
diff changeset
   175
        assert isinstance(self.message, unicode)
15178bf89fb6 [server] fix unicode conversion capability in UndoException
Julien Jehannet <julien.jehannet@logilab.fr>
parents: 7507
diff changeset
   176
        return self.message
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   177
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   178
def _undo_check_relation_target(tentity, rdef, role):
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   179
    """check linked entity has not been redirected for this relation"""
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   180
    card = rdef.role_cardinality(role)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   181
    if card in '?1' and tentity.related(rdef.rtype, role):
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   182
        raise UndoException(tentity._cw._(
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   183
            "Can't restore %(role)s relation %(rtype)s to entity %(eid)s which "
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   184
            "is already linked using this relation.")
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   185
                            % {'role': neg_role(role),
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   186
                               'rtype': rdef.rtype,
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   187
                               'eid': tentity.eid})
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   188
5076
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   189
def _undo_rel_info(session, subj, rtype, obj):
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   190
    entities = []
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   191
    for role, eid in (('subject', subj), ('object', obj)):
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   192
        try:
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   193
            entities.append(session.entity_from_eid(eid))
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   194
        except UnknownEid:
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   195
            raise UndoException(session._(
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   196
                "Can't restore relation %(rtype)s, %(role)s entity %(eid)s"
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   197
                " doesn't exist anymore.")
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   198
                                % {'role': session._(role),
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   199
                                   'rtype': session._(rtype),
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   200
                                   'eid': eid})
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   201
    sentity, oentity = entities
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   202
    try:
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   203
        rschema = session.vreg.schema.rschema(rtype)
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   204
        rdef = rschema.rdefs[(sentity.__regid__, oentity.__regid__)]
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   205
    except KeyError:
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   206
        raise UndoException(session._(
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   207
            "Can't restore relation %(rtype)s between %(subj)s and "
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   208
            "%(obj)s, that relation does not exists anymore in the "
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   209
            "schema.")
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   210
                            % {'rtype': session._(rtype),
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   211
                               'subj': subj,
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   212
                               'obj': obj})
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   213
    return sentity, oentity, rdef
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   214
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   215
def _undo_has_later_transaction(session, eid):
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   216
    return session.system_sql('''\
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   217
SELECT T.tx_uuid FROM transactions AS TREF, transactions AS T
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   218
WHERE TREF.tx_uuid='%(txuuid)s' AND T.tx_uuid!='%(txuuid)s'
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   219
AND T.tx_time>=TREF.tx_time
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   220
AND (EXISTS(SELECT 1 FROM tx_entity_actions AS TEA
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   221
            WHERE TEA.tx_uuid=T.tx_uuid AND TEA.eid=%(eid)s)
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   222
     OR EXISTS(SELECT 1 FROM tx_relation_actions as TRA
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   223
               WHERE TRA.tx_uuid=T.tx_uuid AND (
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   224
                   TRA.eid_from=%(eid)s OR TRA.eid_to=%(eid)s))
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   225
     )''' % {'txuuid': session.transaction_data['undoing_uuid'],
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   226
             'eid': eid}).fetchone()
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   227
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   228
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   229
class NativeSQLSource(SQLAdapterMixIn, AbstractSource):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   230
    """adapter for source using the native cubicweb schema (see below)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   231
    """
2354
9b4bac626977 ability to map attributes to something else than usual cw mapping on sql generation
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2310
diff changeset
   232
    sqlgen_class = SQLGenerator
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   233
    options = (
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   234
        ('db-driver',
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   235
         {'type' : 'string',
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   236
          'default': 'postgres',
5423
e15abfdcce38 backport default into stable: stable is now cw 3.8
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5397 5421
diff changeset
   237
          # XXX use choice type
e15abfdcce38 backport default into stable: stable is now cw 3.8
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5397 5421
diff changeset
   238
          'help': 'database driver (postgres, mysql, sqlite, sqlserver2005)',
6201
cf445fcaf8fe [c-c create] ask for database driver by default
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6128
diff changeset
   239
          'group': 'native-source', 'level': 0,
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   240
          }),
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   241
        ('db-host',
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   242
         {'type' : 'string',
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   243
          'default': '',
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   244
          'help': 'database host',
5323
329b4f6d18b4 [config] with lgc >= 0.50, option's dict inputlevel becomes level
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5217
diff changeset
   245
          'group': 'native-source', 'level': 1,
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   246
          }),
2566
714a8743d423 missing db-port option to source's option definitions
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2354
diff changeset
   247
        ('db-port',
714a8743d423 missing db-port option to source's option definitions
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2354
diff changeset
   248
         {'type' : 'string',
714a8743d423 missing db-port option to source's option definitions
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2354
diff changeset
   249
          'default': '',
714a8743d423 missing db-port option to source's option definitions
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2354
diff changeset
   250
          'help': 'database port',
5323
329b4f6d18b4 [config] with lgc >= 0.50, option's dict inputlevel becomes level
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5217
diff changeset
   251
          'group': 'native-source', 'level': 1,
2566
714a8743d423 missing db-port option to source's option definitions
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2354
diff changeset
   252
          }),
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   253
        ('db-name',
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   254
         {'type' : 'string',
3835
a191b3b9e455 more sensible default values to c-c "create" inputs
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3689
diff changeset
   255
          'default': Method('default_instance_id'),
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   256
          'help': 'database name',
5323
329b4f6d18b4 [config] with lgc >= 0.50, option's dict inputlevel becomes level
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5217
diff changeset
   257
          'group': 'native-source', 'level': 0,
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   258
          }),
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   259
        ('db-user',
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   260
         {'type' : 'string',
3835
a191b3b9e455 more sensible default values to c-c "create" inputs
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3689
diff changeset
   261
          'default': CubicWebNoAppConfiguration.mode == 'user' and getlogin() or 'cubicweb',
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   262
          'help': 'database user',
5323
329b4f6d18b4 [config] with lgc >= 0.50, option's dict inputlevel becomes level
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5217
diff changeset
   263
          'group': 'native-source', 'level': 0,
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   264
          }),
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   265
        ('db-password',
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   266
         {'type' : 'password',
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   267
          'default': '',
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   268
          'help': 'database password',
5323
329b4f6d18b4 [config] with lgc >= 0.50, option's dict inputlevel becomes level
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5217
diff changeset
   269
          'group': 'native-source', 'level': 0,
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   270
          }),
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   271
        ('db-encoding',
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   272
         {'type' : 'string',
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   273
          'default': 'utf8',
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   274
          'help': 'database encoding',
5323
329b4f6d18b4 [config] with lgc >= 0.50, option's dict inputlevel becomes level
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5217
diff changeset
   275
          'group': 'native-source', 'level': 1,
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   276
          }),
5413
dc896e698ab0 added db-extra-arguments to configuration parameters for native source
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5410
diff changeset
   277
        ('db-extra-arguments',
dc896e698ab0 added db-extra-arguments to configuration parameters for native source
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5410
diff changeset
   278
         {'type' : 'string',
dc896e698ab0 added db-extra-arguments to configuration parameters for native source
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5410
diff changeset
   279
          'default': '',
dc896e698ab0 added db-extra-arguments to configuration parameters for native source
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5410
diff changeset
   280
          'help': 'set to "Trusted_Connection" if you are using SQLServer and '
dc896e698ab0 added db-extra-arguments to configuration parameters for native source
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5410
diff changeset
   281
                  'want trusted authentication for the database connection',
5515
513af9be9e37 [config] rename remaining inputlevel to level in option definitions, as expected by lgc >= 0.50
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   282
          'group': 'native-source', 'level': 2,
5413
dc896e698ab0 added db-extra-arguments to configuration parameters for native source
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5410
diff changeset
   283
          }),
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   284
    )
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   285
6427
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6426
diff changeset
   286
    def __init__(self, repo, source_config, *args, **kwargs):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   287
        SQLAdapterMixIn.__init__(self, source_config)
3647
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
   288
        self.authentifiers = [LoginPasswordAuthentifier(self)]
6849
5a0c2cfc19bf [repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   289
        if repo.config['allow-email-login']:
5a0c2cfc19bf [repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   290
            self.authentifiers.insert(0, EmailPasswordAuthentifier(self))
6427
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6426
diff changeset
   291
        AbstractSource.__init__(self, repo, source_config, *args, **kwargs)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   292
        # sql generator
6427
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6426
diff changeset
   293
        self._rql_sqlgen = self.sqlgen_class(self.schema, self.dbhelper,
4831
c5aec27c1bf7 [repo] use logilab.db instead of lgc.adbh/lgc.db/lgc.sqlgen/indexer, test new date extranction functions
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4829
diff changeset
   294
                                             ATTR_MAP.copy())
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   295
        # full text index helper
4806
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
   296
        self.do_fti = not repo.config['delay-full-text-indexation']
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   297
        # sql queries cache
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   298
        self._cache = Cache(repo.config['rql-cache-size'])
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   299
        self._temp_table_data = {}
5061
cdab5220eac0 [cleanup]
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5059
diff changeset
   300
        # we need a lock to protect eid attribution function (XXX, really?
cdab5220eac0 [cleanup]
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5059
diff changeset
   301
        # explain)
7202
5b80f5ee61b5 [native source] eid_creation_lock actually lock the eid connection creation and usage
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7190
diff changeset
   302
        self._eid_cnx_lock = Lock()
5690
6de8437b06bd [source] don't create the eid_creation connection at source initiliaztion time
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 5649
diff changeset
   303
        self._eid_creation_cnx = None
4964
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   304
        # (etype, attr) / storage mapping
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   305
        self._storages = {}
5059
1d5c81588144 [repo] make etype which should go in deleted_entities configurable: we only need this for types imported from other multi-sources instances
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5013
diff changeset
   306
        # entity types that may be used by other multi-sources instances
1d5c81588144 [repo] make etype which should go in deleted_entities configurable: we only need this for types imported from other multi-sources instances
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5013
diff changeset
   307
        self.multisources_etypes = set(repo.config['multi-sources-etypes'])
2072
8008e8812d76 deactivate sqlite connection wrapping for unittest_multisources for now
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2066
diff changeset
   308
        # XXX no_sqlite_wrap trick since we've a sqlite locking pb when
8008e8812d76 deactivate sqlite connection wrapping for unittest_multisources for now
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2066
diff changeset
   309
        # running unittest_multisources with the wrapping below
8008e8812d76 deactivate sqlite connection wrapping for unittest_multisources for now
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2066
diff changeset
   310
        if self.dbdriver == 'sqlite' and \
8008e8812d76 deactivate sqlite connection wrapping for unittest_multisources for now
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2066
diff changeset
   311
               not getattr(repo.config, 'no_sqlite_wrap', False):
2064
a5cd3a92314a properly call [re]set_pool, fix connection handling so we have a change to get cw running on top of a sqlite dabase with threads activated
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2056
diff changeset
   312
            from cubicweb.server.sources.extlite import ConnectionWrapper
5542
a8ad3df5a8a3 [test] absolute sqlite database names in source initialization, avoid pb when cwd is changed later
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5515
diff changeset
   313
            self.dbhelper.dbname = abspath(self.dbhelper.dbname)
2064
a5cd3a92314a properly call [re]set_pool, fix connection handling so we have a change to get cw running on top of a sqlite dabase with threads activated
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2056
diff changeset
   314
            self.get_connection = lambda: ConnectionWrapper(self)
a5cd3a92314a properly call [re]set_pool, fix connection handling so we have a change to get cw running on top of a sqlite dabase with threads activated
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2056
diff changeset
   315
            self.check_connection = lambda cnx: cnx
7398
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   316
            def cnxset_freed(cnx):
2620
de68f84b8f54 R [sql source] cnx._cnx checked in cnx.close, don't do it here
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2619
diff changeset
   317
                cnx.close()
7398
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   318
            self.cnxset_freed = cnxset_freed
5649
a07dee204187 fix unit tests by not using the new create_eid implementation with sqlite
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5645
diff changeset
   319
        if self.dbdriver == 'sqlite':
a07dee204187 fix unit tests by not using the new create_eid implementation with sqlite
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5645
diff changeset
   320
            self._create_eid = None
a07dee204187 fix unit tests by not using the new create_eid implementation with sqlite
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5645
diff changeset
   321
            self.create_eid = self._create_eid_sqlite
5847
51636c991fb4 [migration] refactor schema migration fix introduced by 5833:d7256ae7c1d1
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5824
diff changeset
   322
        self.binary_to_str = self.dbhelper.dbapi_module.binary_to_str
51636c991fb4 [migration] refactor schema migration fix introduced by 5833:d7256ae7c1d1
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5824
diff changeset
   323
2064
a5cd3a92314a properly call [re]set_pool, fix connection handling so we have a change to get cw running on top of a sqlite dabase with threads activated
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2056
diff changeset
   324
a5cd3a92314a properly call [re]set_pool, fix connection handling so we have a change to get cw running on top of a sqlite dabase with threads activated
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2056
diff changeset
   325
    @property
a5cd3a92314a properly call [re]set_pool, fix connection handling so we have a change to get cw running on top of a sqlite dabase with threads activated
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2056
diff changeset
   326
    def _sqlcnx(self):
a5cd3a92314a properly call [re]set_pool, fix connection handling so we have a change to get cw running on top of a sqlite dabase with threads activated
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2056
diff changeset
   327
        # XXX: sqlite connections can only be used in the same thread, so
a5cd3a92314a properly call [re]set_pool, fix connection handling so we have a change to get cw running on top of a sqlite dabase with threads activated
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2056
diff changeset
   328
        #      create a new one each time necessary. If it appears to be time
a5cd3a92314a properly call [re]set_pool, fix connection handling so we have a change to get cw running on top of a sqlite dabase with threads activated
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2056
diff changeset
   329
        #      consuming, find another way
a5cd3a92314a properly call [re]set_pool, fix connection handling so we have a change to get cw running on top of a sqlite dabase with threads activated
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2056
diff changeset
   330
        return SQLAdapterMixIn.get_connection(self)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   331
6945
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   332
    def check_config(self, source_entity):
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   333
        """check configuration of source entity"""
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   334
        if source_entity.host_config:
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   335
            msg = source_entity._cw._('the system source has its configuration '
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   336
                                      'stored on the file-system')
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   337
            raise ValidationError(source_entity.eid, {role_name('config', 'subject'): msg})
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   338
3647
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
   339
    def add_authentifier(self, authentifier):
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
   340
        self.authentifiers.append(authentifier)
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
   341
        authentifier.source = self
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
   342
        authentifier.set_schema(self.schema)
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
   343
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   344
    def reset_caches(self):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   345
        """method called during test to reset potential source caches"""
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   346
        self._cache = Cache(self.repo.config['rql-cache-size'])
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   347
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   348
    def clear_eid_cache(self, eid, etype):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   349
        """clear potential caches for the given eid"""
2610
2933cc6bf9ad [F native source] fix attempts to clear cache key
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2593
diff changeset
   350
        self._cache.pop('Any X WHERE X eid %s, X is %s' % (eid, etype), None)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   351
        self._cache.pop('Any X WHERE X eid %s' % eid, None)
2610
2933cc6bf9ad [F native source] fix attempts to clear cache key
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2593
diff changeset
   352
        self._cache.pop('Any %s' % eid, None)
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   353
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   354
    def sqlexec(self, session, sql, args=None):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   355
        """execute the query and return its result"""
2306
95da5d9f0870 give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2072
diff changeset
   356
        return self.process_result(self.doexec(session, sql, args))
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   357
7398
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   358
    def init_creating(self, cnxset=None):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   359
        # check full text index availibility
4806
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
   360
        if self.do_fti:
7398
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   361
            if cnxset is None:
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   362
                _cnxset = self.repo._get_cnxset()
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   363
                _cnxset.cnxset_set()
6724
24bf6f181d0e [pyro source] store pyro source mapping file into the database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6427
diff changeset
   364
            else:
7398
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   365
                _cnxset = cnxset
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   366
            if not self.dbhelper.has_fti_table(_cnxset['system']):
4825
cdd979ae1b57 fix name error fixed in wrong branch
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4818
diff changeset
   367
                if not self.repo.config.creating:
4810
3055dc7a7c88 don't issue critical warning about missing text index when we're creating the instance
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4807
diff changeset
   368
                    self.critical('no text index table')
4806
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
   369
                self.do_fti = False
7398
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   370
            if cnxset is None:
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   371
                _cnxset.cnxset_freed()
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   372
                self.repo._free_cnxset(_cnxset)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   373
7342
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
   374
    def backup(self, backupfile, confirm, format='native'):
2759
23d7a75693f8 R refactor backup and use tar.gz to store all sources
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 2638
diff changeset
   375
        """method called to create a backup of the source's data"""
7342
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
   376
        if format == 'portable':
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
   377
            self.repo.fill_schema()
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
   378
            self.set_schema(self.repo.schema)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
   379
            helper = DatabaseIndependentBackupRestore(self)
7398
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   380
            self.close_source_connections()
7342
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
   381
            try:
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
   382
                helper.backup(backupfile)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
   383
            finally:
7398
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   384
                self.open_source_connections()
7342
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
   385
        elif format == 'native':
7398
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   386
            self.close_source_connections()
7342
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
   387
            try:
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
   388
                self.backup_to_file(backupfile, confirm)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
   389
            finally:
7398
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   390
                self.open_source_connections()
7342
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
   391
        else:
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
   392
            raise ValueError('Unknown format %r' % format)
2493
9806571ea790 major refactoring of database dump/restore:
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2476
diff changeset
   393
7342
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
   394
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
   395
    def restore(self, backupfile, confirm, drop, format='native'):
2493
9806571ea790 major refactoring of database dump/restore:
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2476
diff changeset
   396
        """method called to restore a backup of source's data"""
7398
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   397
        if self.repo.config.init_cnxset_pool:
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   398
            self.close_source_connections()
2759
23d7a75693f8 R refactor backup and use tar.gz to store all sources
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 2638
diff changeset
   399
        try:
7342
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
   400
            if format == 'portable':
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
   401
                helper = DatabaseIndependentBackupRestore(self)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
   402
                helper.restore(backupfile)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
   403
            elif format == 'native':
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
   404
                self.restore_from_file(backupfile, confirm, drop=drop)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
   405
            else:
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
   406
                raise ValueError('Unknown format %r' % format)
2759
23d7a75693f8 R refactor backup and use tar.gz to store all sources
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 2638
diff changeset
   407
        finally:
7398
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   408
            if self.repo.config.init_cnxset_pool:
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   409
                self.open_source_connections()
2493
9806571ea790 major refactoring of database dump/restore:
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2476
diff changeset
   410
7342
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
   411
6957
ffda12be2e9f [repository] #1460066: backport datafeed cube as cubicweb source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6945
diff changeset
   412
    def init(self, activated, source_entity):
7398
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   413
        self.init_creating(source_entity._cw.cnxset)
7543
570522300e22 [ms, entity metas] add 'actual source' to entities table / base entity metadata cache. Closes #1767090
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7536
diff changeset
   414
        try:
7786
18a366267612 [repo] use smarter query to check asource presence on startup (closes #1922099)
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 7754
diff changeset
   415
            # test if 'asource' column exists
7839
daf46963f4fe ensure the test for asource column won't crash on SQLServer (closes #1949621)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7787
diff changeset
   416
            query = self.dbhelper.sql_add_limit_offset('SELECT asource FROM entities', 1)
daf46963f4fe ensure the test for asource column won't crash on SQLServer (closes #1949621)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7787
diff changeset
   417
            source_entity._cw.system_sql(query)
7543
570522300e22 [ms, entity metas] add 'actual source' to entities table / base entity metadata cache. Closes #1767090
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7536
diff changeset
   418
        except Exception, ex:
570522300e22 [ms, entity metas] add 'actual source' to entities table / base entity metadata cache. Closes #1767090
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7536
diff changeset
   419
            self.eid_type_source = self.eid_type_source_pre_131
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   420
5749
b4393b681f7a [repo] on repository shutdown, we've to close the new eid creation connection
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5729
diff changeset
   421
    def shutdown(self):
b4393b681f7a [repo] on repository shutdown, we've to close the new eid creation connection
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5729
diff changeset
   422
        if self._eid_creation_cnx:
b4393b681f7a [repo] on repository shutdown, we've to close the new eid creation connection
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5729
diff changeset
   423
            self._eid_creation_cnx.close()
5751
1e5ef464cade [test] properly close connections during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5749
diff changeset
   424
            self._eid_creation_cnx = None
5749
b4393b681f7a [repo] on repository shutdown, we've to close the new eid creation connection
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5729
diff changeset
   425
5013
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5011
diff changeset
   426
    # XXX deprecates [un]map_attribute ?
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5011
diff changeset
   427
    def map_attribute(self, etype, attr, cb, sourcedb=True):
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5011
diff changeset
   428
        self._rql_sqlgen.attr_map['%s.%s' % (etype, attr)] = (cb, sourcedb)
2493
9806571ea790 major refactoring of database dump/restore:
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2476
diff changeset
   429
4512
e7ac20bf3629 unset_attribute_storage, for testing purpose at least
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4252
diff changeset
   430
    def unmap_attribute(self, etype, attr):
e7ac20bf3629 unset_attribute_storage, for testing purpose at least
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4252
diff changeset
   431
        self._rql_sqlgen.attr_map.pop('%s.%s' % (etype, attr), None)
e7ac20bf3629 unset_attribute_storage, for testing purpose at least
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4252
diff changeset
   432
4964
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   433
    def set_storage(self, etype, attr, storage):
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   434
        storage_dict = self._storages.setdefault(etype, {})
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   435
        storage_dict[attr] = storage
5013
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5011
diff changeset
   436
        self.map_attribute(etype, attr,
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5011
diff changeset
   437
                           storage.callback, storage.is_source_callback)
4964
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   438
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   439
    def unset_storage(self, etype, attr):
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   440
        self._storages[etype].pop(attr)
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   441
        # if etype has no storage left, remove the entry
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   442
        if not self._storages[etype]:
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   443
            del self._storages[etype]
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   444
        self.unmap_attribute(etype, attr)
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   445
5397
cdbf823450aa [bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5323
diff changeset
   446
    def storage(self, etype, attr):
cdbf823450aa [bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5323
diff changeset
   447
        """return the storage for the given entity type / attribute
cdbf823450aa [bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5323
diff changeset
   448
        """
cdbf823450aa [bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5323
diff changeset
   449
        try:
cdbf823450aa [bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5323
diff changeset
   450
            return self._storages[etype][attr]
cdbf823450aa [bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5323
diff changeset
   451
        except KeyError:
cdbf823450aa [bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5323
diff changeset
   452
            raise Exception('no custom storage set for %s.%s' % (etype, attr))
cdbf823450aa [bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5323
diff changeset
   453
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   454
    # ISource interface #######################################################
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   455
3647
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
   456
    def compile_rql(self, rql, sols):
3240
8604a15995d1 refactor so that rql rewriter may be used outside the server. Enhance it to be usable for RRQLExpression as well
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3041
diff changeset
   457
        rqlst = self.repo.vreg.rqlhelper.parse(rql)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   458
        rqlst.restricted_vars = ()
3647
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
   459
        rqlst.children[0].solutions = sols
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   460
        self.repo.querier.sqlgen_annotate(rqlst)
438
69b79faefa94 need_intersect test and fixes
sylvain.thenault@logilab.fr
parents: 0
diff changeset
   461
        set_qdata(self.schema.rschema, rqlst, ())
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   462
        return rqlst
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   463
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   464
    def set_schema(self, schema):
2476
1294a6bdf3bf application -> instance where it makes sense
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2354
diff changeset
   465
        """set the instance'schema"""
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   466
        self._cache = Cache(self.repo.config['rql-cache-size'])
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   467
        self.cache_hit, self.cache_miss, self.no_cache = 0, 0, 0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   468
        self.schema = schema
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   469
        try:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   470
            self._rql_sqlgen.schema = schema
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   471
        except AttributeError:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   472
            pass # __init__
3647
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
   473
        for authentifier in self.authentifiers:
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
   474
            authentifier.set_schema(self.schema)
4806
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
   475
        clear_cache(self, 'need_fti_indexation')
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   476
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   477
    def support_entity(self, etype, write=False):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   478
        """return true if the given entity's type is handled by this adapter
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   479
        if write is true, return true only if it's a RW support
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   480
        """
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   481
        return not etype in NONSYSTEM_ETYPES
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   482
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   483
    def support_relation(self, rtype, write=False):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   484
        """return true if the given relation's type is handled by this adapter
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   485
        if write is true, return true only if it's a RW support
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   486
        """
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   487
        if write:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   488
            return not rtype in NONSYSTEM_RELATIONS
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   489
        # due to current multi-sources implementation, the system source
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   490
        # can't claim not supporting a relation
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   491
        return True #not rtype == 'content_for'
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   492
3041
782fa7566a22 [multi-sources] new may_cross_relation method on sources
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2967
diff changeset
   493
    def may_cross_relation(self, rtype):
782fa7566a22 [multi-sources] new may_cross_relation method on sources
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2967
diff changeset
   494
        return True
782fa7566a22 [multi-sources] new may_cross_relation method on sources
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2967
diff changeset
   495
3647
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
   496
    def authenticate(self, session, login, **kwargs):
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
   497
        """return CWUser eid for the given login and other authentication
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
   498
        information found in kwargs, else raise `AuthenticationError`
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   499
        """
3647
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
   500
        for authentifier in self.authentifiers:
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   501
            try:
3647
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
   502
                return authentifier.authenticate(session, login, **kwargs)
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
   503
            except AuthenticationError:
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
   504
                continue
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
   505
        raise AuthenticationError()
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   506
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   507
    def syntax_tree_search(self, session, union, args=None, cachekey=None,
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   508
                           varmap=None):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   509
        """return result from this source for a rql query (actually from
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   510
        a rql syntax tree and a solution dictionary mapping each used
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   511
        variable to a possible type). If cachekey is given, the query
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   512
        necessary to fetch the results (but not the results themselves)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   513
        may be cached using this key.
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   514
        """
2625
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2620
diff changeset
   515
        assert dbg_st_search(self.uri, union, varmap, args, cachekey)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   516
        # remember number of actually selected term (sql generation may append some)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   517
        if cachekey is None:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   518
            self.no_cache += 1
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   519
            # generate sql query if we are able to do so (not supported types...)
5013
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5011
diff changeset
   520
            sql, qargs, cbs = self._rql_sqlgen.generate(union, args, varmap)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   521
        else:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   522
            # sql may be cached
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   523
            try:
5013
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5011
diff changeset
   524
                sql, qargs, cbs = self._cache[cachekey]
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   525
                self.cache_hit += 1
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   526
            except KeyError:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   527
                self.cache_miss += 1
5013
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5011
diff changeset
   528
                sql, qargs, cbs = self._rql_sqlgen.generate(union, args, varmap)
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5011
diff changeset
   529
                self._cache[cachekey] = sql, qargs, cbs
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5011
diff changeset
   530
        args = self.merge_args(args, qargs)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   531
        assert isinstance(sql, basestring), repr(sql)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   532
        try:
2306
95da5d9f0870 give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2072
diff changeset
   533
            cursor = self.doexec(session, sql, args)
4831
c5aec27c1bf7 [repo] use logilab.db instead of lgc.adbh/lgc.db/lgc.sqlgen/indexer, test new date extranction functions
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4829
diff changeset
   534
        except (self.OperationalError, self.InterfaceError):
5975
5120d97e2f7e [transaction] do not attempt to reconnect if there has been some write during the transaction
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5847
diff changeset
   535
            if session.mode == 'write':
5120d97e2f7e [transaction] do not attempt to reconnect if there has been some write during the transaction
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5847
diff changeset
   536
                # do not attempt to reconnect if there has been some write
5120d97e2f7e [transaction] do not attempt to reconnect if there has been some write during the transaction
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5847
diff changeset
   537
                # during the transaction
5120d97e2f7e [transaction] do not attempt to reconnect if there has been some write during the transaction
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5847
diff changeset
   538
                raise
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   539
            # FIXME: better detection of deconnection pb
5107
3694bd379513 [source] log attempt to reconnect using warning level. Also, don't relog failed sql (already logged)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5098
diff changeset
   540
            self.warning("trying to reconnect")
7398
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   541
            session.cnxset.reconnect(self)
2306
95da5d9f0870 give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2072
diff changeset
   542
            cursor = self.doexec(session, sql, args)
5605
2604545d7dd9 [win32 SQLServer] connection lost detection
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5515
diff changeset
   543
        except (self.DbapiError,), exc:
2604545d7dd9 [win32 SQLServer] connection lost detection
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5515
diff changeset
   544
            # We get this one with pyodbc and SQL Server when connection was reset
5975
5120d97e2f7e [transaction] do not attempt to reconnect if there has been some write during the transaction
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5847
diff changeset
   545
            if exc.args[0] == '08S01' and session.mode != 'write':
5605
2604545d7dd9 [win32 SQLServer] connection lost detection
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5515
diff changeset
   546
                self.warning("trying to reconnect")
7398
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   547
                session.cnxset.reconnect(self)
5605
2604545d7dd9 [win32 SQLServer] connection lost detection
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5515
diff changeset
   548
                cursor = self.doexec(session, sql, args)
2604545d7dd9 [win32 SQLServer] connection lost detection
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5515
diff changeset
   549
            else:
2604545d7dd9 [win32 SQLServer] connection lost detection
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5515
diff changeset
   550
                raise
5625
6ee2a7b6f194 [external storage] refactor to give session to storage's callback (needed by vcsfile storage)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5606
diff changeset
   551
        results = self.process_result(cursor, cbs, session=session)
2625
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2620
diff changeset
   552
        assert dbg_results(results)
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2620
diff changeset
   553
        return results
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   554
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   555
    def flying_insert(self, table, session, union, args=None, varmap=None):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   556
        """similar as .syntax_tree_search, but inserts data in the
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   557
        temporary table (on-the-fly if possible, eg for the system
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   558
        source whose the given cursor come from). If not possible,
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   559
        inserts all data by calling .executemany().
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   560
        """
2625
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2620
diff changeset
   561
        assert dbg_st_search(
2638
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2627
diff changeset
   562
            self.uri, union, varmap, args,
2625
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2620
diff changeset
   563
            prefix='ON THE FLY temp data insertion into %s from' % table)
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2620
diff changeset
   564
        # generate sql queries if we are able to do so
5013
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5011
diff changeset
   565
        sql, qargs, cbs = self._rql_sqlgen.generate(union, args, varmap)
4831
c5aec27c1bf7 [repo] use logilab.db instead of lgc.adbh/lgc.db/lgc.sqlgen/indexer, test new date extranction functions
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4829
diff changeset
   566
        query = 'INSERT INTO %s %s' % (table, sql.encode(self._dbencoding))
5013
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5011
diff changeset
   567
        self.doexec(session, query, self.merge_args(args, qargs))
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   568
2627
d710278e0c1c manual_insert is a public method
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2625
diff changeset
   569
    def manual_insert(self, results, table, session):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   570
        """insert given result into a temporary table on the system source"""
2625
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2620
diff changeset
   571
        if server.DEBUG & server.DBG_RQL:
6128
fbb8398f80dc cleanups
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6019
diff changeset
   572
            print '  manual insertion of', len(results), 'results into', table
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   573
        if not results:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   574
            return
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   575
        query_args = ['%%(%s)s' % i for i in xrange(len(results[0]))]
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   576
        query = 'INSERT INTO %s VALUES(%s)' % (table, ','.join(query_args))
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   577
        kwargs_list = []
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   578
        for row in results:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   579
            kwargs = {}
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   580
            row = tuple(row)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   581
            for index, cell in enumerate(row):
2066
2c4bf4ee88a2 cleanup, stop encoding unicode string in manual_insert, no more necessary and make crash recent sqlite w/ 8bit string
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2064
diff changeset
   582
                if isinstance(cell, Binary):
4831
c5aec27c1bf7 [repo] use logilab.db instead of lgc.adbh/lgc.db/lgc.sqlgen/indexer, test new date extranction functions
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4829
diff changeset
   583
                    cell = self._binary(cell.getvalue())
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   584
                kwargs[str(index)] = cell
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   585
            kwargs_list.append(kwargs)
2306
95da5d9f0870 give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2072
diff changeset
   586
        self.doexecmany(session, query, kwargs_list)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   587
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   588
    def clean_temp_data(self, session, temptables):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   589
        """remove temporary data, usually associated to temporary tables"""
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   590
        if temptables:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   591
            for table in temptables:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   592
                try:
2306
95da5d9f0870 give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2072
diff changeset
   593
                    self.doexec(session,'DROP TABLE %s' % table)
7645
cb2459aaa7d7 [source] closes #1816412: try to reconnect to dbms if connection is reset in eid_type_source()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 7562
diff changeset
   594
                except Exception:
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   595
                    pass
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   596
                try:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   597
                    del self._temp_table_data[table]
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   598
                except KeyError:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   599
                    continue
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   600
4964
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   601
    @contextmanager
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   602
    def _storage_handler(self, entity, event):
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   603
        # 1/ memorize values as they are before the storage is called.
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   604
        #    For instance, the BFSStorage will replace the `data`
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   605
        #    binary value with a Binary containing the destination path
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   606
        #    on the filesystem. To make the entity.data usage absolutely
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   607
        #    transparent, we'll have to reset entity.data to its binary
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   608
        #    value once the SQL query will be executed
6889
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
   609
        restore_values = []
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
   610
        if isinstance(entity, list):
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
   611
            entities = entity
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
   612
        else:
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
   613
            entities = [entity]
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
   614
        etype = entities[0].__regid__
4964
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   615
        for attr, storage in self._storages.get(etype, {}).items():
6889
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
   616
            for entity in entities:
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
   617
                try:
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
   618
                    edited = entity.cw_edited
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
   619
                except AttributeError:
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
   620
                    assert event == 'deleted'
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
   621
                    getattr(storage, 'entity_deleted')(entity, attr)
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
   622
                else:
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
   623
                    if attr in edited:
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
   624
                        handler = getattr(storage, 'entity_%s' % event)
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
   625
                        to_restore = handler(entity, attr)
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
   626
                        restore_values.append((entity, attr, to_restore))
5128
e5d300d75519 [python] take care to this detail of @contextmanager: if an unhandled exception occurs in the block, it is reraised inside the generator at the point where the yield occurred
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5107
diff changeset
   627
        try:
e5d300d75519 [python] take care to this detail of @contextmanager: if an unhandled exception occurs in the block, it is reraised inside the generator at the point where the yield occurred
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5107
diff changeset
   628
            yield # 2/ execute the source's instructions
e5d300d75519 [python] take care to this detail of @contextmanager: if an unhandled exception occurs in the block, it is reraised inside the generator at the point where the yield occurred
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5107
diff changeset
   629
        finally:
e5d300d75519 [python] take care to this detail of @contextmanager: if an unhandled exception occurs in the block, it is reraised inside the generator at the point where the yield occurred
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5107
diff changeset
   630
            # 3/ restore original values
6889
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
   631
            for entity, attr, value in restore_values:
6142
8bc6eac1fac1 [session] cleanup hook / operation / entity edition api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6128
diff changeset
   632
                entity.cw_edited.edited_attribute(attr, value)
4964
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   633
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   634
    def add_entity(self, session, entity):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   635
        """add a new entity to the source"""
4964
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   636
        with self._storage_handler(entity, 'added'):
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   637
            attrs = self.preprocess_entity(entity)
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   638
            sql = self.sqlgen.insert(SQL_PREFIX + entity.__regid__, attrs)
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   639
            self.doexec(session, sql, attrs)
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   640
            if session.undoable_action('C', entity.__regid__):
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   641
                self._record_tx_action(session, 'tx_entity_actions', 'C',
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   642
                                       etype=entity.__regid__, eid=entity.eid)
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   643
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   644
    def update_entity(self, session, entity):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   645
        """replace an entity in the source"""
4964
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   646
        with self._storage_handler(entity, 'updated'):
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   647
            attrs = self.preprocess_entity(entity)
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   648
            if session.undoable_action('U', entity.__regid__):
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   649
                changes = self._save_attrs(session, entity, attrs)
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   650
                self._record_tx_action(session, 'tx_entity_actions', 'U',
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   651
                                       etype=entity.__regid__, eid=entity.eid,
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   652
                                       changes=self._binary(dumps(changes)))
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   653
            sql = self.sqlgen.update(SQL_PREFIX + entity.__regid__, attrs,
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   654
                                     ['cw_eid'])
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   655
            self.doexec(session, sql, attrs)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   656
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   657
    def delete_entity(self, session, entity):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   658
        """delete an entity from the source"""
4964
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   659
        with self._storage_handler(entity, 'deleted'):
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   660
            if session.undoable_action('D', entity.__regid__):
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   661
                attrs = [SQL_PREFIX + r.type
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   662
                         for r in entity.e_schema.subject_relations()
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   663
                         if (r.final or r.inlined) and not r in VIRTUAL_RTYPES]
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   664
                changes = self._save_attrs(session, entity, attrs)
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   665
                self._record_tx_action(session, 'tx_entity_actions', 'D',
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   666
                                       etype=entity.__regid__, eid=entity.eid,
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   667
                                       changes=self._binary(dumps(changes)))
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   668
            attrs = {'cw_eid': entity.eid}
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   669
            sql = self.sqlgen.delete(SQL_PREFIX + entity.__regid__, attrs)
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   670
            self.doexec(session, sql, attrs)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   671
5072
072ae171aeb0 [cleanup] style fixes, add nodes, 0.2 cents refactorings
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5071
diff changeset
   672
    def add_relation(self, session, subject, rtype, object, inlined=False):
072ae171aeb0 [cleanup] style fixes, add nodes, 0.2 cents refactorings
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5071
diff changeset
   673
        """add a relation to the source"""
7237
9f619715665b [server] improve the speed of setting relations between entities (closes #1625257)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7202
diff changeset
   674
        self._add_relations(session,  rtype, [(subject, object)], inlined)
5072
072ae171aeb0 [cleanup] style fixes, add nodes, 0.2 cents refactorings
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5071
diff changeset
   675
        if session.undoable_action('A', rtype):
072ae171aeb0 [cleanup] style fixes, add nodes, 0.2 cents refactorings
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5071
diff changeset
   676
            self._record_tx_action(session, 'tx_relation_actions', 'A',
072ae171aeb0 [cleanup] style fixes, add nodes, 0.2 cents refactorings
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5071
diff changeset
   677
                                   eid_from=subject, rtype=rtype, eid_to=object)
072ae171aeb0 [cleanup] style fixes, add nodes, 0.2 cents refactorings
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5071
diff changeset
   678
7237
9f619715665b [server] improve the speed of setting relations between entities (closes #1625257)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7202
diff changeset
   679
    def add_relations(self, session,  rtype, subj_obj_list, inlined=False):
9f619715665b [server] improve the speed of setting relations between entities (closes #1625257)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7202
diff changeset
   680
        """add a relations to the source"""
9f619715665b [server] improve the speed of setting relations between entities (closes #1625257)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7202
diff changeset
   681
        self._add_relations(session, rtype, subj_obj_list, inlined)
9f619715665b [server] improve the speed of setting relations between entities (closes #1625257)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7202
diff changeset
   682
        if session.undoable_action('A', rtype):
9f619715665b [server] improve the speed of setting relations between entities (closes #1625257)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7202
diff changeset
   683
            for subject, object in subj_obj_list:
9f619715665b [server] improve the speed of setting relations between entities (closes #1625257)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7202
diff changeset
   684
                self._record_tx_action(session, 'tx_relation_actions', 'A',
9f619715665b [server] improve the speed of setting relations between entities (closes #1625257)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7202
diff changeset
   685
                                       eid_from=subject, rtype=rtype, eid_to=object)
7551
2d4ba5b984dc cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7530
diff changeset
   686
7237
9f619715665b [server] improve the speed of setting relations between entities (closes #1625257)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7202
diff changeset
   687
    def _add_relations(self, session, rtype, subj_obj_list, inlined=False):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   688
        """add a relation to the source"""
7243
9ab01bf84eac fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7237
diff changeset
   689
        sql = []
4818
9f9bfbcdecfd le patch massiveimport a été importé
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4810
diff changeset
   690
        if inlined is False:
7237
9f619715665b [server] improve the speed of setting relations between entities (closes #1625257)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7202
diff changeset
   691
            attrs = [{'eid_from': subject, 'eid_to': object}
9f619715665b [server] improve the speed of setting relations between entities (closes #1625257)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7202
diff changeset
   692
                     for subject, object in subj_obj_list]
7243
9ab01bf84eac fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7237
diff changeset
   693
            sql.append((self.sqlgen.insert('%s_relation' % rtype, attrs[0]), attrs))
4818
9f9bfbcdecfd le patch massiveimport a été importé
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4810
diff changeset
   694
        else: # used by data import
7243
9ab01bf84eac fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7237
diff changeset
   695
            etypes = {}
9ab01bf84eac fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7237
diff changeset
   696
            for subject, object in subj_obj_list:
9ab01bf84eac fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7237
diff changeset
   697
                etype = session.describe(subject)[0]
9ab01bf84eac fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7237
diff changeset
   698
                if etype in etypes:
9ab01bf84eac fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7237
diff changeset
   699
                    etypes[etype].append((subject, object))
9ab01bf84eac fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7237
diff changeset
   700
                else:
9ab01bf84eac fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7237
diff changeset
   701
                    etypes[etype] = [(subject, object)]
9ab01bf84eac fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7237
diff changeset
   702
            for subj_etype, subj_obj_list in etypes.iteritems():
9ab01bf84eac fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7237
diff changeset
   703
                attrs = [{'cw_eid': subject, SQL_PREFIX + rtype: object}
9ab01bf84eac fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7237
diff changeset
   704
                         for subject, object in subj_obj_list]
9ab01bf84eac fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7237
diff changeset
   705
                sql.append((self.sqlgen.update(SQL_PREFIX + etype, attrs[0],
9ab01bf84eac fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7237
diff changeset
   706
                                     ['cw_eid']),
9ab01bf84eac fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7237
diff changeset
   707
                            attrs))
9ab01bf84eac fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7237
diff changeset
   708
        for statement, attrs in sql:
9ab01bf84eac fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7237
diff changeset
   709
            self.doexecmany(session, statement, attrs)
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   710
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   711
    def delete_relation(self, session, subject, rtype, object):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   712
        """delete a relation from the source"""
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   713
        rschema = self.schema.rschema(rtype)
5076
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   714
        self._delete_relation(session, subject, rtype, object, rschema.inlined)
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   715
        if session.undoable_action('R', rtype):
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   716
            self._record_tx_action(session, 'tx_relation_actions', 'R',
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   717
                                   eid_from=subject, rtype=rtype, eid_to=object)
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   718
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   719
    def _delete_relation(self, session, subject, rtype, object, inlined=False):
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   720
        """delete a relation from the source"""
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   721
        if inlined:
1251
af40e615dc89 introduce a 'cw_' prefix on entity table and column names so we don't conflict with sql or DBMS specific keywords
sylvain.thenault@logilab.fr
parents: 1079
diff changeset
   722
            table = SQL_PREFIX + session.describe(subject)[0]
af40e615dc89 introduce a 'cw_' prefix on entity table and column names so we don't conflict with sql or DBMS specific keywords
sylvain.thenault@logilab.fr
parents: 1079
diff changeset
   723
            column = SQL_PREFIX + rtype
af40e615dc89 introduce a 'cw_' prefix on entity table and column names so we don't conflict with sql or DBMS specific keywords
sylvain.thenault@logilab.fr
parents: 1079
diff changeset
   724
            sql = 'UPDATE %s SET %s=NULL WHERE %seid=%%(eid)s' % (table, column,
af40e615dc89 introduce a 'cw_' prefix on entity table and column names so we don't conflict with sql or DBMS specific keywords
sylvain.thenault@logilab.fr
parents: 1079
diff changeset
   725
                                                                  SQL_PREFIX)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   726
            attrs = {'eid' : subject}
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   727
        else:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   728
            attrs = {'eid_from': subject, 'eid_to': object}
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   729
            sql = self.sqlgen.delete('%s_relation' % rtype, attrs)
2306
95da5d9f0870 give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2072
diff changeset
   730
        self.doexec(session, sql, attrs)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   731
2618
ff9b0d5bd884 [F repo sqlite schema changes] don't rollback on potentially expected schema changes failure
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2611
diff changeset
   732
    def doexec(self, session, query, args=None, rollback=True):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   733
        """Execute a query.
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   734
        it's a function just so that it shows up in profiling
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   735
        """
7398
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   736
        cursor = session.cnxset[self.uri]
2593
16d9419a4a79 F: start to handle binary debug log level on the server side
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2589
diff changeset
   737
        if server.DEBUG & server.DBG_SQL:
7398
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   738
            cnx = session.cnxset.connection(self.uri)
2625
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2620
diff changeset
   739
            # getattr to get the actual connection if cnx is a ConnectionWrapper
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2620
diff changeset
   740
            # instance
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2620
diff changeset
   741
            print 'exec', query, args, getattr(cnx, '_cnx', cnx)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   742
        try:
2306
95da5d9f0870 give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2072
diff changeset
   743
            # str(query) to avoid error if it's an unicode string
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   744
            cursor.execute(str(query), args)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   745
        except Exception, ex:
4682
4994901b7379 don't issue critical message when trying to alter sqlite db during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4517
diff changeset
   746
            if self.repo.config.mode != 'test':
4994901b7379 don't issue critical message when trying to alter sqlite db during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4517
diff changeset
   747
                # during test we get those message when trying to alter sqlite
4994901b7379 don't issue critical message when trying to alter sqlite db during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4517
diff changeset
   748
                # db schema
4994901b7379 don't issue critical message when trying to alter sqlite db during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4517
diff changeset
   749
                self.critical("sql: %r\n args: %s\ndbms message: %r",
4994901b7379 don't issue critical message when trying to alter sqlite db during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4517
diff changeset
   750
                              query, args, ex.args[0])
2618
ff9b0d5bd884 [F repo sqlite schema changes] don't rollback on potentially expected schema changes failure
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2611
diff changeset
   751
            if rollback:
ff9b0d5bd884 [F repo sqlite schema changes] don't rollback on potentially expected schema changes failure
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2611
diff changeset
   752
                try:
7398
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   753
                    session.cnxset.connection(self.uri).rollback()
4692
11a040e2601c [test] also hide this message during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4682
diff changeset
   754
                    if self.repo.config.mode != 'test':
11a040e2601c [test] also hide this message during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4682
diff changeset
   755
                        self.critical('transaction has been rollbacked')
7551
2d4ba5b984dc cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7530
diff changeset
   756
                except Exception, ex:
2618
ff9b0d5bd884 [F repo sqlite schema changes] don't rollback on potentially expected schema changes failure
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2611
diff changeset
   757
                    pass
6211
e9d125fd1465 nicer error reporting for unique together constraints
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6201
diff changeset
   758
            if ex.__class__.__name__ == 'IntegrityError':
e9d125fd1465 nicer error reporting for unique together constraints
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6201
diff changeset
   759
                # need string comparison because of various backends
e9d125fd1465 nicer error reporting for unique together constraints
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6201
diff changeset
   760
                for arg in ex.args:
e9d125fd1465 nicer error reporting for unique together constraints
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6201
diff changeset
   761
                    mo = re.search('unique_cw_[^ ]+_idx', arg)
e9d125fd1465 nicer error reporting for unique together constraints
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6201
diff changeset
   762
                    if mo is not None:
e9d125fd1465 nicer error reporting for unique together constraints
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6201
diff changeset
   763
                        index_name = mo.group(0)
e9d125fd1465 nicer error reporting for unique together constraints
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6201
diff changeset
   764
                        elements = index_name.rstrip('_idx').split('_cw_')[1:]
e9d125fd1465 nicer error reporting for unique together constraints
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6201
diff changeset
   765
                        etype = elements[0]
6297
23c1e50ff97b [rql] fix bug with query like 'Any 1 WHERE NOT X in_group G': tables should be kept in EXISTS() even when there are no restriction
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6247
diff changeset
   766
                        rtypes = elements[1:]
6211
e9d125fd1465 nicer error reporting for unique together constraints
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6201
diff changeset
   767
                        raise UniqueTogetherError(etype, rtypes)
6247
f7cb092d2296 unique_together: recast exception raised by sqlite
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6211
diff changeset
   768
                    mo = re.search('columns (.*) are not unique', arg)
f7cb092d2296 unique_together: recast exception raised by sqlite
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6211
diff changeset
   769
                    if mo is not None: # sqlite in use
f7cb092d2296 unique_together: recast exception raised by sqlite
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6211
diff changeset
   770
                        rtypes = [c.strip().lstrip('cw_') for c in mo.group(1).split(',')]
f7cb092d2296 unique_together: recast exception raised by sqlite
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6211
diff changeset
   771
                        etype = '???'
f7cb092d2296 unique_together: recast exception raised by sqlite
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6211
diff changeset
   772
                        raise UniqueTogetherError(etype, rtypes)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   773
            raise
2306
95da5d9f0870 give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2072
diff changeset
   774
        return cursor
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   775
2306
95da5d9f0870 give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2072
diff changeset
   776
    def doexecmany(self, session, query, args):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   777
        """Execute a query.
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   778
        it's a function just so that it shows up in profiling
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   779
        """
2593
16d9419a4a79 F: start to handle binary debug log level on the server side
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2589
diff changeset
   780
        if server.DEBUG & server.DBG_SQL:
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   781
            print 'execmany', query, 'with', len(args), 'arguments'
7398
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   782
        cursor = session.cnxset[self.uri]
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   783
        try:
2306
95da5d9f0870 give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2072
diff changeset
   784
            # str(query) to avoid error if it's an unicode string
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   785
            cursor.executemany(str(query), args)
2306
95da5d9f0870 give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2072
diff changeset
   786
        except Exception, ex:
4682
4994901b7379 don't issue critical message when trying to alter sqlite db during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4517
diff changeset
   787
            if self.repo.config.mode != 'test':
4994901b7379 don't issue critical message when trying to alter sqlite db during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4517
diff changeset
   788
                # during test we get those message when trying to alter sqlite
4994901b7379 don't issue critical message when trying to alter sqlite db during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4517
diff changeset
   789
                # db schema
4994901b7379 don't issue critical message when trying to alter sqlite db during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4517
diff changeset
   790
                self.critical("sql many: %r\n args: %s\ndbms message: %r",
4994901b7379 don't issue critical message when trying to alter sqlite db during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4517
diff changeset
   791
                              query, args, ex.args[0])
2306
95da5d9f0870 give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2072
diff changeset
   792
            try:
7398
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   793
                session.cnxset.connection(self.uri).rollback()
4692
11a040e2601c [test] also hide this message during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4682
diff changeset
   794
                if self.repo.config.mode != 'test':
11a040e2601c [test] also hide this message during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4682
diff changeset
   795
                    self.critical('transaction has been rollbacked')
7645
cb2459aaa7d7 [source] closes #1816412: try to reconnect to dbms if connection is reset in eid_type_source()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 7562
diff changeset
   796
            except Exception:
2306
95da5d9f0870 give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2072
diff changeset
   797
                pass
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   798
            raise
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   799
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   800
    # short cut to method requiring advanced db helper usage ##################
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   801
5891
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   802
    def update_rdef_column(self, session, rdef):
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   803
        """update physical column for a relation definition (final or inlined)
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   804
        """
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   805
        table, column = rdef_table_column(rdef)
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   806
        coltype, allownull = rdef_physical_info(self.dbhelper, rdef)
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   807
        if not self.dbhelper.alter_column_support:
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   808
            self.error("backend can't alter %s.%s to %s%s", table, column, coltype,
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   809
                       not allownull and 'NOT NULL' or '')
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   810
            return
7398
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   811
        self.dbhelper.change_col_type(LogCursor(session.cnxset[self.uri]),
5891
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   812
                                      table, column, coltype, allownull)
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   813
        self.info('altered %s.%s: now %s%s', table, column, coltype,
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   814
                  not allownull and 'NOT NULL' or '')
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   815
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   816
    def update_rdef_null_allowed(self, session, rdef):
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   817
        """update NULL / NOT NULL of physical column for a relation definition
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   818
        (final or inlined)
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   819
        """
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   820
        if not self.dbhelper.alter_column_support:
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   821
            # not supported (and NOT NULL not set by yams in that case, so no
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   822
            # worry)
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   823
            return
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   824
        table, column = rdef_table_column(rdef)
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   825
        coltype, allownull = rdef_physical_info(self.dbhelper, rdef)
7398
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   826
        self.dbhelper.set_null_allowed(LogCursor(session.cnxset[self.uri]),
5891
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   827
                                       table, column, coltype, allownull)
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   828
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   829
    def update_rdef_indexed(self, session, rdef):
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   830
        table, column = rdef_table_column(rdef)
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   831
        if rdef.indexed:
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   832
            self.create_index(session, table, column)
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   833
        else:
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   834
            self.drop_index(session, table, column)
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   835
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   836
    def update_rdef_unique(self, session, rdef):
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   837
        table, column = rdef_table_column(rdef)
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   838
        if rdef.constraint_by_type('UniqueConstraint'):
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   839
            self.create_index(session, table, column, unique=True)
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   840
        else:
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   841
            self.drop_index(session, table, column, unique=True)
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   842
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   843
    def create_index(self, session, table, column, unique=False):
7398
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   844
        cursor = LogCursor(session.cnxset[self.uri])
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   845
        self.dbhelper.create_index(cursor, table, column, unique)
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   846
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   847
    def drop_index(self, session, table, column, unique=False):
7398
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   848
        cursor = LogCursor(session.cnxset[self.uri])
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   849
        self.dbhelper.drop_index(cursor, table, column, unique)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   850
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   851
    # system source interface #################################################
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   852
7645
cb2459aaa7d7 [source] closes #1816412: try to reconnect to dbms if connection is reset in eid_type_source()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 7562
diff changeset
   853
    def _eid_type_source(self, session, eid, sql, _retry=True):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   854
        try:
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   855
            res = self.doexec(session, sql).fetchone()
7646
91a984ba420f [native source] fix bug introduced in 7645:cb2459aaa7d7: fetchone() may return None
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7645
diff changeset
   856
            if res is not None:
91a984ba420f [native source] fix bug introduced in 7645:cb2459aaa7d7: fetchone() may return None
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7645
diff changeset
   857
                return res
7645
cb2459aaa7d7 [source] closes #1816412: try to reconnect to dbms if connection is reset in eid_type_source()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 7562
diff changeset
   858
        except (self.OperationalError, self.InterfaceError):
cb2459aaa7d7 [source] closes #1816412: try to reconnect to dbms if connection is reset in eid_type_source()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 7562
diff changeset
   859
            if session.mode == 'read' and _retry:
cb2459aaa7d7 [source] closes #1816412: try to reconnect to dbms if connection is reset in eid_type_source()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 7562
diff changeset
   860
                self.warning("trying to reconnect (eid_type_source())")
cb2459aaa7d7 [source] closes #1816412: try to reconnect to dbms if connection is reset in eid_type_source()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 7562
diff changeset
   861
                session.cnxset.reconnect(self)
cb2459aaa7d7 [source] closes #1816412: try to reconnect to dbms if connection is reset in eid_type_source()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 7562
diff changeset
   862
                return self._eid_type_source(session, eid, sql, _retry=False)
cb2459aaa7d7 [source] closes #1816412: try to reconnect to dbms if connection is reset in eid_type_source()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 7562
diff changeset
   863
        except Exception:
cb2459aaa7d7 [source] closes #1816412: try to reconnect to dbms if connection is reset in eid_type_source()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 7562
diff changeset
   864
            assert session.cnxset, 'session has no connections set'
cb2459aaa7d7 [source] closes #1816412: try to reconnect to dbms if connection is reset in eid_type_source()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 7562
diff changeset
   865
            self.exception('failed to query entities table for eid %s', eid)
cb2459aaa7d7 [source] closes #1816412: try to reconnect to dbms if connection is reset in eid_type_source()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 7562
diff changeset
   866
        raise UnknownEid(eid)
cb2459aaa7d7 [source] closes #1816412: try to reconnect to dbms if connection is reset in eid_type_source()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 7562
diff changeset
   867
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   868
    def eid_type_source(self, session, eid):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   869
        """return a tuple (type, source, extid) for the entity with id <eid>"""
7543
570522300e22 [ms, entity metas] add 'actual source' to entities table / base entity metadata cache. Closes #1767090
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7536
diff changeset
   870
        sql = 'SELECT type, source, extid, asource FROM entities WHERE eid=%s' % eid
7645
cb2459aaa7d7 [source] closes #1816412: try to reconnect to dbms if connection is reset in eid_type_source()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 7562
diff changeset
   871
        res = self._eid_type_source(session, eid, sql)
7543
570522300e22 [ms, entity metas] add 'actual source' to entities table / base entity metadata cache. Closes #1767090
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7536
diff changeset
   872
        if res[-2] is not None:
1952
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1792
diff changeset
   873
            if not isinstance(res, list):
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1792
diff changeset
   874
                res = list(res)
7543
570522300e22 [ms, entity metas] add 'actual source' to entities table / base entity metadata cache. Closes #1767090
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7536
diff changeset
   875
            res[-2] = b64decode(res[-2])
570522300e22 [ms, entity metas] add 'actual source' to entities table / base entity metadata cache. Closes #1767090
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7536
diff changeset
   876
        return res
570522300e22 [ms, entity metas] add 'actual source' to entities table / base entity metadata cache. Closes #1767090
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7536
diff changeset
   877
570522300e22 [ms, entity metas] add 'actual source' to entities table / base entity metadata cache. Closes #1767090
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7536
diff changeset
   878
    def eid_type_source_pre_131(self, session, eid):
570522300e22 [ms, entity metas] add 'actual source' to entities table / base entity metadata cache. Closes #1767090
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7536
diff changeset
   879
        """return a tuple (type, source, extid) for the entity with id <eid>"""
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   880
        sql = 'SELECT type, source, extid FROM entities WHERE eid=%s' % eid
7645
cb2459aaa7d7 [source] closes #1816412: try to reconnect to dbms if connection is reset in eid_type_source()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 7562
diff changeset
   881
        res = self._eid_type_source(session, eid, sql)
7543
570522300e22 [ms, entity metas] add 'actual source' to entities table / base entity metadata cache. Closes #1767090
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7536
diff changeset
   882
        if not isinstance(res, list):
570522300e22 [ms, entity metas] add 'actual source' to entities table / base entity metadata cache. Closes #1767090
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7536
diff changeset
   883
            res = list(res)
1952
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1792
diff changeset
   884
        if res[-1] is not None:
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1792
diff changeset
   885
            res[-1] = b64decode(res[-1])
7543
570522300e22 [ms, entity metas] add 'actual source' to entities table / base entity metadata cache. Closes #1767090
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7536
diff changeset
   886
        res.append(res[1])
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   887
        return res
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   888
6957
ffda12be2e9f [repository] #1460066: backport datafeed cube as cubicweb source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6945
diff changeset
   889
    def extid2eid(self, session, source_uri, extid):
1952
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1792
diff changeset
   890
        """get eid from an external id. Return None if no record found."""
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1792
diff changeset
   891
        assert isinstance(extid, str)
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   892
        cursor = self.doexec(session,
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   893
                             'SELECT eid FROM entities '
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   894
                             'WHERE extid=%(x)s AND source=%(s)s',
6957
ffda12be2e9f [repository] #1460066: backport datafeed cube as cubicweb source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6945
diff changeset
   895
                             {'x': b64encode(extid), 's': source_uri})
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   896
        # XXX testing rowcount cause strange bug with sqlite, results are there
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   897
        #     but rowcount is 0
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   898
        #if cursor.rowcount > 0:
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   899
        try:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   900
            result = cursor.fetchone()
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   901
            if result:
1954
9b20f3504af8 cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1952
diff changeset
   902
                return result[0]
7645
cb2459aaa7d7 [source] closes #1816412: try to reconnect to dbms if connection is reset in eid_type_source()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 7562
diff changeset
   903
        except Exception:
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   904
            pass
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   905
        return None
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   906
5168
1ab032df5ca3 SQL Server port: temporary table handling
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5131
diff changeset
   907
    def make_temp_table_name(self, table):
7190
f72e3f4666b0 [system source] drop now useless bw compat
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7118
diff changeset
   908
        return self.dbhelper.temporary_table_name(table)
5168
1ab032df5ca3 SQL Server port: temporary table handling
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5131
diff changeset
   909
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   910
    def temp_table_def(self, selected, sol, table):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   911
        return make_schema(selected, sol, table, self.dbhelper.TYPE_MAPPING)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   912
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   913
    def create_temp_table(self, session, table, schema):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   914
        # we don't want on commit drop, this may cause problem when
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   915
        # running with an ldap source, and table will be deleted manually any way
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   916
        # on commit
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   917
        sql = self.dbhelper.sql_temporary_table(table, schema, False)
2306
95da5d9f0870 give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2072
diff changeset
   918
        self.doexec(session, sql)
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   919
5649
a07dee204187 fix unit tests by not using the new create_eid implementation with sqlite
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5645
diff changeset
   920
    def _create_eid_sqlite(self, session):
7202
5b80f5ee61b5 [native source] eid_creation_lock actually lock the eid connection creation and usage
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7190
diff changeset
   921
        with self._eid_cnx_lock:
5644
73d8a757db80 fix create_eid for sqlite (and bring back tests)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5639
diff changeset
   922
            for sql in self.dbhelper.sqls_increment_sequence('entities_id_seq'):
73d8a757db80 fix create_eid for sqlite (and bring back tests)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5639
diff changeset
   923
                cursor = self.doexec(session, sql)
73d8a757db80 fix create_eid for sqlite (and bring back tests)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5639
diff changeset
   924
            return cursor.fetchone()[0]
73d8a757db80 fix create_eid for sqlite (and bring back tests)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5639
diff changeset
   925
73d8a757db80 fix create_eid for sqlite (and bring back tests)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5639
diff changeset
   926
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   927
    def create_eid(self, session):
7202
5b80f5ee61b5 [native source] eid_creation_lock actually lock the eid connection creation and usage
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7190
diff changeset
   928
        # lock needed to prevent 'Connection is busy with results for another
5b80f5ee61b5 [native source] eid_creation_lock actually lock the eid connection creation and usage
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7190
diff changeset
   929
        # command (0)' errors with SQLServer
5b80f5ee61b5 [native source] eid_creation_lock actually lock the eid connection creation and usage
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7190
diff changeset
   930
        with self._eid_cnx_lock:
5649
a07dee204187 fix unit tests by not using the new create_eid implementation with sqlite
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5645
diff changeset
   931
            return self._create_eid()
a07dee204187 fix unit tests by not using the new create_eid implementation with sqlite
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5645
diff changeset
   932
a07dee204187 fix unit tests by not using the new create_eid implementation with sqlite
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5645
diff changeset
   933
    def _create_eid(self):
a07dee204187 fix unit tests by not using the new create_eid implementation with sqlite
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5645
diff changeset
   934
        # internal function doing the eid creation without locking.
a07dee204187 fix unit tests by not using the new create_eid implementation with sqlite
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5645
diff changeset
   935
        # needed for the recursive handling of disconnections (otherwise we
7202
5b80f5ee61b5 [native source] eid_creation_lock actually lock the eid connection creation and usage
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7190
diff changeset
   936
        # deadlock on self._eid_cnx_lock
5639
4acb860159e4 [win32] fix deadlock occuring on the sequence tables with SQLServer
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5605
diff changeset
   937
        if self._eid_creation_cnx is None:
4acb860159e4 [win32] fix deadlock occuring on the sequence tables with SQLServer
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5605
diff changeset
   938
            self._eid_creation_cnx = self.get_connection()
4acb860159e4 [win32] fix deadlock occuring on the sequence tables with SQLServer
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5605
diff changeset
   939
        cnx = self._eid_creation_cnx
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   940
        try:
6850
2b9e58174327 [repo] move cursor creation into the try/except, it may raises an InterfaceError if the connection is closed
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6427
diff changeset
   941
            cursor = cnx.cursor()
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   942
            for sql in self.dbhelper.sqls_increment_sequence('entities_id_seq'):
5639
4acb860159e4 [win32] fix deadlock occuring on the sequence tables with SQLServer
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5605
diff changeset
   943
                cursor.execute(sql)
4acb860159e4 [win32] fix deadlock occuring on the sequence tables with SQLServer
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5605
diff changeset
   944
            eid = cursor.fetchone()[0]
4acb860159e4 [win32] fix deadlock occuring on the sequence tables with SQLServer
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5605
diff changeset
   945
        except (self.OperationalError, self.InterfaceError):
4acb860159e4 [win32] fix deadlock occuring on the sequence tables with SQLServer
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5605
diff changeset
   946
            # FIXME: better detection of deconnection pb
4acb860159e4 [win32] fix deadlock occuring on the sequence tables with SQLServer
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5605
diff changeset
   947
            self.warning("trying to reconnect create eid connection")
4acb860159e4 [win32] fix deadlock occuring on the sequence tables with SQLServer
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5605
diff changeset
   948
            self._eid_creation_cnx = None
5649
a07dee204187 fix unit tests by not using the new create_eid implementation with sqlite
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5645
diff changeset
   949
            return self._create_eid()
5639
4acb860159e4 [win32] fix deadlock occuring on the sequence tables with SQLServer
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5605
diff changeset
   950
        except (self.DbapiError,), exc:
4acb860159e4 [win32] fix deadlock occuring on the sequence tables with SQLServer
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5605
diff changeset
   951
            # We get this one with pyodbc and SQL Server when connection was reset
4acb860159e4 [win32] fix deadlock occuring on the sequence tables with SQLServer
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5605
diff changeset
   952
            if exc.args[0] == '08S01':
4acb860159e4 [win32] fix deadlock occuring on the sequence tables with SQLServer
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5605
diff changeset
   953
                self.warning("trying to reconnect create eid connection")
4acb860159e4 [win32] fix deadlock occuring on the sequence tables with SQLServer
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5605
diff changeset
   954
                self._eid_creation_cnx = None
5649
a07dee204187 fix unit tests by not using the new create_eid implementation with sqlite
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5645
diff changeset
   955
                return self._create_eid()
5639
4acb860159e4 [win32] fix deadlock occuring on the sequence tables with SQLServer
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5605
diff changeset
   956
            else:
4acb860159e4 [win32] fix deadlock occuring on the sequence tables with SQLServer
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5605
diff changeset
   957
                raise
7645
cb2459aaa7d7 [source] closes #1816412: try to reconnect to dbms if connection is reset in eid_type_source()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 7562
diff changeset
   958
        except Exception: # WTF?
5639
4acb860159e4 [win32] fix deadlock occuring on the sequence tables with SQLServer
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5605
diff changeset
   959
            cnx.rollback()
4acb860159e4 [win32] fix deadlock occuring on the sequence tables with SQLServer
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5605
diff changeset
   960
            self._eid_creation_cnx = None
4acb860159e4 [win32] fix deadlock occuring on the sequence tables with SQLServer
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5605
diff changeset
   961
            self.exception('create eid failed in an unforeseen way on SQL statement %s', sql)
4acb860159e4 [win32] fix deadlock occuring on the sequence tables with SQLServer
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5605
diff changeset
   962
            raise
4acb860159e4 [win32] fix deadlock occuring on the sequence tables with SQLServer
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5605
diff changeset
   963
        else:
4acb860159e4 [win32] fix deadlock occuring on the sequence tables with SQLServer
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5605
diff changeset
   964
            cnx.commit()
4acb860159e4 [win32] fix deadlock occuring on the sequence tables with SQLServer
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5605
diff changeset
   965
            return eid
5649
a07dee204187 fix unit tests by not using the new create_eid implementation with sqlite
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5645
diff changeset
   966
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   967
    def add_info(self, session, entity, source, extid, complete):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   968
        """add type and source info for an eid into the system table"""
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   969
        # begin by inserting eid/type/source/extid into the entities table
1952
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1792
diff changeset
   970
        if extid is not None:
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1792
diff changeset
   971
            assert isinstance(extid, str)
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1792
diff changeset
   972
            extid = b64encode(extid)
6957
ffda12be2e9f [repository] #1460066: backport datafeed cube as cubicweb source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6945
diff changeset
   973
        uri = 'system' if source.copy_based_source else source.uri
3399
2b84f4adb6f8 use __regid__ instead of id on appobject classes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3240
diff changeset
   974
        attrs = {'type': entity.__regid__, 'eid': entity.eid, 'extid': extid,
7543
570522300e22 [ms, entity metas] add 'actual source' to entities table / base entity metadata cache. Closes #1767090
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7536
diff changeset
   975
                 'source': uri, 'asource': source.uri, 'mtime': datetime.now()}
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   976
        self.doexec(session, self.sqlgen.insert('entities', attrs), attrs)
6427
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6426
diff changeset
   977
        # insert core relations: is, is_instance_of and cw_source
6851
824d5b6eae7f [repo] kill no more needed 'recreate' feature
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6849
diff changeset
   978
        try:
824d5b6eae7f [repo] kill no more needed 'recreate' feature
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6849
diff changeset
   979
            self.doexec(session, 'INSERT INTO is_relation(eid_from,eid_to) VALUES (%s,%s)'
824d5b6eae7f [repo] kill no more needed 'recreate' feature
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6849
diff changeset
   980
                        % (entity.eid, eschema_eid(session, entity.e_schema)))
824d5b6eae7f [repo] kill no more needed 'recreate' feature
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6849
diff changeset
   981
        except IndexError:
824d5b6eae7f [repo] kill no more needed 'recreate' feature
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6849
diff changeset
   982
            # during schema serialization, skip
824d5b6eae7f [repo] kill no more needed 'recreate' feature
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6849
diff changeset
   983
            pass
824d5b6eae7f [repo] kill no more needed 'recreate' feature
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6849
diff changeset
   984
        else:
824d5b6eae7f [repo] kill no more needed 'recreate' feature
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6849
diff changeset
   985
            for eschema in entity.e_schema.ancestors() + [entity.e_schema]:
824d5b6eae7f [repo] kill no more needed 'recreate' feature
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6849
diff changeset
   986
                self.doexec(session, 'INSERT INTO is_instance_of_relation(eid_from,eid_to) VALUES (%s,%s)'
824d5b6eae7f [repo] kill no more needed 'recreate' feature
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6849
diff changeset
   987
                           % (entity.eid, eschema_eid(session, eschema)))
824d5b6eae7f [repo] kill no more needed 'recreate' feature
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6849
diff changeset
   988
        if 'CWSource' in self.schema and source.eid is not None: # else, cw < 3.10
824d5b6eae7f [repo] kill no more needed 'recreate' feature
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6849
diff changeset
   989
            self.doexec(session, 'INSERT INTO cw_source_relation(eid_from,eid_to) '
824d5b6eae7f [repo] kill no more needed 'recreate' feature
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6849
diff changeset
   990
                        'VALUES (%s,%s)' % (entity.eid, source.eid))
4806
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
   991
        # now we can update the full text index
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
   992
        if self.do_fti and self.need_fti_indexation(entity.__regid__):
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
   993
            if complete:
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
   994
                entity.complete(entity.e_schema.indexable_attributes())
5070
b1f80ccadda3 [repo] refactor fti operation to use set_operation and a single operation whatever the number of entities to ft index
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5067
diff changeset
   995
            self.index_entity(session, entity=entity)
4806
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
   996
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
   997
    def update_info(self, session, entity, need_fti_update):
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   998
        """mark entity as being modified, fulltext reindex if needed"""
4806
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
   999
        if self.do_fti and need_fti_update:
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1000
            # reindex the entity only if this query is updating at least
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1001
            # one indexable attribute
5070
b1f80ccadda3 [repo] refactor fti operation to use set_operation and a single operation whatever the number of entities to ft index
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5067
diff changeset
  1002
            self.index_entity(session, entity=entity)
5059
1d5c81588144 [repo] make etype which should go in deleted_entities configurable: we only need this for types imported from other multi-sources instances
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5013
diff changeset
  1003
        # update entities.mtime.
1d5c81588144 [repo] make etype which should go in deleted_entities configurable: we only need this for types imported from other multi-sources instances
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5013
diff changeset
  1004
        # XXX Only if entity.__regid__ in self.multisources_etypes?
4806
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1005
        attrs = {'eid': entity.eid, 'mtime': datetime.now()}
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1006
        self.doexec(session, self.sqlgen.update('entities', attrs, ['eid']), attrs)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1007
7501
2983dd24494a [repository] refactor/cleanup entity deletion methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
  1008
    def delete_info_multi(self, session, entities, uri):
2983dd24494a [repository] refactor/cleanup entity deletion methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
  1009
        """delete system information on deletion of a list of entities with the
2983dd24494a [repository] refactor/cleanup entity deletion methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
  1010
        same etype and belinging to the same source
2983dd24494a [repository] refactor/cleanup entity deletion methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
  1011
5067
adc2122eed03 [repo] more efficient eid cache operations handling based on set_operation; refactor
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5061
diff changeset
  1012
        * update the fti
7501
2983dd24494a [repository] refactor/cleanup entity deletion methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
  1013
        * remove record from the `entities` table
2983dd24494a [repository] refactor/cleanup entity deletion methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
  1014
        * transfer it to the `deleted_entities`
6889
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
  1015
        """
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
  1016
        self.fti_unindex_entities(session, entities)
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
  1017
        attrs = {'eid': '(%s)' % ','.join([str(_e.eid) for _e in entities])}
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
  1018
        self.doexec(session, self.sqlgen.delete_many('entities', attrs), attrs)
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
  1019
        if entities[0].__regid__ not in self.multisources_etypes:
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
  1020
            return
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
  1021
        attrs = {'type': entities[0].__regid__,
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
  1022
                 'source': uri, 'dtime': datetime.now()}
7501
2983dd24494a [repository] refactor/cleanup entity deletion methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
  1023
        for entity in entities:
2983dd24494a [repository] refactor/cleanup entity deletion methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
  1024
            extid = entity.cw_metainformation()['extid']
6889
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
  1025
            if extid is not None:
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
  1026
                assert isinstance(extid, str), type(extid)
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
  1027
                extid = b64encode(extid)
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
  1028
            attrs.update({'eid': entity.eid, 'extid': extid})
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
  1029
            self.doexec(session, self.sqlgen.insert('deleted_entities', attrs), attrs)
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
  1030
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1031
    def modified_entities(self, session, etypes, mtime):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1032
        """return a 2-uple:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1033
        * list of (etype, eid) of entities of the given types which have been
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1034
          modified since the given timestamp (actually entities whose full text
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1035
          index content has changed)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1036
        * list of (etype, eid) of entities of the given types which have been
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1037
          deleted since the given timestamp
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1038
        """
5059
1d5c81588144 [repo] make etype which should go in deleted_entities configurable: we only need this for types imported from other multi-sources instances
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5013
diff changeset
  1039
        for etype in etypes:
1d5c81588144 [repo] make etype which should go in deleted_entities configurable: we only need this for types imported from other multi-sources instances
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5013
diff changeset
  1040
            if not etype in self.multisources_etypes:
6427
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6426
diff changeset
  1041
                self.error('%s not listed as a multi-sources entity types. '
5072
072ae171aeb0 [cleanup] style fixes, add nodes, 0.2 cents refactorings
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5071
diff changeset
  1042
                              'Modify your configuration' % etype)
5059
1d5c81588144 [repo] make etype which should go in deleted_entities configurable: we only need this for types imported from other multi-sources instances
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5013
diff changeset
  1043
                self.multisources_etypes.add(etype)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1044
        modsql = _modified_sql('entities', etypes)
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1045
        cursor = self.doexec(session, modsql, {'time': mtime})
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1046
        modentities = cursor.fetchall()
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1047
        delsql = _modified_sql('deleted_entities', etypes)
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1048
        cursor = self.doexec(session, delsql, {'time': mtime})
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1049
        delentities = cursor.fetchall()
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1050
        return modentities, delentities
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1051
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1052
    # undo support #############################################################
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1053
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1054
    def undoable_transactions(self, session, ueid=None, **actionfilters):
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1055
        """See :class:`cubicweb.dbapi.Connection.undoable_transactions`"""
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1056
        # force filtering to session's user if not a manager
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1057
        if not session.user.is_in_group('managers'):
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1058
            ueid = session.user.eid
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1059
        restr = {}
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1060
        if ueid is not None:
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1061
            restr['tx_user'] = ueid
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1062
        sql = self.sqlgen.select('transactions', restr, ('tx_uuid', 'tx_time', 'tx_user'))
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1063
        if actionfilters:
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1064
            # we will need subqueries to filter transactions according to
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1065
            # actions done
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1066
            tearestr = {} # filters on the tx_entity_actions table
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1067
            trarestr = {} # filters on the tx_relation_actions table
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1068
            genrestr = {} # generic filters, appliyable to both table
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1069
            # unless public explicitly set to false, we only consider public
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1070
            # actions
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1071
            if actionfilters.pop('public', True):
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1072
                genrestr['txa_public'] = True
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1073
            # put additional filters in trarestr and/or tearestr
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1074
            for key, val in actionfilters.iteritems():
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1075
                if key == 'etype':
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1076
                    # filtering on etype implies filtering on entity actions
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1077
                    # only, and with no eid specified
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1078
                    assert actionfilters.get('action', 'C') in 'CUD'
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1079
                    assert not 'eid' in actionfilters
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1080
                    tearestr['etype'] = val
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1081
                elif key == 'eid':
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1082
                    # eid filter may apply to 'eid' of tx_entity_actions or to
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1083
                    # 'eid_from' OR 'eid_to' of tx_relation_actions
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1084
                    if actionfilters.get('action', 'C') in 'CUD':
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1085
                        tearestr['eid'] = val
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1086
                    if actionfilters.get('action', 'A') in 'AR':
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1087
                        trarestr['eid_from'] = val
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1088
                        trarestr['eid_to'] = val
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1089
                elif key == 'action':
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1090
                    if val in 'CUD':
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1091
                        tearestr['txa_action'] = val
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1092
                    else:
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1093
                        assert val in 'AR'
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1094
                        trarestr['txa_action'] = val
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1095
                else:
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1096
                    raise AssertionError('unknow filter %s' % key)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1097
            assert trarestr or tearestr, "can't only filter on 'public'"
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1098
            subqsqls = []
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1099
            # append subqueries to the original query, using EXISTS()
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1100
            if trarestr or (genrestr and not tearestr):
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1101
                trarestr.update(genrestr)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1102
                trasql = self.sqlgen.select('tx_relation_actions', trarestr, ('1',))
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1103
                if 'eid_from' in trarestr:
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1104
                    # replace AND by OR between eid_from/eid_to restriction
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1105
                    trasql = sql_or_clauses(trasql, ['eid_from = %(eid_from)s',
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1106
                                                     'eid_to = %(eid_to)s'])
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1107
                trasql += ' AND transactions.tx_uuid=tx_relation_actions.tx_uuid'
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1108
                subqsqls.append('EXISTS(%s)' % trasql)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1109
            if tearestr or (genrestr and not trarestr):
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1110
                tearestr.update(genrestr)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1111
                teasql = self.sqlgen.select('tx_entity_actions', tearestr, ('1',))
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1112
                teasql += ' AND transactions.tx_uuid=tx_entity_actions.tx_uuid'
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1113
                subqsqls.append('EXISTS(%s)' % teasql)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1114
            if restr:
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1115
                sql += ' AND %s' % ' OR '.join(subqsqls)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1116
            else:
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1117
                sql += ' WHERE %s' % ' OR '.join(subqsqls)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1118
            restr.update(trarestr)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1119
            restr.update(tearestr)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1120
        # we want results ordered by transaction's time descendant
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1121
        sql += ' ORDER BY tx_time DESC'
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1122
        cu = self.doexec(session, sql, restr)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1123
        # turn results into transaction objects
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1124
        return [tx.Transaction(*args) for args in cu.fetchall()]
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1125
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1126
    def tx_info(self, session, txuuid):
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1127
        """See :class:`cubicweb.dbapi.Connection.transaction_info`"""
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1128
        return tx.Transaction(txuuid, *self._tx_info(session, txuuid))
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1129
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1130
    def tx_actions(self, session, txuuid, public):
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1131
        """See :class:`cubicweb.dbapi.Connection.transaction_actions`"""
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1132
        self._tx_info(session, txuuid)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1133
        restr = {'tx_uuid': txuuid}
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1134
        if public:
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1135
            restr['txa_public'] = True
5072
072ae171aeb0 [cleanup] style fixes, add nodes, 0.2 cents refactorings
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5071
diff changeset
  1136
        # XXX use generator to avoid loading everything in memory?
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1137
        sql = self.sqlgen.select('tx_entity_actions', restr,
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1138
                                 ('txa_action', 'txa_public', 'txa_order',
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1139
                                  'etype', 'eid', 'changes'))
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1140
        cu = self.doexec(session, sql, restr)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1141
        actions = [tx.EntityAction(a,p,o,et,e,c and loads(self.binary_to_str(c)))
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1142
                   for a,p,o,et,e,c in cu.fetchall()]
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1143
        sql = self.sqlgen.select('tx_relation_actions', restr,
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1144
                                 ('txa_action', 'txa_public', 'txa_order',
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1145
                                  'rtype', 'eid_from', 'eid_to'))
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1146
        cu = self.doexec(session, sql, restr)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1147
        actions += [tx.RelationAction(*args) for args in cu.fetchall()]
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1148
        return sorted(actions, key=lambda x: x.order)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1149
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1150
    def undo_transaction(self, session, txuuid):
5071
8631bb9f6e73 [undo] during undoing, call hooks in the [active]integrity and undo categories
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5070
diff changeset
  1151
        """See :class:`cubicweb.dbapi.Connection.undo_transaction`
8631bb9f6e73 [undo] during undoing, call hooks in the [active]integrity and undo categories
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5070
diff changeset
  1152
8631bb9f6e73 [undo] during undoing, call hooks in the [active]integrity and undo categories
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5070
diff changeset
  1153
        important note: while undoing of a transaction, only hooks in the
8631bb9f6e73 [undo] during undoing, call hooks in the [active]integrity and undo categories
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5070
diff changeset
  1154
        'integrity', 'activeintegrity' and 'undo' categories are called.
8631bb9f6e73 [undo] during undoing, call hooks in the [active]integrity and undo categories
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5070
diff changeset
  1155
        """
7398
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
  1156
        # set mode so connections set isn't released subsquently until commit/rollback
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1157
        session.mode = 'write'
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1158
        errors = []
5071
8631bb9f6e73 [undo] during undoing, call hooks in the [active]integrity and undo categories
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5070
diff changeset
  1159
        session.transaction_data['undoing_uuid'] = txuuid
8631bb9f6e73 [undo] during undoing, call hooks in the [active]integrity and undo categories
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5070
diff changeset
  1160
        with hooks_control(session, session.HOOKS_DENY_ALL,
8631bb9f6e73 [undo] during undoing, call hooks in the [active]integrity and undo categories
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5070
diff changeset
  1161
                           'integrity', 'activeintegrity', 'undo'):
4943
7f5b83578fec disable security when undoing
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4913
diff changeset
  1162
            with security_enabled(session, read=False):
7f5b83578fec disable security when undoing
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4913
diff changeset
  1163
                for action in reversed(self.tx_actions(session, txuuid, False)):
7f5b83578fec disable security when undoing
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4913
diff changeset
  1164
                    undomethod = getattr(self, '_undo_%s' % action.action.lower())
7f5b83578fec disable security when undoing
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4913
diff changeset
  1165
                    errors += undomethod(session, action)
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1166
        # remove the transactions record
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1167
        self.doexec(session,
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1168
                    "DELETE FROM transactions WHERE tx_uuid='%s'" % txuuid)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1169
        return errors
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1170
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1171
    def start_undoable_transaction(self, session, uuid):
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1172
        """session callback to insert a transaction record in the transactions
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1173
        table when some undoable transaction is started
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1174
        """
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1175
        ueid = session.user.eid
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1176
        attrs = {'tx_uuid': uuid, 'tx_user': ueid, 'tx_time': datetime.now()}
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1177
        self.doexec(session, self.sqlgen.insert('transactions', attrs), attrs)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1178
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1179
    def _save_attrs(self, session, entity, attrs):
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1180
        """return a pickleable dictionary containing current values for given
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1181
        attributes of the entity
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1182
        """
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1183
        restr = {'cw_eid': entity.eid}
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1184
        sql = self.sqlgen.select(SQL_PREFIX + entity.__regid__, restr, attrs)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1185
        cu = self.doexec(session, sql, restr)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1186
        values = dict(zip(attrs, cu.fetchone()))
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1187
        # ensure backend specific binary are converted back to string
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1188
        eschema = entity.e_schema
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1189
        for column in attrs:
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1190
            # [3:] remove 'cw_' prefix
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1191
            attr = column[3:]
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1192
            if not eschema.subjrels[attr].final:
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1193
                continue
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1194
            if eschema.destination(attr) in ('Password', 'Bytes'):
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1195
                value = values[column]
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1196
                if value is not None:
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1197
                    values[column] = self.binary_to_str(value)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1198
        return values
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1199
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1200
    def _record_tx_action(self, session, table, action, **kwargs):
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1201
        """record a transaction action in the given table (either
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1202
        'tx_entity_actions' or 'tx_relation_action')
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1203
        """
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1204
        kwargs['tx_uuid'] = session.transaction_uuid()
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1205
        kwargs['txa_action'] = action
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1206
        kwargs['txa_order'] = session.transaction_inc_action_counter()
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1207
        kwargs['txa_public'] = session.running_dbapi_query
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1208
        self.doexec(session, self.sqlgen.insert(table, kwargs), kwargs)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1209
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1210
    def _tx_info(self, session, txuuid):
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1211
        """return transaction's time and user of the transaction with the given uuid.
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1212
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1213
        raise `NoSuchTransaction` if there is no such transaction of if the
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1214
        session's user isn't allowed to see it.
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1215
        """
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1216
        restr = {'tx_uuid': txuuid}
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1217
        sql = self.sqlgen.select('transactions', restr, ('tx_time', 'tx_user'))
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1218
        cu = self.doexec(session, sql, restr)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1219
        try:
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1220
            time, ueid = cu.fetchone()
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1221
        except TypeError:
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1222
            raise tx.NoSuchTransaction()
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1223
        if not (session.user.is_in_group('managers')
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1224
                or session.user.eid == ueid):
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1225
            raise tx.NoSuchTransaction()
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1226
        return time, ueid
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1227
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1228
    def _undo_d(self, session, action):
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1229
        """undo an entity deletion"""
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1230
        errors = []
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1231
        err = errors.append
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1232
        eid = action.eid
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1233
        etype = action.etype
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1234
        _ = session._
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1235
        # get an entity instance
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1236
        try:
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1237
            entity = self.repo.vreg['etypes'].etype_class(etype)(session)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1238
        except Exception:
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1239
            err("can't restore entity %s of type %s, type no more supported"
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1240
                % (eid, etype))
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1241
            return errors
6142
8bc6eac1fac1 [session] cleanup hook / operation / entity edition api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6128
diff changeset
  1242
        entity.cw_edited = edited = EditedEntity(entity)
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1243
        # check for schema changes, entities linked through inlined relation
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1244
        # still exists, rewrap binary values
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1245
        eschema = entity.e_schema
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1246
        getrschema = eschema.subjrels
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1247
        for column, value in action.changes.items():
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1248
            rtype = column[3:] # remove cw_ prefix
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1249
            try:
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1250
                rschema = getrschema[rtype]
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1251
            except KeyError:
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1252
                err(_("Can't restore relation %(rtype)s of entity %(eid)s, "
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1253
                      "this relation does not exists anymore in the schema.")
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1254
                    % {'rtype': rtype, 'eid': eid})
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1255
            if not rschema.final:
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1256
                assert value is None
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1257
            elif eschema.destination(rtype) in ('Bytes', 'Password'):
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1258
                action.changes[column] = self._binary(value)
6142
8bc6eac1fac1 [session] cleanup hook / operation / entity edition api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6128
diff changeset
  1259
                edited[rtype] = Binary(value)
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1260
            elif isinstance(value, str):
6142
8bc6eac1fac1 [session] cleanup hook / operation / entity edition api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6128
diff changeset
  1261
                edited[rtype] = unicode(value, session.encoding, 'replace')
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1262
            else:
6142
8bc6eac1fac1 [session] cleanup hook / operation / entity edition api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6128
diff changeset
  1263
                edited[rtype] = value
5557
1a534c596bff [entity] continue cleanup of Entity/AnyEntity namespace
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5556
diff changeset
  1264
        entity.eid = eid
5075
a4b735e76c66 [undo] init entity cache when undoing an entity deletion
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5074
diff changeset
  1265
        session.repo.init_entity_caches(session, entity, self)
6142
8bc6eac1fac1 [session] cleanup hook / operation / entity edition api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6128
diff changeset
  1266
        edited.check()
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1267
        self.repo.hm.call_hooks('before_add_entity', session, entity=entity)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1268
        # restore the entity
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1269
        action.changes['cw_eid'] = eid
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1270
        sql = self.sqlgen.insert(SQL_PREFIX + etype, action.changes)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1271
        self.doexec(session, sql, action.changes)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1272
        # restore record in entities (will update fti if needed)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1273
        self.add_info(session, entity, self, None, True)
5059
1d5c81588144 [repo] make etype which should go in deleted_entities configurable: we only need this for types imported from other multi-sources instances
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5013
diff changeset
  1274
        # remove record from deleted_entities if entity's type is multi-sources
1d5c81588144 [repo] make etype which should go in deleted_entities configurable: we only need this for types imported from other multi-sources instances
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5013
diff changeset
  1275
        if entity.__regid__ in self.multisources_etypes:
1d5c81588144 [repo] make etype which should go in deleted_entities configurable: we only need this for types imported from other multi-sources instances
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5013
diff changeset
  1276
            self.doexec(session,
1d5c81588144 [repo] make etype which should go in deleted_entities configurable: we only need this for types imported from other multi-sources instances
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5013
diff changeset
  1277
                        'DELETE FROM deleted_entities WHERE eid=%s' % eid)
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1278
        self.repo.hm.call_hooks('after_add_entity', session, entity=entity)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1279
        return errors
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1280
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1281
    def _undo_r(self, session, action):
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1282
        """undo a relation removal"""
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1283
        errors = []
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1284
        subj, rtype, obj = action.eid_from, action.rtype, action.eid_to
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1285
        try:
5076
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1286
            sentity, oentity, rdef = _undo_rel_info(session, subj, rtype, obj)
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1287
        except UndoException, ex:
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1288
            errors.append(unicode(ex))
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1289
        else:
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1290
            for role, entity in (('subject', sentity),
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1291
                                 ('object', oentity)):
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1292
                try:
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1293
                    _undo_check_relation_target(entity, rdef, role)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1294
                except UndoException, ex:
5076
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1295
                    errors.append(unicode(ex))
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1296
                    continue
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1297
        if not errors:
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1298
            self.repo.hm.call_hooks('before_add_relation', session,
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1299
                                    eidfrom=subj, rtype=rtype, eidto=obj)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1300
            # add relation in the database
7243
9ab01bf84eac fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7237
diff changeset
  1301
            self._add_relations(session, rtype, [(subj, obj)], rdef.rtype.inlined)
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1302
            # set related cache
5076
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1303
            session.update_rel_cache_add(subj, rtype, obj, rdef.rtype.symmetric)
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1304
            self.repo.hm.call_hooks('after_add_relation', session,
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1305
                                    eidfrom=subj, rtype=rtype, eidto=obj)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1306
        return errors
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1307
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1308
    def _undo_c(self, session, action):
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1309
        """undo an entity creation"""
5076
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1310
        eid = action.eid
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1311
        # XXX done to avoid fetching all remaining relation for the entity
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1312
        # we should find an efficient way to do this (keeping current veolidf
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1313
        # massive deletion performance)
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1314
        if _undo_has_later_transaction(session, eid):
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1315
            msg = session._('some later transaction(s) touch entity, undo them '
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1316
                            'first')
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1317
            raise ValidationError(eid, {None: msg})
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1318
        etype = action.etype
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1319
        # get an entity instance
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1320
        try:
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1321
            entity = self.repo.vreg['etypes'].etype_class(etype)(session)
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1322
        except Exception:
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1323
            return [session._(
5098
32b1adfb6b92 [i18n] use named substitution to avoid gettext warnings
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5076
diff changeset
  1324
                "Can't undo creation of entity %(eid)s of type %(etype)s, type "
32b1adfb6b92 [i18n] use named substitution to avoid gettext warnings
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5076
diff changeset
  1325
                "no more supported" % {'eid': eid, 'etype': etype})]
5557
1a534c596bff [entity] continue cleanup of Entity/AnyEntity namespace
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5556
diff changeset
  1326
        entity.eid = eid
5076
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1327
        # for proper eid/type cache update
6426
541659c39f6a [hook/operation] nicer api to achieve same result as set_operation, as described in #1253630
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6366
diff changeset
  1328
        CleanupDeletedEidsCacheOp.get_instance(session).add_data(eid)
5076
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1329
        self.repo.hm.call_hooks('before_delete_entity', session, entity=entity)
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1330
        # remove is / is_instance_of which are added using sql by hooks, hence
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1331
        # unvisible as transaction action
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1332
        self.doexec(session, 'DELETE FROM is_relation WHERE eid_from=%s' % eid)
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1333
        self.doexec(session, 'DELETE FROM is_instance_of_relation WHERE eid_from=%s' % eid)
6427
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6426
diff changeset
  1334
        self.doexec(session, 'DELETE FROM cw_source_relation WHERE eid_from=%s' % self.eid)
5076
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1335
        # XXX check removal of inlined relation?
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1336
        # delete the entity
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1337
        attrs = {'cw_eid': eid}
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1338
        sql = self.sqlgen.delete(SQL_PREFIX + entity.__regid__, attrs)
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1339
        self.doexec(session, sql, attrs)
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1340
        # remove record from entities (will update fti if needed)
7507
4c043afb104a fix failures introduced by recent refactoring
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7501
diff changeset
  1341
        self.delete_info_multi(session, [entity], self.uri)
5076
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1342
        self.repo.hm.call_hooks('after_delete_entity', session, entity=entity)
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1343
        return ()
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1344
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1345
    def _undo_u(self, session, action):
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1346
        """undo an entity update"""
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1347
        return ['undoing of entity updating not yet supported.']
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1348
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1349
    def _undo_a(self, session, action):
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1350
        """undo a relation addition"""
5076
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1351
        errors = []
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1352
        subj, rtype, obj = action.eid_from, action.rtype, action.eid_to
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1353
        try:
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1354
            sentity, oentity, rdef = _undo_rel_info(session, subj, rtype, obj)
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1355
        except UndoException, ex:
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1356
            errors.append(unicode(ex))
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1357
        else:
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1358
            rschema = rdef.rtype
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1359
            if rschema.inlined:
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1360
                sql = 'SELECT 1 FROM cw_%s WHERE cw_eid=%s and cw_%s=%s'\
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1361
                      % (sentity.__regid__, subj, rtype, obj)
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1362
            else:
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1363
                sql = 'SELECT 1 FROM %s_relation WHERE eid_from=%s and eid_to=%s'\
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1364
                      % (rtype, subj, obj)
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1365
            cu = self.doexec(session, sql)
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1366
            if cu.fetchone() is None:
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1367
                errors.append(session._(
5098
32b1adfb6b92 [i18n] use named substitution to avoid gettext warnings
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5076
diff changeset
  1368
                    "Can't undo addition of relation %(rtype)s from %(subj)s to"
32b1adfb6b92 [i18n] use named substitution to avoid gettext warnings
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5076
diff changeset
  1369
                    " %(obj)s, doesn't exist anymore" % locals()))
5076
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1370
        if not errors:
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1371
            self.repo.hm.call_hooks('before_delete_relation', session,
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1372
                                    eidfrom=subj, rtype=rtype, eidto=obj)
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1373
            # delete relation from the database
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1374
            self._delete_relation(session, subj, rtype, obj, rschema.inlined)
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1375
            # set related cache
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1376
            session.update_rel_cache_del(subj, rtype, obj, rschema.symmetric)
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1377
            self.repo.hm.call_hooks('after_delete_relation', session,
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1378
                                    eidfrom=subj, rtype=rtype, eidto=obj)
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1379
        return errors
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1380
4806
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1381
    # full text index handling #################################################
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1382
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1383
    @cached
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1384
    def need_fti_indexation(self, etype):
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1385
        eschema = self.schema.eschema(etype)
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1386
        if any(eschema.indexable_attributes()):
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1387
            return True
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1388
        if any(eschema.fulltext_containers()):
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1389
            return True
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1390
        return False
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1391
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1392
    def index_entity(self, session, entity):
4807
5642bfa43236 [cleanup] add index_entity to abstract source, add docstring
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4806
diff changeset
  1393
        """create an operation to [re]index textual content of the given entity
5642bfa43236 [cleanup] add index_entity to abstract source, add docstring
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4806
diff changeset
  1394
        on commit
5642bfa43236 [cleanup] add index_entity to abstract source, add docstring
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4806
diff changeset
  1395
        """
6426
541659c39f6a [hook/operation] nicer api to achieve same result as set_operation, as described in #1253630
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6366
diff changeset
  1396
        FTIndexEntityOp.get_instance(session).add_data(entity.eid)
4806
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1397
6889
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
  1398
    def fti_unindex_entities(self, session, entities):
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
  1399
        """remove text content for entities from the full text index
4806
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1400
        """
7398
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
  1401
        cursor = session.cnxset['system']
6889
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
  1402
        cursor_unindex_object = self.dbhelper.cursor_unindex_object
4806
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1403
        try:
6889
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
  1404
            for entity in entities:
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
  1405
                cursor_unindex_object(entity.eid, cursor)
4806
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1406
        except Exception: # let KeyboardInterrupt / SystemExit propagate
6889
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
  1407
            self.exception('error while unindexing %s', entity)
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
  1408
4806
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1409
6889
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
  1410
    def fti_index_entities(self, session, entities):
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
  1411
        """add text content of created/modified entities to the full text index
4806
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1412
        """
6889
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
  1413
        cursor_index_object = self.dbhelper.cursor_index_object
7398
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
  1414
        cursor = session.cnxset['system']
4806
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1415
        try:
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1416
            # use cursor_index_object, not cursor_reindex_object since
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1417
            # unindexing done in the FTIndexEntityOp
6889
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
  1418
            for entity in entities:
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
  1419
                cursor_index_object(entity.eid,
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
  1420
                                    entity.cw_adapt_to('IFTIndexable'),
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
  1421
                                    cursor)
4806
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1422
        except Exception: # let KeyboardInterrupt / SystemExit propagate
6889
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
  1423
            self.exception('error while indexing %s', entity)
4806
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1424
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1425
6426
541659c39f6a [hook/operation] nicer api to achieve same result as set_operation, as described in #1253630
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6366
diff changeset
  1426
class FTIndexEntityOp(hook.DataOperationMixIn, hook.LateOperation):
4806
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1427
    """operation to delay entity full text indexation to commit
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1428
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1429
    since fti indexing may trigger discovery of other entities, it should be
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1430
    triggered on precommit, not commit, and this should be done after other
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1431
    precommit operation which may add relations to the entity
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1432
    """
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1433
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1434
    def precommit_event(self):
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1435
        session = self.session
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1436
        source = session.repo.system_source
5070
b1f80ccadda3 [repo] refactor fti operation to use set_operation and a single operation whatever the number of entities to ft index
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5067
diff changeset
  1437
        pendingeids = session.transaction_data.get('pendingeids', ())
b1f80ccadda3 [repo] refactor fti operation to use set_operation and a single operation whatever the number of entities to ft index
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5067
diff changeset
  1438
        done = session.transaction_data.setdefault('indexedeids', set())
6889
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
  1439
        to_reindex = set()
6426
541659c39f6a [hook/operation] nicer api to achieve same result as set_operation, as described in #1253630
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6366
diff changeset
  1440
        for eid in self.get_data():
5070
b1f80ccadda3 [repo] refactor fti operation to use set_operation and a single operation whatever the number of entities to ft index
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5067
diff changeset
  1441
            if eid in pendingeids or eid in done:
b1f80ccadda3 [repo] refactor fti operation to use set_operation and a single operation whatever the number of entities to ft index
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5067
diff changeset
  1442
                # entity added and deleted in the same transaction or already
b1f80ccadda3 [repo] refactor fti operation to use set_operation and a single operation whatever the number of entities to ft index
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5067
diff changeset
  1443
                # processed
6889
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
  1444
                continue
5070
b1f80ccadda3 [repo] refactor fti operation to use set_operation and a single operation whatever the number of entities to ft index
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5067
diff changeset
  1445
            done.add(eid)
5556
9ab2b4c74baf [entity] introduce a new 'adapters' registry
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5542
diff changeset
  1446
            iftindexable = session.entity_from_eid(eid).cw_adapt_to('IFTIndexable')
6889
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
  1447
            to_reindex |= set(iftindexable.fti_containers())
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
  1448
        source.fti_unindex_entities(session, to_reindex)
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
  1449
        source.fti_index_entities(session, to_reindex)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1450
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1451
def sql_schema(driver):
4831
c5aec27c1bf7 [repo] use logilab.db instead of lgc.adbh/lgc.db/lgc.sqlgen/indexer, test new date extranction functions
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4829
diff changeset
  1452
    helper = get_db_helper(driver)
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1453
    typemap = helper.TYPE_MAPPING
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1454
    schema = """
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1455
/* Create the repository's system database */
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1456
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1457
%s
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1458
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1459
CREATE TABLE entities (
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1460
  eid INTEGER PRIMARY KEY NOT NULL,
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1461
  type VARCHAR(64) NOT NULL,
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1462
  source VARCHAR(64) NOT NULL,
7543
570522300e22 [ms, entity metas] add 'actual source' to entities table / base entity metadata cache. Closes #1767090
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7536
diff changeset
  1463
  asource VARCHAR(64) NOT NULL,
4113
986fc01be83c TIMESTAMP column type has a special meaning for SQLServer,
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 3958
diff changeset
  1464
  mtime %s NOT NULL,
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1465
  extid VARCHAR(256)
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1466
);;
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1467
CREATE INDEX entities_type_idx ON entities(type);;
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1468
CREATE INDEX entities_mtime_idx ON entities(mtime);;
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1469
CREATE INDEX entities_extid_idx ON entities(extid);;
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1470
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1471
CREATE TABLE deleted_entities (
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1472
  eid INTEGER PRIMARY KEY NOT NULL,
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1473
  type VARCHAR(64) NOT NULL,
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1474
  source VARCHAR(64) NOT NULL,
4113
986fc01be83c TIMESTAMP column type has a special meaning for SQLServer,
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 3958
diff changeset
  1475
  dtime %s NOT NULL,
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1476
  extid VARCHAR(256)
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1477
);;
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1478
CREATE INDEX deleted_entities_type_idx ON deleted_entities(type);;
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1479
CREATE INDEX deleted_entities_dtime_idx ON deleted_entities(dtime);;
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1480
CREATE INDEX deleted_entities_extid_idx ON deleted_entities(extid);;
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1481
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1482
CREATE TABLE transactions (
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1483
  tx_uuid CHAR(32) PRIMARY KEY NOT NULL,
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1484
  tx_user INTEGER NOT NULL,
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1485
  tx_time %s NOT NULL
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1486
);;
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1487
CREATE INDEX transactions_tx_user_idx ON transactions(tx_user);;
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1488
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1489
CREATE TABLE tx_entity_actions (
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1490
  tx_uuid CHAR(32) REFERENCES transactions(tx_uuid) ON DELETE CASCADE,
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1491
  txa_action CHAR(1) NOT NULL,
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1492
  txa_public %s NOT NULL,
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1493
  txa_order INTEGER,
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1494
  eid INTEGER NOT NULL,
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1495
  etype VARCHAR(64) NOT NULL,
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1496
  changes %s
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1497
);;
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1498
CREATE INDEX tx_entity_actions_txa_action_idx ON tx_entity_actions(txa_action);;
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1499
CREATE INDEX tx_entity_actions_txa_public_idx ON tx_entity_actions(txa_public);;
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1500
CREATE INDEX tx_entity_actions_eid_idx ON tx_entity_actions(eid);;
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1501
CREATE INDEX tx_entity_actions_etype_idx ON tx_entity_actions(etype);;
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1502
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1503
CREATE TABLE tx_relation_actions (
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1504
  tx_uuid CHAR(32) REFERENCES transactions(tx_uuid) ON DELETE CASCADE,
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1505
  txa_action CHAR(1) NOT NULL,
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1506
  txa_public %s NOT NULL,
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1507
  txa_order INTEGER,
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1508
  eid_from INTEGER NOT NULL,
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1509
  eid_to INTEGER NOT NULL,
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1510
  rtype VARCHAR(256) NOT NULL
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1511
);;
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1512
CREATE INDEX tx_relation_actions_txa_action_idx ON tx_relation_actions(txa_action);;
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1513
CREATE INDEX tx_relation_actions_txa_public_idx ON tx_relation_actions(txa_public);;
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1514
CREATE INDEX tx_relation_actions_eid_from_idx ON tx_relation_actions(eid_from);;
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1515
CREATE INDEX tx_relation_actions_eid_to_idx ON tx_relation_actions(eid_to);;
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1516
""" % (helper.sql_create_sequence('entities_id_seq').replace(';', ';;'),
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1517
       typemap['Datetime'], typemap['Datetime'], typemap['Datetime'],
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1518
       typemap['Boolean'], typemap['Bytes'], typemap['Boolean'])
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1519
    if helper.backend_name == 'sqlite':
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1520
        # sqlite support the ON DELETE CASCADE syntax but do nothing
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1521
        schema += '''
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1522
CREATE TRIGGER fkd_transactions
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1523
BEFORE DELETE ON transactions
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1524
FOR EACH ROW BEGIN
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1525
    DELETE FROM tx_entity_actions WHERE tx_uuid=OLD.tx_uuid;
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1526
    DELETE FROM tx_relation_actions WHERE tx_uuid=OLD.tx_uuid;
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1527
END;;
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1528
'''
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1529
    return schema
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1530
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1531
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1532
def sql_drop_schema(driver):
4831
c5aec27c1bf7 [repo] use logilab.db instead of lgc.adbh/lgc.db/lgc.sqlgen/indexer, test new date extranction functions
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4829
diff changeset
  1533
    helper = get_db_helper(driver)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1534
    return """
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1535
%s
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1536
DROP TABLE entities;
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1537
DROP TABLE deleted_entities;
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1538
DROP TABLE tx_entity_actions;
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1539
DROP TABLE tx_relation_actions;
5214
3285b6e3b930 fix cwctl db-init -d on SQL Server
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5168
diff changeset
  1540
DROP TABLE transactions;
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1541
""" % helper.sql_drop_sequence('entities_id_seq')
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1542
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1543
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1544
def grant_schema(user, set_owner=True):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1545
    result = ''
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1546
    for table in ('entities', 'deleted_entities', 'entities_id_seq',
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1547
                  'transactions', 'tx_entity_actions', 'tx_relation_actions'):
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1548
        if set_owner:
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1549
            result = 'ALTER TABLE %s OWNER TO %s;\n' % (table, user)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1550
        result += 'GRANT ALL ON %s TO %s;\n' % (table, user)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1551
    return result
3647
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1552
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1553
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1554
class BaseAuthentifier(object):
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1555
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1556
    def __init__(self, source=None):
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1557
        self.source = source
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1558
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1559
    def set_schema(self, schema):
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1560
        """set the instance'schema"""
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1561
        pass
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1562
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1563
class LoginPasswordAuthentifier(BaseAuthentifier):
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1564
    passwd_rql = "Any P WHERE X is CWUser, X login %(login)s, X upassword P"
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1565
    auth_rql = "Any X WHERE X is CWUser, X login %(login)s, X upassword %(pwd)s"
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1566
    _sols = ({'X': 'CWUser', 'P': 'Password'},)
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1567
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1568
    def set_schema(self, schema):
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1569
        """set the instance'schema"""
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1570
        if 'CWUser' in schema: # probably an empty schema if not true...
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1571
            # rql syntax trees used to authenticate users
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1572
            self._passwd_rqlst = self.source.compile_rql(self.passwd_rql, self._sols)
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1573
            self._auth_rqlst = self.source.compile_rql(self.auth_rql, self._sols)
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1574
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1575
    def authenticate(self, session, login, password=None, **kwargs):
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1576
        """return CWUser eid for the given login/password if this account is
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1577
        defined in this source, else raise `AuthenticationError`
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1578
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1579
        two queries are needed since passwords are stored crypted, so we have
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1580
        to fetch the salt first
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1581
        """
6018
f4d1d5d9ccbb [security] don't put uncrypted password in query parameters, else it may be logged on error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5975
diff changeset
  1582
        args = {'login': login, 'pwd' : None}
3647
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1583
        if password is not None:
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1584
            rset = self.source.syntax_tree_search(session, self._passwd_rqlst, args)
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1585
            try:
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1586
                pwd = rset[0][0]
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1587
            except IndexError:
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1588
                raise AuthenticationError('bad login')
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1589
            # passwords are stored using the Bytes type, so we get a StringIO
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1590
            if pwd is not None:
4204
60256056bda6 backport stable branch
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4181 4198
diff changeset
  1591
                args['pwd'] = Binary(crypt_password(password, pwd.getvalue()[:2]))
3647
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1592
        # get eid from login and (crypted) password
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1593
        rset = self.source.syntax_tree_search(session, self._auth_rqlst, args)
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1594
        try:
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1595
            return rset[0][0]
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1596
        except IndexError:
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1597
            raise AuthenticationError('bad password')
6849
5a0c2cfc19bf [repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
  1598
5a0c2cfc19bf [repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
  1599
5a0c2cfc19bf [repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
  1600
class EmailPasswordAuthentifier(BaseAuthentifier):
5a0c2cfc19bf [repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
  1601
    def authenticate(self, session, login, **authinfo):
5a0c2cfc19bf [repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
  1602
        # email_auth flag prevent from infinite recursion (call to
5a0c2cfc19bf [repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
  1603
        # repo.check_auth_info at the end of this method may lead us here again)
5a0c2cfc19bf [repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
  1604
        if not '@' in login or authinfo.pop('email_auth', None):
5a0c2cfc19bf [repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
  1605
            raise AuthenticationError('not an email')
5a0c2cfc19bf [repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
  1606
        rset = session.execute('Any L WHERE U login L, U primary_email M, '
5a0c2cfc19bf [repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
  1607
                               'M address %(login)s', {'login': login},
5a0c2cfc19bf [repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
  1608
                               build_descr=False)
5a0c2cfc19bf [repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
  1609
        if rset.rowcount != 1:
5a0c2cfc19bf [repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
  1610
            raise AuthenticationError('unexisting email')
5a0c2cfc19bf [repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
  1611
        login = rset.rows[0][0]
5a0c2cfc19bf [repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
  1612
        authinfo['email_auth'] = True
5a0c2cfc19bf [repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
  1613
        return self.source.repo.check_auth_info(session, login, authinfo)
7342
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1614
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1615
class DatabaseIndependentBackupRestore(object):
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1616
    """Helper class to perform db backend agnostic backup and restore
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1617
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1618
    The backup and restore methods are used to dump / restore the
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1619
    system database in a database independent format. The file is a
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1620
    Zip archive containing the following files:
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1621
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1622
    * format.txt: the format of the archive. Currently '1.0'
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1623
    * tables.txt: list of filenames in the archive tables/ directory
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1624
    * sequences.txt: list of filenames in the archive sequences/ directory
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1625
    * versions.txt: the list of cube versions from CWProperty
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1626
    * tables/<tablename>.<chunkno>: pickled data
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1627
    * sequences/<sequencename>: pickled data
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1628
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1629
    The pickled data format for tables and sequences is a tuple of 3 elements:
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1630
    * the table name
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1631
    * a tuple of column names
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1632
    * a list of rows (as tuples with one element per column)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1633
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1634
    Tables are saved in chunks in different files in order to prevent
7551
2d4ba5b984dc cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7530
diff changeset
  1635
    a too high memory consumption.
7342
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1636
    """
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1637
    def __init__(self, source):
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1638
        """
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1639
        :param: source an instance of the system source
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1640
        """
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1641
        self._source = source
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1642
        self.logger = logging.getLogger('cubicweb.ctl')
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1643
        self.logger.setLevel(logging.INFO)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1644
        self.logger.addHandler(logging.StreamHandler(sys.stdout))
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1645
        self.schema = self._source.schema
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1646
        self.dbhelper = self._source.dbhelper
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1647
        self.cnx = None
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1648
        self.cursor = None
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1649
        self.sql_generator = sqlgen.SQLGenerator()
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1650
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1651
    def get_connection(self):
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1652
        return self._source.get_connection()
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1653
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1654
    def backup(self, backupfile):
7787
d9607ae447d2 [server] portable dump format supports now ZIP64 extensions by default (closes #1912535)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7786
diff changeset
  1655
        archive=zipfile.ZipFile(backupfile, 'w', allowZip64=True)
7342
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1656
        self.cnx = self.get_connection()
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1657
        try:
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1658
            self.cursor = self.cnx.cursor()
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1659
            self.cursor.arraysize=100
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1660
            self.logger.info('writing metadata')
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1661
            self.write_metadata(archive)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1662
            for seq in self.get_sequences():
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1663
                self.logger.info('processing sequence %s', seq)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1664
                self.write_sequence(archive, seq)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1665
            for table in self.get_tables():
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1666
                self.logger.info('processing table %s', table)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1667
                self.write_table(archive, table)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1668
        finally:
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1669
            archive.close()
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1670
            self.cnx.close()
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1671
        self.logger.info('done')
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1672
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1673
    def get_tables(self):
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1674
        non_entity_tables = ['entities',
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1675
                             'deleted_entities',
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1676
                             'transactions',
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1677
                             'tx_entity_actions',
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1678
                             'tx_relation_actions',
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1679
                             ]
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1680
        etype_tables = []
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1681
        relation_tables = []
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1682
        prefix = 'cw_'
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1683
        for etype in self.schema.entities():
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1684
            eschema = self.schema.eschema(etype)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1685
            print etype, eschema.final
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1686
            if eschema.final:
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1687
                continue
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1688
            etype_tables.append('%s%s'%(prefix, etype))
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1689
        for rtype in self.schema.relations():
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1690
            rschema = self.schema.rschema(rtype)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1691
            if rschema.final or rschema.inlined:
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1692
                continue
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1693
            relation_tables.append('%s_relation' % rtype)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1694
        return non_entity_tables + etype_tables + relation_tables
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1695
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1696
    def get_sequences(self):
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1697
        return ['entities_id_seq']
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1698
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1699
    def write_metadata(self, archive):
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1700
        archive.writestr('format.txt', '1.0')
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1701
        archive.writestr('tables.txt', '\n'.join(self.get_tables()))
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1702
        archive.writestr('sequences.txt', '\n'.join(self.get_sequences()))
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1703
        versions = self._get_versions()
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1704
        versions_str = '\n'.join('%s %s' % (k,v)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1705
                                 for k,v in versions)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1706
        archive.writestr('versions.txt', versions_str)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1707
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1708
    def write_sequence(self, archive, seq):
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1709
        sql = self.dbhelper.sql_sequence_current_state(seq)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1710
        columns, rows_iterator = self._get_cols_and_rows(sql)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1711
        rows = list(rows_iterator)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1712
        serialized = self._serialize(seq, columns, rows)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1713
        archive.writestr('sequences/%s' % seq, serialized)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1714
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1715
    def write_table(self, archive, table):
7753
681ef2a664dd work around cursor.rowcount not returning anything useful before 1st fetch with mssql (closes #1910869)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7551
diff changeset
  1716
        nb_lines_sql = 'SELECT COUNT(*) FROM %s' % table
681ef2a664dd work around cursor.rowcount not returning anything useful before 1st fetch with mssql (closes #1910869)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7551
diff changeset
  1717
        self.cursor.execute(nb_lines_sql)
681ef2a664dd work around cursor.rowcount not returning anything useful before 1st fetch with mssql (closes #1910869)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7551
diff changeset
  1718
        rowcount = self.cursor.fetchone()[0]
7342
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1719
        sql = 'SELECT * FROM %s' % table
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1720
        columns, rows_iterator = self._get_cols_and_rows(sql)
7753
681ef2a664dd work around cursor.rowcount not returning anything useful before 1st fetch with mssql (closes #1910869)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7551
diff changeset
  1721
        self.logger.info('number of rows: %d', rowcount)
7342
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1722
        if table.startswith('cw_'): # entities
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1723
            blocksize = 2000
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1724
        else: # relations and metadata
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1725
            blocksize = 10000
7753
681ef2a664dd work around cursor.rowcount not returning anything useful before 1st fetch with mssql (closes #1910869)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7551
diff changeset
  1726
        if rowcount > 0:
681ef2a664dd work around cursor.rowcount not returning anything useful before 1st fetch with mssql (closes #1910869)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7551
diff changeset
  1727
            for i, start in enumerate(xrange(0, rowcount, blocksize)):
7342
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1728
                rows = list(itertools.islice(rows_iterator, blocksize))
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1729
                serialized = self._serialize(table, columns, rows)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1730
                archive.writestr('tables/%s.%04d' % (table, i), serialized)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1731
                self.logger.debug('wrote rows %d to %d (out of %d) to %s.%04d',
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1732
                                  start, start+len(rows)-1,
7753
681ef2a664dd work around cursor.rowcount not returning anything useful before 1st fetch with mssql (closes #1910869)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7551
diff changeset
  1733
                                  rowcount,
7342
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1734
                                  table, i)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1735
        else:
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1736
            rows = []
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1737
            serialized = self._serialize(table, columns, rows)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1738
            archive.writestr('tables/%s.%04d' % (table, 0), serialized)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1739
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1740
    def _get_cols_and_rows(self, sql):
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1741
        process_result = self._source.iter_process_result
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1742
        self.cursor.execute(sql)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1743
        columns = (d[0] for d in self.cursor.description)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1744
        rows = process_result(self.cursor)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1745
        return tuple(columns), rows
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1746
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1747
    def _serialize(self, name, columns, rows):
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1748
        return dumps((name, columns, rows), pickle.HIGHEST_PROTOCOL)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1749
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1750
    def restore(self, backupfile):
7787
d9607ae447d2 [server] portable dump format supports now ZIP64 extensions by default (closes #1912535)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7786
diff changeset
  1751
        archive = zipfile.ZipFile(backupfile, 'r', allowZip64=True)
7342
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1752
        self.cnx = self.get_connection()
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1753
        self.cursor = self.cnx.cursor()
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1754
        sequences, tables, table_chunks = self.read_metadata(archive, backupfile)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1755
        for seq in sequences:
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1756
            self.logger.info('restoring sequence %s', seq)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1757
            self.read_sequence(archive, seq)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1758
        for table in tables:
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1759
            self.logger.info('restoring table %s', table)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1760
            self.read_table(archive, table, sorted(table_chunks[table]))
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1761
        self.cnx.close()
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1762
        archive.close()
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1763
        self.logger.info('done')
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1764
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1765
    def read_metadata(self, archive, backupfile):
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1766
        formatinfo = archive.read('format.txt')
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1767
        self.logger.info('checking metadata')
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1768
        if formatinfo.strip() != "1.0":
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1769
            self.logger.critical('Unsupported format in archive: %s', formatinfo)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1770
            raise ValueError('Unknown format in %s: %s' % (backupfile, formatinfo))
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1771
        tables = archive.read('tables.txt').splitlines()
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1772
        sequences = archive.read('sequences.txt').splitlines()
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1773
        file_versions = self._parse_versions(archive.read('versions.txt'))
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1774
        versions = set(self._get_versions())
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1775
        if file_versions != versions:
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1776
            self.logger.critical('Unable to restore : versions do not match')
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1777
            self.logger.critical('Expected:\n%s', '\n'.join(list(sorted(versions))))
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1778
            self.logger.critical('Found:\n%s', '\n'.join(list(sorted(file_versions))))
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1779
            raise ValueError('Unable to restore : versions do not match')
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1780
        table_chunks = {}
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1781
        for name in archive.namelist():
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1782
            if not name.startswith('tables/'):
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1783
                continue
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1784
            filename = basename(name)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1785
            tablename, _ext = filename.rsplit('.', 1)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1786
            table_chunks.setdefault(tablename, []).append(name)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1787
        return sequences, tables, table_chunks
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1788
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1789
    def read_sequence(self, archive, seq):
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1790
        seqname, columns, rows = loads(archive.read('sequences/%s' % seq))
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1791
        assert seqname == seq
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1792
        assert len(rows) == 1
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1793
        assert len(rows[0]) == 1
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1794
        value = rows[0][0]
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1795
        sql = self.dbhelper.sql_restart_sequence(seq, value)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1796
        self.cursor.execute(sql)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1797
        self.cnx.commit()
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1798
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1799
    def read_table(self, archive, table, filenames):
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1800
        merge_args = self._source.merge_args
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1801
        self.cursor.execute('DELETE FROM %s' % table)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1802
        self.cnx.commit()
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1803
        row_count = 0
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1804
        for filename in filenames:
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1805
            tablename, columns, rows = loads(archive.read(filename))
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1806
            assert tablename == table
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1807
            if not rows:
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1808
                continue
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1809
            insert = self.sql_generator.insert(table,
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1810
                                               dict(zip(columns, rows[0])))
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1811
            for row in rows:
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1812
                self.cursor.execute(insert, merge_args(dict(zip(columns, row)), {}))
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1813
            row_count += len(rows)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1814
            self.cnx.commit()
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1815
        self.logger.info('inserted %d rows', row_count)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1816
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1817
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1818
    def _parse_versions(self, version_str):
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1819
        versions = set()
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1820
        for line in version_str.splitlines():
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1821
            versions.add(tuple(line.split()))
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1822
        return versions
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1823
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1824
    def _get_versions(self):
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1825
        version_sql = 'SELECT cw_pkey, cw_value FROM cw_CWProperty'
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1826
        versions = []
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1827
        self.cursor.execute(version_sql)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1828
        for pkey, value in self.cursor.fetchall():
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1829
            if pkey.startswith(u'system.version'):
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1830
                versions.append((pkey, value))
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1831
        return versions