server/sources/native.py
author David Douard <david.douard@logilab.fr>
Wed, 26 Nov 2014 17:58:23 +0100
changeset 10061 ecbfec2da8a2
parent 10037 6b83ceda8323
parent 10053 079b5279a2cd
child 10074 ab956b780d4e
child 10076 3810332ef42c
permissions -rw-r--r--
[merge] backport 3.18 fixes
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
9583
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
     1
# copyright 2003-2014 LOGILAB S.A. (Paris, FRANCE), all rights reserved.
5421
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5413
diff changeset
     2
# contact http://www.logilab.fr/ -- mailto:contact@logilab.fr
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5413
diff changeset
     3
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5413
diff changeset
     4
# This file is part of CubicWeb.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5413
diff changeset
     5
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5413
diff changeset
     6
# CubicWeb is free software: you can redistribute it and/or modify it under the
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5413
diff changeset
     7
# terms of the GNU Lesser General Public License as published by the Free
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5413
diff changeset
     8
# Software Foundation, either version 2.1 of the License, or (at your option)
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5413
diff changeset
     9
# any later version.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5413
diff changeset
    10
#
5424
8ecbcbff9777 replace logilab-common by CubicWeb in disclaimer
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5421
diff changeset
    11
# CubicWeb is distributed in the hope that it will be useful, but WITHOUT
5421
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5413
diff changeset
    12
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5413
diff changeset
    13
# FOR A PARTICULAR PURPOSE.  See the GNU Lesser General Public License for more
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5413
diff changeset
    14
# details.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5413
diff changeset
    15
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5413
diff changeset
    16
# You should have received a copy of the GNU Lesser General Public License along
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5413
diff changeset
    17
# with CubicWeb.  If not, see <http://www.gnu.org/licenses/>.
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    18
"""Adapters for native cubicweb sources.
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    19
1952
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1792
diff changeset
    20
Notes:
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1792
diff changeset
    21
* extid (aka external id, the primary key of an entity in the external source
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1792
diff changeset
    22
  from which it comes from) are stored in a varchar column encoded as a base64
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1792
diff changeset
    23
  string. This is because it should actually be Bytes but we want an index on
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1792
diff changeset
    24
  it for fast querying.
5824
de9b7e88660e cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5751
diff changeset
    25
"""
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    26
__docformat__ = "restructuredtext en"
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    27
9932
c493767679b2 [source/native] cPickle is available in all supported pythons
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9802
diff changeset
    28
from cPickle import loads, dumps
c493767679b2 [source/native] cPickle is available in all supported pythons
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9802
diff changeset
    29
import cPickle as pickle
4900
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4893
diff changeset
    30
from threading import Lock
1016
26387b836099 use datetime instead of mx.DateTime
sylvain.thenault@logilab.fr
parents: 973
diff changeset
    31
from datetime import datetime
1952
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1792
diff changeset
    32
from base64 import b64decode, b64encode
4964
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
    33
from contextlib import contextmanager
9468
39b7a91a3f4c [repo] pylint cleanup, mainly of imports, with a bit of style
Julien Cristau <julien.cristau@logilab.fr>
parents: 9467
diff changeset
    34
from os.path import basename
6211
e9d125fd1465 nicer error reporting for unique together constraints
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6201
diff changeset
    35
import re
6889
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
    36
import itertools
7342
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
    37
import zipfile
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
    38
import logging
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
    39
import sys
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    40
4806
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
    41
from logilab.common.decorators import cached, clear_cache
3835
a191b3b9e455 more sensible default values to c-c "create" inputs
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3689
diff changeset
    42
from logilab.common.configuration import Method
a191b3b9e455 more sensible default values to c-c "create" inputs
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3689
diff changeset
    43
from logilab.common.shellutils import getlogin
7342
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
    44
from logilab.database import get_db_helper, sqlgen
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    45
5891
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
    46
from yams import schema2sql as y2sql
6945
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
    47
from yams.schema import role_name
5891
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
    48
6945
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
    49
from cubicweb import (UnknownEid, AuthenticationError, ValidationError, Binary,
9468
39b7a91a3f4c [repo] pylint cleanup, mainly of imports, with a bit of style
Julien Cristau <julien.cristau@logilab.fr>
parents: 9467
diff changeset
    50
                      UniqueTogetherError, UndoTransactionException)
5076
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
    51
from cubicweb import transaction as tx, server, neg_role
7954
a3d3220669d6 [cache] replace lgc.cache with something more appropriate (closes #1921713)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 7922
diff changeset
    52
from cubicweb.utils import QueryCache
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
    53
from cubicweb.schema import VIRTUAL_RTYPES
3835
a191b3b9e455 more sensible default values to c-c "create" inputs
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3689
diff changeset
    54
from cubicweb.cwconfig import CubicWebNoAppConfiguration
4806
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
    55
from cubicweb.server import hook
8546
3d2038d6f20d [sources/native] automatically update passwords using deprecated hashes on login
Julien Cristau <julien.cristau@logilab.fr>
parents: 8520
diff changeset
    56
from cubicweb.server.utils import crypt_password, eschema_eid, verify_and_update
2759
23d7a75693f8 R refactor backup and use tar.gz to store all sources
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 2638
diff changeset
    57
from cubicweb.server.sqlutils import SQL_PREFIX, SQLAdapterMixIn
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    58
from cubicweb.server.rqlannotation import set_qdata
5076
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
    59
from cubicweb.server.hook import CleanupDeletedEidsCacheOp
7118
e094b3d4eb95 [server] move EditedEntity class to its own module, to avoid cyclic dependency when needed from e.g. session.py
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6957
diff changeset
    60
from cubicweb.server.edition import EditedEntity
2625
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2620
diff changeset
    61
from cubicweb.server.sources import AbstractSource, dbg_st_search, dbg_results
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    62
from cubicweb.server.sources.rql2sql import SQLGenerator
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    63
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    64
2354
9b4bac626977 ability to map attributes to something else than usual cw mapping on sql generation
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2310
diff changeset
    65
ATTR_MAP = {}
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    66
NONSYSTEM_ETYPES = set()
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    67
NONSYSTEM_RELATIONS = set()
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    68
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    69
class LogCursor(object):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    70
    def __init__(self, cursor):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    71
        self.cu = cursor
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
    72
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    73
    def execute(self, query, args=None):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    74
        """Execute a query.
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    75
        it's a function just so that it shows up in profiling
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    76
        """
2593
16d9419a4a79 F: start to handle binary debug log level on the server side
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2589
diff changeset
    77
        if server.DEBUG & server.DBG_SQL:
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    78
            print 'exec', query, args
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    79
        try:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    80
            self.cu.execute(str(query), args)
8695
358d8bed9626 [toward-py3k] rewrite to "except AnException as exc:" (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8694
diff changeset
    81
        except Exception as ex:
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    82
            print "sql: %r\n args: %s\ndbms message: %r" % (
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    83
                query, args, ex.args[0])
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    84
            raise
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
    85
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    86
    def fetchall(self):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    87
        return self.cu.fetchall()
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
    88
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    89
    def fetchone(self):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    90
        return self.cu.fetchone()
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
    91
2625
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2620
diff changeset
    92
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
    93
def sql_or_clauses(sql, clauses):
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
    94
    select, restr = sql.split(' WHERE ', 1)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
    95
    restrclauses = restr.split(' AND ')
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
    96
    for clause in clauses:
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
    97
        restrclauses.remove(clause)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
    98
    if restrclauses:
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
    99
        restr = '%s AND (%s)' % (' AND '.join(restrclauses),
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   100
                                 ' OR '.join(clauses))
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   101
    else:
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   102
        restr = '(%s)' % ' OR '.join(clauses)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   103
    return '%s WHERE %s' % (select, restr)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   104
9450
af4b93bc38a5 [multi-sources-removal] Drop deleted_entities system table and entities.mtime column
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9448
diff changeset
   105
5891
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   106
def rdef_table_column(rdef):
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   107
    """return table and column used to store the given relation definition in
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   108
    the database
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   109
    """
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   110
    return (SQL_PREFIX + str(rdef.subject),
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   111
            SQL_PREFIX + str(rdef.rtype))
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   112
9450
af4b93bc38a5 [multi-sources-removal] Drop deleted_entities system table and entities.mtime column
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9448
diff changeset
   113
5891
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   114
def rdef_physical_info(dbhelper, rdef):
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   115
    """return backend type and a boolean flag if NULL values should be allowed
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   116
    for a given relation definition
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   117
    """
6359
0bff5a05385c [sync schema] take care rdef may not be final, in which case we want to use type of eid attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6297
diff changeset
   118
    if rdef.object.final:
0bff5a05385c [sync schema] take care rdef may not be final, in which case we want to use type of eid attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6297
diff changeset
   119
        ttype = rdef.object
0bff5a05385c [sync schema] take care rdef may not be final, in which case we want to use type of eid attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6297
diff changeset
   120
    else:
0bff5a05385c [sync schema] take care rdef may not be final, in which case we want to use type of eid attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6297
diff changeset
   121
        ttype = 'Int' # eid type
0bff5a05385c [sync schema] take care rdef may not be final, in which case we want to use type of eid attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6297
diff changeset
   122
    coltype = y2sql.type_from_constraints(dbhelper, ttype,
5891
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   123
                                          rdef.constraints, creating=False)
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   124
    allownull = rdef.cardinality[0] != '1'
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   125
    return coltype, allownull
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   126
8265
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
   127
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
   128
class _UndoException(Exception):
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   129
    """something went wrong during undoing"""
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   130
7530
15178bf89fb6 [server] fix unicode conversion capability in UndoException
Julien Jehannet <julien.jehannet@logilab.fr>
parents: 7507
diff changeset
   131
    def __unicode__(self):
15178bf89fb6 [server] fix unicode conversion capability in UndoException
Julien Jehannet <julien.jehannet@logilab.fr>
parents: 7507
diff changeset
   132
        """Called by the unicode builtin; should return a Unicode object
15178bf89fb6 [server] fix unicode conversion capability in UndoException
Julien Jehannet <julien.jehannet@logilab.fr>
parents: 7507
diff changeset
   133
8265
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
   134
        Type of _UndoException message must be `unicode` by design in CubicWeb.
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
   135
        """
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
   136
        assert isinstance(self.args[0], unicode)
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
   137
        return self.args[0]
7530
15178bf89fb6 [server] fix unicode conversion capability in UndoException
Julien Jehannet <julien.jehannet@logilab.fr>
parents: 7507
diff changeset
   138
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   139
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   140
def _undo_check_relation_target(tentity, rdef, role):
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   141
    """check linked entity has not been redirected for this relation"""
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   142
    card = rdef.role_cardinality(role)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   143
    if card in '?1' and tentity.related(rdef.rtype, role):
8265
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
   144
        raise _UndoException(tentity._cw._(
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   145
            "Can't restore %(role)s relation %(rtype)s to entity %(eid)s which "
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   146
            "is already linked using this relation.")
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   147
                            % {'role': neg_role(role),
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   148
                               'rtype': rdef.rtype,
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   149
                               'eid': tentity.eid})
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   150
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   151
def _undo_rel_info(cnx, subj, rtype, obj):
5076
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   152
    entities = []
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   153
    for role, eid in (('subject', subj), ('object', obj)):
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   154
        try:
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   155
            entities.append(cnx.entity_from_eid(eid))
5076
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   156
        except UnknownEid:
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   157
            raise _UndoException(cnx._(
5076
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   158
                "Can't restore relation %(rtype)s, %(role)s entity %(eid)s"
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   159
                " doesn't exist anymore.")
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   160
                                % {'role': cnx._(role),
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   161
                                   'rtype': cnx._(rtype),
5076
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   162
                                   'eid': eid})
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   163
    sentity, oentity = entities
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   164
    try:
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   165
        rschema = cnx.vreg.schema.rschema(rtype)
8900
010a59e12d89 use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8821
diff changeset
   166
        rdef = rschema.rdefs[(sentity.cw_etype, oentity.cw_etype)]
5076
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   167
    except KeyError:
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   168
        raise _UndoException(cnx._(
5076
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   169
            "Can't restore relation %(rtype)s between %(subj)s and "
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   170
            "%(obj)s, that relation does not exists anymore in the "
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   171
            "schema.")
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   172
                            % {'rtype': cnx._(rtype),
5076
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   173
                               'subj': subj,
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   174
                               'obj': obj})
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   175
    return sentity, oentity, rdef
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   176
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   177
def _undo_has_later_transaction(cnx, eid):
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   178
    return cnx.system_sql('''\
5076
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   179
SELECT T.tx_uuid FROM transactions AS TREF, transactions AS T
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   180
WHERE TREF.tx_uuid='%(txuuid)s' AND T.tx_uuid!='%(txuuid)s'
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   181
AND T.tx_time>=TREF.tx_time
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   182
AND (EXISTS(SELECT 1 FROM tx_entity_actions AS TEA
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   183
            WHERE TEA.tx_uuid=T.tx_uuid AND TEA.eid=%(eid)s)
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   184
     OR EXISTS(SELECT 1 FROM tx_relation_actions as TRA
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   185
               WHERE TRA.tx_uuid=T.tx_uuid AND (
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   186
                   TRA.eid_from=%(eid)s OR TRA.eid_to=%(eid)s))
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   187
     )''' % {'txuuid': cnx.transaction_data['undoing_uuid'],
5076
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   188
             'eid': eid}).fetchone()
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   189
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   190
9583
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   191
class DefaultEidGenerator(object):
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   192
    __slots__ = ('source', 'cnx', 'lock')
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   193
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   194
    def __init__(self, source):
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   195
        self.source = source
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   196
        self.cnx = None
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   197
        self.lock = Lock()
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   198
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   199
    def close(self):
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   200
        if self.cnx:
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   201
            self.cnx.close()
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   202
        self.cnx = None
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   203
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   204
    def create_eid(self, _cnx, count=1):
9583
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   205
        # lock needed to prevent 'Connection is busy with results for another
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   206
        # command (0)' errors with SQLServer
9585
3f5b59527d31 [source/native] allow many eid creation per .create_eid call (closes #3526594)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9583
diff changeset
   207
        assert count > 0
9583
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   208
        with self.lock:
9585
3f5b59527d31 [source/native] allow many eid creation per .create_eid call (closes #3526594)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9583
diff changeset
   209
            return self._create_eid(count)
9583
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   210
9585
3f5b59527d31 [source/native] allow many eid creation per .create_eid call (closes #3526594)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9583
diff changeset
   211
    def _create_eid(self, count):
9583
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   212
        # internal function doing the eid creation without locking.
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   213
        # needed for the recursive handling of disconnections (otherwise we
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   214
        # deadlock on self._eid_cnx_lock
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   215
        source = self.source
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   216
        if self.cnx is None:
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   217
            self.cnx = source.get_connection()
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   218
        cnx = self.cnx
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   219
        try:
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   220
            cursor = cnx.cursor()
9585
3f5b59527d31 [source/native] allow many eid creation per .create_eid call (closes #3526594)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9583
diff changeset
   221
            for sql in source.dbhelper.sqls_increment_numrange('entities_id_seq', count):
9583
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   222
                cursor.execute(sql)
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   223
            eid = cursor.fetchone()[0]
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   224
        except (source.OperationalError, source.InterfaceError):
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   225
            # FIXME: better detection of deconnection pb
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   226
            source.warning("trying to reconnect create eid connection")
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   227
            self.cnx = None
9585
3f5b59527d31 [source/native] allow many eid creation per .create_eid call (closes #3526594)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9583
diff changeset
   228
            return self._create_eid(count)
9583
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   229
        except source.DbapiError as exc:
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   230
            # We get this one with pyodbc and SQL Server when connection was reset
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   231
            if exc.args[0] == '08S01':
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   232
                source.warning("trying to reconnect create eid connection")
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   233
                self.cnx = None
9585
3f5b59527d31 [source/native] allow many eid creation per .create_eid call (closes #3526594)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9583
diff changeset
   234
                return self._create_eid(count)
9583
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   235
            else:
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   236
                raise
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   237
        except Exception: # WTF?
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   238
            cnx.rollback()
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   239
            self.cnx = None
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   240
            source.exception('create eid failed in an unforeseen way on SQL statement %s', sql)
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   241
            raise
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   242
        else:
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   243
            cnx.commit()
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   244
            return eid
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   245
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   246
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   247
class SQLITEEidGenerator(object):
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   248
    __slots__ = ('source', 'lock')
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   249
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   250
    def __init__(self, source):
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   251
        self.source = source
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   252
        self.lock = Lock()
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   253
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   254
    def close(self):
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   255
        pass
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   256
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   257
    def create_eid(self, cnx, count=1):
9585
3f5b59527d31 [source/native] allow many eid creation per .create_eid call (closes #3526594)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9583
diff changeset
   258
        assert count > 0
9583
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   259
        source = self.source
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   260
        with self.lock:
9585
3f5b59527d31 [source/native] allow many eid creation per .create_eid call (closes #3526594)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9583
diff changeset
   261
            for sql in source.dbhelper.sqls_increment_numrange('entities_id_seq', count):
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   262
                cursor = source.doexec(cnx, sql)
9583
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   263
            return cursor.fetchone()[0]
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   264
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   265
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   266
class NativeSQLSource(SQLAdapterMixIn, AbstractSource):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   267
    """adapter for source using the native cubicweb schema (see below)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   268
    """
2354
9b4bac626977 ability to map attributes to something else than usual cw mapping on sql generation
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2310
diff changeset
   269
    sqlgen_class = SQLGenerator
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   270
    options = (
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   271
        ('db-driver',
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   272
         {'type' : 'string',
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   273
          'default': 'postgres',
5423
e15abfdcce38 backport default into stable: stable is now cw 3.8
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5397 5421
diff changeset
   274
          # XXX use choice type
e15abfdcce38 backport default into stable: stable is now cw 3.8
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5397 5421
diff changeset
   275
          'help': 'database driver (postgres, mysql, sqlite, sqlserver2005)',
6201
cf445fcaf8fe [c-c create] ask for database driver by default
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6128
diff changeset
   276
          'group': 'native-source', 'level': 0,
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   277
          }),
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   278
        ('db-host',
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   279
         {'type' : 'string',
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   280
          'default': '',
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   281
          'help': 'database host',
5323
329b4f6d18b4 [config] with lgc >= 0.50, option's dict inputlevel becomes level
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5217
diff changeset
   282
          'group': 'native-source', 'level': 1,
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   283
          }),
2566
714a8743d423 missing db-port option to source's option definitions
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2354
diff changeset
   284
        ('db-port',
714a8743d423 missing db-port option to source's option definitions
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2354
diff changeset
   285
         {'type' : 'string',
714a8743d423 missing db-port option to source's option definitions
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2354
diff changeset
   286
          'default': '',
714a8743d423 missing db-port option to source's option definitions
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2354
diff changeset
   287
          'help': 'database port',
5323
329b4f6d18b4 [config] with lgc >= 0.50, option's dict inputlevel becomes level
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5217
diff changeset
   288
          'group': 'native-source', 'level': 1,
2566
714a8743d423 missing db-port option to source's option definitions
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2354
diff changeset
   289
          }),
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   290
        ('db-name',
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   291
         {'type' : 'string',
3835
a191b3b9e455 more sensible default values to c-c "create" inputs
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3689
diff changeset
   292
          'default': Method('default_instance_id'),
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   293
          'help': 'database name',
5323
329b4f6d18b4 [config] with lgc >= 0.50, option's dict inputlevel becomes level
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5217
diff changeset
   294
          'group': 'native-source', 'level': 0,
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   295
          }),
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   296
        ('db-user',
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   297
         {'type' : 'string',
3835
a191b3b9e455 more sensible default values to c-c "create" inputs
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3689
diff changeset
   298
          'default': CubicWebNoAppConfiguration.mode == 'user' and getlogin() or 'cubicweb',
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   299
          'help': 'database user',
5323
329b4f6d18b4 [config] with lgc >= 0.50, option's dict inputlevel becomes level
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5217
diff changeset
   300
          'group': 'native-source', 'level': 0,
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   301
          }),
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   302
        ('db-password',
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   303
         {'type' : 'password',
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   304
          'default': '',
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   305
          'help': 'database password',
5323
329b4f6d18b4 [config] with lgc >= 0.50, option's dict inputlevel becomes level
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5217
diff changeset
   306
          'group': 'native-source', 'level': 0,
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   307
          }),
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   308
        ('db-encoding',
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   309
         {'type' : 'string',
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   310
          'default': 'utf8',
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   311
          'help': 'database encoding',
5323
329b4f6d18b4 [config] with lgc >= 0.50, option's dict inputlevel becomes level
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5217
diff changeset
   312
          'group': 'native-source', 'level': 1,
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   313
          }),
5413
dc896e698ab0 added db-extra-arguments to configuration parameters for native source
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5410
diff changeset
   314
        ('db-extra-arguments',
dc896e698ab0 added db-extra-arguments to configuration parameters for native source
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5410
diff changeset
   315
         {'type' : 'string',
dc896e698ab0 added db-extra-arguments to configuration parameters for native source
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5410
diff changeset
   316
          'default': '',
dc896e698ab0 added db-extra-arguments to configuration parameters for native source
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5410
diff changeset
   317
          'help': 'set to "Trusted_Connection" if you are using SQLServer and '
dc896e698ab0 added db-extra-arguments to configuration parameters for native source
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5410
diff changeset
   318
                  'want trusted authentication for the database connection',
5515
513af9be9e37 [config] rename remaining inputlevel to level in option definitions, as expected by lgc >= 0.50
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   319
          'group': 'native-source', 'level': 2,
5413
dc896e698ab0 added db-extra-arguments to configuration parameters for native source
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5410
diff changeset
   320
          }),
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   321
    )
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   322
6427
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6426
diff changeset
   323
    def __init__(self, repo, source_config, *args, **kwargs):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   324
        SQLAdapterMixIn.__init__(self, source_config)
3647
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
   325
        self.authentifiers = [LoginPasswordAuthentifier(self)]
6849
5a0c2cfc19bf [repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   326
        if repo.config['allow-email-login']:
5a0c2cfc19bf [repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   327
            self.authentifiers.insert(0, EmailPasswordAuthentifier(self))
6427
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6426
diff changeset
   328
        AbstractSource.__init__(self, repo, source_config, *args, **kwargs)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   329
        # sql generator
6427
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6426
diff changeset
   330
        self._rql_sqlgen = self.sqlgen_class(self.schema, self.dbhelper,
4831
c5aec27c1bf7 [repo] use logilab.db instead of lgc.adbh/lgc.db/lgc.sqlgen/indexer, test new date extranction functions
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4829
diff changeset
   331
                                             ATTR_MAP.copy())
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   332
        # full text index helper
4806
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
   333
        self.do_fti = not repo.config['delay-full-text-indexation']
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   334
        # sql queries cache
7954
a3d3220669d6 [cache] replace lgc.cache with something more appropriate (closes #1921713)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 7922
diff changeset
   335
        self._cache = QueryCache(repo.config['rql-cache-size'])
4964
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   336
        # (etype, attr) / storage mapping
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   337
        self._storages = {}
9583
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   338
        self.binary_to_str = self.dbhelper.dbapi_module.binary_to_str
5649
a07dee204187 fix unit tests by not using the new create_eid implementation with sqlite
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5645
diff changeset
   339
        if self.dbdriver == 'sqlite':
9583
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   340
            self.eid_generator = SQLITEEidGenerator(self)
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   341
        else:
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   342
            self.eid_generator = DefaultEidGenerator(self)
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   343
        self.create_eid = self.eid_generator.create_eid
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   344
6945
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   345
    def check_config(self, source_entity):
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   346
        """check configuration of source entity"""
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   347
        if source_entity.host_config:
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   348
            msg = source_entity._cw._('the system source has its configuration '
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   349
                                      'stored on the file-system')
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   350
            raise ValidationError(source_entity.eid, {role_name('config', 'subject'): msg})
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   351
3647
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
   352
    def add_authentifier(self, authentifier):
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
   353
        self.authentifiers.append(authentifier)
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
   354
        authentifier.source = self
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
   355
        authentifier.set_schema(self.schema)
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
   356
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   357
    def reset_caches(self):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   358
        """method called during test to reset potential source caches"""
7954
a3d3220669d6 [cache] replace lgc.cache with something more appropriate (closes #1921713)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 7922
diff changeset
   359
        self._cache = QueryCache(self.repo.config['rql-cache-size'])
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   360
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   361
    def clear_eid_cache(self, eid, etype):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   362
        """clear potential caches for the given eid"""
2610
2933cc6bf9ad [F native source] fix attempts to clear cache key
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2593
diff changeset
   363
        self._cache.pop('Any X WHERE X eid %s, X is %s' % (eid, etype), None)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   364
        self._cache.pop('Any X WHERE X eid %s' % eid, None)
2610
2933cc6bf9ad [F native source] fix attempts to clear cache key
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2593
diff changeset
   365
        self._cache.pop('Any %s' % eid, None)
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   366
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   367
    def sqlexec(self, cnx, sql, args=None):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   368
        """execute the query and return its result"""
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   369
        return self.process_result(self.doexec(cnx, sql, args))
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   370
7398
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   371
    def init_creating(self, cnxset=None):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   372
        # check full text index availibility
4806
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
   373
        if self.do_fti:
7398
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   374
            if cnxset is None:
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   375
                _cnxset = self.repo._get_cnxset()
6724
24bf6f181d0e [pyro source] store pyro source mapping file into the database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6427
diff changeset
   376
            else:
7398
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   377
                _cnxset = cnxset
9463
d62e13eba033 [multi-sources-removal] Simplify ConnectionsSet internal structures and public methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9452
diff changeset
   378
            if not self.dbhelper.has_fti_table(_cnxset.cu):
4825
cdd979ae1b57 fix name error fixed in wrong branch
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4818
diff changeset
   379
                if not self.repo.config.creating:
4810
3055dc7a7c88 don't issue critical warning about missing text index when we're creating the instance
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4807
diff changeset
   380
                    self.critical('no text index table')
4806
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
   381
                self.do_fti = False
7398
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   382
            if cnxset is None:
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   383
                _cnxset.cnxset_freed()
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   384
                self.repo._free_cnxset(_cnxset)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   385
7342
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
   386
    def backup(self, backupfile, confirm, format='native'):
2759
23d7a75693f8 R refactor backup and use tar.gz to store all sources
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 2638
diff changeset
   387
        """method called to create a backup of the source's data"""
7342
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
   388
        if format == 'portable':
8947
3bbd416b09ec [repo] straightforward bootstrap sequence. Closes #2841188
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8916
diff changeset
   389
            # ensure the schema is the one stored in the database: if repository
3bbd416b09ec [repo] straightforward bootstrap sequence. Closes #2841188
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8916
diff changeset
   390
            # started in quick_start mode, the file system's one has been loaded
3bbd416b09ec [repo] straightforward bootstrap sequence. Closes #2841188
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8916
diff changeset
   391
            # so force reload
3bbd416b09ec [repo] straightforward bootstrap sequence. Closes #2841188
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8916
diff changeset
   392
            if self.repo.config.quick_start:
3bbd416b09ec [repo] straightforward bootstrap sequence. Closes #2841188
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8916
diff changeset
   393
                self.repo.set_schema(self.repo.deserialize_schema(),
3bbd416b09ec [repo] straightforward bootstrap sequence. Closes #2841188
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8916
diff changeset
   394
                                     resetvreg=False)
7342
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
   395
            helper = DatabaseIndependentBackupRestore(self)
7398
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   396
            self.close_source_connections()
7342
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
   397
            try:
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
   398
                helper.backup(backupfile)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
   399
            finally:
7398
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   400
                self.open_source_connections()
7342
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
   401
        elif format == 'native':
7398
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   402
            self.close_source_connections()
7342
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
   403
            try:
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
   404
                self.backup_to_file(backupfile, confirm)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
   405
            finally:
7398
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   406
                self.open_source_connections()
7342
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
   407
        else:
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
   408
            raise ValueError('Unknown format %r' % format)
2493
9806571ea790 major refactoring of database dump/restore:
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2476
diff changeset
   409
7342
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
   410
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
   411
    def restore(self, backupfile, confirm, drop, format='native'):
2493
9806571ea790 major refactoring of database dump/restore:
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2476
diff changeset
   412
        """method called to restore a backup of source's data"""
7398
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   413
        if self.repo.config.init_cnxset_pool:
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   414
            self.close_source_connections()
2759
23d7a75693f8 R refactor backup and use tar.gz to store all sources
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 2638
diff changeset
   415
        try:
7342
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
   416
            if format == 'portable':
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
   417
                helper = DatabaseIndependentBackupRestore(self)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
   418
                helper.restore(backupfile)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
   419
            elif format == 'native':
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
   420
                self.restore_from_file(backupfile, confirm, drop=drop)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
   421
            else:
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
   422
                raise ValueError('Unknown format %r' % format)
2759
23d7a75693f8 R refactor backup and use tar.gz to store all sources
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 2638
diff changeset
   423
        finally:
7398
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   424
            if self.repo.config.init_cnxset_pool:
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   425
                self.open_source_connections()
2493
9806571ea790 major refactoring of database dump/restore:
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2476
diff changeset
   426
7342
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
   427
6957
ffda12be2e9f [repository] #1460066: backport datafeed cube as cubicweb source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6945
diff changeset
   428
    def init(self, activated, source_entity):
7543
570522300e22 [ms, entity metas] add 'actual source' to entities table / base entity metadata cache. Closes #1767090
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7536
diff changeset
   429
        try:
7786
18a366267612 [repo] use smarter query to check asource presence on startup (closes #1922099)
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 7754
diff changeset
   430
            # test if 'asource' column exists
7839
daf46963f4fe ensure the test for asource column won't crash on SQLServer (closes #1949621)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7787
diff changeset
   431
            query = self.dbhelper.sql_add_limit_offset('SELECT asource FROM entities', 1)
daf46963f4fe ensure the test for asource column won't crash on SQLServer (closes #1949621)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7787
diff changeset
   432
            source_entity._cw.system_sql(query)
8695
358d8bed9626 [toward-py3k] rewrite to "except AnException as exc:" (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8694
diff changeset
   433
        except Exception as ex:
7543
570522300e22 [ms, entity metas] add 'actual source' to entities table / base entity metadata cache. Closes #1767090
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7536
diff changeset
   434
            self.eid_type_source = self.eid_type_source_pre_131
9131
b3ad80aa645f fix migration from pre-3.13.1 versions (closes #2846978)
Julien Cristau <julien.cristau@logilab.fr>
parents: 9130
diff changeset
   435
        super(NativeSQLSource, self).init(activated, source_entity)
b3ad80aa645f fix migration from pre-3.13.1 versions (closes #2846978)
Julien Cristau <julien.cristau@logilab.fr>
parents: 9130
diff changeset
   436
        self.init_creating(source_entity._cw.cnxset)
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   437
5749
b4393b681f7a [repo] on repository shutdown, we've to close the new eid creation connection
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5729
diff changeset
   438
    def shutdown(self):
9583
e337a9f658e0 [source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9574
diff changeset
   439
        self.eid_generator.close()
5749
b4393b681f7a [repo] on repository shutdown, we've to close the new eid creation connection
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5729
diff changeset
   440
9492
c7fc56eecd1a English typography
Dimitri Papadopoulos <dimitri.papadopoulos@cea.fr>
parents: 9375
diff changeset
   441
    # XXX deprecates [un]map_attribute?
5013
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5011
diff changeset
   442
    def map_attribute(self, etype, attr, cb, sourcedb=True):
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5011
diff changeset
   443
        self._rql_sqlgen.attr_map['%s.%s' % (etype, attr)] = (cb, sourcedb)
2493
9806571ea790 major refactoring of database dump/restore:
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2476
diff changeset
   444
4512
e7ac20bf3629 unset_attribute_storage, for testing purpose at least
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4252
diff changeset
   445
    def unmap_attribute(self, etype, attr):
e7ac20bf3629 unset_attribute_storage, for testing purpose at least
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4252
diff changeset
   446
        self._rql_sqlgen.attr_map.pop('%s.%s' % (etype, attr), None)
e7ac20bf3629 unset_attribute_storage, for testing purpose at least
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4252
diff changeset
   447
4964
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   448
    def set_storage(self, etype, attr, storage):
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   449
        storage_dict = self._storages.setdefault(etype, {})
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   450
        storage_dict[attr] = storage
5013
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5011
diff changeset
   451
        self.map_attribute(etype, attr,
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5011
diff changeset
   452
                           storage.callback, storage.is_source_callback)
4964
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   453
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   454
    def unset_storage(self, etype, attr):
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   455
        self._storages[etype].pop(attr)
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   456
        # if etype has no storage left, remove the entry
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   457
        if not self._storages[etype]:
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   458
            del self._storages[etype]
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   459
        self.unmap_attribute(etype, attr)
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   460
5397
cdbf823450aa [bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5323
diff changeset
   461
    def storage(self, etype, attr):
cdbf823450aa [bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5323
diff changeset
   462
        """return the storage for the given entity type / attribute
cdbf823450aa [bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5323
diff changeset
   463
        """
cdbf823450aa [bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5323
diff changeset
   464
        try:
cdbf823450aa [bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5323
diff changeset
   465
            return self._storages[etype][attr]
cdbf823450aa [bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5323
diff changeset
   466
        except KeyError:
cdbf823450aa [bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5323
diff changeset
   467
            raise Exception('no custom storage set for %s.%s' % (etype, attr))
cdbf823450aa [bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5323
diff changeset
   468
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   469
    # ISource interface #######################################################
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   470
3647
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
   471
    def compile_rql(self, rql, sols):
3240
8604a15995d1 refactor so that rql rewriter may be used outside the server. Enhance it to be usable for RRQLExpression as well
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3041
diff changeset
   472
        rqlst = self.repo.vreg.rqlhelper.parse(rql)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   473
        rqlst.restricted_vars = ()
3647
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
   474
        rqlst.children[0].solutions = sols
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   475
        self.repo.querier.sqlgen_annotate(rqlst)
438
69b79faefa94 need_intersect test and fixes
sylvain.thenault@logilab.fr
parents: 0
diff changeset
   476
        set_qdata(self.schema.rschema, rqlst, ())
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   477
        return rqlst
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   478
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   479
    def set_schema(self, schema):
2476
1294a6bdf3bf application -> instance where it makes sense
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2354
diff changeset
   480
        """set the instance'schema"""
7954
a3d3220669d6 [cache] replace lgc.cache with something more appropriate (closes #1921713)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 7922
diff changeset
   481
        self._cache = QueryCache(self.repo.config['rql-cache-size'])
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   482
        self.cache_hit, self.cache_miss, self.no_cache = 0, 0, 0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   483
        self.schema = schema
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   484
        try:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   485
            self._rql_sqlgen.schema = schema
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   486
        except AttributeError:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   487
            pass # __init__
3647
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
   488
        for authentifier in self.authentifiers:
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
   489
            authentifier.set_schema(self.schema)
4806
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
   490
        clear_cache(self, 'need_fti_indexation')
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   491
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   492
    def support_entity(self, etype, write=False):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   493
        """return true if the given entity's type is handled by this adapter
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   494
        if write is true, return true only if it's a RW support
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   495
        """
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   496
        return not etype in NONSYSTEM_ETYPES
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   497
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   498
    def support_relation(self, rtype, write=False):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   499
        """return true if the given relation's type is handled by this adapter
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   500
        if write is true, return true only if it's a RW support
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   501
        """
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   502
        if write:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   503
            return not rtype in NONSYSTEM_RELATIONS
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   504
        # due to current multi-sources implementation, the system source
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   505
        # can't claim not supporting a relation
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   506
        return True #not rtype == 'content_for'
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   507
9512
88dc96fc9fc1 [server] use a connection instead of a session for user authentication
Julien Cristau <julien.cristau@logilab.fr>
parents: 9487
diff changeset
   508
    def authenticate(self, cnx, login, **kwargs):
3647
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
   509
        """return CWUser eid for the given login and other authentication
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
   510
        information found in kwargs, else raise `AuthenticationError`
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   511
        """
3647
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
   512
        for authentifier in self.authentifiers:
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   513
            try:
9512
88dc96fc9fc1 [server] use a connection instead of a session for user authentication
Julien Cristau <julien.cristau@logilab.fr>
parents: 9487
diff changeset
   514
                return authentifier.authenticate(cnx, login, **kwargs)
3647
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
   515
            except AuthenticationError:
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
   516
                continue
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
   517
        raise AuthenticationError()
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   518
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   519
    def syntax_tree_search(self, cnx, union, args=None, cachekey=None,
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   520
                           varmap=None):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   521
        """return result from this source for a rql query (actually from
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   522
        a rql syntax tree and a solution dictionary mapping each used
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   523
        variable to a possible type). If cachekey is given, the query
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   524
        necessary to fetch the results (but not the results themselves)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   525
        may be cached using this key.
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   526
        """
2625
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2620
diff changeset
   527
        assert dbg_st_search(self.uri, union, varmap, args, cachekey)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   528
        # remember number of actually selected term (sql generation may append some)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   529
        if cachekey is None:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   530
            self.no_cache += 1
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   531
            # generate sql query if we are able to do so (not supported types...)
5013
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5011
diff changeset
   532
            sql, qargs, cbs = self._rql_sqlgen.generate(union, args, varmap)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   533
        else:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   534
            # sql may be cached
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   535
            try:
5013
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5011
diff changeset
   536
                sql, qargs, cbs = self._cache[cachekey]
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   537
                self.cache_hit += 1
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   538
            except KeyError:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   539
                self.cache_miss += 1
5013
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5011
diff changeset
   540
                sql, qargs, cbs = self._rql_sqlgen.generate(union, args, varmap)
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5011
diff changeset
   541
                self._cache[cachekey] = sql, qargs, cbs
ad91f93bbb93 [source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5011
diff changeset
   542
        args = self.merge_args(args, qargs)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   543
        assert isinstance(sql, basestring), repr(sql)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   544
        try:
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   545
            cursor = self.doexec(cnx, sql, args)
4831
c5aec27c1bf7 [repo] use logilab.db instead of lgc.adbh/lgc.db/lgc.sqlgen/indexer, test new date extranction functions
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4829
diff changeset
   546
        except (self.OperationalError, self.InterfaceError):
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   547
            if cnx.mode == 'write':
5975
5120d97e2f7e [transaction] do not attempt to reconnect if there has been some write during the transaction
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5847
diff changeset
   548
                # do not attempt to reconnect if there has been some write
5120d97e2f7e [transaction] do not attempt to reconnect if there has been some write during the transaction
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5847
diff changeset
   549
                # during the transaction
5120d97e2f7e [transaction] do not attempt to reconnect if there has been some write during the transaction
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5847
diff changeset
   550
                raise
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   551
            # FIXME: better detection of deconnection pb
5107
3694bd379513 [source] log attempt to reconnect using warning level. Also, don't relog failed sql (already logged)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5098
diff changeset
   552
            self.warning("trying to reconnect")
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   553
            cnx.cnxset.reconnect()
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   554
            cursor = self.doexec(cnx, sql, args)
8695
358d8bed9626 [toward-py3k] rewrite to "except AnException as exc:" (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8694
diff changeset
   555
        except self.DbapiError as exc:
5605
2604545d7dd9 [win32 SQLServer] connection lost detection
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5515
diff changeset
   556
            # We get this one with pyodbc and SQL Server when connection was reset
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   557
            if exc.args[0] == '08S01' and cnx.mode != 'write':
5605
2604545d7dd9 [win32 SQLServer] connection lost detection
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5515
diff changeset
   558
                self.warning("trying to reconnect")
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   559
                cnx.cnxset.reconnect()
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   560
                cursor = self.doexec(cnx, sql, args)
5605
2604545d7dd9 [win32 SQLServer] connection lost detection
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5515
diff changeset
   561
            else:
2604545d7dd9 [win32 SQLServer] connection lost detection
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5515
diff changeset
   562
                raise
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   563
        results = self.process_result(cursor, cbs, session=cnx)
2625
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2620
diff changeset
   564
        assert dbg_results(results)
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2620
diff changeset
   565
        return results
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   566
4964
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   567
    @contextmanager
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   568
    def _storage_handler(self, entity, event):
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   569
        # 1/ memorize values as they are before the storage is called.
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   570
        #    For instance, the BFSStorage will replace the `data`
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   571
        #    binary value with a Binary containing the destination path
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   572
        #    on the filesystem. To make the entity.data usage absolutely
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   573
        #    transparent, we'll have to reset entity.data to its binary
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   574
        #    value once the SQL query will be executed
6889
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
   575
        restore_values = []
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
   576
        if isinstance(entity, list):
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
   577
            entities = entity
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
   578
        else:
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
   579
            entities = [entity]
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
   580
        etype = entities[0].__regid__
4964
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   581
        for attr, storage in self._storages.get(etype, {}).items():
6889
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
   582
            for entity in entities:
8501
b922dd08eb79 [storage] relies on event=='deleted' to detect "delete" event (closes #2450680)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8379
diff changeset
   583
                if event == 'deleted':
b922dd08eb79 [storage] relies on event=='deleted' to detect "delete" event (closes #2450680)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8379
diff changeset
   584
                    storage.entity_deleted(entity, attr)
b922dd08eb79 [storage] relies on event=='deleted' to detect "delete" event (closes #2450680)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8379
diff changeset
   585
                else:
6889
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
   586
                    edited = entity.cw_edited
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
   587
                    if attr in edited:
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
   588
                        handler = getattr(storage, 'entity_%s' % event)
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
   589
                        to_restore = handler(entity, attr)
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
   590
                        restore_values.append((entity, attr, to_restore))
5128
e5d300d75519 [python] take care to this detail of @contextmanager: if an unhandled exception occurs in the block, it is reraised inside the generator at the point where the yield occurred
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5107
diff changeset
   591
        try:
e5d300d75519 [python] take care to this detail of @contextmanager: if an unhandled exception occurs in the block, it is reraised inside the generator at the point where the yield occurred
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5107
diff changeset
   592
            yield # 2/ execute the source's instructions
e5d300d75519 [python] take care to this detail of @contextmanager: if an unhandled exception occurs in the block, it is reraised inside the generator at the point where the yield occurred
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5107
diff changeset
   593
        finally:
e5d300d75519 [python] take care to this detail of @contextmanager: if an unhandled exception occurs in the block, it is reraised inside the generator at the point where the yield occurred
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5107
diff changeset
   594
            # 3/ restore original values
6889
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
   595
            for entity, attr, value in restore_values:
6142
8bc6eac1fac1 [session] cleanup hook / operation / entity edition api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6128
diff changeset
   596
                entity.cw_edited.edited_attribute(attr, value)
4964
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   597
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   598
    def add_entity(self, cnx, entity):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   599
        """add a new entity to the source"""
4964
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   600
        with self._storage_handler(entity, 'added'):
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   601
            attrs = self.preprocess_entity(entity)
8900
010a59e12d89 use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8821
diff changeset
   602
            sql = self.sqlgen.insert(SQL_PREFIX + entity.cw_etype, attrs)
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   603
            self.doexec(cnx, sql, attrs)
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   604
            if cnx.ertype_supports_undo(entity.cw_etype):
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   605
                self._record_tx_action(cnx, 'tx_entity_actions', 'C',
8900
010a59e12d89 use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8821
diff changeset
   606
                                       etype=entity.cw_etype, eid=entity.eid)
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   607
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   608
    def update_entity(self, cnx, entity):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   609
        """replace an entity in the source"""
4964
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   610
        with self._storage_handler(entity, 'updated'):
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   611
            attrs = self.preprocess_entity(entity)
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   612
            if cnx.ertype_supports_undo(entity.cw_etype):
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   613
                changes = self._save_attrs(cnx, entity, attrs)
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   614
                self._record_tx_action(cnx, 'tx_entity_actions', 'U',
8900
010a59e12d89 use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8821
diff changeset
   615
                                       etype=entity.cw_etype, eid=entity.eid,
4964
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   616
                                       changes=self._binary(dumps(changes)))
8900
010a59e12d89 use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8821
diff changeset
   617
            sql = self.sqlgen.update(SQL_PREFIX + entity.cw_etype, attrs,
4964
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   618
                                     ['cw_eid'])
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   619
            self.doexec(cnx, sql, attrs)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   620
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   621
    def delete_entity(self, cnx, entity):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   622
        """delete an entity from the source"""
4964
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   623
        with self._storage_handler(entity, 'deleted'):
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   624
            if cnx.ertype_supports_undo(entity.cw_etype):
4964
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   625
                attrs = [SQL_PREFIX + r.type
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   626
                         for r in entity.e_schema.subject_relations()
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   627
                         if (r.final or r.inlined) and not r in VIRTUAL_RTYPES]
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   628
                changes = self._save_attrs(cnx, entity, attrs)
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   629
                self._record_tx_action(cnx, 'tx_entity_actions', 'D',
8900
010a59e12d89 use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8821
diff changeset
   630
                                       etype=entity.cw_etype, eid=entity.eid,
4964
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   631
                                       changes=self._binary(dumps(changes)))
d9e8af8a7a42 [source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 4943
diff changeset
   632
            attrs = {'cw_eid': entity.eid}
8900
010a59e12d89 use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8821
diff changeset
   633
            sql = self.sqlgen.delete(SQL_PREFIX + entity.cw_etype, attrs)
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   634
            self.doexec(cnx, sql, attrs)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   635
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   636
    def add_relation(self, cnx, subject, rtype, object, inlined=False):
5072
072ae171aeb0 [cleanup] style fixes, add nodes, 0.2 cents refactorings
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5071
diff changeset
   637
        """add a relation to the source"""
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   638
        self._add_relations(cnx,  rtype, [(subject, object)], inlined)
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   639
        if cnx.ertype_supports_undo(rtype):
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   640
            self._record_tx_action(cnx, 'tx_relation_actions', 'A',
5072
072ae171aeb0 [cleanup] style fixes, add nodes, 0.2 cents refactorings
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5071
diff changeset
   641
                                   eid_from=subject, rtype=rtype, eid_to=object)
072ae171aeb0 [cleanup] style fixes, add nodes, 0.2 cents refactorings
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5071
diff changeset
   642
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   643
    def add_relations(self, cnx,  rtype, subj_obj_list, inlined=False):
7237
9f619715665b [server] improve the speed of setting relations between entities (closes #1625257)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7202
diff changeset
   644
        """add a relations to the source"""
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   645
        self._add_relations(cnx, rtype, subj_obj_list, inlined)
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   646
        if cnx.ertype_supports_undo(rtype):
7237
9f619715665b [server] improve the speed of setting relations between entities (closes #1625257)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7202
diff changeset
   647
            for subject, object in subj_obj_list:
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   648
                self._record_tx_action(cnx, 'tx_relation_actions', 'A',
7237
9f619715665b [server] improve the speed of setting relations between entities (closes #1625257)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7202
diff changeset
   649
                                       eid_from=subject, rtype=rtype, eid_to=object)
7551
2d4ba5b984dc cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7530
diff changeset
   650
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   651
    def _add_relations(self, cnx, rtype, subj_obj_list, inlined=False):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   652
        """add a relation to the source"""
7243
9ab01bf84eac fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7237
diff changeset
   653
        sql = []
4818
9f9bfbcdecfd le patch massiveimport a été importé
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4810
diff changeset
   654
        if inlined is False:
7237
9f619715665b [server] improve the speed of setting relations between entities (closes #1625257)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7202
diff changeset
   655
            attrs = [{'eid_from': subject, 'eid_to': object}
9f619715665b [server] improve the speed of setting relations between entities (closes #1625257)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7202
diff changeset
   656
                     for subject, object in subj_obj_list]
7243
9ab01bf84eac fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7237
diff changeset
   657
            sql.append((self.sqlgen.insert('%s_relation' % rtype, attrs[0]), attrs))
4818
9f9bfbcdecfd le patch massiveimport a été importé
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4810
diff changeset
   658
        else: # used by data import
7243
9ab01bf84eac fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7237
diff changeset
   659
            etypes = {}
9ab01bf84eac fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7237
diff changeset
   660
            for subject, object in subj_obj_list:
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   661
                etype = cnx.entity_metas(subject)['type']
7243
9ab01bf84eac fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7237
diff changeset
   662
                if etype in etypes:
9ab01bf84eac fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7237
diff changeset
   663
                    etypes[etype].append((subject, object))
9ab01bf84eac fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7237
diff changeset
   664
                else:
9ab01bf84eac fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7237
diff changeset
   665
                    etypes[etype] = [(subject, object)]
9ab01bf84eac fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7237
diff changeset
   666
            for subj_etype, subj_obj_list in etypes.iteritems():
9ab01bf84eac fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7237
diff changeset
   667
                attrs = [{'cw_eid': subject, SQL_PREFIX + rtype: object}
9ab01bf84eac fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7237
diff changeset
   668
                         for subject, object in subj_obj_list]
9ab01bf84eac fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7237
diff changeset
   669
                sql.append((self.sqlgen.update(SQL_PREFIX + etype, attrs[0],
9ab01bf84eac fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7237
diff changeset
   670
                                     ['cw_eid']),
9ab01bf84eac fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7237
diff changeset
   671
                            attrs))
9ab01bf84eac fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7237
diff changeset
   672
        for statement, attrs in sql:
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   673
            self.doexecmany(cnx, statement, attrs)
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   674
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   675
    def delete_relation(self, cnx, subject, rtype, object):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   676
        """delete a relation from the source"""
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   677
        rschema = self.schema.rschema(rtype)
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   678
        self._delete_relation(cnx, subject, rtype, object, rschema.inlined)
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   679
        if cnx.ertype_supports_undo(rtype):
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   680
            self._record_tx_action(cnx, 'tx_relation_actions', 'R',
5076
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   681
                                   eid_from=subject, rtype=rtype, eid_to=object)
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   682
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   683
    def _delete_relation(self, cnx, subject, rtype, object, inlined=False):
5076
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   684
        """delete a relation from the source"""
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
   685
        if inlined:
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   686
            table = SQL_PREFIX + cnx.entity_metas(subject)['type']
1251
af40e615dc89 introduce a 'cw_' prefix on entity table and column names so we don't conflict with sql or DBMS specific keywords
sylvain.thenault@logilab.fr
parents: 1079
diff changeset
   687
            column = SQL_PREFIX + rtype
af40e615dc89 introduce a 'cw_' prefix on entity table and column names so we don't conflict with sql or DBMS specific keywords
sylvain.thenault@logilab.fr
parents: 1079
diff changeset
   688
            sql = 'UPDATE %s SET %s=NULL WHERE %seid=%%(eid)s' % (table, column,
af40e615dc89 introduce a 'cw_' prefix on entity table and column names so we don't conflict with sql or DBMS specific keywords
sylvain.thenault@logilab.fr
parents: 1079
diff changeset
   689
                                                                  SQL_PREFIX)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   690
            attrs = {'eid' : subject}
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   691
        else:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   692
            attrs = {'eid_from': subject, 'eid_to': object}
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   693
            sql = self.sqlgen.delete('%s_relation' % rtype, attrs)
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   694
        self.doexec(cnx, sql, attrs)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   695
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   696
    def doexec(self, cnx, query, args=None, rollback=True):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   697
        """Execute a query.
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   698
        it's a function just so that it shows up in profiling
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   699
        """
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   700
        cursor = cnx.cnxset.cu
2593
16d9419a4a79 F: start to handle binary debug log level on the server side
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2589
diff changeset
   701
        if server.DEBUG & server.DBG_SQL:
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   702
            cnx = cnx.cnxset.cnx
9447
0636c4960259 [multi-sources-removal] Drop cw.server.sources.extlite
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9402
diff changeset
   703
            # getattr to get the actual connection if cnx is a CnxLoggingWrapper
2625
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2620
diff changeset
   704
            # instance
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2620
diff changeset
   705
            print 'exec', query, args, getattr(cnx, '_cnx', cnx)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   706
        try:
9701
46c8d8701240 an unicode string -> a unicode string
Rémi Cardona <remi.cardona@logilab.fr>
parents: 9492
diff changeset
   707
            # str(query) to avoid error if it's a unicode string
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   708
            cursor.execute(str(query), args)
8695
358d8bed9626 [toward-py3k] rewrite to "except AnException as exc:" (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8694
diff changeset
   709
        except Exception as ex:
4682
4994901b7379 don't issue critical message when trying to alter sqlite db during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4517
diff changeset
   710
            if self.repo.config.mode != 'test':
4994901b7379 don't issue critical message when trying to alter sqlite db during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4517
diff changeset
   711
                # during test we get those message when trying to alter sqlite
4994901b7379 don't issue critical message when trying to alter sqlite db during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4517
diff changeset
   712
                # db schema
4994901b7379 don't issue critical message when trying to alter sqlite db during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4517
diff changeset
   713
                self.critical("sql: %r\n args: %s\ndbms message: %r",
4994901b7379 don't issue critical message when trying to alter sqlite db during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4517
diff changeset
   714
                              query, args, ex.args[0])
2618
ff9b0d5bd884 [F repo sqlite schema changes] don't rollback on potentially expected schema changes failure
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2611
diff changeset
   715
            if rollback:
ff9b0d5bd884 [F repo sqlite schema changes] don't rollback on potentially expected schema changes failure
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2611
diff changeset
   716
                try:
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   717
                    cnx.cnxset.rollback()
4692
11a040e2601c [test] also hide this message during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4682
diff changeset
   718
                    if self.repo.config.mode != 'test':
9267
24d9b86dfa54 spelling: rollbacked -> rolled back
Rémi Cardona <remi.cardona@logilab.fr>
parents: 9131
diff changeset
   719
                        self.critical('transaction has been rolled back')
8695
358d8bed9626 [toward-py3k] rewrite to "except AnException as exc:" (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8694
diff changeset
   720
                except Exception as ex:
2618
ff9b0d5bd884 [F repo sqlite schema changes] don't rollback on potentially expected schema changes failure
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2611
diff changeset
   721
                    pass
6211
e9d125fd1465 nicer error reporting for unique together constraints
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6201
diff changeset
   722
            if ex.__class__.__name__ == 'IntegrityError':
e9d125fd1465 nicer error reporting for unique together constraints
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6201
diff changeset
   723
                # need string comparison because of various backends
e9d125fd1465 nicer error reporting for unique together constraints
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6201
diff changeset
   724
                for arg in ex.args:
9375
8e88576787c3 [schema] fix unique together index handling
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9336
diff changeset
   725
                    # postgres, sqlserver
8e88576787c3 [schema] fix unique together index handling
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9336
diff changeset
   726
                    mo = re.search("unique_[a-z0-9]{32}", arg)
6211
e9d125fd1465 nicer error reporting for unique together constraints
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6201
diff changeset
   727
                    if mo is not None:
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   728
                        raise UniqueTogetherError(cnx, cstrname=mo.group(0))
10053
079b5279a2cd [server] Handle unique constraint violations under recent sqlite
Julien Cristau <julien.cristau@logilab.fr>
parents: 10052
diff changeset
   729
                    # old sqlite
6247
f7cb092d2296 unique_together: recast exception raised by sqlite
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6211
diff changeset
   730
                    mo = re.search('columns (.*) are not unique', arg)
f7cb092d2296 unique_together: recast exception raised by sqlite
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6211
diff changeset
   731
                    if mo is not None: # sqlite in use
8820
dbffb6959564 server/source/native: fix wrong usage of .lstrip that produce garbled error messages (closes #2777641)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 8546
diff changeset
   732
                        # we left chop the 'cw_' prefix of attribute names
dbffb6959564 server/source/native: fix wrong usage of .lstrip that produce garbled error messages (closes #2777641)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 8546
diff changeset
   733
                        rtypes = [c.strip()[3:]
dbffb6959564 server/source/native: fix wrong usage of .lstrip that produce garbled error messages (closes #2777641)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 8546
diff changeset
   734
                                  for c in mo.group(1).split(',')]
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   735
                        raise UniqueTogetherError(cnx, rtypes=rtypes)
10053
079b5279a2cd [server] Handle unique constraint violations under recent sqlite
Julien Cristau <julien.cristau@logilab.fr>
parents: 10052
diff changeset
   736
                    # sqlite after http://www.sqlite.org/cgi/src/info/c80e229dd9c1230a
079b5279a2cd [server] Handle unique constraint violations under recent sqlite
Julien Cristau <julien.cristau@logilab.fr>
parents: 10052
diff changeset
   737
                    if arg.startswith('UNIQUE constraint failed:'):
079b5279a2cd [server] Handle unique constraint violations under recent sqlite
Julien Cristau <julien.cristau@logilab.fr>
parents: 10052
diff changeset
   738
                        # message looks like: "UNIQUE constraint failed: foo.cw_bar, foo.cw_baz"
079b5279a2cd [server] Handle unique constraint violations under recent sqlite
Julien Cristau <julien.cristau@logilab.fr>
parents: 10052
diff changeset
   739
                        # so drop the prefix, split on comma, drop the tablenames, and drop "cw_"
079b5279a2cd [server] Handle unique constraint violations under recent sqlite
Julien Cristau <julien.cristau@logilab.fr>
parents: 10052
diff changeset
   740
                        columns = arg.split(':', 1)[1].split(',')
079b5279a2cd [server] Handle unique constraint violations under recent sqlite
Julien Cristau <julien.cristau@logilab.fr>
parents: 10052
diff changeset
   741
                        rtypes = [c.split('.', 1)[1].strip()[3:] for c in columns]
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   742
                        raise UniqueTogetherError(cnx, rtypes=rtypes)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   743
            raise
2306
95da5d9f0870 give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2072
diff changeset
   744
        return cursor
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   745
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   746
    def doexecmany(self, cnx, query, args):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   747
        """Execute a query.
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   748
        it's a function just so that it shows up in profiling
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   749
        """
2593
16d9419a4a79 F: start to handle binary debug log level on the server side
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2589
diff changeset
   750
        if server.DEBUG & server.DBG_SQL:
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   751
            print 'execmany', query, 'with', len(args), 'arguments'
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   752
        cursor = cnx.cnxset.cu
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   753
        try:
9701
46c8d8701240 an unicode string -> a unicode string
Rémi Cardona <remi.cardona@logilab.fr>
parents: 9492
diff changeset
   754
            # str(query) to avoid error if it's a unicode string
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   755
            cursor.executemany(str(query), args)
8695
358d8bed9626 [toward-py3k] rewrite to "except AnException as exc:" (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8694
diff changeset
   756
        except Exception as ex:
4682
4994901b7379 don't issue critical message when trying to alter sqlite db during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4517
diff changeset
   757
            if self.repo.config.mode != 'test':
4994901b7379 don't issue critical message when trying to alter sqlite db during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4517
diff changeset
   758
                # during test we get those message when trying to alter sqlite
4994901b7379 don't issue critical message when trying to alter sqlite db during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4517
diff changeset
   759
                # db schema
4994901b7379 don't issue critical message when trying to alter sqlite db during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4517
diff changeset
   760
                self.critical("sql many: %r\n args: %s\ndbms message: %r",
4994901b7379 don't issue critical message when trying to alter sqlite db during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4517
diff changeset
   761
                              query, args, ex.args[0])
2306
95da5d9f0870 give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2072
diff changeset
   762
            try:
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   763
                cnx.cnxset.rollback()
4692
11a040e2601c [test] also hide this message during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4682
diff changeset
   764
                if self.repo.config.mode != 'test':
9267
24d9b86dfa54 spelling: rollbacked -> rolled back
Rémi Cardona <remi.cardona@logilab.fr>
parents: 9131
diff changeset
   765
                    self.critical('transaction has been rolled back')
7645
cb2459aaa7d7 [source] closes #1816412: try to reconnect to dbms if connection is reset in eid_type_source()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 7562
diff changeset
   766
            except Exception:
2306
95da5d9f0870 give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2072
diff changeset
   767
                pass
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   768
            raise
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   769
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   770
    # short cut to method requiring advanced db helper usage ##################
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   771
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   772
    def update_rdef_column(self, cnx, rdef):
5891
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   773
        """update physical column for a relation definition (final or inlined)
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   774
        """
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   775
        table, column = rdef_table_column(rdef)
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   776
        coltype, allownull = rdef_physical_info(self.dbhelper, rdef)
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   777
        if not self.dbhelper.alter_column_support:
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   778
            self.error("backend can't alter %s.%s to %s%s", table, column, coltype,
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   779
                       not allownull and 'NOT NULL' or '')
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   780
            return
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   781
        self.dbhelper.change_col_type(LogCursor(cnx.cnxset.cu),
5891
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   782
                                      table, column, coltype, allownull)
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   783
        self.info('altered %s.%s: now %s%s', table, column, coltype,
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   784
                  not allownull and 'NOT NULL' or '')
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   785
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   786
    def update_rdef_null_allowed(self, cnx, rdef):
5891
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   787
        """update NULL / NOT NULL of physical column for a relation definition
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   788
        (final or inlined)
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   789
        """
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   790
        if not self.dbhelper.alter_column_support:
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   791
            # not supported (and NOT NULL not set by yams in that case, so no
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   792
            # worry)
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   793
            return
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   794
        table, column = rdef_table_column(rdef)
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   795
        coltype, allownull = rdef_physical_info(self.dbhelper, rdef)
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   796
        self.dbhelper.set_null_allowed(LogCursor(cnx.cnxset.cu),
5891
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   797
                                       table, column, coltype, allownull)
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   798
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   799
    def update_rdef_indexed(self, cnx, rdef):
5891
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   800
        table, column = rdef_table_column(rdef)
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   801
        if rdef.indexed:
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   802
            self.create_index(cnx, table, column)
5891
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   803
        else:
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   804
            self.drop_index(cnx, table, column)
5891
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   805
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   806
    def update_rdef_unique(self, cnx, rdef):
5891
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   807
        table, column = rdef_table_column(rdef)
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   808
        if rdef.constraint_by_type('UniqueConstraint'):
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   809
            self.create_index(cnx, table, column, unique=True)
5891
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   810
        else:
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   811
            self.drop_index(cnx, table, column, unique=True)
5891
99024ad59223 [schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5849
diff changeset
   812
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   813
    def create_index(self, cnx, table, column, unique=False):
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   814
        cursor = LogCursor(cnx.cnxset.cu)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   815
        self.dbhelper.create_index(cursor, table, column, unique)
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   816
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   817
    def drop_index(self, cnx, table, column, unique=False):
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   818
        cursor = LogCursor(cnx.cnxset.cu)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   819
        self.dbhelper.drop_index(cursor, table, column, unique)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   820
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   821
    # system source interface #################################################
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   822
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   823
    def _eid_type_source(self, cnx, eid, sql, _retry=True):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   824
        try:
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   825
            res = self.doexec(cnx, sql).fetchone()
7646
91a984ba420f [native source] fix bug introduced in 7645:cb2459aaa7d7: fetchone() may return None
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7645
diff changeset
   826
            if res is not None:
91a984ba420f [native source] fix bug introduced in 7645:cb2459aaa7d7: fetchone() may return None
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7645
diff changeset
   827
                return res
7645
cb2459aaa7d7 [source] closes #1816412: try to reconnect to dbms if connection is reset in eid_type_source()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 7562
diff changeset
   828
        except (self.OperationalError, self.InterfaceError):
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   829
            if cnx.mode == 'read' and _retry:
7645
cb2459aaa7d7 [source] closes #1816412: try to reconnect to dbms if connection is reset in eid_type_source()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 7562
diff changeset
   830
                self.warning("trying to reconnect (eid_type_source())")
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   831
                cnx.cnxset.reconnect()
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   832
                return self._eid_type_source(cnx, eid, sql, _retry=False)
7645
cb2459aaa7d7 [source] closes #1816412: try to reconnect to dbms if connection is reset in eid_type_source()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 7562
diff changeset
   833
        except Exception:
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   834
            assert cnx.cnxset, 'connection has no connections set'
7645
cb2459aaa7d7 [source] closes #1816412: try to reconnect to dbms if connection is reset in eid_type_source()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 7562
diff changeset
   835
            self.exception('failed to query entities table for eid %s', eid)
cb2459aaa7d7 [source] closes #1816412: try to reconnect to dbms if connection is reset in eid_type_source()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 7562
diff changeset
   836
        raise UnknownEid(eid)
cb2459aaa7d7 [source] closes #1816412: try to reconnect to dbms if connection is reset in eid_type_source()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 7562
diff changeset
   837
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   838
    def eid_type_source(self, cnx, eid): # pylint: disable=E0202
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   839
        """return a tuple (type, source, extid) for the entity with id <eid>"""
9469
032825bbacab [multi-sources-removal] Drop entities.source column
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9468
diff changeset
   840
        sql = 'SELECT type, extid, asource FROM entities WHERE eid=%s' % eid
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   841
        res = self._eid_type_source(cnx, eid, sql)
7543
570522300e22 [ms, entity metas] add 'actual source' to entities table / base entity metadata cache. Closes #1767090
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7536
diff changeset
   842
        if res[-2] is not None:
1952
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1792
diff changeset
   843
            if not isinstance(res, list):
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1792
diff changeset
   844
                res = list(res)
7543
570522300e22 [ms, entity metas] add 'actual source' to entities table / base entity metadata cache. Closes #1767090
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7536
diff changeset
   845
            res[-2] = b64decode(res[-2])
570522300e22 [ms, entity metas] add 'actual source' to entities table / base entity metadata cache. Closes #1767090
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7536
diff changeset
   846
        return res
570522300e22 [ms, entity metas] add 'actual source' to entities table / base entity metadata cache. Closes #1767090
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7536
diff changeset
   847
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   848
    def eid_type_source_pre_131(self, cnx, eid):
7543
570522300e22 [ms, entity metas] add 'actual source' to entities table / base entity metadata cache. Closes #1767090
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7536
diff changeset
   849
        """return a tuple (type, source, extid) for the entity with id <eid>"""
9469
032825bbacab [multi-sources-removal] Drop entities.source column
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9468
diff changeset
   850
        sql = 'SELECT type, extid FROM entities WHERE eid=%s' % eid
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   851
        res = self._eid_type_source(cnx, eid, sql)
7543
570522300e22 [ms, entity metas] add 'actual source' to entities table / base entity metadata cache. Closes #1767090
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7536
diff changeset
   852
        if not isinstance(res, list):
570522300e22 [ms, entity metas] add 'actual source' to entities table / base entity metadata cache. Closes #1767090
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7536
diff changeset
   853
            res = list(res)
1952
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1792
diff changeset
   854
        if res[-1] is not None:
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1792
diff changeset
   855
            res[-1] = b64decode(res[-1])
7543
570522300e22 [ms, entity metas] add 'actual source' to entities table / base entity metadata cache. Closes #1767090
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7536
diff changeset
   856
        res.append(res[1])
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   857
        return res
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   858
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   859
    def extid2eid(self, cnx, extid):
1952
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1792
diff changeset
   860
        """get eid from an external id. Return None if no record found."""
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1792
diff changeset
   861
        assert isinstance(extid, str)
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   862
        cursor = self.doexec(cnx,
9469
032825bbacab [multi-sources-removal] Drop entities.source column
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9468
diff changeset
   863
                             'SELECT eid FROM entities WHERE extid=%(x)s',
032825bbacab [multi-sources-removal] Drop entities.source column
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9468
diff changeset
   864
                             {'x': b64encode(extid)})
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   865
        # XXX testing rowcount cause strange bug with sqlite, results are there
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   866
        #     but rowcount is 0
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   867
        #if cursor.rowcount > 0:
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   868
        try:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   869
            result = cursor.fetchone()
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   870
            if result:
1954
9b20f3504af8 cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1952
diff changeset
   871
                return result[0]
7645
cb2459aaa7d7 [source] closes #1816412: try to reconnect to dbms if connection is reset in eid_type_source()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 7562
diff changeset
   872
        except Exception:
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   873
            pass
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   874
        return None
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   875
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   876
    def _handle_is_relation_sql(self, cnx, sql, attrs):
8625
7ee0752178e5 [dataimport] Add SQL Store for faster import - works ONLY with Postgres for now, as it requires "copy from" command - closes #2410822
Vincent Michel <vincent.michel@logilab.fr>
parents: 8573
diff changeset
   877
        """ Handler for specific is_relation sql that may be
7ee0752178e5 [dataimport] Add SQL Store for faster import - works ONLY with Postgres for now, as it requires "copy from" command - closes #2410822
Vincent Michel <vincent.michel@logilab.fr>
parents: 8573
diff changeset
   878
        overwritten in some stores"""
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   879
        self.doexec(cnx, sql % attrs)
8625
7ee0752178e5 [dataimport] Add SQL Store for faster import - works ONLY with Postgres for now, as it requires "copy from" command - closes #2410822
Vincent Michel <vincent.michel@logilab.fr>
parents: 8573
diff changeset
   880
7ee0752178e5 [dataimport] Add SQL Store for faster import - works ONLY with Postgres for now, as it requires "copy from" command - closes #2410822
Vincent Michel <vincent.michel@logilab.fr>
parents: 8573
diff changeset
   881
    _handle_insert_entity_sql = doexec
7ee0752178e5 [dataimport] Add SQL Store for faster import - works ONLY with Postgres for now, as it requires "copy from" command - closes #2410822
Vincent Michel <vincent.michel@logilab.fr>
parents: 8573
diff changeset
   882
    _handle_is_instance_of_sql = _handle_source_relation_sql = _handle_is_relation_sql
7ee0752178e5 [dataimport] Add SQL Store for faster import - works ONLY with Postgres for now, as it requires "copy from" command - closes #2410822
Vincent Michel <vincent.michel@logilab.fr>
parents: 8573
diff changeset
   883
9664
5ef5494b6b0b [repository] complete argument is no more needed since we have no more 'true' multisource.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9611
diff changeset
   884
    def add_info(self, cnx, entity, source, extid):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   885
        """add type and source info for an eid into the system table"""
9802
9f815b1f2516 [repository,tests] use the new connection api (part 1/2)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9770
diff changeset
   886
        assert cnx.cnxset is not None
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   887
        # begin by inserting eid/type/source/extid into the entities table
1952
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1792
diff changeset
   888
        if extid is not None:
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1792
diff changeset
   889
            assert isinstance(extid, str)
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1792
diff changeset
   890
            extid = b64encode(extid)
8900
010a59e12d89 use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8821
diff changeset
   891
        attrs = {'type': entity.cw_etype, 'eid': entity.eid, 'extid': extid,
9802
9f815b1f2516 [repository,tests] use the new connection api (part 1/2)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9770
diff changeset
   892
                 'asource': source.uri}
9f815b1f2516 [repository,tests] use the new connection api (part 1/2)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9770
diff changeset
   893
        self._handle_insert_entity_sql(cnx, self.sqlgen.insert('entities', attrs), attrs)
6427
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6426
diff changeset
   894
        # insert core relations: is, is_instance_of and cw_source
6851
824d5b6eae7f [repo] kill no more needed 'recreate' feature
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6849
diff changeset
   895
        try:
9802
9f815b1f2516 [repository,tests] use the new connection api (part 1/2)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9770
diff changeset
   896
            self._handle_is_relation_sql(cnx, 'INSERT INTO is_relation(eid_from,eid_to) VALUES (%s,%s)',
9f815b1f2516 [repository,tests] use the new connection api (part 1/2)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9770
diff changeset
   897
                                         (entity.eid, eschema_eid(cnx, entity.e_schema)))
6851
824d5b6eae7f [repo] kill no more needed 'recreate' feature
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6849
diff changeset
   898
        except IndexError:
824d5b6eae7f [repo] kill no more needed 'recreate' feature
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6849
diff changeset
   899
            # during schema serialization, skip
824d5b6eae7f [repo] kill no more needed 'recreate' feature
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6849
diff changeset
   900
            pass
824d5b6eae7f [repo] kill no more needed 'recreate' feature
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6849
diff changeset
   901
        else:
824d5b6eae7f [repo] kill no more needed 'recreate' feature
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6849
diff changeset
   902
            for eschema in entity.e_schema.ancestors() + [entity.e_schema]:
9802
9f815b1f2516 [repository,tests] use the new connection api (part 1/2)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9770
diff changeset
   903
                self._handle_is_relation_sql(cnx,
8625
7ee0752178e5 [dataimport] Add SQL Store for faster import - works ONLY with Postgres for now, as it requires "copy from" command - closes #2410822
Vincent Michel <vincent.michel@logilab.fr>
parents: 8573
diff changeset
   904
                                             'INSERT INTO is_instance_of_relation(eid_from,eid_to) VALUES (%s,%s)',
9802
9f815b1f2516 [repository,tests] use the new connection api (part 1/2)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9770
diff changeset
   905
                                             (entity.eid, eschema_eid(cnx, eschema)))
6851
824d5b6eae7f [repo] kill no more needed 'recreate' feature
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6849
diff changeset
   906
        if 'CWSource' in self.schema and source.eid is not None: # else, cw < 3.10
9802
9f815b1f2516 [repository,tests] use the new connection api (part 1/2)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9770
diff changeset
   907
            self._handle_is_relation_sql(cnx, 'INSERT INTO cw_source_relation(eid_from,eid_to) VALUES (%s,%s)',
8625
7ee0752178e5 [dataimport] Add SQL Store for faster import - works ONLY with Postgres for now, as it requires "copy from" command - closes #2410822
Vincent Michel <vincent.michel@logilab.fr>
parents: 8573
diff changeset
   908
                                         (entity.eid, source.eid))
4806
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
   909
        # now we can update the full text index
8900
010a59e12d89 use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8821
diff changeset
   910
        if self.do_fti and self.need_fti_indexation(entity.cw_etype):
9802
9f815b1f2516 [repository,tests] use the new connection api (part 1/2)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9770
diff changeset
   911
            self.index_entity(cnx, entity=entity)
4806
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
   912
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   913
    def update_info(self, cnx, entity, need_fti_update):
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   914
        """mark entity as being modified, fulltext reindex if needed"""
4806
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
   915
        if self.do_fti and need_fti_update:
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
   916
            # reindex the entity only if this query is updating at least
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
   917
            # one indexable attribute
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   918
            self.index_entity(cnx, entity=entity)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   919
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   920
    def delete_info_multi(self, cnx, entities):
7501
2983dd24494a [repository] refactor/cleanup entity deletion methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   921
        """delete system information on deletion of a list of entities with the
2983dd24494a [repository] refactor/cleanup entity deletion methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   922
        same etype and belinging to the same source
2983dd24494a [repository] refactor/cleanup entity deletion methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   923
5067
adc2122eed03 [repo] more efficient eid cache operations handling based on set_operation; refactor
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5061
diff changeset
   924
        * update the fti
7501
2983dd24494a [repository] refactor/cleanup entity deletion methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   925
        * remove record from the `entities` table
6889
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
   926
        """
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   927
        self.fti_unindex_entities(cnx, entities)
6889
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
   928
        attrs = {'eid': '(%s)' % ','.join([str(_e.eid) for _e in entities])}
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   929
        self.doexec(cnx, self.sqlgen.delete_many('entities', attrs), attrs)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   930
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   931
    # undo support #############################################################
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   932
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   933
    def undoable_transactions(self, cnx, ueid=None, **actionfilters):
9487
88a092a665f4 [server/sources] replace references to dbapi with repoapi
Julien Cristau <julien.cristau@logilab.fr>
parents: 9482
diff changeset
   934
        """See :class:`cubicweb.repoapi.ClientConnection.undoable_transactions`"""
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   935
        # force filtering to connection's user if not a manager
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   936
        if not cnx.user.is_in_group('managers'):
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
   937
            ueid = cnx.user.eid
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   938
        restr = {}
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   939
        if ueid is not None:
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   940
            restr['tx_user'] = ueid
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   941
        sql = self.sqlgen.select('transactions', restr, ('tx_uuid', 'tx_time', 'tx_user'))
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   942
        if actionfilters:
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   943
            # we will need subqueries to filter transactions according to
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   944
            # actions done
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   945
            tearestr = {} # filters on the tx_entity_actions table
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   946
            trarestr = {} # filters on the tx_relation_actions table
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   947
            genrestr = {} # generic filters, appliyable to both table
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   948
            # unless public explicitly set to false, we only consider public
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   949
            # actions
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   950
            if actionfilters.pop('public', True):
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   951
                genrestr['txa_public'] = True
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   952
            # put additional filters in trarestr and/or tearestr
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   953
            for key, val in actionfilters.iteritems():
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   954
                if key == 'etype':
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   955
                    # filtering on etype implies filtering on entity actions
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   956
                    # only, and with no eid specified
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   957
                    assert actionfilters.get('action', 'C') in 'CUD'
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   958
                    assert not 'eid' in actionfilters
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   959
                    tearestr['etype'] = val
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   960
                elif key == 'eid':
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   961
                    # eid filter may apply to 'eid' of tx_entity_actions or to
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   962
                    # 'eid_from' OR 'eid_to' of tx_relation_actions
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   963
                    if actionfilters.get('action', 'C') in 'CUD':
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   964
                        tearestr['eid'] = val
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   965
                    if actionfilters.get('action', 'A') in 'AR':
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   966
                        trarestr['eid_from'] = val
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   967
                        trarestr['eid_to'] = val
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   968
                elif key == 'action':
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   969
                    if val in 'CUD':
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   970
                        tearestr['txa_action'] = val
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   971
                    else:
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   972
                        assert val in 'AR'
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   973
                        trarestr['txa_action'] = val
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   974
                else:
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   975
                    raise AssertionError('unknow filter %s' % key)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   976
            assert trarestr or tearestr, "can't only filter on 'public'"
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   977
            subqsqls = []
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   978
            # append subqueries to the original query, using EXISTS()
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   979
            if trarestr or (genrestr and not tearestr):
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   980
                trarestr.update(genrestr)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   981
                trasql = self.sqlgen.select('tx_relation_actions', trarestr, ('1',))
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   982
                if 'eid_from' in trarestr:
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   983
                    # replace AND by OR between eid_from/eid_to restriction
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   984
                    trasql = sql_or_clauses(trasql, ['eid_from = %(eid_from)s',
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   985
                                                     'eid_to = %(eid_to)s'])
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   986
                trasql += ' AND transactions.tx_uuid=tx_relation_actions.tx_uuid'
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   987
                subqsqls.append('EXISTS(%s)' % trasql)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   988
            if tearestr or (genrestr and not trarestr):
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   989
                tearestr.update(genrestr)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   990
                teasql = self.sqlgen.select('tx_entity_actions', tearestr, ('1',))
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   991
                teasql += ' AND transactions.tx_uuid=tx_entity_actions.tx_uuid'
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   992
                subqsqls.append('EXISTS(%s)' % teasql)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   993
            if restr:
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   994
                sql += ' AND %s' % ' OR '.join(subqsqls)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   995
            else:
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   996
                sql += ' WHERE %s' % ' OR '.join(subqsqls)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   997
            restr.update(trarestr)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   998
            restr.update(tearestr)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
   999
        # we want results ordered by transaction's time descendant
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1000
        sql += ' ORDER BY tx_time DESC'
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1001
        with cnx.ensure_cnx_set:
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1002
            cu = self.doexec(cnx, sql, restr)
9605
e6b8931abe17 [sources] ensure we have a cnxset in undoable_transactions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9585
diff changeset
  1003
            # turn results into transaction objects
e6b8931abe17 [sources] ensure we have a cnxset in undoable_transactions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9585
diff changeset
  1004
            return [tx.Transaction(*args) for args in cu.fetchall()]
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1005
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1006
    def tx_info(self, cnx, txuuid):
9487
88a092a665f4 [server/sources] replace references to dbapi with repoapi
Julien Cristau <julien.cristau@logilab.fr>
parents: 9482
diff changeset
  1007
        """See :class:`cubicweb.repoapi.ClientConnection.transaction_info`"""
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1008
        return tx.Transaction(txuuid, *self._tx_info(cnx, txuuid))
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1009
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1010
    def tx_actions(self, cnx, txuuid, public):
9487
88a092a665f4 [server/sources] replace references to dbapi with repoapi
Julien Cristau <julien.cristau@logilab.fr>
parents: 9482
diff changeset
  1011
        """See :class:`cubicweb.repoapi.ClientConnection.transaction_actions`"""
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1012
        self._tx_info(cnx, txuuid)
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1013
        restr = {'tx_uuid': txuuid}
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1014
        if public:
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1015
            restr['txa_public'] = True
5072
072ae171aeb0 [cleanup] style fixes, add nodes, 0.2 cents refactorings
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5071
diff changeset
  1016
        # XXX use generator to avoid loading everything in memory?
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1017
        sql = self.sqlgen.select('tx_entity_actions', restr,
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1018
                                 ('txa_action', 'txa_public', 'txa_order',
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1019
                                  'etype', 'eid', 'changes'))
9996
65bb21409292 [server] hold connection to the db in tx_actions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9984
diff changeset
  1020
        with cnx.ensure_cnx_set:
65bb21409292 [server] hold connection to the db in tx_actions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9984
diff changeset
  1021
            cu = self.doexec(cnx, sql, restr)
65bb21409292 [server] hold connection to the db in tx_actions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9984
diff changeset
  1022
            actions = [tx.EntityAction(a,p,o,et,e,c and loads(self.binary_to_str(c)))
65bb21409292 [server] hold connection to the db in tx_actions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9984
diff changeset
  1023
                       for a,p,o,et,e,c in cu.fetchall()]
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1024
        sql = self.sqlgen.select('tx_relation_actions', restr,
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1025
                                 ('txa_action', 'txa_public', 'txa_order',
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1026
                                  'rtype', 'eid_from', 'eid_to'))
9996
65bb21409292 [server] hold connection to the db in tx_actions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9984
diff changeset
  1027
        with cnx.ensure_cnx_set:
65bb21409292 [server] hold connection to the db in tx_actions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9984
diff changeset
  1028
            cu = self.doexec(cnx, sql, restr)
65bb21409292 [server] hold connection to the db in tx_actions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9984
diff changeset
  1029
            actions += [tx.RelationAction(*args) for args in cu.fetchall()]
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1030
        return sorted(actions, key=lambda x: x.order)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1031
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1032
    def undo_transaction(self, cnx, txuuid):
9487
88a092a665f4 [server/sources] replace references to dbapi with repoapi
Julien Cristau <julien.cristau@logilab.fr>
parents: 9482
diff changeset
  1033
        """See :class:`cubicweb.repoapi.ClientConnection.undo_transaction`
5071
8631bb9f6e73 [undo] during undoing, call hooks in the [active]integrity and undo categories
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5070
diff changeset
  1034
8631bb9f6e73 [undo] during undoing, call hooks in the [active]integrity and undo categories
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5070
diff changeset
  1035
        important note: while undoing of a transaction, only hooks in the
8631bb9f6e73 [undo] during undoing, call hooks in the [active]integrity and undo categories
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5070
diff changeset
  1036
        'integrity', 'activeintegrity' and 'undo' categories are called.
8631bb9f6e73 [undo] during undoing, call hooks in the [active]integrity and undo categories
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5070
diff changeset
  1037
        """
7398
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
  1038
        # set mode so connections set isn't released subsquently until commit/rollback
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1039
        cnx.mode = 'write'
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1040
        errors = []
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1041
        cnx.transaction_data['undoing_uuid'] = txuuid
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1042
        with cnx.deny_all_hooks_but('integrity', 'activeintegrity', 'undo'):
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1043
            with cnx.security_enabled(read=False):
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1044
                for action in reversed(self.tx_actions(cnx, txuuid, False)):
4943
7f5b83578fec disable security when undoing
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4913
diff changeset
  1045
                    undomethod = getattr(self, '_undo_%s' % action.action.lower())
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1046
                    errors += undomethod(cnx, action)
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1047
        # remove the transactions record
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1048
        self.doexec(cnx,
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1049
                    "DELETE FROM transactions WHERE tx_uuid='%s'" % txuuid)
8265
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1050
        if errors:
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1051
            raise UndoTransactionException(txuuid, errors)
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1052
        else:
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1053
            return
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1054
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1055
    def start_undoable_transaction(self, cnx, uuid):
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1056
        """connection callback to insert a transaction record in the transactions
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1057
        table when some undoable transaction is started
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1058
        """
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1059
        ueid = cnx.user.eid
7922
d307c3817782 [repository] use utcnow instead of now for some internal timetamps (closes #1988458)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7879
diff changeset
  1060
        attrs = {'tx_uuid': uuid, 'tx_user': ueid, 'tx_time': datetime.utcnow()}
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1061
        self.doexec(cnx, self.sqlgen.insert('transactions', attrs), attrs)
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1062
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1063
    def _save_attrs(self, cnx, entity, attrs):
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1064
        """return a pickleable dictionary containing current values for given
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1065
        attributes of the entity
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1066
        """
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1067
        restr = {'cw_eid': entity.eid}
8900
010a59e12d89 use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8821
diff changeset
  1068
        sql = self.sqlgen.select(SQL_PREFIX + entity.cw_etype, restr, attrs)
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1069
        cu = self.doexec(cnx, sql, restr)
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1070
        values = dict(zip(attrs, cu.fetchone()))
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1071
        # ensure backend specific binary are converted back to string
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1072
        eschema = entity.e_schema
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1073
        for column in attrs:
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1074
            # [3:] remove 'cw_' prefix
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1075
            attr = column[3:]
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1076
            if not eschema.subjrels[attr].final:
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1077
                continue
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1078
            if eschema.destination(attr) in ('Password', 'Bytes'):
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1079
                value = values[column]
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1080
                if value is not None:
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1081
                    values[column] = self.binary_to_str(value)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1082
        return values
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1083
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1084
    def _record_tx_action(self, cnx, table, action, **kwargs):
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1085
        """record a transaction action in the given table (either
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1086
        'tx_entity_actions' or 'tx_relation_action')
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1087
        """
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1088
        kwargs['tx_uuid'] = cnx.transaction_uuid()
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1089
        kwargs['txa_action'] = action
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1090
        kwargs['txa_order'] = cnx.transaction_inc_action_counter()
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1091
        kwargs['txa_public'] = cnx.running_dbapi_query
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1092
        self.doexec(cnx, self.sqlgen.insert(table, kwargs), kwargs)
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1093
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1094
    def _tx_info(self, cnx, txuuid):
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1095
        """return transaction's time and user of the transaction with the given uuid.
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1096
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1097
        raise `NoSuchTransaction` if there is no such transaction of if the
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1098
        connection's user isn't allowed to see it.
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1099
        """
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1100
        with cnx.ensure_cnx_set:
9098
5467fb901931 [connection] transparent cnx_set handling
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8947
diff changeset
  1101
            restr = {'tx_uuid': txuuid}
5467fb901931 [connection] transparent cnx_set handling
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8947
diff changeset
  1102
            sql = self.sqlgen.select('transactions', restr,
5467fb901931 [connection] transparent cnx_set handling
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8947
diff changeset
  1103
                                     ('tx_time', 'tx_user'))
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1104
            cu = self.doexec(cnx, sql, restr)
9098
5467fb901931 [connection] transparent cnx_set handling
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8947
diff changeset
  1105
            try:
5467fb901931 [connection] transparent cnx_set handling
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8947
diff changeset
  1106
                time, ueid = cu.fetchone()
5467fb901931 [connection] transparent cnx_set handling
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8947
diff changeset
  1107
            except TypeError:
5467fb901931 [connection] transparent cnx_set handling
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8947
diff changeset
  1108
                raise tx.NoSuchTransaction(txuuid)
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1109
            if not (cnx.user.is_in_group('managers')
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1110
                    or cnx.user.eid == ueid):
9098
5467fb901931 [connection] transparent cnx_set handling
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8947
diff changeset
  1111
                raise tx.NoSuchTransaction(txuuid)
5467fb901931 [connection] transparent cnx_set handling
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8947
diff changeset
  1112
            return time, ueid
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1113
8265
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1114
    def _reedit_entity(self, entity, changes, err):
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1115
        cnx = entity._cw
8265
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1116
        eid = entity.eid
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1117
        entity.cw_edited = edited = EditedEntity(entity)
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1118
        # check for schema changes, entities linked through inlined relation
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1119
        # still exists, rewrap binary values
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1120
        eschema = entity.e_schema
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1121
        getrschema = eschema.subjrels
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1122
        for column, value in changes.items():
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1123
            rtype = column[len(SQL_PREFIX):]
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1124
            if rtype == "eid":
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1125
                continue # XXX should even `eid` be stored in action changes?
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1126
            try:
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1127
                rschema = getrschema[rtype]
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1128
            except KeyError:
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1129
                err(cnx._("can't restore relation %(rtype)s of entity %(eid)s, "
8265
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1130
                              "this relation does not exist in the schema anymore.")
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1131
                    % {'rtype': rtype, 'eid': eid})
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1132
            if not rschema.final:
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1133
                if not rschema.inlined:
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1134
                    assert value is None
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1135
                # rschema is an inlined relation
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1136
                elif value is not None:
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1137
                    # not a deletion: we must put something in edited
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1138
                    try:
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1139
                        entity._cw.entity_from_eid(value) # check target exists
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1140
                        edited[rtype] = value
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1141
                    except UnknownEid:
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1142
                        err(cnx._("can't restore entity %(eid)s of type %(eschema)s, "
8265
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1143
                                      "target of %(rtype)s (eid %(value)s) does not exist any longer")
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1144
                            % locals())
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1145
            elif eschema.destination(rtype) in ('Bytes', 'Password'):
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1146
                changes[column] = self._binary(value)
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1147
                edited[rtype] = Binary(value)
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1148
            elif isinstance(value, str):
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1149
                edited[rtype] = unicode(value, cnx.encoding, 'replace')
8265
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1150
            else:
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1151
                edited[rtype] = value
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1152
        # This must only be done after init_entitiy_caches : defered in calling functions
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1153
        # edited.check()
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1154
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1155
    def _undo_d(self, cnx, action):
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1156
        """undo an entity deletion"""
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1157
        errors = []
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1158
        err = errors.append
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1159
        eid = action.eid
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1160
        etype = action.etype
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1161
        _ = cnx._
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1162
        # get an entity instance
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1163
        try:
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1164
            entity = self.repo.vreg['etypes'].etype_class(etype)(cnx)
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1165
        except Exception:
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1166
            err("can't restore entity %s of type %s, type no more supported"
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1167
                % (eid, etype))
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1168
            return errors
8265
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1169
        self._reedit_entity(entity, action.changes, err)
5557
1a534c596bff [entity] continue cleanup of Entity/AnyEntity namespace
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5556
diff changeset
  1170
        entity.eid = eid
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1171
        cnx.repo.init_entity_caches(cnx, entity, self)
8265
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1172
        entity.cw_edited.check()
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1173
        self.repo.hm.call_hooks('before_add_entity', cnx, entity=entity)
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1174
        # restore the entity
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1175
        action.changes['cw_eid'] = eid
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1176
        sql = self.sqlgen.insert(SQL_PREFIX + etype, action.changes)
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1177
        self.doexec(cnx, sql, action.changes)
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1178
        # restore record in entities (will update fti if needed)
9664
5ef5494b6b0b [repository] complete argument is no more needed since we have no more 'true' multisource.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9611
diff changeset
  1179
        self.add_info(cnx, entity, self, None)
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1180
        self.repo.hm.call_hooks('after_add_entity', cnx, entity=entity)
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1181
        return errors
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1182
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1183
    def _undo_r(self, cnx, action):
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1184
        """undo a relation removal"""
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1185
        errors = []
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1186
        subj, rtype, obj = action.eid_from, action.rtype, action.eid_to
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1187
        try:
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1188
            sentity, oentity, rdef = _undo_rel_info(cnx, subj, rtype, obj)
8695
358d8bed9626 [toward-py3k] rewrite to "except AnException as exc:" (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8694
diff changeset
  1189
        except _UndoException as ex:
5076
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1190
            errors.append(unicode(ex))
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1191
        else:
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1192
            for role, entity in (('subject', sentity),
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1193
                                 ('object', oentity)):
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1194
                try:
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1195
                    _undo_check_relation_target(entity, rdef, role)
8695
358d8bed9626 [toward-py3k] rewrite to "except AnException as exc:" (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8694
diff changeset
  1196
                except _UndoException as ex:
5076
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1197
                    errors.append(unicode(ex))
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1198
                    continue
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1199
        if not errors:
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1200
            self.repo.hm.call_hooks('before_add_relation', cnx,
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1201
                                    eidfrom=subj, rtype=rtype, eidto=obj)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1202
            # add relation in the database
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1203
            self._add_relations(cnx, rtype, [(subj, obj)], rdef.rtype.inlined)
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1204
            # set related cache
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1205
            cnx.update_rel_cache_add(subj, rtype, obj, rdef.rtype.symmetric)
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1206
            self.repo.hm.call_hooks('after_add_relation', cnx,
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1207
                                    eidfrom=subj, rtype=rtype, eidto=obj)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1208
        return errors
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1209
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1210
    def _undo_c(self, cnx, action):
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1211
        """undo an entity creation"""
5076
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1212
        eid = action.eid
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1213
        # XXX done to avoid fetching all remaining relation for the entity
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1214
        # we should find an efficient way to do this (keeping current veolidf
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1215
        # massive deletion performance)
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1216
        if _undo_has_later_transaction(cnx, eid):
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1217
            msg = cnx._('some later transaction(s) touch entity, undo them '
5076
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1218
                            'first')
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1219
            raise ValidationError(eid, {None: msg})
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1220
        etype = action.etype
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1221
        # get an entity instance
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1222
        try:
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1223
            entity = self.repo.vreg['etypes'].etype_class(etype)(cnx)
5076
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1224
        except Exception:
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1225
            return [cnx._(
5098
32b1adfb6b92 [i18n] use named substitution to avoid gettext warnings
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5076
diff changeset
  1226
                "Can't undo creation of entity %(eid)s of type %(etype)s, type "
32b1adfb6b92 [i18n] use named substitution to avoid gettext warnings
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5076
diff changeset
  1227
                "no more supported" % {'eid': eid, 'etype': etype})]
5557
1a534c596bff [entity] continue cleanup of Entity/AnyEntity namespace
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5556
diff changeset
  1228
        entity.eid = eid
5076
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1229
        # for proper eid/type cache update
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1230
        CleanupDeletedEidsCacheOp.get_instance(cnx).add_data(eid)
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1231
        self.repo.hm.call_hooks('before_delete_entity', cnx, entity=entity)
5076
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1232
        # remove is / is_instance_of which are added using sql by hooks, hence
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1233
        # unvisible as transaction action
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1234
        self.doexec(cnx, 'DELETE FROM is_relation WHERE eid_from=%s' % eid)
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1235
        self.doexec(cnx, 'DELETE FROM is_instance_of_relation WHERE eid_from=%s' % eid)
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1236
        self.doexec(cnx, 'DELETE FROM cw_source_relation WHERE eid_from=%s' % self.eid)
5076
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1237
        # XXX check removal of inlined relation?
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1238
        # delete the entity
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1239
        attrs = {'cw_eid': eid}
8900
010a59e12d89 use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8821
diff changeset
  1240
        sql = self.sqlgen.delete(SQL_PREFIX + entity.cw_etype, attrs)
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1241
        self.doexec(cnx, sql, attrs)
5076
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1242
        # remove record from entities (will update fti if needed)
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1243
        self.delete_info_multi(cnx, [entity])
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1244
        self.repo.hm.call_hooks('after_delete_entity', cnx, entity=entity)
5076
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1245
        return ()
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1246
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1247
    def _undo_u(self, cnx, action):
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1248
        """undo an entity update"""
8265
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1249
        errors = []
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1250
        err = errors.append
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1251
        try:
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1252
            entity = cnx.entity_from_eid(action.eid)
8265
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1253
        except UnknownEid:
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1254
            err(cnx._("can't restore state of entity %s, it has been "
8265
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1255
                          "deleted inbetween") % action.eid)
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1256
            return errors
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1257
        self._reedit_entity(entity, action.changes, err)
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1258
        entity.cw_edited.check()
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1259
        self.repo.hm.call_hooks('before_update_entity', cnx, entity=entity)
8900
010a59e12d89 use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8821
diff changeset
  1260
        sql = self.sqlgen.update(SQL_PREFIX + entity.cw_etype, action.changes,
8265
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1261
                                 ['cw_eid'])
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1262
        self.doexec(cnx, sql, action.changes)
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1263
        self.repo.hm.call_hooks('after_update_entity', cnx, entity=entity)
8265
9747ab9230ad [repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents: 8235
diff changeset
  1264
        return errors
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1265
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1266
    def _undo_a(self, cnx, action):
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1267
        """undo a relation addition"""
5076
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1268
        errors = []
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1269
        subj, rtype, obj = action.eid_from, action.rtype, action.eid_to
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1270
        try:
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1271
            sentity, oentity, rdef = _undo_rel_info(cnx, subj, rtype, obj)
8695
358d8bed9626 [toward-py3k] rewrite to "except AnException as exc:" (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8694
diff changeset
  1272
        except _UndoException as ex:
5076
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1273
            errors.append(unicode(ex))
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1274
        else:
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1275
            rschema = rdef.rtype
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1276
            if rschema.inlined:
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1277
                sql = 'SELECT 1 FROM cw_%s WHERE cw_eid=%s and cw_%s=%s'\
8900
010a59e12d89 use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8821
diff changeset
  1278
                      % (sentity.cw_etype, subj, rtype, obj)
5076
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1279
            else:
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1280
                sql = 'SELECT 1 FROM %s_relation WHERE eid_from=%s and eid_to=%s'\
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1281
                      % (rtype, subj, obj)
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1282
            cu = self.doexec(cnx, sql)
5076
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1283
            if cu.fetchone() is None:
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1284
                errors.append(cnx._(
5098
32b1adfb6b92 [i18n] use named substitution to avoid gettext warnings
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5076
diff changeset
  1285
                    "Can't undo addition of relation %(rtype)s from %(subj)s to"
32b1adfb6b92 [i18n] use named substitution to avoid gettext warnings
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5076
diff changeset
  1286
                    " %(obj)s, doesn't exist anymore" % locals()))
5076
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1287
        if not errors:
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1288
            self.repo.hm.call_hooks('before_delete_relation', cnx,
5076
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1289
                                    eidfrom=subj, rtype=rtype, eidto=obj)
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1290
            # delete relation from the database
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1291
            self._delete_relation(cnx, subj, rtype, obj, rschema.inlined)
5076
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1292
            # set related cache
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1293
            cnx.update_rel_cache_del(subj, rtype, obj, rschema.symmetric)
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1294
            self.repo.hm.call_hooks('after_delete_relation', cnx,
5076
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1295
                                    eidfrom=subj, rtype=rtype, eidto=obj)
b0e6134b4324 [undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5075
diff changeset
  1296
        return errors
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1297
4806
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1298
    # full text index handling #################################################
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1299
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1300
    @cached
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1301
    def need_fti_indexation(self, etype):
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1302
        eschema = self.schema.eschema(etype)
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1303
        if any(eschema.indexable_attributes()):
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1304
            return True
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1305
        if any(eschema.fulltext_containers()):
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1306
            return True
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1307
        return False
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1308
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1309
    def index_entity(self, cnx, entity):
4807
5642bfa43236 [cleanup] add index_entity to abstract source, add docstring
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4806
diff changeset
  1310
        """create an operation to [re]index textual content of the given entity
5642bfa43236 [cleanup] add index_entity to abstract source, add docstring
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4806
diff changeset
  1311
        on commit
5642bfa43236 [cleanup] add index_entity to abstract source, add docstring
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4806
diff changeset
  1312
        """
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1313
        FTIndexEntityOp.get_instance(cnx).add_data(entity.eid)
4806
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1314
9574
2d4c4842bd04 [server] some s/session/cnx/
Julien Cristau <julien.cristau@logilab.fr>
parents: 9546
diff changeset
  1315
    def fti_unindex_entities(self, cnx, entities):
6889
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
  1316
        """remove text content for entities from the full text index
4806
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1317
        """
9574
2d4c4842bd04 [server] some s/session/cnx/
Julien Cristau <julien.cristau@logilab.fr>
parents: 9546
diff changeset
  1318
        cursor = cnx.cnxset.cu
6889
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
  1319
        cursor_unindex_object = self.dbhelper.cursor_unindex_object
4806
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1320
        try:
6889
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
  1321
            for entity in entities:
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
  1322
                cursor_unindex_object(entity.eid, cursor)
4806
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1323
        except Exception: # let KeyboardInterrupt / SystemExit propagate
6889
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
  1324
            self.exception('error while unindexing %s', entity)
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
  1325
4806
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1326
9574
2d4c4842bd04 [server] some s/session/cnx/
Julien Cristau <julien.cristau@logilab.fr>
parents: 9546
diff changeset
  1327
    def fti_index_entities(self, cnx, entities):
6889
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
  1328
        """add text content of created/modified entities to the full text index
4806
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1329
        """
6889
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
  1330
        cursor_index_object = self.dbhelper.cursor_index_object
9574
2d4c4842bd04 [server] some s/session/cnx/
Julien Cristau <julien.cristau@logilab.fr>
parents: 9546
diff changeset
  1331
        cursor = cnx.cnxset.cu
4806
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1332
        try:
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1333
            # use cursor_index_object, not cursor_reindex_object since
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1334
            # unindexing done in the FTIndexEntityOp
6889
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
  1335
            for entity in entities:
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
  1336
                cursor_index_object(entity.eid,
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
  1337
                                    entity.cw_adapt_to('IFTIndexable'),
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
  1338
                                    cursor)
4806
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1339
        except Exception: # let KeyboardInterrupt / SystemExit propagate
6889
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
  1340
            self.exception('error while indexing %s', entity)
4806
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1341
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1342
6426
541659c39f6a [hook/operation] nicer api to achieve same result as set_operation, as described in #1253630
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6366
diff changeset
  1343
class FTIndexEntityOp(hook.DataOperationMixIn, hook.LateOperation):
4806
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1344
    """operation to delay entity full text indexation to commit
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1345
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1346
    since fti indexing may trigger discovery of other entities, it should be
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1347
    triggered on precommit, not commit, and this should be done after other
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1348
    precommit operation which may add relations to the entity
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1349
    """
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1350
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4692
diff changeset
  1351
    def precommit_event(self):
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1352
        cnx = self.cnx
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1353
        source = cnx.repo.system_source
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1354
        pendingeids = cnx.transaction_data.get('pendingeids', ())
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1355
        done = cnx.transaction_data.setdefault('indexedeids', set())
6889
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
  1356
        to_reindex = set()
6426
541659c39f6a [hook/operation] nicer api to achieve same result as set_operation, as described in #1253630
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6366
diff changeset
  1357
        for eid in self.get_data():
5070
b1f80ccadda3 [repo] refactor fti operation to use set_operation and a single operation whatever the number of entities to ft index
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5067
diff changeset
  1358
            if eid in pendingeids or eid in done:
b1f80ccadda3 [repo] refactor fti operation to use set_operation and a single operation whatever the number of entities to ft index
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5067
diff changeset
  1359
                # entity added and deleted in the same transaction or already
b1f80ccadda3 [repo] refactor fti operation to use set_operation and a single operation whatever the number of entities to ft index
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5067
diff changeset
  1360
                # processed
6889
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
  1361
                continue
5070
b1f80ccadda3 [repo] refactor fti operation to use set_operation and a single operation whatever the number of entities to ft index
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5067
diff changeset
  1362
            done.add(eid)
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1363
            iftindexable = cnx.entity_from_eid(eid).cw_adapt_to('IFTIndexable')
6889
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6850
diff changeset
  1364
            to_reindex |= set(iftindexable.fti_containers())
9611
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1365
        source.fti_unindex_entities(cnx, to_reindex)
ca853478aaa8 [server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents: 9605
diff changeset
  1366
        source.fti_index_entities(cnx, to_reindex)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1367
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1368
def sql_schema(driver):
4831
c5aec27c1bf7 [repo] use logilab.db instead of lgc.adbh/lgc.db/lgc.sqlgen/indexer, test new date extranction functions
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4829
diff changeset
  1369
    helper = get_db_helper(driver)
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1370
    typemap = helper.TYPE_MAPPING
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1371
    schema = """
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1372
/* Create the repository's system database */
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1373
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1374
%s
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1375
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1376
CREATE TABLE entities (
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1377
  eid INTEGER PRIMARY KEY NOT NULL,
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1378
  type VARCHAR(64) NOT NULL,
8205
14c52c5ee706 [entities system table] change asource and source column size (closes #1951819)
Alexandre Richardson <alexandre.richardson@logilab.fr>
parents: 7967
diff changeset
  1379
  asource VARCHAR(128) NOT NULL,
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1380
  extid VARCHAR(256)
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1381
);;
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1382
CREATE INDEX entities_type_idx ON entities(type);;
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1383
CREATE INDEX entities_extid_idx ON entities(extid);;
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1384
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1385
CREATE TABLE transactions (
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1386
  tx_uuid CHAR(32) PRIMARY KEY NOT NULL,
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1387
  tx_user INTEGER NOT NULL,
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1388
  tx_time %s NOT NULL
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1389
);;
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1390
CREATE INDEX transactions_tx_user_idx ON transactions(tx_user);;
9336
722635e530a0 [server] Add missing indices for undo support (closes #3259691)
Julien Cristau <julien.cristau@logilab.fr>
parents: 9298
diff changeset
  1391
CREATE INDEX transactions_tx_time_idx ON transactions(tx_time);;
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1392
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1393
CREATE TABLE tx_entity_actions (
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1394
  tx_uuid CHAR(32) REFERENCES transactions(tx_uuid) ON DELETE CASCADE,
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1395
  txa_action CHAR(1) NOT NULL,
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1396
  txa_public %s NOT NULL,
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1397
  txa_order INTEGER,
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1398
  eid INTEGER NOT NULL,
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1399
  etype VARCHAR(64) NOT NULL,
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1400
  changes %s
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1401
);;
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1402
CREATE INDEX tx_entity_actions_txa_action_idx ON tx_entity_actions(txa_action);;
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1403
CREATE INDEX tx_entity_actions_txa_public_idx ON tx_entity_actions(txa_public);;
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1404
CREATE INDEX tx_entity_actions_eid_idx ON tx_entity_actions(eid);;
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1405
CREATE INDEX tx_entity_actions_etype_idx ON tx_entity_actions(etype);;
9336
722635e530a0 [server] Add missing indices for undo support (closes #3259691)
Julien Cristau <julien.cristau@logilab.fr>
parents: 9298
diff changeset
  1406
CREATE INDEX tx_entity_actions_tx_uuid_idx ON tx_entity_actions(tx_uuid);;
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1407
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1408
CREATE TABLE tx_relation_actions (
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1409
  tx_uuid CHAR(32) REFERENCES transactions(tx_uuid) ON DELETE CASCADE,
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1410
  txa_action CHAR(1) NOT NULL,
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1411
  txa_public %s NOT NULL,
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1412
  txa_order INTEGER,
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1413
  eid_from INTEGER NOT NULL,
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1414
  eid_to INTEGER NOT NULL,
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1415
  rtype VARCHAR(256) NOT NULL
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1416
);;
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1417
CREATE INDEX tx_relation_actions_txa_action_idx ON tx_relation_actions(txa_action);;
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1418
CREATE INDEX tx_relation_actions_txa_public_idx ON tx_relation_actions(txa_public);;
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1419
CREATE INDEX tx_relation_actions_eid_from_idx ON tx_relation_actions(eid_from);;
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1420
CREATE INDEX tx_relation_actions_eid_to_idx ON tx_relation_actions(eid_to);;
9336
722635e530a0 [server] Add missing indices for undo support (closes #3259691)
Julien Cristau <julien.cristau@logilab.fr>
parents: 9298
diff changeset
  1421
CREATE INDEX tx_relation_actions_tx_uuid_idx ON tx_relation_actions(tx_uuid);;
9585
3f5b59527d31 [source/native] allow many eid creation per .create_eid call (closes #3526594)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9583
diff changeset
  1422
""" % (helper.sql_create_numrange('entities_id_seq').replace(';', ';;'),
9450
af4b93bc38a5 [multi-sources-removal] Drop deleted_entities system table and entities.mtime column
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9448
diff changeset
  1423
       typemap['Datetime'],
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1424
       typemap['Boolean'], typemap['Bytes'], typemap['Boolean'])
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1425
    if helper.backend_name == 'sqlite':
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1426
        # sqlite support the ON DELETE CASCADE syntax but do nothing
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1427
        schema += '''
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1428
CREATE TRIGGER fkd_transactions
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1429
BEFORE DELETE ON transactions
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1430
FOR EACH ROW BEGIN
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1431
    DELETE FROM tx_entity_actions WHERE tx_uuid=OLD.tx_uuid;
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1432
    DELETE FROM tx_relation_actions WHERE tx_uuid=OLD.tx_uuid;
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1433
END;;
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1434
'''
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1435
    return schema
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1436
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1437
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1438
def sql_drop_schema(driver):
4831
c5aec27c1bf7 [repo] use logilab.db instead of lgc.adbh/lgc.db/lgc.sqlgen/indexer, test new date extranction functions
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4829
diff changeset
  1439
    helper = get_db_helper(driver)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1440
    return """
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1441
%s
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1442
DROP TABLE entities;
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1443
DROP TABLE tx_entity_actions;
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1444
DROP TABLE tx_relation_actions;
5214
3285b6e3b930 fix cwctl db-init -d on SQL Server
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5168
diff changeset
  1445
DROP TABLE transactions;
9585
3f5b59527d31 [source/native] allow many eid creation per .create_eid call (closes #3526594)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9583
diff changeset
  1446
""" % helper.sql_drop_numrange('entities_id_seq')
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1447
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1448
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1449
def grant_schema(user, set_owner=True):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1450
    result = ''
9450
af4b93bc38a5 [multi-sources-removal] Drop deleted_entities system table and entities.mtime column
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9448
diff changeset
  1451
    for table in ('entities', 'entities_id_seq',
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1452
                  'transactions', 'tx_entity_actions', 'tx_relation_actions'):
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1453
        if set_owner:
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1454
            result = 'ALTER TABLE %s OWNER TO %s;\n' % (table, user)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4902
diff changeset
  1455
        result += 'GRANT ALL ON %s TO %s;\n' % (table, user)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
  1456
    return result
3647
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1457
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1458
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1459
class BaseAuthentifier(object):
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1460
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1461
    def __init__(self, source=None):
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1462
        self.source = source
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1463
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1464
    def set_schema(self, schema):
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1465
        """set the instance'schema"""
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1466
        pass
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1467
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1468
class LoginPasswordAuthentifier(BaseAuthentifier):
8520
fcd048fa6e6d [source/native] only system source users should be authenticated by the native source (closes #2465883)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 8508
diff changeset
  1469
    passwd_rql = 'Any P WHERE X is CWUser, X login %(login)s, X upassword P'
fcd048fa6e6d [source/native] only system source users should be authenticated by the native source (closes #2465883)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 8508
diff changeset
  1470
    auth_rql = ('Any X WHERE X is CWUser, X login %(login)s, X upassword %(pwd)s, '
fcd048fa6e6d [source/native] only system source users should be authenticated by the native source (closes #2465883)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 8508
diff changeset
  1471
                'X cw_source S, S name "system"')
fcd048fa6e6d [source/native] only system source users should be authenticated by the native source (closes #2465883)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 8508
diff changeset
  1472
    _sols = ({'X': 'CWUser', 'P': 'Password', 'S': 'CWSource'},)
3647
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1473
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1474
    def set_schema(self, schema):
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1475
        """set the instance'schema"""
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1476
        if 'CWUser' in schema: # probably an empty schema if not true...
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1477
            # rql syntax trees used to authenticate users
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1478
            self._passwd_rqlst = self.source.compile_rql(self.passwd_rql, self._sols)
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1479
            self._auth_rqlst = self.source.compile_rql(self.auth_rql, self._sols)
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1480
9512
88dc96fc9fc1 [server] use a connection instead of a session for user authentication
Julien Cristau <julien.cristau@logilab.fr>
parents: 9487
diff changeset
  1481
    def authenticate(self, cnx, login, password=None, **kwargs):
3647
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1482
        """return CWUser eid for the given login/password if this account is
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1483
        defined in this source, else raise `AuthenticationError`
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1484
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1485
        two queries are needed since passwords are stored crypted, so we have
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1486
        to fetch the salt first
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1487
        """
6018
f4d1d5d9ccbb [security] don't put uncrypted password in query parameters, else it may be logged on error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5975
diff changeset
  1488
        args = {'login': login, 'pwd' : None}
3647
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1489
        if password is not None:
9512
88dc96fc9fc1 [server] use a connection instead of a session for user authentication
Julien Cristau <julien.cristau@logilab.fr>
parents: 9487
diff changeset
  1490
            rset = self.source.syntax_tree_search(cnx, self._passwd_rqlst, args)
3647
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1491
            try:
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1492
                pwd = rset[0][0]
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1493
            except IndexError:
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1494
                raise AuthenticationError('bad login')
8235
c2a91d6639d8 [system source] fix authentication: don't allow login whatever the given password if user password is NULL in database. Closes #2186099
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8205
diff changeset
  1495
            if pwd is None:
c2a91d6639d8 [system source] fix authentication: don't allow login whatever the given password if user password is NULL in database. Closes #2186099
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8205
diff changeset
  1496
                # if pwd is None but a password is provided, something is wrong
c2a91d6639d8 [system source] fix authentication: don't allow login whatever the given password if user password is NULL in database. Closes #2186099
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8205
diff changeset
  1497
                raise AuthenticationError('bad password')
3647
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1498
            # passwords are stored using the Bytes type, so we get a StringIO
8317
9c59258e7798 [security] use a stronger encryption algorythm for password, keeping bw compat
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8235
diff changeset
  1499
            args['pwd'] = Binary(crypt_password(password, pwd.getvalue()))
3647
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1500
        # get eid from login and (crypted) password
9512
88dc96fc9fc1 [server] use a connection instead of a session for user authentication
Julien Cristau <julien.cristau@logilab.fr>
parents: 9487
diff changeset
  1501
        rset = self.source.syntax_tree_search(cnx, self._auth_rqlst, args)
10037
6b83ceda8323 [server] fix LoginPasswordAuthentifier.authenticate
David Douard <david.douard@logilab.fr>
parents: 9996
diff changeset
  1502
        pwd = args['pwd']
3647
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1503
        try:
8546
3d2038d6f20d [sources/native] automatically update passwords using deprecated hashes on login
Julien Cristau <julien.cristau@logilab.fr>
parents: 8520
diff changeset
  1504
            user = rset[0][0]
3d2038d6f20d [sources/native] automatically update passwords using deprecated hashes on login
Julien Cristau <julien.cristau@logilab.fr>
parents: 8520
diff changeset
  1505
            # If the stored hash uses a deprecated scheme (e.g. DES or MD5 used
3d2038d6f20d [sources/native] automatically update passwords using deprecated hashes on login
Julien Cristau <julien.cristau@logilab.fr>
parents: 8520
diff changeset
  1506
            # before 3.14.7), update with a fresh one
10037
6b83ceda8323 [server] fix LoginPasswordAuthentifier.authenticate
David Douard <david.douard@logilab.fr>
parents: 9996
diff changeset
  1507
            if pwd is not None and pwd.getvalue():
8546
3d2038d6f20d [sources/native] automatically update passwords using deprecated hashes on login
Julien Cristau <julien.cristau@logilab.fr>
parents: 8520
diff changeset
  1508
                verify, newhash = verify_and_update(password, pwd.getvalue())
3d2038d6f20d [sources/native] automatically update passwords using deprecated hashes on login
Julien Cristau <julien.cristau@logilab.fr>
parents: 8520
diff changeset
  1509
                if not verify: # should not happen, but...
3d2038d6f20d [sources/native] automatically update passwords using deprecated hashes on login
Julien Cristau <julien.cristau@logilab.fr>
parents: 8520
diff changeset
  1510
                    raise AuthenticationError('bad password')
3d2038d6f20d [sources/native] automatically update passwords using deprecated hashes on login
Julien Cristau <julien.cristau@logilab.fr>
parents: 8520
diff changeset
  1511
                if newhash:
9512
88dc96fc9fc1 [server] use a connection instead of a session for user authentication
Julien Cristau <julien.cristau@logilab.fr>
parents: 9487
diff changeset
  1512
                    cnx.system_sql("UPDATE %s SET %s=%%(newhash)s WHERE %s=%%(login)s" % (
8546
3d2038d6f20d [sources/native] automatically update passwords using deprecated hashes on login
Julien Cristau <julien.cristau@logilab.fr>
parents: 8520
diff changeset
  1513
                                        SQL_PREFIX + 'CWUser',
3d2038d6f20d [sources/native] automatically update passwords using deprecated hashes on login
Julien Cristau <julien.cristau@logilab.fr>
parents: 8520
diff changeset
  1514
                                        SQL_PREFIX + 'upassword',
3d2038d6f20d [sources/native] automatically update passwords using deprecated hashes on login
Julien Cristau <julien.cristau@logilab.fr>
parents: 8520
diff changeset
  1515
                                        SQL_PREFIX + 'login'),
3d2038d6f20d [sources/native] automatically update passwords using deprecated hashes on login
Julien Cristau <julien.cristau@logilab.fr>
parents: 8520
diff changeset
  1516
                                       {'newhash': self.source._binary(newhash),
3d2038d6f20d [sources/native] automatically update passwords using deprecated hashes on login
Julien Cristau <julien.cristau@logilab.fr>
parents: 8520
diff changeset
  1517
                                        'login': login})
9512
88dc96fc9fc1 [server] use a connection instead of a session for user authentication
Julien Cristau <julien.cristau@logilab.fr>
parents: 9487
diff changeset
  1518
                    cnx.commit(free_cnxset=False)
8546
3d2038d6f20d [sources/native] automatically update passwords using deprecated hashes on login
Julien Cristau <julien.cristau@logilab.fr>
parents: 8520
diff changeset
  1519
            return user
3647
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1520
        except IndexError:
2941f4a0aab9 refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3503
diff changeset
  1521
            raise AuthenticationError('bad password')
6849
5a0c2cfc19bf [repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
  1522
5a0c2cfc19bf [repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
  1523
5a0c2cfc19bf [repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
  1524
class EmailPasswordAuthentifier(BaseAuthentifier):
9512
88dc96fc9fc1 [server] use a connection instead of a session for user authentication
Julien Cristau <julien.cristau@logilab.fr>
parents: 9487
diff changeset
  1525
    def authenticate(self, cnx, login, **authinfo):
6849
5a0c2cfc19bf [repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
  1526
        # email_auth flag prevent from infinite recursion (call to
5a0c2cfc19bf [repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
  1527
        # repo.check_auth_info at the end of this method may lead us here again)
5a0c2cfc19bf [repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
  1528
        if not '@' in login or authinfo.pop('email_auth', None):
5a0c2cfc19bf [repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
  1529
            raise AuthenticationError('not an email')
9512
88dc96fc9fc1 [server] use a connection instead of a session for user authentication
Julien Cristau <julien.cristau@logilab.fr>
parents: 9487
diff changeset
  1530
        rset = cnx.execute('Any L WHERE U login L, U primary_email M, '
6849
5a0c2cfc19bf [repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
  1531
                               'M address %(login)s', {'login': login},
5a0c2cfc19bf [repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
  1532
                               build_descr=False)
5a0c2cfc19bf [repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
  1533
        if rset.rowcount != 1:
5a0c2cfc19bf [repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
  1534
            raise AuthenticationError('unexisting email')
5a0c2cfc19bf [repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
  1535
        login = rset.rows[0][0]
5a0c2cfc19bf [repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
  1536
        authinfo['email_auth'] = True
9512
88dc96fc9fc1 [server] use a connection instead of a session for user authentication
Julien Cristau <julien.cristau@logilab.fr>
parents: 9487
diff changeset
  1537
        return self.source.repo.check_auth_info(cnx, login, authinfo)
7342
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1538
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1539
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1540
class DatabaseIndependentBackupRestore(object):
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1541
    """Helper class to perform db backend agnostic backup and restore
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1542
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1543
    The backup and restore methods are used to dump / restore the
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1544
    system database in a database independent format. The file is a
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1545
    Zip archive containing the following files:
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1546
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1547
    * format.txt: the format of the archive. Currently '1.0'
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1548
    * tables.txt: list of filenames in the archive tables/ directory
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1549
    * sequences.txt: list of filenames in the archive sequences/ directory
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1550
    * versions.txt: the list of cube versions from CWProperty
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1551
    * tables/<tablename>.<chunkno>: pickled data
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1552
    * sequences/<sequencename>: pickled data
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1553
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1554
    The pickled data format for tables and sequences is a tuple of 3 elements:
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1555
    * the table name
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1556
    * a tuple of column names
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1557
    * a list of rows (as tuples with one element per column)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1558
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1559
    Tables are saved in chunks in different files in order to prevent
7551
2d4ba5b984dc cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7530
diff changeset
  1560
    a too high memory consumption.
7342
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1561
    """
10048
7d9256aab337 [source/native/backup_restore] have a unique tunable blocksize for the dump phase
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9336
diff changeset
  1562
    blocksize = 100
7d9256aab337 [source/native/backup_restore] have a unique tunable blocksize for the dump phase
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9336
diff changeset
  1563
7342
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1564
    def __init__(self, source):
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1565
        """
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1566
        :param: source an instance of the system source
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1567
        """
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1568
        self._source = source
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1569
        self.logger = logging.getLogger('cubicweb.ctl')
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1570
        self.logger.setLevel(logging.INFO)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1571
        self.logger.addHandler(logging.StreamHandler(sys.stdout))
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1572
        self.schema = self._source.schema
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1573
        self.dbhelper = self._source.dbhelper
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1574
        self.cnx = None
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1575
        self.cursor = None
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1576
        self.sql_generator = sqlgen.SQLGenerator()
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1577
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1578
    def get_connection(self):
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1579
        return self._source.get_connection()
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1580
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1581
    def backup(self, backupfile):
8910
7652c3d46ba3 [portable dump] skip virtual relations, they have no table associated. Closes #2841199
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8821
diff changeset
  1582
        archive = zipfile.ZipFile(backupfile, 'w', allowZip64=True)
7342
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1583
        self.cnx = self.get_connection()
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1584
        try:
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1585
            self.cursor = self.cnx.cursor()
9468
39b7a91a3f4c [repo] pylint cleanup, mainly of imports, with a bit of style
Julien Cristau <julien.cristau@logilab.fr>
parents: 9467
diff changeset
  1586
            self.cursor.arraysize = 100
7342
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1587
            self.logger.info('writing metadata')
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1588
            self.write_metadata(archive)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1589
            for seq in self.get_sequences():
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1590
                self.logger.info('processing sequence %s', seq)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1591
                self.write_sequence(archive, seq)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1592
            for table in self.get_tables():
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1593
                self.logger.info('processing table %s', table)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1594
                self.write_table(archive, table)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1595
        finally:
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1596
            archive.close()
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1597
            self.cnx.close()
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1598
        self.logger.info('done')
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1599
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1600
    def get_tables(self):
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1601
        non_entity_tables = ['entities',
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1602
                             'transactions',
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1603
                             'tx_entity_actions',
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1604
                             'tx_relation_actions',
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1605
                             ]
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1606
        etype_tables = []
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1607
        relation_tables = []
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1608
        prefix = 'cw_'
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1609
        for etype in self.schema.entities():
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1610
            eschema = self.schema.eschema(etype)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1611
            if eschema.final:
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1612
                continue
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1613
            etype_tables.append('%s%s'%(prefix, etype))
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1614
        for rtype in self.schema.relations():
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1615
            rschema = self.schema.rschema(rtype)
8910
7652c3d46ba3 [portable dump] skip virtual relations, they have no table associated. Closes #2841199
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8821
diff changeset
  1616
            if rschema.final or rschema.inlined or rschema in VIRTUAL_RTYPES:
7342
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1617
                continue
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1618
            relation_tables.append('%s_relation' % rtype)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1619
        return non_entity_tables + etype_tables + relation_tables
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1620
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1621
    def get_sequences(self):
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1622
        return ['entities_id_seq']
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1623
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1624
    def write_metadata(self, archive):
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1625
        archive.writestr('format.txt', '1.0')
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1626
        archive.writestr('tables.txt', '\n'.join(self.get_tables()))
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1627
        archive.writestr('sequences.txt', '\n'.join(self.get_sequences()))
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1628
        versions = self._get_versions()
9468
39b7a91a3f4c [repo] pylint cleanup, mainly of imports, with a bit of style
Julien Cristau <julien.cristau@logilab.fr>
parents: 9467
diff changeset
  1629
        versions_str = '\n'.join('%s %s' % (k, v)
39b7a91a3f4c [repo] pylint cleanup, mainly of imports, with a bit of style
Julien Cristau <julien.cristau@logilab.fr>
parents: 9467
diff changeset
  1630
                                 for k, v in versions)
7342
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1631
        archive.writestr('versions.txt', versions_str)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1632
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1633
    def write_sequence(self, archive, seq):
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1634
        sql = self.dbhelper.sql_sequence_current_state(seq)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1635
        columns, rows_iterator = self._get_cols_and_rows(sql)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1636
        rows = list(rows_iterator)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1637
        serialized = self._serialize(seq, columns, rows)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1638
        archive.writestr('sequences/%s' % seq, serialized)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1639
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1640
    def write_table(self, archive, table):
7753
681ef2a664dd work around cursor.rowcount not returning anything useful before 1st fetch with mssql (closes #1910869)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7551
diff changeset
  1641
        nb_lines_sql = 'SELECT COUNT(*) FROM %s' % table
681ef2a664dd work around cursor.rowcount not returning anything useful before 1st fetch with mssql (closes #1910869)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7551
diff changeset
  1642
        self.cursor.execute(nb_lines_sql)
681ef2a664dd work around cursor.rowcount not returning anything useful before 1st fetch with mssql (closes #1910869)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7551
diff changeset
  1643
        rowcount = self.cursor.fetchone()[0]
7342
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1644
        sql = 'SELECT * FROM %s' % table
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1645
        columns, rows_iterator = self._get_cols_and_rows(sql)
7753
681ef2a664dd work around cursor.rowcount not returning anything useful before 1st fetch with mssql (closes #1910869)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7551
diff changeset
  1646
        self.logger.info('number of rows: %d', rowcount)
10048
7d9256aab337 [source/native/backup_restore] have a unique tunable blocksize for the dump phase
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9336
diff changeset
  1647
        blocksize = self.blocksize
7753
681ef2a664dd work around cursor.rowcount not returning anything useful before 1st fetch with mssql (closes #1910869)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7551
diff changeset
  1648
        if rowcount > 0:
681ef2a664dd work around cursor.rowcount not returning anything useful before 1st fetch with mssql (closes #1910869)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7551
diff changeset
  1649
            for i, start in enumerate(xrange(0, rowcount, blocksize)):
7342
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1650
                rows = list(itertools.islice(rows_iterator, blocksize))
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1651
                serialized = self._serialize(table, columns, rows)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1652
                archive.writestr('tables/%s.%04d' % (table, i), serialized)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1653
                self.logger.debug('wrote rows %d to %d (out of %d) to %s.%04d',
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1654
                                  start, start+len(rows)-1,
7753
681ef2a664dd work around cursor.rowcount not returning anything useful before 1st fetch with mssql (closes #1910869)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7551
diff changeset
  1655
                                  rowcount,
7342
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1656
                                  table, i)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1657
        else:
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1658
            rows = []
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1659
            serialized = self._serialize(table, columns, rows)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1660
            archive.writestr('tables/%s.%04d' % (table, 0), serialized)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1661
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1662
    def _get_cols_and_rows(self, sql):
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1663
        process_result = self._source.iter_process_result
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1664
        self.cursor.execute(sql)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1665
        columns = (d[0] for d in self.cursor.description)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1666
        rows = process_result(self.cursor)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1667
        return tuple(columns), rows
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1668
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1669
    def _serialize(self, name, columns, rows):
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1670
        return dumps((name, columns, rows), pickle.HIGHEST_PROTOCOL)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1671
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1672
    def restore(self, backupfile):
7787
d9607ae447d2 [server] portable dump format supports now ZIP64 extensions by default (closes #1912535)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7786
diff changeset
  1673
        archive = zipfile.ZipFile(backupfile, 'r', allowZip64=True)
7342
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1674
        self.cnx = self.get_connection()
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1675
        self.cursor = self.cnx.cursor()
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1676
        sequences, tables, table_chunks = self.read_metadata(archive, backupfile)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1677
        for seq in sequences:
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1678
            self.logger.info('restoring sequence %s', seq)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1679
            self.read_sequence(archive, seq)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1680
        for table in tables:
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1681
            self.logger.info('restoring table %s', table)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1682
            self.read_table(archive, table, sorted(table_chunks[table]))
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1683
        self.cnx.close()
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1684
        archive.close()
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1685
        self.logger.info('done')
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1686
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1687
    def read_metadata(self, archive, backupfile):
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1688
        formatinfo = archive.read('format.txt')
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1689
        self.logger.info('checking metadata')
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1690
        if formatinfo.strip() != "1.0":
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1691
            self.logger.critical('Unsupported format in archive: %s', formatinfo)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1692
            raise ValueError('Unknown format in %s: %s' % (backupfile, formatinfo))
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1693
        tables = archive.read('tables.txt').splitlines()
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1694
        sequences = archive.read('sequences.txt').splitlines()
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1695
        file_versions = self._parse_versions(archive.read('versions.txt'))
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1696
        versions = set(self._get_versions())
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1697
        if file_versions != versions:
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1698
            self.logger.critical('Unable to restore : versions do not match')
8379
1a3746ec4d65 [sources/native] do not let a log message crash on us
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 8317
diff changeset
  1699
            self.logger.critical('Expected:\n%s', '\n'.join('%s : %s' % (cube, ver)
1a3746ec4d65 [sources/native] do not let a log message crash on us
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 8317
diff changeset
  1700
                                                            for cube, ver in sorted(versions)))
1a3746ec4d65 [sources/native] do not let a log message crash on us
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 8317
diff changeset
  1701
            self.logger.critical('Found:\n%s', '\n'.join('%s : %s' % (cube, ver)
1a3746ec4d65 [sources/native] do not let a log message crash on us
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 8317
diff changeset
  1702
                                                         for cube, ver in sorted(file_versions)))
7342
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1703
            raise ValueError('Unable to restore : versions do not match')
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1704
        table_chunks = {}
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1705
        for name in archive.namelist():
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1706
            if not name.startswith('tables/'):
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1707
                continue
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1708
            filename = basename(name)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1709
            tablename, _ext = filename.rsplit('.', 1)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1710
            table_chunks.setdefault(tablename, []).append(name)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1711
        return sequences, tables, table_chunks
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1712
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1713
    def read_sequence(self, archive, seq):
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1714
        seqname, columns, rows = loads(archive.read('sequences/%s' % seq))
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1715
        assert seqname == seq
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1716
        assert len(rows) == 1
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1717
        assert len(rows[0]) == 1
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1718
        value = rows[0][0]
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1719
        sql = self.dbhelper.sql_restart_sequence(seq, value)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1720
        self.cursor.execute(sql)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1721
        self.cnx.commit()
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1722
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1723
    def read_table(self, archive, table, filenames):
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1724
        merge_args = self._source.merge_args
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1725
        self.cursor.execute('DELETE FROM %s' % table)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1726
        self.cnx.commit()
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1727
        row_count = 0
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1728
        for filename in filenames:
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1729
            tablename, columns, rows = loads(archive.read(filename))
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1730
            assert tablename == table
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1731
            if not rows:
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1732
                continue
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1733
            insert = self.sql_generator.insert(table,
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1734
                                               dict(zip(columns, rows[0])))
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1735
            for row in rows:
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1736
                self.cursor.execute(insert, merge_args(dict(zip(columns, row)), {}))
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1737
            row_count += len(rows)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1738
            self.cnx.commit()
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1739
        self.logger.info('inserted %d rows', row_count)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1740
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1741
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1742
    def _parse_versions(self, version_str):
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1743
        versions = set()
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1744
        for line in version_str.splitlines():
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1745
            versions.add(tuple(line.split()))
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1746
        return versions
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1747
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1748
    def _get_versions(self):
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1749
        version_sql = 'SELECT cw_pkey, cw_value FROM cw_CWProperty'
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1750
        versions = []
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1751
        self.cursor.execute(version_sql)
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1752
        for pkey, value in self.cursor.fetchall():
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1753
            if pkey.startswith(u'system.version'):
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1754
                versions.append((pkey, value))
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7243
diff changeset
  1755
        return versions