author | Christophe de Vienne <christophe@unlish.com> |
Tue, 09 Dec 2014 15:18:58 +0100 | |
changeset 10113 | e1e8df594d58 |
parent 10076 | 3810332ef42c |
child 10136 | c33dcd18e0bb |
child 10178 | 4b33246cb43c |
permissions | -rw-r--r-- |
9583
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
1 |
# copyright 2003-2014 LOGILAB S.A. (Paris, FRANCE), all rights reserved. |
5421
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5413
diff
changeset
|
2 |
# contact http://www.logilab.fr/ -- mailto:contact@logilab.fr |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5413
diff
changeset
|
3 |
# |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5413
diff
changeset
|
4 |
# This file is part of CubicWeb. |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5413
diff
changeset
|
5 |
# |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5413
diff
changeset
|
6 |
# CubicWeb is free software: you can redistribute it and/or modify it under the |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5413
diff
changeset
|
7 |
# terms of the GNU Lesser General Public License as published by the Free |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5413
diff
changeset
|
8 |
# Software Foundation, either version 2.1 of the License, or (at your option) |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5413
diff
changeset
|
9 |
# any later version. |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5413
diff
changeset
|
10 |
# |
5424
8ecbcbff9777
replace logilab-common by CubicWeb in disclaimer
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5421
diff
changeset
|
11 |
# CubicWeb is distributed in the hope that it will be useful, but WITHOUT |
5421
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5413
diff
changeset
|
12 |
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5413
diff
changeset
|
13 |
# FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5413
diff
changeset
|
14 |
# details. |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5413
diff
changeset
|
15 |
# |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5413
diff
changeset
|
16 |
# You should have received a copy of the GNU Lesser General Public License along |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5413
diff
changeset
|
17 |
# with CubicWeb. If not, see <http://www.gnu.org/licenses/>. |
0 | 18 |
"""Adapters for native cubicweb sources. |
19 |
||
1952
8e19c813750d
fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1792
diff
changeset
|
20 |
Notes: |
8e19c813750d
fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1792
diff
changeset
|
21 |
* extid (aka external id, the primary key of an entity in the external source |
8e19c813750d
fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1792
diff
changeset
|
22 |
from which it comes from) are stored in a varchar column encoded as a base64 |
8e19c813750d
fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1792
diff
changeset
|
23 |
string. This is because it should actually be Bytes but we want an index on |
8e19c813750d
fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1792
diff
changeset
|
24 |
it for fast querying. |
5824 | 25 |
""" |
0 | 26 |
__docformat__ = "restructuredtext en" |
27 |
||
9932
c493767679b2
[source/native] cPickle is available in all supported pythons
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9802
diff
changeset
|
28 |
from cPickle import loads, dumps |
c493767679b2
[source/native] cPickle is available in all supported pythons
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9802
diff
changeset
|
29 |
import cPickle as pickle |
4900 | 30 |
from threading import Lock |
1016
26387b836099
use datetime instead of mx.DateTime
sylvain.thenault@logilab.fr
parents:
973
diff
changeset
|
31 |
from datetime import datetime |
1952
8e19c813750d
fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1792
diff
changeset
|
32 |
from base64 import b64decode, b64encode |
4964
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
33 |
from contextlib import contextmanager |
9468
39b7a91a3f4c
[repo] pylint cleanup, mainly of imports, with a bit of style
Julien Cristau <julien.cristau@logilab.fr>
parents:
9467
diff
changeset
|
34 |
from os.path import basename |
6211
e9d125fd1465
nicer error reporting for unique together constraints
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6201
diff
changeset
|
35 |
import re |
6889
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
36 |
import itertools |
7342
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
37 |
import zipfile |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
38 |
import logging |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
39 |
import sys |
0 | 40 |
|
4806
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
41 |
from logilab.common.decorators import cached, clear_cache |
3835
a191b3b9e455
more sensible default values to c-c "create" inputs
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3689
diff
changeset
|
42 |
from logilab.common.configuration import Method |
a191b3b9e455
more sensible default values to c-c "create" inputs
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3689
diff
changeset
|
43 |
from logilab.common.shellutils import getlogin |
7342
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
44 |
from logilab.database import get_db_helper, sqlgen |
0 | 45 |
|
5891
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
46 |
from yams import schema2sql as y2sql |
6945
28bf94d062a9
[sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6931
diff
changeset
|
47 |
from yams.schema import role_name |
5891
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
48 |
|
6945
28bf94d062a9
[sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6931
diff
changeset
|
49 |
from cubicweb import (UnknownEid, AuthenticationError, ValidationError, Binary, |
9468
39b7a91a3f4c
[repo] pylint cleanup, mainly of imports, with a bit of style
Julien Cristau <julien.cristau@logilab.fr>
parents:
9467
diff
changeset
|
50 |
UniqueTogetherError, UndoTransactionException) |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
51 |
from cubicweb import transaction as tx, server, neg_role |
7954
a3d3220669d6
[cache] replace lgc.cache with something more appropriate (closes #1921713)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
7922
diff
changeset
|
52 |
from cubicweb.utils import QueryCache |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
53 |
from cubicweb.schema import VIRTUAL_RTYPES |
3835
a191b3b9e455
more sensible default values to c-c "create" inputs
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3689
diff
changeset
|
54 |
from cubicweb.cwconfig import CubicWebNoAppConfiguration |
4806
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
55 |
from cubicweb.server import hook |
8546
3d2038d6f20d
[sources/native] automatically update passwords using deprecated hashes on login
Julien Cristau <julien.cristau@logilab.fr>
parents:
8520
diff
changeset
|
56 |
from cubicweb.server.utils import crypt_password, eschema_eid, verify_and_update |
2759
23d7a75693f8
R refactor backup and use tar.gz to store all sources
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents:
2638
diff
changeset
|
57 |
from cubicweb.server.sqlutils import SQL_PREFIX, SQLAdapterMixIn |
0 | 58 |
from cubicweb.server.rqlannotation import set_qdata |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
59 |
from cubicweb.server.hook import CleanupDeletedEidsCacheOp |
7118
e094b3d4eb95
[server] move EditedEntity class to its own module, to avoid cyclic dependency when needed from e.g. session.py
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6957
diff
changeset
|
60 |
from cubicweb.server.edition import EditedEntity |
2625
d6012db7b93e
R [server debug]Â more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2620
diff
changeset
|
61 |
from cubicweb.server.sources import AbstractSource, dbg_st_search, dbg_results |
0 | 62 |
from cubicweb.server.sources.rql2sql import SQLGenerator |
63 |
||
64 |
||
2354
9b4bac626977
ability to map attributes to something else than usual cw mapping on sql generation
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2310
diff
changeset
|
65 |
ATTR_MAP = {} |
0 | 66 |
NONSYSTEM_ETYPES = set() |
67 |
NONSYSTEM_RELATIONS = set() |
|
68 |
||
69 |
class LogCursor(object): |
|
70 |
def __init__(self, cursor): |
|
71 |
self.cu = cursor |
|
1792 | 72 |
|
0 | 73 |
def execute(self, query, args=None): |
74 |
"""Execute a query. |
|
75 |
it's a function just so that it shows up in profiling |
|
76 |
""" |
|
2593
16d9419a4a79
F: start to handle binary debug log level on the server side
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2589
diff
changeset
|
77 |
if server.DEBUG & server.DBG_SQL: |
0 | 78 |
print 'exec', query, args |
79 |
try: |
|
80 |
self.cu.execute(str(query), args) |
|
8695
358d8bed9626
[toward-py3k] rewrite to "except AnException as exc:" (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents:
8694
diff
changeset
|
81 |
except Exception as ex: |
0 | 82 |
print "sql: %r\n args: %s\ndbms message: %r" % ( |
83 |
query, args, ex.args[0]) |
|
84 |
raise |
|
1792 | 85 |
|
0 | 86 |
def fetchall(self): |
87 |
return self.cu.fetchall() |
|
1792 | 88 |
|
0 | 89 |
def fetchone(self): |
90 |
return self.cu.fetchone() |
|
1792 | 91 |
|
2625
d6012db7b93e
R [server debug]Â more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2620
diff
changeset
|
92 |
|
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
93 |
def sql_or_clauses(sql, clauses): |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
94 |
select, restr = sql.split(' WHERE ', 1) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
95 |
restrclauses = restr.split(' AND ') |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
96 |
for clause in clauses: |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
97 |
restrclauses.remove(clause) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
98 |
if restrclauses: |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
99 |
restr = '%s AND (%s)' % (' AND '.join(restrclauses), |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
100 |
' OR '.join(clauses)) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
101 |
else: |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
102 |
restr = '(%s)' % ' OR '.join(clauses) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
103 |
return '%s WHERE %s' % (select, restr) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
104 |
|
9450
af4b93bc38a5
[multi-sources-removal] Drop deleted_entities system table and entities.mtime column
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
9448
diff
changeset
|
105 |
|
5891
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
106 |
def rdef_table_column(rdef): |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
107 |
"""return table and column used to store the given relation definition in |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
108 |
the database |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
109 |
""" |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
110 |
return (SQL_PREFIX + str(rdef.subject), |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
111 |
SQL_PREFIX + str(rdef.rtype)) |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
112 |
|
9450
af4b93bc38a5
[multi-sources-removal] Drop deleted_entities system table and entities.mtime column
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
9448
diff
changeset
|
113 |
|
5891
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
114 |
def rdef_physical_info(dbhelper, rdef): |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
115 |
"""return backend type and a boolean flag if NULL values should be allowed |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
116 |
for a given relation definition |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
117 |
""" |
6359
0bff5a05385c
[sync schema] take care rdef may not be final, in which case we want to use type of eid attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6297
diff
changeset
|
118 |
if rdef.object.final: |
0bff5a05385c
[sync schema] take care rdef may not be final, in which case we want to use type of eid attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6297
diff
changeset
|
119 |
ttype = rdef.object |
0bff5a05385c
[sync schema] take care rdef may not be final, in which case we want to use type of eid attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6297
diff
changeset
|
120 |
else: |
0bff5a05385c
[sync schema] take care rdef may not be final, in which case we want to use type of eid attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6297
diff
changeset
|
121 |
ttype = 'Int' # eid type |
0bff5a05385c
[sync schema] take care rdef may not be final, in which case we want to use type of eid attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6297
diff
changeset
|
122 |
coltype = y2sql.type_from_constraints(dbhelper, ttype, |
5891
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
123 |
rdef.constraints, creating=False) |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
124 |
allownull = rdef.cardinality[0] != '1' |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
125 |
return coltype, allownull |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
126 |
|
8265
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
127 |
|
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
128 |
class _UndoException(Exception): |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
129 |
"""something went wrong during undoing""" |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
130 |
|
7530
15178bf89fb6
[server] fix unicode conversion capability in UndoException
Julien Jehannet <julien.jehannet@logilab.fr>
parents:
7507
diff
changeset
|
131 |
def __unicode__(self): |
15178bf89fb6
[server] fix unicode conversion capability in UndoException
Julien Jehannet <julien.jehannet@logilab.fr>
parents:
7507
diff
changeset
|
132 |
"""Called by the unicode builtin; should return a Unicode object |
15178bf89fb6
[server] fix unicode conversion capability in UndoException
Julien Jehannet <julien.jehannet@logilab.fr>
parents:
7507
diff
changeset
|
133 |
|
8265
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
134 |
Type of _UndoException message must be `unicode` by design in CubicWeb. |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
135 |
""" |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
136 |
assert isinstance(self.args[0], unicode) |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
137 |
return self.args[0] |
7530
15178bf89fb6
[server] fix unicode conversion capability in UndoException
Julien Jehannet <julien.jehannet@logilab.fr>
parents:
7507
diff
changeset
|
138 |
|
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
139 |
|
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
140 |
def _undo_check_relation_target(tentity, rdef, role): |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
141 |
"""check linked entity has not been redirected for this relation""" |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
142 |
card = rdef.role_cardinality(role) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
143 |
if card in '?1' and tentity.related(rdef.rtype, role): |
8265
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
144 |
raise _UndoException(tentity._cw._( |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
145 |
"Can't restore %(role)s relation %(rtype)s to entity %(eid)s which " |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
146 |
"is already linked using this relation.") |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
147 |
% {'role': neg_role(role), |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
148 |
'rtype': rdef.rtype, |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
149 |
'eid': tentity.eid}) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
150 |
|
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
151 |
def _undo_rel_info(cnx, subj, rtype, obj): |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
152 |
entities = [] |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
153 |
for role, eid in (('subject', subj), ('object', obj)): |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
154 |
try: |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
155 |
entities.append(cnx.entity_from_eid(eid)) |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
156 |
except UnknownEid: |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
157 |
raise _UndoException(cnx._( |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
158 |
"Can't restore relation %(rtype)s, %(role)s entity %(eid)s" |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
159 |
" doesn't exist anymore.") |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
160 |
% {'role': cnx._(role), |
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
161 |
'rtype': cnx._(rtype), |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
162 |
'eid': eid}) |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
163 |
sentity, oentity = entities |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
164 |
try: |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
165 |
rschema = cnx.vreg.schema.rschema(rtype) |
8900
010a59e12d89
use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
8821
diff
changeset
|
166 |
rdef = rschema.rdefs[(sentity.cw_etype, oentity.cw_etype)] |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
167 |
except KeyError: |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
168 |
raise _UndoException(cnx._( |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
169 |
"Can't restore relation %(rtype)s between %(subj)s and " |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
170 |
"%(obj)s, that relation does not exists anymore in the " |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
171 |
"schema.") |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
172 |
% {'rtype': cnx._(rtype), |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
173 |
'subj': subj, |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
174 |
'obj': obj}) |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
175 |
return sentity, oentity, rdef |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
176 |
|
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
177 |
def _undo_has_later_transaction(cnx, eid): |
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
178 |
return cnx.system_sql('''\ |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
179 |
SELECT T.tx_uuid FROM transactions AS TREF, transactions AS T |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
180 |
WHERE TREF.tx_uuid='%(txuuid)s' AND T.tx_uuid!='%(txuuid)s' |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
181 |
AND T.tx_time>=TREF.tx_time |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
182 |
AND (EXISTS(SELECT 1 FROM tx_entity_actions AS TEA |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
183 |
WHERE TEA.tx_uuid=T.tx_uuid AND TEA.eid=%(eid)s) |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
184 |
OR EXISTS(SELECT 1 FROM tx_relation_actions as TRA |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
185 |
WHERE TRA.tx_uuid=T.tx_uuid AND ( |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
186 |
TRA.eid_from=%(eid)s OR TRA.eid_to=%(eid)s)) |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
187 |
)''' % {'txuuid': cnx.transaction_data['undoing_uuid'], |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
188 |
'eid': eid}).fetchone() |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
189 |
|
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
190 |
|
9583
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
191 |
class DefaultEidGenerator(object): |
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
192 |
__slots__ = ('source', 'cnx', 'lock') |
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
193 |
|
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
194 |
def __init__(self, source): |
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
195 |
self.source = source |
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
196 |
self.cnx = None |
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
197 |
self.lock = Lock() |
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
198 |
|
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
199 |
def close(self): |
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
200 |
if self.cnx: |
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
201 |
self.cnx.close() |
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
202 |
self.cnx = None |
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
203 |
|
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
204 |
def create_eid(self, _cnx, count=1): |
9583
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
205 |
# lock needed to prevent 'Connection is busy with results for another |
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
206 |
# command (0)' errors with SQLServer |
9585
3f5b59527d31
[source/native] allow many eid creation per .create_eid call (closes #3526594)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9583
diff
changeset
|
207 |
assert count > 0 |
9583
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
208 |
with self.lock: |
9585
3f5b59527d31
[source/native] allow many eid creation per .create_eid call (closes #3526594)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9583
diff
changeset
|
209 |
return self._create_eid(count) |
9583
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
210 |
|
9585
3f5b59527d31
[source/native] allow many eid creation per .create_eid call (closes #3526594)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9583
diff
changeset
|
211 |
def _create_eid(self, count): |
9583
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
212 |
# internal function doing the eid creation without locking. |
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
213 |
# needed for the recursive handling of disconnections (otherwise we |
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
214 |
# deadlock on self._eid_cnx_lock |
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
215 |
source = self.source |
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
216 |
if self.cnx is None: |
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
217 |
self.cnx = source.get_connection() |
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
218 |
cnx = self.cnx |
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
219 |
try: |
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
220 |
cursor = cnx.cursor() |
9585
3f5b59527d31
[source/native] allow many eid creation per .create_eid call (closes #3526594)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9583
diff
changeset
|
221 |
for sql in source.dbhelper.sqls_increment_numrange('entities_id_seq', count): |
9583
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
222 |
cursor.execute(sql) |
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
223 |
eid = cursor.fetchone()[0] |
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
224 |
except (source.OperationalError, source.InterfaceError): |
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
225 |
# FIXME: better detection of deconnection pb |
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
226 |
source.warning("trying to reconnect create eid connection") |
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
227 |
self.cnx = None |
9585
3f5b59527d31
[source/native] allow many eid creation per .create_eid call (closes #3526594)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9583
diff
changeset
|
228 |
return self._create_eid(count) |
9583
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
229 |
except source.DbapiError as exc: |
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
230 |
# We get this one with pyodbc and SQL Server when connection was reset |
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
231 |
if exc.args[0] == '08S01': |
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
232 |
source.warning("trying to reconnect create eid connection") |
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
233 |
self.cnx = None |
9585
3f5b59527d31
[source/native] allow many eid creation per .create_eid call (closes #3526594)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9583
diff
changeset
|
234 |
return self._create_eid(count) |
9583
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
235 |
else: |
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
236 |
raise |
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
237 |
except Exception: # WTF? |
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
238 |
cnx.rollback() |
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
239 |
self.cnx = None |
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
240 |
source.exception('create eid failed in an unforeseen way on SQL statement %s', sql) |
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
241 |
raise |
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
242 |
else: |
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
243 |
cnx.commit() |
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
244 |
return eid |
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
245 |
|
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
246 |
|
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
247 |
class SQLITEEidGenerator(object): |
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
248 |
__slots__ = ('source', 'lock') |
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
249 |
|
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
250 |
def __init__(self, source): |
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
251 |
self.source = source |
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
252 |
self.lock = Lock() |
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
253 |
|
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
254 |
def close(self): |
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
255 |
pass |
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
256 |
|
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
257 |
def create_eid(self, cnx, count=1): |
9585
3f5b59527d31
[source/native] allow many eid creation per .create_eid call (closes #3526594)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9583
diff
changeset
|
258 |
assert count > 0 |
9583
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
259 |
source = self.source |
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
260 |
with self.lock: |
9585
3f5b59527d31
[source/native] allow many eid creation per .create_eid call (closes #3526594)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9583
diff
changeset
|
261 |
for sql in source.dbhelper.sqls_increment_numrange('entities_id_seq', count): |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
262 |
cursor = source.doexec(cnx, sql) |
9583
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
263 |
return cursor.fetchone()[0] |
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
264 |
|
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
265 |
|
0 | 266 |
class NativeSQLSource(SQLAdapterMixIn, AbstractSource): |
267 |
"""adapter for source using the native cubicweb schema (see below) |
|
268 |
""" |
|
2354
9b4bac626977
ability to map attributes to something else than usual cw mapping on sql generation
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2310
diff
changeset
|
269 |
sqlgen_class = SQLGenerator |
0 | 270 |
options = ( |
271 |
('db-driver', |
|
272 |
{'type' : 'string', |
|
273 |
'default': 'postgres', |
|
5423
e15abfdcce38
backport default into stable: stable is now cw 3.8
Sylvain Thénault <sylvain.thenault@logilab.fr>
diff
changeset
|
274 |
# XXX use choice type |
e15abfdcce38
backport default into stable: stable is now cw 3.8
Sylvain Thénault <sylvain.thenault@logilab.fr>
diff
changeset
|
275 |
'help': 'database driver (postgres, mysql, sqlite, sqlserver2005)', |
6201
cf445fcaf8fe
[c-c create] ask for database driver by default
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6128
diff
changeset
|
276 |
'group': 'native-source', 'level': 0, |
0 | 277 |
}), |
278 |
('db-host', |
|
279 |
{'type' : 'string', |
|
280 |
'default': '', |
|
281 |
'help': 'database host', |
|
5323
329b4f6d18b4
[config] with lgc >= 0.50, option's dict inputlevel becomes level
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5217
diff
changeset
|
282 |
'group': 'native-source', 'level': 1, |
0 | 283 |
}), |
2566
714a8743d423
missing db-port option to source's option definitions
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2354
diff
changeset
|
284 |
('db-port', |
714a8743d423
missing db-port option to source's option definitions
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2354
diff
changeset
|
285 |
{'type' : 'string', |
714a8743d423
missing db-port option to source's option definitions
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2354
diff
changeset
|
286 |
'default': '', |
714a8743d423
missing db-port option to source's option definitions
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2354
diff
changeset
|
287 |
'help': 'database port', |
5323
329b4f6d18b4
[config] with lgc >= 0.50, option's dict inputlevel becomes level
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5217
diff
changeset
|
288 |
'group': 'native-source', 'level': 1, |
2566
714a8743d423
missing db-port option to source's option definitions
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2354
diff
changeset
|
289 |
}), |
0 | 290 |
('db-name', |
291 |
{'type' : 'string', |
|
3835
a191b3b9e455
more sensible default values to c-c "create" inputs
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3689
diff
changeset
|
292 |
'default': Method('default_instance_id'), |
0 | 293 |
'help': 'database name', |
5323
329b4f6d18b4
[config] with lgc >= 0.50, option's dict inputlevel becomes level
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5217
diff
changeset
|
294 |
'group': 'native-source', 'level': 0, |
0 | 295 |
}), |
296 |
('db-user', |
|
297 |
{'type' : 'string', |
|
3835
a191b3b9e455
more sensible default values to c-c "create" inputs
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3689
diff
changeset
|
298 |
'default': CubicWebNoAppConfiguration.mode == 'user' and getlogin() or 'cubicweb', |
0 | 299 |
'help': 'database user', |
5323
329b4f6d18b4
[config] with lgc >= 0.50, option's dict inputlevel becomes level
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5217
diff
changeset
|
300 |
'group': 'native-source', 'level': 0, |
0 | 301 |
}), |
302 |
('db-password', |
|
303 |
{'type' : 'password', |
|
304 |
'default': '', |
|
305 |
'help': 'database password', |
|
5323
329b4f6d18b4
[config] with lgc >= 0.50, option's dict inputlevel becomes level
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5217
diff
changeset
|
306 |
'group': 'native-source', 'level': 0, |
0 | 307 |
}), |
308 |
('db-encoding', |
|
309 |
{'type' : 'string', |
|
310 |
'default': 'utf8', |
|
311 |
'help': 'database encoding', |
|
5323
329b4f6d18b4
[config] with lgc >= 0.50, option's dict inputlevel becomes level
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5217
diff
changeset
|
312 |
'group': 'native-source', 'level': 1, |
0 | 313 |
}), |
5413
dc896e698ab0
added db-extra-arguments to configuration parameters for native source
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5410
diff
changeset
|
314 |
('db-extra-arguments', |
dc896e698ab0
added db-extra-arguments to configuration parameters for native source
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5410
diff
changeset
|
315 |
{'type' : 'string', |
dc896e698ab0
added db-extra-arguments to configuration parameters for native source
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5410
diff
changeset
|
316 |
'default': '', |
dc896e698ab0
added db-extra-arguments to configuration parameters for native source
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5410
diff
changeset
|
317 |
'help': 'set to "Trusted_Connection" if you are using SQLServer and ' |
dc896e698ab0
added db-extra-arguments to configuration parameters for native source
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5410
diff
changeset
|
318 |
'want trusted authentication for the database connection', |
5515
513af9be9e37
[config] rename remaining inputlevel to level in option definitions, as expected by lgc >= 0.50
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
319 |
'group': 'native-source', 'level': 2, |
5413
dc896e698ab0
added db-extra-arguments to configuration parameters for native source
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5410
diff
changeset
|
320 |
}), |
0 | 321 |
) |
1792 | 322 |
|
6427
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6426
diff
changeset
|
323 |
def __init__(self, repo, source_config, *args, **kwargs): |
0 | 324 |
SQLAdapterMixIn.__init__(self, source_config) |
3647
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
325 |
self.authentifiers = [LoginPasswordAuthentifier(self)] |
6849
5a0c2cfc19bf
[repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6724
diff
changeset
|
326 |
if repo.config['allow-email-login']: |
5a0c2cfc19bf
[repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6724
diff
changeset
|
327 |
self.authentifiers.insert(0, EmailPasswordAuthentifier(self)) |
6427
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6426
diff
changeset
|
328 |
AbstractSource.__init__(self, repo, source_config, *args, **kwargs) |
0 | 329 |
# sql generator |
6427
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6426
diff
changeset
|
330 |
self._rql_sqlgen = self.sqlgen_class(self.schema, self.dbhelper, |
4831
c5aec27c1bf7
[repo] use logilab.db instead of lgc.adbh/lgc.db/lgc.sqlgen/indexer, test new date extranction functions
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4829
diff
changeset
|
331 |
ATTR_MAP.copy()) |
0 | 332 |
# full text index helper |
4806
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
333 |
self.do_fti = not repo.config['delay-full-text-indexation'] |
0 | 334 |
# sql queries cache |
7954
a3d3220669d6
[cache] replace lgc.cache with something more appropriate (closes #1921713)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
7922
diff
changeset
|
335 |
self._cache = QueryCache(repo.config['rql-cache-size']) |
4964
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
336 |
# (etype, attr) / storage mapping |
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
337 |
self._storages = {} |
9583
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
338 |
self.binary_to_str = self.dbhelper.dbapi_module.binary_to_str |
5649
a07dee204187
fix unit tests by not using the new create_eid implementation with sqlite
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5645
diff
changeset
|
339 |
if self.dbdriver == 'sqlite': |
9583
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
340 |
self.eid_generator = SQLITEEidGenerator(self) |
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
341 |
else: |
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
342 |
self.eid_generator = DefaultEidGenerator(self) |
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
343 |
self.create_eid = self.eid_generator.create_eid |
0 | 344 |
|
6945
28bf94d062a9
[sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6931
diff
changeset
|
345 |
def check_config(self, source_entity): |
28bf94d062a9
[sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6931
diff
changeset
|
346 |
"""check configuration of source entity""" |
28bf94d062a9
[sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6931
diff
changeset
|
347 |
if source_entity.host_config: |
28bf94d062a9
[sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6931
diff
changeset
|
348 |
msg = source_entity._cw._('the system source has its configuration ' |
28bf94d062a9
[sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6931
diff
changeset
|
349 |
'stored on the file-system') |
28bf94d062a9
[sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6931
diff
changeset
|
350 |
raise ValidationError(source_entity.eid, {role_name('config', 'subject'): msg}) |
28bf94d062a9
[sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6931
diff
changeset
|
351 |
|
3647
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
352 |
def add_authentifier(self, authentifier): |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
353 |
self.authentifiers.append(authentifier) |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
354 |
authentifier.source = self |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
355 |
authentifier.set_schema(self.schema) |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
356 |
|
0 | 357 |
def reset_caches(self): |
358 |
"""method called during test to reset potential source caches""" |
|
7954
a3d3220669d6
[cache] replace lgc.cache with something more appropriate (closes #1921713)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
7922
diff
changeset
|
359 |
self._cache = QueryCache(self.repo.config['rql-cache-size']) |
1792 | 360 |
|
0 | 361 |
def clear_eid_cache(self, eid, etype): |
362 |
"""clear potential caches for the given eid""" |
|
2610
2933cc6bf9ad
[F native source] fix attempts to clear cache key
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2593
diff
changeset
|
363 |
self._cache.pop('Any X WHERE X eid %s, X is %s' % (eid, etype), None) |
0 | 364 |
self._cache.pop('Any X WHERE X eid %s' % eid, None) |
2610
2933cc6bf9ad
[F native source] fix attempts to clear cache key
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2593
diff
changeset
|
365 |
self._cache.pop('Any %s' % eid, None) |
1792 | 366 |
|
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
367 |
def sqlexec(self, cnx, sql, args=None): |
0 | 368 |
"""execute the query and return its result""" |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
369 |
return self.process_result(self.doexec(cnx, sql, args)) |
1792 | 370 |
|
7398
26695dd703d8
[repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
371 |
def init_creating(self, cnxset=None): |
0 | 372 |
# check full text index availibility |
4806
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
373 |
if self.do_fti: |
7398
26695dd703d8
[repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
374 |
if cnxset is None: |
26695dd703d8
[repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
375 |
_cnxset = self.repo._get_cnxset() |
6724
24bf6f181d0e
[pyro source] store pyro source mapping file into the database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6427
diff
changeset
|
376 |
else: |
7398
26695dd703d8
[repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
377 |
_cnxset = cnxset |
9463
d62e13eba033
[multi-sources-removal] Simplify ConnectionsSet internal structures and public methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
9452
diff
changeset
|
378 |
if not self.dbhelper.has_fti_table(_cnxset.cu): |
4825
cdd979ae1b57
fix name error fixed in wrong branch
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4818
diff
changeset
|
379 |
if not self.repo.config.creating: |
4810
3055dc7a7c88
don't issue critical warning about missing text index when we're creating the instance
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4807
diff
changeset
|
380 |
self.critical('no text index table') |
4806
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
381 |
self.do_fti = False |
7398
26695dd703d8
[repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
382 |
if cnxset is None: |
26695dd703d8
[repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
383 |
_cnxset.cnxset_freed() |
26695dd703d8
[repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
384 |
self.repo._free_cnxset(_cnxset) |
0 | 385 |
|
7342
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
386 |
def backup(self, backupfile, confirm, format='native'): |
2759
23d7a75693f8
R refactor backup and use tar.gz to store all sources
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents:
2638
diff
changeset
|
387 |
"""method called to create a backup of the source's data""" |
7342
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
388 |
if format == 'portable': |
8947
3bbd416b09ec
[repo] straightforward bootstrap sequence. Closes #2841188
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
8916
diff
changeset
|
389 |
# ensure the schema is the one stored in the database: if repository |
3bbd416b09ec
[repo] straightforward bootstrap sequence. Closes #2841188
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
8916
diff
changeset
|
390 |
# started in quick_start mode, the file system's one has been loaded |
3bbd416b09ec
[repo] straightforward bootstrap sequence. Closes #2841188
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
8916
diff
changeset
|
391 |
# so force reload |
3bbd416b09ec
[repo] straightforward bootstrap sequence. Closes #2841188
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
8916
diff
changeset
|
392 |
if self.repo.config.quick_start: |
3bbd416b09ec
[repo] straightforward bootstrap sequence. Closes #2841188
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
8916
diff
changeset
|
393 |
self.repo.set_schema(self.repo.deserialize_schema(), |
3bbd416b09ec
[repo] straightforward bootstrap sequence. Closes #2841188
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
8916
diff
changeset
|
394 |
resetvreg=False) |
7342
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
395 |
helper = DatabaseIndependentBackupRestore(self) |
7398
26695dd703d8
[repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
396 |
self.close_source_connections() |
7342
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
397 |
try: |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
398 |
helper.backup(backupfile) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
399 |
finally: |
7398
26695dd703d8
[repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
400 |
self.open_source_connections() |
7342
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
401 |
elif format == 'native': |
7398
26695dd703d8
[repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
402 |
self.close_source_connections() |
7342
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
403 |
try: |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
404 |
self.backup_to_file(backupfile, confirm) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
405 |
finally: |
7398
26695dd703d8
[repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
406 |
self.open_source_connections() |
7342
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
407 |
else: |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
408 |
raise ValueError('Unknown format %r' % format) |
2493
9806571ea790
major refactoring of database dump/restore:
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2476
diff
changeset
|
409 |
|
7342
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
410 |
|
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
411 |
def restore(self, backupfile, confirm, drop, format='native'): |
2493
9806571ea790
major refactoring of database dump/restore:
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2476
diff
changeset
|
412 |
"""method called to restore a backup of source's data""" |
7398
26695dd703d8
[repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
413 |
if self.repo.config.init_cnxset_pool: |
26695dd703d8
[repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
414 |
self.close_source_connections() |
2759
23d7a75693f8
R refactor backup and use tar.gz to store all sources
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents:
2638
diff
changeset
|
415 |
try: |
7342
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
416 |
if format == 'portable': |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
417 |
helper = DatabaseIndependentBackupRestore(self) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
418 |
helper.restore(backupfile) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
419 |
elif format == 'native': |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
420 |
self.restore_from_file(backupfile, confirm, drop=drop) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
421 |
else: |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
422 |
raise ValueError('Unknown format %r' % format) |
2759
23d7a75693f8
R refactor backup and use tar.gz to store all sources
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents:
2638
diff
changeset
|
423 |
finally: |
7398
26695dd703d8
[repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
424 |
if self.repo.config.init_cnxset_pool: |
26695dd703d8
[repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
425 |
self.open_source_connections() |
2493
9806571ea790
major refactoring of database dump/restore:
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2476
diff
changeset
|
426 |
|
7342
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
427 |
|
6957
ffda12be2e9f
[repository] #1460066: backport datafeed cube as cubicweb source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6945
diff
changeset
|
428 |
def init(self, activated, source_entity): |
7543
570522300e22
[ms, entity metas] add 'actual source' to entities table / base entity metadata cache. Closes #1767090
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7536
diff
changeset
|
429 |
try: |
7786
18a366267612
[repo] use smarter query to check asource presence on startup (closes #1922099)
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
7754
diff
changeset
|
430 |
# test if 'asource' column exists |
7839
daf46963f4fe
ensure the test for asource column won't crash on SQLServer (closes #1949621)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7787
diff
changeset
|
431 |
query = self.dbhelper.sql_add_limit_offset('SELECT asource FROM entities', 1) |
daf46963f4fe
ensure the test for asource column won't crash on SQLServer (closes #1949621)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7787
diff
changeset
|
432 |
source_entity._cw.system_sql(query) |
8695
358d8bed9626
[toward-py3k] rewrite to "except AnException as exc:" (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents:
8694
diff
changeset
|
433 |
except Exception as ex: |
7543
570522300e22
[ms, entity metas] add 'actual source' to entities table / base entity metadata cache. Closes #1767090
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7536
diff
changeset
|
434 |
self.eid_type_source = self.eid_type_source_pre_131 |
9131
b3ad80aa645f
fix migration from pre-3.13.1 versions (closes #2846978)
Julien Cristau <julien.cristau@logilab.fr>
parents:
9130
diff
changeset
|
435 |
super(NativeSQLSource, self).init(activated, source_entity) |
b3ad80aa645f
fix migration from pre-3.13.1 versions (closes #2846978)
Julien Cristau <julien.cristau@logilab.fr>
parents:
9130
diff
changeset
|
436 |
self.init_creating(source_entity._cw.cnxset) |
1792 | 437 |
|
5749
b4393b681f7a
[repo] on repository shutdown, we've to close the new eid creation connection
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5729
diff
changeset
|
438 |
def shutdown(self): |
9583
e337a9f658e0
[source/native] refactor eid generation code
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9574
diff
changeset
|
439 |
self.eid_generator.close() |
5749
b4393b681f7a
[repo] on repository shutdown, we've to close the new eid creation connection
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5729
diff
changeset
|
440 |
|
9492
c7fc56eecd1a
English typography
Dimitri Papadopoulos <dimitri.papadopoulos@cea.fr>
parents:
9375
diff
changeset
|
441 |
# XXX deprecates [un]map_attribute? |
5013
ad91f93bbb93
[source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5011
diff
changeset
|
442 |
def map_attribute(self, etype, attr, cb, sourcedb=True): |
ad91f93bbb93
[source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5011
diff
changeset
|
443 |
self._rql_sqlgen.attr_map['%s.%s' % (etype, attr)] = (cb, sourcedb) |
2493
9806571ea790
major refactoring of database dump/restore:
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2476
diff
changeset
|
444 |
|
4512
e7ac20bf3629
unset_attribute_storage, for testing purpose at least
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4252
diff
changeset
|
445 |
def unmap_attribute(self, etype, attr): |
e7ac20bf3629
unset_attribute_storage, for testing purpose at least
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4252
diff
changeset
|
446 |
self._rql_sqlgen.attr_map.pop('%s.%s' % (etype, attr), None) |
e7ac20bf3629
unset_attribute_storage, for testing purpose at least
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4252
diff
changeset
|
447 |
|
4964
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
448 |
def set_storage(self, etype, attr, storage): |
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
449 |
storage_dict = self._storages.setdefault(etype, {}) |
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
450 |
storage_dict[attr] = storage |
5013
ad91f93bbb93
[source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5011
diff
changeset
|
451 |
self.map_attribute(etype, attr, |
ad91f93bbb93
[source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5011
diff
changeset
|
452 |
storage.callback, storage.is_source_callback) |
4964
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
453 |
|
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
454 |
def unset_storage(self, etype, attr): |
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
455 |
self._storages[etype].pop(attr) |
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
456 |
# if etype has no storage left, remove the entry |
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
457 |
if not self._storages[etype]: |
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
458 |
del self._storages[etype] |
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
459 |
self.unmap_attribute(etype, attr) |
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
460 |
|
5397
cdbf823450aa
[bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5323
diff
changeset
|
461 |
def storage(self, etype, attr): |
cdbf823450aa
[bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5323
diff
changeset
|
462 |
"""return the storage for the given entity type / attribute |
cdbf823450aa
[bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5323
diff
changeset
|
463 |
""" |
cdbf823450aa
[bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5323
diff
changeset
|
464 |
try: |
cdbf823450aa
[bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5323
diff
changeset
|
465 |
return self._storages[etype][attr] |
cdbf823450aa
[bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5323
diff
changeset
|
466 |
except KeyError: |
cdbf823450aa
[bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5323
diff
changeset
|
467 |
raise Exception('no custom storage set for %s.%s' % (etype, attr)) |
cdbf823450aa
[bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5323
diff
changeset
|
468 |
|
0 | 469 |
# ISource interface ####################################################### |
470 |
||
3647
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
471 |
def compile_rql(self, rql, sols): |
3240
8604a15995d1
refactor so that rql rewriter may be used outside the server. Enhance it to be usable for RRQLExpression as well
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3041
diff
changeset
|
472 |
rqlst = self.repo.vreg.rqlhelper.parse(rql) |
0 | 473 |
rqlst.restricted_vars = () |
3647
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
474 |
rqlst.children[0].solutions = sols |
0 | 475 |
self.repo.querier.sqlgen_annotate(rqlst) |
438 | 476 |
set_qdata(self.schema.rschema, rqlst, ()) |
0 | 477 |
return rqlst |
1792 | 478 |
|
0 | 479 |
def set_schema(self, schema): |
2476
1294a6bdf3bf
application -> instance where it makes sense
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2354
diff
changeset
|
480 |
"""set the instance'schema""" |
7954
a3d3220669d6
[cache] replace lgc.cache with something more appropriate (closes #1921713)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
7922
diff
changeset
|
481 |
self._cache = QueryCache(self.repo.config['rql-cache-size']) |
0 | 482 |
self.cache_hit, self.cache_miss, self.no_cache = 0, 0, 0 |
483 |
self.schema = schema |
|
484 |
try: |
|
485 |
self._rql_sqlgen.schema = schema |
|
486 |
except AttributeError: |
|
487 |
pass # __init__ |
|
3647
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
488 |
for authentifier in self.authentifiers: |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
489 |
authentifier.set_schema(self.schema) |
4806
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
490 |
clear_cache(self, 'need_fti_indexation') |
1792 | 491 |
|
0 | 492 |
def support_entity(self, etype, write=False): |
493 |
"""return true if the given entity's type is handled by this adapter |
|
494 |
if write is true, return true only if it's a RW support |
|
495 |
""" |
|
496 |
return not etype in NONSYSTEM_ETYPES |
|
1792 | 497 |
|
0 | 498 |
def support_relation(self, rtype, write=False): |
499 |
"""return true if the given relation's type is handled by this adapter |
|
500 |
if write is true, return true only if it's a RW support |
|
501 |
""" |
|
502 |
if write: |
|
503 |
return not rtype in NONSYSTEM_RELATIONS |
|
504 |
# due to current multi-sources implementation, the system source |
|
1792 | 505 |
# can't claim not supporting a relation |
0 | 506 |
return True #not rtype == 'content_for' |
507 |
||
9512
88dc96fc9fc1
[server] use a connection instead of a session for user authentication
Julien Cristau <julien.cristau@logilab.fr>
parents:
9487
diff
changeset
|
508 |
def authenticate(self, cnx, login, **kwargs): |
3647
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
509 |
"""return CWUser eid for the given login and other authentication |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
510 |
information found in kwargs, else raise `AuthenticationError` |
0 | 511 |
""" |
3647
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
512 |
for authentifier in self.authentifiers: |
0 | 513 |
try: |
9512
88dc96fc9fc1
[server] use a connection instead of a session for user authentication
Julien Cristau <julien.cristau@logilab.fr>
parents:
9487
diff
changeset
|
514 |
return authentifier.authenticate(cnx, login, **kwargs) |
3647
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
515 |
except AuthenticationError: |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
516 |
continue |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
517 |
raise AuthenticationError() |
1792 | 518 |
|
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
519 |
def syntax_tree_search(self, cnx, union, args=None, cachekey=None, |
0 | 520 |
varmap=None): |
521 |
"""return result from this source for a rql query (actually from |
|
522 |
a rql syntax tree and a solution dictionary mapping each used |
|
523 |
variable to a possible type). If cachekey is given, the query |
|
524 |
necessary to fetch the results (but not the results themselves) |
|
525 |
may be cached using this key. |
|
526 |
""" |
|
2625
d6012db7b93e
R [server debug]Â more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2620
diff
changeset
|
527 |
assert dbg_st_search(self.uri, union, varmap, args, cachekey) |
0 | 528 |
# remember number of actually selected term (sql generation may append some) |
529 |
if cachekey is None: |
|
530 |
self.no_cache += 1 |
|
531 |
# generate sql query if we are able to do so (not supported types...) |
|
5013
ad91f93bbb93
[source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5011
diff
changeset
|
532 |
sql, qargs, cbs = self._rql_sqlgen.generate(union, args, varmap) |
0 | 533 |
else: |
534 |
# sql may be cached |
|
535 |
try: |
|
5013
ad91f93bbb93
[source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5011
diff
changeset
|
536 |
sql, qargs, cbs = self._cache[cachekey] |
0 | 537 |
self.cache_hit += 1 |
538 |
except KeyError: |
|
539 |
self.cache_miss += 1 |
|
5013
ad91f93bbb93
[source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5011
diff
changeset
|
540 |
sql, qargs, cbs = self._rql_sqlgen.generate(union, args, varmap) |
ad91f93bbb93
[source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5011
diff
changeset
|
541 |
self._cache[cachekey] = sql, qargs, cbs |
ad91f93bbb93
[source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5011
diff
changeset
|
542 |
args = self.merge_args(args, qargs) |
0 | 543 |
assert isinstance(sql, basestring), repr(sql) |
544 |
try: |
|
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
545 |
cursor = self.doexec(cnx, sql, args) |
4831
c5aec27c1bf7
[repo] use logilab.db instead of lgc.adbh/lgc.db/lgc.sqlgen/indexer, test new date extranction functions
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4829
diff
changeset
|
546 |
except (self.OperationalError, self.InterfaceError): |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
547 |
if cnx.mode == 'write': |
5975
5120d97e2f7e
[transaction] do not attempt to reconnect if there has been some write during the transaction
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5847
diff
changeset
|
548 |
# do not attempt to reconnect if there has been some write |
5120d97e2f7e
[transaction] do not attempt to reconnect if there has been some write during the transaction
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5847
diff
changeset
|
549 |
# during the transaction |
5120d97e2f7e
[transaction] do not attempt to reconnect if there has been some write during the transaction
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5847
diff
changeset
|
550 |
raise |
0 | 551 |
# FIXME: better detection of deconnection pb |
5107
3694bd379513
[source] log attempt to reconnect using warning level. Also, don't relog failed sql (already logged)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5098
diff
changeset
|
552 |
self.warning("trying to reconnect") |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
553 |
cnx.cnxset.reconnect() |
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
554 |
cursor = self.doexec(cnx, sql, args) |
8695
358d8bed9626
[toward-py3k] rewrite to "except AnException as exc:" (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents:
8694
diff
changeset
|
555 |
except self.DbapiError as exc: |
5605
2604545d7dd9
[win32 SQLServer] connection lost detection
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5515
diff
changeset
|
556 |
# We get this one with pyodbc and SQL Server when connection was reset |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
557 |
if exc.args[0] == '08S01' and cnx.mode != 'write': |
5605
2604545d7dd9
[win32 SQLServer] connection lost detection
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5515
diff
changeset
|
558 |
self.warning("trying to reconnect") |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
559 |
cnx.cnxset.reconnect() |
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
560 |
cursor = self.doexec(cnx, sql, args) |
5605
2604545d7dd9
[win32 SQLServer] connection lost detection
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5515
diff
changeset
|
561 |
else: |
2604545d7dd9
[win32 SQLServer] connection lost detection
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5515
diff
changeset
|
562 |
raise |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
563 |
results = self.process_result(cursor, cbs, session=cnx) |
2625
d6012db7b93e
R [server debug]Â more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2620
diff
changeset
|
564 |
assert dbg_results(results) |
d6012db7b93e
R [server debug]Â more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2620
diff
changeset
|
565 |
return results |
1792 | 566 |
|
4964
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
567 |
@contextmanager |
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
568 |
def _storage_handler(self, entity, event): |
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
569 |
# 1/ memorize values as they are before the storage is called. |
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
570 |
# For instance, the BFSStorage will replace the `data` |
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
571 |
# binary value with a Binary containing the destination path |
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
572 |
# on the filesystem. To make the entity.data usage absolutely |
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
573 |
# transparent, we'll have to reset entity.data to its binary |
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
574 |
# value once the SQL query will be executed |
6889
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
575 |
restore_values = [] |
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
576 |
if isinstance(entity, list): |
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
577 |
entities = entity |
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
578 |
else: |
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
579 |
entities = [entity] |
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
580 |
etype = entities[0].__regid__ |
4964
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
581 |
for attr, storage in self._storages.get(etype, {}).items(): |
6889
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
582 |
for entity in entities: |
8501
b922dd08eb79
[storage] relies on event=='deleted' to detect "delete" event (closes #2450680)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
8379
diff
changeset
|
583 |
if event == 'deleted': |
b922dd08eb79
[storage] relies on event=='deleted' to detect "delete" event (closes #2450680)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
8379
diff
changeset
|
584 |
storage.entity_deleted(entity, attr) |
b922dd08eb79
[storage] relies on event=='deleted' to detect "delete" event (closes #2450680)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
8379
diff
changeset
|
585 |
else: |
6889
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
586 |
edited = entity.cw_edited |
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
587 |
if attr in edited: |
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
588 |
handler = getattr(storage, 'entity_%s' % event) |
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
589 |
to_restore = handler(entity, attr) |
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
590 |
restore_values.append((entity, attr, to_restore)) |
5128
e5d300d75519
[python] take care to this detail of @contextmanager: if an unhandled exception occurs in the block, it is reraised inside the generator at the point where the yield occurred
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5107
diff
changeset
|
591 |
try: |
e5d300d75519
[python] take care to this detail of @contextmanager: if an unhandled exception occurs in the block, it is reraised inside the generator at the point where the yield occurred
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5107
diff
changeset
|
592 |
yield # 2/ execute the source's instructions |
e5d300d75519
[python] take care to this detail of @contextmanager: if an unhandled exception occurs in the block, it is reraised inside the generator at the point where the yield occurred
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5107
diff
changeset
|
593 |
finally: |
e5d300d75519
[python] take care to this detail of @contextmanager: if an unhandled exception occurs in the block, it is reraised inside the generator at the point where the yield occurred
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5107
diff
changeset
|
594 |
# 3/ restore original values |
6889
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
595 |
for entity, attr, value in restore_values: |
6142
8bc6eac1fac1
[session] cleanup hook / operation / entity edition api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6128
diff
changeset
|
596 |
entity.cw_edited.edited_attribute(attr, value) |
4964
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
597 |
|
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
598 |
def add_entity(self, cnx, entity): |
0 | 599 |
"""add a new entity to the source""" |
4964
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
600 |
with self._storage_handler(entity, 'added'): |
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
601 |
attrs = self.preprocess_entity(entity) |
8900
010a59e12d89
use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
8821
diff
changeset
|
602 |
sql = self.sqlgen.insert(SQL_PREFIX + entity.cw_etype, attrs) |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
603 |
self.doexec(cnx, sql, attrs) |
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
604 |
if cnx.ertype_supports_undo(entity.cw_etype): |
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
605 |
self._record_tx_action(cnx, 'tx_entity_actions', 'C', |
8900
010a59e12d89
use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
8821
diff
changeset
|
606 |
etype=entity.cw_etype, eid=entity.eid) |
1792 | 607 |
|
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
608 |
def update_entity(self, cnx, entity): |
0 | 609 |
"""replace an entity in the source""" |
4964
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
610 |
with self._storage_handler(entity, 'updated'): |
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
611 |
attrs = self.preprocess_entity(entity) |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
612 |
if cnx.ertype_supports_undo(entity.cw_etype): |
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
613 |
changes = self._save_attrs(cnx, entity, attrs) |
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
614 |
self._record_tx_action(cnx, 'tx_entity_actions', 'U', |
8900
010a59e12d89
use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
8821
diff
changeset
|
615 |
etype=entity.cw_etype, eid=entity.eid, |
4964
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
616 |
changes=self._binary(dumps(changes))) |
8900
010a59e12d89
use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
8821
diff
changeset
|
617 |
sql = self.sqlgen.update(SQL_PREFIX + entity.cw_etype, attrs, |
4964
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
618 |
['cw_eid']) |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
619 |
self.doexec(cnx, sql, attrs) |
0 | 620 |
|
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
621 |
def delete_entity(self, cnx, entity): |
0 | 622 |
"""delete an entity from the source""" |
4964
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
623 |
with self._storage_handler(entity, 'deleted'): |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
624 |
if cnx.ertype_supports_undo(entity.cw_etype): |
4964
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
625 |
attrs = [SQL_PREFIX + r.type |
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
626 |
for r in entity.e_schema.subject_relations() |
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
627 |
if (r.final or r.inlined) and not r in VIRTUAL_RTYPES] |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
628 |
changes = self._save_attrs(cnx, entity, attrs) |
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
629 |
self._record_tx_action(cnx, 'tx_entity_actions', 'D', |
8900
010a59e12d89
use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
8821
diff
changeset
|
630 |
etype=entity.cw_etype, eid=entity.eid, |
4964
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
631 |
changes=self._binary(dumps(changes))) |
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
632 |
attrs = {'cw_eid': entity.eid} |
8900
010a59e12d89
use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
8821
diff
changeset
|
633 |
sql = self.sqlgen.delete(SQL_PREFIX + entity.cw_etype, attrs) |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
634 |
self.doexec(cnx, sql, attrs) |
0 | 635 |
|
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
636 |
def add_relation(self, cnx, subject, rtype, object, inlined=False): |
5072
072ae171aeb0
[cleanup] style fixes, add nodes, 0.2 cents refactorings
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5071
diff
changeset
|
637 |
"""add a relation to the source""" |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
638 |
self._add_relations(cnx, rtype, [(subject, object)], inlined) |
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
639 |
if cnx.ertype_supports_undo(rtype): |
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
640 |
self._record_tx_action(cnx, 'tx_relation_actions', 'A', |
5072
072ae171aeb0
[cleanup] style fixes, add nodes, 0.2 cents refactorings
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5071
diff
changeset
|
641 |
eid_from=subject, rtype=rtype, eid_to=object) |
072ae171aeb0
[cleanup] style fixes, add nodes, 0.2 cents refactorings
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5071
diff
changeset
|
642 |
|
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
643 |
def add_relations(self, cnx, rtype, subj_obj_list, inlined=False): |
7237
9f619715665b
[server] improve the speed of setting relations between entities (closes #1625257)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7202
diff
changeset
|
644 |
"""add a relations to the source""" |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
645 |
self._add_relations(cnx, rtype, subj_obj_list, inlined) |
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
646 |
if cnx.ertype_supports_undo(rtype): |
7237
9f619715665b
[server] improve the speed of setting relations between entities (closes #1625257)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7202
diff
changeset
|
647 |
for subject, object in subj_obj_list: |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
648 |
self._record_tx_action(cnx, 'tx_relation_actions', 'A', |
7237
9f619715665b
[server] improve the speed of setting relations between entities (closes #1625257)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7202
diff
changeset
|
649 |
eid_from=subject, rtype=rtype, eid_to=object) |
7551 | 650 |
|
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
651 |
def _add_relations(self, cnx, rtype, subj_obj_list, inlined=False): |
0 | 652 |
"""add a relation to the source""" |
7243
9ab01bf84eac
fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7237
diff
changeset
|
653 |
sql = [] |
4818
9f9bfbcdecfd
le patch massiveimport a été importé
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4810
diff
changeset
|
654 |
if inlined is False: |
7237
9f619715665b
[server] improve the speed of setting relations between entities (closes #1625257)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7202
diff
changeset
|
655 |
attrs = [{'eid_from': subject, 'eid_to': object} |
9f619715665b
[server] improve the speed of setting relations between entities (closes #1625257)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7202
diff
changeset
|
656 |
for subject, object in subj_obj_list] |
7243
9ab01bf84eac
fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7237
diff
changeset
|
657 |
sql.append((self.sqlgen.insert('%s_relation' % rtype, attrs[0]), attrs)) |
4818
9f9bfbcdecfd
le patch massiveimport a été importé
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4810
diff
changeset
|
658 |
else: # used by data import |
7243
9ab01bf84eac
fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7237
diff
changeset
|
659 |
etypes = {} |
9ab01bf84eac
fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7237
diff
changeset
|
660 |
for subject, object in subj_obj_list: |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
661 |
etype = cnx.entity_metas(subject)['type'] |
7243
9ab01bf84eac
fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7237
diff
changeset
|
662 |
if etype in etypes: |
9ab01bf84eac
fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7237
diff
changeset
|
663 |
etypes[etype].append((subject, object)) |
9ab01bf84eac
fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7237
diff
changeset
|
664 |
else: |
9ab01bf84eac
fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7237
diff
changeset
|
665 |
etypes[etype] = [(subject, object)] |
9ab01bf84eac
fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7237
diff
changeset
|
666 |
for subj_etype, subj_obj_list in etypes.iteritems(): |
9ab01bf84eac
fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7237
diff
changeset
|
667 |
attrs = [{'cw_eid': subject, SQL_PREFIX + rtype: object} |
9ab01bf84eac
fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7237
diff
changeset
|
668 |
for subject, object in subj_obj_list] |
9ab01bf84eac
fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7237
diff
changeset
|
669 |
sql.append((self.sqlgen.update(SQL_PREFIX + etype, attrs[0], |
9ab01bf84eac
fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7237
diff
changeset
|
670 |
['cw_eid']), |
9ab01bf84eac
fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7237
diff
changeset
|
671 |
attrs)) |
9ab01bf84eac
fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7237
diff
changeset
|
672 |
for statement, attrs in sql: |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
673 |
self.doexecmany(cnx, statement, attrs) |
1792 | 674 |
|
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
675 |
def delete_relation(self, cnx, subject, rtype, object): |
0 | 676 |
"""delete a relation from the source""" |
677 |
rschema = self.schema.rschema(rtype) |
|
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
678 |
self._delete_relation(cnx, subject, rtype, object, rschema.inlined) |
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
679 |
if cnx.ertype_supports_undo(rtype): |
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
680 |
self._record_tx_action(cnx, 'tx_relation_actions', 'R', |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
681 |
eid_from=subject, rtype=rtype, eid_to=object) |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
682 |
|
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
683 |
def _delete_relation(self, cnx, subject, rtype, object, inlined=False): |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
684 |
"""delete a relation from the source""" |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
685 |
if inlined: |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
686 |
table = SQL_PREFIX + cnx.entity_metas(subject)['type'] |
1251
af40e615dc89
introduce a 'cw_' prefix on entity table and column names so we don't conflict with sql or DBMS specific keywords
sylvain.thenault@logilab.fr
parents:
1079
diff
changeset
|
687 |
column = SQL_PREFIX + rtype |
af40e615dc89
introduce a 'cw_' prefix on entity table and column names so we don't conflict with sql or DBMS specific keywords
sylvain.thenault@logilab.fr
parents:
1079
diff
changeset
|
688 |
sql = 'UPDATE %s SET %s=NULL WHERE %seid=%%(eid)s' % (table, column, |
af40e615dc89
introduce a 'cw_' prefix on entity table and column names so we don't conflict with sql or DBMS specific keywords
sylvain.thenault@logilab.fr
parents:
1079
diff
changeset
|
689 |
SQL_PREFIX) |
0 | 690 |
attrs = {'eid' : subject} |
691 |
else: |
|
692 |
attrs = {'eid_from': subject, 'eid_to': object} |
|
693 |
sql = self.sqlgen.delete('%s_relation' % rtype, attrs) |
|
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
694 |
self.doexec(cnx, sql, attrs) |
0 | 695 |
|
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
696 |
def doexec(self, cnx, query, args=None, rollback=True): |
0 | 697 |
"""Execute a query. |
698 |
it's a function just so that it shows up in profiling |
|
699 |
""" |
|
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
700 |
cursor = cnx.cnxset.cu |
2593
16d9419a4a79
F: start to handle binary debug log level on the server side
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2589
diff
changeset
|
701 |
if server.DEBUG & server.DBG_SQL: |
10076
3810332ef42c
[server] fix 'cnx' variable confusion in DBG_SQL exception case
Julien Cristau <julien.cristau@logilab.fr>
parents:
10061
diff
changeset
|
702 |
print 'exec', query, args, cnx.cnxset.cnx |
0 | 703 |
try: |
9701
46c8d8701240
an unicode string -> a unicode string
Rémi Cardona <remi.cardona@logilab.fr>
parents:
9492
diff
changeset
|
704 |
# str(query) to avoid error if it's a unicode string |
0 | 705 |
cursor.execute(str(query), args) |
8695
358d8bed9626
[toward-py3k] rewrite to "except AnException as exc:" (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents:
8694
diff
changeset
|
706 |
except Exception as ex: |
4682
4994901b7379
don't issue critical message when trying to alter sqlite db during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4517
diff
changeset
|
707 |
if self.repo.config.mode != 'test': |
4994901b7379
don't issue critical message when trying to alter sqlite db during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4517
diff
changeset
|
708 |
# during test we get those message when trying to alter sqlite |
4994901b7379
don't issue critical message when trying to alter sqlite db during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4517
diff
changeset
|
709 |
# db schema |
4994901b7379
don't issue critical message when trying to alter sqlite db during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4517
diff
changeset
|
710 |
self.critical("sql: %r\n args: %s\ndbms message: %r", |
4994901b7379
don't issue critical message when trying to alter sqlite db during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4517
diff
changeset
|
711 |
query, args, ex.args[0]) |
2618
ff9b0d5bd884
[F repo sqlite schema changes] don't rollback on potentially expected schema changes failure
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2611
diff
changeset
|
712 |
if rollback: |
ff9b0d5bd884
[F repo sqlite schema changes] don't rollback on potentially expected schema changes failure
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2611
diff
changeset
|
713 |
try: |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
714 |
cnx.cnxset.rollback() |
4692
11a040e2601c
[test] also hide this message during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4682
diff
changeset
|
715 |
if self.repo.config.mode != 'test': |
9267
24d9b86dfa54
spelling: rollbacked -> rolled back
Rémi Cardona <remi.cardona@logilab.fr>
parents:
9131
diff
changeset
|
716 |
self.critical('transaction has been rolled back') |
8695
358d8bed9626
[toward-py3k] rewrite to "except AnException as exc:" (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents:
8694
diff
changeset
|
717 |
except Exception as ex: |
2618
ff9b0d5bd884
[F repo sqlite schema changes] don't rollback on potentially expected schema changes failure
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2611
diff
changeset
|
718 |
pass |
6211
e9d125fd1465
nicer error reporting for unique together constraints
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6201
diff
changeset
|
719 |
if ex.__class__.__name__ == 'IntegrityError': |
e9d125fd1465
nicer error reporting for unique together constraints
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6201
diff
changeset
|
720 |
# need string comparison because of various backends |
e9d125fd1465
nicer error reporting for unique together constraints
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6201
diff
changeset
|
721 |
for arg in ex.args: |
9375
8e88576787c3
[schema] fix unique together index handling
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9336
diff
changeset
|
722 |
# postgres, sqlserver |
8e88576787c3
[schema] fix unique together index handling
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9336
diff
changeset
|
723 |
mo = re.search("unique_[a-z0-9]{32}", arg) |
6211
e9d125fd1465
nicer error reporting for unique together constraints
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6201
diff
changeset
|
724 |
if mo is not None: |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
725 |
raise UniqueTogetherError(cnx, cstrname=mo.group(0)) |
10053
079b5279a2cd
[server] Handle unique constraint violations under recent sqlite
Julien Cristau <julien.cristau@logilab.fr>
parents:
10052
diff
changeset
|
726 |
# old sqlite |
6247
f7cb092d2296
unique_together: recast exception raised by sqlite
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6211
diff
changeset
|
727 |
mo = re.search('columns (.*) are not unique', arg) |
f7cb092d2296
unique_together: recast exception raised by sqlite
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6211
diff
changeset
|
728 |
if mo is not None: # sqlite in use |
8820
dbffb6959564
server/source/native: fix wrong usage of .lstrip that produce garbled error messages (closes #2777641)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
8546
diff
changeset
|
729 |
# we left chop the 'cw_' prefix of attribute names |
dbffb6959564
server/source/native: fix wrong usage of .lstrip that produce garbled error messages (closes #2777641)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
8546
diff
changeset
|
730 |
rtypes = [c.strip()[3:] |
dbffb6959564
server/source/native: fix wrong usage of .lstrip that produce garbled error messages (closes #2777641)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
8546
diff
changeset
|
731 |
for c in mo.group(1).split(',')] |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
732 |
raise UniqueTogetherError(cnx, rtypes=rtypes) |
10053
079b5279a2cd
[server] Handle unique constraint violations under recent sqlite
Julien Cristau <julien.cristau@logilab.fr>
parents:
10052
diff
changeset
|
733 |
# sqlite after http://www.sqlite.org/cgi/src/info/c80e229dd9c1230a |
079b5279a2cd
[server] Handle unique constraint violations under recent sqlite
Julien Cristau <julien.cristau@logilab.fr>
parents:
10052
diff
changeset
|
734 |
if arg.startswith('UNIQUE constraint failed:'): |
079b5279a2cd
[server] Handle unique constraint violations under recent sqlite
Julien Cristau <julien.cristau@logilab.fr>
parents:
10052
diff
changeset
|
735 |
# message looks like: "UNIQUE constraint failed: foo.cw_bar, foo.cw_baz" |
079b5279a2cd
[server] Handle unique constraint violations under recent sqlite
Julien Cristau <julien.cristau@logilab.fr>
parents:
10052
diff
changeset
|
736 |
# so drop the prefix, split on comma, drop the tablenames, and drop "cw_" |
079b5279a2cd
[server] Handle unique constraint violations under recent sqlite
Julien Cristau <julien.cristau@logilab.fr>
parents:
10052
diff
changeset
|
737 |
columns = arg.split(':', 1)[1].split(',') |
079b5279a2cd
[server] Handle unique constraint violations under recent sqlite
Julien Cristau <julien.cristau@logilab.fr>
parents:
10052
diff
changeset
|
738 |
rtypes = [c.split('.', 1)[1].strip()[3:] for c in columns] |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
739 |
raise UniqueTogetherError(cnx, rtypes=rtypes) |
0 | 740 |
raise |
2306
95da5d9f0870
give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2072
diff
changeset
|
741 |
return cursor |
1792 | 742 |
|
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
743 |
def doexecmany(self, cnx, query, args): |
0 | 744 |
"""Execute a query. |
745 |
it's a function just so that it shows up in profiling |
|
746 |
""" |
|
2593
16d9419a4a79
F: start to handle binary debug log level on the server side
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2589
diff
changeset
|
747 |
if server.DEBUG & server.DBG_SQL: |
0 | 748 |
print 'execmany', query, 'with', len(args), 'arguments' |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
749 |
cursor = cnx.cnxset.cu |
0 | 750 |
try: |
9701
46c8d8701240
an unicode string -> a unicode string
Rémi Cardona <remi.cardona@logilab.fr>
parents:
9492
diff
changeset
|
751 |
# str(query) to avoid error if it's a unicode string |
0 | 752 |
cursor.executemany(str(query), args) |
8695
358d8bed9626
[toward-py3k] rewrite to "except AnException as exc:" (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents:
8694
diff
changeset
|
753 |
except Exception as ex: |
4682
4994901b7379
don't issue critical message when trying to alter sqlite db during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4517
diff
changeset
|
754 |
if self.repo.config.mode != 'test': |
4994901b7379
don't issue critical message when trying to alter sqlite db during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4517
diff
changeset
|
755 |
# during test we get those message when trying to alter sqlite |
4994901b7379
don't issue critical message when trying to alter sqlite db during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4517
diff
changeset
|
756 |
# db schema |
4994901b7379
don't issue critical message when trying to alter sqlite db during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4517
diff
changeset
|
757 |
self.critical("sql many: %r\n args: %s\ndbms message: %r", |
4994901b7379
don't issue critical message when trying to alter sqlite db during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4517
diff
changeset
|
758 |
query, args, ex.args[0]) |
2306
95da5d9f0870
give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2072
diff
changeset
|
759 |
try: |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
760 |
cnx.cnxset.rollback() |
4692
11a040e2601c
[test] also hide this message during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4682
diff
changeset
|
761 |
if self.repo.config.mode != 'test': |
9267
24d9b86dfa54
spelling: rollbacked -> rolled back
Rémi Cardona <remi.cardona@logilab.fr>
parents:
9131
diff
changeset
|
762 |
self.critical('transaction has been rolled back') |
7645
cb2459aaa7d7
[source] closes #1816412: try to reconnect to dbms if connection is reset in eid_type_source()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
7562
diff
changeset
|
763 |
except Exception: |
2306
95da5d9f0870
give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2072
diff
changeset
|
764 |
pass |
0 | 765 |
raise |
1792 | 766 |
|
0 | 767 |
# short cut to method requiring advanced db helper usage ################## |
1792 | 768 |
|
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
769 |
def update_rdef_column(self, cnx, rdef): |
5891
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
770 |
"""update physical column for a relation definition (final or inlined) |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
771 |
""" |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
772 |
table, column = rdef_table_column(rdef) |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
773 |
coltype, allownull = rdef_physical_info(self.dbhelper, rdef) |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
774 |
if not self.dbhelper.alter_column_support: |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
775 |
self.error("backend can't alter %s.%s to %s%s", table, column, coltype, |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
776 |
not allownull and 'NOT NULL' or '') |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
777 |
return |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
778 |
self.dbhelper.change_col_type(LogCursor(cnx.cnxset.cu), |
5891
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
779 |
table, column, coltype, allownull) |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
780 |
self.info('altered %s.%s: now %s%s', table, column, coltype, |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
781 |
not allownull and 'NOT NULL' or '') |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
782 |
|
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
783 |
def update_rdef_null_allowed(self, cnx, rdef): |
5891
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
784 |
"""update NULL / NOT NULL of physical column for a relation definition |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
785 |
(final or inlined) |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
786 |
""" |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
787 |
if not self.dbhelper.alter_column_support: |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
788 |
# not supported (and NOT NULL not set by yams in that case, so no |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
789 |
# worry) |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
790 |
return |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
791 |
table, column = rdef_table_column(rdef) |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
792 |
coltype, allownull = rdef_physical_info(self.dbhelper, rdef) |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
793 |
self.dbhelper.set_null_allowed(LogCursor(cnx.cnxset.cu), |
5891
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
794 |
table, column, coltype, allownull) |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
795 |
|
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
796 |
def update_rdef_indexed(self, cnx, rdef): |
5891
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
797 |
table, column = rdef_table_column(rdef) |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
798 |
if rdef.indexed: |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
799 |
self.create_index(cnx, table, column) |
5891
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
800 |
else: |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
801 |
self.drop_index(cnx, table, column) |
5891
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
802 |
|
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
803 |
def update_rdef_unique(self, cnx, rdef): |
5891
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
804 |
table, column = rdef_table_column(rdef) |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
805 |
if rdef.constraint_by_type('UniqueConstraint'): |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
806 |
self.create_index(cnx, table, column, unique=True) |
5891
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
807 |
else: |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
808 |
self.drop_index(cnx, table, column, unique=True) |
5891
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
809 |
|
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
810 |
def create_index(self, cnx, table, column, unique=False): |
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
811 |
cursor = LogCursor(cnx.cnxset.cu) |
0 | 812 |
self.dbhelper.create_index(cursor, table, column, unique) |
1792 | 813 |
|
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
814 |
def drop_index(self, cnx, table, column, unique=False): |
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
815 |
cursor = LogCursor(cnx.cnxset.cu) |
0 | 816 |
self.dbhelper.drop_index(cursor, table, column, unique) |
817 |
||
818 |
# system source interface ################################################# |
|
819 |
||
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
820 |
def _eid_type_source(self, cnx, eid, sql, _retry=True): |
0 | 821 |
try: |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
822 |
res = self.doexec(cnx, sql).fetchone() |
7646
91a984ba420f
[native source] fix bug introduced in 7645:cb2459aaa7d7: fetchone() may return None
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7645
diff
changeset
|
823 |
if res is not None: |
91a984ba420f
[native source] fix bug introduced in 7645:cb2459aaa7d7: fetchone() may return None
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7645
diff
changeset
|
824 |
return res |
7645
cb2459aaa7d7
[source] closes #1816412: try to reconnect to dbms if connection is reset in eid_type_source()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
7562
diff
changeset
|
825 |
except (self.OperationalError, self.InterfaceError): |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
826 |
if cnx.mode == 'read' and _retry: |
7645
cb2459aaa7d7
[source] closes #1816412: try to reconnect to dbms if connection is reset in eid_type_source()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
7562
diff
changeset
|
827 |
self.warning("trying to reconnect (eid_type_source())") |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
828 |
cnx.cnxset.reconnect() |
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
829 |
return self._eid_type_source(cnx, eid, sql, _retry=False) |
7645
cb2459aaa7d7
[source] closes #1816412: try to reconnect to dbms if connection is reset in eid_type_source()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
7562
diff
changeset
|
830 |
except Exception: |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
831 |
assert cnx.cnxset, 'connection has no connections set' |
7645
cb2459aaa7d7
[source] closes #1816412: try to reconnect to dbms if connection is reset in eid_type_source()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
7562
diff
changeset
|
832 |
self.exception('failed to query entities table for eid %s', eid) |
cb2459aaa7d7
[source] closes #1816412: try to reconnect to dbms if connection is reset in eid_type_source()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
7562
diff
changeset
|
833 |
raise UnknownEid(eid) |
cb2459aaa7d7
[source] closes #1816412: try to reconnect to dbms if connection is reset in eid_type_source()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
7562
diff
changeset
|
834 |
|
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
835 |
def eid_type_source(self, cnx, eid): # pylint: disable=E0202 |
0 | 836 |
"""return a tuple (type, source, extid) for the entity with id <eid>""" |
9469
032825bbacab
[multi-sources-removal] Drop entities.source column
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
9468
diff
changeset
|
837 |
sql = 'SELECT type, extid, asource FROM entities WHERE eid=%s' % eid |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
838 |
res = self._eid_type_source(cnx, eid, sql) |
7543
570522300e22
[ms, entity metas] add 'actual source' to entities table / base entity metadata cache. Closes #1767090
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7536
diff
changeset
|
839 |
if res[-2] is not None: |
1952
8e19c813750d
fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1792
diff
changeset
|
840 |
if not isinstance(res, list): |
8e19c813750d
fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1792
diff
changeset
|
841 |
res = list(res) |
7543
570522300e22
[ms, entity metas] add 'actual source' to entities table / base entity metadata cache. Closes #1767090
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7536
diff
changeset
|
842 |
res[-2] = b64decode(res[-2]) |
570522300e22
[ms, entity metas] add 'actual source' to entities table / base entity metadata cache. Closes #1767090
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7536
diff
changeset
|
843 |
return res |
570522300e22
[ms, entity metas] add 'actual source' to entities table / base entity metadata cache. Closes #1767090
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7536
diff
changeset
|
844 |
|
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
845 |
def eid_type_source_pre_131(self, cnx, eid): |
7543
570522300e22
[ms, entity metas] add 'actual source' to entities table / base entity metadata cache. Closes #1767090
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7536
diff
changeset
|
846 |
"""return a tuple (type, source, extid) for the entity with id <eid>""" |
9469
032825bbacab
[multi-sources-removal] Drop entities.source column
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
9468
diff
changeset
|
847 |
sql = 'SELECT type, extid FROM entities WHERE eid=%s' % eid |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
848 |
res = self._eid_type_source(cnx, eid, sql) |
7543
570522300e22
[ms, entity metas] add 'actual source' to entities table / base entity metadata cache. Closes #1767090
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7536
diff
changeset
|
849 |
if not isinstance(res, list): |
570522300e22
[ms, entity metas] add 'actual source' to entities table / base entity metadata cache. Closes #1767090
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7536
diff
changeset
|
850 |
res = list(res) |
1952
8e19c813750d
fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1792
diff
changeset
|
851 |
if res[-1] is not None: |
8e19c813750d
fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1792
diff
changeset
|
852 |
res[-1] = b64decode(res[-1]) |
7543
570522300e22
[ms, entity metas] add 'actual source' to entities table / base entity metadata cache. Closes #1767090
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7536
diff
changeset
|
853 |
res.append(res[1]) |
0 | 854 |
return res |
855 |
||
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
856 |
def extid2eid(self, cnx, extid): |
1952
8e19c813750d
fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1792
diff
changeset
|
857 |
"""get eid from an external id. Return None if no record found.""" |
8e19c813750d
fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1792
diff
changeset
|
858 |
assert isinstance(extid, str) |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
859 |
cursor = self.doexec(cnx, |
9469
032825bbacab
[multi-sources-removal] Drop entities.source column
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
9468
diff
changeset
|
860 |
'SELECT eid FROM entities WHERE extid=%(x)s', |
032825bbacab
[multi-sources-removal] Drop entities.source column
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
9468
diff
changeset
|
861 |
{'x': b64encode(extid)}) |
0 | 862 |
# XXX testing rowcount cause strange bug with sqlite, results are there |
863 |
# but rowcount is 0 |
|
1792 | 864 |
#if cursor.rowcount > 0: |
0 | 865 |
try: |
866 |
result = cursor.fetchone() |
|
867 |
if result: |
|
1954 | 868 |
return result[0] |
7645
cb2459aaa7d7
[source] closes #1816412: try to reconnect to dbms if connection is reset in eid_type_source()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
7562
diff
changeset
|
869 |
except Exception: |
0 | 870 |
pass |
871 |
return None |
|
1792 | 872 |
|
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
873 |
def _handle_is_relation_sql(self, cnx, sql, attrs): |
8625
7ee0752178e5
[dataimport] Add SQL Store for faster import - works ONLY with Postgres for now, as it requires "copy from" command - closes #2410822
Vincent Michel <vincent.michel@logilab.fr>
parents:
8573
diff
changeset
|
874 |
""" Handler for specific is_relation sql that may be |
7ee0752178e5
[dataimport] Add SQL Store for faster import - works ONLY with Postgres for now, as it requires "copy from" command - closes #2410822
Vincent Michel <vincent.michel@logilab.fr>
parents:
8573
diff
changeset
|
875 |
overwritten in some stores""" |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
876 |
self.doexec(cnx, sql % attrs) |
8625
7ee0752178e5
[dataimport] Add SQL Store for faster import - works ONLY with Postgres for now, as it requires "copy from" command - closes #2410822
Vincent Michel <vincent.michel@logilab.fr>
parents:
8573
diff
changeset
|
877 |
|
7ee0752178e5
[dataimport] Add SQL Store for faster import - works ONLY with Postgres for now, as it requires "copy from" command - closes #2410822
Vincent Michel <vincent.michel@logilab.fr>
parents:
8573
diff
changeset
|
878 |
_handle_insert_entity_sql = doexec |
7ee0752178e5
[dataimport] Add SQL Store for faster import - works ONLY with Postgres for now, as it requires "copy from" command - closes #2410822
Vincent Michel <vincent.michel@logilab.fr>
parents:
8573
diff
changeset
|
879 |
_handle_is_instance_of_sql = _handle_source_relation_sql = _handle_is_relation_sql |
7ee0752178e5
[dataimport] Add SQL Store for faster import - works ONLY with Postgres for now, as it requires "copy from" command - closes #2410822
Vincent Michel <vincent.michel@logilab.fr>
parents:
8573
diff
changeset
|
880 |
|
9664
5ef5494b6b0b
[repository] complete argument is no more needed since we have no more 'true' multisource.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
9611
diff
changeset
|
881 |
def add_info(self, cnx, entity, source, extid): |
0 | 882 |
"""add type and source info for an eid into the system table""" |
9802
9f815b1f2516
[repository,tests] use the new connection api (part 1/2)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9770
diff
changeset
|
883 |
assert cnx.cnxset is not None |
0 | 884 |
# begin by inserting eid/type/source/extid into the entities table |
1952
8e19c813750d
fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1792
diff
changeset
|
885 |
if extid is not None: |
8e19c813750d
fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1792
diff
changeset
|
886 |
assert isinstance(extid, str) |
8e19c813750d
fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1792
diff
changeset
|
887 |
extid = b64encode(extid) |
8900
010a59e12d89
use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
8821
diff
changeset
|
888 |
attrs = {'type': entity.cw_etype, 'eid': entity.eid, 'extid': extid, |
9802
9f815b1f2516
[repository,tests] use the new connection api (part 1/2)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9770
diff
changeset
|
889 |
'asource': source.uri} |
9f815b1f2516
[repository,tests] use the new connection api (part 1/2)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9770
diff
changeset
|
890 |
self._handle_insert_entity_sql(cnx, self.sqlgen.insert('entities', attrs), attrs) |
6427
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6426
diff
changeset
|
891 |
# insert core relations: is, is_instance_of and cw_source |
6851
824d5b6eae7f
[repo] kill no more needed 'recreate' feature
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6849
diff
changeset
|
892 |
try: |
9802
9f815b1f2516
[repository,tests] use the new connection api (part 1/2)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9770
diff
changeset
|
893 |
self._handle_is_relation_sql(cnx, 'INSERT INTO is_relation(eid_from,eid_to) VALUES (%s,%s)', |
9f815b1f2516
[repository,tests] use the new connection api (part 1/2)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9770
diff
changeset
|
894 |
(entity.eid, eschema_eid(cnx, entity.e_schema))) |
6851
824d5b6eae7f
[repo] kill no more needed 'recreate' feature
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6849
diff
changeset
|
895 |
except IndexError: |
824d5b6eae7f
[repo] kill no more needed 'recreate' feature
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6849
diff
changeset
|
896 |
# during schema serialization, skip |
824d5b6eae7f
[repo] kill no more needed 'recreate' feature
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6849
diff
changeset
|
897 |
pass |
824d5b6eae7f
[repo] kill no more needed 'recreate' feature
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6849
diff
changeset
|
898 |
else: |
824d5b6eae7f
[repo] kill no more needed 'recreate' feature
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6849
diff
changeset
|
899 |
for eschema in entity.e_schema.ancestors() + [entity.e_schema]: |
9802
9f815b1f2516
[repository,tests] use the new connection api (part 1/2)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9770
diff
changeset
|
900 |
self._handle_is_relation_sql(cnx, |
8625
7ee0752178e5
[dataimport] Add SQL Store for faster import - works ONLY with Postgres for now, as it requires "copy from" command - closes #2410822
Vincent Michel <vincent.michel@logilab.fr>
parents:
8573
diff
changeset
|
901 |
'INSERT INTO is_instance_of_relation(eid_from,eid_to) VALUES (%s,%s)', |
9802
9f815b1f2516
[repository,tests] use the new connection api (part 1/2)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9770
diff
changeset
|
902 |
(entity.eid, eschema_eid(cnx, eschema))) |
6851
824d5b6eae7f
[repo] kill no more needed 'recreate' feature
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6849
diff
changeset
|
903 |
if 'CWSource' in self.schema and source.eid is not None: # else, cw < 3.10 |
9802
9f815b1f2516
[repository,tests] use the new connection api (part 1/2)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9770
diff
changeset
|
904 |
self._handle_is_relation_sql(cnx, 'INSERT INTO cw_source_relation(eid_from,eid_to) VALUES (%s,%s)', |
8625
7ee0752178e5
[dataimport] Add SQL Store for faster import - works ONLY with Postgres for now, as it requires "copy from" command - closes #2410822
Vincent Michel <vincent.michel@logilab.fr>
parents:
8573
diff
changeset
|
905 |
(entity.eid, source.eid)) |
4806
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
906 |
# now we can update the full text index |
8900
010a59e12d89
use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
8821
diff
changeset
|
907 |
if self.do_fti and self.need_fti_indexation(entity.cw_etype): |
9802
9f815b1f2516
[repository,tests] use the new connection api (part 1/2)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9770
diff
changeset
|
908 |
self.index_entity(cnx, entity=entity) |
4806
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
909 |
|
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
910 |
def update_info(self, cnx, entity, need_fti_update): |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
911 |
"""mark entity as being modified, fulltext reindex if needed""" |
4806
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
912 |
if self.do_fti and need_fti_update: |
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
913 |
# reindex the entity only if this query is updating at least |
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
914 |
# one indexable attribute |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
915 |
self.index_entity(cnx, entity=entity) |
0 | 916 |
|
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
917 |
def delete_info_multi(self, cnx, entities): |
7501
2983dd24494a
[repository] refactor/cleanup entity deletion methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
918 |
"""delete system information on deletion of a list of entities with the |
2983dd24494a
[repository] refactor/cleanup entity deletion methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
919 |
same etype and belinging to the same source |
2983dd24494a
[repository] refactor/cleanup entity deletion methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
920 |
|
5067
adc2122eed03
[repo] more efficient eid cache operations handling based on set_operation; refactor
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5061
diff
changeset
|
921 |
* update the fti |
7501
2983dd24494a
[repository] refactor/cleanup entity deletion methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
922 |
* remove record from the `entities` table |
6889
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
923 |
""" |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
924 |
self.fti_unindex_entities(cnx, entities) |
6889
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
925 |
attrs = {'eid': '(%s)' % ','.join([str(_e.eid) for _e in entities])} |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
926 |
self.doexec(cnx, self.sqlgen.delete_many('entities', attrs), attrs) |
0 | 927 |
|
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
928 |
# undo support ############################################################# |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
929 |
|
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
930 |
def undoable_transactions(self, cnx, ueid=None, **actionfilters): |
9487
88a092a665f4
[server/sources] replace references to dbapi with repoapi
Julien Cristau <julien.cristau@logilab.fr>
parents:
9482
diff
changeset
|
931 |
"""See :class:`cubicweb.repoapi.ClientConnection.undoable_transactions`""" |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
932 |
# force filtering to connection's user if not a manager |
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
933 |
if not cnx.user.is_in_group('managers'): |
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
934 |
ueid = cnx.user.eid |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
935 |
restr = {} |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
936 |
if ueid is not None: |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
937 |
restr['tx_user'] = ueid |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
938 |
sql = self.sqlgen.select('transactions', restr, ('tx_uuid', 'tx_time', 'tx_user')) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
939 |
if actionfilters: |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
940 |
# we will need subqueries to filter transactions according to |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
941 |
# actions done |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
942 |
tearestr = {} # filters on the tx_entity_actions table |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
943 |
trarestr = {} # filters on the tx_relation_actions table |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
944 |
genrestr = {} # generic filters, appliyable to both table |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
945 |
# unless public explicitly set to false, we only consider public |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
946 |
# actions |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
947 |
if actionfilters.pop('public', True): |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
948 |
genrestr['txa_public'] = True |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
949 |
# put additional filters in trarestr and/or tearestr |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
950 |
for key, val in actionfilters.iteritems(): |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
951 |
if key == 'etype': |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
952 |
# filtering on etype implies filtering on entity actions |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
953 |
# only, and with no eid specified |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
954 |
assert actionfilters.get('action', 'C') in 'CUD' |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
955 |
assert not 'eid' in actionfilters |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
956 |
tearestr['etype'] = val |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
957 |
elif key == 'eid': |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
958 |
# eid filter may apply to 'eid' of tx_entity_actions or to |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
959 |
# 'eid_from' OR 'eid_to' of tx_relation_actions |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
960 |
if actionfilters.get('action', 'C') in 'CUD': |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
961 |
tearestr['eid'] = val |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
962 |
if actionfilters.get('action', 'A') in 'AR': |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
963 |
trarestr['eid_from'] = val |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
964 |
trarestr['eid_to'] = val |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
965 |
elif key == 'action': |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
966 |
if val in 'CUD': |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
967 |
tearestr['txa_action'] = val |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
968 |
else: |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
969 |
assert val in 'AR' |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
970 |
trarestr['txa_action'] = val |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
971 |
else: |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
972 |
raise AssertionError('unknow filter %s' % key) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
973 |
assert trarestr or tearestr, "can't only filter on 'public'" |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
974 |
subqsqls = [] |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
975 |
# append subqueries to the original query, using EXISTS() |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
976 |
if trarestr or (genrestr and not tearestr): |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
977 |
trarestr.update(genrestr) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
978 |
trasql = self.sqlgen.select('tx_relation_actions', trarestr, ('1',)) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
979 |
if 'eid_from' in trarestr: |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
980 |
# replace AND by OR between eid_from/eid_to restriction |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
981 |
trasql = sql_or_clauses(trasql, ['eid_from = %(eid_from)s', |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
982 |
'eid_to = %(eid_to)s']) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
983 |
trasql += ' AND transactions.tx_uuid=tx_relation_actions.tx_uuid' |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
984 |
subqsqls.append('EXISTS(%s)' % trasql) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
985 |
if tearestr or (genrestr and not trarestr): |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
986 |
tearestr.update(genrestr) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
987 |
teasql = self.sqlgen.select('tx_entity_actions', tearestr, ('1',)) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
988 |
teasql += ' AND transactions.tx_uuid=tx_entity_actions.tx_uuid' |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
989 |
subqsqls.append('EXISTS(%s)' % teasql) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
990 |
if restr: |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
991 |
sql += ' AND %s' % ' OR '.join(subqsqls) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
992 |
else: |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
993 |
sql += ' WHERE %s' % ' OR '.join(subqsqls) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
994 |
restr.update(trarestr) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
995 |
restr.update(tearestr) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
996 |
# we want results ordered by transaction's time descendant |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
997 |
sql += ' ORDER BY tx_time DESC' |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
998 |
with cnx.ensure_cnx_set: |
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
999 |
cu = self.doexec(cnx, sql, restr) |
9605
e6b8931abe17
[sources] ensure we have a cnxset in undoable_transactions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9585
diff
changeset
|
1000 |
# turn results into transaction objects |
e6b8931abe17
[sources] ensure we have a cnxset in undoable_transactions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9585
diff
changeset
|
1001 |
return [tx.Transaction(*args) for args in cu.fetchall()] |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1002 |
|
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1003 |
def tx_info(self, cnx, txuuid): |
9487
88a092a665f4
[server/sources] replace references to dbapi with repoapi
Julien Cristau <julien.cristau@logilab.fr>
parents:
9482
diff
changeset
|
1004 |
"""See :class:`cubicweb.repoapi.ClientConnection.transaction_info`""" |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1005 |
return tx.Transaction(txuuid, *self._tx_info(cnx, txuuid)) |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1006 |
|
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1007 |
def tx_actions(self, cnx, txuuid, public): |
9487
88a092a665f4
[server/sources] replace references to dbapi with repoapi
Julien Cristau <julien.cristau@logilab.fr>
parents:
9482
diff
changeset
|
1008 |
"""See :class:`cubicweb.repoapi.ClientConnection.transaction_actions`""" |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1009 |
self._tx_info(cnx, txuuid) |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1010 |
restr = {'tx_uuid': txuuid} |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1011 |
if public: |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1012 |
restr['txa_public'] = True |
5072
072ae171aeb0
[cleanup] style fixes, add nodes, 0.2 cents refactorings
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5071
diff
changeset
|
1013 |
# XXX use generator to avoid loading everything in memory? |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1014 |
sql = self.sqlgen.select('tx_entity_actions', restr, |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1015 |
('txa_action', 'txa_public', 'txa_order', |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1016 |
'etype', 'eid', 'changes')) |
9996
65bb21409292
[server] hold connection to the db in tx_actions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9984
diff
changeset
|
1017 |
with cnx.ensure_cnx_set: |
65bb21409292
[server] hold connection to the db in tx_actions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9984
diff
changeset
|
1018 |
cu = self.doexec(cnx, sql, restr) |
65bb21409292
[server] hold connection to the db in tx_actions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9984
diff
changeset
|
1019 |
actions = [tx.EntityAction(a,p,o,et,e,c and loads(self.binary_to_str(c))) |
65bb21409292
[server] hold connection to the db in tx_actions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9984
diff
changeset
|
1020 |
for a,p,o,et,e,c in cu.fetchall()] |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1021 |
sql = self.sqlgen.select('tx_relation_actions', restr, |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1022 |
('txa_action', 'txa_public', 'txa_order', |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1023 |
'rtype', 'eid_from', 'eid_to')) |
9996
65bb21409292
[server] hold connection to the db in tx_actions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9984
diff
changeset
|
1024 |
with cnx.ensure_cnx_set: |
65bb21409292
[server] hold connection to the db in tx_actions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9984
diff
changeset
|
1025 |
cu = self.doexec(cnx, sql, restr) |
65bb21409292
[server] hold connection to the db in tx_actions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9984
diff
changeset
|
1026 |
actions += [tx.RelationAction(*args) for args in cu.fetchall()] |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1027 |
return sorted(actions, key=lambda x: x.order) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1028 |
|
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1029 |
def undo_transaction(self, cnx, txuuid): |
9487
88a092a665f4
[server/sources] replace references to dbapi with repoapi
Julien Cristau <julien.cristau@logilab.fr>
parents:
9482
diff
changeset
|
1030 |
"""See :class:`cubicweb.repoapi.ClientConnection.undo_transaction` |
5071
8631bb9f6e73
[undo] during undoing, call hooks in the [active]integrity and undo categories
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5070
diff
changeset
|
1031 |
|
8631bb9f6e73
[undo] during undoing, call hooks in the [active]integrity and undo categories
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5070
diff
changeset
|
1032 |
important note: while undoing of a transaction, only hooks in the |
8631bb9f6e73
[undo] during undoing, call hooks in the [active]integrity and undo categories
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5070
diff
changeset
|
1033 |
'integrity', 'activeintegrity' and 'undo' categories are called. |
8631bb9f6e73
[undo] during undoing, call hooks in the [active]integrity and undo categories
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5070
diff
changeset
|
1034 |
""" |
7398
26695dd703d8
[repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
1035 |
# set mode so connections set isn't released subsquently until commit/rollback |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1036 |
cnx.mode = 'write' |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1037 |
errors = [] |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1038 |
cnx.transaction_data['undoing_uuid'] = txuuid |
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1039 |
with cnx.deny_all_hooks_but('integrity', 'activeintegrity', 'undo'): |
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1040 |
with cnx.security_enabled(read=False): |
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1041 |
for action in reversed(self.tx_actions(cnx, txuuid, False)): |
4943
7f5b83578fec
disable security when undoing
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4913
diff
changeset
|
1042 |
undomethod = getattr(self, '_undo_%s' % action.action.lower()) |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1043 |
errors += undomethod(cnx, action) |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1044 |
# remove the transactions record |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1045 |
self.doexec(cnx, |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1046 |
"DELETE FROM transactions WHERE tx_uuid='%s'" % txuuid) |
8265
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1047 |
if errors: |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1048 |
raise UndoTransactionException(txuuid, errors) |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1049 |
else: |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1050 |
return |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1051 |
|
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1052 |
def start_undoable_transaction(self, cnx, uuid): |
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1053 |
"""connection callback to insert a transaction record in the transactions |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1054 |
table when some undoable transaction is started |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1055 |
""" |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1056 |
ueid = cnx.user.eid |
7922
d307c3817782
[repository] use utcnow instead of now for some internal timetamps (closes #1988458)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7879
diff
changeset
|
1057 |
attrs = {'tx_uuid': uuid, 'tx_user': ueid, 'tx_time': datetime.utcnow()} |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1058 |
self.doexec(cnx, self.sqlgen.insert('transactions', attrs), attrs) |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1059 |
|
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1060 |
def _save_attrs(self, cnx, entity, attrs): |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1061 |
"""return a pickleable dictionary containing current values for given |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1062 |
attributes of the entity |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1063 |
""" |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1064 |
restr = {'cw_eid': entity.eid} |
8900
010a59e12d89
use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
8821
diff
changeset
|
1065 |
sql = self.sqlgen.select(SQL_PREFIX + entity.cw_etype, restr, attrs) |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1066 |
cu = self.doexec(cnx, sql, restr) |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1067 |
values = dict(zip(attrs, cu.fetchone())) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1068 |
# ensure backend specific binary are converted back to string |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1069 |
eschema = entity.e_schema |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1070 |
for column in attrs: |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1071 |
# [3:] remove 'cw_' prefix |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1072 |
attr = column[3:] |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1073 |
if not eschema.subjrels[attr].final: |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1074 |
continue |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1075 |
if eschema.destination(attr) in ('Password', 'Bytes'): |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1076 |
value = values[column] |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1077 |
if value is not None: |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1078 |
values[column] = self.binary_to_str(value) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1079 |
return values |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1080 |
|
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1081 |
def _record_tx_action(self, cnx, table, action, **kwargs): |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1082 |
"""record a transaction action in the given table (either |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1083 |
'tx_entity_actions' or 'tx_relation_action') |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1084 |
""" |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1085 |
kwargs['tx_uuid'] = cnx.transaction_uuid() |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1086 |
kwargs['txa_action'] = action |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1087 |
kwargs['txa_order'] = cnx.transaction_inc_action_counter() |
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1088 |
kwargs['txa_public'] = cnx.running_dbapi_query |
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1089 |
self.doexec(cnx, self.sqlgen.insert(table, kwargs), kwargs) |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1090 |
|
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1091 |
def _tx_info(self, cnx, txuuid): |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1092 |
"""return transaction's time and user of the transaction with the given uuid. |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1093 |
|
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1094 |
raise `NoSuchTransaction` if there is no such transaction of if the |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1095 |
connection's user isn't allowed to see it. |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1096 |
""" |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1097 |
with cnx.ensure_cnx_set: |
9098
5467fb901931
[connection] transparent cnx_set handling
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
8947
diff
changeset
|
1098 |
restr = {'tx_uuid': txuuid} |
5467fb901931
[connection] transparent cnx_set handling
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
8947
diff
changeset
|
1099 |
sql = self.sqlgen.select('transactions', restr, |
5467fb901931
[connection] transparent cnx_set handling
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
8947
diff
changeset
|
1100 |
('tx_time', 'tx_user')) |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1101 |
cu = self.doexec(cnx, sql, restr) |
9098
5467fb901931
[connection] transparent cnx_set handling
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
8947
diff
changeset
|
1102 |
try: |
5467fb901931
[connection] transparent cnx_set handling
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
8947
diff
changeset
|
1103 |
time, ueid = cu.fetchone() |
5467fb901931
[connection] transparent cnx_set handling
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
8947
diff
changeset
|
1104 |
except TypeError: |
5467fb901931
[connection] transparent cnx_set handling
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
8947
diff
changeset
|
1105 |
raise tx.NoSuchTransaction(txuuid) |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1106 |
if not (cnx.user.is_in_group('managers') |
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1107 |
or cnx.user.eid == ueid): |
9098
5467fb901931
[connection] transparent cnx_set handling
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
8947
diff
changeset
|
1108 |
raise tx.NoSuchTransaction(txuuid) |
5467fb901931
[connection] transparent cnx_set handling
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
8947
diff
changeset
|
1109 |
return time, ueid |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1110 |
|
8265
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1111 |
def _reedit_entity(self, entity, changes, err): |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1112 |
cnx = entity._cw |
8265
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1113 |
eid = entity.eid |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1114 |
entity.cw_edited = edited = EditedEntity(entity) |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1115 |
# check for schema changes, entities linked through inlined relation |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1116 |
# still exists, rewrap binary values |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1117 |
eschema = entity.e_schema |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1118 |
getrschema = eschema.subjrels |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1119 |
for column, value in changes.items(): |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1120 |
rtype = column[len(SQL_PREFIX):] |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1121 |
if rtype == "eid": |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1122 |
continue # XXX should even `eid` be stored in action changes? |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1123 |
try: |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1124 |
rschema = getrschema[rtype] |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1125 |
except KeyError: |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1126 |
err(cnx._("can't restore relation %(rtype)s of entity %(eid)s, " |
8265
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1127 |
"this relation does not exist in the schema anymore.") |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1128 |
% {'rtype': rtype, 'eid': eid}) |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1129 |
if not rschema.final: |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1130 |
if not rschema.inlined: |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1131 |
assert value is None |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1132 |
# rschema is an inlined relation |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1133 |
elif value is not None: |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1134 |
# not a deletion: we must put something in edited |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1135 |
try: |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1136 |
entity._cw.entity_from_eid(value) # check target exists |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1137 |
edited[rtype] = value |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1138 |
except UnknownEid: |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1139 |
err(cnx._("can't restore entity %(eid)s of type %(eschema)s, " |
8265
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1140 |
"target of %(rtype)s (eid %(value)s) does not exist any longer") |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1141 |
% locals()) |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1142 |
elif eschema.destination(rtype) in ('Bytes', 'Password'): |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1143 |
changes[column] = self._binary(value) |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1144 |
edited[rtype] = Binary(value) |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1145 |
elif isinstance(value, str): |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1146 |
edited[rtype] = unicode(value, cnx.encoding, 'replace') |
8265
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1147 |
else: |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1148 |
edited[rtype] = value |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1149 |
# This must only be done after init_entitiy_caches : defered in calling functions |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1150 |
# edited.check() |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1151 |
|
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1152 |
def _undo_d(self, cnx, action): |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1153 |
"""undo an entity deletion""" |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1154 |
errors = [] |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1155 |
err = errors.append |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1156 |
eid = action.eid |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1157 |
etype = action.etype |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1158 |
_ = cnx._ |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1159 |
# get an entity instance |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1160 |
try: |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1161 |
entity = self.repo.vreg['etypes'].etype_class(etype)(cnx) |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1162 |
except Exception: |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1163 |
err("can't restore entity %s of type %s, type no more supported" |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1164 |
% (eid, etype)) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1165 |
return errors |
8265
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1166 |
self._reedit_entity(entity, action.changes, err) |
5557
1a534c596bff
[entity] continue cleanup of Entity/AnyEntity namespace
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5556
diff
changeset
|
1167 |
entity.eid = eid |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1168 |
cnx.repo.init_entity_caches(cnx, entity, self) |
8265
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1169 |
entity.cw_edited.check() |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1170 |
self.repo.hm.call_hooks('before_add_entity', cnx, entity=entity) |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1171 |
# restore the entity |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1172 |
action.changes['cw_eid'] = eid |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1173 |
sql = self.sqlgen.insert(SQL_PREFIX + etype, action.changes) |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1174 |
self.doexec(cnx, sql, action.changes) |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1175 |
# restore record in entities (will update fti if needed) |
9664
5ef5494b6b0b
[repository] complete argument is no more needed since we have no more 'true' multisource.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
9611
diff
changeset
|
1176 |
self.add_info(cnx, entity, self, None) |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1177 |
self.repo.hm.call_hooks('after_add_entity', cnx, entity=entity) |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1178 |
return errors |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1179 |
|
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1180 |
def _undo_r(self, cnx, action): |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1181 |
"""undo a relation removal""" |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1182 |
errors = [] |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1183 |
subj, rtype, obj = action.eid_from, action.rtype, action.eid_to |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1184 |
try: |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1185 |
sentity, oentity, rdef = _undo_rel_info(cnx, subj, rtype, obj) |
8695
358d8bed9626
[toward-py3k] rewrite to "except AnException as exc:" (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents:
8694
diff
changeset
|
1186 |
except _UndoException as ex: |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1187 |
errors.append(unicode(ex)) |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1188 |
else: |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1189 |
for role, entity in (('subject', sentity), |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1190 |
('object', oentity)): |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1191 |
try: |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1192 |
_undo_check_relation_target(entity, rdef, role) |
8695
358d8bed9626
[toward-py3k] rewrite to "except AnException as exc:" (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents:
8694
diff
changeset
|
1193 |
except _UndoException as ex: |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1194 |
errors.append(unicode(ex)) |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1195 |
continue |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1196 |
if not errors: |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1197 |
self.repo.hm.call_hooks('before_add_relation', cnx, |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1198 |
eidfrom=subj, rtype=rtype, eidto=obj) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1199 |
# add relation in the database |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1200 |
self._add_relations(cnx, rtype, [(subj, obj)], rdef.rtype.inlined) |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1201 |
# set related cache |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1202 |
cnx.update_rel_cache_add(subj, rtype, obj, rdef.rtype.symmetric) |
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1203 |
self.repo.hm.call_hooks('after_add_relation', cnx, |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1204 |
eidfrom=subj, rtype=rtype, eidto=obj) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1205 |
return errors |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1206 |
|
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1207 |
def _undo_c(self, cnx, action): |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1208 |
"""undo an entity creation""" |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1209 |
eid = action.eid |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1210 |
# XXX done to avoid fetching all remaining relation for the entity |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1211 |
# we should find an efficient way to do this (keeping current veolidf |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1212 |
# massive deletion performance) |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1213 |
if _undo_has_later_transaction(cnx, eid): |
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1214 |
msg = cnx._('some later transaction(s) touch entity, undo them ' |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1215 |
'first') |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1216 |
raise ValidationError(eid, {None: msg}) |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1217 |
etype = action.etype |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1218 |
# get an entity instance |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1219 |
try: |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1220 |
entity = self.repo.vreg['etypes'].etype_class(etype)(cnx) |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1221 |
except Exception: |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1222 |
return [cnx._( |
5098
32b1adfb6b92
[i18n] use named substitution to avoid gettext warnings
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5076
diff
changeset
|
1223 |
"Can't undo creation of entity %(eid)s of type %(etype)s, type " |
32b1adfb6b92
[i18n] use named substitution to avoid gettext warnings
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5076
diff
changeset
|
1224 |
"no more supported" % {'eid': eid, 'etype': etype})] |
5557
1a534c596bff
[entity] continue cleanup of Entity/AnyEntity namespace
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5556
diff
changeset
|
1225 |
entity.eid = eid |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1226 |
# for proper eid/type cache update |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1227 |
CleanupDeletedEidsCacheOp.get_instance(cnx).add_data(eid) |
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1228 |
self.repo.hm.call_hooks('before_delete_entity', cnx, entity=entity) |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1229 |
# remove is / is_instance_of which are added using sql by hooks, hence |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1230 |
# unvisible as transaction action |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1231 |
self.doexec(cnx, 'DELETE FROM is_relation WHERE eid_from=%s' % eid) |
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1232 |
self.doexec(cnx, 'DELETE FROM is_instance_of_relation WHERE eid_from=%s' % eid) |
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1233 |
self.doexec(cnx, 'DELETE FROM cw_source_relation WHERE eid_from=%s' % self.eid) |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1234 |
# XXX check removal of inlined relation? |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1235 |
# delete the entity |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1236 |
attrs = {'cw_eid': eid} |
8900
010a59e12d89
use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
8821
diff
changeset
|
1237 |
sql = self.sqlgen.delete(SQL_PREFIX + entity.cw_etype, attrs) |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1238 |
self.doexec(cnx, sql, attrs) |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1239 |
# remove record from entities (will update fti if needed) |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1240 |
self.delete_info_multi(cnx, [entity]) |
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1241 |
self.repo.hm.call_hooks('after_delete_entity', cnx, entity=entity) |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1242 |
return () |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1243 |
|
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1244 |
def _undo_u(self, cnx, action): |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1245 |
"""undo an entity update""" |
8265
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1246 |
errors = [] |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1247 |
err = errors.append |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1248 |
try: |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1249 |
entity = cnx.entity_from_eid(action.eid) |
8265
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1250 |
except UnknownEid: |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1251 |
err(cnx._("can't restore state of entity %s, it has been " |
8265
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1252 |
"deleted inbetween") % action.eid) |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1253 |
return errors |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1254 |
self._reedit_entity(entity, action.changes, err) |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1255 |
entity.cw_edited.check() |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1256 |
self.repo.hm.call_hooks('before_update_entity', cnx, entity=entity) |
8900
010a59e12d89
use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
8821
diff
changeset
|
1257 |
sql = self.sqlgen.update(SQL_PREFIX + entity.cw_etype, action.changes, |
8265
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1258 |
['cw_eid']) |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1259 |
self.doexec(cnx, sql, action.changes) |
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1260 |
self.repo.hm.call_hooks('after_update_entity', cnx, entity=entity) |
8265
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1261 |
return errors |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1262 |
|
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1263 |
def _undo_a(self, cnx, action): |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1264 |
"""undo a relation addition""" |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1265 |
errors = [] |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1266 |
subj, rtype, obj = action.eid_from, action.rtype, action.eid_to |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1267 |
try: |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1268 |
sentity, oentity, rdef = _undo_rel_info(cnx, subj, rtype, obj) |
8695
358d8bed9626
[toward-py3k] rewrite to "except AnException as exc:" (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents:
8694
diff
changeset
|
1269 |
except _UndoException as ex: |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1270 |
errors.append(unicode(ex)) |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1271 |
else: |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1272 |
rschema = rdef.rtype |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1273 |
if rschema.inlined: |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1274 |
sql = 'SELECT 1 FROM cw_%s WHERE cw_eid=%s and cw_%s=%s'\ |
8900
010a59e12d89
use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
8821
diff
changeset
|
1275 |
% (sentity.cw_etype, subj, rtype, obj) |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1276 |
else: |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1277 |
sql = 'SELECT 1 FROM %s_relation WHERE eid_from=%s and eid_to=%s'\ |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1278 |
% (rtype, subj, obj) |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1279 |
cu = self.doexec(cnx, sql) |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1280 |
if cu.fetchone() is None: |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1281 |
errors.append(cnx._( |
5098
32b1adfb6b92
[i18n] use named substitution to avoid gettext warnings
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5076
diff
changeset
|
1282 |
"Can't undo addition of relation %(rtype)s from %(subj)s to" |
32b1adfb6b92
[i18n] use named substitution to avoid gettext warnings
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5076
diff
changeset
|
1283 |
" %(obj)s, doesn't exist anymore" % locals())) |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1284 |
if not errors: |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1285 |
self.repo.hm.call_hooks('before_delete_relation', cnx, |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1286 |
eidfrom=subj, rtype=rtype, eidto=obj) |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1287 |
# delete relation from the database |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1288 |
self._delete_relation(cnx, subj, rtype, obj, rschema.inlined) |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1289 |
# set related cache |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1290 |
cnx.update_rel_cache_del(subj, rtype, obj, rschema.symmetric) |
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1291 |
self.repo.hm.call_hooks('after_delete_relation', cnx, |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1292 |
eidfrom=subj, rtype=rtype, eidto=obj) |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1293 |
return errors |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1294 |
|
4806
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1295 |
# full text index handling ################################################# |
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1296 |
|
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1297 |
@cached |
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1298 |
def need_fti_indexation(self, etype): |
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1299 |
eschema = self.schema.eschema(etype) |
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1300 |
if any(eschema.indexable_attributes()): |
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1301 |
return True |
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1302 |
if any(eschema.fulltext_containers()): |
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1303 |
return True |
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1304 |
return False |
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1305 |
|
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1306 |
def index_entity(self, cnx, entity): |
4807
5642bfa43236
[cleanup] add index_entity to abstract source, add docstring
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4806
diff
changeset
|
1307 |
"""create an operation to [re]index textual content of the given entity |
5642bfa43236
[cleanup] add index_entity to abstract source, add docstring
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4806
diff
changeset
|
1308 |
on commit |
5642bfa43236
[cleanup] add index_entity to abstract source, add docstring
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4806
diff
changeset
|
1309 |
""" |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1310 |
FTIndexEntityOp.get_instance(cnx).add_data(entity.eid) |
4806
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1311 |
|
9574
2d4c4842bd04
[server] some s/session/cnx/
Julien Cristau <julien.cristau@logilab.fr>
parents:
9546
diff
changeset
|
1312 |
def fti_unindex_entities(self, cnx, entities): |
6889
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
1313 |
"""remove text content for entities from the full text index |
4806
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1314 |
""" |
9574
2d4c4842bd04
[server] some s/session/cnx/
Julien Cristau <julien.cristau@logilab.fr>
parents:
9546
diff
changeset
|
1315 |
cursor = cnx.cnxset.cu |
6889
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
1316 |
cursor_unindex_object = self.dbhelper.cursor_unindex_object |
4806
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1317 |
try: |
6889
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
1318 |
for entity in entities: |
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
1319 |
cursor_unindex_object(entity.eid, cursor) |
4806
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1320 |
except Exception: # let KeyboardInterrupt / SystemExit propagate |
6889
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
1321 |
self.exception('error while unindexing %s', entity) |
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
1322 |
|
4806
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1323 |
|
9574
2d4c4842bd04
[server] some s/session/cnx/
Julien Cristau <julien.cristau@logilab.fr>
parents:
9546
diff
changeset
|
1324 |
def fti_index_entities(self, cnx, entities): |
6889
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
1325 |
"""add text content of created/modified entities to the full text index |
4806
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1326 |
""" |
6889
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
1327 |
cursor_index_object = self.dbhelper.cursor_index_object |
9574
2d4c4842bd04
[server] some s/session/cnx/
Julien Cristau <julien.cristau@logilab.fr>
parents:
9546
diff
changeset
|
1328 |
cursor = cnx.cnxset.cu |
4806
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1329 |
try: |
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1330 |
# use cursor_index_object, not cursor_reindex_object since |
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1331 |
# unindexing done in the FTIndexEntityOp |
6889
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
1332 |
for entity in entities: |
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
1333 |
cursor_index_object(entity.eid, |
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
1334 |
entity.cw_adapt_to('IFTIndexable'), |
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
1335 |
cursor) |
4806
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1336 |
except Exception: # let KeyboardInterrupt / SystemExit propagate |
6889
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
1337 |
self.exception('error while indexing %s', entity) |
4806
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1338 |
|
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1339 |
|
6426
541659c39f6a
[hook/operation] nicer api to achieve same result as set_operation, as described in #1253630
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6366
diff
changeset
|
1340 |
class FTIndexEntityOp(hook.DataOperationMixIn, hook.LateOperation): |
4806
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1341 |
"""operation to delay entity full text indexation to commit |
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1342 |
|
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1343 |
since fti indexing may trigger discovery of other entities, it should be |
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1344 |
triggered on precommit, not commit, and this should be done after other |
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1345 |
precommit operation which may add relations to the entity |
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1346 |
""" |
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1347 |
|
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1348 |
def precommit_event(self): |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1349 |
cnx = self.cnx |
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1350 |
source = cnx.repo.system_source |
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1351 |
pendingeids = cnx.transaction_data.get('pendingeids', ()) |
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1352 |
done = cnx.transaction_data.setdefault('indexedeids', set()) |
6889
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
1353 |
to_reindex = set() |
6426
541659c39f6a
[hook/operation] nicer api to achieve same result as set_operation, as described in #1253630
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6366
diff
changeset
|
1354 |
for eid in self.get_data(): |
5070
b1f80ccadda3
[repo] refactor fti operation to use set_operation and a single operation whatever the number of entities to ft index
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5067
diff
changeset
|
1355 |
if eid in pendingeids or eid in done: |
b1f80ccadda3
[repo] refactor fti operation to use set_operation and a single operation whatever the number of entities to ft index
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5067
diff
changeset
|
1356 |
# entity added and deleted in the same transaction or already |
b1f80ccadda3
[repo] refactor fti operation to use set_operation and a single operation whatever the number of entities to ft index
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5067
diff
changeset
|
1357 |
# processed |
6889
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
1358 |
continue |
5070
b1f80ccadda3
[repo] refactor fti operation to use set_operation and a single operation whatever the number of entities to ft index
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5067
diff
changeset
|
1359 |
done.add(eid) |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1360 |
iftindexable = cnx.entity_from_eid(eid).cw_adapt_to('IFTIndexable') |
6889
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
1361 |
to_reindex |= set(iftindexable.fti_containers()) |
9611
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1362 |
source.fti_unindex_entities(cnx, to_reindex) |
ca853478aaa8
[server/sources/native] deal with connections, not sessions
Julien Cristau <julien.cristau@logilab.fr>
parents:
9605
diff
changeset
|
1363 |
source.fti_index_entities(cnx, to_reindex) |
0 | 1364 |
|
1365 |
def sql_schema(driver): |
|
4831
c5aec27c1bf7
[repo] use logilab.db instead of lgc.adbh/lgc.db/lgc.sqlgen/indexer, test new date extranction functions
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4829
diff
changeset
|
1366 |
helper = get_db_helper(driver) |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1367 |
typemap = helper.TYPE_MAPPING |
0 | 1368 |
schema = """ |
1369 |
/* Create the repository's system database */ |
|
1370 |
||
1371 |
%s |
|
1372 |
||
1373 |
CREATE TABLE entities ( |
|
1374 |
eid INTEGER PRIMARY KEY NOT NULL, |
|
1375 |
type VARCHAR(64) NOT NULL, |
|
8205
14c52c5ee706
[entities system table] change asource and source column size (closes #1951819)
Alexandre Richardson <alexandre.richardson@logilab.fr>
parents:
7967
diff
changeset
|
1376 |
asource VARCHAR(128) NOT NULL, |
0 | 1377 |
extid VARCHAR(256) |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1378 |
);; |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1379 |
CREATE INDEX entities_type_idx ON entities(type);; |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1380 |
CREATE INDEX entities_extid_idx ON entities(extid);; |
0 | 1381 |
|
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1382 |
CREATE TABLE transactions ( |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1383 |
tx_uuid CHAR(32) PRIMARY KEY NOT NULL, |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1384 |
tx_user INTEGER NOT NULL, |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1385 |
tx_time %s NOT NULL |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1386 |
);; |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1387 |
CREATE INDEX transactions_tx_user_idx ON transactions(tx_user);; |
9336
722635e530a0
[server] Add missing indices for undo support (closes #3259691)
Julien Cristau <julien.cristau@logilab.fr>
parents:
9298
diff
changeset
|
1388 |
CREATE INDEX transactions_tx_time_idx ON transactions(tx_time);; |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1389 |
|
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1390 |
CREATE TABLE tx_entity_actions ( |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1391 |
tx_uuid CHAR(32) REFERENCES transactions(tx_uuid) ON DELETE CASCADE, |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1392 |
txa_action CHAR(1) NOT NULL, |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1393 |
txa_public %s NOT NULL, |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1394 |
txa_order INTEGER, |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1395 |
eid INTEGER NOT NULL, |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1396 |
etype VARCHAR(64) NOT NULL, |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1397 |
changes %s |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1398 |
);; |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1399 |
CREATE INDEX tx_entity_actions_txa_action_idx ON tx_entity_actions(txa_action);; |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1400 |
CREATE INDEX tx_entity_actions_txa_public_idx ON tx_entity_actions(txa_public);; |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1401 |
CREATE INDEX tx_entity_actions_eid_idx ON tx_entity_actions(eid);; |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1402 |
CREATE INDEX tx_entity_actions_etype_idx ON tx_entity_actions(etype);; |
9336
722635e530a0
[server] Add missing indices for undo support (closes #3259691)
Julien Cristau <julien.cristau@logilab.fr>
parents:
9298
diff
changeset
|
1403 |
CREATE INDEX tx_entity_actions_tx_uuid_idx ON tx_entity_actions(tx_uuid);; |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1404 |
|
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1405 |
CREATE TABLE tx_relation_actions ( |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1406 |
tx_uuid CHAR(32) REFERENCES transactions(tx_uuid) ON DELETE CASCADE, |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1407 |
txa_action CHAR(1) NOT NULL, |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1408 |
txa_public %s NOT NULL, |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1409 |
txa_order INTEGER, |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1410 |
eid_from INTEGER NOT NULL, |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1411 |
eid_to INTEGER NOT NULL, |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1412 |
rtype VARCHAR(256) NOT NULL |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1413 |
);; |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1414 |
CREATE INDEX tx_relation_actions_txa_action_idx ON tx_relation_actions(txa_action);; |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1415 |
CREATE INDEX tx_relation_actions_txa_public_idx ON tx_relation_actions(txa_public);; |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1416 |
CREATE INDEX tx_relation_actions_eid_from_idx ON tx_relation_actions(eid_from);; |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1417 |
CREATE INDEX tx_relation_actions_eid_to_idx ON tx_relation_actions(eid_to);; |
9336
722635e530a0
[server] Add missing indices for undo support (closes #3259691)
Julien Cristau <julien.cristau@logilab.fr>
parents:
9298
diff
changeset
|
1418 |
CREATE INDEX tx_relation_actions_tx_uuid_idx ON tx_relation_actions(tx_uuid);; |
9585
3f5b59527d31
[source/native] allow many eid creation per .create_eid call (closes #3526594)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9583
diff
changeset
|
1419 |
""" % (helper.sql_create_numrange('entities_id_seq').replace(';', ';;'), |
9450
af4b93bc38a5
[multi-sources-removal] Drop deleted_entities system table and entities.mtime column
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
9448
diff
changeset
|
1420 |
typemap['Datetime'], |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1421 |
typemap['Boolean'], typemap['Bytes'], typemap['Boolean']) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1422 |
if helper.backend_name == 'sqlite': |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1423 |
# sqlite support the ON DELETE CASCADE syntax but do nothing |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1424 |
schema += ''' |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1425 |
CREATE TRIGGER fkd_transactions |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1426 |
BEFORE DELETE ON transactions |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1427 |
FOR EACH ROW BEGIN |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1428 |
DELETE FROM tx_entity_actions WHERE tx_uuid=OLD.tx_uuid; |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1429 |
DELETE FROM tx_relation_actions WHERE tx_uuid=OLD.tx_uuid; |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1430 |
END;; |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1431 |
''' |
0 | 1432 |
return schema |
1433 |
||
1434 |
||
1435 |
def sql_drop_schema(driver): |
|
4831
c5aec27c1bf7
[repo] use logilab.db instead of lgc.adbh/lgc.db/lgc.sqlgen/indexer, test new date extranction functions
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4829
diff
changeset
|
1436 |
helper = get_db_helper(driver) |
0 | 1437 |
return """ |
1438 |
%s |
|
1439 |
DROP TABLE entities; |
|
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1440 |
DROP TABLE tx_entity_actions; |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1441 |
DROP TABLE tx_relation_actions; |
5214
3285b6e3b930
fix cwctl db-init -d on SQL Server
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5168
diff
changeset
|
1442 |
DROP TABLE transactions; |
9585
3f5b59527d31
[source/native] allow many eid creation per .create_eid call (closes #3526594)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9583
diff
changeset
|
1443 |
""" % helper.sql_drop_numrange('entities_id_seq') |
0 | 1444 |
|
1445 |
||
1446 |
def grant_schema(user, set_owner=True): |
|
1447 |
result = '' |
|
9450
af4b93bc38a5
[multi-sources-removal] Drop deleted_entities system table and entities.mtime column
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
9448
diff
changeset
|
1448 |
for table in ('entities', 'entities_id_seq', |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1449 |
'transactions', 'tx_entity_actions', 'tx_relation_actions'): |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1450 |
if set_owner: |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1451 |
result = 'ALTER TABLE %s OWNER TO %s;\n' % (table, user) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1452 |
result += 'GRANT ALL ON %s TO %s;\n' % (table, user) |
0 | 1453 |
return result |
3647
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1454 |
|
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1455 |
|
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1456 |
class BaseAuthentifier(object): |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1457 |
|
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1458 |
def __init__(self, source=None): |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1459 |
self.source = source |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1460 |
|
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1461 |
def set_schema(self, schema): |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1462 |
"""set the instance'schema""" |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1463 |
pass |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1464 |
|
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1465 |
class LoginPasswordAuthentifier(BaseAuthentifier): |
8520
fcd048fa6e6d
[source/native] only system source users should be authenticated by the native source (closes #2465883)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
8508
diff
changeset
|
1466 |
passwd_rql = 'Any P WHERE X is CWUser, X login %(login)s, X upassword P' |
fcd048fa6e6d
[source/native] only system source users should be authenticated by the native source (closes #2465883)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
8508
diff
changeset
|
1467 |
auth_rql = ('Any X WHERE X is CWUser, X login %(login)s, X upassword %(pwd)s, ' |
fcd048fa6e6d
[source/native] only system source users should be authenticated by the native source (closes #2465883)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
8508
diff
changeset
|
1468 |
'X cw_source S, S name "system"') |
fcd048fa6e6d
[source/native] only system source users should be authenticated by the native source (closes #2465883)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
8508
diff
changeset
|
1469 |
_sols = ({'X': 'CWUser', 'P': 'Password', 'S': 'CWSource'},) |
3647
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1470 |
|
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1471 |
def set_schema(self, schema): |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1472 |
"""set the instance'schema""" |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1473 |
if 'CWUser' in schema: # probably an empty schema if not true... |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1474 |
# rql syntax trees used to authenticate users |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1475 |
self._passwd_rqlst = self.source.compile_rql(self.passwd_rql, self._sols) |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1476 |
self._auth_rqlst = self.source.compile_rql(self.auth_rql, self._sols) |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1477 |
|
9512
88dc96fc9fc1
[server] use a connection instead of a session for user authentication
Julien Cristau <julien.cristau@logilab.fr>
parents:
9487
diff
changeset
|
1478 |
def authenticate(self, cnx, login, password=None, **kwargs): |
3647
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1479 |
"""return CWUser eid for the given login/password if this account is |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1480 |
defined in this source, else raise `AuthenticationError` |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1481 |
|
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1482 |
two queries are needed since passwords are stored crypted, so we have |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1483 |
to fetch the salt first |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1484 |
""" |
6018
f4d1d5d9ccbb
[security] don't put uncrypted password in query parameters, else it may be logged on error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5975
diff
changeset
|
1485 |
args = {'login': login, 'pwd' : None} |
3647
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1486 |
if password is not None: |
9512
88dc96fc9fc1
[server] use a connection instead of a session for user authentication
Julien Cristau <julien.cristau@logilab.fr>
parents:
9487
diff
changeset
|
1487 |
rset = self.source.syntax_tree_search(cnx, self._passwd_rqlst, args) |
3647
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1488 |
try: |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1489 |
pwd = rset[0][0] |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1490 |
except IndexError: |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1491 |
raise AuthenticationError('bad login') |
8235
c2a91d6639d8
[system source]Â fix authentication: don't allow login whatever the given password if user password is NULL in database. Closes #2186099
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
8205
diff
changeset
|
1492 |
if pwd is None: |
c2a91d6639d8
[system source]Â fix authentication: don't allow login whatever the given password if user password is NULL in database. Closes #2186099
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
8205
diff
changeset
|
1493 |
# if pwd is None but a password is provided, something is wrong |
c2a91d6639d8
[system source]Â fix authentication: don't allow login whatever the given password if user password is NULL in database. Closes #2186099
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
8205
diff
changeset
|
1494 |
raise AuthenticationError('bad password') |
3647
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1495 |
# passwords are stored using the Bytes type, so we get a StringIO |
8317
9c59258e7798
[security] use a stronger encryption algorythm for password, keeping bw compat
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
8235
diff
changeset
|
1496 |
args['pwd'] = Binary(crypt_password(password, pwd.getvalue())) |
3647
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1497 |
# get eid from login and (crypted) password |
9512
88dc96fc9fc1
[server] use a connection instead of a session for user authentication
Julien Cristau <julien.cristau@logilab.fr>
parents:
9487
diff
changeset
|
1498 |
rset = self.source.syntax_tree_search(cnx, self._auth_rqlst, args) |
10037
6b83ceda8323
[server] fix LoginPasswordAuthentifier.authenticate
David Douard <david.douard@logilab.fr>
parents:
9996
diff
changeset
|
1499 |
pwd = args['pwd'] |
3647
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1500 |
try: |
8546
3d2038d6f20d
[sources/native] automatically update passwords using deprecated hashes on login
Julien Cristau <julien.cristau@logilab.fr>
parents:
8520
diff
changeset
|
1501 |
user = rset[0][0] |
3d2038d6f20d
[sources/native] automatically update passwords using deprecated hashes on login
Julien Cristau <julien.cristau@logilab.fr>
parents:
8520
diff
changeset
|
1502 |
# If the stored hash uses a deprecated scheme (e.g. DES or MD5 used |
3d2038d6f20d
[sources/native] automatically update passwords using deprecated hashes on login
Julien Cristau <julien.cristau@logilab.fr>
parents:
8520
diff
changeset
|
1503 |
# before 3.14.7), update with a fresh one |
10037
6b83ceda8323
[server] fix LoginPasswordAuthentifier.authenticate
David Douard <david.douard@logilab.fr>
parents:
9996
diff
changeset
|
1504 |
if pwd is not None and pwd.getvalue(): |
8546
3d2038d6f20d
[sources/native] automatically update passwords using deprecated hashes on login
Julien Cristau <julien.cristau@logilab.fr>
parents:
8520
diff
changeset
|
1505 |
verify, newhash = verify_and_update(password, pwd.getvalue()) |
3d2038d6f20d
[sources/native] automatically update passwords using deprecated hashes on login
Julien Cristau <julien.cristau@logilab.fr>
parents:
8520
diff
changeset
|
1506 |
if not verify: # should not happen, but... |
3d2038d6f20d
[sources/native] automatically update passwords using deprecated hashes on login
Julien Cristau <julien.cristau@logilab.fr>
parents:
8520
diff
changeset
|
1507 |
raise AuthenticationError('bad password') |
3d2038d6f20d
[sources/native] automatically update passwords using deprecated hashes on login
Julien Cristau <julien.cristau@logilab.fr>
parents:
8520
diff
changeset
|
1508 |
if newhash: |
9512
88dc96fc9fc1
[server] use a connection instead of a session for user authentication
Julien Cristau <julien.cristau@logilab.fr>
parents:
9487
diff
changeset
|
1509 |
cnx.system_sql("UPDATE %s SET %s=%%(newhash)s WHERE %s=%%(login)s" % ( |
8546
3d2038d6f20d
[sources/native] automatically update passwords using deprecated hashes on login
Julien Cristau <julien.cristau@logilab.fr>
parents:
8520
diff
changeset
|
1510 |
SQL_PREFIX + 'CWUser', |
3d2038d6f20d
[sources/native] automatically update passwords using deprecated hashes on login
Julien Cristau <julien.cristau@logilab.fr>
parents:
8520
diff
changeset
|
1511 |
SQL_PREFIX + 'upassword', |
3d2038d6f20d
[sources/native] automatically update passwords using deprecated hashes on login
Julien Cristau <julien.cristau@logilab.fr>
parents:
8520
diff
changeset
|
1512 |
SQL_PREFIX + 'login'), |
3d2038d6f20d
[sources/native] automatically update passwords using deprecated hashes on login
Julien Cristau <julien.cristau@logilab.fr>
parents:
8520
diff
changeset
|
1513 |
{'newhash': self.source._binary(newhash), |
3d2038d6f20d
[sources/native] automatically update passwords using deprecated hashes on login
Julien Cristau <julien.cristau@logilab.fr>
parents:
8520
diff
changeset
|
1514 |
'login': login}) |
9512
88dc96fc9fc1
[server] use a connection instead of a session for user authentication
Julien Cristau <julien.cristau@logilab.fr>
parents:
9487
diff
changeset
|
1515 |
cnx.commit(free_cnxset=False) |
8546
3d2038d6f20d
[sources/native] automatically update passwords using deprecated hashes on login
Julien Cristau <julien.cristau@logilab.fr>
parents:
8520
diff
changeset
|
1516 |
return user |
3647
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1517 |
except IndexError: |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1518 |
raise AuthenticationError('bad password') |
6849
5a0c2cfc19bf
[repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6724
diff
changeset
|
1519 |
|
5a0c2cfc19bf
[repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6724
diff
changeset
|
1520 |
|
5a0c2cfc19bf
[repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6724
diff
changeset
|
1521 |
class EmailPasswordAuthentifier(BaseAuthentifier): |
9512
88dc96fc9fc1
[server] use a connection instead of a session for user authentication
Julien Cristau <julien.cristau@logilab.fr>
parents:
9487
diff
changeset
|
1522 |
def authenticate(self, cnx, login, **authinfo): |
6849
5a0c2cfc19bf
[repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6724
diff
changeset
|
1523 |
# email_auth flag prevent from infinite recursion (call to |
5a0c2cfc19bf
[repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6724
diff
changeset
|
1524 |
# repo.check_auth_info at the end of this method may lead us here again) |
5a0c2cfc19bf
[repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6724
diff
changeset
|
1525 |
if not '@' in login or authinfo.pop('email_auth', None): |
5a0c2cfc19bf
[repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6724
diff
changeset
|
1526 |
raise AuthenticationError('not an email') |
9512
88dc96fc9fc1
[server] use a connection instead of a session for user authentication
Julien Cristau <julien.cristau@logilab.fr>
parents:
9487
diff
changeset
|
1527 |
rset = cnx.execute('Any L WHERE U login L, U primary_email M, ' |
6849
5a0c2cfc19bf
[repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6724
diff
changeset
|
1528 |
'M address %(login)s', {'login': login}, |
5a0c2cfc19bf
[repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6724
diff
changeset
|
1529 |
build_descr=False) |
5a0c2cfc19bf
[repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6724
diff
changeset
|
1530 |
if rset.rowcount != 1: |
5a0c2cfc19bf
[repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6724
diff
changeset
|
1531 |
raise AuthenticationError('unexisting email') |
5a0c2cfc19bf
[repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6724
diff
changeset
|
1532 |
login = rset.rows[0][0] |
5a0c2cfc19bf
[repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6724
diff
changeset
|
1533 |
authinfo['email_auth'] = True |
9512
88dc96fc9fc1
[server] use a connection instead of a session for user authentication
Julien Cristau <julien.cristau@logilab.fr>
parents:
9487
diff
changeset
|
1534 |
return self.source.repo.check_auth_info(cnx, login, authinfo) |
7342
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1535 |
|
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1536 |
|
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1537 |
class DatabaseIndependentBackupRestore(object): |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1538 |
"""Helper class to perform db backend agnostic backup and restore |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1539 |
|
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1540 |
The backup and restore methods are used to dump / restore the |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1541 |
system database in a database independent format. The file is a |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1542 |
Zip archive containing the following files: |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1543 |
|
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1544 |
* format.txt: the format of the archive. Currently '1.0' |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1545 |
* tables.txt: list of filenames in the archive tables/ directory |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1546 |
* sequences.txt: list of filenames in the archive sequences/ directory |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1547 |
* versions.txt: the list of cube versions from CWProperty |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1548 |
* tables/<tablename>.<chunkno>: pickled data |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1549 |
* sequences/<sequencename>: pickled data |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1550 |
|
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1551 |
The pickled data format for tables and sequences is a tuple of 3 elements: |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1552 |
* the table name |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1553 |
* a tuple of column names |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1554 |
* a list of rows (as tuples with one element per column) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1555 |
|
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1556 |
Tables are saved in chunks in different files in order to prevent |
7551 | 1557 |
a too high memory consumption. |
7342
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1558 |
""" |
10048
7d9256aab337
[source/native/backup_restore] have a unique tunable blocksize for the dump phase
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9336
diff
changeset
|
1559 |
blocksize = 100 |
7d9256aab337
[source/native/backup_restore] have a unique tunable blocksize for the dump phase
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9336
diff
changeset
|
1560 |
|
7342
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1561 |
def __init__(self, source): |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1562 |
""" |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1563 |
:param: source an instance of the system source |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1564 |
""" |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1565 |
self._source = source |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1566 |
self.logger = logging.getLogger('cubicweb.ctl') |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1567 |
self.logger.setLevel(logging.INFO) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1568 |
self.logger.addHandler(logging.StreamHandler(sys.stdout)) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1569 |
self.schema = self._source.schema |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1570 |
self.dbhelper = self._source.dbhelper |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1571 |
self.cnx = None |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1572 |
self.cursor = None |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1573 |
self.sql_generator = sqlgen.SQLGenerator() |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1574 |
|
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1575 |
def get_connection(self): |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1576 |
return self._source.get_connection() |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1577 |
|
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1578 |
def backup(self, backupfile): |
8910
7652c3d46ba3
[portable dump] skip virtual relations, they have no table associated. Closes #2841199
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
8821
diff
changeset
|
1579 |
archive = zipfile.ZipFile(backupfile, 'w', allowZip64=True) |
7342
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1580 |
self.cnx = self.get_connection() |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1581 |
try: |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1582 |
self.cursor = self.cnx.cursor() |
9468
39b7a91a3f4c
[repo] pylint cleanup, mainly of imports, with a bit of style
Julien Cristau <julien.cristau@logilab.fr>
parents:
9467
diff
changeset
|
1583 |
self.cursor.arraysize = 100 |
7342
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1584 |
self.logger.info('writing metadata') |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1585 |
self.write_metadata(archive) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1586 |
for seq in self.get_sequences(): |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1587 |
self.logger.info('processing sequence %s', seq) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1588 |
self.write_sequence(archive, seq) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1589 |
for table in self.get_tables(): |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1590 |
self.logger.info('processing table %s', table) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1591 |
self.write_table(archive, table) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1592 |
finally: |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1593 |
archive.close() |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1594 |
self.cnx.close() |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1595 |
self.logger.info('done') |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1596 |
|
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1597 |
def get_tables(self): |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1598 |
non_entity_tables = ['entities', |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1599 |
'transactions', |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1600 |
'tx_entity_actions', |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1601 |
'tx_relation_actions', |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1602 |
] |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1603 |
etype_tables = [] |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1604 |
relation_tables = [] |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1605 |
prefix = 'cw_' |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1606 |
for etype in self.schema.entities(): |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1607 |
eschema = self.schema.eschema(etype) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1608 |
if eschema.final: |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1609 |
continue |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1610 |
etype_tables.append('%s%s'%(prefix, etype)) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1611 |
for rtype in self.schema.relations(): |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1612 |
rschema = self.schema.rschema(rtype) |
8910
7652c3d46ba3
[portable dump] skip virtual relations, they have no table associated. Closes #2841199
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
8821
diff
changeset
|
1613 |
if rschema.final or rschema.inlined or rschema in VIRTUAL_RTYPES: |
7342
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1614 |
continue |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1615 |
relation_tables.append('%s_relation' % rtype) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1616 |
return non_entity_tables + etype_tables + relation_tables |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1617 |
|
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1618 |
def get_sequences(self): |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1619 |
return ['entities_id_seq'] |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1620 |
|
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1621 |
def write_metadata(self, archive): |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1622 |
archive.writestr('format.txt', '1.0') |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1623 |
archive.writestr('tables.txt', '\n'.join(self.get_tables())) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1624 |
archive.writestr('sequences.txt', '\n'.join(self.get_sequences())) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1625 |
versions = self._get_versions() |
9468
39b7a91a3f4c
[repo] pylint cleanup, mainly of imports, with a bit of style
Julien Cristau <julien.cristau@logilab.fr>
parents:
9467
diff
changeset
|
1626 |
versions_str = '\n'.join('%s %s' % (k, v) |
39b7a91a3f4c
[repo] pylint cleanup, mainly of imports, with a bit of style
Julien Cristau <julien.cristau@logilab.fr>
parents:
9467
diff
changeset
|
1627 |
for k, v in versions) |
7342
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1628 |
archive.writestr('versions.txt', versions_str) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1629 |
|
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1630 |
def write_sequence(self, archive, seq): |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1631 |
sql = self.dbhelper.sql_sequence_current_state(seq) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1632 |
columns, rows_iterator = self._get_cols_and_rows(sql) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1633 |
rows = list(rows_iterator) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1634 |
serialized = self._serialize(seq, columns, rows) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1635 |
archive.writestr('sequences/%s' % seq, serialized) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1636 |
|
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1637 |
def write_table(self, archive, table): |
7753
681ef2a664dd
work around cursor.rowcount not returning anything useful before 1st fetch with mssql (closes #1910869)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7551
diff
changeset
|
1638 |
nb_lines_sql = 'SELECT COUNT(*) FROM %s' % table |
681ef2a664dd
work around cursor.rowcount not returning anything useful before 1st fetch with mssql (closes #1910869)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7551
diff
changeset
|
1639 |
self.cursor.execute(nb_lines_sql) |
681ef2a664dd
work around cursor.rowcount not returning anything useful before 1st fetch with mssql (closes #1910869)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7551
diff
changeset
|
1640 |
rowcount = self.cursor.fetchone()[0] |
7342
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1641 |
sql = 'SELECT * FROM %s' % table |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1642 |
columns, rows_iterator = self._get_cols_and_rows(sql) |
7753
681ef2a664dd
work around cursor.rowcount not returning anything useful before 1st fetch with mssql (closes #1910869)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7551
diff
changeset
|
1643 |
self.logger.info('number of rows: %d', rowcount) |
10048
7d9256aab337
[source/native/backup_restore] have a unique tunable blocksize for the dump phase
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
9336
diff
changeset
|
1644 |
blocksize = self.blocksize |
7753
681ef2a664dd
work around cursor.rowcount not returning anything useful before 1st fetch with mssql (closes #1910869)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7551
diff
changeset
|
1645 |
if rowcount > 0: |
681ef2a664dd
work around cursor.rowcount not returning anything useful before 1st fetch with mssql (closes #1910869)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7551
diff
changeset
|
1646 |
for i, start in enumerate(xrange(0, rowcount, blocksize)): |
7342
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1647 |
rows = list(itertools.islice(rows_iterator, blocksize)) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1648 |
serialized = self._serialize(table, columns, rows) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1649 |
archive.writestr('tables/%s.%04d' % (table, i), serialized) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1650 |
self.logger.debug('wrote rows %d to %d (out of %d) to %s.%04d', |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1651 |
start, start+len(rows)-1, |
7753
681ef2a664dd
work around cursor.rowcount not returning anything useful before 1st fetch with mssql (closes #1910869)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7551
diff
changeset
|
1652 |
rowcount, |
7342
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1653 |
table, i) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1654 |
else: |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1655 |
rows = [] |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1656 |
serialized = self._serialize(table, columns, rows) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1657 |
archive.writestr('tables/%s.%04d' % (table, 0), serialized) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1658 |
|
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1659 |
def _get_cols_and_rows(self, sql): |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1660 |
process_result = self._source.iter_process_result |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1661 |
self.cursor.execute(sql) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1662 |
columns = (d[0] for d in self.cursor.description) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1663 |
rows = process_result(self.cursor) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1664 |
return tuple(columns), rows |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1665 |
|
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1666 |
def _serialize(self, name, columns, rows): |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1667 |
return dumps((name, columns, rows), pickle.HIGHEST_PROTOCOL) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1668 |
|
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1669 |
def restore(self, backupfile): |
7787
d9607ae447d2
[server] portable dump format supports now ZIP64 extensions by default (closes #1912535)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7786
diff
changeset
|
1670 |
archive = zipfile.ZipFile(backupfile, 'r', allowZip64=True) |
7342
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1671 |
self.cnx = self.get_connection() |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1672 |
self.cursor = self.cnx.cursor() |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1673 |
sequences, tables, table_chunks = self.read_metadata(archive, backupfile) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1674 |
for seq in sequences: |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1675 |
self.logger.info('restoring sequence %s', seq) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1676 |
self.read_sequence(archive, seq) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1677 |
for table in tables: |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1678 |
self.logger.info('restoring table %s', table) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1679 |
self.read_table(archive, table, sorted(table_chunks[table])) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1680 |
self.cnx.close() |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1681 |
archive.close() |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1682 |
self.logger.info('done') |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1683 |
|
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1684 |
def read_metadata(self, archive, backupfile): |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1685 |
formatinfo = archive.read('format.txt') |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1686 |
self.logger.info('checking metadata') |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1687 |
if formatinfo.strip() != "1.0": |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1688 |
self.logger.critical('Unsupported format in archive: %s', formatinfo) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1689 |
raise ValueError('Unknown format in %s: %s' % (backupfile, formatinfo)) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1690 |
tables = archive.read('tables.txt').splitlines() |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1691 |
sequences = archive.read('sequences.txt').splitlines() |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1692 |
file_versions = self._parse_versions(archive.read('versions.txt')) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1693 |
versions = set(self._get_versions()) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1694 |
if file_versions != versions: |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1695 |
self.logger.critical('Unable to restore : versions do not match') |
8379
1a3746ec4d65
[sources/native] do not let a log message crash on us
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
8317
diff
changeset
|
1696 |
self.logger.critical('Expected:\n%s', '\n'.join('%s : %s' % (cube, ver) |
1a3746ec4d65
[sources/native] do not let a log message crash on us
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
8317
diff
changeset
|
1697 |
for cube, ver in sorted(versions))) |
1a3746ec4d65
[sources/native] do not let a log message crash on us
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
8317
diff
changeset
|
1698 |
self.logger.critical('Found:\n%s', '\n'.join('%s : %s' % (cube, ver) |
1a3746ec4d65
[sources/native] do not let a log message crash on us
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
8317
diff
changeset
|
1699 |
for cube, ver in sorted(file_versions))) |
7342
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1700 |
raise ValueError('Unable to restore : versions do not match') |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1701 |
table_chunks = {} |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1702 |
for name in archive.namelist(): |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1703 |
if not name.startswith('tables/'): |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1704 |
continue |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1705 |
filename = basename(name) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1706 |
tablename, _ext = filename.rsplit('.', 1) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1707 |
table_chunks.setdefault(tablename, []).append(name) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1708 |
return sequences, tables, table_chunks |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1709 |
|
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1710 |
def read_sequence(self, archive, seq): |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1711 |
seqname, columns, rows = loads(archive.read('sequences/%s' % seq)) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1712 |
assert seqname == seq |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1713 |
assert len(rows) == 1 |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1714 |
assert len(rows[0]) == 1 |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1715 |
value = rows[0][0] |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1716 |
sql = self.dbhelper.sql_restart_sequence(seq, value) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1717 |
self.cursor.execute(sql) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1718 |
self.cnx.commit() |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1719 |
|
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1720 |
def read_table(self, archive, table, filenames): |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1721 |
merge_args = self._source.merge_args |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1722 |
self.cursor.execute('DELETE FROM %s' % table) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1723 |
self.cnx.commit() |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1724 |
row_count = 0 |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1725 |
for filename in filenames: |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1726 |
tablename, columns, rows = loads(archive.read(filename)) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1727 |
assert tablename == table |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1728 |
if not rows: |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1729 |
continue |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1730 |
insert = self.sql_generator.insert(table, |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1731 |
dict(zip(columns, rows[0]))) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1732 |
for row in rows: |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1733 |
self.cursor.execute(insert, merge_args(dict(zip(columns, row)), {})) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1734 |
row_count += len(rows) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1735 |
self.cnx.commit() |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1736 |
self.logger.info('inserted %d rows', row_count) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1737 |
|
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1738 |
|
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1739 |
def _parse_versions(self, version_str): |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1740 |
versions = set() |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1741 |
for line in version_str.splitlines(): |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1742 |
versions.add(tuple(line.split())) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1743 |
return versions |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1744 |
|
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1745 |
def _get_versions(self): |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1746 |
version_sql = 'SELECT cw_pkey, cw_value FROM cw_CWProperty' |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1747 |
versions = [] |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1748 |
self.cursor.execute(version_sql) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1749 |
for pkey, value in self.cursor.fetchall(): |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1750 |
if pkey.startswith(u'system.version'): |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1751 |
versions.append((pkey, value)) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1752 |
return versions |