author | Pierre-Yves David <pierre-yves.david@logilab.fr> |
Fri, 14 Jun 2013 12:10:45 +0200 | |
changeset 9025 | 768eb9a6a2db |
parent 8947 | 3bbd416b09ec |
child 9098 | 5467fb901931 |
child 9130 | 0f1504a9fb51 |
permissions | -rw-r--r-- |
8910
7652c3d46ba3
[portable dump] skip virtual relations, they have no table associated. Closes #2841199
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
8821
diff
changeset
|
1 |
# copyright 2003-2013 LOGILAB S.A. (Paris, FRANCE), all rights reserved. |
5421
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5413
diff
changeset
|
2 |
# contact http://www.logilab.fr/ -- mailto:contact@logilab.fr |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5413
diff
changeset
|
3 |
# |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5413
diff
changeset
|
4 |
# This file is part of CubicWeb. |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5413
diff
changeset
|
5 |
# |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5413
diff
changeset
|
6 |
# CubicWeb is free software: you can redistribute it and/or modify it under the |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5413
diff
changeset
|
7 |
# terms of the GNU Lesser General Public License as published by the Free |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5413
diff
changeset
|
8 |
# Software Foundation, either version 2.1 of the License, or (at your option) |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5413
diff
changeset
|
9 |
# any later version. |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5413
diff
changeset
|
10 |
# |
5424
8ecbcbff9777
replace logilab-common by CubicWeb in disclaimer
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5421
diff
changeset
|
11 |
# CubicWeb is distributed in the hope that it will be useful, but WITHOUT |
5421
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5413
diff
changeset
|
12 |
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5413
diff
changeset
|
13 |
# FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5413
diff
changeset
|
14 |
# details. |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5413
diff
changeset
|
15 |
# |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5413
diff
changeset
|
16 |
# You should have received a copy of the GNU Lesser General Public License along |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5413
diff
changeset
|
17 |
# with CubicWeb. If not, see <http://www.gnu.org/licenses/>. |
0 | 18 |
"""Adapters for native cubicweb sources. |
19 |
||
1952
8e19c813750d
fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1792
diff
changeset
|
20 |
Notes: |
8e19c813750d
fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1792
diff
changeset
|
21 |
* extid (aka external id, the primary key of an entity in the external source |
8e19c813750d
fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1792
diff
changeset
|
22 |
from which it comes from) are stored in a varchar column encoded as a base64 |
8e19c813750d
fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1792
diff
changeset
|
23 |
string. This is because it should actually be Bytes but we want an index on |
8e19c813750d
fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1792
diff
changeset
|
24 |
it for fast querying. |
5824 | 25 |
""" |
0 | 26 |
__docformat__ = "restructuredtext en" |
27 |
||
7342
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
28 |
try: |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
29 |
from cPickle import loads, dumps |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
30 |
import cPickle as pickle |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
31 |
except ImportError: |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
32 |
from pickle import loads, dumps |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
33 |
import pickle |
4900 | 34 |
from threading import Lock |
1016
26387b836099
use datetime instead of mx.DateTime
sylvain.thenault@logilab.fr
parents:
973
diff
changeset
|
35 |
from datetime import datetime |
1952
8e19c813750d
fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1792
diff
changeset
|
36 |
from base64 import b64decode, b64encode |
4964
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
37 |
from contextlib import contextmanager |
7342
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
38 |
from os.path import abspath, basename |
6211
e9d125fd1465
nicer error reporting for unique together constraints
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6201
diff
changeset
|
39 |
import re |
6889
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
40 |
import itertools |
7342
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
41 |
import zipfile |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
42 |
import logging |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
43 |
import sys |
0 | 44 |
|
4806
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
45 |
from logilab.common.compat import any |
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
46 |
from logilab.common.decorators import cached, clear_cache |
3835
a191b3b9e455
more sensible default values to c-c "create" inputs
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3689
diff
changeset
|
47 |
from logilab.common.configuration import Method |
a191b3b9e455
more sensible default values to c-c "create" inputs
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3689
diff
changeset
|
48 |
from logilab.common.shellutils import getlogin |
7342
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
49 |
from logilab.database import get_db_helper, sqlgen |
0 | 50 |
|
5891
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
51 |
from yams import schema2sql as y2sql |
6945
28bf94d062a9
[sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6931
diff
changeset
|
52 |
from yams.schema import role_name |
5891
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
53 |
|
6945
28bf94d062a9
[sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6931
diff
changeset
|
54 |
from cubicweb import (UnknownEid, AuthenticationError, ValidationError, Binary, |
8265
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
55 |
UniqueTogetherError, QueryError, UndoTransactionException) |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
56 |
from cubicweb import transaction as tx, server, neg_role |
7954
a3d3220669d6
[cache] replace lgc.cache with something more appropriate (closes #1921713)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
7922
diff
changeset
|
57 |
from cubicweb.utils import QueryCache |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
58 |
from cubicweb.schema import VIRTUAL_RTYPES |
3835
a191b3b9e455
more sensible default values to c-c "create" inputs
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3689
diff
changeset
|
59 |
from cubicweb.cwconfig import CubicWebNoAppConfiguration |
4806
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
60 |
from cubicweb.server import hook |
8546
3d2038d6f20d
[sources/native] automatically update passwords using deprecated hashes on login
Julien Cristau <julien.cristau@logilab.fr>
parents:
8520
diff
changeset
|
61 |
from cubicweb.server.utils import crypt_password, eschema_eid, verify_and_update |
2759
23d7a75693f8
R refactor backup and use tar.gz to store all sources
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents:
2638
diff
changeset
|
62 |
from cubicweb.server.sqlutils import SQL_PREFIX, SQLAdapterMixIn |
0 | 63 |
from cubicweb.server.rqlannotation import set_qdata |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
64 |
from cubicweb.server.hook import CleanupDeletedEidsCacheOp |
7118
e094b3d4eb95
[server] move EditedEntity class to its own module, to avoid cyclic dependency when needed from e.g. session.py
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6957
diff
changeset
|
65 |
from cubicweb.server.edition import EditedEntity |
2625
d6012db7b93e
R [server debug]Â more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2620
diff
changeset
|
66 |
from cubicweb.server.sources import AbstractSource, dbg_st_search, dbg_results |
0 | 67 |
from cubicweb.server.sources.rql2sql import SQLGenerator |
68 |
||
69 |
||
2354
9b4bac626977
ability to map attributes to something else than usual cw mapping on sql generation
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2310
diff
changeset
|
70 |
ATTR_MAP = {} |
0 | 71 |
NONSYSTEM_ETYPES = set() |
72 |
NONSYSTEM_RELATIONS = set() |
|
73 |
||
74 |
class LogCursor(object): |
|
75 |
def __init__(self, cursor): |
|
76 |
self.cu = cursor |
|
1792 | 77 |
|
0 | 78 |
def execute(self, query, args=None): |
79 |
"""Execute a query. |
|
80 |
it's a function just so that it shows up in profiling |
|
81 |
""" |
|
2593
16d9419a4a79
F: start to handle binary debug log level on the server side
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2589
diff
changeset
|
82 |
if server.DEBUG & server.DBG_SQL: |
0 | 83 |
print 'exec', query, args |
84 |
try: |
|
85 |
self.cu.execute(str(query), args) |
|
8695
358d8bed9626
[toward-py3k] rewrite to "except AnException as exc:" (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents:
8694
diff
changeset
|
86 |
except Exception as ex: |
0 | 87 |
print "sql: %r\n args: %s\ndbms message: %r" % ( |
88 |
query, args, ex.args[0]) |
|
89 |
raise |
|
1792 | 90 |
|
0 | 91 |
def fetchall(self): |
92 |
return self.cu.fetchall() |
|
1792 | 93 |
|
0 | 94 |
def fetchone(self): |
95 |
return self.cu.fetchone() |
|
1792 | 96 |
|
2625
d6012db7b93e
R [server debug]Â more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2620
diff
changeset
|
97 |
|
0 | 98 |
def make_schema(selected, solution, table, typemap): |
99 |
"""return a sql schema to store RQL query result""" |
|
100 |
sql = [] |
|
101 |
varmap = {} |
|
102 |
for i, term in enumerate(selected): |
|
103 |
name = 'C%s' % i |
|
104 |
key = term.as_string() |
|
105 |
varmap[key] = '%s.%s' % (table, name) |
|
106 |
ttype = term.get_type(solution) |
|
107 |
try: |
|
108 |
sql.append('%s %s' % (name, typemap[ttype])) |
|
109 |
except KeyError: |
|
3689
deb13e88e037
follow yams 0.25 api changes to improve performance
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3497
diff
changeset
|
110 |
# assert not schema(ttype).final |
0 | 111 |
sql.append('%s %s' % (name, typemap['Int'])) |
112 |
return ','.join(sql), varmap |
|
113 |
||
2625
d6012db7b93e
R [server debug]Â more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2620
diff
changeset
|
114 |
|
0 | 115 |
def _modified_sql(table, etypes): |
116 |
# XXX protect against sql injection |
|
117 |
if len(etypes) > 1: |
|
118 |
restr = 'type IN (%s)' % ','.join("'%s'" % etype for etype in etypes) |
|
119 |
else: |
|
120 |
restr = "type='%s'" % etypes[0] |
|
121 |
if table == 'entities': |
|
122 |
attr = 'mtime' |
|
123 |
else: |
|
124 |
attr = 'dtime' |
|
125 |
return 'SELECT type, eid FROM %s WHERE %s AND %s > %%(time)s' % ( |
|
126 |
table, restr, attr) |
|
127 |
||
128 |
||
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
129 |
def sql_or_clauses(sql, clauses): |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
130 |
select, restr = sql.split(' WHERE ', 1) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
131 |
restrclauses = restr.split(' AND ') |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
132 |
for clause in clauses: |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
133 |
restrclauses.remove(clause) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
134 |
if restrclauses: |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
135 |
restr = '%s AND (%s)' % (' AND '.join(restrclauses), |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
136 |
' OR '.join(clauses)) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
137 |
else: |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
138 |
restr = '(%s)' % ' OR '.join(clauses) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
139 |
return '%s WHERE %s' % (select, restr) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
140 |
|
5891
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
141 |
def rdef_table_column(rdef): |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
142 |
"""return table and column used to store the given relation definition in |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
143 |
the database |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
144 |
""" |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
145 |
return (SQL_PREFIX + str(rdef.subject), |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
146 |
SQL_PREFIX + str(rdef.rtype)) |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
147 |
|
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
148 |
def rdef_physical_info(dbhelper, rdef): |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
149 |
"""return backend type and a boolean flag if NULL values should be allowed |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
150 |
for a given relation definition |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
151 |
""" |
6359
0bff5a05385c
[sync schema] take care rdef may not be final, in which case we want to use type of eid attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6297
diff
changeset
|
152 |
if rdef.object.final: |
0bff5a05385c
[sync schema] take care rdef may not be final, in which case we want to use type of eid attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6297
diff
changeset
|
153 |
ttype = rdef.object |
0bff5a05385c
[sync schema] take care rdef may not be final, in which case we want to use type of eid attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6297
diff
changeset
|
154 |
else: |
0bff5a05385c
[sync schema] take care rdef may not be final, in which case we want to use type of eid attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6297
diff
changeset
|
155 |
ttype = 'Int' # eid type |
0bff5a05385c
[sync schema] take care rdef may not be final, in which case we want to use type of eid attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6297
diff
changeset
|
156 |
coltype = y2sql.type_from_constraints(dbhelper, ttype, |
5891
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
157 |
rdef.constraints, creating=False) |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
158 |
allownull = rdef.cardinality[0] != '1' |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
159 |
return coltype, allownull |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
160 |
|
8265
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
161 |
|
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
162 |
class _UndoException(Exception): |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
163 |
"""something went wrong during undoing""" |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
164 |
|
7530
15178bf89fb6
[server] fix unicode conversion capability in UndoException
Julien Jehannet <julien.jehannet@logilab.fr>
parents:
7507
diff
changeset
|
165 |
def __unicode__(self): |
15178bf89fb6
[server] fix unicode conversion capability in UndoException
Julien Jehannet <julien.jehannet@logilab.fr>
parents:
7507
diff
changeset
|
166 |
"""Called by the unicode builtin; should return a Unicode object |
15178bf89fb6
[server] fix unicode conversion capability in UndoException
Julien Jehannet <julien.jehannet@logilab.fr>
parents:
7507
diff
changeset
|
167 |
|
8265
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
168 |
Type of _UndoException message must be `unicode` by design in CubicWeb. |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
169 |
""" |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
170 |
assert isinstance(self.args[0], unicode) |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
171 |
return self.args[0] |
7530
15178bf89fb6
[server] fix unicode conversion capability in UndoException
Julien Jehannet <julien.jehannet@logilab.fr>
parents:
7507
diff
changeset
|
172 |
|
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
173 |
|
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
174 |
def _undo_check_relation_target(tentity, rdef, role): |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
175 |
"""check linked entity has not been redirected for this relation""" |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
176 |
card = rdef.role_cardinality(role) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
177 |
if card in '?1' and tentity.related(rdef.rtype, role): |
8265
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
178 |
raise _UndoException(tentity._cw._( |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
179 |
"Can't restore %(role)s relation %(rtype)s to entity %(eid)s which " |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
180 |
"is already linked using this relation.") |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
181 |
% {'role': neg_role(role), |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
182 |
'rtype': rdef.rtype, |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
183 |
'eid': tentity.eid}) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
184 |
|
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
185 |
def _undo_rel_info(session, subj, rtype, obj): |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
186 |
entities = [] |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
187 |
for role, eid in (('subject', subj), ('object', obj)): |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
188 |
try: |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
189 |
entities.append(session.entity_from_eid(eid)) |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
190 |
except UnknownEid: |
8265
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
191 |
raise _UndoException(session._( |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
192 |
"Can't restore relation %(rtype)s, %(role)s entity %(eid)s" |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
193 |
" doesn't exist anymore.") |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
194 |
% {'role': session._(role), |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
195 |
'rtype': session._(rtype), |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
196 |
'eid': eid}) |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
197 |
sentity, oentity = entities |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
198 |
try: |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
199 |
rschema = session.vreg.schema.rschema(rtype) |
8900
010a59e12d89
use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
8821
diff
changeset
|
200 |
rdef = rschema.rdefs[(sentity.cw_etype, oentity.cw_etype)] |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
201 |
except KeyError: |
8265
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
202 |
raise _UndoException(session._( |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
203 |
"Can't restore relation %(rtype)s between %(subj)s and " |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
204 |
"%(obj)s, that relation does not exists anymore in the " |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
205 |
"schema.") |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
206 |
% {'rtype': session._(rtype), |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
207 |
'subj': subj, |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
208 |
'obj': obj}) |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
209 |
return sentity, oentity, rdef |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
210 |
|
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
211 |
def _undo_has_later_transaction(session, eid): |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
212 |
return session.system_sql('''\ |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
213 |
SELECT T.tx_uuid FROM transactions AS TREF, transactions AS T |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
214 |
WHERE TREF.tx_uuid='%(txuuid)s' AND T.tx_uuid!='%(txuuid)s' |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
215 |
AND T.tx_time>=TREF.tx_time |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
216 |
AND (EXISTS(SELECT 1 FROM tx_entity_actions AS TEA |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
217 |
WHERE TEA.tx_uuid=T.tx_uuid AND TEA.eid=%(eid)s) |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
218 |
OR EXISTS(SELECT 1 FROM tx_relation_actions as TRA |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
219 |
WHERE TRA.tx_uuid=T.tx_uuid AND ( |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
220 |
TRA.eid_from=%(eid)s OR TRA.eid_to=%(eid)s)) |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
221 |
)''' % {'txuuid': session.transaction_data['undoing_uuid'], |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
222 |
'eid': eid}).fetchone() |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
223 |
|
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
224 |
|
0 | 225 |
class NativeSQLSource(SQLAdapterMixIn, AbstractSource): |
226 |
"""adapter for source using the native cubicweb schema (see below) |
|
227 |
""" |
|
2354
9b4bac626977
ability to map attributes to something else than usual cw mapping on sql generation
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2310
diff
changeset
|
228 |
sqlgen_class = SQLGenerator |
0 | 229 |
options = ( |
230 |
('db-driver', |
|
231 |
{'type' : 'string', |
|
232 |
'default': 'postgres', |
|
5423
e15abfdcce38
backport default into stable: stable is now cw 3.8
Sylvain Thénault <sylvain.thenault@logilab.fr>
diff
changeset
|
233 |
# XXX use choice type |
e15abfdcce38
backport default into stable: stable is now cw 3.8
Sylvain Thénault <sylvain.thenault@logilab.fr>
diff
changeset
|
234 |
'help': 'database driver (postgres, mysql, sqlite, sqlserver2005)', |
6201
cf445fcaf8fe
[c-c create] ask for database driver by default
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6128
diff
changeset
|
235 |
'group': 'native-source', 'level': 0, |
0 | 236 |
}), |
237 |
('db-host', |
|
238 |
{'type' : 'string', |
|
239 |
'default': '', |
|
240 |
'help': 'database host', |
|
5323
329b4f6d18b4
[config] with lgc >= 0.50, option's dict inputlevel becomes level
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5217
diff
changeset
|
241 |
'group': 'native-source', 'level': 1, |
0 | 242 |
}), |
2566
714a8743d423
missing db-port option to source's option definitions
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2354
diff
changeset
|
243 |
('db-port', |
714a8743d423
missing db-port option to source's option definitions
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2354
diff
changeset
|
244 |
{'type' : 'string', |
714a8743d423
missing db-port option to source's option definitions
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2354
diff
changeset
|
245 |
'default': '', |
714a8743d423
missing db-port option to source's option definitions
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2354
diff
changeset
|
246 |
'help': 'database port', |
5323
329b4f6d18b4
[config] with lgc >= 0.50, option's dict inputlevel becomes level
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5217
diff
changeset
|
247 |
'group': 'native-source', 'level': 1, |
2566
714a8743d423
missing db-port option to source's option definitions
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2354
diff
changeset
|
248 |
}), |
0 | 249 |
('db-name', |
250 |
{'type' : 'string', |
|
3835
a191b3b9e455
more sensible default values to c-c "create" inputs
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3689
diff
changeset
|
251 |
'default': Method('default_instance_id'), |
0 | 252 |
'help': 'database name', |
5323
329b4f6d18b4
[config] with lgc >= 0.50, option's dict inputlevel becomes level
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5217
diff
changeset
|
253 |
'group': 'native-source', 'level': 0, |
0 | 254 |
}), |
255 |
('db-user', |
|
256 |
{'type' : 'string', |
|
3835
a191b3b9e455
more sensible default values to c-c "create" inputs
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3689
diff
changeset
|
257 |
'default': CubicWebNoAppConfiguration.mode == 'user' and getlogin() or 'cubicweb', |
0 | 258 |
'help': 'database user', |
5323
329b4f6d18b4
[config] with lgc >= 0.50, option's dict inputlevel becomes level
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5217
diff
changeset
|
259 |
'group': 'native-source', 'level': 0, |
0 | 260 |
}), |
261 |
('db-password', |
|
262 |
{'type' : 'password', |
|
263 |
'default': '', |
|
264 |
'help': 'database password', |
|
5323
329b4f6d18b4
[config] with lgc >= 0.50, option's dict inputlevel becomes level
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5217
diff
changeset
|
265 |
'group': 'native-source', 'level': 0, |
0 | 266 |
}), |
267 |
('db-encoding', |
|
268 |
{'type' : 'string', |
|
269 |
'default': 'utf8', |
|
270 |
'help': 'database encoding', |
|
5323
329b4f6d18b4
[config] with lgc >= 0.50, option's dict inputlevel becomes level
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5217
diff
changeset
|
271 |
'group': 'native-source', 'level': 1, |
0 | 272 |
}), |
5413
dc896e698ab0
added db-extra-arguments to configuration parameters for native source
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5410
diff
changeset
|
273 |
('db-extra-arguments', |
dc896e698ab0
added db-extra-arguments to configuration parameters for native source
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5410
diff
changeset
|
274 |
{'type' : 'string', |
dc896e698ab0
added db-extra-arguments to configuration parameters for native source
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5410
diff
changeset
|
275 |
'default': '', |
dc896e698ab0
added db-extra-arguments to configuration parameters for native source
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5410
diff
changeset
|
276 |
'help': 'set to "Trusted_Connection" if you are using SQLServer and ' |
dc896e698ab0
added db-extra-arguments to configuration parameters for native source
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5410
diff
changeset
|
277 |
'want trusted authentication for the database connection', |
5515
513af9be9e37
[config] rename remaining inputlevel to level in option definitions, as expected by lgc >= 0.50
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
278 |
'group': 'native-source', 'level': 2, |
5413
dc896e698ab0
added db-extra-arguments to configuration parameters for native source
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5410
diff
changeset
|
279 |
}), |
0 | 280 |
) |
1792 | 281 |
|
6427
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6426
diff
changeset
|
282 |
def __init__(self, repo, source_config, *args, **kwargs): |
0 | 283 |
SQLAdapterMixIn.__init__(self, source_config) |
3647
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
284 |
self.authentifiers = [LoginPasswordAuthentifier(self)] |
6849
5a0c2cfc19bf
[repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6724
diff
changeset
|
285 |
if repo.config['allow-email-login']: |
5a0c2cfc19bf
[repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6724
diff
changeset
|
286 |
self.authentifiers.insert(0, EmailPasswordAuthentifier(self)) |
6427
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6426
diff
changeset
|
287 |
AbstractSource.__init__(self, repo, source_config, *args, **kwargs) |
0 | 288 |
# sql generator |
6427
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6426
diff
changeset
|
289 |
self._rql_sqlgen = self.sqlgen_class(self.schema, self.dbhelper, |
4831
c5aec27c1bf7
[repo] use logilab.db instead of lgc.adbh/lgc.db/lgc.sqlgen/indexer, test new date extranction functions
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4829
diff
changeset
|
290 |
ATTR_MAP.copy()) |
0 | 291 |
# full text index helper |
4806
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
292 |
self.do_fti = not repo.config['delay-full-text-indexation'] |
0 | 293 |
# sql queries cache |
7954
a3d3220669d6
[cache] replace lgc.cache with something more appropriate (closes #1921713)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
7922
diff
changeset
|
294 |
self._cache = QueryCache(repo.config['rql-cache-size']) |
0 | 295 |
self._temp_table_data = {} |
5061 | 296 |
# we need a lock to protect eid attribution function (XXX, really? |
297 |
# explain) |
|
7202
5b80f5ee61b5
[native source] eid_creation_lock actually lock the eid connection creation and usage
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7190
diff
changeset
|
298 |
self._eid_cnx_lock = Lock() |
5690
6de8437b06bd
[source] don't create the eid_creation connection at source initiliaztion time
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
5649
diff
changeset
|
299 |
self._eid_creation_cnx = None |
4964
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
300 |
# (etype, attr) / storage mapping |
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
301 |
self._storages = {} |
5059
1d5c81588144
[repo] make etype which should go in deleted_entities configurable: we only need this for types imported from other multi-sources instances
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5013
diff
changeset
|
302 |
# entity types that may be used by other multi-sources instances |
1d5c81588144
[repo] make etype which should go in deleted_entities configurable: we only need this for types imported from other multi-sources instances
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5013
diff
changeset
|
303 |
self.multisources_etypes = set(repo.config['multi-sources-etypes']) |
2072
8008e8812d76
deactivate sqlite connection wrapping for unittest_multisources for now
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2066
diff
changeset
|
304 |
# XXX no_sqlite_wrap trick since we've a sqlite locking pb when |
8008e8812d76
deactivate sqlite connection wrapping for unittest_multisources for now
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2066
diff
changeset
|
305 |
# running unittest_multisources with the wrapping below |
8008e8812d76
deactivate sqlite connection wrapping for unittest_multisources for now
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2066
diff
changeset
|
306 |
if self.dbdriver == 'sqlite' and \ |
8008e8812d76
deactivate sqlite connection wrapping for unittest_multisources for now
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2066
diff
changeset
|
307 |
not getattr(repo.config, 'no_sqlite_wrap', False): |
2064
a5cd3a92314a
properly call [re]set_pool, fix connection handling so we have a change to get cw running on top of a sqlite dabase with threads activated
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2056
diff
changeset
|
308 |
from cubicweb.server.sources.extlite import ConnectionWrapper |
5542
a8ad3df5a8a3
[test] absolute sqlite database names in source initialization, avoid pb when cwd is changed later
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5515
diff
changeset
|
309 |
self.dbhelper.dbname = abspath(self.dbhelper.dbname) |
2064
a5cd3a92314a
properly call [re]set_pool, fix connection handling so we have a change to get cw running on top of a sqlite dabase with threads activated
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2056
diff
changeset
|
310 |
self.get_connection = lambda: ConnectionWrapper(self) |
a5cd3a92314a
properly call [re]set_pool, fix connection handling so we have a change to get cw running on top of a sqlite dabase with threads activated
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2056
diff
changeset
|
311 |
self.check_connection = lambda cnx: cnx |
7398
26695dd703d8
[repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
312 |
def cnxset_freed(cnx): |
2620
de68f84b8f54
R [sql source] cnx._cnx checked in cnx.close, don't do it here
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2619
diff
changeset
|
313 |
cnx.close() |
7398
26695dd703d8
[repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
314 |
self.cnxset_freed = cnxset_freed |
5649
a07dee204187
fix unit tests by not using the new create_eid implementation with sqlite
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5645
diff
changeset
|
315 |
if self.dbdriver == 'sqlite': |
a07dee204187
fix unit tests by not using the new create_eid implementation with sqlite
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5645
diff
changeset
|
316 |
self._create_eid = None |
a07dee204187
fix unit tests by not using the new create_eid implementation with sqlite
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5645
diff
changeset
|
317 |
self.create_eid = self._create_eid_sqlite |
5847
51636c991fb4
[migration] refactor schema migration fix introduced by 5833:d7256ae7c1d1
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5824
diff
changeset
|
318 |
self.binary_to_str = self.dbhelper.dbapi_module.binary_to_str |
51636c991fb4
[migration] refactor schema migration fix introduced by 5833:d7256ae7c1d1
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5824
diff
changeset
|
319 |
|
2064
a5cd3a92314a
properly call [re]set_pool, fix connection handling so we have a change to get cw running on top of a sqlite dabase with threads activated
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2056
diff
changeset
|
320 |
|
a5cd3a92314a
properly call [re]set_pool, fix connection handling so we have a change to get cw running on top of a sqlite dabase with threads activated
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2056
diff
changeset
|
321 |
@property |
a5cd3a92314a
properly call [re]set_pool, fix connection handling so we have a change to get cw running on top of a sqlite dabase with threads activated
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2056
diff
changeset
|
322 |
def _sqlcnx(self): |
a5cd3a92314a
properly call [re]set_pool, fix connection handling so we have a change to get cw running on top of a sqlite dabase with threads activated
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2056
diff
changeset
|
323 |
# XXX: sqlite connections can only be used in the same thread, so |
a5cd3a92314a
properly call [re]set_pool, fix connection handling so we have a change to get cw running on top of a sqlite dabase with threads activated
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2056
diff
changeset
|
324 |
# create a new one each time necessary. If it appears to be time |
a5cd3a92314a
properly call [re]set_pool, fix connection handling so we have a change to get cw running on top of a sqlite dabase with threads activated
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2056
diff
changeset
|
325 |
# consuming, find another way |
a5cd3a92314a
properly call [re]set_pool, fix connection handling so we have a change to get cw running on top of a sqlite dabase with threads activated
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2056
diff
changeset
|
326 |
return SQLAdapterMixIn.get_connection(self) |
0 | 327 |
|
6945
28bf94d062a9
[sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6931
diff
changeset
|
328 |
def check_config(self, source_entity): |
28bf94d062a9
[sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6931
diff
changeset
|
329 |
"""check configuration of source entity""" |
28bf94d062a9
[sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6931
diff
changeset
|
330 |
if source_entity.host_config: |
28bf94d062a9
[sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6931
diff
changeset
|
331 |
msg = source_entity._cw._('the system source has its configuration ' |
28bf94d062a9
[sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6931
diff
changeset
|
332 |
'stored on the file-system') |
28bf94d062a9
[sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6931
diff
changeset
|
333 |
raise ValidationError(source_entity.eid, {role_name('config', 'subject'): msg}) |
28bf94d062a9
[sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6931
diff
changeset
|
334 |
|
3647
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
335 |
def add_authentifier(self, authentifier): |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
336 |
self.authentifiers.append(authentifier) |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
337 |
authentifier.source = self |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
338 |
authentifier.set_schema(self.schema) |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
339 |
|
0 | 340 |
def reset_caches(self): |
341 |
"""method called during test to reset potential source caches""" |
|
7954
a3d3220669d6
[cache] replace lgc.cache with something more appropriate (closes #1921713)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
7922
diff
changeset
|
342 |
self._cache = QueryCache(self.repo.config['rql-cache-size']) |
1792 | 343 |
|
0 | 344 |
def clear_eid_cache(self, eid, etype): |
345 |
"""clear potential caches for the given eid""" |
|
2610
2933cc6bf9ad
[F native source] fix attempts to clear cache key
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2593
diff
changeset
|
346 |
self._cache.pop('Any X WHERE X eid %s, X is %s' % (eid, etype), None) |
0 | 347 |
self._cache.pop('Any X WHERE X eid %s' % eid, None) |
2610
2933cc6bf9ad
[F native source] fix attempts to clear cache key
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2593
diff
changeset
|
348 |
self._cache.pop('Any %s' % eid, None) |
1792 | 349 |
|
0 | 350 |
def sqlexec(self, session, sql, args=None): |
351 |
"""execute the query and return its result""" |
|
2306
95da5d9f0870
give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2072
diff
changeset
|
352 |
return self.process_result(self.doexec(session, sql, args)) |
1792 | 353 |
|
7398
26695dd703d8
[repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
354 |
def init_creating(self, cnxset=None): |
0 | 355 |
# check full text index availibility |
4806
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
356 |
if self.do_fti: |
7398
26695dd703d8
[repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
357 |
if cnxset is None: |
26695dd703d8
[repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
358 |
_cnxset = self.repo._get_cnxset() |
26695dd703d8
[repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
359 |
_cnxset.cnxset_set() |
6724
24bf6f181d0e
[pyro source] store pyro source mapping file into the database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6427
diff
changeset
|
360 |
else: |
7398
26695dd703d8
[repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
361 |
_cnxset = cnxset |
26695dd703d8
[repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
362 |
if not self.dbhelper.has_fti_table(_cnxset['system']): |
4825
cdd979ae1b57
fix name error fixed in wrong branch
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4818
diff
changeset
|
363 |
if not self.repo.config.creating: |
4810
3055dc7a7c88
don't issue critical warning about missing text index when we're creating the instance
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4807
diff
changeset
|
364 |
self.critical('no text index table') |
4806
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
365 |
self.do_fti = False |
7398
26695dd703d8
[repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
366 |
if cnxset is None: |
26695dd703d8
[repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
367 |
_cnxset.cnxset_freed() |
26695dd703d8
[repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
368 |
self.repo._free_cnxset(_cnxset) |
0 | 369 |
|
7342
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
370 |
def backup(self, backupfile, confirm, format='native'): |
2759
23d7a75693f8
R refactor backup and use tar.gz to store all sources
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents:
2638
diff
changeset
|
371 |
"""method called to create a backup of the source's data""" |
7342
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
372 |
if format == 'portable': |
8947
3bbd416b09ec
[repo] straightforward bootstrap sequence. Closes #2841188
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
8916
diff
changeset
|
373 |
# ensure the schema is the one stored in the database: if repository |
3bbd416b09ec
[repo] straightforward bootstrap sequence. Closes #2841188
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
8916
diff
changeset
|
374 |
# started in quick_start mode, the file system's one has been loaded |
3bbd416b09ec
[repo] straightforward bootstrap sequence. Closes #2841188
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
8916
diff
changeset
|
375 |
# so force reload |
3bbd416b09ec
[repo] straightforward bootstrap sequence. Closes #2841188
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
8916
diff
changeset
|
376 |
if self.repo.config.quick_start: |
3bbd416b09ec
[repo] straightforward bootstrap sequence. Closes #2841188
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
8916
diff
changeset
|
377 |
self.repo.set_schema(self.repo.deserialize_schema(), |
3bbd416b09ec
[repo] straightforward bootstrap sequence. Closes #2841188
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
8916
diff
changeset
|
378 |
resetvreg=False) |
7342
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
379 |
helper = DatabaseIndependentBackupRestore(self) |
7398
26695dd703d8
[repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
380 |
self.close_source_connections() |
7342
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
381 |
try: |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
382 |
helper.backup(backupfile) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
383 |
finally: |
7398
26695dd703d8
[repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
384 |
self.open_source_connections() |
7342
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
385 |
elif format == 'native': |
7398
26695dd703d8
[repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
386 |
self.close_source_connections() |
7342
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
387 |
try: |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
388 |
self.backup_to_file(backupfile, confirm) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
389 |
finally: |
7398
26695dd703d8
[repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
390 |
self.open_source_connections() |
7342
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
391 |
else: |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
392 |
raise ValueError('Unknown format %r' % format) |
2493
9806571ea790
major refactoring of database dump/restore:
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2476
diff
changeset
|
393 |
|
7342
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
394 |
|
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
395 |
def restore(self, backupfile, confirm, drop, format='native'): |
2493
9806571ea790
major refactoring of database dump/restore:
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2476
diff
changeset
|
396 |
"""method called to restore a backup of source's data""" |
7398
26695dd703d8
[repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
397 |
if self.repo.config.init_cnxset_pool: |
26695dd703d8
[repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
398 |
self.close_source_connections() |
2759
23d7a75693f8
R refactor backup and use tar.gz to store all sources
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents:
2638
diff
changeset
|
399 |
try: |
7342
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
400 |
if format == 'portable': |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
401 |
helper = DatabaseIndependentBackupRestore(self) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
402 |
helper.restore(backupfile) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
403 |
elif format == 'native': |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
404 |
self.restore_from_file(backupfile, confirm, drop=drop) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
405 |
else: |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
406 |
raise ValueError('Unknown format %r' % format) |
2759
23d7a75693f8
R refactor backup and use tar.gz to store all sources
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents:
2638
diff
changeset
|
407 |
finally: |
7398
26695dd703d8
[repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
408 |
if self.repo.config.init_cnxset_pool: |
26695dd703d8
[repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
409 |
self.open_source_connections() |
2493
9806571ea790
major refactoring of database dump/restore:
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2476
diff
changeset
|
410 |
|
7342
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
411 |
|
6957
ffda12be2e9f
[repository] #1460066: backport datafeed cube as cubicweb source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6945
diff
changeset
|
412 |
def init(self, activated, source_entity): |
8707
28cbd267e96b
[sources] fix classes that inherit from AbstractSource (closes #2718669)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents:
8695
diff
changeset
|
413 |
super(NativeSQLSource, self).init(activated, source_entity) |
7398
26695dd703d8
[repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
414 |
self.init_creating(source_entity._cw.cnxset) |
7543
570522300e22
[ms, entity metas] add 'actual source' to entities table / base entity metadata cache. Closes #1767090
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7536
diff
changeset
|
415 |
try: |
7786
18a366267612
[repo] use smarter query to check asource presence on startup (closes #1922099)
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
7754
diff
changeset
|
416 |
# test if 'asource' column exists |
7839
daf46963f4fe
ensure the test for asource column won't crash on SQLServer (closes #1949621)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7787
diff
changeset
|
417 |
query = self.dbhelper.sql_add_limit_offset('SELECT asource FROM entities', 1) |
daf46963f4fe
ensure the test for asource column won't crash on SQLServer (closes #1949621)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7787
diff
changeset
|
418 |
source_entity._cw.system_sql(query) |
8695
358d8bed9626
[toward-py3k] rewrite to "except AnException as exc:" (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents:
8694
diff
changeset
|
419 |
except Exception as ex: |
7543
570522300e22
[ms, entity metas] add 'actual source' to entities table / base entity metadata cache. Closes #1767090
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7536
diff
changeset
|
420 |
self.eid_type_source = self.eid_type_source_pre_131 |
1792 | 421 |
|
5749
b4393b681f7a
[repo] on repository shutdown, we've to close the new eid creation connection
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5729
diff
changeset
|
422 |
def shutdown(self): |
b4393b681f7a
[repo] on repository shutdown, we've to close the new eid creation connection
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5729
diff
changeset
|
423 |
if self._eid_creation_cnx: |
b4393b681f7a
[repo] on repository shutdown, we've to close the new eid creation connection
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5729
diff
changeset
|
424 |
self._eid_creation_cnx.close() |
5751
1e5ef464cade
[test]Â properly close connections during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5749
diff
changeset
|
425 |
self._eid_creation_cnx = None |
5749
b4393b681f7a
[repo] on repository shutdown, we've to close the new eid creation connection
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5729
diff
changeset
|
426 |
|
5013
ad91f93bbb93
[source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5011
diff
changeset
|
427 |
# XXX deprecates [un]map_attribute ? |
ad91f93bbb93
[source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5011
diff
changeset
|
428 |
def map_attribute(self, etype, attr, cb, sourcedb=True): |
ad91f93bbb93
[source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5011
diff
changeset
|
429 |
self._rql_sqlgen.attr_map['%s.%s' % (etype, attr)] = (cb, sourcedb) |
2493
9806571ea790
major refactoring of database dump/restore:
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2476
diff
changeset
|
430 |
|
4512
e7ac20bf3629
unset_attribute_storage, for testing purpose at least
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4252
diff
changeset
|
431 |
def unmap_attribute(self, etype, attr): |
e7ac20bf3629
unset_attribute_storage, for testing purpose at least
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4252
diff
changeset
|
432 |
self._rql_sqlgen.attr_map.pop('%s.%s' % (etype, attr), None) |
e7ac20bf3629
unset_attribute_storage, for testing purpose at least
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4252
diff
changeset
|
433 |
|
4964
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
434 |
def set_storage(self, etype, attr, storage): |
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
435 |
storage_dict = self._storages.setdefault(etype, {}) |
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
436 |
storage_dict[attr] = storage |
5013
ad91f93bbb93
[source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5011
diff
changeset
|
437 |
self.map_attribute(etype, attr, |
ad91f93bbb93
[source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5011
diff
changeset
|
438 |
storage.callback, storage.is_source_callback) |
4964
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
439 |
|
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
440 |
def unset_storage(self, etype, attr): |
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
441 |
self._storages[etype].pop(attr) |
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
442 |
# if etype has no storage left, remove the entry |
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
443 |
if not self._storages[etype]: |
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
444 |
del self._storages[etype] |
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
445 |
self.unmap_attribute(etype, attr) |
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
446 |
|
5397
cdbf823450aa
[bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5323
diff
changeset
|
447 |
def storage(self, etype, attr): |
cdbf823450aa
[bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5323
diff
changeset
|
448 |
"""return the storage for the given entity type / attribute |
cdbf823450aa
[bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5323
diff
changeset
|
449 |
""" |
cdbf823450aa
[bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5323
diff
changeset
|
450 |
try: |
cdbf823450aa
[bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5323
diff
changeset
|
451 |
return self._storages[etype][attr] |
cdbf823450aa
[bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5323
diff
changeset
|
452 |
except KeyError: |
cdbf823450aa
[bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5323
diff
changeset
|
453 |
raise Exception('no custom storage set for %s.%s' % (etype, attr)) |
cdbf823450aa
[bfss] new storage_changed migration action to move an attribute to a custom storage. Closes #893941
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5323
diff
changeset
|
454 |
|
0 | 455 |
# ISource interface ####################################################### |
456 |
||
3647
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
457 |
def compile_rql(self, rql, sols): |
3240
8604a15995d1
refactor so that rql rewriter may be used outside the server. Enhance it to be usable for RRQLExpression as well
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3041
diff
changeset
|
458 |
rqlst = self.repo.vreg.rqlhelper.parse(rql) |
0 | 459 |
rqlst.restricted_vars = () |
3647
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
460 |
rqlst.children[0].solutions = sols |
0 | 461 |
self.repo.querier.sqlgen_annotate(rqlst) |
438 | 462 |
set_qdata(self.schema.rschema, rqlst, ()) |
0 | 463 |
return rqlst |
1792 | 464 |
|
0 | 465 |
def set_schema(self, schema): |
2476
1294a6bdf3bf
application -> instance where it makes sense
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2354
diff
changeset
|
466 |
"""set the instance'schema""" |
7954
a3d3220669d6
[cache] replace lgc.cache with something more appropriate (closes #1921713)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
7922
diff
changeset
|
467 |
self._cache = QueryCache(self.repo.config['rql-cache-size']) |
0 | 468 |
self.cache_hit, self.cache_miss, self.no_cache = 0, 0, 0 |
469 |
self.schema = schema |
|
470 |
try: |
|
471 |
self._rql_sqlgen.schema = schema |
|
472 |
except AttributeError: |
|
473 |
pass # __init__ |
|
3647
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
474 |
for authentifier in self.authentifiers: |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
475 |
authentifier.set_schema(self.schema) |
4806
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
476 |
clear_cache(self, 'need_fti_indexation') |
1792 | 477 |
|
0 | 478 |
def support_entity(self, etype, write=False): |
479 |
"""return true if the given entity's type is handled by this adapter |
|
480 |
if write is true, return true only if it's a RW support |
|
481 |
""" |
|
482 |
return not etype in NONSYSTEM_ETYPES |
|
1792 | 483 |
|
0 | 484 |
def support_relation(self, rtype, write=False): |
485 |
"""return true if the given relation's type is handled by this adapter |
|
486 |
if write is true, return true only if it's a RW support |
|
487 |
""" |
|
488 |
if write: |
|
489 |
return not rtype in NONSYSTEM_RELATIONS |
|
490 |
# due to current multi-sources implementation, the system source |
|
1792 | 491 |
# can't claim not supporting a relation |
0 | 492 |
return True #not rtype == 'content_for' |
493 |
||
3041
782fa7566a22
[multi-sources] new may_cross_relation method on sources
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2967
diff
changeset
|
494 |
def may_cross_relation(self, rtype): |
782fa7566a22
[multi-sources] new may_cross_relation method on sources
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2967
diff
changeset
|
495 |
return True |
782fa7566a22
[multi-sources] new may_cross_relation method on sources
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2967
diff
changeset
|
496 |
|
3647
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
497 |
def authenticate(self, session, login, **kwargs): |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
498 |
"""return CWUser eid for the given login and other authentication |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
499 |
information found in kwargs, else raise `AuthenticationError` |
0 | 500 |
""" |
3647
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
501 |
for authentifier in self.authentifiers: |
0 | 502 |
try: |
3647
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
503 |
return authentifier.authenticate(session, login, **kwargs) |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
504 |
except AuthenticationError: |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
505 |
continue |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
506 |
raise AuthenticationError() |
1792 | 507 |
|
508 |
def syntax_tree_search(self, session, union, args=None, cachekey=None, |
|
0 | 509 |
varmap=None): |
510 |
"""return result from this source for a rql query (actually from |
|
511 |
a rql syntax tree and a solution dictionary mapping each used |
|
512 |
variable to a possible type). If cachekey is given, the query |
|
513 |
necessary to fetch the results (but not the results themselves) |
|
514 |
may be cached using this key. |
|
515 |
""" |
|
2625
d6012db7b93e
R [server debug]Â more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2620
diff
changeset
|
516 |
assert dbg_st_search(self.uri, union, varmap, args, cachekey) |
0 | 517 |
# remember number of actually selected term (sql generation may append some) |
518 |
if cachekey is None: |
|
519 |
self.no_cache += 1 |
|
520 |
# generate sql query if we are able to do so (not supported types...) |
|
5013
ad91f93bbb93
[source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5011
diff
changeset
|
521 |
sql, qargs, cbs = self._rql_sqlgen.generate(union, args, varmap) |
0 | 522 |
else: |
523 |
# sql may be cached |
|
524 |
try: |
|
5013
ad91f93bbb93
[source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5011
diff
changeset
|
525 |
sql, qargs, cbs = self._cache[cachekey] |
0 | 526 |
self.cache_hit += 1 |
527 |
except KeyError: |
|
528 |
self.cache_miss += 1 |
|
5013
ad91f93bbb93
[source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5011
diff
changeset
|
529 |
sql, qargs, cbs = self._rql_sqlgen.generate(union, args, varmap) |
ad91f93bbb93
[source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5011
diff
changeset
|
530 |
self._cache[cachekey] = sql, qargs, cbs |
ad91f93bbb93
[source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5011
diff
changeset
|
531 |
args = self.merge_args(args, qargs) |
0 | 532 |
assert isinstance(sql, basestring), repr(sql) |
533 |
try: |
|
2306
95da5d9f0870
give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2072
diff
changeset
|
534 |
cursor = self.doexec(session, sql, args) |
4831
c5aec27c1bf7
[repo] use logilab.db instead of lgc.adbh/lgc.db/lgc.sqlgen/indexer, test new date extranction functions
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4829
diff
changeset
|
535 |
except (self.OperationalError, self.InterfaceError): |
5975
5120d97e2f7e
[transaction] do not attempt to reconnect if there has been some write during the transaction
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5847
diff
changeset
|
536 |
if session.mode == 'write': |
5120d97e2f7e
[transaction] do not attempt to reconnect if there has been some write during the transaction
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5847
diff
changeset
|
537 |
# do not attempt to reconnect if there has been some write |
5120d97e2f7e
[transaction] do not attempt to reconnect if there has been some write during the transaction
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5847
diff
changeset
|
538 |
# during the transaction |
5120d97e2f7e
[transaction] do not attempt to reconnect if there has been some write during the transaction
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5847
diff
changeset
|
539 |
raise |
0 | 540 |
# FIXME: better detection of deconnection pb |
5107
3694bd379513
[source] log attempt to reconnect using warning level. Also, don't relog failed sql (already logged)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5098
diff
changeset
|
541 |
self.warning("trying to reconnect") |
7398
26695dd703d8
[repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
542 |
session.cnxset.reconnect(self) |
2306
95da5d9f0870
give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2072
diff
changeset
|
543 |
cursor = self.doexec(session, sql, args) |
8695
358d8bed9626
[toward-py3k] rewrite to "except AnException as exc:" (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents:
8694
diff
changeset
|
544 |
except self.DbapiError as exc: |
5605
2604545d7dd9
[win32 SQLServer] connection lost detection
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5515
diff
changeset
|
545 |
# We get this one with pyodbc and SQL Server when connection was reset |
5975
5120d97e2f7e
[transaction] do not attempt to reconnect if there has been some write during the transaction
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5847
diff
changeset
|
546 |
if exc.args[0] == '08S01' and session.mode != 'write': |
5605
2604545d7dd9
[win32 SQLServer] connection lost detection
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5515
diff
changeset
|
547 |
self.warning("trying to reconnect") |
7398
26695dd703d8
[repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
548 |
session.cnxset.reconnect(self) |
5605
2604545d7dd9
[win32 SQLServer] connection lost detection
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5515
diff
changeset
|
549 |
cursor = self.doexec(session, sql, args) |
2604545d7dd9
[win32 SQLServer] connection lost detection
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5515
diff
changeset
|
550 |
else: |
2604545d7dd9
[win32 SQLServer] connection lost detection
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5515
diff
changeset
|
551 |
raise |
5625
6ee2a7b6f194
[external storage] refactor to give session to storage's callback (needed by vcsfile storage)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5606
diff
changeset
|
552 |
results = self.process_result(cursor, cbs, session=session) |
2625
d6012db7b93e
R [server debug]Â more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2620
diff
changeset
|
553 |
assert dbg_results(results) |
d6012db7b93e
R [server debug]Â more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2620
diff
changeset
|
554 |
return results |
1792 | 555 |
|
0 | 556 |
def flying_insert(self, table, session, union, args=None, varmap=None): |
557 |
"""similar as .syntax_tree_search, but inserts data in the |
|
558 |
temporary table (on-the-fly if possible, eg for the system |
|
559 |
source whose the given cursor come from). If not possible, |
|
560 |
inserts all data by calling .executemany(). |
|
561 |
""" |
|
2625
d6012db7b93e
R [server debug]Â more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2620
diff
changeset
|
562 |
assert dbg_st_search( |
2638 | 563 |
self.uri, union, varmap, args, |
2625
d6012db7b93e
R [server debug]Â more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2620
diff
changeset
|
564 |
prefix='ON THE FLY temp data insertion into %s from' % table) |
d6012db7b93e
R [server debug]Â more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2620
diff
changeset
|
565 |
# generate sql queries if we are able to do so |
5013
ad91f93bbb93
[source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5011
diff
changeset
|
566 |
sql, qargs, cbs = self._rql_sqlgen.generate(union, args, varmap) |
4831
c5aec27c1bf7
[repo] use logilab.db instead of lgc.adbh/lgc.db/lgc.sqlgen/indexer, test new date extranction functions
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4829
diff
changeset
|
567 |
query = 'INSERT INTO %s %s' % (table, sql.encode(self._dbencoding)) |
5013
ad91f93bbb93
[source storage] refactor source sql generation and results handling to allow repository side callbacks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5011
diff
changeset
|
568 |
self.doexec(session, query, self.merge_args(args, qargs)) |
1792 | 569 |
|
2627
d710278e0c1c
manual_insert is a public method
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2625
diff
changeset
|
570 |
def manual_insert(self, results, table, session): |
0 | 571 |
"""insert given result into a temporary table on the system source""" |
2625
d6012db7b93e
R [server debug]Â more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2620
diff
changeset
|
572 |
if server.DEBUG & server.DBG_RQL: |
6128 | 573 |
print ' manual insertion of', len(results), 'results into', table |
0 | 574 |
if not results: |
575 |
return |
|
576 |
query_args = ['%%(%s)s' % i for i in xrange(len(results[0]))] |
|
577 |
query = 'INSERT INTO %s VALUES(%s)' % (table, ','.join(query_args)) |
|
578 |
kwargs_list = [] |
|
579 |
for row in results: |
|
580 |
kwargs = {} |
|
581 |
row = tuple(row) |
|
582 |
for index, cell in enumerate(row): |
|
2066
2c4bf4ee88a2
cleanup, stop encoding unicode string in manual_insert, no more necessary and make crash recent sqlite w/ 8bit string
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2064
diff
changeset
|
583 |
if isinstance(cell, Binary): |
4831
c5aec27c1bf7
[repo] use logilab.db instead of lgc.adbh/lgc.db/lgc.sqlgen/indexer, test new date extranction functions
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4829
diff
changeset
|
584 |
cell = self._binary(cell.getvalue()) |
0 | 585 |
kwargs[str(index)] = cell |
586 |
kwargs_list.append(kwargs) |
|
2306
95da5d9f0870
give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2072
diff
changeset
|
587 |
self.doexecmany(session, query, kwargs_list) |
0 | 588 |
|
589 |
def clean_temp_data(self, session, temptables): |
|
590 |
"""remove temporary data, usually associated to temporary tables""" |
|
591 |
if temptables: |
|
592 |
for table in temptables: |
|
593 |
try: |
|
2306
95da5d9f0870
give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2072
diff
changeset
|
594 |
self.doexec(session,'DROP TABLE %s' % table) |
7645
cb2459aaa7d7
[source] closes #1816412: try to reconnect to dbms if connection is reset in eid_type_source()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
7562
diff
changeset
|
595 |
except Exception: |
0 | 596 |
pass |
597 |
try: |
|
598 |
del self._temp_table_data[table] |
|
599 |
except KeyError: |
|
600 |
continue |
|
1792 | 601 |
|
4964
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
602 |
@contextmanager |
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
603 |
def _storage_handler(self, entity, event): |
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
604 |
# 1/ memorize values as they are before the storage is called. |
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
605 |
# For instance, the BFSStorage will replace the `data` |
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
606 |
# binary value with a Binary containing the destination path |
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
607 |
# on the filesystem. To make the entity.data usage absolutely |
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
608 |
# transparent, we'll have to reset entity.data to its binary |
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
609 |
# value once the SQL query will be executed |
6889
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
610 |
restore_values = [] |
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
611 |
if isinstance(entity, list): |
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
612 |
entities = entity |
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
613 |
else: |
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
614 |
entities = [entity] |
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
615 |
etype = entities[0].__regid__ |
4964
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
616 |
for attr, storage in self._storages.get(etype, {}).items(): |
6889
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
617 |
for entity in entities: |
8501
b922dd08eb79
[storage] relies on event=='deleted' to detect "delete" event (closes #2450680)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
8379
diff
changeset
|
618 |
if event == 'deleted': |
b922dd08eb79
[storage] relies on event=='deleted' to detect "delete" event (closes #2450680)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
8379
diff
changeset
|
619 |
storage.entity_deleted(entity, attr) |
b922dd08eb79
[storage] relies on event=='deleted' to detect "delete" event (closes #2450680)
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
8379
diff
changeset
|
620 |
else: |
6889
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
621 |
edited = entity.cw_edited |
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
622 |
if attr in edited: |
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
623 |
handler = getattr(storage, 'entity_%s' % event) |
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
624 |
to_restore = handler(entity, attr) |
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
625 |
restore_values.append((entity, attr, to_restore)) |
5128
e5d300d75519
[python] take care to this detail of @contextmanager: if an unhandled exception occurs in the block, it is reraised inside the generator at the point where the yield occurred
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5107
diff
changeset
|
626 |
try: |
e5d300d75519
[python] take care to this detail of @contextmanager: if an unhandled exception occurs in the block, it is reraised inside the generator at the point where the yield occurred
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5107
diff
changeset
|
627 |
yield # 2/ execute the source's instructions |
e5d300d75519
[python] take care to this detail of @contextmanager: if an unhandled exception occurs in the block, it is reraised inside the generator at the point where the yield occurred
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5107
diff
changeset
|
628 |
finally: |
e5d300d75519
[python] take care to this detail of @contextmanager: if an unhandled exception occurs in the block, it is reraised inside the generator at the point where the yield occurred
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5107
diff
changeset
|
629 |
# 3/ restore original values |
6889
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
630 |
for entity, attr, value in restore_values: |
6142
8bc6eac1fac1
[session] cleanup hook / operation / entity edition api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6128
diff
changeset
|
631 |
entity.cw_edited.edited_attribute(attr, value) |
4964
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
632 |
|
0 | 633 |
def add_entity(self, session, entity): |
634 |
"""add a new entity to the source""" |
|
4964
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
635 |
with self._storage_handler(entity, 'added'): |
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
636 |
attrs = self.preprocess_entity(entity) |
8900
010a59e12d89
use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
8821
diff
changeset
|
637 |
sql = self.sqlgen.insert(SQL_PREFIX + entity.cw_etype, attrs) |
4964
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
638 |
self.doexec(session, sql, attrs) |
8900
010a59e12d89
use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
8821
diff
changeset
|
639 |
if session.ertype_supports_undo(entity.cw_etype): |
4964
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
640 |
self._record_tx_action(session, 'tx_entity_actions', 'C', |
8900
010a59e12d89
use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
8821
diff
changeset
|
641 |
etype=entity.cw_etype, eid=entity.eid) |
1792 | 642 |
|
0 | 643 |
def update_entity(self, session, entity): |
644 |
"""replace an entity in the source""" |
|
4964
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
645 |
with self._storage_handler(entity, 'updated'): |
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
646 |
attrs = self.preprocess_entity(entity) |
8900
010a59e12d89
use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
8821
diff
changeset
|
647 |
if session.ertype_supports_undo(entity.cw_etype): |
4964
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
648 |
changes = self._save_attrs(session, entity, attrs) |
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
649 |
self._record_tx_action(session, 'tx_entity_actions', 'U', |
8900
010a59e12d89
use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
8821
diff
changeset
|
650 |
etype=entity.cw_etype, eid=entity.eid, |
4964
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
651 |
changes=self._binary(dumps(changes))) |
8900
010a59e12d89
use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
8821
diff
changeset
|
652 |
sql = self.sqlgen.update(SQL_PREFIX + entity.cw_etype, attrs, |
4964
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
653 |
['cw_eid']) |
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
654 |
self.doexec(session, sql, attrs) |
0 | 655 |
|
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
656 |
def delete_entity(self, session, entity): |
0 | 657 |
"""delete an entity from the source""" |
4964
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
658 |
with self._storage_handler(entity, 'deleted'): |
8900
010a59e12d89
use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
8821
diff
changeset
|
659 |
if session.ertype_supports_undo(entity.cw_etype): |
4964
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
660 |
attrs = [SQL_PREFIX + r.type |
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
661 |
for r in entity.e_schema.subject_relations() |
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
662 |
if (r.final or r.inlined) and not r in VIRTUAL_RTYPES] |
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
663 |
changes = self._save_attrs(session, entity, attrs) |
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
664 |
self._record_tx_action(session, 'tx_entity_actions', 'D', |
8900
010a59e12d89
use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
8821
diff
changeset
|
665 |
etype=entity.cw_etype, eid=entity.eid, |
4964
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
666 |
changes=self._binary(dumps(changes))) |
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
667 |
attrs = {'cw_eid': entity.eid} |
8900
010a59e12d89
use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
8821
diff
changeset
|
668 |
sql = self.sqlgen.delete(SQL_PREFIX + entity.cw_etype, attrs) |
4964
d9e8af8a7a42
[source] implement storages right in the source rather than in hooks
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
4943
diff
changeset
|
669 |
self.doexec(session, sql, attrs) |
0 | 670 |
|
5072
072ae171aeb0
[cleanup] style fixes, add nodes, 0.2 cents refactorings
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5071
diff
changeset
|
671 |
def add_relation(self, session, subject, rtype, object, inlined=False): |
072ae171aeb0
[cleanup] style fixes, add nodes, 0.2 cents refactorings
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5071
diff
changeset
|
672 |
"""add a relation to the source""" |
7237
9f619715665b
[server] improve the speed of setting relations between entities (closes #1625257)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7202
diff
changeset
|
673 |
self._add_relations(session, rtype, [(subject, object)], inlined) |
8265
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
674 |
if session.ertype_supports_undo(rtype): |
5072
072ae171aeb0
[cleanup] style fixes, add nodes, 0.2 cents refactorings
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5071
diff
changeset
|
675 |
self._record_tx_action(session, 'tx_relation_actions', 'A', |
072ae171aeb0
[cleanup] style fixes, add nodes, 0.2 cents refactorings
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5071
diff
changeset
|
676 |
eid_from=subject, rtype=rtype, eid_to=object) |
072ae171aeb0
[cleanup] style fixes, add nodes, 0.2 cents refactorings
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5071
diff
changeset
|
677 |
|
7237
9f619715665b
[server] improve the speed of setting relations between entities (closes #1625257)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7202
diff
changeset
|
678 |
def add_relations(self, session, rtype, subj_obj_list, inlined=False): |
9f619715665b
[server] improve the speed of setting relations between entities (closes #1625257)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7202
diff
changeset
|
679 |
"""add a relations to the source""" |
9f619715665b
[server] improve the speed of setting relations between entities (closes #1625257)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7202
diff
changeset
|
680 |
self._add_relations(session, rtype, subj_obj_list, inlined) |
8265
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
681 |
if session.ertype_supports_undo(rtype): |
7237
9f619715665b
[server] improve the speed of setting relations between entities (closes #1625257)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7202
diff
changeset
|
682 |
for subject, object in subj_obj_list: |
9f619715665b
[server] improve the speed of setting relations between entities (closes #1625257)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7202
diff
changeset
|
683 |
self._record_tx_action(session, 'tx_relation_actions', 'A', |
9f619715665b
[server] improve the speed of setting relations between entities (closes #1625257)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7202
diff
changeset
|
684 |
eid_from=subject, rtype=rtype, eid_to=object) |
7551 | 685 |
|
7237
9f619715665b
[server] improve the speed of setting relations between entities (closes #1625257)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7202
diff
changeset
|
686 |
def _add_relations(self, session, rtype, subj_obj_list, inlined=False): |
0 | 687 |
"""add a relation to the source""" |
7243
9ab01bf84eac
fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7237
diff
changeset
|
688 |
sql = [] |
4818
9f9bfbcdecfd
le patch massiveimport a été importé
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4810
diff
changeset
|
689 |
if inlined is False: |
7237
9f619715665b
[server] improve the speed of setting relations between entities (closes #1625257)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7202
diff
changeset
|
690 |
attrs = [{'eid_from': subject, 'eid_to': object} |
9f619715665b
[server] improve the speed of setting relations between entities (closes #1625257)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7202
diff
changeset
|
691 |
for subject, object in subj_obj_list] |
7243
9ab01bf84eac
fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7237
diff
changeset
|
692 |
sql.append((self.sqlgen.insert('%s_relation' % rtype, attrs[0]), attrs)) |
4818
9f9bfbcdecfd
le patch massiveimport a été importé
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4810
diff
changeset
|
693 |
else: # used by data import |
7243
9ab01bf84eac
fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7237
diff
changeset
|
694 |
etypes = {} |
9ab01bf84eac
fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7237
diff
changeset
|
695 |
for subject, object in subj_obj_list: |
9ab01bf84eac
fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7237
diff
changeset
|
696 |
etype = session.describe(subject)[0] |
9ab01bf84eac
fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7237
diff
changeset
|
697 |
if etype in etypes: |
9ab01bf84eac
fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7237
diff
changeset
|
698 |
etypes[etype].append((subject, object)) |
9ab01bf84eac
fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7237
diff
changeset
|
699 |
else: |
9ab01bf84eac
fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7237
diff
changeset
|
700 |
etypes[etype] = [(subject, object)] |
9ab01bf84eac
fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7237
diff
changeset
|
701 |
for subj_etype, subj_obj_list in etypes.iteritems(): |
9ab01bf84eac
fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7237
diff
changeset
|
702 |
attrs = [{'cw_eid': subject, SQL_PREFIX + rtype: object} |
9ab01bf84eac
fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7237
diff
changeset
|
703 |
for subject, object in subj_obj_list] |
9ab01bf84eac
fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7237
diff
changeset
|
704 |
sql.append((self.sqlgen.update(SQL_PREFIX + etype, attrs[0], |
9ab01bf84eac
fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7237
diff
changeset
|
705 |
['cw_eid']), |
9ab01bf84eac
fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7237
diff
changeset
|
706 |
attrs)) |
9ab01bf84eac
fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7237
diff
changeset
|
707 |
for statement, attrs in sql: |
9ab01bf84eac
fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7237
diff
changeset
|
708 |
self.doexecmany(session, statement, attrs) |
1792 | 709 |
|
0 | 710 |
def delete_relation(self, session, subject, rtype, object): |
711 |
"""delete a relation from the source""" |
|
712 |
rschema = self.schema.rschema(rtype) |
|
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
713 |
self._delete_relation(session, subject, rtype, object, rschema.inlined) |
8265
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
714 |
if session.ertype_supports_undo(rtype): |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
715 |
self._record_tx_action(session, 'tx_relation_actions', 'R', |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
716 |
eid_from=subject, rtype=rtype, eid_to=object) |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
717 |
|
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
718 |
def _delete_relation(self, session, subject, rtype, object, inlined=False): |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
719 |
"""delete a relation from the source""" |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
720 |
if inlined: |
1251
af40e615dc89
introduce a 'cw_' prefix on entity table and column names so we don't conflict with sql or DBMS specific keywords
sylvain.thenault@logilab.fr
parents:
1079
diff
changeset
|
721 |
table = SQL_PREFIX + session.describe(subject)[0] |
af40e615dc89
introduce a 'cw_' prefix on entity table and column names so we don't conflict with sql or DBMS specific keywords
sylvain.thenault@logilab.fr
parents:
1079
diff
changeset
|
722 |
column = SQL_PREFIX + rtype |
af40e615dc89
introduce a 'cw_' prefix on entity table and column names so we don't conflict with sql or DBMS specific keywords
sylvain.thenault@logilab.fr
parents:
1079
diff
changeset
|
723 |
sql = 'UPDATE %s SET %s=NULL WHERE %seid=%%(eid)s' % (table, column, |
af40e615dc89
introduce a 'cw_' prefix on entity table and column names so we don't conflict with sql or DBMS specific keywords
sylvain.thenault@logilab.fr
parents:
1079
diff
changeset
|
724 |
SQL_PREFIX) |
0 | 725 |
attrs = {'eid' : subject} |
726 |
else: |
|
727 |
attrs = {'eid_from': subject, 'eid_to': object} |
|
728 |
sql = self.sqlgen.delete('%s_relation' % rtype, attrs) |
|
2306
95da5d9f0870
give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2072
diff
changeset
|
729 |
self.doexec(session, sql, attrs) |
0 | 730 |
|
2618
ff9b0d5bd884
[F repo sqlite schema changes] don't rollback on potentially expected schema changes failure
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2611
diff
changeset
|
731 |
def doexec(self, session, query, args=None, rollback=True): |
0 | 732 |
"""Execute a query. |
733 |
it's a function just so that it shows up in profiling |
|
734 |
""" |
|
7398
26695dd703d8
[repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
735 |
cursor = session.cnxset[self.uri] |
2593
16d9419a4a79
F: start to handle binary debug log level on the server side
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2589
diff
changeset
|
736 |
if server.DEBUG & server.DBG_SQL: |
7398
26695dd703d8
[repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
737 |
cnx = session.cnxset.connection(self.uri) |
2625
d6012db7b93e
R [server debug]Â more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2620
diff
changeset
|
738 |
# getattr to get the actual connection if cnx is a ConnectionWrapper |
d6012db7b93e
R [server debug]Â more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2620
diff
changeset
|
739 |
# instance |
d6012db7b93e
R [server debug]Â more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2620
diff
changeset
|
740 |
print 'exec', query, args, getattr(cnx, '_cnx', cnx) |
0 | 741 |
try: |
2306
95da5d9f0870
give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2072
diff
changeset
|
742 |
# str(query) to avoid error if it's an unicode string |
0 | 743 |
cursor.execute(str(query), args) |
8695
358d8bed9626
[toward-py3k] rewrite to "except AnException as exc:" (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents:
8694
diff
changeset
|
744 |
except Exception as ex: |
4682
4994901b7379
don't issue critical message when trying to alter sqlite db during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4517
diff
changeset
|
745 |
if self.repo.config.mode != 'test': |
4994901b7379
don't issue critical message when trying to alter sqlite db during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4517
diff
changeset
|
746 |
# during test we get those message when trying to alter sqlite |
4994901b7379
don't issue critical message when trying to alter sqlite db during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4517
diff
changeset
|
747 |
# db schema |
4994901b7379
don't issue critical message when trying to alter sqlite db during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4517
diff
changeset
|
748 |
self.critical("sql: %r\n args: %s\ndbms message: %r", |
4994901b7379
don't issue critical message when trying to alter sqlite db during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4517
diff
changeset
|
749 |
query, args, ex.args[0]) |
2618
ff9b0d5bd884
[F repo sqlite schema changes] don't rollback on potentially expected schema changes failure
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2611
diff
changeset
|
750 |
if rollback: |
ff9b0d5bd884
[F repo sqlite schema changes] don't rollback on potentially expected schema changes failure
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2611
diff
changeset
|
751 |
try: |
7398
26695dd703d8
[repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
752 |
session.cnxset.connection(self.uri).rollback() |
4692
11a040e2601c
[test] also hide this message during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4682
diff
changeset
|
753 |
if self.repo.config.mode != 'test': |
11a040e2601c
[test] also hide this message during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4682
diff
changeset
|
754 |
self.critical('transaction has been rollbacked') |
8695
358d8bed9626
[toward-py3k] rewrite to "except AnException as exc:" (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents:
8694
diff
changeset
|
755 |
except Exception as ex: |
2618
ff9b0d5bd884
[F repo sqlite schema changes] don't rollback on potentially expected schema changes failure
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2611
diff
changeset
|
756 |
pass |
6211
e9d125fd1465
nicer error reporting for unique together constraints
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6201
diff
changeset
|
757 |
if ex.__class__.__name__ == 'IntegrityError': |
e9d125fd1465
nicer error reporting for unique together constraints
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6201
diff
changeset
|
758 |
# need string comparison because of various backends |
e9d125fd1465
nicer error reporting for unique together constraints
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6201
diff
changeset
|
759 |
for arg in ex.args: |
e9d125fd1465
nicer error reporting for unique together constraints
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6201
diff
changeset
|
760 |
mo = re.search('unique_cw_[^ ]+_idx', arg) |
e9d125fd1465
nicer error reporting for unique together constraints
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6201
diff
changeset
|
761 |
if mo is not None: |
e9d125fd1465
nicer error reporting for unique together constraints
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6201
diff
changeset
|
762 |
index_name = mo.group(0) |
8820
dbffb6959564
server/source/native: fix wrong usage of .lstrip that produce garbled error messages (closes #2777641)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
8546
diff
changeset
|
763 |
# right-chop '_idx' postfix |
dbffb6959564
server/source/native: fix wrong usage of .lstrip that produce garbled error messages (closes #2777641)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
8546
diff
changeset
|
764 |
# (garanteed to be there, see regexp above) |
dbffb6959564
server/source/native: fix wrong usage of .lstrip that produce garbled error messages (closes #2777641)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
8546
diff
changeset
|
765 |
elements = index_name[:-4].split('_cw_')[1:] |
6211
e9d125fd1465
nicer error reporting for unique together constraints
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6201
diff
changeset
|
766 |
etype = elements[0] |
6297
23c1e50ff97b
[rql] fix bug with query like 'Any 1 WHERE NOT X in_group G': tables should be kept in EXISTS() even when there are no restriction
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6247
diff
changeset
|
767 |
rtypes = elements[1:] |
6211
e9d125fd1465
nicer error reporting for unique together constraints
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6201
diff
changeset
|
768 |
raise UniqueTogetherError(etype, rtypes) |
6247
f7cb092d2296
unique_together: recast exception raised by sqlite
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6211
diff
changeset
|
769 |
mo = re.search('columns (.*) are not unique', arg) |
f7cb092d2296
unique_together: recast exception raised by sqlite
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6211
diff
changeset
|
770 |
if mo is not None: # sqlite in use |
8820
dbffb6959564
server/source/native: fix wrong usage of .lstrip that produce garbled error messages (closes #2777641)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
8546
diff
changeset
|
771 |
# we left chop the 'cw_' prefix of attribute names |
dbffb6959564
server/source/native: fix wrong usage of .lstrip that produce garbled error messages (closes #2777641)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
8546
diff
changeset
|
772 |
rtypes = [c.strip()[3:] |
dbffb6959564
server/source/native: fix wrong usage of .lstrip that produce garbled error messages (closes #2777641)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
8546
diff
changeset
|
773 |
for c in mo.group(1).split(',')] |
6247
f7cb092d2296
unique_together: recast exception raised by sqlite
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6211
diff
changeset
|
774 |
etype = '???' |
f7cb092d2296
unique_together: recast exception raised by sqlite
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6211
diff
changeset
|
775 |
raise UniqueTogetherError(etype, rtypes) |
0 | 776 |
raise |
2306
95da5d9f0870
give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2072
diff
changeset
|
777 |
return cursor |
1792 | 778 |
|
2306
95da5d9f0870
give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2072
diff
changeset
|
779 |
def doexecmany(self, session, query, args): |
0 | 780 |
"""Execute a query. |
781 |
it's a function just so that it shows up in profiling |
|
782 |
""" |
|
2593
16d9419a4a79
F: start to handle binary debug log level on the server side
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2589
diff
changeset
|
783 |
if server.DEBUG & server.DBG_SQL: |
0 | 784 |
print 'execmany', query, 'with', len(args), 'arguments' |
7398
26695dd703d8
[repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
785 |
cursor = session.cnxset[self.uri] |
0 | 786 |
try: |
2306
95da5d9f0870
give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2072
diff
changeset
|
787 |
# str(query) to avoid error if it's an unicode string |
0 | 788 |
cursor.executemany(str(query), args) |
8695
358d8bed9626
[toward-py3k] rewrite to "except AnException as exc:" (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents:
8694
diff
changeset
|
789 |
except Exception as ex: |
4682
4994901b7379
don't issue critical message when trying to alter sqlite db during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4517
diff
changeset
|
790 |
if self.repo.config.mode != 'test': |
4994901b7379
don't issue critical message when trying to alter sqlite db during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4517
diff
changeset
|
791 |
# during test we get those message when trying to alter sqlite |
4994901b7379
don't issue critical message when trying to alter sqlite db during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4517
diff
changeset
|
792 |
# db schema |
4994901b7379
don't issue critical message when trying to alter sqlite db during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4517
diff
changeset
|
793 |
self.critical("sql many: %r\n args: %s\ndbms message: %r", |
4994901b7379
don't issue critical message when trying to alter sqlite db during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4517
diff
changeset
|
794 |
query, args, ex.args[0]) |
2306
95da5d9f0870
give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2072
diff
changeset
|
795 |
try: |
7398
26695dd703d8
[repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
796 |
session.cnxset.connection(self.uri).rollback() |
4692
11a040e2601c
[test] also hide this message during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4682
diff
changeset
|
797 |
if self.repo.config.mode != 'test': |
11a040e2601c
[test] also hide this message during test
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4682
diff
changeset
|
798 |
self.critical('transaction has been rollbacked') |
7645
cb2459aaa7d7
[source] closes #1816412: try to reconnect to dbms if connection is reset in eid_type_source()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
7562
diff
changeset
|
799 |
except Exception: |
2306
95da5d9f0870
give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2072
diff
changeset
|
800 |
pass |
0 | 801 |
raise |
1792 | 802 |
|
0 | 803 |
# short cut to method requiring advanced db helper usage ################## |
1792 | 804 |
|
5891
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
805 |
def update_rdef_column(self, session, rdef): |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
806 |
"""update physical column for a relation definition (final or inlined) |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
807 |
""" |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
808 |
table, column = rdef_table_column(rdef) |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
809 |
coltype, allownull = rdef_physical_info(self.dbhelper, rdef) |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
810 |
if not self.dbhelper.alter_column_support: |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
811 |
self.error("backend can't alter %s.%s to %s%s", table, column, coltype, |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
812 |
not allownull and 'NOT NULL' or '') |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
813 |
return |
7398
26695dd703d8
[repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
814 |
self.dbhelper.change_col_type(LogCursor(session.cnxset[self.uri]), |
5891
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
815 |
table, column, coltype, allownull) |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
816 |
self.info('altered %s.%s: now %s%s', table, column, coltype, |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
817 |
not allownull and 'NOT NULL' or '') |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
818 |
|
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
819 |
def update_rdef_null_allowed(self, session, rdef): |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
820 |
"""update NULL / NOT NULL of physical column for a relation definition |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
821 |
(final or inlined) |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
822 |
""" |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
823 |
if not self.dbhelper.alter_column_support: |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
824 |
# not supported (and NOT NULL not set by yams in that case, so no |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
825 |
# worry) |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
826 |
return |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
827 |
table, column = rdef_table_column(rdef) |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
828 |
coltype, allownull = rdef_physical_info(self.dbhelper, rdef) |
7398
26695dd703d8
[repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
829 |
self.dbhelper.set_null_allowed(LogCursor(session.cnxset[self.uri]), |
5891
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
830 |
table, column, coltype, allownull) |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
831 |
|
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
832 |
def update_rdef_indexed(self, session, rdef): |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
833 |
table, column = rdef_table_column(rdef) |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
834 |
if rdef.indexed: |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
835 |
self.create_index(session, table, column) |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
836 |
else: |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
837 |
self.drop_index(session, table, column) |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
838 |
|
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
839 |
def update_rdef_unique(self, session, rdef): |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
840 |
table, column = rdef_table_column(rdef) |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
841 |
if rdef.constraint_by_type('UniqueConstraint'): |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
842 |
self.create_index(session, table, column, unique=True) |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
843 |
else: |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
844 |
self.drop_index(session, table, column, unique=True) |
99024ad59223
[schema migration] import refactoring to fix #1109558 and enhances things on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5849
diff
changeset
|
845 |
|
0 | 846 |
def create_index(self, session, table, column, unique=False): |
7398
26695dd703d8
[repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
847 |
cursor = LogCursor(session.cnxset[self.uri]) |
0 | 848 |
self.dbhelper.create_index(cursor, table, column, unique) |
1792 | 849 |
|
0 | 850 |
def drop_index(self, session, table, column, unique=False): |
7398
26695dd703d8
[repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
851 |
cursor = LogCursor(session.cnxset[self.uri]) |
0 | 852 |
self.dbhelper.drop_index(cursor, table, column, unique) |
853 |
||
854 |
# system source interface ################################################# |
|
855 |
||
7645
cb2459aaa7d7
[source] closes #1816412: try to reconnect to dbms if connection is reset in eid_type_source()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
7562
diff
changeset
|
856 |
def _eid_type_source(self, session, eid, sql, _retry=True): |
0 | 857 |
try: |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
858 |
res = self.doexec(session, sql).fetchone() |
7646
91a984ba420f
[native source] fix bug introduced in 7645:cb2459aaa7d7: fetchone() may return None
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7645
diff
changeset
|
859 |
if res is not None: |
91a984ba420f
[native source] fix bug introduced in 7645:cb2459aaa7d7: fetchone() may return None
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7645
diff
changeset
|
860 |
return res |
7645
cb2459aaa7d7
[source] closes #1816412: try to reconnect to dbms if connection is reset in eid_type_source()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
7562
diff
changeset
|
861 |
except (self.OperationalError, self.InterfaceError): |
cb2459aaa7d7
[source] closes #1816412: try to reconnect to dbms if connection is reset in eid_type_source()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
7562
diff
changeset
|
862 |
if session.mode == 'read' and _retry: |
cb2459aaa7d7
[source] closes #1816412: try to reconnect to dbms if connection is reset in eid_type_source()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
7562
diff
changeset
|
863 |
self.warning("trying to reconnect (eid_type_source())") |
cb2459aaa7d7
[source] closes #1816412: try to reconnect to dbms if connection is reset in eid_type_source()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
7562
diff
changeset
|
864 |
session.cnxset.reconnect(self) |
cb2459aaa7d7
[source] closes #1816412: try to reconnect to dbms if connection is reset in eid_type_source()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
7562
diff
changeset
|
865 |
return self._eid_type_source(session, eid, sql, _retry=False) |
cb2459aaa7d7
[source] closes #1816412: try to reconnect to dbms if connection is reset in eid_type_source()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
7562
diff
changeset
|
866 |
except Exception: |
cb2459aaa7d7
[source] closes #1816412: try to reconnect to dbms if connection is reset in eid_type_source()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
7562
diff
changeset
|
867 |
assert session.cnxset, 'session has no connections set' |
cb2459aaa7d7
[source] closes #1816412: try to reconnect to dbms if connection is reset in eid_type_source()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
7562
diff
changeset
|
868 |
self.exception('failed to query entities table for eid %s', eid) |
cb2459aaa7d7
[source] closes #1816412: try to reconnect to dbms if connection is reset in eid_type_source()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
7562
diff
changeset
|
869 |
raise UnknownEid(eid) |
cb2459aaa7d7
[source] closes #1816412: try to reconnect to dbms if connection is reset in eid_type_source()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
7562
diff
changeset
|
870 |
|
7879
9aae456abab5
[pylint] fix pylint detected errors and tweak it so that pylint -E will be much less verbose next time (+ update some copyrights on the way)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7839
diff
changeset
|
871 |
def eid_type_source(self, session, eid): # pylint: disable=E0202 |
0 | 872 |
"""return a tuple (type, source, extid) for the entity with id <eid>""" |
7543
570522300e22
[ms, entity metas] add 'actual source' to entities table / base entity metadata cache. Closes #1767090
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7536
diff
changeset
|
873 |
sql = 'SELECT type, source, extid, asource FROM entities WHERE eid=%s' % eid |
7645
cb2459aaa7d7
[source] closes #1816412: try to reconnect to dbms if connection is reset in eid_type_source()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
7562
diff
changeset
|
874 |
res = self._eid_type_source(session, eid, sql) |
7543
570522300e22
[ms, entity metas] add 'actual source' to entities table / base entity metadata cache. Closes #1767090
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7536
diff
changeset
|
875 |
if res[-2] is not None: |
1952
8e19c813750d
fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1792
diff
changeset
|
876 |
if not isinstance(res, list): |
8e19c813750d
fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1792
diff
changeset
|
877 |
res = list(res) |
7543
570522300e22
[ms, entity metas] add 'actual source' to entities table / base entity metadata cache. Closes #1767090
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7536
diff
changeset
|
878 |
res[-2] = b64decode(res[-2]) |
570522300e22
[ms, entity metas] add 'actual source' to entities table / base entity metadata cache. Closes #1767090
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7536
diff
changeset
|
879 |
return res |
570522300e22
[ms, entity metas] add 'actual source' to entities table / base entity metadata cache. Closes #1767090
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7536
diff
changeset
|
880 |
|
570522300e22
[ms, entity metas] add 'actual source' to entities table / base entity metadata cache. Closes #1767090
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7536
diff
changeset
|
881 |
def eid_type_source_pre_131(self, session, eid): |
570522300e22
[ms, entity metas] add 'actual source' to entities table / base entity metadata cache. Closes #1767090
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7536
diff
changeset
|
882 |
"""return a tuple (type, source, extid) for the entity with id <eid>""" |
0 | 883 |
sql = 'SELECT type, source, extid FROM entities WHERE eid=%s' % eid |
7645
cb2459aaa7d7
[source] closes #1816412: try to reconnect to dbms if connection is reset in eid_type_source()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
7562
diff
changeset
|
884 |
res = self._eid_type_source(session, eid, sql) |
7543
570522300e22
[ms, entity metas] add 'actual source' to entities table / base entity metadata cache. Closes #1767090
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7536
diff
changeset
|
885 |
if not isinstance(res, list): |
570522300e22
[ms, entity metas] add 'actual source' to entities table / base entity metadata cache. Closes #1767090
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7536
diff
changeset
|
886 |
res = list(res) |
1952
8e19c813750d
fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1792
diff
changeset
|
887 |
if res[-1] is not None: |
8e19c813750d
fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1792
diff
changeset
|
888 |
res[-1] = b64decode(res[-1]) |
7543
570522300e22
[ms, entity metas] add 'actual source' to entities table / base entity metadata cache. Closes #1767090
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7536
diff
changeset
|
889 |
res.append(res[1]) |
0 | 890 |
return res |
891 |
||
6957
ffda12be2e9f
[repository] #1460066: backport datafeed cube as cubicweb source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6945
diff
changeset
|
892 |
def extid2eid(self, session, source_uri, extid): |
1952
8e19c813750d
fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1792
diff
changeset
|
893 |
"""get eid from an external id. Return None if no record found.""" |
8e19c813750d
fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1792
diff
changeset
|
894 |
assert isinstance(extid, str) |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
895 |
cursor = self.doexec(session, |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
896 |
'SELECT eid FROM entities ' |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
897 |
'WHERE extid=%(x)s AND source=%(s)s', |
6957
ffda12be2e9f
[repository] #1460066: backport datafeed cube as cubicweb source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6945
diff
changeset
|
898 |
{'x': b64encode(extid), 's': source_uri}) |
0 | 899 |
# XXX testing rowcount cause strange bug with sqlite, results are there |
900 |
# but rowcount is 0 |
|
1792 | 901 |
#if cursor.rowcount > 0: |
0 | 902 |
try: |
903 |
result = cursor.fetchone() |
|
904 |
if result: |
|
1954 | 905 |
return result[0] |
7645
cb2459aaa7d7
[source] closes #1816412: try to reconnect to dbms if connection is reset in eid_type_source()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
7562
diff
changeset
|
906 |
except Exception: |
0 | 907 |
pass |
908 |
return None |
|
1792 | 909 |
|
5168
1ab032df5ca3
SQL Server port: temporary table handling
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5131
diff
changeset
|
910 |
def make_temp_table_name(self, table): |
7190
f72e3f4666b0
[system source] drop now useless bw compat
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7118
diff
changeset
|
911 |
return self.dbhelper.temporary_table_name(table) |
5168
1ab032df5ca3
SQL Server port: temporary table handling
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5131
diff
changeset
|
912 |
|
0 | 913 |
def temp_table_def(self, selected, sol, table): |
914 |
return make_schema(selected, sol, table, self.dbhelper.TYPE_MAPPING) |
|
915 |
||
916 |
def create_temp_table(self, session, table, schema): |
|
917 |
# we don't want on commit drop, this may cause problem when |
|
918 |
# running with an ldap source, and table will be deleted manually any way |
|
919 |
# on commit |
|
920 |
sql = self.dbhelper.sql_temporary_table(table, schema, False) |
|
2306
95da5d9f0870
give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2072
diff
changeset
|
921 |
self.doexec(session, sql) |
1792 | 922 |
|
5649
a07dee204187
fix unit tests by not using the new create_eid implementation with sqlite
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5645
diff
changeset
|
923 |
def _create_eid_sqlite(self, session): |
7202
5b80f5ee61b5
[native source] eid_creation_lock actually lock the eid connection creation and usage
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7190
diff
changeset
|
924 |
with self._eid_cnx_lock: |
5644
73d8a757db80
fix create_eid for sqlite (and bring back tests)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5639
diff
changeset
|
925 |
for sql in self.dbhelper.sqls_increment_sequence('entities_id_seq'): |
73d8a757db80
fix create_eid for sqlite (and bring back tests)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5639
diff
changeset
|
926 |
cursor = self.doexec(session, sql) |
73d8a757db80
fix create_eid for sqlite (and bring back tests)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5639
diff
changeset
|
927 |
return cursor.fetchone()[0] |
73d8a757db80
fix create_eid for sqlite (and bring back tests)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5639
diff
changeset
|
928 |
|
73d8a757db80
fix create_eid for sqlite (and bring back tests)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5639
diff
changeset
|
929 |
|
7879
9aae456abab5
[pylint] fix pylint detected errors and tweak it so that pylint -E will be much less verbose next time (+ update some copyrights on the way)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7839
diff
changeset
|
930 |
def create_eid(self, session): # pylint: disable=E0202 |
7202
5b80f5ee61b5
[native source] eid_creation_lock actually lock the eid connection creation and usage
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7190
diff
changeset
|
931 |
# lock needed to prevent 'Connection is busy with results for another |
5b80f5ee61b5
[native source] eid_creation_lock actually lock the eid connection creation and usage
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7190
diff
changeset
|
932 |
# command (0)' errors with SQLServer |
5b80f5ee61b5
[native source] eid_creation_lock actually lock the eid connection creation and usage
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7190
diff
changeset
|
933 |
with self._eid_cnx_lock: |
7879
9aae456abab5
[pylint] fix pylint detected errors and tweak it so that pylint -E will be much less verbose next time (+ update some copyrights on the way)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7839
diff
changeset
|
934 |
return self._create_eid() # pylint: disable=E1102 |
5649
a07dee204187
fix unit tests by not using the new create_eid implementation with sqlite
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5645
diff
changeset
|
935 |
|
7879
9aae456abab5
[pylint] fix pylint detected errors and tweak it so that pylint -E will be much less verbose next time (+ update some copyrights on the way)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7839
diff
changeset
|
936 |
def _create_eid(self): # pylint: disable=E0202 |
5649
a07dee204187
fix unit tests by not using the new create_eid implementation with sqlite
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5645
diff
changeset
|
937 |
# internal function doing the eid creation without locking. |
a07dee204187
fix unit tests by not using the new create_eid implementation with sqlite
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5645
diff
changeset
|
938 |
# needed for the recursive handling of disconnections (otherwise we |
7202
5b80f5ee61b5
[native source] eid_creation_lock actually lock the eid connection creation and usage
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7190
diff
changeset
|
939 |
# deadlock on self._eid_cnx_lock |
5639
4acb860159e4
[win32] fix deadlock occuring on the sequence tables with SQLServer
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5605
diff
changeset
|
940 |
if self._eid_creation_cnx is None: |
4acb860159e4
[win32] fix deadlock occuring on the sequence tables with SQLServer
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5605
diff
changeset
|
941 |
self._eid_creation_cnx = self.get_connection() |
4acb860159e4
[win32] fix deadlock occuring on the sequence tables with SQLServer
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5605
diff
changeset
|
942 |
cnx = self._eid_creation_cnx |
0 | 943 |
try: |
6850
2b9e58174327
[repo] move cursor creation into the try/except, it may raises an InterfaceError if the connection is closed
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6427
diff
changeset
|
944 |
cursor = cnx.cursor() |
0 | 945 |
for sql in self.dbhelper.sqls_increment_sequence('entities_id_seq'): |
5639
4acb860159e4
[win32] fix deadlock occuring on the sequence tables with SQLServer
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5605
diff
changeset
|
946 |
cursor.execute(sql) |
4acb860159e4
[win32] fix deadlock occuring on the sequence tables with SQLServer
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5605
diff
changeset
|
947 |
eid = cursor.fetchone()[0] |
4acb860159e4
[win32] fix deadlock occuring on the sequence tables with SQLServer
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5605
diff
changeset
|
948 |
except (self.OperationalError, self.InterfaceError): |
4acb860159e4
[win32] fix deadlock occuring on the sequence tables with SQLServer
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5605
diff
changeset
|
949 |
# FIXME: better detection of deconnection pb |
4acb860159e4
[win32] fix deadlock occuring on the sequence tables with SQLServer
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5605
diff
changeset
|
950 |
self.warning("trying to reconnect create eid connection") |
4acb860159e4
[win32] fix deadlock occuring on the sequence tables with SQLServer
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5605
diff
changeset
|
951 |
self._eid_creation_cnx = None |
7879
9aae456abab5
[pylint] fix pylint detected errors and tweak it so that pylint -E will be much less verbose next time (+ update some copyrights on the way)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7839
diff
changeset
|
952 |
return self._create_eid() # pylint: disable=E1102 |
8695
358d8bed9626
[toward-py3k] rewrite to "except AnException as exc:" (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents:
8694
diff
changeset
|
953 |
except self.DbapiError as exc: |
5639
4acb860159e4
[win32] fix deadlock occuring on the sequence tables with SQLServer
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5605
diff
changeset
|
954 |
# We get this one with pyodbc and SQL Server when connection was reset |
4acb860159e4
[win32] fix deadlock occuring on the sequence tables with SQLServer
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5605
diff
changeset
|
955 |
if exc.args[0] == '08S01': |
4acb860159e4
[win32] fix deadlock occuring on the sequence tables with SQLServer
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5605
diff
changeset
|
956 |
self.warning("trying to reconnect create eid connection") |
4acb860159e4
[win32] fix deadlock occuring on the sequence tables with SQLServer
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5605
diff
changeset
|
957 |
self._eid_creation_cnx = None |
7879
9aae456abab5
[pylint] fix pylint detected errors and tweak it so that pylint -E will be much less verbose next time (+ update some copyrights on the way)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7839
diff
changeset
|
958 |
return self._create_eid() # pylint: disable=E1102 |
5639
4acb860159e4
[win32] fix deadlock occuring on the sequence tables with SQLServer
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5605
diff
changeset
|
959 |
else: |
4acb860159e4
[win32] fix deadlock occuring on the sequence tables with SQLServer
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5605
diff
changeset
|
960 |
raise |
7645
cb2459aaa7d7
[source] closes #1816412: try to reconnect to dbms if connection is reset in eid_type_source()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
7562
diff
changeset
|
961 |
except Exception: # WTF? |
5639
4acb860159e4
[win32] fix deadlock occuring on the sequence tables with SQLServer
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5605
diff
changeset
|
962 |
cnx.rollback() |
4acb860159e4
[win32] fix deadlock occuring on the sequence tables with SQLServer
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5605
diff
changeset
|
963 |
self._eid_creation_cnx = None |
4acb860159e4
[win32] fix deadlock occuring on the sequence tables with SQLServer
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5605
diff
changeset
|
964 |
self.exception('create eid failed in an unforeseen way on SQL statement %s', sql) |
4acb860159e4
[win32] fix deadlock occuring on the sequence tables with SQLServer
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5605
diff
changeset
|
965 |
raise |
4acb860159e4
[win32] fix deadlock occuring on the sequence tables with SQLServer
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5605
diff
changeset
|
966 |
else: |
4acb860159e4
[win32] fix deadlock occuring on the sequence tables with SQLServer
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5605
diff
changeset
|
967 |
cnx.commit() |
4acb860159e4
[win32] fix deadlock occuring on the sequence tables with SQLServer
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5605
diff
changeset
|
968 |
return eid |
5649
a07dee204187
fix unit tests by not using the new create_eid implementation with sqlite
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5645
diff
changeset
|
969 |
|
8625
7ee0752178e5
[dataimport] Add SQL Store for faster import - works ONLY with Postgres for now, as it requires "copy from" command - closes #2410822
Vincent Michel <vincent.michel@logilab.fr>
parents:
8573
diff
changeset
|
970 |
def _handle_is_relation_sql(self, session, sql, attrs): |
7ee0752178e5
[dataimport] Add SQL Store for faster import - works ONLY with Postgres for now, as it requires "copy from" command - closes #2410822
Vincent Michel <vincent.michel@logilab.fr>
parents:
8573
diff
changeset
|
971 |
""" Handler for specific is_relation sql that may be |
7ee0752178e5
[dataimport] Add SQL Store for faster import - works ONLY with Postgres for now, as it requires "copy from" command - closes #2410822
Vincent Michel <vincent.michel@logilab.fr>
parents:
8573
diff
changeset
|
972 |
overwritten in some stores""" |
7ee0752178e5
[dataimport] Add SQL Store for faster import - works ONLY with Postgres for now, as it requires "copy from" command - closes #2410822
Vincent Michel <vincent.michel@logilab.fr>
parents:
8573
diff
changeset
|
973 |
self.doexec(session, sql % attrs) |
7ee0752178e5
[dataimport] Add SQL Store for faster import - works ONLY with Postgres for now, as it requires "copy from" command - closes #2410822
Vincent Michel <vincent.michel@logilab.fr>
parents:
8573
diff
changeset
|
974 |
|
7ee0752178e5
[dataimport] Add SQL Store for faster import - works ONLY with Postgres for now, as it requires "copy from" command - closes #2410822
Vincent Michel <vincent.michel@logilab.fr>
parents:
8573
diff
changeset
|
975 |
_handle_insert_entity_sql = doexec |
7ee0752178e5
[dataimport] Add SQL Store for faster import - works ONLY with Postgres for now, as it requires "copy from" command - closes #2410822
Vincent Michel <vincent.michel@logilab.fr>
parents:
8573
diff
changeset
|
976 |
_handle_is_instance_of_sql = _handle_source_relation_sql = _handle_is_relation_sql |
7ee0752178e5
[dataimport] Add SQL Store for faster import - works ONLY with Postgres for now, as it requires "copy from" command - closes #2410822
Vincent Michel <vincent.michel@logilab.fr>
parents:
8573
diff
changeset
|
977 |
|
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
978 |
def add_info(self, session, entity, source, extid, complete): |
0 | 979 |
"""add type and source info for an eid into the system table""" |
980 |
# begin by inserting eid/type/source/extid into the entities table |
|
1952
8e19c813750d
fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1792
diff
changeset
|
981 |
if extid is not None: |
8e19c813750d
fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1792
diff
changeset
|
982 |
assert isinstance(extid, str) |
8e19c813750d
fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1792
diff
changeset
|
983 |
extid = b64encode(extid) |
6957
ffda12be2e9f
[repository] #1460066: backport datafeed cube as cubicweb source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6945
diff
changeset
|
984 |
uri = 'system' if source.copy_based_source else source.uri |
8900
010a59e12d89
use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
8821
diff
changeset
|
985 |
attrs = {'type': entity.cw_etype, 'eid': entity.eid, 'extid': extid, |
7922
d307c3817782
[repository] use utcnow instead of now for some internal timetamps (closes #1988458)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7879
diff
changeset
|
986 |
'source': uri, 'asource': source.uri, 'mtime': datetime.utcnow()} |
8625
7ee0752178e5
[dataimport] Add SQL Store for faster import - works ONLY with Postgres for now, as it requires "copy from" command - closes #2410822
Vincent Michel <vincent.michel@logilab.fr>
parents:
8573
diff
changeset
|
987 |
self._handle_insert_entity_sql(session, self.sqlgen.insert('entities', attrs), attrs) |
6427
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6426
diff
changeset
|
988 |
# insert core relations: is, is_instance_of and cw_source |
6851
824d5b6eae7f
[repo] kill no more needed 'recreate' feature
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6849
diff
changeset
|
989 |
try: |
8625
7ee0752178e5
[dataimport] Add SQL Store for faster import - works ONLY with Postgres for now, as it requires "copy from" command - closes #2410822
Vincent Michel <vincent.michel@logilab.fr>
parents:
8573
diff
changeset
|
990 |
self._handle_is_relation_sql(session, 'INSERT INTO is_relation(eid_from,eid_to) VALUES (%s,%s)', |
7ee0752178e5
[dataimport] Add SQL Store for faster import - works ONLY with Postgres for now, as it requires "copy from" command - closes #2410822
Vincent Michel <vincent.michel@logilab.fr>
parents:
8573
diff
changeset
|
991 |
(entity.eid, eschema_eid(session, entity.e_schema))) |
6851
824d5b6eae7f
[repo] kill no more needed 'recreate' feature
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6849
diff
changeset
|
992 |
except IndexError: |
824d5b6eae7f
[repo] kill no more needed 'recreate' feature
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6849
diff
changeset
|
993 |
# during schema serialization, skip |
824d5b6eae7f
[repo] kill no more needed 'recreate' feature
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6849
diff
changeset
|
994 |
pass |
824d5b6eae7f
[repo] kill no more needed 'recreate' feature
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6849
diff
changeset
|
995 |
else: |
824d5b6eae7f
[repo] kill no more needed 'recreate' feature
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6849
diff
changeset
|
996 |
for eschema in entity.e_schema.ancestors() + [entity.e_schema]: |
8625
7ee0752178e5
[dataimport] Add SQL Store for faster import - works ONLY with Postgres for now, as it requires "copy from" command - closes #2410822
Vincent Michel <vincent.michel@logilab.fr>
parents:
8573
diff
changeset
|
997 |
self._handle_is_relation_sql(session, |
7ee0752178e5
[dataimport] Add SQL Store for faster import - works ONLY with Postgres for now, as it requires "copy from" command - closes #2410822
Vincent Michel <vincent.michel@logilab.fr>
parents:
8573
diff
changeset
|
998 |
'INSERT INTO is_instance_of_relation(eid_from,eid_to) VALUES (%s,%s)', |
7ee0752178e5
[dataimport] Add SQL Store for faster import - works ONLY with Postgres for now, as it requires "copy from" command - closes #2410822
Vincent Michel <vincent.michel@logilab.fr>
parents:
8573
diff
changeset
|
999 |
(entity.eid, eschema_eid(session, eschema))) |
6851
824d5b6eae7f
[repo] kill no more needed 'recreate' feature
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6849
diff
changeset
|
1000 |
if 'CWSource' in self.schema and source.eid is not None: # else, cw < 3.10 |
8625
7ee0752178e5
[dataimport] Add SQL Store for faster import - works ONLY with Postgres for now, as it requires "copy from" command - closes #2410822
Vincent Michel <vincent.michel@logilab.fr>
parents:
8573
diff
changeset
|
1001 |
self._handle_is_relation_sql(session, 'INSERT INTO cw_source_relation(eid_from,eid_to) VALUES (%s,%s)', |
7ee0752178e5
[dataimport] Add SQL Store for faster import - works ONLY with Postgres for now, as it requires "copy from" command - closes #2410822
Vincent Michel <vincent.michel@logilab.fr>
parents:
8573
diff
changeset
|
1002 |
(entity.eid, source.eid)) |
4806
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1003 |
# now we can update the full text index |
8900
010a59e12d89
use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
8821
diff
changeset
|
1004 |
if self.do_fti and self.need_fti_indexation(entity.cw_etype): |
4806
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1005 |
if complete: |
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1006 |
entity.complete(entity.e_schema.indexable_attributes()) |
5070
b1f80ccadda3
[repo] refactor fti operation to use set_operation and a single operation whatever the number of entities to ft index
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5067
diff
changeset
|
1007 |
self.index_entity(session, entity=entity) |
4806
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1008 |
|
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1009 |
def update_info(self, session, entity, need_fti_update): |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1010 |
"""mark entity as being modified, fulltext reindex if needed""" |
4806
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1011 |
if self.do_fti and need_fti_update: |
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1012 |
# reindex the entity only if this query is updating at least |
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1013 |
# one indexable attribute |
5070
b1f80ccadda3
[repo] refactor fti operation to use set_operation and a single operation whatever the number of entities to ft index
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5067
diff
changeset
|
1014 |
self.index_entity(session, entity=entity) |
5059
1d5c81588144
[repo] make etype which should go in deleted_entities configurable: we only need this for types imported from other multi-sources instances
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5013
diff
changeset
|
1015 |
# update entities.mtime. |
8900
010a59e12d89
use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
8821
diff
changeset
|
1016 |
# XXX Only if entity.cw_etype in self.multisources_etypes? |
7922
d307c3817782
[repository] use utcnow instead of now for some internal timetamps (closes #1988458)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7879
diff
changeset
|
1017 |
attrs = {'eid': entity.eid, 'mtime': datetime.utcnow()} |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1018 |
self.doexec(session, self.sqlgen.update('entities', attrs, ['eid']), attrs) |
0 | 1019 |
|
7501
2983dd24494a
[repository] refactor/cleanup entity deletion methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
1020 |
def delete_info_multi(self, session, entities, uri): |
2983dd24494a
[repository] refactor/cleanup entity deletion methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
1021 |
"""delete system information on deletion of a list of entities with the |
2983dd24494a
[repository] refactor/cleanup entity deletion methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
1022 |
same etype and belinging to the same source |
2983dd24494a
[repository] refactor/cleanup entity deletion methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
1023 |
|
5067
adc2122eed03
[repo] more efficient eid cache operations handling based on set_operation; refactor
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5061
diff
changeset
|
1024 |
* update the fti |
7501
2983dd24494a
[repository] refactor/cleanup entity deletion methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
1025 |
* remove record from the `entities` table |
2983dd24494a
[repository] refactor/cleanup entity deletion methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
1026 |
* transfer it to the `deleted_entities` |
6889
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
1027 |
""" |
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
1028 |
self.fti_unindex_entities(session, entities) |
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
1029 |
attrs = {'eid': '(%s)' % ','.join([str(_e.eid) for _e in entities])} |
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
1030 |
self.doexec(session, self.sqlgen.delete_many('entities', attrs), attrs) |
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
1031 |
if entities[0].__regid__ not in self.multisources_etypes: |
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
1032 |
return |
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
1033 |
attrs = {'type': entities[0].__regid__, |
7922
d307c3817782
[repository] use utcnow instead of now for some internal timetamps (closes #1988458)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7879
diff
changeset
|
1034 |
'source': uri, 'dtime': datetime.utcnow()} |
7501
2983dd24494a
[repository] refactor/cleanup entity deletion methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
1035 |
for entity in entities: |
2983dd24494a
[repository] refactor/cleanup entity deletion methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
1036 |
extid = entity.cw_metainformation()['extid'] |
6889
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
1037 |
if extid is not None: |
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
1038 |
assert isinstance(extid, str), type(extid) |
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
1039 |
extid = b64encode(extid) |
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
1040 |
attrs.update({'eid': entity.eid, 'extid': extid}) |
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
1041 |
self.doexec(session, self.sqlgen.insert('deleted_entities', attrs), attrs) |
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
1042 |
|
0 | 1043 |
def modified_entities(self, session, etypes, mtime): |
1044 |
"""return a 2-uple: |
|
1045 |
* list of (etype, eid) of entities of the given types which have been |
|
1046 |
modified since the given timestamp (actually entities whose full text |
|
1047 |
index content has changed) |
|
1048 |
* list of (etype, eid) of entities of the given types which have been |
|
1049 |
deleted since the given timestamp |
|
1050 |
""" |
|
5059
1d5c81588144
[repo] make etype which should go in deleted_entities configurable: we only need this for types imported from other multi-sources instances
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5013
diff
changeset
|
1051 |
for etype in etypes: |
1d5c81588144
[repo] make etype which should go in deleted_entities configurable: we only need this for types imported from other multi-sources instances
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5013
diff
changeset
|
1052 |
if not etype in self.multisources_etypes: |
6427
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6426
diff
changeset
|
1053 |
self.error('%s not listed as a multi-sources entity types. ' |
5072
072ae171aeb0
[cleanup] style fixes, add nodes, 0.2 cents refactorings
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5071
diff
changeset
|
1054 |
'Modify your configuration' % etype) |
5059
1d5c81588144
[repo] make etype which should go in deleted_entities configurable: we only need this for types imported from other multi-sources instances
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5013
diff
changeset
|
1055 |
self.multisources_etypes.add(etype) |
0 | 1056 |
modsql = _modified_sql('entities', etypes) |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1057 |
cursor = self.doexec(session, modsql, {'time': mtime}) |
0 | 1058 |
modentities = cursor.fetchall() |
1059 |
delsql = _modified_sql('deleted_entities', etypes) |
|
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1060 |
cursor = self.doexec(session, delsql, {'time': mtime}) |
0 | 1061 |
delentities = cursor.fetchall() |
1062 |
return modentities, delentities |
|
1063 |
||
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1064 |
# undo support ############################################################# |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1065 |
|
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1066 |
def undoable_transactions(self, session, ueid=None, **actionfilters): |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1067 |
"""See :class:`cubicweb.dbapi.Connection.undoable_transactions`""" |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1068 |
# force filtering to session's user if not a manager |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1069 |
if not session.user.is_in_group('managers'): |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1070 |
ueid = session.user.eid |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1071 |
restr = {} |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1072 |
if ueid is not None: |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1073 |
restr['tx_user'] = ueid |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1074 |
sql = self.sqlgen.select('transactions', restr, ('tx_uuid', 'tx_time', 'tx_user')) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1075 |
if actionfilters: |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1076 |
# we will need subqueries to filter transactions according to |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1077 |
# actions done |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1078 |
tearestr = {} # filters on the tx_entity_actions table |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1079 |
trarestr = {} # filters on the tx_relation_actions table |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1080 |
genrestr = {} # generic filters, appliyable to both table |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1081 |
# unless public explicitly set to false, we only consider public |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1082 |
# actions |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1083 |
if actionfilters.pop('public', True): |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1084 |
genrestr['txa_public'] = True |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1085 |
# put additional filters in trarestr and/or tearestr |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1086 |
for key, val in actionfilters.iteritems(): |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1087 |
if key == 'etype': |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1088 |
# filtering on etype implies filtering on entity actions |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1089 |
# only, and with no eid specified |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1090 |
assert actionfilters.get('action', 'C') in 'CUD' |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1091 |
assert not 'eid' in actionfilters |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1092 |
tearestr['etype'] = val |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1093 |
elif key == 'eid': |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1094 |
# eid filter may apply to 'eid' of tx_entity_actions or to |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1095 |
# 'eid_from' OR 'eid_to' of tx_relation_actions |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1096 |
if actionfilters.get('action', 'C') in 'CUD': |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1097 |
tearestr['eid'] = val |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1098 |
if actionfilters.get('action', 'A') in 'AR': |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1099 |
trarestr['eid_from'] = val |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1100 |
trarestr['eid_to'] = val |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1101 |
elif key == 'action': |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1102 |
if val in 'CUD': |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1103 |
tearestr['txa_action'] = val |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1104 |
else: |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1105 |
assert val in 'AR' |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1106 |
trarestr['txa_action'] = val |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1107 |
else: |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1108 |
raise AssertionError('unknow filter %s' % key) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1109 |
assert trarestr or tearestr, "can't only filter on 'public'" |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1110 |
subqsqls = [] |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1111 |
# append subqueries to the original query, using EXISTS() |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1112 |
if trarestr or (genrestr and not tearestr): |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1113 |
trarestr.update(genrestr) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1114 |
trasql = self.sqlgen.select('tx_relation_actions', trarestr, ('1',)) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1115 |
if 'eid_from' in trarestr: |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1116 |
# replace AND by OR between eid_from/eid_to restriction |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1117 |
trasql = sql_or_clauses(trasql, ['eid_from = %(eid_from)s', |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1118 |
'eid_to = %(eid_to)s']) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1119 |
trasql += ' AND transactions.tx_uuid=tx_relation_actions.tx_uuid' |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1120 |
subqsqls.append('EXISTS(%s)' % trasql) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1121 |
if tearestr or (genrestr and not trarestr): |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1122 |
tearestr.update(genrestr) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1123 |
teasql = self.sqlgen.select('tx_entity_actions', tearestr, ('1',)) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1124 |
teasql += ' AND transactions.tx_uuid=tx_entity_actions.tx_uuid' |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1125 |
subqsqls.append('EXISTS(%s)' % teasql) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1126 |
if restr: |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1127 |
sql += ' AND %s' % ' OR '.join(subqsqls) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1128 |
else: |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1129 |
sql += ' WHERE %s' % ' OR '.join(subqsqls) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1130 |
restr.update(trarestr) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1131 |
restr.update(tearestr) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1132 |
# we want results ordered by transaction's time descendant |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1133 |
sql += ' ORDER BY tx_time DESC' |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1134 |
cu = self.doexec(session, sql, restr) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1135 |
# turn results into transaction objects |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1136 |
return [tx.Transaction(*args) for args in cu.fetchall()] |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1137 |
|
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1138 |
def tx_info(self, session, txuuid): |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1139 |
"""See :class:`cubicweb.dbapi.Connection.transaction_info`""" |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1140 |
return tx.Transaction(txuuid, *self._tx_info(session, txuuid)) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1141 |
|
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1142 |
def tx_actions(self, session, txuuid, public): |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1143 |
"""See :class:`cubicweb.dbapi.Connection.transaction_actions`""" |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1144 |
self._tx_info(session, txuuid) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1145 |
restr = {'tx_uuid': txuuid} |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1146 |
if public: |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1147 |
restr['txa_public'] = True |
5072
072ae171aeb0
[cleanup] style fixes, add nodes, 0.2 cents refactorings
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5071
diff
changeset
|
1148 |
# XXX use generator to avoid loading everything in memory? |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1149 |
sql = self.sqlgen.select('tx_entity_actions', restr, |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1150 |
('txa_action', 'txa_public', 'txa_order', |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1151 |
'etype', 'eid', 'changes')) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1152 |
cu = self.doexec(session, sql, restr) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1153 |
actions = [tx.EntityAction(a,p,o,et,e,c and loads(self.binary_to_str(c))) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1154 |
for a,p,o,et,e,c in cu.fetchall()] |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1155 |
sql = self.sqlgen.select('tx_relation_actions', restr, |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1156 |
('txa_action', 'txa_public', 'txa_order', |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1157 |
'rtype', 'eid_from', 'eid_to')) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1158 |
cu = self.doexec(session, sql, restr) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1159 |
actions += [tx.RelationAction(*args) for args in cu.fetchall()] |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1160 |
return sorted(actions, key=lambda x: x.order) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1161 |
|
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1162 |
def undo_transaction(self, session, txuuid): |
5071
8631bb9f6e73
[undo] during undoing, call hooks in the [active]integrity and undo categories
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5070
diff
changeset
|
1163 |
"""See :class:`cubicweb.dbapi.Connection.undo_transaction` |
8631bb9f6e73
[undo] during undoing, call hooks in the [active]integrity and undo categories
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5070
diff
changeset
|
1164 |
|
8631bb9f6e73
[undo] during undoing, call hooks in the [active]integrity and undo categories
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5070
diff
changeset
|
1165 |
important note: while undoing of a transaction, only hooks in the |
8631bb9f6e73
[undo] during undoing, call hooks in the [active]integrity and undo categories
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5070
diff
changeset
|
1166 |
'integrity', 'activeintegrity' and 'undo' categories are called. |
8631bb9f6e73
[undo] during undoing, call hooks in the [active]integrity and undo categories
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5070
diff
changeset
|
1167 |
""" |
7398
26695dd703d8
[repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
1168 |
# set mode so connections set isn't released subsquently until commit/rollback |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1169 |
session.mode = 'write' |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1170 |
errors = [] |
5071
8631bb9f6e73
[undo] during undoing, call hooks in the [active]integrity and undo categories
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5070
diff
changeset
|
1171 |
session.transaction_data['undoing_uuid'] = txuuid |
8265
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1172 |
with session.deny_all_hooks_but('integrity', 'activeintegrity', 'undo'): |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1173 |
with session.security_enabled(read=False): |
4943
7f5b83578fec
disable security when undoing
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4913
diff
changeset
|
1174 |
for action in reversed(self.tx_actions(session, txuuid, False)): |
7f5b83578fec
disable security when undoing
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4913
diff
changeset
|
1175 |
undomethod = getattr(self, '_undo_%s' % action.action.lower()) |
7f5b83578fec
disable security when undoing
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4913
diff
changeset
|
1176 |
errors += undomethod(session, action) |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1177 |
# remove the transactions record |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1178 |
self.doexec(session, |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1179 |
"DELETE FROM transactions WHERE tx_uuid='%s'" % txuuid) |
8265
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1180 |
if errors: |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1181 |
raise UndoTransactionException(txuuid, errors) |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1182 |
else: |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1183 |
return |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1184 |
|
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1185 |
def start_undoable_transaction(self, session, uuid): |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1186 |
"""session callback to insert a transaction record in the transactions |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1187 |
table when some undoable transaction is started |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1188 |
""" |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1189 |
ueid = session.user.eid |
7922
d307c3817782
[repository] use utcnow instead of now for some internal timetamps (closes #1988458)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7879
diff
changeset
|
1190 |
attrs = {'tx_uuid': uuid, 'tx_user': ueid, 'tx_time': datetime.utcnow()} |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1191 |
self.doexec(session, self.sqlgen.insert('transactions', attrs), attrs) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1192 |
|
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1193 |
def _save_attrs(self, session, entity, attrs): |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1194 |
"""return a pickleable dictionary containing current values for given |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1195 |
attributes of the entity |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1196 |
""" |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1197 |
restr = {'cw_eid': entity.eid} |
8900
010a59e12d89
use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
8821
diff
changeset
|
1198 |
sql = self.sqlgen.select(SQL_PREFIX + entity.cw_etype, restr, attrs) |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1199 |
cu = self.doexec(session, sql, restr) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1200 |
values = dict(zip(attrs, cu.fetchone())) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1201 |
# ensure backend specific binary are converted back to string |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1202 |
eschema = entity.e_schema |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1203 |
for column in attrs: |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1204 |
# [3:] remove 'cw_' prefix |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1205 |
attr = column[3:] |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1206 |
if not eschema.subjrels[attr].final: |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1207 |
continue |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1208 |
if eschema.destination(attr) in ('Password', 'Bytes'): |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1209 |
value = values[column] |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1210 |
if value is not None: |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1211 |
values[column] = self.binary_to_str(value) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1212 |
return values |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1213 |
|
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1214 |
def _record_tx_action(self, session, table, action, **kwargs): |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1215 |
"""record a transaction action in the given table (either |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1216 |
'tx_entity_actions' or 'tx_relation_action') |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1217 |
""" |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1218 |
kwargs['tx_uuid'] = session.transaction_uuid() |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1219 |
kwargs['txa_action'] = action |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1220 |
kwargs['txa_order'] = session.transaction_inc_action_counter() |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1221 |
kwargs['txa_public'] = session.running_dbapi_query |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1222 |
self.doexec(session, self.sqlgen.insert(table, kwargs), kwargs) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1223 |
|
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1224 |
def _tx_info(self, session, txuuid): |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1225 |
"""return transaction's time and user of the transaction with the given uuid. |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1226 |
|
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1227 |
raise `NoSuchTransaction` if there is no such transaction of if the |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1228 |
session's user isn't allowed to see it. |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1229 |
""" |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1230 |
restr = {'tx_uuid': txuuid} |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1231 |
sql = self.sqlgen.select('transactions', restr, ('tx_time', 'tx_user')) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1232 |
cu = self.doexec(session, sql, restr) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1233 |
try: |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1234 |
time, ueid = cu.fetchone() |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1235 |
except TypeError: |
8265
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1236 |
raise tx.NoSuchTransaction(txuuid) |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1237 |
if not (session.user.is_in_group('managers') |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1238 |
or session.user.eid == ueid): |
8265
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1239 |
raise tx.NoSuchTransaction(txuuid) |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1240 |
return time, ueid |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1241 |
|
8265
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1242 |
def _reedit_entity(self, entity, changes, err): |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1243 |
session = entity._cw |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1244 |
eid = entity.eid |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1245 |
entity.cw_edited = edited = EditedEntity(entity) |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1246 |
# check for schema changes, entities linked through inlined relation |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1247 |
# still exists, rewrap binary values |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1248 |
eschema = entity.e_schema |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1249 |
getrschema = eschema.subjrels |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1250 |
for column, value in changes.items(): |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1251 |
rtype = column[len(SQL_PREFIX):] |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1252 |
if rtype == "eid": |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1253 |
continue # XXX should even `eid` be stored in action changes? |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1254 |
try: |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1255 |
rschema = getrschema[rtype] |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1256 |
except KeyError: |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1257 |
err(session._("can't restore relation %(rtype)s of entity %(eid)s, " |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1258 |
"this relation does not exist in the schema anymore.") |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1259 |
% {'rtype': rtype, 'eid': eid}) |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1260 |
if not rschema.final: |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1261 |
if not rschema.inlined: |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1262 |
assert value is None |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1263 |
# rschema is an inlined relation |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1264 |
elif value is not None: |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1265 |
# not a deletion: we must put something in edited |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1266 |
try: |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1267 |
entity._cw.entity_from_eid(value) # check target exists |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1268 |
edited[rtype] = value |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1269 |
except UnknownEid: |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1270 |
err(session._("can't restore entity %(eid)s of type %(eschema)s, " |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1271 |
"target of %(rtype)s (eid %(value)s) does not exist any longer") |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1272 |
% locals()) |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1273 |
elif eschema.destination(rtype) in ('Bytes', 'Password'): |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1274 |
changes[column] = self._binary(value) |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1275 |
edited[rtype] = Binary(value) |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1276 |
elif isinstance(value, str): |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1277 |
edited[rtype] = unicode(value, session.encoding, 'replace') |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1278 |
else: |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1279 |
edited[rtype] = value |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1280 |
# This must only be done after init_entitiy_caches : defered in calling functions |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1281 |
# edited.check() |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1282 |
|
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1283 |
def _undo_d(self, session, action): |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1284 |
"""undo an entity deletion""" |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1285 |
errors = [] |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1286 |
err = errors.append |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1287 |
eid = action.eid |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1288 |
etype = action.etype |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1289 |
_ = session._ |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1290 |
# get an entity instance |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1291 |
try: |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1292 |
entity = self.repo.vreg['etypes'].etype_class(etype)(session) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1293 |
except Exception: |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1294 |
err("can't restore entity %s of type %s, type no more supported" |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1295 |
% (eid, etype)) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1296 |
return errors |
8265
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1297 |
self._reedit_entity(entity, action.changes, err) |
5557
1a534c596bff
[entity] continue cleanup of Entity/AnyEntity namespace
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5556
diff
changeset
|
1298 |
entity.eid = eid |
5075
a4b735e76c66
[undo] init entity cache when undoing an entity deletion
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5074
diff
changeset
|
1299 |
session.repo.init_entity_caches(session, entity, self) |
8265
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1300 |
entity.cw_edited.check() |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1301 |
self.repo.hm.call_hooks('before_add_entity', session, entity=entity) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1302 |
# restore the entity |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1303 |
action.changes['cw_eid'] = eid |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1304 |
sql = self.sqlgen.insert(SQL_PREFIX + etype, action.changes) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1305 |
self.doexec(session, sql, action.changes) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1306 |
# restore record in entities (will update fti if needed) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1307 |
self.add_info(session, entity, self, None, True) |
5059
1d5c81588144
[repo] make etype which should go in deleted_entities configurable: we only need this for types imported from other multi-sources instances
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5013
diff
changeset
|
1308 |
# remove record from deleted_entities if entity's type is multi-sources |
8900
010a59e12d89
use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
8821
diff
changeset
|
1309 |
if entity.cw_etype in self.multisources_etypes: |
5059
1d5c81588144
[repo] make etype which should go in deleted_entities configurable: we only need this for types imported from other multi-sources instances
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5013
diff
changeset
|
1310 |
self.doexec(session, |
1d5c81588144
[repo] make etype which should go in deleted_entities configurable: we only need this for types imported from other multi-sources instances
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5013
diff
changeset
|
1311 |
'DELETE FROM deleted_entities WHERE eid=%s' % eid) |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1312 |
self.repo.hm.call_hooks('after_add_entity', session, entity=entity) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1313 |
return errors |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1314 |
|
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1315 |
def _undo_r(self, session, action): |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1316 |
"""undo a relation removal""" |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1317 |
errors = [] |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1318 |
subj, rtype, obj = action.eid_from, action.rtype, action.eid_to |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1319 |
try: |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1320 |
sentity, oentity, rdef = _undo_rel_info(session, subj, rtype, obj) |
8695
358d8bed9626
[toward-py3k] rewrite to "except AnException as exc:" (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents:
8694
diff
changeset
|
1321 |
except _UndoException as ex: |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1322 |
errors.append(unicode(ex)) |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1323 |
else: |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1324 |
for role, entity in (('subject', sentity), |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1325 |
('object', oentity)): |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1326 |
try: |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1327 |
_undo_check_relation_target(entity, rdef, role) |
8695
358d8bed9626
[toward-py3k] rewrite to "except AnException as exc:" (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents:
8694
diff
changeset
|
1328 |
except _UndoException as ex: |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1329 |
errors.append(unicode(ex)) |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1330 |
continue |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1331 |
if not errors: |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1332 |
self.repo.hm.call_hooks('before_add_relation', session, |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1333 |
eidfrom=subj, rtype=rtype, eidto=obj) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1334 |
# add relation in the database |
7243
9ab01bf84eac
fix bad handling of inlined relations when entities of different etypes are passed as subject to add_relations
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7237
diff
changeset
|
1335 |
self._add_relations(session, rtype, [(subj, obj)], rdef.rtype.inlined) |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1336 |
# set related cache |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1337 |
session.update_rel_cache_add(subj, rtype, obj, rdef.rtype.symmetric) |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1338 |
self.repo.hm.call_hooks('after_add_relation', session, |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1339 |
eidfrom=subj, rtype=rtype, eidto=obj) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1340 |
return errors |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1341 |
|
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1342 |
def _undo_c(self, session, action): |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1343 |
"""undo an entity creation""" |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1344 |
eid = action.eid |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1345 |
# XXX done to avoid fetching all remaining relation for the entity |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1346 |
# we should find an efficient way to do this (keeping current veolidf |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1347 |
# massive deletion performance) |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1348 |
if _undo_has_later_transaction(session, eid): |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1349 |
msg = session._('some later transaction(s) touch entity, undo them ' |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1350 |
'first') |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1351 |
raise ValidationError(eid, {None: msg}) |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1352 |
etype = action.etype |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1353 |
# get an entity instance |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1354 |
try: |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1355 |
entity = self.repo.vreg['etypes'].etype_class(etype)(session) |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1356 |
except Exception: |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1357 |
return [session._( |
5098
32b1adfb6b92
[i18n] use named substitution to avoid gettext warnings
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5076
diff
changeset
|
1358 |
"Can't undo creation of entity %(eid)s of type %(etype)s, type " |
32b1adfb6b92
[i18n] use named substitution to avoid gettext warnings
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5076
diff
changeset
|
1359 |
"no more supported" % {'eid': eid, 'etype': etype})] |
5557
1a534c596bff
[entity] continue cleanup of Entity/AnyEntity namespace
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5556
diff
changeset
|
1360 |
entity.eid = eid |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1361 |
# for proper eid/type cache update |
6426
541659c39f6a
[hook/operation] nicer api to achieve same result as set_operation, as described in #1253630
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6366
diff
changeset
|
1362 |
CleanupDeletedEidsCacheOp.get_instance(session).add_data(eid) |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1363 |
self.repo.hm.call_hooks('before_delete_entity', session, entity=entity) |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1364 |
# remove is / is_instance_of which are added using sql by hooks, hence |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1365 |
# unvisible as transaction action |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1366 |
self.doexec(session, 'DELETE FROM is_relation WHERE eid_from=%s' % eid) |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1367 |
self.doexec(session, 'DELETE FROM is_instance_of_relation WHERE eid_from=%s' % eid) |
6427
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6426
diff
changeset
|
1368 |
self.doexec(session, 'DELETE FROM cw_source_relation WHERE eid_from=%s' % self.eid) |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1369 |
# XXX check removal of inlined relation? |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1370 |
# delete the entity |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1371 |
attrs = {'cw_eid': eid} |
8900
010a59e12d89
use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
8821
diff
changeset
|
1372 |
sql = self.sqlgen.delete(SQL_PREFIX + entity.cw_etype, attrs) |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1373 |
self.doexec(session, sql, attrs) |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1374 |
# remove record from entities (will update fti if needed) |
7507
4c043afb104a
fix failures introduced by recent refactoring
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7501
diff
changeset
|
1375 |
self.delete_info_multi(session, [entity], self.uri) |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1376 |
self.repo.hm.call_hooks('after_delete_entity', session, entity=entity) |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1377 |
return () |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1378 |
|
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1379 |
def _undo_u(self, session, action): |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1380 |
"""undo an entity update""" |
8265
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1381 |
errors = [] |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1382 |
err = errors.append |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1383 |
try: |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1384 |
entity = session.entity_from_eid(action.eid) |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1385 |
except UnknownEid: |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1386 |
err(session._("can't restore state of entity %s, it has been " |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1387 |
"deleted inbetween") % action.eid) |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1388 |
return errors |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1389 |
self._reedit_entity(entity, action.changes, err) |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1390 |
entity.cw_edited.check() |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1391 |
self.repo.hm.call_hooks('before_update_entity', session, entity=entity) |
8900
010a59e12d89
use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
8821
diff
changeset
|
1392 |
sql = self.sqlgen.update(SQL_PREFIX + entity.cw_etype, action.changes, |
8265
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1393 |
['cw_eid']) |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1394 |
self.doexec(session, sql, action.changes) |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1395 |
self.repo.hm.call_hooks('after_update_entity', session, entity=entity) |
9747ab9230ad
[repo, undo] Finish repository-side implementation of the undo feature (closes #893940)
Anthony Truchet <anthony.truchet@logilab.fr>
parents:
8235
diff
changeset
|
1396 |
return errors |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1397 |
|
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1398 |
def _undo_a(self, session, action): |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1399 |
"""undo a relation addition""" |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1400 |
errors = [] |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1401 |
subj, rtype, obj = action.eid_from, action.rtype, action.eid_to |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1402 |
try: |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1403 |
sentity, oentity, rdef = _undo_rel_info(session, subj, rtype, obj) |
8695
358d8bed9626
[toward-py3k] rewrite to "except AnException as exc:" (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents:
8694
diff
changeset
|
1404 |
except _UndoException as ex: |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1405 |
errors.append(unicode(ex)) |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1406 |
else: |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1407 |
rschema = rdef.rtype |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1408 |
if rschema.inlined: |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1409 |
sql = 'SELECT 1 FROM cw_%s WHERE cw_eid=%s and cw_%s=%s'\ |
8900
010a59e12d89
use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents:
8821
diff
changeset
|
1410 |
% (sentity.cw_etype, subj, rtype, obj) |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1411 |
else: |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1412 |
sql = 'SELECT 1 FROM %s_relation WHERE eid_from=%s and eid_to=%s'\ |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1413 |
% (rtype, subj, obj) |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1414 |
cu = self.doexec(session, sql) |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1415 |
if cu.fetchone() is None: |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1416 |
errors.append(session._( |
5098
32b1adfb6b92
[i18n] use named substitution to avoid gettext warnings
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5076
diff
changeset
|
1417 |
"Can't undo addition of relation %(rtype)s from %(subj)s to" |
32b1adfb6b92
[i18n] use named substitution to avoid gettext warnings
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5076
diff
changeset
|
1418 |
" %(obj)s, doesn't exist anymore" % locals())) |
5076
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1419 |
if not errors: |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1420 |
self.repo.hm.call_hooks('before_delete_relation', session, |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1421 |
eidfrom=subj, rtype=rtype, eidto=obj) |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1422 |
# delete relation from the database |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1423 |
self._delete_relation(session, subj, rtype, obj, rschema.inlined) |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1424 |
# set related cache |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1425 |
session.update_rel_cache_del(subj, rtype, obj, rschema.symmetric) |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1426 |
self.repo.hm.call_hooks('after_delete_relation', session, |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1427 |
eidfrom=subj, rtype=rtype, eidto=obj) |
b0e6134b4324
[undo] basic support for undoing of entity creation / relation addition
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5075
diff
changeset
|
1428 |
return errors |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1429 |
|
4806
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1430 |
# full text index handling ################################################# |
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1431 |
|
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1432 |
@cached |
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1433 |
def need_fti_indexation(self, etype): |
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1434 |
eschema = self.schema.eschema(etype) |
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1435 |
if any(eschema.indexable_attributes()): |
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1436 |
return True |
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1437 |
if any(eschema.fulltext_containers()): |
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1438 |
return True |
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1439 |
return False |
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1440 |
|
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1441 |
def index_entity(self, session, entity): |
4807
5642bfa43236
[cleanup] add index_entity to abstract source, add docstring
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4806
diff
changeset
|
1442 |
"""create an operation to [re]index textual content of the given entity |
5642bfa43236
[cleanup] add index_entity to abstract source, add docstring
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4806
diff
changeset
|
1443 |
on commit |
5642bfa43236
[cleanup] add index_entity to abstract source, add docstring
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4806
diff
changeset
|
1444 |
""" |
6426
541659c39f6a
[hook/operation] nicer api to achieve same result as set_operation, as described in #1253630
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6366
diff
changeset
|
1445 |
FTIndexEntityOp.get_instance(session).add_data(entity.eid) |
4806
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1446 |
|
6889
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
1447 |
def fti_unindex_entities(self, session, entities): |
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
1448 |
"""remove text content for entities from the full text index |
4806
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1449 |
""" |
7398
26695dd703d8
[repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
1450 |
cursor = session.cnxset['system'] |
6889
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
1451 |
cursor_unindex_object = self.dbhelper.cursor_unindex_object |
4806
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1452 |
try: |
6889
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
1453 |
for entity in entities: |
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
1454 |
cursor_unindex_object(entity.eid, cursor) |
4806
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1455 |
except Exception: # let KeyboardInterrupt / SystemExit propagate |
6889
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
1456 |
self.exception('error while unindexing %s', entity) |
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
1457 |
|
4806
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1458 |
|
6889
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
1459 |
def fti_index_entities(self, session, entities): |
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
1460 |
"""add text content of created/modified entities to the full text index |
4806
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1461 |
""" |
6889
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
1462 |
cursor_index_object = self.dbhelper.cursor_index_object |
7398
26695dd703d8
[repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7342
diff
changeset
|
1463 |
cursor = session.cnxset['system'] |
4806
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1464 |
try: |
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1465 |
# use cursor_index_object, not cursor_reindex_object since |
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1466 |
# unindexing done in the FTIndexEntityOp |
6889
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
1467 |
for entity in entities: |
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
1468 |
cursor_index_object(entity.eid, |
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
1469 |
entity.cw_adapt_to('IFTIndexable'), |
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
1470 |
cursor) |
4806
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1471 |
except Exception: # let KeyboardInterrupt / SystemExit propagate |
6889
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
1472 |
self.exception('error while indexing %s', entity) |
4806
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1473 |
|
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1474 |
|
6426
541659c39f6a
[hook/operation] nicer api to achieve same result as set_operation, as described in #1253630
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6366
diff
changeset
|
1475 |
class FTIndexEntityOp(hook.DataOperationMixIn, hook.LateOperation): |
4806
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1476 |
"""operation to delay entity full text indexation to commit |
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1477 |
|
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1478 |
since fti indexing may trigger discovery of other entities, it should be |
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1479 |
triggered on precommit, not commit, and this should be done after other |
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1480 |
precommit operation which may add relations to the entity |
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1481 |
""" |
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1482 |
|
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1483 |
def precommit_event(self): |
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1484 |
session = self.session |
4f12f59b1a13
[fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4692
diff
changeset
|
1485 |
source = session.repo.system_source |
5070
b1f80ccadda3
[repo] refactor fti operation to use set_operation and a single operation whatever the number of entities to ft index
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5067
diff
changeset
|
1486 |
pendingeids = session.transaction_data.get('pendingeids', ()) |
b1f80ccadda3
[repo] refactor fti operation to use set_operation and a single operation whatever the number of entities to ft index
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5067
diff
changeset
|
1487 |
done = session.transaction_data.setdefault('indexedeids', set()) |
6889
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
1488 |
to_reindex = set() |
6426
541659c39f6a
[hook/operation] nicer api to achieve same result as set_operation, as described in #1253630
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6366
diff
changeset
|
1489 |
for eid in self.get_data(): |
5070
b1f80ccadda3
[repo] refactor fti operation to use set_operation and a single operation whatever the number of entities to ft index
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5067
diff
changeset
|
1490 |
if eid in pendingeids or eid in done: |
b1f80ccadda3
[repo] refactor fti operation to use set_operation and a single operation whatever the number of entities to ft index
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5067
diff
changeset
|
1491 |
# entity added and deleted in the same transaction or already |
b1f80ccadda3
[repo] refactor fti operation to use set_operation and a single operation whatever the number of entities to ft index
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5067
diff
changeset
|
1492 |
# processed |
6889
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
1493 |
continue |
5070
b1f80ccadda3
[repo] refactor fti operation to use set_operation and a single operation whatever the number of entities to ft index
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5067
diff
changeset
|
1494 |
done.add(eid) |
5556
9ab2b4c74baf
[entity] introduce a new 'adapters' registry
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5542
diff
changeset
|
1495 |
iftindexable = session.entity_from_eid(eid).cw_adapt_to('IFTIndexable') |
6889
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
1496 |
to_reindex |= set(iftindexable.fti_containers()) |
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
1497 |
source.fti_unindex_entities(session, to_reindex) |
37668bf302f5
improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
6850
diff
changeset
|
1498 |
source.fti_index_entities(session, to_reindex) |
0 | 1499 |
|
1500 |
def sql_schema(driver): |
|
4831
c5aec27c1bf7
[repo] use logilab.db instead of lgc.adbh/lgc.db/lgc.sqlgen/indexer, test new date extranction functions
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4829
diff
changeset
|
1501 |
helper = get_db_helper(driver) |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1502 |
typemap = helper.TYPE_MAPPING |
0 | 1503 |
schema = """ |
1504 |
/* Create the repository's system database */ |
|
1505 |
||
1506 |
%s |
|
1507 |
||
1508 |
CREATE TABLE entities ( |
|
1509 |
eid INTEGER PRIMARY KEY NOT NULL, |
|
1510 |
type VARCHAR(64) NOT NULL, |
|
8205
14c52c5ee706
[entities system table] change asource and source column size (closes #1951819)
Alexandre Richardson <alexandre.richardson@logilab.fr>
parents:
7967
diff
changeset
|
1511 |
source VARCHAR(128) NOT NULL, |
14c52c5ee706
[entities system table] change asource and source column size (closes #1951819)
Alexandre Richardson <alexandre.richardson@logilab.fr>
parents:
7967
diff
changeset
|
1512 |
asource VARCHAR(128) NOT NULL, |
4113
986fc01be83c
TIMESTAMP column type has a special meaning for SQLServer,
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
3958
diff
changeset
|
1513 |
mtime %s NOT NULL, |
0 | 1514 |
extid VARCHAR(256) |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1515 |
);; |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1516 |
CREATE INDEX entities_type_idx ON entities(type);; |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1517 |
CREATE INDEX entities_mtime_idx ON entities(mtime);; |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1518 |
CREATE INDEX entities_extid_idx ON entities(extid);; |
0 | 1519 |
|
1520 |
CREATE TABLE deleted_entities ( |
|
1521 |
eid INTEGER PRIMARY KEY NOT NULL, |
|
1522 |
type VARCHAR(64) NOT NULL, |
|
8205
14c52c5ee706
[entities system table] change asource and source column size (closes #1951819)
Alexandre Richardson <alexandre.richardson@logilab.fr>
parents:
7967
diff
changeset
|
1523 |
source VARCHAR(128) NOT NULL, |
4113
986fc01be83c
TIMESTAMP column type has a special meaning for SQLServer,
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
3958
diff
changeset
|
1524 |
dtime %s NOT NULL, |
0 | 1525 |
extid VARCHAR(256) |
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1526 |
);; |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1527 |
CREATE INDEX deleted_entities_type_idx ON deleted_entities(type);; |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1528 |
CREATE INDEX deleted_entities_dtime_idx ON deleted_entities(dtime);; |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1529 |
CREATE INDEX deleted_entities_extid_idx ON deleted_entities(extid);; |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1530 |
|
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1531 |
CREATE TABLE transactions ( |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1532 |
tx_uuid CHAR(32) PRIMARY KEY NOT NULL, |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1533 |
tx_user INTEGER NOT NULL, |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1534 |
tx_time %s NOT NULL |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1535 |
);; |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1536 |
CREATE INDEX transactions_tx_user_idx ON transactions(tx_user);; |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1537 |
|
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1538 |
CREATE TABLE tx_entity_actions ( |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1539 |
tx_uuid CHAR(32) REFERENCES transactions(tx_uuid) ON DELETE CASCADE, |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1540 |
txa_action CHAR(1) NOT NULL, |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1541 |
txa_public %s NOT NULL, |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1542 |
txa_order INTEGER, |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1543 |
eid INTEGER NOT NULL, |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1544 |
etype VARCHAR(64) NOT NULL, |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1545 |
changes %s |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1546 |
);; |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1547 |
CREATE INDEX tx_entity_actions_txa_action_idx ON tx_entity_actions(txa_action);; |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1548 |
CREATE INDEX tx_entity_actions_txa_public_idx ON tx_entity_actions(txa_public);; |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1549 |
CREATE INDEX tx_entity_actions_eid_idx ON tx_entity_actions(eid);; |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1550 |
CREATE INDEX tx_entity_actions_etype_idx ON tx_entity_actions(etype);; |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1551 |
|
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1552 |
CREATE TABLE tx_relation_actions ( |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1553 |
tx_uuid CHAR(32) REFERENCES transactions(tx_uuid) ON DELETE CASCADE, |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1554 |
txa_action CHAR(1) NOT NULL, |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1555 |
txa_public %s NOT NULL, |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1556 |
txa_order INTEGER, |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1557 |
eid_from INTEGER NOT NULL, |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1558 |
eid_to INTEGER NOT NULL, |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1559 |
rtype VARCHAR(256) NOT NULL |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1560 |
);; |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1561 |
CREATE INDEX tx_relation_actions_txa_action_idx ON tx_relation_actions(txa_action);; |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1562 |
CREATE INDEX tx_relation_actions_txa_public_idx ON tx_relation_actions(txa_public);; |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1563 |
CREATE INDEX tx_relation_actions_eid_from_idx ON tx_relation_actions(eid_from);; |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1564 |
CREATE INDEX tx_relation_actions_eid_to_idx ON tx_relation_actions(eid_to);; |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1565 |
""" % (helper.sql_create_sequence('entities_id_seq').replace(';', ';;'), |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1566 |
typemap['Datetime'], typemap['Datetime'], typemap['Datetime'], |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1567 |
typemap['Boolean'], typemap['Bytes'], typemap['Boolean']) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1568 |
if helper.backend_name == 'sqlite': |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1569 |
# sqlite support the ON DELETE CASCADE syntax but do nothing |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1570 |
schema += ''' |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1571 |
CREATE TRIGGER fkd_transactions |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1572 |
BEFORE DELETE ON transactions |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1573 |
FOR EACH ROW BEGIN |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1574 |
DELETE FROM tx_entity_actions WHERE tx_uuid=OLD.tx_uuid; |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1575 |
DELETE FROM tx_relation_actions WHERE tx_uuid=OLD.tx_uuid; |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1576 |
END;; |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1577 |
''' |
0 | 1578 |
return schema |
1579 |
||
1580 |
||
1581 |
def sql_drop_schema(driver): |
|
4831
c5aec27c1bf7
[repo] use logilab.db instead of lgc.adbh/lgc.db/lgc.sqlgen/indexer, test new date extranction functions
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4829
diff
changeset
|
1582 |
helper = get_db_helper(driver) |
0 | 1583 |
return """ |
1584 |
%s |
|
1585 |
DROP TABLE entities; |
|
1586 |
DROP TABLE deleted_entities; |
|
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1587 |
DROP TABLE tx_entity_actions; |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1588 |
DROP TABLE tx_relation_actions; |
5214
3285b6e3b930
fix cwctl db-init -d on SQL Server
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
5168
diff
changeset
|
1589 |
DROP TABLE transactions; |
0 | 1590 |
""" % helper.sql_drop_sequence('entities_id_seq') |
1591 |
||
1592 |
||
1593 |
def grant_schema(user, set_owner=True): |
|
1594 |
result = '' |
|
4913
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1595 |
for table in ('entities', 'deleted_entities', 'entities_id_seq', |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1596 |
'transactions', 'tx_entity_actions', 'tx_relation_actions'): |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1597 |
if set_owner: |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1598 |
result = 'ALTER TABLE %s OWNER TO %s;\n' % (table, user) |
083b4d454192
server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents:
4902
diff
changeset
|
1599 |
result += 'GRANT ALL ON %s TO %s;\n' % (table, user) |
0 | 1600 |
return result |
3647
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1601 |
|
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1602 |
|
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1603 |
class BaseAuthentifier(object): |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1604 |
|
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1605 |
def __init__(self, source=None): |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1606 |
self.source = source |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1607 |
|
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1608 |
def set_schema(self, schema): |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1609 |
"""set the instance'schema""" |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1610 |
pass |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1611 |
|
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1612 |
class LoginPasswordAuthentifier(BaseAuthentifier): |
8520
fcd048fa6e6d
[source/native] only system source users should be authenticated by the native source (closes #2465883)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
8508
diff
changeset
|
1613 |
passwd_rql = 'Any P WHERE X is CWUser, X login %(login)s, X upassword P' |
fcd048fa6e6d
[source/native] only system source users should be authenticated by the native source (closes #2465883)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
8508
diff
changeset
|
1614 |
auth_rql = ('Any X WHERE X is CWUser, X login %(login)s, X upassword %(pwd)s, ' |
fcd048fa6e6d
[source/native] only system source users should be authenticated by the native source (closes #2465883)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
8508
diff
changeset
|
1615 |
'X cw_source S, S name "system"') |
fcd048fa6e6d
[source/native] only system source users should be authenticated by the native source (closes #2465883)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
8508
diff
changeset
|
1616 |
_sols = ({'X': 'CWUser', 'P': 'Password', 'S': 'CWSource'},) |
3647
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1617 |
|
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1618 |
def set_schema(self, schema): |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1619 |
"""set the instance'schema""" |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1620 |
if 'CWUser' in schema: # probably an empty schema if not true... |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1621 |
# rql syntax trees used to authenticate users |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1622 |
self._passwd_rqlst = self.source.compile_rql(self.passwd_rql, self._sols) |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1623 |
self._auth_rqlst = self.source.compile_rql(self.auth_rql, self._sols) |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1624 |
|
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1625 |
def authenticate(self, session, login, password=None, **kwargs): |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1626 |
"""return CWUser eid for the given login/password if this account is |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1627 |
defined in this source, else raise `AuthenticationError` |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1628 |
|
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1629 |
two queries are needed since passwords are stored crypted, so we have |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1630 |
to fetch the salt first |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1631 |
""" |
6018
f4d1d5d9ccbb
[security] don't put uncrypted password in query parameters, else it may be logged on error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5975
diff
changeset
|
1632 |
args = {'login': login, 'pwd' : None} |
3647
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1633 |
if password is not None: |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1634 |
rset = self.source.syntax_tree_search(session, self._passwd_rqlst, args) |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1635 |
try: |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1636 |
pwd = rset[0][0] |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1637 |
except IndexError: |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1638 |
raise AuthenticationError('bad login') |
8235
c2a91d6639d8
[system source]Â fix authentication: don't allow login whatever the given password if user password is NULL in database. Closes #2186099
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
8205
diff
changeset
|
1639 |
if pwd is None: |
c2a91d6639d8
[system source]Â fix authentication: don't allow login whatever the given password if user password is NULL in database. Closes #2186099
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
8205
diff
changeset
|
1640 |
# if pwd is None but a password is provided, something is wrong |
c2a91d6639d8
[system source]Â fix authentication: don't allow login whatever the given password if user password is NULL in database. Closes #2186099
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
8205
diff
changeset
|
1641 |
raise AuthenticationError('bad password') |
3647
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1642 |
# passwords are stored using the Bytes type, so we get a StringIO |
8317
9c59258e7798
[security] use a stronger encryption algorythm for password, keeping bw compat
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
8235
diff
changeset
|
1643 |
args['pwd'] = Binary(crypt_password(password, pwd.getvalue())) |
3647
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1644 |
# get eid from login and (crypted) password |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1645 |
rset = self.source.syntax_tree_search(session, self._auth_rqlst, args) |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1646 |
try: |
8546
3d2038d6f20d
[sources/native] automatically update passwords using deprecated hashes on login
Julien Cristau <julien.cristau@logilab.fr>
parents:
8520
diff
changeset
|
1647 |
user = rset[0][0] |
3d2038d6f20d
[sources/native] automatically update passwords using deprecated hashes on login
Julien Cristau <julien.cristau@logilab.fr>
parents:
8520
diff
changeset
|
1648 |
# If the stored hash uses a deprecated scheme (e.g. DES or MD5 used |
3d2038d6f20d
[sources/native] automatically update passwords using deprecated hashes on login
Julien Cristau <julien.cristau@logilab.fr>
parents:
8520
diff
changeset
|
1649 |
# before 3.14.7), update with a fresh one |
3d2038d6f20d
[sources/native] automatically update passwords using deprecated hashes on login
Julien Cristau <julien.cristau@logilab.fr>
parents:
8520
diff
changeset
|
1650 |
if pwd.getvalue(): |
3d2038d6f20d
[sources/native] automatically update passwords using deprecated hashes on login
Julien Cristau <julien.cristau@logilab.fr>
parents:
8520
diff
changeset
|
1651 |
verify, newhash = verify_and_update(password, pwd.getvalue()) |
3d2038d6f20d
[sources/native] automatically update passwords using deprecated hashes on login
Julien Cristau <julien.cristau@logilab.fr>
parents:
8520
diff
changeset
|
1652 |
if not verify: # should not happen, but... |
3d2038d6f20d
[sources/native] automatically update passwords using deprecated hashes on login
Julien Cristau <julien.cristau@logilab.fr>
parents:
8520
diff
changeset
|
1653 |
raise AuthenticationError('bad password') |
3d2038d6f20d
[sources/native] automatically update passwords using deprecated hashes on login
Julien Cristau <julien.cristau@logilab.fr>
parents:
8520
diff
changeset
|
1654 |
if newhash: |
3d2038d6f20d
[sources/native] automatically update passwords using deprecated hashes on login
Julien Cristau <julien.cristau@logilab.fr>
parents:
8520
diff
changeset
|
1655 |
session.system_sql("UPDATE %s SET %s=%%(newhash)s WHERE %s=%%(login)s" % ( |
3d2038d6f20d
[sources/native] automatically update passwords using deprecated hashes on login
Julien Cristau <julien.cristau@logilab.fr>
parents:
8520
diff
changeset
|
1656 |
SQL_PREFIX + 'CWUser', |
3d2038d6f20d
[sources/native] automatically update passwords using deprecated hashes on login
Julien Cristau <julien.cristau@logilab.fr>
parents:
8520
diff
changeset
|
1657 |
SQL_PREFIX + 'upassword', |
3d2038d6f20d
[sources/native] automatically update passwords using deprecated hashes on login
Julien Cristau <julien.cristau@logilab.fr>
parents:
8520
diff
changeset
|
1658 |
SQL_PREFIX + 'login'), |
3d2038d6f20d
[sources/native] automatically update passwords using deprecated hashes on login
Julien Cristau <julien.cristau@logilab.fr>
parents:
8520
diff
changeset
|
1659 |
{'newhash': self.source._binary(newhash), |
3d2038d6f20d
[sources/native] automatically update passwords using deprecated hashes on login
Julien Cristau <julien.cristau@logilab.fr>
parents:
8520
diff
changeset
|
1660 |
'login': login}) |
3d2038d6f20d
[sources/native] automatically update passwords using deprecated hashes on login
Julien Cristau <julien.cristau@logilab.fr>
parents:
8520
diff
changeset
|
1661 |
session.commit(free_cnxset=False) |
3d2038d6f20d
[sources/native] automatically update passwords using deprecated hashes on login
Julien Cristau <julien.cristau@logilab.fr>
parents:
8520
diff
changeset
|
1662 |
return user |
3647
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1663 |
except IndexError: |
2941f4a0aab9
refactor repo authentication to allow pluggable authentifier to login with something else than a password
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3503
diff
changeset
|
1664 |
raise AuthenticationError('bad password') |
6849
5a0c2cfc19bf
[repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6724
diff
changeset
|
1665 |
|
5a0c2cfc19bf
[repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6724
diff
changeset
|
1666 |
|
5a0c2cfc19bf
[repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6724
diff
changeset
|
1667 |
class EmailPasswordAuthentifier(BaseAuthentifier): |
5a0c2cfc19bf
[repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6724
diff
changeset
|
1668 |
def authenticate(self, session, login, **authinfo): |
5a0c2cfc19bf
[repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6724
diff
changeset
|
1669 |
# email_auth flag prevent from infinite recursion (call to |
5a0c2cfc19bf
[repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6724
diff
changeset
|
1670 |
# repo.check_auth_info at the end of this method may lead us here again) |
5a0c2cfc19bf
[repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6724
diff
changeset
|
1671 |
if not '@' in login or authinfo.pop('email_auth', None): |
5a0c2cfc19bf
[repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6724
diff
changeset
|
1672 |
raise AuthenticationError('not an email') |
5a0c2cfc19bf
[repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6724
diff
changeset
|
1673 |
rset = session.execute('Any L WHERE U login L, U primary_email M, ' |
5a0c2cfc19bf
[repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6724
diff
changeset
|
1674 |
'M address %(login)s', {'login': login}, |
5a0c2cfc19bf
[repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6724
diff
changeset
|
1675 |
build_descr=False) |
5a0c2cfc19bf
[repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6724
diff
changeset
|
1676 |
if rset.rowcount != 1: |
5a0c2cfc19bf
[repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6724
diff
changeset
|
1677 |
raise AuthenticationError('unexisting email') |
5a0c2cfc19bf
[repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6724
diff
changeset
|
1678 |
login = rset.rows[0][0] |
5a0c2cfc19bf
[repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6724
diff
changeset
|
1679 |
authinfo['email_auth'] = True |
5a0c2cfc19bf
[repository auth] cleanup email login by turning it into a proper repo-side authentication plugin
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6724
diff
changeset
|
1680 |
return self.source.repo.check_auth_info(session, login, authinfo) |
7342
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1681 |
|
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1682 |
class DatabaseIndependentBackupRestore(object): |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1683 |
"""Helper class to perform db backend agnostic backup and restore |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1684 |
|
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1685 |
The backup and restore methods are used to dump / restore the |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1686 |
system database in a database independent format. The file is a |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1687 |
Zip archive containing the following files: |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1688 |
|
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1689 |
* format.txt: the format of the archive. Currently '1.0' |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1690 |
* tables.txt: list of filenames in the archive tables/ directory |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1691 |
* sequences.txt: list of filenames in the archive sequences/ directory |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1692 |
* versions.txt: the list of cube versions from CWProperty |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1693 |
* tables/<tablename>.<chunkno>: pickled data |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1694 |
* sequences/<sequencename>: pickled data |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1695 |
|
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1696 |
The pickled data format for tables and sequences is a tuple of 3 elements: |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1697 |
* the table name |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1698 |
* a tuple of column names |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1699 |
* a list of rows (as tuples with one element per column) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1700 |
|
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1701 |
Tables are saved in chunks in different files in order to prevent |
7551 | 1702 |
a too high memory consumption. |
7342
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1703 |
""" |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1704 |
def __init__(self, source): |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1705 |
""" |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1706 |
:param: source an instance of the system source |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1707 |
""" |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1708 |
self._source = source |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1709 |
self.logger = logging.getLogger('cubicweb.ctl') |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1710 |
self.logger.setLevel(logging.INFO) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1711 |
self.logger.addHandler(logging.StreamHandler(sys.stdout)) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1712 |
self.schema = self._source.schema |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1713 |
self.dbhelper = self._source.dbhelper |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1714 |
self.cnx = None |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1715 |
self.cursor = None |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1716 |
self.sql_generator = sqlgen.SQLGenerator() |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1717 |
|
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1718 |
def get_connection(self): |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1719 |
return self._source.get_connection() |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1720 |
|
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1721 |
def backup(self, backupfile): |
8910
7652c3d46ba3
[portable dump] skip virtual relations, they have no table associated. Closes #2841199
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
8821
diff
changeset
|
1722 |
archive = zipfile.ZipFile(backupfile, 'w', allowZip64=True) |
7342
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1723 |
self.cnx = self.get_connection() |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1724 |
try: |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1725 |
self.cursor = self.cnx.cursor() |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1726 |
self.cursor.arraysize=100 |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1727 |
self.logger.info('writing metadata') |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1728 |
self.write_metadata(archive) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1729 |
for seq in self.get_sequences(): |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1730 |
self.logger.info('processing sequence %s', seq) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1731 |
self.write_sequence(archive, seq) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1732 |
for table in self.get_tables(): |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1733 |
self.logger.info('processing table %s', table) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1734 |
self.write_table(archive, table) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1735 |
finally: |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1736 |
archive.close() |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1737 |
self.cnx.close() |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1738 |
self.logger.info('done') |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1739 |
|
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1740 |
def get_tables(self): |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1741 |
non_entity_tables = ['entities', |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1742 |
'deleted_entities', |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1743 |
'transactions', |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1744 |
'tx_entity_actions', |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1745 |
'tx_relation_actions', |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1746 |
] |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1747 |
etype_tables = [] |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1748 |
relation_tables = [] |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1749 |
prefix = 'cw_' |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1750 |
for etype in self.schema.entities(): |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1751 |
eschema = self.schema.eschema(etype) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1752 |
if eschema.final: |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1753 |
continue |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1754 |
etype_tables.append('%s%s'%(prefix, etype)) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1755 |
for rtype in self.schema.relations(): |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1756 |
rschema = self.schema.rschema(rtype) |
8910
7652c3d46ba3
[portable dump] skip virtual relations, they have no table associated. Closes #2841199
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
8821
diff
changeset
|
1757 |
if rschema.final or rschema.inlined or rschema in VIRTUAL_RTYPES: |
7342
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1758 |
continue |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1759 |
relation_tables.append('%s_relation' % rtype) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1760 |
return non_entity_tables + etype_tables + relation_tables |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1761 |
|
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1762 |
def get_sequences(self): |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1763 |
return ['entities_id_seq'] |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1764 |
|
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1765 |
def write_metadata(self, archive): |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1766 |
archive.writestr('format.txt', '1.0') |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1767 |
archive.writestr('tables.txt', '\n'.join(self.get_tables())) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1768 |
archive.writestr('sequences.txt', '\n'.join(self.get_sequences())) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1769 |
versions = self._get_versions() |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1770 |
versions_str = '\n'.join('%s %s' % (k,v) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1771 |
for k,v in versions) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1772 |
archive.writestr('versions.txt', versions_str) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1773 |
|
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1774 |
def write_sequence(self, archive, seq): |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1775 |
sql = self.dbhelper.sql_sequence_current_state(seq) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1776 |
columns, rows_iterator = self._get_cols_and_rows(sql) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1777 |
rows = list(rows_iterator) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1778 |
serialized = self._serialize(seq, columns, rows) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1779 |
archive.writestr('sequences/%s' % seq, serialized) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1780 |
|
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1781 |
def write_table(self, archive, table): |
7753
681ef2a664dd
work around cursor.rowcount not returning anything useful before 1st fetch with mssql (closes #1910869)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7551
diff
changeset
|
1782 |
nb_lines_sql = 'SELECT COUNT(*) FROM %s' % table |
681ef2a664dd
work around cursor.rowcount not returning anything useful before 1st fetch with mssql (closes #1910869)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7551
diff
changeset
|
1783 |
self.cursor.execute(nb_lines_sql) |
681ef2a664dd
work around cursor.rowcount not returning anything useful before 1st fetch with mssql (closes #1910869)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7551
diff
changeset
|
1784 |
rowcount = self.cursor.fetchone()[0] |
7342
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1785 |
sql = 'SELECT * FROM %s' % table |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1786 |
columns, rows_iterator = self._get_cols_and_rows(sql) |
7753
681ef2a664dd
work around cursor.rowcount not returning anything useful before 1st fetch with mssql (closes #1910869)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7551
diff
changeset
|
1787 |
self.logger.info('number of rows: %d', rowcount) |
7342
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1788 |
if table.startswith('cw_'): # entities |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1789 |
blocksize = 2000 |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1790 |
else: # relations and metadata |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1791 |
blocksize = 10000 |
7753
681ef2a664dd
work around cursor.rowcount not returning anything useful before 1st fetch with mssql (closes #1910869)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7551
diff
changeset
|
1792 |
if rowcount > 0: |
681ef2a664dd
work around cursor.rowcount not returning anything useful before 1st fetch with mssql (closes #1910869)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7551
diff
changeset
|
1793 |
for i, start in enumerate(xrange(0, rowcount, blocksize)): |
7342
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1794 |
rows = list(itertools.islice(rows_iterator, blocksize)) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1795 |
serialized = self._serialize(table, columns, rows) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1796 |
archive.writestr('tables/%s.%04d' % (table, i), serialized) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1797 |
self.logger.debug('wrote rows %d to %d (out of %d) to %s.%04d', |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1798 |
start, start+len(rows)-1, |
7753
681ef2a664dd
work around cursor.rowcount not returning anything useful before 1st fetch with mssql (closes #1910869)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
7551
diff
changeset
|
1799 |
rowcount, |
7342
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1800 |
table, i) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1801 |
else: |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1802 |
rows = [] |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1803 |
serialized = self._serialize(table, columns, rows) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1804 |
archive.writestr('tables/%s.%04d' % (table, 0), serialized) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1805 |
|
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1806 |
def _get_cols_and_rows(self, sql): |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1807 |
process_result = self._source.iter_process_result |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1808 |
self.cursor.execute(sql) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1809 |
columns = (d[0] for d in self.cursor.description) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1810 |
rows = process_result(self.cursor) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1811 |
return tuple(columns), rows |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1812 |
|
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1813 |
def _serialize(self, name, columns, rows): |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1814 |
return dumps((name, columns, rows), pickle.HIGHEST_PROTOCOL) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1815 |
|
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1816 |
def restore(self, backupfile): |
7787
d9607ae447d2
[server] portable dump format supports now ZIP64 extensions by default (closes #1912535)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7786
diff
changeset
|
1817 |
archive = zipfile.ZipFile(backupfile, 'r', allowZip64=True) |
7342
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1818 |
self.cnx = self.get_connection() |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1819 |
self.cursor = self.cnx.cursor() |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1820 |
sequences, tables, table_chunks = self.read_metadata(archive, backupfile) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1821 |
for seq in sequences: |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1822 |
self.logger.info('restoring sequence %s', seq) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1823 |
self.read_sequence(archive, seq) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1824 |
for table in tables: |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1825 |
self.logger.info('restoring table %s', table) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1826 |
self.read_table(archive, table, sorted(table_chunks[table])) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1827 |
self.cnx.close() |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1828 |
archive.close() |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1829 |
self.logger.info('done') |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1830 |
|
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1831 |
def read_metadata(self, archive, backupfile): |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1832 |
formatinfo = archive.read('format.txt') |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1833 |
self.logger.info('checking metadata') |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1834 |
if formatinfo.strip() != "1.0": |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1835 |
self.logger.critical('Unsupported format in archive: %s', formatinfo) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1836 |
raise ValueError('Unknown format in %s: %s' % (backupfile, formatinfo)) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1837 |
tables = archive.read('tables.txt').splitlines() |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1838 |
sequences = archive.read('sequences.txt').splitlines() |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1839 |
file_versions = self._parse_versions(archive.read('versions.txt')) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1840 |
versions = set(self._get_versions()) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1841 |
if file_versions != versions: |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1842 |
self.logger.critical('Unable to restore : versions do not match') |
8379
1a3746ec4d65
[sources/native] do not let a log message crash on us
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
8317
diff
changeset
|
1843 |
self.logger.critical('Expected:\n%s', '\n'.join('%s : %s' % (cube, ver) |
1a3746ec4d65
[sources/native] do not let a log message crash on us
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
8317
diff
changeset
|
1844 |
for cube, ver in sorted(versions))) |
1a3746ec4d65
[sources/native] do not let a log message crash on us
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
8317
diff
changeset
|
1845 |
self.logger.critical('Found:\n%s', '\n'.join('%s : %s' % (cube, ver) |
1a3746ec4d65
[sources/native] do not let a log message crash on us
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
8317
diff
changeset
|
1846 |
for cube, ver in sorted(file_versions))) |
7342
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1847 |
raise ValueError('Unable to restore : versions do not match') |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1848 |
table_chunks = {} |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1849 |
for name in archive.namelist(): |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1850 |
if not name.startswith('tables/'): |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1851 |
continue |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1852 |
filename = basename(name) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1853 |
tablename, _ext = filename.rsplit('.', 1) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1854 |
table_chunks.setdefault(tablename, []).append(name) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1855 |
return sequences, tables, table_chunks |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1856 |
|
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1857 |
def read_sequence(self, archive, seq): |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1858 |
seqname, columns, rows = loads(archive.read('sequences/%s' % seq)) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1859 |
assert seqname == seq |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1860 |
assert len(rows) == 1 |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1861 |
assert len(rows[0]) == 1 |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1862 |
value = rows[0][0] |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1863 |
sql = self.dbhelper.sql_restart_sequence(seq, value) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1864 |
self.cursor.execute(sql) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1865 |
self.cnx.commit() |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1866 |
|
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1867 |
def read_table(self, archive, table, filenames): |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1868 |
merge_args = self._source.merge_args |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1869 |
self.cursor.execute('DELETE FROM %s' % table) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1870 |
self.cnx.commit() |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1871 |
row_count = 0 |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1872 |
for filename in filenames: |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1873 |
tablename, columns, rows = loads(archive.read(filename)) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1874 |
assert tablename == table |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1875 |
if not rows: |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1876 |
continue |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1877 |
insert = self.sql_generator.insert(table, |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1878 |
dict(zip(columns, rows[0]))) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1879 |
for row in rows: |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1880 |
self.cursor.execute(insert, merge_args(dict(zip(columns, row)), {})) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1881 |
row_count += len(rows) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1882 |
self.cnx.commit() |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1883 |
self.logger.info('inserted %d rows', row_count) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1884 |
|
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1885 |
|
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1886 |
def _parse_versions(self, version_str): |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1887 |
versions = set() |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1888 |
for line in version_str.splitlines(): |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1889 |
versions.add(tuple(line.split())) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1890 |
return versions |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1891 |
|
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1892 |
def _get_versions(self): |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1893 |
version_sql = 'SELECT cw_pkey, cw_value FROM cw_CWProperty' |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1894 |
versions = [] |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1895 |
self.cursor.execute(version_sql) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1896 |
for pkey, value in self.cursor.fetchall(): |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1897 |
if pkey.startswith(u'system.version'): |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1898 |
versions.append((pkey, value)) |
d1c8b5b3531c
adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7243
diff
changeset
|
1899 |
return versions |