server/sources/native.py
author Sylvain Thénault <sylvain.thenault@logilab.fr>
Tue, 01 Dec 2009 19:29:56 +0100
branchstable
changeset 3958 505025eb0d37
parent 3835 a191b3b9e455
child 3998 94cc7cad3d2d
child 4113 986fc01be83c
permissions -rw-r--r--
fix name error while debugging
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
     1
"""Adapters for native cubicweb sources.
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
     2
1952
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1792
diff changeset
     3
Notes:
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1792
diff changeset
     4
* extid (aka external id, the primary key of an entity in the external source
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1792
diff changeset
     5
  from which it comes from) are stored in a varchar column encoded as a base64
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1792
diff changeset
     6
  string. This is because it should actually be Bytes but we want an index on
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1792
diff changeset
     7
  it for fast querying.
2056
57d287a2132a reminder
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1977
diff changeset
     8
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
     9
:organization: Logilab
1977
606923dff11b big bunch of copyright / docstring update
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 1954
diff changeset
    10
:copyright: 2001-2009 LOGILAB S.A. (Paris, FRANCE), license is LGPL v2.
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    11
:contact: http://www.logilab.fr/ -- mailto:contact@logilab.fr
1977
606923dff11b big bunch of copyright / docstring update
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 1954
diff changeset
    12
:license: GNU Lesser General Public License, v2.1 - http://www.gnu.org/licenses
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    13
"""
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    14
__docformat__ = "restructuredtext en"
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    15
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    16
from threading import Lock
1016
26387b836099 use datetime instead of mx.DateTime
sylvain.thenault@logilab.fr
parents: 973
diff changeset
    17
from datetime import datetime
1952
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1792
diff changeset
    18
from base64 import b64decode, b64encode
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    19
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    20
from logilab.common.cache import Cache
3835
a191b3b9e455 more sensible default values to c-c "create" inputs
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3689
diff changeset
    21
from logilab.common.configuration import Method
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    22
from logilab.common.adbh import get_adv_func_helper
3835
a191b3b9e455 more sensible default values to c-c "create" inputs
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3689
diff changeset
    23
from logilab.common.shellutils import getlogin
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    24
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    25
from indexer import get_indexer
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    26
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    27
from cubicweb import UnknownEid, AuthenticationError, Binary, server
3835
a191b3b9e455 more sensible default values to c-c "create" inputs
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3689
diff changeset
    28
from cubicweb.cwconfig import CubicWebNoAppConfiguration
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    29
from cubicweb.server.utils import crypt_password
2759
23d7a75693f8 R refactor backup and use tar.gz to store all sources
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 2638
diff changeset
    30
from cubicweb.server.sqlutils import SQL_PREFIX, SQLAdapterMixIn
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    31
from cubicweb.server.rqlannotation import set_qdata
2625
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2620
diff changeset
    32
from cubicweb.server.sources import AbstractSource, dbg_st_search, dbg_results
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    33
from cubicweb.server.sources.rql2sql import SQLGenerator
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    34
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    35
2354
9b4bac626977 ability to map attributes to something else than usual cw mapping on sql generation
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2310
diff changeset
    36
ATTR_MAP = {}
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    37
NONSYSTEM_ETYPES = set()
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    38
NONSYSTEM_RELATIONS = set()
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    39
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    40
class LogCursor(object):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    41
    def __init__(self, cursor):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    42
        self.cu = cursor
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
    43
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    44
    def execute(self, query, args=None):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    45
        """Execute a query.
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    46
        it's a function just so that it shows up in profiling
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    47
        """
2593
16d9419a4a79 F: start to handle binary debug log level on the server side
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2589
diff changeset
    48
        if server.DEBUG & server.DBG_SQL:
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    49
            print 'exec', query, args
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    50
        try:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    51
            self.cu.execute(str(query), args)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    52
        except Exception, ex:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    53
            print "sql: %r\n args: %s\ndbms message: %r" % (
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    54
                query, args, ex.args[0])
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    55
            raise
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
    56
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    57
    def fetchall(self):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    58
        return self.cu.fetchall()
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
    59
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    60
    def fetchone(self):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    61
        return self.cu.fetchone()
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
    62
2625
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2620
diff changeset
    63
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    64
def make_schema(selected, solution, table, typemap):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    65
    """return a sql schema to store RQL query result"""
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    66
    sql = []
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    67
    varmap = {}
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    68
    for i, term in enumerate(selected):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    69
        name = 'C%s' % i
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    70
        key = term.as_string()
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    71
        varmap[key] = '%s.%s' % (table, name)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    72
        ttype = term.get_type(solution)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    73
        try:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    74
            sql.append('%s %s' % (name, typemap[ttype]))
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    75
        except KeyError:
3689
deb13e88e037 follow yams 0.25 api changes to improve performance
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3497
diff changeset
    76
            # assert not schema(ttype).final
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    77
            sql.append('%s %s' % (name, typemap['Int']))
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    78
    return ','.join(sql), varmap
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    79
2625
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2620
diff changeset
    80
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    81
def _modified_sql(table, etypes):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    82
    # XXX protect against sql injection
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    83
    if len(etypes) > 1:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    84
        restr = 'type IN (%s)' % ','.join("'%s'" % etype for etype in etypes)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    85
    else:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    86
        restr = "type='%s'" % etypes[0]
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    87
    if table == 'entities':
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    88
        attr = 'mtime'
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    89
    else:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    90
        attr = 'dtime'
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    91
    return 'SELECT type, eid FROM %s WHERE %s AND %s > %%(time)s' % (
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    92
        table, restr, attr)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    93
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    94
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    95
class NativeSQLSource(SQLAdapterMixIn, AbstractSource):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    96
    """adapter for source using the native cubicweb schema (see below)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    97
    """
2354
9b4bac626977 ability to map attributes to something else than usual cw mapping on sql generation
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2310
diff changeset
    98
    sqlgen_class = SQLGenerator
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
    99
1398
5fe84a5f7035 rename internal entity types to have CW prefix instead of E
sylvain.thenault@logilab.fr
parents: 1263
diff changeset
   100
    passwd_rql = "Any P WHERE X is CWUser, X login %(login)s, X upassword P"
5fe84a5f7035 rename internal entity types to have CW prefix instead of E
sylvain.thenault@logilab.fr
parents: 1263
diff changeset
   101
    auth_rql = "Any X WHERE X is CWUser, X login %(login)s, X upassword %(pwd)s"
5fe84a5f7035 rename internal entity types to have CW prefix instead of E
sylvain.thenault@logilab.fr
parents: 1263
diff changeset
   102
    _sols = ({'X': 'CWUser', 'P': 'Password'},)
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   103
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   104
    options = (
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   105
        ('db-driver',
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   106
         {'type' : 'string',
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   107
          'default': 'postgres',
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   108
          'help': 'database driver (postgres or sqlite)',
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   109
          'group': 'native-source', 'inputlevel': 1,
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   110
          }),
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   111
        ('db-host',
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   112
         {'type' : 'string',
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   113
          'default': '',
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   114
          'help': 'database host',
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   115
          'group': 'native-source', 'inputlevel': 1,
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   116
          }),
2566
714a8743d423 missing db-port option to source's option definitions
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2354
diff changeset
   117
        ('db-port',
714a8743d423 missing db-port option to source's option definitions
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2354
diff changeset
   118
         {'type' : 'string',
714a8743d423 missing db-port option to source's option definitions
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2354
diff changeset
   119
          'default': '',
714a8743d423 missing db-port option to source's option definitions
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2354
diff changeset
   120
          'help': 'database port',
714a8743d423 missing db-port option to source's option definitions
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2354
diff changeset
   121
          'group': 'native-source', 'inputlevel': 1,
714a8743d423 missing db-port option to source's option definitions
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2354
diff changeset
   122
          }),
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   123
        ('db-name',
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   124
         {'type' : 'string',
3835
a191b3b9e455 more sensible default values to c-c "create" inputs
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3689
diff changeset
   125
          'default': Method('default_instance_id'),
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   126
          'help': 'database name',
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   127
          'group': 'native-source', 'inputlevel': 0,
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   128
          }),
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   129
        ('db-user',
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   130
         {'type' : 'string',
3835
a191b3b9e455 more sensible default values to c-c "create" inputs
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3689
diff changeset
   131
          'default': CubicWebNoAppConfiguration.mode == 'user' and getlogin() or 'cubicweb',
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   132
          'help': 'database user',
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   133
          'group': 'native-source', 'inputlevel': 0,
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   134
          }),
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   135
        ('db-password',
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   136
         {'type' : 'password',
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   137
          'default': '',
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   138
          'help': 'database password',
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   139
          'group': 'native-source', 'inputlevel': 0,
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   140
          }),
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   141
        ('db-encoding',
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   142
         {'type' : 'string',
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   143
          'default': 'utf8',
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   144
          'help': 'database encoding',
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   145
          'group': 'native-source', 'inputlevel': 1,
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   146
          }),
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   147
    )
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   148
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   149
    def __init__(self, repo, appschema, source_config, *args, **kwargs):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   150
        SQLAdapterMixIn.__init__(self, source_config)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   151
        AbstractSource.__init__(self, repo, appschema, source_config,
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   152
                                *args, **kwargs)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   153
        # sql generator
2354
9b4bac626977 ability to map attributes to something else than usual cw mapping on sql generation
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2310
diff changeset
   154
        self._rql_sqlgen = self.sqlgen_class(appschema, self.dbhelper,
9b4bac626977 ability to map attributes to something else than usual cw mapping on sql generation
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2310
diff changeset
   155
                                             self.encoding, ATTR_MAP.copy())
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   156
        # full text index helper
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   157
        self.indexer = get_indexer(self.dbdriver, self.encoding)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   158
        # advanced functionality helper
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   159
        self.dbhelper.fti_uid_attr = self.indexer.uid_attr
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   160
        self.dbhelper.fti_table = self.indexer.table
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   161
        self.dbhelper.fti_restriction_sql = self.indexer.restriction_sql
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   162
        self.dbhelper.fti_need_distinct_query = self.indexer.need_distinct
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   163
        # sql queries cache
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   164
        self._cache = Cache(repo.config['rql-cache-size'])
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   165
        self._temp_table_data = {}
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   166
        self._eid_creation_lock = Lock()
2072
8008e8812d76 deactivate sqlite connection wrapping for unittest_multisources for now
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2066
diff changeset
   167
        # XXX no_sqlite_wrap trick since we've a sqlite locking pb when
8008e8812d76 deactivate sqlite connection wrapping for unittest_multisources for now
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2066
diff changeset
   168
        # running unittest_multisources with the wrapping below
8008e8812d76 deactivate sqlite connection wrapping for unittest_multisources for now
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2066
diff changeset
   169
        if self.dbdriver == 'sqlite' and \
8008e8812d76 deactivate sqlite connection wrapping for unittest_multisources for now
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2066
diff changeset
   170
               not getattr(repo.config, 'no_sqlite_wrap', False):
2064
a5cd3a92314a properly call [re]set_pool, fix connection handling so we have a change to get cw running on top of a sqlite dabase with threads activated
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2056
diff changeset
   171
            from cubicweb.server.sources.extlite import ConnectionWrapper
a5cd3a92314a properly call [re]set_pool, fix connection handling so we have a change to get cw running on top of a sqlite dabase with threads activated
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2056
diff changeset
   172
            self.get_connection = lambda: ConnectionWrapper(self)
a5cd3a92314a properly call [re]set_pool, fix connection handling so we have a change to get cw running on top of a sqlite dabase with threads activated
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2056
diff changeset
   173
            self.check_connection = lambda cnx: cnx
a5cd3a92314a properly call [re]set_pool, fix connection handling so we have a change to get cw running on top of a sqlite dabase with threads activated
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2056
diff changeset
   174
            def pool_reset(cnx):
2620
de68f84b8f54 R [sql source] cnx._cnx checked in cnx.close, don't do it here
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2619
diff changeset
   175
                cnx.close()
2064
a5cd3a92314a properly call [re]set_pool, fix connection handling so we have a change to get cw running on top of a sqlite dabase with threads activated
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2056
diff changeset
   176
            self.pool_reset = pool_reset
a5cd3a92314a properly call [re]set_pool, fix connection handling so we have a change to get cw running on top of a sqlite dabase with threads activated
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2056
diff changeset
   177
a5cd3a92314a properly call [re]set_pool, fix connection handling so we have a change to get cw running on top of a sqlite dabase with threads activated
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2056
diff changeset
   178
    @property
a5cd3a92314a properly call [re]set_pool, fix connection handling so we have a change to get cw running on top of a sqlite dabase with threads activated
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2056
diff changeset
   179
    def _sqlcnx(self):
a5cd3a92314a properly call [re]set_pool, fix connection handling so we have a change to get cw running on top of a sqlite dabase with threads activated
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2056
diff changeset
   180
        # XXX: sqlite connections can only be used in the same thread, so
a5cd3a92314a properly call [re]set_pool, fix connection handling so we have a change to get cw running on top of a sqlite dabase with threads activated
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2056
diff changeset
   181
        #      create a new one each time necessary. If it appears to be time
a5cd3a92314a properly call [re]set_pool, fix connection handling so we have a change to get cw running on top of a sqlite dabase with threads activated
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2056
diff changeset
   182
        #      consuming, find another way
a5cd3a92314a properly call [re]set_pool, fix connection handling so we have a change to get cw running on top of a sqlite dabase with threads activated
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2056
diff changeset
   183
        return SQLAdapterMixIn.get_connection(self)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   184
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   185
    def reset_caches(self):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   186
        """method called during test to reset potential source caches"""
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   187
        self._cache = Cache(self.repo.config['rql-cache-size'])
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   188
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   189
    def clear_eid_cache(self, eid, etype):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   190
        """clear potential caches for the given eid"""
2610
2933cc6bf9ad [F native source] fix attempts to clear cache key
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2593
diff changeset
   191
        self._cache.pop('Any X WHERE X eid %s, X is %s' % (eid, etype), None)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   192
        self._cache.pop('Any X WHERE X eid %s' % eid, None)
2610
2933cc6bf9ad [F native source] fix attempts to clear cache key
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2593
diff changeset
   193
        self._cache.pop('Any %s' % eid, None)
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   194
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   195
    def sqlexec(self, session, sql, args=None):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   196
        """execute the query and return its result"""
2306
95da5d9f0870 give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2072
diff changeset
   197
        return self.process_result(self.doexec(session, sql, args))
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   198
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   199
    def init_creating(self):
2064
a5cd3a92314a properly call [re]set_pool, fix connection handling so we have a change to get cw running on top of a sqlite dabase with threads activated
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2056
diff changeset
   200
        pool = self.repo._get_pool()
a5cd3a92314a properly call [re]set_pool, fix connection handling so we have a change to get cw running on top of a sqlite dabase with threads activated
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2056
diff changeset
   201
        pool.pool_set()
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   202
        # check full text index availibility
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   203
        if not self.indexer.has_fti_table(pool['system']):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   204
            self.error('no text index table')
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   205
            self.indexer = None
2064
a5cd3a92314a properly call [re]set_pool, fix connection handling so we have a change to get cw running on top of a sqlite dabase with threads activated
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2056
diff changeset
   206
        pool.pool_reset()
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   207
        self.repo._free_pool(pool)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   208
2759
23d7a75693f8 R refactor backup and use tar.gz to store all sources
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 2638
diff changeset
   209
    def backup(self, backupfile):
23d7a75693f8 R refactor backup and use tar.gz to store all sources
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 2638
diff changeset
   210
        """method called to create a backup of the source's data"""
23d7a75693f8 R refactor backup and use tar.gz to store all sources
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 2638
diff changeset
   211
        self.close_pool_connections()
23d7a75693f8 R refactor backup and use tar.gz to store all sources
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 2638
diff changeset
   212
        try:
23d7a75693f8 R refactor backup and use tar.gz to store all sources
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 2638
diff changeset
   213
            self.backup_to_file(backupfile)
23d7a75693f8 R refactor backup and use tar.gz to store all sources
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 2638
diff changeset
   214
        finally:
23d7a75693f8 R refactor backup and use tar.gz to store all sources
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 2638
diff changeset
   215
            self.open_pool_connections()
2493
9806571ea790 major refactoring of database dump/restore:
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2476
diff changeset
   216
2759
23d7a75693f8 R refactor backup and use tar.gz to store all sources
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 2638
diff changeset
   217
    def restore(self, backupfile, drop):
2493
9806571ea790 major refactoring of database dump/restore:
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2476
diff changeset
   218
        """method called to restore a backup of source's data"""
2959
daabb9bc5233 make db-restore command work even with no/corrupted database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2759
diff changeset
   219
        if self.repo.config.open_connections_pools:
daabb9bc5233 make db-restore command work even with no/corrupted database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2759
diff changeset
   220
            self.close_pool_connections()
2759
23d7a75693f8 R refactor backup and use tar.gz to store all sources
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 2638
diff changeset
   221
        try:
23d7a75693f8 R refactor backup and use tar.gz to store all sources
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 2638
diff changeset
   222
            self.restore_from_file(backupfile, drop)
23d7a75693f8 R refactor backup and use tar.gz to store all sources
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 2638
diff changeset
   223
        finally:
2959
daabb9bc5233 make db-restore command work even with no/corrupted database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2759
diff changeset
   224
            if self.repo.config.open_connections_pools:
daabb9bc5233 make db-restore command work even with no/corrupted database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2759
diff changeset
   225
                self.open_pool_connections()
2493
9806571ea790 major refactoring of database dump/restore:
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2476
diff changeset
   226
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   227
    def init(self):
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   228
        self.init_creating()
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   229
2354
9b4bac626977 ability to map attributes to something else than usual cw mapping on sql generation
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2310
diff changeset
   230
    def map_attribute(self, etype, attr, cb):
9b4bac626977 ability to map attributes to something else than usual cw mapping on sql generation
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2310
diff changeset
   231
        self._rql_sqlgen.attr_map['%s.%s' % (etype, attr)] = cb
2493
9806571ea790 major refactoring of database dump/restore:
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2476
diff changeset
   232
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   233
    # ISource interface #######################################################
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   234
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   235
    def compile_rql(self, rql):
3240
8604a15995d1 refactor so that rql rewriter may be used outside the server. Enhance it to be usable for RRQLExpression as well
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3041
diff changeset
   236
        rqlst = self.repo.vreg.rqlhelper.parse(rql)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   237
        rqlst.restricted_vars = ()
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   238
        rqlst.children[0].solutions = self._sols
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   239
        self.repo.querier.sqlgen_annotate(rqlst)
438
69b79faefa94 need_intersect test and fixes
sylvain.thenault@logilab.fr
parents: 0
diff changeset
   240
        set_qdata(self.schema.rschema, rqlst, ())
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   241
        return rqlst
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   242
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   243
    def set_schema(self, schema):
2476
1294a6bdf3bf application -> instance where it makes sense
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2354
diff changeset
   244
        """set the instance'schema"""
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   245
        self._cache = Cache(self.repo.config['rql-cache-size'])
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   246
        self.cache_hit, self.cache_miss, self.no_cache = 0, 0, 0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   247
        self.schema = schema
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   248
        try:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   249
            self._rql_sqlgen.schema = schema
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   250
        except AttributeError:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   251
            pass # __init__
1398
5fe84a5f7035 rename internal entity types to have CW prefix instead of E
sylvain.thenault@logilab.fr
parents: 1263
diff changeset
   252
        if 'CWUser' in schema: # probably an empty schema if not true...
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   253
            # rql syntax trees used to authenticate users
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   254
            self._passwd_rqlst = self.compile_rql(self.passwd_rql)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   255
            self._auth_rqlst = self.compile_rql(self.auth_rql)
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   256
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   257
    def support_entity(self, etype, write=False):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   258
        """return true if the given entity's type is handled by this adapter
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   259
        if write is true, return true only if it's a RW support
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   260
        """
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   261
        return not etype in NONSYSTEM_ETYPES
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   262
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   263
    def support_relation(self, rtype, write=False):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   264
        """return true if the given relation's type is handled by this adapter
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   265
        if write is true, return true only if it's a RW support
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   266
        """
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   267
        if write:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   268
            return not rtype in NONSYSTEM_RELATIONS
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   269
        # due to current multi-sources implementation, the system source
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   270
        # can't claim not supporting a relation
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   271
        return True #not rtype == 'content_for'
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   272
3041
782fa7566a22 [multi-sources] new may_cross_relation method on sources
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2967
diff changeset
   273
    def may_cross_relation(self, rtype):
782fa7566a22 [multi-sources] new may_cross_relation method on sources
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2967
diff changeset
   274
        return True
782fa7566a22 [multi-sources] new may_cross_relation method on sources
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2967
diff changeset
   275
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   276
    def authenticate(self, session, login, password):
1398
5fe84a5f7035 rename internal entity types to have CW prefix instead of E
sylvain.thenault@logilab.fr
parents: 1263
diff changeset
   277
        """return CWUser eid for the given login/password if this account is
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   278
        defined in this source, else raise `AuthenticationError`
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   279
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   280
        two queries are needed since passwords are stored crypted, so we have
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   281
        to fetch the salt first
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   282
        """
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   283
        args = {'login': login, 'pwd' : password}
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   284
        if password is not None:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   285
            rset = self.syntax_tree_search(session, self._passwd_rqlst, args)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   286
            try:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   287
                pwd = rset[0][0]
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   288
            except IndexError:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   289
                raise AuthenticationError('bad login')
1954
9b20f3504af8 cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1952
diff changeset
   290
            # passwords are stored using the Bytes type, so we get a StringIO
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   291
            if pwd is not None:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   292
                args['pwd'] = crypt_password(password, pwd.getvalue()[:2])
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   293
        # get eid from login and (crypted) password
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   294
        rset = self.syntax_tree_search(session, self._auth_rqlst, args)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   295
        try:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   296
            return rset[0][0]
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   297
        except IndexError:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   298
            raise AuthenticationError('bad password')
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   299
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   300
    def syntax_tree_search(self, session, union, args=None, cachekey=None,
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   301
                           varmap=None):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   302
        """return result from this source for a rql query (actually from
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   303
        a rql syntax tree and a solution dictionary mapping each used
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   304
        variable to a possible type). If cachekey is given, the query
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   305
        necessary to fetch the results (but not the results themselves)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   306
        may be cached using this key.
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   307
        """
2625
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2620
diff changeset
   308
        assert dbg_st_search(self.uri, union, varmap, args, cachekey)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   309
        # remember number of actually selected term (sql generation may append some)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   310
        if cachekey is None:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   311
            self.no_cache += 1
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   312
            # generate sql query if we are able to do so (not supported types...)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   313
            sql, query_args = self._rql_sqlgen.generate(union, args, varmap)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   314
        else:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   315
            # sql may be cached
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   316
            try:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   317
                sql, query_args = self._cache[cachekey]
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   318
                self.cache_hit += 1
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   319
            except KeyError:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   320
                self.cache_miss += 1
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   321
                sql, query_args = self._rql_sqlgen.generate(union, args, varmap)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   322
                self._cache[cachekey] = sql, query_args
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   323
        args = self.merge_args(args, query_args)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   324
        assert isinstance(sql, basestring), repr(sql)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   325
        try:
2306
95da5d9f0870 give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2072
diff changeset
   326
            cursor = self.doexec(session, sql, args)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   327
        except (self.dbapi_module.OperationalError,
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   328
                self.dbapi_module.InterfaceError):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   329
            # FIXME: better detection of deconnection pb
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   330
            self.info("request failed '%s' ... retry with a new cursor", sql)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   331
            session.pool.reconnect(self)
2306
95da5d9f0870 give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2072
diff changeset
   332
            cursor = self.doexec(session, sql, args)
2625
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2620
diff changeset
   333
        results = self.process_result(cursor)
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2620
diff changeset
   334
        assert dbg_results(results)
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2620
diff changeset
   335
        return results
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   336
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   337
    def flying_insert(self, table, session, union, args=None, varmap=None):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   338
        """similar as .syntax_tree_search, but inserts data in the
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   339
        temporary table (on-the-fly if possible, eg for the system
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   340
        source whose the given cursor come from). If not possible,
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   341
        inserts all data by calling .executemany().
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   342
        """
2625
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2620
diff changeset
   343
        assert dbg_st_search(
2638
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2627
diff changeset
   344
            self.uri, union, varmap, args,
2625
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2620
diff changeset
   345
            prefix='ON THE FLY temp data insertion into %s from' % table)
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2620
diff changeset
   346
        # generate sql queries if we are able to do so
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2620
diff changeset
   347
        sql, query_args = self._rql_sqlgen.generate(union, args, varmap)
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2620
diff changeset
   348
        query = 'INSERT INTO %s %s' % (table, sql.encode(self.encoding))
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2620
diff changeset
   349
        self.doexec(session, query, self.merge_args(args, query_args))
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   350
2627
d710278e0c1c manual_insert is a public method
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2625
diff changeset
   351
    def manual_insert(self, results, table, session):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   352
        """insert given result into a temporary table on the system source"""
2625
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2620
diff changeset
   353
        if server.DEBUG & server.DBG_RQL:
3958
505025eb0d37 fix name error while debugging
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3835
diff changeset
   354
            print '  manual insertion of', results, 'into', table
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   355
        if not results:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   356
            return
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   357
        query_args = ['%%(%s)s' % i for i in xrange(len(results[0]))]
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   358
        query = 'INSERT INTO %s VALUES(%s)' % (table, ','.join(query_args))
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   359
        kwargs_list = []
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   360
        for row in results:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   361
            kwargs = {}
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   362
            row = tuple(row)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   363
            for index, cell in enumerate(row):
2066
2c4bf4ee88a2 cleanup, stop encoding unicode string in manual_insert, no more necessary and make crash recent sqlite w/ 8bit string
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2064
diff changeset
   364
                if isinstance(cell, Binary):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   365
                    cell = self.binary(cell.getvalue())
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   366
                kwargs[str(index)] = cell
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   367
            kwargs_list.append(kwargs)
2306
95da5d9f0870 give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2072
diff changeset
   368
        self.doexecmany(session, query, kwargs_list)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   369
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   370
    def clean_temp_data(self, session, temptables):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   371
        """remove temporary data, usually associated to temporary tables"""
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   372
        if temptables:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   373
            for table in temptables:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   374
                try:
2306
95da5d9f0870 give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2072
diff changeset
   375
                    self.doexec(session,'DROP TABLE %s' % table)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   376
                except:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   377
                    pass
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   378
                try:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   379
                    del self._temp_table_data[table]
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   380
                except KeyError:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   381
                    continue
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   382
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   383
    def add_entity(self, session, entity):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   384
        """add a new entity to the source"""
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   385
        attrs = self.preprocess_entity(entity)
1251
af40e615dc89 introduce a 'cw_' prefix on entity table and column names so we don't conflict with sql or DBMS specific keywords
sylvain.thenault@logilab.fr
parents: 1079
diff changeset
   386
        sql = self.sqlgen.insert(SQL_PREFIX + str(entity.e_schema), attrs)
2306
95da5d9f0870 give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2072
diff changeset
   387
        self.doexec(session, sql, attrs)
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   388
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   389
    def update_entity(self, session, entity):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   390
        """replace an entity in the source"""
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   391
        attrs = self.preprocess_entity(entity)
1251
af40e615dc89 introduce a 'cw_' prefix on entity table and column names so we don't conflict with sql or DBMS specific keywords
sylvain.thenault@logilab.fr
parents: 1079
diff changeset
   392
        sql = self.sqlgen.update(SQL_PREFIX + str(entity.e_schema), attrs, [SQL_PREFIX + 'eid'])
2306
95da5d9f0870 give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2072
diff changeset
   393
        self.doexec(session, sql, attrs)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   394
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   395
    def delete_entity(self, session, etype, eid):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   396
        """delete an entity from the source"""
1251
af40e615dc89 introduce a 'cw_' prefix on entity table and column names so we don't conflict with sql or DBMS specific keywords
sylvain.thenault@logilab.fr
parents: 1079
diff changeset
   397
        attrs = {SQL_PREFIX + 'eid': eid}
af40e615dc89 introduce a 'cw_' prefix on entity table and column names so we don't conflict with sql or DBMS specific keywords
sylvain.thenault@logilab.fr
parents: 1079
diff changeset
   398
        sql = self.sqlgen.delete(SQL_PREFIX + etype, attrs)
2306
95da5d9f0870 give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2072
diff changeset
   399
        self.doexec(session, sql, attrs)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   400
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   401
    def add_relation(self, session, subject, rtype, object):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   402
        """add a relation to the source"""
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   403
        attrs = {'eid_from': subject, 'eid_to': object}
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   404
        sql = self.sqlgen.insert('%s_relation' % rtype, attrs)
2306
95da5d9f0870 give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2072
diff changeset
   405
        self.doexec(session, sql, attrs)
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   406
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   407
    def delete_relation(self, session, subject, rtype, object):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   408
        """delete a relation from the source"""
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   409
        rschema = self.schema.rschema(rtype)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   410
        if rschema.inlined:
1251
af40e615dc89 introduce a 'cw_' prefix on entity table and column names so we don't conflict with sql or DBMS specific keywords
sylvain.thenault@logilab.fr
parents: 1079
diff changeset
   411
            table = SQL_PREFIX + session.describe(subject)[0]
af40e615dc89 introduce a 'cw_' prefix on entity table and column names so we don't conflict with sql or DBMS specific keywords
sylvain.thenault@logilab.fr
parents: 1079
diff changeset
   412
            column = SQL_PREFIX + rtype
af40e615dc89 introduce a 'cw_' prefix on entity table and column names so we don't conflict with sql or DBMS specific keywords
sylvain.thenault@logilab.fr
parents: 1079
diff changeset
   413
            sql = 'UPDATE %s SET %s=NULL WHERE %seid=%%(eid)s' % (table, column,
af40e615dc89 introduce a 'cw_' prefix on entity table and column names so we don't conflict with sql or DBMS specific keywords
sylvain.thenault@logilab.fr
parents: 1079
diff changeset
   414
                                                                  SQL_PREFIX)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   415
            attrs = {'eid' : subject}
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   416
        else:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   417
            attrs = {'eid_from': subject, 'eid_to': object}
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   418
            sql = self.sqlgen.delete('%s_relation' % rtype, attrs)
2306
95da5d9f0870 give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2072
diff changeset
   419
        self.doexec(session, sql, attrs)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   420
2618
ff9b0d5bd884 [F repo sqlite schema changes] don't rollback on potentially expected schema changes failure
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2611
diff changeset
   421
    def doexec(self, session, query, args=None, rollback=True):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   422
        """Execute a query.
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   423
        it's a function just so that it shows up in profiling
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   424
        """
2306
95da5d9f0870 give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2072
diff changeset
   425
        cursor = session.pool[self.uri]
2593
16d9419a4a79 F: start to handle binary debug log level on the server side
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2589
diff changeset
   426
        if server.DEBUG & server.DBG_SQL:
2625
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2620
diff changeset
   427
            cnx = session.pool.connection(self.uri)
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2620
diff changeset
   428
            # getattr to get the actual connection if cnx is a ConnectionWrapper
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2620
diff changeset
   429
            # instance
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2620
diff changeset
   430
            print 'exec', query, args, getattr(cnx, '_cnx', cnx)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   431
        try:
2306
95da5d9f0870 give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2072
diff changeset
   432
            # str(query) to avoid error if it's an unicode string
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   433
            cursor.execute(str(query), args)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   434
        except Exception, ex:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   435
            self.critical("sql: %r\n args: %s\ndbms message: %r",
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   436
                          query, args, ex.args[0])
2618
ff9b0d5bd884 [F repo sqlite schema changes] don't rollback on potentially expected schema changes failure
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2611
diff changeset
   437
            if rollback:
ff9b0d5bd884 [F repo sqlite schema changes] don't rollback on potentially expected schema changes failure
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2611
diff changeset
   438
                try:
ff9b0d5bd884 [F repo sqlite schema changes] don't rollback on potentially expected schema changes failure
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2611
diff changeset
   439
                    session.pool.connection(self.uri).rollback()
ff9b0d5bd884 [F repo sqlite schema changes] don't rollback on potentially expected schema changes failure
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2611
diff changeset
   440
                    self.critical('transaction has been rollbacked')
ff9b0d5bd884 [F repo sqlite schema changes] don't rollback on potentially expected schema changes failure
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2611
diff changeset
   441
                except:
ff9b0d5bd884 [F repo sqlite schema changes] don't rollback on potentially expected schema changes failure
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2611
diff changeset
   442
                    pass
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   443
            raise
2306
95da5d9f0870 give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2072
diff changeset
   444
        return cursor
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   445
2306
95da5d9f0870 give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2072
diff changeset
   446
    def doexecmany(self, session, query, args):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   447
        """Execute a query.
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   448
        it's a function just so that it shows up in profiling
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   449
        """
2593
16d9419a4a79 F: start to handle binary debug log level on the server side
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2589
diff changeset
   450
        if server.DEBUG & server.DBG_SQL:
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   451
            print 'execmany', query, 'with', len(args), 'arguments'
2306
95da5d9f0870 give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2072
diff changeset
   452
        cursor = session.pool[self.uri]
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   453
        try:
2306
95da5d9f0870 give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2072
diff changeset
   454
            # str(query) to avoid error if it's an unicode string
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   455
            cursor.executemany(str(query), args)
2306
95da5d9f0870 give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2072
diff changeset
   456
        except Exception, ex:
95da5d9f0870 give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2072
diff changeset
   457
            self.critical("sql many: %r\n args: %s\ndbms message: %r",
95da5d9f0870 give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2072
diff changeset
   458
                          query, args, ex.args[0])
95da5d9f0870 give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2072
diff changeset
   459
            try:
95da5d9f0870 give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2072
diff changeset
   460
                session.pool.connection(self.uri).rollback()
95da5d9f0870 give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2072
diff changeset
   461
                self.critical('transaction has been rollbacked')
95da5d9f0870 give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2072
diff changeset
   462
            except:
95da5d9f0870 give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2072
diff changeset
   463
                pass
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   464
            raise
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   465
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   466
    # short cut to method requiring advanced db helper usage ##################
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   467
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   468
    def create_index(self, session, table, column, unique=False):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   469
        cursor = LogCursor(session.pool[self.uri])
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   470
        self.dbhelper.create_index(cursor, table, column, unique)
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   471
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   472
    def drop_index(self, session, table, column, unique=False):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   473
        cursor = LogCursor(session.pool[self.uri])
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   474
        self.dbhelper.drop_index(cursor, table, column, unique)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   475
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   476
    # system source interface #################################################
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   477
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   478
    def eid_type_source(self, session, eid):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   479
        """return a tuple (type, source, extid) for the entity with id <eid>"""
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   480
        sql = 'SELECT type, source, extid FROM entities WHERE eid=%s' % eid
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   481
        try:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   482
            res = session.system_sql(sql).fetchone()
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   483
        except:
1079
452cb76fe07a backport typo fix
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 973
diff changeset
   484
            assert session.pool, 'session has no pool set'
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   485
            raise UnknownEid(eid)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   486
        if res is None:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   487
            raise UnknownEid(eid)
1952
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1792
diff changeset
   488
        if res[-1] is not None:
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1792
diff changeset
   489
            if not isinstance(res, list):
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1792
diff changeset
   490
                res = list(res)
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1792
diff changeset
   491
            res[-1] = b64decode(res[-1])
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   492
        return res
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   493
1952
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1792
diff changeset
   494
    def extid2eid(self, session, source, extid):
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1792
diff changeset
   495
        """get eid from an external id. Return None if no record found."""
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1792
diff changeset
   496
        assert isinstance(extid, str)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   497
        cursor = session.system_sql('SELECT eid FROM entities WHERE '
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   498
                                    'extid=%(x)s AND source=%(s)s',
1952
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1792
diff changeset
   499
                                    {'x': b64encode(extid), 's': source.uri})
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   500
        # XXX testing rowcount cause strange bug with sqlite, results are there
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   501
        #     but rowcount is 0
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   502
        #if cursor.rowcount > 0:
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   503
        try:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   504
            result = cursor.fetchone()
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   505
            if result:
1954
9b20f3504af8 cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1952
diff changeset
   506
                return result[0]
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   507
        except:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   508
            pass
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   509
        return None
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   510
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   511
    def temp_table_def(self, selected, sol, table):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   512
        return make_schema(selected, sol, table, self.dbhelper.TYPE_MAPPING)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   513
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   514
    def create_temp_table(self, session, table, schema):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   515
        # we don't want on commit drop, this may cause problem when
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   516
        # running with an ldap source, and table will be deleted manually any way
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   517
        # on commit
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   518
        sql = self.dbhelper.sql_temporary_table(table, schema, False)
2306
95da5d9f0870 give session to doexec so it's able to rollback the connection on unexpected error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2072
diff changeset
   519
        self.doexec(session, sql)
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   520
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   521
    def create_eid(self, session):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   522
        self._eid_creation_lock.acquire()
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   523
        try:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   524
            for sql in self.dbhelper.sqls_increment_sequence('entities_id_seq'):
2310
80fcdfbb8eed oops, needs the cursor
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2306
diff changeset
   525
                cursor = self.doexec(session, sql)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   526
            return cursor.fetchone()[0]
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   527
        finally:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   528
            self._eid_creation_lock.release()
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   529
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   530
    def add_info(self, session, entity, source, extid=None):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   531
        """add type and source info for an eid into the system table"""
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   532
        # begin by inserting eid/type/source/extid into the entities table
1952
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1792
diff changeset
   533
        if extid is not None:
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1792
diff changeset
   534
            assert isinstance(extid, str)
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1792
diff changeset
   535
            extid = b64encode(extid)
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1792
diff changeset
   536
        attrs = {'type': entity.id, 'eid': entity.eid, 'extid': extid,
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1792
diff changeset
   537
                 'source': source.uri, 'mtime': datetime.now()}
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   538
        session.system_sql(self.sqlgen.insert('entities', attrs), attrs)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   539
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   540
    def delete_info(self, session, eid, etype, uri, extid):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   541
        """delete system information on deletion of an entity by transfering
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   542
        record from the entities table to the deleted_entities table
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   543
        """
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   544
        attrs = {'eid': eid}
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   545
        session.system_sql(self.sqlgen.delete('entities', attrs), attrs)
2955
6bb5025c9fc7 remove some pretty old deprecation code
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2759
diff changeset
   546
        if extid is not None:
6bb5025c9fc7 remove some pretty old deprecation code
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2759
diff changeset
   547
            assert isinstance(extid, str), type(extid)
6bb5025c9fc7 remove some pretty old deprecation code
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2759
diff changeset
   548
            extid = b64encode(extid)
6bb5025c9fc7 remove some pretty old deprecation code
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2759
diff changeset
   549
        attrs = {'type': etype, 'eid': eid, 'extid': extid,
6bb5025c9fc7 remove some pretty old deprecation code
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2759
diff changeset
   550
                 'source': uri, 'dtime': datetime.now()}
6bb5025c9fc7 remove some pretty old deprecation code
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2759
diff changeset
   551
        session.system_sql(self.sqlgen.insert('deleted_entities', attrs), attrs)
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   552
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   553
    def fti_unindex_entity(self, session, eid):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   554
        """remove text content for entity with the given eid from the full text
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   555
        index
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   556
        """
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   557
        try:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   558
            self.indexer.cursor_unindex_object(eid, session.pool['system'])
3497
63b9541dd36f [fti] don't catch BaseException subclasses
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3240
diff changeset
   559
        except Exception: # let KeyboardInterrupt / SystemExit propagate
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   560
            if self.indexer is not None:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   561
                self.exception('error while unindexing %s', eid)
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   562
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   563
    def fti_index_entity(self, session, entity):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   564
        """add text content of a created/modified entity to the full text index
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   565
        """
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   566
        self.info('reindexing %r', entity.eid)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   567
        try:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   568
            self.indexer.cursor_reindex_object(entity.eid, entity,
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   569
                                               session.pool['system'])
3497
63b9541dd36f [fti] don't catch BaseException subclasses
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3240
diff changeset
   570
        except Exception: # let KeyboardInterrupt / SystemExit propagate
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   571
            if self.indexer is not None:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   572
                self.exception('error while reindexing %s', entity)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   573
        # update entities.mtime
1016
26387b836099 use datetime instead of mx.DateTime
sylvain.thenault@logilab.fr
parents: 973
diff changeset
   574
        attrs = {'eid': entity.eid, 'mtime': datetime.now()}
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   575
        session.system_sql(self.sqlgen.update('entities', attrs, ['eid']), attrs)
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   576
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   577
    def modified_entities(self, session, etypes, mtime):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   578
        """return a 2-uple:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   579
        * list of (etype, eid) of entities of the given types which have been
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   580
          modified since the given timestamp (actually entities whose full text
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   581
          index content has changed)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   582
        * list of (etype, eid) of entities of the given types which have been
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   583
          deleted since the given timestamp
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   584
        """
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   585
        modsql = _modified_sql('entities', etypes)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   586
        cursor = session.system_sql(modsql, {'time': mtime})
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   587
        modentities = cursor.fetchall()
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   588
        delsql = _modified_sql('deleted_entities', etypes)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   589
        cursor = session.system_sql(delsql, {'time': mtime})
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   590
        delentities = cursor.fetchall()
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   591
        return modentities, delentities
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   592
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   593
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   594
def sql_schema(driver):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   595
    helper = get_adv_func_helper(driver)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   596
    schema = """
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   597
/* Create the repository's system database */
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   598
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   599
%s
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   600
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   601
CREATE TABLE entities (
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   602
  eid INTEGER PRIMARY KEY NOT NULL,
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   603
  type VARCHAR(64) NOT NULL,
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   604
  source VARCHAR(64) NOT NULL,
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   605
  mtime TIMESTAMP NOT NULL,
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   606
  extid VARCHAR(256)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   607
);
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   608
CREATE INDEX entities_type_idx ON entities(type);
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   609
CREATE INDEX entities_mtime_idx ON entities(mtime);
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   610
CREATE INDEX entities_extid_idx ON entities(extid);
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   611
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   612
CREATE TABLE deleted_entities (
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   613
  eid INTEGER PRIMARY KEY NOT NULL,
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   614
  type VARCHAR(64) NOT NULL,
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   615
  source VARCHAR(64) NOT NULL,
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   616
  dtime TIMESTAMP NOT NULL,
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   617
  extid VARCHAR(256)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   618
);
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   619
CREATE INDEX deleted_entities_type_idx ON deleted_entities(type);
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   620
CREATE INDEX deleted_entities_dtime_idx ON deleted_entities(dtime);
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   621
CREATE INDEX deleted_entities_extid_idx ON deleted_entities(extid);
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   622
""" % helper.sql_create_sequence('entities_id_seq')
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   623
    return schema
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   624
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   625
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   626
def sql_drop_schema(driver):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   627
    helper = get_adv_func_helper(driver)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   628
    return """
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   629
%s
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   630
DROP TABLE entities;
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   631
DROP TABLE deleted_entities;
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   632
""" % helper.sql_drop_sequence('entities_id_seq')
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   633
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   634
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   635
def grant_schema(user, set_owner=True):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   636
    result = ''
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   637
    if set_owner:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   638
        result = 'ALTER TABLE entities OWNER TO %s;\n' % user
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   639
        result += 'ALTER TABLE deleted_entities OWNER TO %s;\n' % user
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   640
        result += 'ALTER TABLE entities_id_seq OWNER TO %s;\n' % user
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   641
    result += 'GRANT ALL ON entities TO %s;\n' % user
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   642
    result += 'GRANT ALL ON deleted_entities TO %s;\n' % user
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   643
    result += 'GRANT ALL ON entities_id_seq TO %s;\n' % user
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   644
    return result