server/sources/pyrorql.py
author Sylvain Thénault <sylvain.thenault@logilab.fr>
Wed, 26 May 2010 10:28:48 +0200
branchstable
changeset 5582 3e133b29a1a4
parent 5426 0d4853a6e5ee
child 5643 fd240f98a3ee
permissions -rw-r--r--
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
5421
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4913
diff changeset
     1
# copyright 2003-2010 LOGILAB S.A. (Paris, FRANCE), all rights reserved.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4913
diff changeset
     2
# contact http://www.logilab.fr/ -- mailto:contact@logilab.fr
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4913
diff changeset
     3
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4913
diff changeset
     4
# This file is part of CubicWeb.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4913
diff changeset
     5
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4913
diff changeset
     6
# CubicWeb is free software: you can redistribute it and/or modify it under the
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4913
diff changeset
     7
# terms of the GNU Lesser General Public License as published by the Free
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4913
diff changeset
     8
# Software Foundation, either version 2.1 of the License, or (at your option)
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4913
diff changeset
     9
# any later version.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4913
diff changeset
    10
#
5424
8ecbcbff9777 replace logilab-common by CubicWeb in disclaimer
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5421
diff changeset
    11
# CubicWeb is distributed in the hope that it will be useful, but WITHOUT
5421
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4913
diff changeset
    12
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4913
diff changeset
    13
# FOR A PARTICULAR PURPOSE.  See the GNU Lesser General Public License for more
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4913
diff changeset
    14
# details.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4913
diff changeset
    15
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4913
diff changeset
    16
# You should have received a copy of the GNU Lesser General Public License along
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4913
diff changeset
    17
# with CubicWeb.  If not, see <http://www.gnu.org/licenses/>.
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    18
"""Source to query another RQL repository using pyro
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    19
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    20
"""
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    21
__docformat__ = "restructuredtext en"
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    22
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    23
import threading
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    24
from os.path import join
1016
26387b836099 use datetime instead of mx.DateTime
sylvain.thenault@logilab.fr
parents: 520
diff changeset
    25
from time import mktime
26387b836099 use datetime instead of mx.DateTime
sylvain.thenault@logilab.fr
parents: 520
diff changeset
    26
from datetime import datetime
1952
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1805
diff changeset
    27
from base64 import b64decode
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    28
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    29
from Pyro.errors import PyroError, ConnectionClosedError
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    30
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    31
from logilab.common.configuration import REQUIRED
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    32
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    33
from rql.nodes import Constant
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    34
from rql.utils import rqlvar_maker
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    35
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    36
from cubicweb import dbapi, server
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    37
from cubicweb import BadConnectionId, UnknownEid, ConnectionError
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    38
from cubicweb.cwconfig import register_persistent_options
2625
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2593
diff changeset
    39
from cubicweb.server.sources import (AbstractSource, ConnectionWrapper,
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2593
diff changeset
    40
                                     TimedCache, dbg_st_search, dbg_results)
5582
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
    41
from cubicweb.server.msplanner import neged_relation
3016
5787d1cc8106 [rset] fix #231354 w/ rql 0.22.3 api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2763
diff changeset
    42
5787d1cc8106 [rset] fix #231354 w/ rql 0.22.3 api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2763
diff changeset
    43
def uidtype(union, col, etype, args):
5787d1cc8106 [rset] fix #231354 w/ rql 0.22.3 api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2763
diff changeset
    44
    select, col = union.locate_subquery(col, etype, args)
5787d1cc8106 [rset] fix #231354 w/ rql 0.22.3 api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2763
diff changeset
    45
    return getattr(select.selection[col], 'uidtype', None)
5787d1cc8106 [rset] fix #231354 w/ rql 0.22.3 api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2763
diff changeset
    46
5787d1cc8106 [rset] fix #231354 w/ rql 0.22.3 api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2763
diff changeset
    47
1138
22f634977c95 make pylint happy, fix some bugs on the way
sylvain.thenault@logilab.fr
parents: 1016
diff changeset
    48
class ReplaceByInOperator(Exception):
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    49
    def __init__(self, eids):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    50
        self.eids = eids
1791
c77629112437 clear cache before synchronization
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
    51
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    52
class PyroRQLSource(AbstractSource):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    53
    """External repository source, using Pyro connection"""
1791
c77629112437 clear cache before synchronization
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
    54
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    55
    # boolean telling if modification hooks should be called when something is
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    56
    # modified in this source
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    57
    should_call_hooks = False
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    58
    # boolean telling if the repository should connect to this source during
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    59
    # migration
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    60
    connect_for_migration = False
1791
c77629112437 clear cache before synchronization
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
    61
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    62
    support_entities = None
1791
c77629112437 clear cache before synchronization
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
    63
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    64
    options = (
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    65
        # XXX pyro-ns host/port
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    66
        ('pyro-ns-id',
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    67
         {'type' : 'string',
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    68
          'default': REQUIRED,
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    69
          'help': 'identifier of the repository in the pyro name server',
5323
329b4f6d18b4 [config] with lgc >= 0.50, option's dict inputlevel becomes level
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5174
diff changeset
    70
          'group': 'pyro-source', 'level': 0,
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    71
          }),
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    72
        ('mapping-file',
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    73
         {'type' : 'string',
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    74
          'default': REQUIRED,
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    75
          'help': 'path to a python file with the schema mapping definition',
5323
329b4f6d18b4 [config] with lgc >= 0.50, option's dict inputlevel becomes level
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5174
diff changeset
    76
          'group': 'pyro-source', 'level': 1,
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    77
          }),
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    78
        ('cubicweb-user',
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    79
         {'type' : 'string',
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    80
          'default': REQUIRED,
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    81
          'help': 'user to use for connection on the distant repository',
5323
329b4f6d18b4 [config] with lgc >= 0.50, option's dict inputlevel becomes level
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5174
diff changeset
    82
          'group': 'pyro-source', 'level': 0,
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    83
          }),
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    84
        ('cubicweb-password',
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    85
         {'type' : 'password',
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    86
          'default': '',
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    87
          'help': 'user to use for connection on the distant repository',
5323
329b4f6d18b4 [config] with lgc >= 0.50, option's dict inputlevel becomes level
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5174
diff changeset
    88
          'group': 'pyro-source', 'level': 0,
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    89
          }),
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    90
        ('base-url',
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    91
         {'type' : 'string',
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    92
          'default': '',
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    93
          'help': 'url of the web site for the distant repository, if you want '
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    94
          'to generate external link to entities from this repository',
5323
329b4f6d18b4 [config] with lgc >= 0.50, option's dict inputlevel becomes level
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5174
diff changeset
    95
          'group': 'pyro-source', 'level': 1,
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    96
          }),
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    97
        ('pyro-ns-host',
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    98
         {'type' : 'string',
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    99
          'default': None,
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   100
          'help': 'Pyro name server\'s host. If not set, default to the value \
2665
0c6281487f90 [pyro] use lgc.pyro_ext, simplify pyro related options
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2625
diff changeset
   101
from all_in_one.conf. It may contains port information using <host>:<port> notation.',
5323
329b4f6d18b4 [config] with lgc >= 0.50, option's dict inputlevel becomes level
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5174
diff changeset
   102
          'group': 'pyro-source', 'level': 1,
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   103
          }),
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   104
        ('pyro-ns-group',
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   105
         {'type' : 'string',
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   106
          'default': None,
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   107
          'help': 'Pyro name server\'s group where the repository will be \
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   108
registered. If not set, default to the value from all_in_one.conf.',
5323
329b4f6d18b4 [config] with lgc >= 0.50, option's dict inputlevel becomes level
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5174
diff changeset
   109
          'group': 'pyro-source', 'level': 1,
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   110
          }),
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   111
        ('synchronization-interval',
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   112
         {'type' : 'int',
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   113
          'default': 5*60,
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   114
          'help': 'interval between synchronization with the external \
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   115
repository (default to 5 minutes).',
5323
329b4f6d18b4 [config] with lgc >= 0.50, option's dict inputlevel becomes level
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5174
diff changeset
   116
          'group': 'pyro-source', 'level': 2,
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   117
          }),
1791
c77629112437 clear cache before synchronization
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   118
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   119
    )
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   120
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   121
    PUBLIC_KEYS = AbstractSource.PUBLIC_KEYS + ('base-url',)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   122
    _conn = None
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   123
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   124
    def __init__(self, repo, appschema, source_config, *args, **kwargs):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   125
        AbstractSource.__init__(self, repo, appschema, source_config,
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   126
                                *args, **kwargs)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   127
        mappingfile = source_config['mapping-file']
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   128
        if not mappingfile[0] == '/':
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   129
            mappingfile = join(repo.config.apphome, mappingfile)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   130
        mapping = {}
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   131
        execfile(mappingfile, mapping)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   132
        self.support_entities = mapping['support_entities']
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   133
        self.support_relations = mapping.get('support_relations', {})
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   134
        self.dont_cross_relations = mapping.get('dont_cross_relations', ())
417
828424d2eb3c oops, get back cross_relation from mapping to rql source
sylvain.thenault@logilab.fr
parents: 391
diff changeset
   135
        self.cross_relations = mapping.get('cross_relations', ())
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   136
        baseurl = source_config.get('base-url')
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   137
        if baseurl and not baseurl.endswith('/'):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   138
            source_config['base-url'] += '/'
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   139
        self.config = source_config
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   140
        myoptions = (('%s.latest-update-time' % self.uri,
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   141
                      {'type' : 'int', 'sitewide': True,
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   142
                       'default': 0,
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   143
                       'help': _('timestamp of the latest source synchronization.'),
1791
c77629112437 clear cache before synchronization
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   144
                       'group': 'sources',
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   145
                       }),)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   146
        register_persistent_options(myoptions)
1238
fa29b5b60107 set 30sec query cache on pyro source, important speedup for pages generating multiple time the same external query
sylvain.thenault@logilab.fr
parents: 520
diff changeset
   147
        self._query_cache = TimedCache(30)
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   148
2763
39b42e158249 [ms] proper reset cache on external source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2665
diff changeset
   149
    def reset_caches(self):
39b42e158249 [ms] proper reset cache on external source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2665
diff changeset
   150
        """method called during test to reset potential source caches"""
39b42e158249 [ms] proper reset cache on external source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2665
diff changeset
   151
        self._query_cache = TimedCache(30)
39b42e158249 [ms] proper reset cache on external source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2665
diff changeset
   152
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   153
    def last_update_time(self):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   154
        pkey = u'sources.%s.latest-update-time' % self.uri
1398
5fe84a5f7035 rename internal entity types to have CW prefix instead of E
sylvain.thenault@logilab.fr
parents: 1263
diff changeset
   155
        rql = 'Any V WHERE X is CWProperty, X value V, X pkey %(k)s'
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   156
        session = self.repo.internal_session()
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   157
        try:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   158
            rset = session.execute(rql, {'k': pkey})
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   159
            if not rset:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   160
                # insert it
1398
5fe84a5f7035 rename internal entity types to have CW prefix instead of E
sylvain.thenault@logilab.fr
parents: 1263
diff changeset
   161
                session.execute('INSERT CWProperty X: X pkey %(k)s, X value %(v)s',
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   162
                                {'k': pkey, 'v': u'0'})
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   163
                session.commit()
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   164
                timestamp = 0
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   165
            else:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   166
                assert len(rset) == 1
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   167
                timestamp = int(rset[0][0])
1016
26387b836099 use datetime instead of mx.DateTime
sylvain.thenault@logilab.fr
parents: 520
diff changeset
   168
            return datetime.fromtimestamp(timestamp)
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   169
        finally:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   170
            session.close()
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   171
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   172
    def init(self):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   173
        """method called by the repository once ready to handle request"""
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   174
        interval = int(self.config.get('synchronization-interval', 5*60))
1791
c77629112437 clear cache before synchronization
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   175
        self.repo.looping_task(interval, self.synchronize)
c77629112437 clear cache before synchronization
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   176
        self.repo.looping_task(self._query_cache.ttl.seconds/10, self._query_cache.clear_expired)
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   177
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   178
    def synchronize(self, mtime=None):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   179
        """synchronize content known by this repository with content in the
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   180
        external repository
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   181
        """
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   182
        self.info('synchronizing pyro source %s', self.uri)
519
06390418cd9a pyrorql source now ignore external eids which are themselves coming from another external source already in use by the repository (should have the same uri)
sylvain.thenault@logilab.fr
parents: 417
diff changeset
   183
        cnx = self.get_connection()
2271
5483155a1e17 avoid additional tb in logs when we can't connect to the external source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2269
diff changeset
   184
        try:
5483155a1e17 avoid additional tb in logs when we can't connect to the external source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2269
diff changeset
   185
            extrepo = cnx._repo
5483155a1e17 avoid additional tb in logs when we can't connect to the external source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2269
diff changeset
   186
        except AttributeError:
5483155a1e17 avoid additional tb in logs when we can't connect to the external source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2269
diff changeset
   187
            # fake connection wrapper returned when we can't connect to the
5483155a1e17 avoid additional tb in logs when we can't connect to the external source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2269
diff changeset
   188
            # external source (hence we've no chance to synchronize...)
5483155a1e17 avoid additional tb in logs when we can't connect to the external source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2269
diff changeset
   189
            return
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   190
        etypes = self.support_entities.keys()
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   191
        if mtime is None:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   192
            mtime = self.last_update_time()
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   193
        updatetime, modified, deleted = extrepo.entities_modified_since(etypes,
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   194
                                                                        mtime)
1791
c77629112437 clear cache before synchronization
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   195
        self._query_cache.clear()
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   196
        repo = self.repo
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   197
        session = repo.internal_session()
4816
c02583cb80a9 repair stuff broken by fti handling changes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4212
diff changeset
   198
        source = repo.system_source
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   199
        try:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   200
            for etype, extid in modified:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   201
                try:
519
06390418cd9a pyrorql source now ignore external eids which are themselves coming from another external source already in use by the repository (should have the same uri)
sylvain.thenault@logilab.fr
parents: 417
diff changeset
   202
                    exturi = cnx.describe(extid)[1]
06390418cd9a pyrorql source now ignore external eids which are themselves coming from another external source already in use by the repository (should have the same uri)
sylvain.thenault@logilab.fr
parents: 417
diff changeset
   203
                    if exturi == 'system' or not exturi in repo.sources_by_uri:
1952
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1805
diff changeset
   204
                        eid = self.extid2eid(str(extid), etype, session)
519
06390418cd9a pyrorql source now ignore external eids which are themselves coming from another external source already in use by the repository (should have the same uri)
sylvain.thenault@logilab.fr
parents: 417
diff changeset
   205
                        rset = session.eid_rset(eid, etype)
06390418cd9a pyrorql source now ignore external eids which are themselves coming from another external source already in use by the repository (should have the same uri)
sylvain.thenault@logilab.fr
parents: 417
diff changeset
   206
                        entity = rset.get_entity(0, 0)
06390418cd9a pyrorql source now ignore external eids which are themselves coming from another external source already in use by the repository (should have the same uri)
sylvain.thenault@logilab.fr
parents: 417
diff changeset
   207
                        entity.complete(entity.e_schema.indexable_attributes())
4816
c02583cb80a9 repair stuff broken by fti handling changes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4212
diff changeset
   208
                        source.index_entity(session, entity)
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   209
                except:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   210
                    self.exception('while updating %s with external id %s of source %s',
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   211
                                   etype, extid, self.uri)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   212
                    continue
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   213
            for etype, extid in deleted:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   214
                try:
1952
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1805
diff changeset
   215
                    eid = self.extid2eid(str(extid), etype, session,
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1805
diff changeset
   216
                                         insert=False)
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   217
                    # entity has been deleted from external repository but is not known here
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   218
                    if eid is not None:
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4816
diff changeset
   219
                        entity = session.entity_from_eid(eid, etype)
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4816
diff changeset
   220
                        repo.delete_info(session, entity, self.uri, extid)
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   221
                except:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   222
                    self.exception('while updating %s with external id %s of source %s',
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   223
                                   etype, extid, self.uri)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   224
                    continue
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   225
            session.execute('SET X value %(v)s WHERE X pkey %(k)s',
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   226
                            {'k': u'sources.%s.latest-update-time' % self.uri,
1016
26387b836099 use datetime instead of mx.DateTime
sylvain.thenault@logilab.fr
parents: 520
diff changeset
   227
                             'v': unicode(int(mktime(updatetime.timetuple())))})
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   228
            session.commit()
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   229
        finally:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   230
            session.close()
1791
c77629112437 clear cache before synchronization
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   231
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   232
    def _get_connection(self):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   233
        """open and return a connection to the source"""
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   234
        nshost = self.config.get('pyro-ns-host') or self.repo.config['pyro-ns-host']
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   235
        nsgroup = self.config.get('pyro-ns-group') or self.repo.config['pyro-ns-group']
3527
efeb16ff93f3 log when connecting to an external pyro source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3043
diff changeset
   236
        self.info('connecting to instance :%s.%s for user %s',
3605
1466323ddb9c fix debug msg
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3527
diff changeset
   237
                  nsgroup, self.config['pyro-ns-id'], self.config['cubicweb-user'])
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   238
        #cnxprops = ConnectionProperties(cnxtype=self.config['cnx-type'])
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   239
        return dbapi.connect(database=self.config['pyro-ns-id'],
2269
0d1b7206fdc9 renamed arg
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1977
diff changeset
   240
                             login=self.config['cubicweb-user'],
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   241
                             password=self.config['cubicweb-password'],
2665
0c6281487f90 [pyro] use lgc.pyro_ext, simplify pyro related options
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2625
diff changeset
   242
                             host=nshost, group=nsgroup,
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   243
                             setvreg=False) #cnxprops=cnxprops)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   244
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   245
    def get_connection(self):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   246
        try:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   247
            return self._get_connection()
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   248
        except (ConnectionError, PyroError):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   249
            self.critical("can't get connection to source %s", self.uri,
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   250
                          exc_info=1)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   251
            return ConnectionWrapper()
1791
c77629112437 clear cache before synchronization
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   252
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   253
    def check_connection(self, cnx):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   254
        """check connection validity, return None if the connection is still valid
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   255
        else a new connection
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   256
        """
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   257
        # we have to transfer manually thread ownership. This can be done safely
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   258
        # since the pool to which belong the connection is affected to one
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   259
        # session/thread and can't be called simultaneously
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   260
        try:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   261
            cnx._repo._transferThread(threading.currentThread())
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   262
        except AttributeError:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   263
            # inmemory connection
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   264
            pass
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   265
        if not isinstance(cnx, ConnectionWrapper):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   266
            try:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   267
                cnx.check()
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   268
                return # ok
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   269
            except (BadConnectionId, ConnectionClosedError):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   270
                pass
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   271
        # try to reconnect
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   272
        return self.get_connection()
1791
c77629112437 clear cache before synchronization
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   273
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   274
    def syntax_tree_search(self, session, union, args=None, cachekey=None,
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   275
                           varmap=None):
2625
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2593
diff changeset
   276
        assert dbg_st_search(self.uri, union, varmap, args, cachekey)
1238
fa29b5b60107 set 30sec query cache on pyro source, important speedup for pages generating multiple time the same external query
sylvain.thenault@logilab.fr
parents: 520
diff changeset
   277
        rqlkey = union.as_string(kwargs=args)
fa29b5b60107 set 30sec query cache on pyro source, important speedup for pages generating multiple time the same external query
sylvain.thenault@logilab.fr
parents: 520
diff changeset
   278
        try:
fa29b5b60107 set 30sec query cache on pyro source, important speedup for pages generating multiple time the same external query
sylvain.thenault@logilab.fr
parents: 520
diff changeset
   279
            results = self._query_cache[rqlkey]
fa29b5b60107 set 30sec query cache on pyro source, important speedup for pages generating multiple time the same external query
sylvain.thenault@logilab.fr
parents: 520
diff changeset
   280
        except KeyError:
fa29b5b60107 set 30sec query cache on pyro source, important speedup for pages generating multiple time the same external query
sylvain.thenault@logilab.fr
parents: 520
diff changeset
   281
            results = self._syntax_tree_search(session, union, args)
fa29b5b60107 set 30sec query cache on pyro source, important speedup for pages generating multiple time the same external query
sylvain.thenault@logilab.fr
parents: 520
diff changeset
   282
            self._query_cache[rqlkey] = results
2625
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2593
diff changeset
   283
        assert dbg_results(results)
1238
fa29b5b60107 set 30sec query cache on pyro source, important speedup for pages generating multiple time the same external query
sylvain.thenault@logilab.fr
parents: 520
diff changeset
   284
        return results
1791
c77629112437 clear cache before synchronization
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   285
1238
fa29b5b60107 set 30sec query cache on pyro source, important speedup for pages generating multiple time the same external query
sylvain.thenault@logilab.fr
parents: 520
diff changeset
   286
    def _syntax_tree_search(self, session, union, args):
1791
c77629112437 clear cache before synchronization
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   287
        """return result from this source for a rql query (actually from a rql
c77629112437 clear cache before synchronization
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   288
        syntax tree and a solution dictionary mapping each used variable to a
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   289
        possible type). If cachekey is given, the query necessary to fetch the
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   290
        results (but not the results themselves) may be cached using this key.
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   291
        """
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   292
        if not args is None:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   293
            args = args.copy()
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   294
        # get cached cursor anyway
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   295
        cu = session.pool[self.uri]
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   296
        if cu is None:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   297
            # this is a ConnectionWrapper instance
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   298
            msg = session._("can't connect to source %s, some data may be missing")
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   299
            session.set_shared_data('sources_error', msg % self.uri)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   300
            return []
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   301
        try:
5174
78438ad513ca #759035: Automate addition of eid cachekey in RQL analysis
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4913
diff changeset
   302
            rql = RQL2RQL(self).generate(session, union, args)
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   303
        except UnknownEid, ex:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   304
            if server.DEBUG:
2625
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2593
diff changeset
   305
                print '  unknown eid', ex, 'no results'
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   306
            return []
2593
16d9419a4a79 F: start to handle binary debug log level on the server side
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2271
diff changeset
   307
        if server.DEBUG & server.DBG_RQL:
2625
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2593
diff changeset
   308
            print '  translated rql', rql
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   309
        try:
5174
78438ad513ca #759035: Automate addition of eid cachekey in RQL analysis
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4913
diff changeset
   310
            rset = cu.execute(rql, args)
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   311
        except Exception, ex:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   312
            self.exception(str(ex))
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   313
            msg = session._("error while querying source %s, some data may be missing")
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   314
            session.set_shared_data('sources_error', msg % self.uri)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   315
            return []
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   316
        descr = rset.description
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   317
        if rset:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   318
            needtranslation = []
519
06390418cd9a pyrorql source now ignore external eids which are themselves coming from another external source already in use by the repository (should have the same uri)
sylvain.thenault@logilab.fr
parents: 417
diff changeset
   319
            rows = rset.rows
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   320
            for i, etype in enumerate(descr[0]):
3689
deb13e88e037 follow yams 0.25 api changes to improve performance
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3605
diff changeset
   321
                if (etype is None or not self.schema.eschema(etype).final
3016
5787d1cc8106 [rset] fix #231354 w/ rql 0.22.3 api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2763
diff changeset
   322
                    or uidtype(union, i, etype, args)):
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   323
                    needtranslation.append(i)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   324
            if needtranslation:
519
06390418cd9a pyrorql source now ignore external eids which are themselves coming from another external source already in use by the repository (should have the same uri)
sylvain.thenault@logilab.fr
parents: 417
diff changeset
   325
                cnx = session.pool.connection(self.uri)
06390418cd9a pyrorql source now ignore external eids which are themselves coming from another external source already in use by the repository (should have the same uri)
sylvain.thenault@logilab.fr
parents: 417
diff changeset
   326
                for rowindex in xrange(rset.rowcount - 1, -1, -1):
06390418cd9a pyrorql source now ignore external eids which are themselves coming from another external source already in use by the repository (should have the same uri)
sylvain.thenault@logilab.fr
parents: 417
diff changeset
   327
                    row = rows[rowindex]
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   328
                    for colindex in needtranslation:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   329
                        if row[colindex] is not None: # optional variable
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   330
                            etype = descr[rowindex][colindex]
519
06390418cd9a pyrorql source now ignore external eids which are themselves coming from another external source already in use by the repository (should have the same uri)
sylvain.thenault@logilab.fr
parents: 417
diff changeset
   331
                            exttype, exturi, extid = cnx.describe(row[colindex])
06390418cd9a pyrorql source now ignore external eids which are themselves coming from another external source already in use by the repository (should have the same uri)
sylvain.thenault@logilab.fr
parents: 417
diff changeset
   332
                            if exturi == 'system' or not exturi in self.repo.sources_by_uri:
1952
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1805
diff changeset
   333
                                eid = self.extid2eid(str(row[colindex]), etype,
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1805
diff changeset
   334
                                                     session)
519
06390418cd9a pyrorql source now ignore external eids which are themselves coming from another external source already in use by the repository (should have the same uri)
sylvain.thenault@logilab.fr
parents: 417
diff changeset
   335
                                row[colindex] = eid
06390418cd9a pyrorql source now ignore external eids which are themselves coming from another external source already in use by the repository (should have the same uri)
sylvain.thenault@logilab.fr
parents: 417
diff changeset
   336
                            else:
06390418cd9a pyrorql source now ignore external eids which are themselves coming from another external source already in use by the repository (should have the same uri)
sylvain.thenault@logilab.fr
parents: 417
diff changeset
   337
                                # skip this row
06390418cd9a pyrorql source now ignore external eids which are themselves coming from another external source already in use by the repository (should have the same uri)
sylvain.thenault@logilab.fr
parents: 417
diff changeset
   338
                                del rows[rowindex]
06390418cd9a pyrorql source now ignore external eids which are themselves coming from another external source already in use by the repository (should have the same uri)
sylvain.thenault@logilab.fr
parents: 417
diff changeset
   339
                                del descr[rowindex]
06390418cd9a pyrorql source now ignore external eids which are themselves coming from another external source already in use by the repository (should have the same uri)
sylvain.thenault@logilab.fr
parents: 417
diff changeset
   340
                                break
06390418cd9a pyrorql source now ignore external eids which are themselves coming from another external source already in use by the repository (should have the same uri)
sylvain.thenault@logilab.fr
parents: 417
diff changeset
   341
            results = rows
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   342
        else:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   343
            results = []
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   344
        return results
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   345
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   346
    def _entity_relations_and_kwargs(self, session, entity):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   347
        relations = []
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   348
        kwargs = {'x': self.eid2extid(entity.eid, session)}
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   349
        for key, val in entity.iteritems():
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   350
            relations.append('X %s %%(%s)s' % (key, key))
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   351
            kwargs[key] = val
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   352
        return relations, kwargs
1791
c77629112437 clear cache before synchronization
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   353
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   354
    def add_entity(self, session, entity):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   355
        """add a new entity to the source"""
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   356
        raise NotImplementedError()
1791
c77629112437 clear cache before synchronization
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   357
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   358
    def update_entity(self, session, entity):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   359
        """update an entity in the source"""
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   360
        relations, kwargs = self._entity_relations_and_kwargs(session, entity)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   361
        cu = session.pool[self.uri]
5363
d336476dcf57 [3.8 api update] remaining cache keys
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5323
diff changeset
   362
        cu.execute('SET %s WHERE X eid %%(x)s' % ','.join(relations), kwargs)
1805
7bad2785d167 clear query cache on write
sylvain.thenault@logilab.fr
parents: 1791
diff changeset
   363
        self._query_cache.clear()
3043
7bcb7f1c0f46 force cache clear when external entity is edited, we don't know what may occurs in external repository hooks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3016
diff changeset
   364
        entity.clear_all_caches()
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   365
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4816
diff changeset
   366
    def delete_entity(self, session, entity):
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   367
        """delete an entity from the source"""
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   368
        cu = session.pool[self.uri]
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4816
diff changeset
   369
        cu.execute('DELETE %s X WHERE X eid %%(x)s' % entity.__regid__,
5363
d336476dcf57 [3.8 api update] remaining cache keys
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5323
diff changeset
   370
                   {'x': self.eid2extid(entity.eid, session)})
1805
7bad2785d167 clear query cache on write
sylvain.thenault@logilab.fr
parents: 1791
diff changeset
   371
        self._query_cache.clear()
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   372
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   373
    def add_relation(self, session, subject, rtype, object):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   374
        """add a relation to the source"""
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   375
        cu = session.pool[self.uri]
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   376
        cu.execute('SET X %s Y WHERE X eid %%(x)s, Y eid %%(y)s' % rtype,
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   377
                   {'x': self.eid2extid(subject, session),
5363
d336476dcf57 [3.8 api update] remaining cache keys
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5323
diff changeset
   378
                    'y': self.eid2extid(object, session)})
1805
7bad2785d167 clear query cache on write
sylvain.thenault@logilab.fr
parents: 1791
diff changeset
   379
        self._query_cache.clear()
3043
7bcb7f1c0f46 force cache clear when external entity is edited, we don't know what may occurs in external repository hooks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3016
diff changeset
   380
        session.entity_from_eid(subject).clear_all_caches()
7bcb7f1c0f46 force cache clear when external entity is edited, we don't know what may occurs in external repository hooks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3016
diff changeset
   381
        session.entity_from_eid(object).clear_all_caches()
1791
c77629112437 clear cache before synchronization
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   382
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   383
    def delete_relation(self, session, subject, rtype, object):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   384
        """delete a relation from the source"""
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   385
        cu = session.pool[self.uri]
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   386
        cu.execute('DELETE X %s Y WHERE X eid %%(x)s, Y eid %%(y)s' % rtype,
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   387
                   {'x': self.eid2extid(subject, session),
5363
d336476dcf57 [3.8 api update] remaining cache keys
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5323
diff changeset
   388
                    'y': self.eid2extid(object, session)})
1805
7bad2785d167 clear query cache on write
sylvain.thenault@logilab.fr
parents: 1791
diff changeset
   389
        self._query_cache.clear()
3043
7bcb7f1c0f46 force cache clear when external entity is edited, we don't know what may occurs in external repository hooks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3016
diff changeset
   390
        session.entity_from_eid(subject).clear_all_caches()
7bcb7f1c0f46 force cache clear when external entity is edited, we don't know what may occurs in external repository hooks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3016
diff changeset
   391
        session.entity_from_eid(object).clear_all_caches()
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   392
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   393
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   394
class RQL2RQL(object):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   395
    """translate a local rql query to be executed on a distant repository"""
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   396
    def __init__(self, source):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   397
        self.source = source
390
739d12586b9d avoid potentialy mis-interpreted attribute error
sylvain.thenault@logilab.fr
parents: 257
diff changeset
   398
        self.current_operator = None
1791
c77629112437 clear cache before synchronization
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   399
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   400
    def _accept_children(self, node):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   401
        res = []
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   402
        for child in node.children:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   403
            rql = child.accept(self)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   404
            if rql is not None:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   405
                res.append(rql)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   406
        return res
1791
c77629112437 clear cache before synchronization
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   407
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   408
    def generate(self, session, rqlst, args):
1791
c77629112437 clear cache before synchronization
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   409
        self._session = session
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   410
        self.kwargs = args
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   411
        self.need_translation = False
5174
78438ad513ca #759035: Automate addition of eid cachekey in RQL analysis
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4913
diff changeset
   412
        return self.visit_union(rqlst)
1791
c77629112437 clear cache before synchronization
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   413
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   414
    def visit_union(self, node):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   415
        s = self._accept_children(node)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   416
        if len(s) > 1:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   417
            return ' UNION '.join('(%s)' % q for q in s)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   418
        return s[0]
1791
c77629112437 clear cache before synchronization
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   419
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   420
    def visit_select(self, node):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   421
        """return the tree as an encoded rql string"""
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   422
        self._varmaker = rqlvar_maker(defined=node.defined_vars.copy())
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   423
        self._const_var = {}
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   424
        if node.distinct:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   425
            base = 'DISTINCT Any'
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   426
        else:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   427
            base = 'Any'
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   428
        s = ['%s %s' % (base, ','.join(v.accept(self) for v in node.selection))]
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   429
        if node.groupby:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   430
            s.append('GROUPBY %s' % ', '.join(group.accept(self)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   431
                                              for group in node.groupby))
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   432
        if node.orderby:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   433
            s.append('ORDERBY %s' % ', '.join(self.visit_sortterm(term)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   434
                                              for term in node.orderby))
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   435
        if node.limit is not None:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   436
            s.append('LIMIT %s' % node.limit)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   437
        if node.offset:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   438
            s.append('OFFSET %s' % node.offset)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   439
        restrictions = []
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   440
        if node.where is not None:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   441
            nr = node.where.accept(self)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   442
            if nr is not None:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   443
                restrictions.append(nr)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   444
        if restrictions:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   445
            s.append('WHERE %s' % ','.join(restrictions))
1791
c77629112437 clear cache before synchronization
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   446
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   447
        if node.having:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   448
            s.append('HAVING %s' % ', '.join(term.accept(self)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   449
                                             for term in node.having))
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   450
        subqueries = []
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   451
        for subquery in node.with_:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   452
            subqueries.append('%s BEING (%s)' % (','.join(ca.name for ca in subquery.aliases),
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   453
                                                 self.visit_union(subquery.query)))
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   454
        if subqueries:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   455
            s.append('WITH %s' % (','.join(subqueries)))
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   456
        return ' '.join(s)
1791
c77629112437 clear cache before synchronization
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   457
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   458
    def visit_and(self, node):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   459
        res = self._accept_children(node)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   460
        if res:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   461
            return ', '.join(res)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   462
        return
1791
c77629112437 clear cache before synchronization
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   463
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   464
    def visit_or(self, node):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   465
        res = self._accept_children(node)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   466
        if len(res) > 1:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   467
            return ' OR '.join('(%s)' % rql for rql in res)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   468
        elif res:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   469
            return res[0]
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   470
        return
1791
c77629112437 clear cache before synchronization
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   471
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   472
    def visit_not(self, node):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   473
        rql = node.children[0].accept(self)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   474
        if rql:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   475
            return 'NOT (%s)' % rql
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   476
        return
1791
c77629112437 clear cache before synchronization
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   477
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   478
    def visit_exists(self, node):
5582
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   479
        rql = node.children[0].accept(self)
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   480
        if rql:
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   481
            return 'EXISTS(%s)' % rql
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   482
        return
1791
c77629112437 clear cache before synchronization
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   483
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   484
    def visit_relation(self, node):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   485
        try:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   486
            if isinstance(node.children[0], Constant):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   487
                # simplified rqlst, reintroduce eid relation
391
42693fe3ef6f skip neged relation here as well
sylvain.thenault@logilab.fr
parents: 390
diff changeset
   488
                try:
42693fe3ef6f skip neged relation here as well
sylvain.thenault@logilab.fr
parents: 390
diff changeset
   489
                    restr, lhs = self.process_eid_const(node.children[0])
42693fe3ef6f skip neged relation here as well
sylvain.thenault@logilab.fr
parents: 390
diff changeset
   490
                except UnknownEid:
42693fe3ef6f skip neged relation here as well
sylvain.thenault@logilab.fr
parents: 390
diff changeset
   491
                    # can safely skip not relation with an unsupported eid
5582
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   492
                    if neged_relation(node):
391
42693fe3ef6f skip neged relation here as well
sylvain.thenault@logilab.fr
parents: 390
diff changeset
   493
                        return
42693fe3ef6f skip neged relation here as well
sylvain.thenault@logilab.fr
parents: 390
diff changeset
   494
                    raise
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   495
            else:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   496
                lhs = node.children[0].accept(self)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   497
                restr = None
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   498
        except UnknownEid:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   499
            # can safely skip not relation with an unsupported eid
5582
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   500
            if neged_relation(node):
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   501
                return
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   502
            # XXX what about optional relation or outer NOT EXISTS()
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   503
            raise
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   504
        if node.optional in ('left', 'both'):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   505
            lhs += '?'
3689
deb13e88e037 follow yams 0.25 api changes to improve performance
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3605
diff changeset
   506
        if node.r_type == 'eid' or not self.source.schema.rschema(node.r_type).final:
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   507
            self.need_translation = True
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   508
            self.current_operator = node.operator()
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   509
            if isinstance(node.children[0], Constant):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   510
                self.current_etypes = (node.children[0].uidtype,)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   511
            else:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   512
                self.current_etypes = node.children[0].variable.stinfo['possibletypes']
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   513
        try:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   514
            rhs = node.children[1].accept(self)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   515
        except UnknownEid:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   516
            # can safely skip not relation with an unsupported eid
5582
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   517
            if neged_relation(node):
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   518
                return
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   519
            # XXX what about optional relation or outer NOT EXISTS()
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   520
            raise
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   521
        except ReplaceByInOperator, ex:
1952
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1805
diff changeset
   522
            rhs = 'IN (%s)' % ','.join(eid for eid in ex.eids)
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   523
        self.need_translation = False
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   524
        self.current_operator = None
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   525
        if node.optional in ('right', 'both'):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   526
            rhs += '?'
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   527
        if restr is not None:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   528
            return '%s %s %s, %s' % (lhs, node.r_type, rhs, restr)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   529
        return '%s %s %s' % (lhs, node.r_type, rhs)
1791
c77629112437 clear cache before synchronization
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   530
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   531
    def visit_comparison(self, node):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   532
        if node.operator in ('=', 'IS'):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   533
            return node.children[0].accept(self)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   534
        return '%s %s' % (node.operator.encode(),
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   535
                          node.children[0].accept(self))
1791
c77629112437 clear cache before synchronization
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   536
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   537
    def visit_mathexpression(self, node):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   538
        return '(%s %s %s)' % (node.children[0].accept(self),
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   539
                               node.operator.encode(),
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   540
                               node.children[1].accept(self))
1791
c77629112437 clear cache before synchronization
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   541
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   542
    def visit_function(self, node):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   543
        #if node.name == 'IN':
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   544
        res = []
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   545
        for child in node.children:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   546
            try:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   547
                rql = child.accept(self)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   548
            except UnknownEid, ex:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   549
                continue
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   550
            res.append(rql)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   551
        if not res:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   552
            raise ex
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   553
        return '%s(%s)' % (node.name, ', '.join(res))
1791
c77629112437 clear cache before synchronization
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   554
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   555
    def visit_constant(self, node):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   556
        if self.need_translation or node.uidtype:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   557
            if node.type == 'Int':
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   558
                return str(self.eid2extid(node.value))
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   559
            if node.type == 'Substitute':
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   560
                key = node.value
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   561
                # ensure we have not yet translated the value...
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   562
                if not key in self._const_var:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   563
                    self.kwargs[key] = self.eid2extid(self.kwargs[key])
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   564
                    self._const_var[key] = None
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   565
        return node.as_string()
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   566
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   567
    def visit_variableref(self, node):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   568
        """get the sql name for a variable reference"""
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   569
        return node.name
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   570
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   571
    def visit_sortterm(self, node):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   572
        if node.asc:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   573
            return node.term.accept(self)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   574
        return '%s DESC' % node.term.accept(self)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   575
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   576
    def process_eid_const(self, const):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   577
        value = const.eval(self.kwargs)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   578
        try:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   579
            return None, self._const_var[value]
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   580
        except:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   581
            var = self._varmaker.next()
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   582
            self.need_translation = True
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   583
            restr = '%s eid %s' % (var, self.visit_constant(const))
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   584
            self.need_translation = False
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   585
            self._const_var[value] = var
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   586
            return restr, var
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   587
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   588
    def eid2extid(self, eid):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   589
        try:
1791
c77629112437 clear cache before synchronization
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   590
            return self.source.eid2extid(eid, self._session)
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   591
        except UnknownEid:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   592
            operator = self.current_operator
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   593
            if operator is not None and operator != '=':
1954
9b20f3504af8 cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1952
diff changeset
   594
                # deal with query like "X eid > 12"
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   595
                #
1954
9b20f3504af8 cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1952
diff changeset
   596
                # The problem is that eid order in the external source may
9b20f3504af8 cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1952
diff changeset
   597
                # differ from the local source
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   598
                #
1954
9b20f3504af8 cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1952
diff changeset
   599
                # So search for all eids from this source matching the condition
9b20f3504af8 cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1952
diff changeset
   600
                # locally and then to replace the "> 12" branch by "IN (eids)"
9b20f3504af8 cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1952
diff changeset
   601
                #
9b20f3504af8 cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1952
diff changeset
   602
                # XXX we may have to insert a huge number of eids...)
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   603
                sql = "SELECT extid FROM entities WHERE source='%s' AND type IN (%s) AND eid%s%s"
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   604
                etypes = ','.join("'%s'" % etype for etype in self.current_etypes)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   605
                cu = self._session.system_sql(sql % (self.source.uri, etypes,
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   606
                                                      operator, eid))
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   607
                # XXX buggy cu.rowcount which may be zero while there are some
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   608
                # results
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   609
                rows = cu.fetchall()
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   610
                if rows:
1952
8e19c813750d fix extid handling: ensure encoded string is given, and store them as base64 (see note in native.py).
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1805
diff changeset
   611
                    raise ReplaceByInOperator((b64decode(r[0]) for r in rows))
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   612
            raise
1791
c77629112437 clear cache before synchronization
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   613