cubicweb/server/sources/__init__.py
author Sylvain Thénault <sylvain.thenault@logilab.fr>
Wed, 12 Apr 2017 16:12:51 +0200
branch3.25
changeset 12155 4ba8fd2cd41a
parent 12153 0ff0aff4413d
child 12156 0d1d22a3a48b
permissions -rw-r--r--
[sources] Turn classmethod into standard method No need for a class method and easier to get proper logger including source's name.
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
11767
432f87a63057 flake8 and all
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11763
diff changeset
     1
# copyright 2003-2016 LOGILAB S.A. (Paris, FRANCE), all rights reserved.
5421
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5124
diff changeset
     2
# contact http://www.logilab.fr/ -- mailto:contact@logilab.fr
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5124
diff changeset
     3
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5124
diff changeset
     4
# This file is part of CubicWeb.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5124
diff changeset
     5
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5124
diff changeset
     6
# CubicWeb is free software: you can redistribute it and/or modify it under the
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5124
diff changeset
     7
# terms of the GNU Lesser General Public License as published by the Free
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5124
diff changeset
     8
# Software Foundation, either version 2.1 of the License, or (at your option)
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5124
diff changeset
     9
# any later version.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5124
diff changeset
    10
#
5424
8ecbcbff9777 replace logilab-common by CubicWeb in disclaimer
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5421
diff changeset
    11
# CubicWeb is distributed in the hope that it will be useful, but WITHOUT
5421
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5124
diff changeset
    12
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5124
diff changeset
    13
# FOR A PARTICULAR PURPOSE.  See the GNU Lesser General Public License for more
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5124
diff changeset
    14
# details.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5124
diff changeset
    15
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5124
diff changeset
    16
# You should have received a copy of the GNU Lesser General Public License along
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5124
diff changeset
    17
# with CubicWeb.  If not, see <http://www.gnu.org/licenses/>.
5824
de9b7e88660e cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5642
diff changeset
    18
"""cubicweb server sources support"""
11767
432f87a63057 flake8 and all
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11763
diff changeset
    19
10589
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 10580
diff changeset
    20
from __future__ import print_function
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    21
9180
13ed6de41774 [querier] Add timings to debug prints (DBG_RQL)
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 8892
diff changeset
    22
from time import time
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    23
from logging import getLogger
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    24
10622
3cc6154b94a3 [py3k] unicode → six.text_type
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10589
diff changeset
    25
from six import text_type
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    26
6945
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
    27
from logilab.common import configuration
10832
2350424585f8 [server] unormalize source logger's name to avoid encoding errors
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 10651
diff changeset
    28
from logilab.common.textutils import unormalize
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    29
6945
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
    30
from yams.schema import role_name
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
    31
12149
649100470733 [sources] Stop translating validation error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12148
diff changeset
    32
from cubicweb import ValidationError, set_log_methods, server, _
10086
98bc2ca1a816 [source/native] session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9822
diff changeset
    33
from cubicweb.server import SOURCE_TYPES
1251
af40e615dc89 introduce a 'cw_' prefix on entity table and column names so we don't conflict with sql or DBMS specific keywords
sylvain.thenault@logilab.fr
parents: 1250
diff changeset
    34
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    35
11237
f32134dd0067 [repository] drop remanescence of old multi-sources code
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11057
diff changeset
    36
def dbg_st_search(uri, union, args, cachekey=None, prefix='rql for'):
2625
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2596
diff changeset
    37
    if server.DEBUG & server.DBG_RQL:
9180
13ed6de41774 [querier] Add timings to debug prints (DBG_RQL)
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 8892
diff changeset
    38
        global t
10589
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 10580
diff changeset
    39
        print('  %s %s source: %s' % (prefix, uri, repr(union.as_string())))
9180
13ed6de41774 [querier] Add timings to debug prints (DBG_RQL)
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 8892
diff changeset
    40
        t = time()
2625
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2596
diff changeset
    41
        if server.DEBUG & server.DBG_MORE:
10589
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 10580
diff changeset
    42
            print('    args', repr(args))
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 10580
diff changeset
    43
            print('    cache key', cachekey)
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 10580
diff changeset
    44
            print('    solutions', ','.join(str(s.solutions)
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 10580
diff changeset
    45
                                            for s in union.children))
2625
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2596
diff changeset
    46
    # return true so it can be used as assertion (and so be killed by python -O)
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2596
diff changeset
    47
    return True
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2596
diff changeset
    48
11767
432f87a63057 flake8 and all
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11763
diff changeset
    49
2625
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2596
diff changeset
    50
def dbg_results(results):
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2596
diff changeset
    51
    if server.DEBUG & server.DBG_RQL:
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2596
diff changeset
    52
        if len(results) > 10:
10589
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 10580
diff changeset
    53
            print('  -->', results[:10], '...', len(results), end=' ')
2625
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2596
diff changeset
    54
        else:
10589
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 10580
diff changeset
    55
            print('  -->', results, end=' ')
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 10580
diff changeset
    56
        print('time: ', time() - t)
2625
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2596
diff changeset
    57
    # return true so it can be used as assertion (and so be killed by python -O)
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2596
diff changeset
    58
    return True
2493
9806571ea790 major refactoring of database dump/restore:
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2476
diff changeset
    59
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    60
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    61
class AbstractSource(object):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    62
    """an abstract class for sources"""
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    63
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    64
    # boolean telling if the repository should connect to this source during
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    65
    # migration
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    66
    connect_for_migration = True
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
    67
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    68
    # a global identifier for this source, which has to be set by the source
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    69
    # instance
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    70
    uri = None
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    71
    # a reference to the system information helper
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    72
    repo = None
2476
1294a6bdf3bf application -> instance where it makes sense
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1977
diff changeset
    73
    # a reference to the instance'schema (may differs from the source'schema)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    74
    schema = None
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
    75
6427
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6162
diff changeset
    76
    # force deactivation (configuration error for instance)
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6162
diff changeset
    77
    disabled = False
3040
e4dc22040f5f [multi-sources] move [dont_]cross_relations to AbstractSource
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2759
diff changeset
    78
7552
82dde8276a5b [datafeed, entities] url for entities from a datafeed source should be on their origin site. Closes #1769391
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7514
diff changeset
    79
    # boolean telling if cwuri of entities from this source is the url that
82dde8276a5b [datafeed, entities] url for entities from a datafeed source should be on their origin site. Closes #1769391
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7514
diff changeset
    80
    # should be used as entity's absolute url
82dde8276a5b [datafeed, entities] url for entities from a datafeed source should be on their origin site. Closes #1769391
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7514
diff changeset
    81
    use_cwuri_as_url = False
82dde8276a5b [datafeed, entities] url for entities from a datafeed source should be on their origin site. Closes #1769391
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7514
diff changeset
    82
6945
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
    83
    # source configuration options
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
    84
    options = ()
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
    85
7879
9aae456abab5 [pylint] fix pylint detected errors and tweak it so that pylint -E will be much less verbose next time (+ update some copyrights on the way)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7815
diff changeset
    86
    # these are overridden by set_log_methods below
9aae456abab5 [pylint] fix pylint detected errors and tweak it so that pylint -E will be much less verbose next time (+ update some copyrights on the way)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7815
diff changeset
    87
    # only defining here to prevent pylint from complaining
9aae456abab5 [pylint] fix pylint detected errors and tweak it so that pylint -E will be much less verbose next time (+ update some copyrights on the way)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7815
diff changeset
    88
    info = warning = error = critical = exception = debug = lambda msg,*a,**kw: None
9aae456abab5 [pylint] fix pylint detected errors and tweak it so that pylint -E will be much less verbose next time (+ update some copyrights on the way)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7815
diff changeset
    89
6945
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
    90
    def __init__(self, repo, source_config, eid=None):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    91
        self.repo = repo
6427
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6162
diff changeset
    92
        self.set_schema(repo.schema)
6945
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
    93
        self.eid = eid
6722
3341521d857b [repo source] rename attribute so it's much easier to grasp its role
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6582
diff changeset
    94
        self.public_config = source_config.copy()
9822
4a118bfd6ab4 [datafeed] Allow to override use_cwuri_as_url in configuration of a datafeed source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9662
diff changeset
    95
        self.public_config['use-cwuri-as-url'] = self.use_cwuri_as_url
6722
3341521d857b [repo source] rename attribute so it's much easier to grasp its role
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6582
diff changeset
    96
        self.remove_sensitive_information(self.public_config)
6945
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
    97
        self.uri = source_config.pop('uri')
10832
2350424585f8 [server] unormalize source logger's name to avoid encoding errors
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 10651
diff changeset
    98
        # unormalize to avoid non-ascii characters in logger's name, this will cause decoding error
2350424585f8 [server] unormalize source logger's name to avoid encoding errors
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 10651
diff changeset
    99
        # on logging
10845
75c60e58ce6b [py3k] Fix unicode added in commit 2350424585f8
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10832
diff changeset
   100
        set_log_methods(self, getLogger('cubicweb.sources.' + unormalize(text_type(self.uri))))
6945
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   101
        source_config.pop('type')
12150
6260c88e0ff5 [sources] Enhance prototype of check_conf_dict
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12149
diff changeset
   102
        self.config = self._check_config_dict(
6260c88e0ff5 [sources] Enhance prototype of check_conf_dict
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12149
diff changeset
   103
            eid, source_config, raise_on_error=False)
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   104
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   105
    def __repr__(self):
8228
fe53c7e1625f [sources] include source's class in its repr
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8134
diff changeset
   106
        return '<%s %s source %s @%#x>' % (self.uri, self.__class__.__name__,
fe53c7e1625f [sources] include source's class in its repr
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8134
diff changeset
   107
                                           self.eid, id(self))
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   108
8892
80783605d270 [toward-py3k] rewrite __cmp__ (closes #2715115)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8696
diff changeset
   109
    def __lt__(self, other):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   110
        """simple comparison function to get predictable source order, with the
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   111
        system source at last
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   112
        """
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   113
        if self.uri == other.uri:
8892
80783605d270 [toward-py3k] rewrite __cmp__ (closes #2715115)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8696
diff changeset
   114
            return False
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   115
        if self.uri == 'system':
8892
80783605d270 [toward-py3k] rewrite __cmp__ (closes #2715115)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8696
diff changeset
   116
            return False
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   117
        if other.uri == 'system':
8892
80783605d270 [toward-py3k] rewrite __cmp__ (closes #2715115)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8696
diff changeset
   118
            return True
80783605d270 [toward-py3k] rewrite __cmp__ (closes #2715115)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8696
diff changeset
   119
        return self.uri < other.uri
80783605d270 [toward-py3k] rewrite __cmp__ (closes #2715115)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8696
diff changeset
   120
80783605d270 [toward-py3k] rewrite __cmp__ (closes #2715115)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8696
diff changeset
   121
    def __eq__(self, other):
80783605d270 [toward-py3k] rewrite __cmp__ (closes #2715115)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8696
diff changeset
   122
        return self.uri == other.uri
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   123
10576
1ee631aedf2f [server/sources] AbstractSource.__ne__
Julien Cristau <julien.cristau@logilab.fr>
parents: 10575
diff changeset
   124
    def __ne__(self, other):
1ee631aedf2f [server/sources] AbstractSource.__ne__
Julien Cristau <julien.cristau@logilab.fr>
parents: 10575
diff changeset
   125
        return not (self == other)
1ee631aedf2f [server/sources] AbstractSource.__ne__
Julien Cristau <julien.cristau@logilab.fr>
parents: 10575
diff changeset
   126
7342
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7237
diff changeset
   127
    def backup(self, backupfile, confirm, format='native'):
6943
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   128
        """method called to create a backup of source's data"""
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   129
        pass
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   130
7342
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7237
diff changeset
   131
    def restore(self, backupfile, confirm, drop, format='native'):
6943
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   132
        """method called to restore a backup of source's data"""
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   133
        pass
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   134
12155
4ba8fd2cd41a [sources] Turn classmethod into standard method
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12153
diff changeset
   135
    def _check_config_dict(self, eid, confdict, raise_on_error=True):
12150
6260c88e0ff5 [sources] Enhance prototype of check_conf_dict
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12149
diff changeset
   136
        """Check configuration of source entity and return config dict properly
6945
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   137
        typed with defaults set.
12150
6260c88e0ff5 [sources] Enhance prototype of check_conf_dict
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12149
diff changeset
   138
6260c88e0ff5 [sources] Enhance prototype of check_conf_dict
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12149
diff changeset
   139
        If `raise_on_error` is True (the default), a ValidationError will be
6260c88e0ff5 [sources] Enhance prototype of check_conf_dict
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12149
diff changeset
   140
        raised if some error is encountered, else the problem will be ignored.
6945
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   141
        """
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   142
        processed = {}
12155
4ba8fd2cd41a [sources] Turn classmethod into standard method
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12153
diff changeset
   143
        for optname, optdict in self.options:
6945
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   144
            value = confdict.pop(optname, optdict.get('default'))
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   145
            if value is configuration.REQUIRED:
12150
6260c88e0ff5 [sources] Enhance prototype of check_conf_dict
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12149
diff changeset
   146
                if not raise_on_error:
6945
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   147
                    continue
12149
649100470733 [sources] Stop translating validation error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12148
diff changeset
   148
                msg = _('specifying %s is mandatory')
649100470733 [sources] Stop translating validation error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12148
diff changeset
   149
                msgargs = optname
649100470733 [sources] Stop translating validation error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12148
diff changeset
   150
                raise ValidationError(eid, {role_name('config', 'subject'): msg}, msgargs)
6945
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   151
            elif value is not None:
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   152
                # type check
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   153
                try:
9292
4b3e657d17ab [source configuration] silent lgc 0.60 deprecation warning
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9180
diff changeset
   154
                    value = configuration._validate(value, optdict, optname)
8695
358d8bed9626 [toward-py3k] rewrite to "except AnException as exc:" (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8674
diff changeset
   155
                except Exception as ex:
12150
6260c88e0ff5 [sources] Enhance prototype of check_conf_dict
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12149
diff changeset
   156
                    if not raise_on_error:
6260c88e0ff5 [sources] Enhance prototype of check_conf_dict
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12149
diff changeset
   157
                        continue
12149
649100470733 [sources] Stop translating validation error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12148
diff changeset
   158
                    msg = text_type(ex)
6945
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   159
                    raise ValidationError(eid, {role_name('config', 'subject'): msg})
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   160
            processed[optname] = value
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   161
        # cw < 3.10 bw compat
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   162
        try:
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   163
            processed['adapter'] = confdict['adapter']
7815
2a164a9cf81c [exceptions] stop catching any exception in various places (closes #1942716)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7779
diff changeset
   164
        except KeyError:
6945
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   165
            pass
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   166
        # check for unknown options
8696
0bb18407c053 [toward py3k] rewrite dict.keys() and dict.values() (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8695
diff changeset
   167
        if confdict and tuple(confdict) != ('adapter',):
12150
6260c88e0ff5 [sources] Enhance prototype of check_conf_dict
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12149
diff changeset
   168
            if raise_on_error:
12149
649100470733 [sources] Stop translating validation error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12148
diff changeset
   169
                msg = _('unknown options %s')
649100470733 [sources] Stop translating validation error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12148
diff changeset
   170
                msgargs = ', '.join(confdict)
649100470733 [sources] Stop translating validation error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12148
diff changeset
   171
                raise ValidationError(eid, {role_name('config', 'subject'): msg}, msgargs)
6945
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   172
            else:
12155
4ba8fd2cd41a [sources] Turn classmethod into standard method
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12153
diff changeset
   173
                self.warning('unknown options %s', ', '.join(confdict))
6945
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   174
                # add options to processed, they may be necessary during migration
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   175
                processed.update(confdict)
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   176
        return processed
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   177
12155
4ba8fd2cd41a [sources] Turn classmethod into standard method
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12153
diff changeset
   178
    def check_config(self, source_entity):
12150
6260c88e0ff5 [sources] Enhance prototype of check_conf_dict
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12149
diff changeset
   179
        """Check configuration of source entity, raise ValidationError if some
6260c88e0ff5 [sources] Enhance prototype of check_conf_dict
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12149
diff changeset
   180
        errors are detected.
6260c88e0ff5 [sources] Enhance prototype of check_conf_dict
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12149
diff changeset
   181
        """
12155
4ba8fd2cd41a [sources] Turn classmethod into standard method
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12153
diff changeset
   182
        return self._check_config_dict(source_entity.eid, source_entity.dictconfig)
6945
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   183
12153
0ff0aff4413d [sources] Check source's url attribute value on creation/modification
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12150
diff changeset
   184
    def check_urls(self, source_entity):
0ff0aff4413d [sources] Check source's url attribute value on creation/modification
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12150
diff changeset
   185
        """Check URL of source entity: `urls` is a string that may contain one
0ff0aff4413d [sources] Check source's url attribute value on creation/modification
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12150
diff changeset
   186
        URL per line), and return a list of at least one validated URL.
0ff0aff4413d [sources] Check source's url attribute value on creation/modification
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12150
diff changeset
   187
        """
0ff0aff4413d [sources] Check source's url attribute value on creation/modification
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12150
diff changeset
   188
        urls = source_entity.url if source_entity.url else ''
0ff0aff4413d [sources] Check source's url attribute value on creation/modification
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12150
diff changeset
   189
        urls = [url.strip() for url in urls.splitlines() if url.strip()]
0ff0aff4413d [sources] Check source's url attribute value on creation/modification
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12150
diff changeset
   190
        if not urls:
0ff0aff4413d [sources] Check source's url attribute value on creation/modification
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12150
diff changeset
   191
            msg = _('specifying an URL is mandatory')
0ff0aff4413d [sources] Check source's url attribute value on creation/modification
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12150
diff changeset
   192
            raise ValidationError(source_entity.eid, {role_name('url', 'subject'): msg})
0ff0aff4413d [sources] Check source's url attribute value on creation/modification
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12150
diff changeset
   193
        return urls
0ff0aff4413d [sources] Check source's url attribute value on creation/modification
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12150
diff changeset
   194
6943
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   195
    # source initialization / finalization #####################################
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   196
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   197
    def set_schema(self, schema):
2476
1294a6bdf3bf application -> instance where it makes sense
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1977
diff changeset
   198
        """set the instance'schema"""
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   199
        self.schema = schema
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   200
6943
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   201
    def init_creating(self):
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   202
        """method called by the repository once ready to create a new instance"""
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   203
        pass
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   204
12148
79160d54662e [sources] Simplify source's init method
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12144
diff changeset
   205
    def init(self, source_entity):
6943
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   206
        """method called by the repository once ready to handle request.
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   207
        `activated` is a boolean flag telling if the source is activated or not.
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   208
        """
12144
f54286c1cef5 [server] Inline _entity_update method into init method of AbstractSource
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12143
diff changeset
   209
        source_entity.complete()
f54286c1cef5 [server] Inline _entity_update method into init method of AbstractSource
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12143
diff changeset
   210
        if source_entity.url:
12153
0ff0aff4413d [sources] Check source's url attribute value on creation/modification
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12150
diff changeset
   211
            self.urls = self.check_urls(source_entity)
12144
f54286c1cef5 [server] Inline _entity_update method into init method of AbstractSource
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12143
diff changeset
   212
        else:
f54286c1cef5 [server] Inline _entity_update method into init method of AbstractSource
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12143
diff changeset
   213
            self.urls = []
6943
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   214
7552
82dde8276a5b [datafeed, entities] url for entities from a datafeed source should be on their origin site. Closes #1769391
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7514
diff changeset
   215
    PUBLIC_KEYS = ('type', 'uri', 'use-cwuri-as-url')
6943
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   216
    def remove_sensitive_information(self, sourcedef):
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   217
        """remove sensitive information such as login / password from source
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   218
        definition
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   219
        """
8696
0bb18407c053 [toward py3k] rewrite dict.keys() and dict.values() (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8695
diff changeset
   220
        for key in list(sourcedef):
6943
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   221
            if not key in self.PUBLIC_KEYS:
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   222
                sourcedef.pop(key)
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   223
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   224
    # connections handling #####################################################
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   225
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   226
    def get_connection(self):
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   227
        """open and return a connection to the source"""
8134
7f93da785e3a add instance to not implemented error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7922
diff changeset
   228
        raise NotImplementedError(self)
6943
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   229
7398
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   230
    def close_source_connections(self):
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   231
        for cnxset in self.repo.cnxsets:
9463
d62e13eba033 [multi-sources-removal] Simplify ConnectionsSet internal structures and public methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9452
diff changeset
   232
            cnxset.cu = None
d62e13eba033 [multi-sources-removal] Simplify ConnectionsSet internal structures and public methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9452
diff changeset
   233
            cnxset.cnx.close()
6943
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   234
7398
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   235
    def open_source_connections(self):
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   236
        for cnxset in self.repo.cnxsets:
9463
d62e13eba033 [multi-sources-removal] Simplify ConnectionsSet internal structures and public methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9452
diff changeset
   237
            cnxset.cnx = self.get_connection()
d62e13eba033 [multi-sources-removal] Simplify ConnectionsSet internal structures and public methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9452
diff changeset
   238
            cnxset.cu = cnxset.cnx.cursor()
6943
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   239
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   240
    # user authentication api ##################################################
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   241
9512
88dc96fc9fc1 [server] use a connection instead of a session for user authentication
Julien Cristau <julien.cristau@logilab.fr>
parents: 9469
diff changeset
   242
    def authenticate(self, cnx, login, **kwargs):
6152
6824f8b61098 use is_instance in a number of places (esp. documentation) rather than the deprecated implements
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6057
diff changeset
   243
        """if the source support CWUser entity type, it should implement
1398
5fe84a5f7035 rename internal entity types to have CW prefix instead of E
sylvain.thenault@logilab.fr
parents: 1263
diff changeset
   244
        this method which should return CWUser eid for the given login/password
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   245
        if this account is defined in this source and valid login / password is
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   246
        given. Else raise `AuthenticationError`
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   247
        """
8134
7f93da785e3a add instance to not implemented error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7922
diff changeset
   248
        raise NotImplementedError(self)
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   249
6943
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   250
    # RQL query api ############################################################
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   251
9512
88dc96fc9fc1 [server] use a connection instead of a session for user authentication
Julien Cristau <julien.cristau@logilab.fr>
parents: 9469
diff changeset
   252
    def syntax_tree_search(self, cnx, union,
11237
f32134dd0067 [repository] drop remanescence of old multi-sources code
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11057
diff changeset
   253
                           args=None, cachekey=None, debug=0):
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   254
        """return result from this source for a rql query (actually from a rql
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   255
        syntax tree and a solution dictionary mapping each used variable to a
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   256
        possible type). If cachekey is given, the query necessary to fetch the
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   257
        results (but not the results themselves) may be cached using this key.
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   258
        """
8134
7f93da785e3a add instance to not implemented error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7922
diff changeset
   259
        raise NotImplementedError(self)
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   260
6943
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   261
    # write modification api ###################################################
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   262
    # read-only sources don't have to implement methods below
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   263
10086
98bc2ca1a816 [source/native] session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9822
diff changeset
   264
    def add_entity(self, cnx, entity):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   265
        """add a new entity to the source"""
8134
7f93da785e3a add instance to not implemented error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7922
diff changeset
   266
        raise NotImplementedError(self)
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   267
10086
98bc2ca1a816 [source/native] session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9822
diff changeset
   268
    def update_entity(self, cnx, entity):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   269
        """update an entity in the source"""
8134
7f93da785e3a add instance to not implemented error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7922
diff changeset
   270
        raise NotImplementedError(self)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   271
10086
98bc2ca1a816 [source/native] session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9822
diff changeset
   272
    def delete_entities(self, cnx, entities):
6889
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6722
diff changeset
   273
        """delete several entities from the source"""
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6722
diff changeset
   274
        for entity in entities:
10086
98bc2ca1a816 [source/native] session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9822
diff changeset
   275
            self.delete_entity(cnx, entity)
6889
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6722
diff changeset
   276
10086
98bc2ca1a816 [source/native] session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9822
diff changeset
   277
    def delete_entity(self, cnx, entity):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   278
        """delete an entity from the source"""
8134
7f93da785e3a add instance to not implemented error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7922
diff changeset
   279
        raise NotImplementedError(self)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   280
10086
98bc2ca1a816 [source/native] session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9822
diff changeset
   281
    def add_relation(self, cnx, subject, rtype, object):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   282
        """add a relation to the source"""
8134
7f93da785e3a add instance to not implemented error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7922
diff changeset
   283
        raise NotImplementedError(self)
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   284
10086
98bc2ca1a816 [source/native] session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9822
diff changeset
   285
    def add_relations(self, cnx,  rtype, subj_obj_list):
7237
9f619715665b [server] improve the speed of setting relations between entities (closes #1625257)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7118
diff changeset
   286
        """add a relations to the source"""
9f619715665b [server] improve the speed of setting relations between entities (closes #1625257)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7118
diff changeset
   287
        # override in derived classes if you feel you can
9f619715665b [server] improve the speed of setting relations between entities (closes #1625257)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7118
diff changeset
   288
        # optimize
9f619715665b [server] improve the speed of setting relations between entities (closes #1625257)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7118
diff changeset
   289
        for subject, object in subj_obj_list:
10086
98bc2ca1a816 [source/native] session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9822
diff changeset
   290
            self.add_relation(cnx, subject, rtype, object)
7237
9f619715665b [server] improve the speed of setting relations between entities (closes #1625257)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7118
diff changeset
   291
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   292
    def delete_relation(self, session, subject, rtype, object):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   293
        """delete a relation from the source"""
8134
7f93da785e3a add instance to not implemented error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7922
diff changeset
   294
        raise NotImplementedError(self)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   295
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   296
    # system source interface #################################################
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   297
11774
51c160677afe [repository] Drop the entities.extid column and associated cache
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11767
diff changeset
   298
    def eid_type(self, cnx, eid):
51c160677afe [repository] Drop the entities.extid column and associated cache
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11767
diff changeset
   299
        """Return the type of entity `eid`."""
8134
7f93da785e3a add instance to not implemented error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7922
diff changeset
   300
        raise NotImplementedError(self)
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   301
10086
98bc2ca1a816 [source/native] session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9822
diff changeset
   302
    def create_eid(self, cnx):
8134
7f93da785e3a add instance to not implemented error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7922
diff changeset
   303
        raise NotImplementedError(self)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   304
11774
51c160677afe [repository] Drop the entities.extid column and associated cache
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11767
diff changeset
   305
    def add_info(self, cnx, entity, source):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   306
        """add type and source info for an eid into the system table"""
8134
7f93da785e3a add instance to not implemented error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7922
diff changeset
   307
        raise NotImplementedError(self)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   308
10086
98bc2ca1a816 [source/native] session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9822
diff changeset
   309
    def update_info(self, cnx, entity, need_fti_update):
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4807
diff changeset
   310
        """mark entity as being modified, fulltext reindex if needed"""
8134
7f93da785e3a add instance to not implemented error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7922
diff changeset
   311
        raise NotImplementedError(self)
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4807
diff changeset
   312
10086
98bc2ca1a816 [source/native] session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9822
diff changeset
   313
    def index_entity(self, cnx, entity):
4807
5642bfa43236 [cleanup] add index_entity to abstract source, add docstring
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4252
diff changeset
   314
        """create an operation to [re]index textual content of the given entity
5642bfa43236 [cleanup] add index_entity to abstract source, add docstring
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4252
diff changeset
   315
        on commit
5642bfa43236 [cleanup] add index_entity to abstract source, add docstring
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4252
diff changeset
   316
        """
8134
7f93da785e3a add instance to not implemented error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7922
diff changeset
   317
        raise NotImplementedError(self)
4807
5642bfa43236 [cleanup] add index_entity to abstract source, add docstring
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4252
diff changeset
   318
10086
98bc2ca1a816 [source/native] session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9822
diff changeset
   319
    def fti_unindex_entities(self, cnx, entities):
6889
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6722
diff changeset
   320
        """remove text content for entities from the full text index
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   321
        """
8134
7f93da785e3a add instance to not implemented error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7922
diff changeset
   322
        raise NotImplementedError(self)
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   323
10086
98bc2ca1a816 [source/native] session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9822
diff changeset
   324
    def fti_index_entities(self, cnx, entities):
6889
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6722
diff changeset
   325
        """add text content of created/modified entities to the full text index
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   326
        """
8134
7f93da785e3a add instance to not implemented error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7922
diff changeset
   327
        raise NotImplementedError(self)
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   328
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   329
    # sql system source interface #############################################
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   330
10086
98bc2ca1a816 [source/native] session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9822
diff changeset
   331
    def sqlexec(self, cnx, sql, args=None):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   332
        """execute the query and return its result"""
8134
7f93da785e3a add instance to not implemented error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7922
diff changeset
   333
        raise NotImplementedError(self)
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   334
10086
98bc2ca1a816 [source/native] session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9822
diff changeset
   335
    def create_index(self, cnx, table, column, unique=False):
8134
7f93da785e3a add instance to not implemented error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7922
diff changeset
   336
        raise NotImplementedError(self)
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   337
10086
98bc2ca1a816 [source/native] session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9822
diff changeset
   338
    def drop_index(self, cnx, table, column, unique=False):
8134
7f93da785e3a add instance to not implemented error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7922
diff changeset
   339
        raise NotImplementedError(self)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   340
7697
ef50074a0314 [repo api] deprecates source.extid2eid, only the system source should implement it,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7552
diff changeset
   341
6427
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6162
diff changeset
   342
def source_adapter(source_type):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   343
    try:
6427
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6162
diff changeset
   344
        return SOURCE_TYPES[source_type]
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   345
    except KeyError:
6427
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6162
diff changeset
   346
        raise RuntimeError('Unknown source type %r' % source_type)
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   347
11767
432f87a63057 flake8 and all
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11763
diff changeset
   348
6945
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   349
def get_source(type, source_config, repo, eid):
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   350
    """return a source adapter according to the adapter field in the source's
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   351
    configuration
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   352
    """
6945
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   353
    return source_adapter(type)(repo, source_config, eid)