cubicweb/server/sources/__init__.py
author Denis Laxalde <denis.laxalde@logilab.fr>
Fri, 05 Apr 2019 17:58:19 +0200
changeset 12567 26744ad37953
parent 12156 0d1d22a3a48b
child 12739 c6f8ca03718f
permissions -rw-r--r--
Drop python2 support This mostly consists in removing the dependency on "six" and updating the code to use only Python3 idioms. Notice that we previously used TemporaryDirectory from cubicweb.devtools.testlib for compatibility with Python2. We now directly import it from tempfile.
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
11767
432f87a63057 flake8 and all
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11763
diff changeset
     1
# copyright 2003-2016 LOGILAB S.A. (Paris, FRANCE), all rights reserved.
5421
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5124
diff changeset
     2
# contact http://www.logilab.fr/ -- mailto:contact@logilab.fr
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5124
diff changeset
     3
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5124
diff changeset
     4
# This file is part of CubicWeb.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5124
diff changeset
     5
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5124
diff changeset
     6
# CubicWeb is free software: you can redistribute it and/or modify it under the
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5124
diff changeset
     7
# terms of the GNU Lesser General Public License as published by the Free
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5124
diff changeset
     8
# Software Foundation, either version 2.1 of the License, or (at your option)
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5124
diff changeset
     9
# any later version.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5124
diff changeset
    10
#
5424
8ecbcbff9777 replace logilab-common by CubicWeb in disclaimer
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5421
diff changeset
    11
# CubicWeb is distributed in the hope that it will be useful, but WITHOUT
5421
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5124
diff changeset
    12
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5124
diff changeset
    13
# FOR A PARTICULAR PURPOSE.  See the GNU Lesser General Public License for more
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5124
diff changeset
    14
# details.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5124
diff changeset
    15
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5124
diff changeset
    16
# You should have received a copy of the GNU Lesser General Public License along
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5124
diff changeset
    17
# with CubicWeb.  If not, see <http://www.gnu.org/licenses/>.
5824
de9b7e88660e cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5642
diff changeset
    18
"""cubicweb server sources support"""
11767
432f87a63057 flake8 and all
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11763
diff changeset
    19
9180
13ed6de41774 [querier] Add timings to debug prints (DBG_RQL)
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 8892
diff changeset
    20
from time import time
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    21
from logging import getLogger
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    22
6945
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
    23
from logilab.common import configuration
10832
2350424585f8 [server] unormalize source logger's name to avoid encoding errors
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 10651
diff changeset
    24
from logilab.common.textutils import unormalize
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    25
6945
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
    26
from yams.schema import role_name
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
    27
12149
649100470733 [sources] Stop translating validation error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12148
diff changeset
    28
from cubicweb import ValidationError, set_log_methods, server, _
10086
98bc2ca1a816 [source/native] session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9822
diff changeset
    29
from cubicweb.server import SOURCE_TYPES
1251
af40e615dc89 introduce a 'cw_' prefix on entity table and column names so we don't conflict with sql or DBMS specific keywords
sylvain.thenault@logilab.fr
parents: 1250
diff changeset
    30
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    31
11237
f32134dd0067 [repository] drop remanescence of old multi-sources code
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11057
diff changeset
    32
def dbg_st_search(uri, union, args, cachekey=None, prefix='rql for'):
2625
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2596
diff changeset
    33
    if server.DEBUG & server.DBG_RQL:
9180
13ed6de41774 [querier] Add timings to debug prints (DBG_RQL)
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 8892
diff changeset
    34
        global t
10589
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 10580
diff changeset
    35
        print('  %s %s source: %s' % (prefix, uri, repr(union.as_string())))
9180
13ed6de41774 [querier] Add timings to debug prints (DBG_RQL)
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 8892
diff changeset
    36
        t = time()
2625
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2596
diff changeset
    37
        if server.DEBUG & server.DBG_MORE:
10589
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 10580
diff changeset
    38
            print('    args', repr(args))
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 10580
diff changeset
    39
            print('    cache key', cachekey)
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 10580
diff changeset
    40
            print('    solutions', ','.join(str(s.solutions)
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 10580
diff changeset
    41
                                            for s in union.children))
2625
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2596
diff changeset
    42
    # return true so it can be used as assertion (and so be killed by python -O)
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2596
diff changeset
    43
    return True
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2596
diff changeset
    44
11767
432f87a63057 flake8 and all
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11763
diff changeset
    45
2625
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2596
diff changeset
    46
def dbg_results(results):
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2596
diff changeset
    47
    if server.DEBUG & server.DBG_RQL:
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2596
diff changeset
    48
        if len(results) > 10:
10589
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 10580
diff changeset
    49
            print('  -->', results[:10], '...', len(results), end=' ')
2625
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2596
diff changeset
    50
        else:
10589
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 10580
diff changeset
    51
            print('  -->', results, end=' ')
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 10580
diff changeset
    52
        print('time: ', time() - t)
2625
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2596
diff changeset
    53
    # return true so it can be used as assertion (and so be killed by python -O)
d6012db7b93e R [server debug] more server side debugging tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2596
diff changeset
    54
    return True
2493
9806571ea790 major refactoring of database dump/restore:
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2476
diff changeset
    55
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    56
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    57
class AbstractSource(object):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    58
    """an abstract class for sources"""
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    59
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    60
    # boolean telling if the repository should connect to this source during
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    61
    # migration
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    62
    connect_for_migration = True
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
    63
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    64
    # a global identifier for this source, which has to be set by the source
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    65
    # instance
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    66
    uri = None
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    67
    # a reference to the system information helper
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    68
    repo = None
2476
1294a6bdf3bf application -> instance where it makes sense
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1977
diff changeset
    69
    # a reference to the instance'schema (may differs from the source'schema)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    70
    schema = None
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
    71
6427
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6162
diff changeset
    72
    # force deactivation (configuration error for instance)
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6162
diff changeset
    73
    disabled = False
3040
e4dc22040f5f [multi-sources] move [dont_]cross_relations to AbstractSource
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2759
diff changeset
    74
7552
82dde8276a5b [datafeed, entities] url for entities from a datafeed source should be on their origin site. Closes #1769391
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7514
diff changeset
    75
    # boolean telling if cwuri of entities from this source is the url that
82dde8276a5b [datafeed, entities] url for entities from a datafeed source should be on their origin site. Closes #1769391
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7514
diff changeset
    76
    # should be used as entity's absolute url
82dde8276a5b [datafeed, entities] url for entities from a datafeed source should be on their origin site. Closes #1769391
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7514
diff changeset
    77
    use_cwuri_as_url = False
82dde8276a5b [datafeed, entities] url for entities from a datafeed source should be on their origin site. Closes #1769391
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7514
diff changeset
    78
6945
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
    79
    # source configuration options
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
    80
    options = ()
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
    81
7879
9aae456abab5 [pylint] fix pylint detected errors and tweak it so that pylint -E will be much less verbose next time (+ update some copyrights on the way)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7815
diff changeset
    82
    # these are overridden by set_log_methods below
9aae456abab5 [pylint] fix pylint detected errors and tweak it so that pylint -E will be much less verbose next time (+ update some copyrights on the way)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7815
diff changeset
    83
    # only defining here to prevent pylint from complaining
12156
0d1d22a3a48b Fix some flake8 errors
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12155
diff changeset
    84
    info = warning = error = critical = exception = debug = lambda msg, *a, **kw: None
7879
9aae456abab5 [pylint] fix pylint detected errors and tweak it so that pylint -E will be much less verbose next time (+ update some copyrights on the way)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7815
diff changeset
    85
6945
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
    86
    def __init__(self, repo, source_config, eid=None):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    87
        self.repo = repo
6427
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6162
diff changeset
    88
        self.set_schema(repo.schema)
6945
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
    89
        self.eid = eid
6722
3341521d857b [repo source] rename attribute so it's much easier to grasp its role
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6582
diff changeset
    90
        self.public_config = source_config.copy()
9822
4a118bfd6ab4 [datafeed] Allow to override use_cwuri_as_url in configuration of a datafeed source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9662
diff changeset
    91
        self.public_config['use-cwuri-as-url'] = self.use_cwuri_as_url
6722
3341521d857b [repo source] rename attribute so it's much easier to grasp its role
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6582
diff changeset
    92
        self.remove_sensitive_information(self.public_config)
6945
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
    93
        self.uri = source_config.pop('uri')
10832
2350424585f8 [server] unormalize source logger's name to avoid encoding errors
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 10651
diff changeset
    94
        # unormalize to avoid non-ascii characters in logger's name, this will cause decoding error
2350424585f8 [server] unormalize source logger's name to avoid encoding errors
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 10651
diff changeset
    95
        # on logging
12567
26744ad37953 Drop python2 support
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12156
diff changeset
    96
        set_log_methods(self, getLogger('cubicweb.sources.' + unormalize(self.uri)))
6945
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
    97
        source_config.pop('type')
12150
6260c88e0ff5 [sources] Enhance prototype of check_conf_dict
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12149
diff changeset
    98
        self.config = self._check_config_dict(
6260c88e0ff5 [sources] Enhance prototype of check_conf_dict
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12149
diff changeset
    99
            eid, source_config, raise_on_error=False)
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   100
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   101
    def __repr__(self):
8228
fe53c7e1625f [sources] include source's class in its repr
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8134
diff changeset
   102
        return '<%s %s source %s @%#x>' % (self.uri, self.__class__.__name__,
fe53c7e1625f [sources] include source's class in its repr
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8134
diff changeset
   103
                                           self.eid, id(self))
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   104
8892
80783605d270 [toward-py3k] rewrite __cmp__ (closes #2715115)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8696
diff changeset
   105
    def __lt__(self, other):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   106
        """simple comparison function to get predictable source order, with the
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   107
        system source at last
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   108
        """
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   109
        if self.uri == other.uri:
8892
80783605d270 [toward-py3k] rewrite __cmp__ (closes #2715115)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8696
diff changeset
   110
            return False
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   111
        if self.uri == 'system':
8892
80783605d270 [toward-py3k] rewrite __cmp__ (closes #2715115)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8696
diff changeset
   112
            return False
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   113
        if other.uri == 'system':
8892
80783605d270 [toward-py3k] rewrite __cmp__ (closes #2715115)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8696
diff changeset
   114
            return True
80783605d270 [toward-py3k] rewrite __cmp__ (closes #2715115)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8696
diff changeset
   115
        return self.uri < other.uri
80783605d270 [toward-py3k] rewrite __cmp__ (closes #2715115)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8696
diff changeset
   116
80783605d270 [toward-py3k] rewrite __cmp__ (closes #2715115)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8696
diff changeset
   117
    def __eq__(self, other):
80783605d270 [toward-py3k] rewrite __cmp__ (closes #2715115)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8696
diff changeset
   118
        return self.uri == other.uri
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   119
10576
1ee631aedf2f [server/sources] AbstractSource.__ne__
Julien Cristau <julien.cristau@logilab.fr>
parents: 10575
diff changeset
   120
    def __ne__(self, other):
1ee631aedf2f [server/sources] AbstractSource.__ne__
Julien Cristau <julien.cristau@logilab.fr>
parents: 10575
diff changeset
   121
        return not (self == other)
1ee631aedf2f [server/sources] AbstractSource.__ne__
Julien Cristau <julien.cristau@logilab.fr>
parents: 10575
diff changeset
   122
7342
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7237
diff changeset
   123
    def backup(self, backupfile, confirm, format='native'):
6943
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   124
        """method called to create a backup of source's data"""
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   125
        pass
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   126
7342
d1c8b5b3531c adds support for a portable db import/export format (closes: #1521112)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7237
diff changeset
   127
    def restore(self, backupfile, confirm, drop, format='native'):
6943
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   128
        """method called to restore a backup of source's data"""
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   129
        pass
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   130
12155
4ba8fd2cd41a [sources] Turn classmethod into standard method
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12153
diff changeset
   131
    def _check_config_dict(self, eid, confdict, raise_on_error=True):
12150
6260c88e0ff5 [sources] Enhance prototype of check_conf_dict
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12149
diff changeset
   132
        """Check configuration of source entity and return config dict properly
6945
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   133
        typed with defaults set.
12150
6260c88e0ff5 [sources] Enhance prototype of check_conf_dict
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12149
diff changeset
   134
6260c88e0ff5 [sources] Enhance prototype of check_conf_dict
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12149
diff changeset
   135
        If `raise_on_error` is True (the default), a ValidationError will be
6260c88e0ff5 [sources] Enhance prototype of check_conf_dict
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12149
diff changeset
   136
        raised if some error is encountered, else the problem will be ignored.
6945
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   137
        """
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   138
        processed = {}
12155
4ba8fd2cd41a [sources] Turn classmethod into standard method
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12153
diff changeset
   139
        for optname, optdict in self.options:
6945
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   140
            value = confdict.pop(optname, optdict.get('default'))
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   141
            if value is configuration.REQUIRED:
12150
6260c88e0ff5 [sources] Enhance prototype of check_conf_dict
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12149
diff changeset
   142
                if not raise_on_error:
6945
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   143
                    continue
12149
649100470733 [sources] Stop translating validation error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12148
diff changeset
   144
                msg = _('specifying %s is mandatory')
649100470733 [sources] Stop translating validation error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12148
diff changeset
   145
                msgargs = optname
649100470733 [sources] Stop translating validation error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12148
diff changeset
   146
                raise ValidationError(eid, {role_name('config', 'subject'): msg}, msgargs)
6945
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   147
            elif value is not None:
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   148
                # type check
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   149
                try:
9292
4b3e657d17ab [source configuration] silent lgc 0.60 deprecation warning
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9180
diff changeset
   150
                    value = configuration._validate(value, optdict, optname)
8695
358d8bed9626 [toward-py3k] rewrite to "except AnException as exc:" (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8674
diff changeset
   151
                except Exception as ex:
12150
6260c88e0ff5 [sources] Enhance prototype of check_conf_dict
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12149
diff changeset
   152
                    if not raise_on_error:
6260c88e0ff5 [sources] Enhance prototype of check_conf_dict
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12149
diff changeset
   153
                        continue
12567
26744ad37953 Drop python2 support
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12156
diff changeset
   154
                    msg = str(ex)
6945
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   155
                    raise ValidationError(eid, {role_name('config', 'subject'): msg})
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   156
            processed[optname] = value
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   157
        # cw < 3.10 bw compat
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   158
        try:
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   159
            processed['adapter'] = confdict['adapter']
7815
2a164a9cf81c [exceptions] stop catching any exception in various places (closes #1942716)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7779
diff changeset
   160
        except KeyError:
6945
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   161
            pass
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   162
        # check for unknown options
8696
0bb18407c053 [toward py3k] rewrite dict.keys() and dict.values() (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8695
diff changeset
   163
        if confdict and tuple(confdict) != ('adapter',):
12150
6260c88e0ff5 [sources] Enhance prototype of check_conf_dict
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12149
diff changeset
   164
            if raise_on_error:
12149
649100470733 [sources] Stop translating validation error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12148
diff changeset
   165
                msg = _('unknown options %s')
649100470733 [sources] Stop translating validation error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12148
diff changeset
   166
                msgargs = ', '.join(confdict)
649100470733 [sources] Stop translating validation error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12148
diff changeset
   167
                raise ValidationError(eid, {role_name('config', 'subject'): msg}, msgargs)
6945
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   168
            else:
12155
4ba8fd2cd41a [sources] Turn classmethod into standard method
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12153
diff changeset
   169
                self.warning('unknown options %s', ', '.join(confdict))
6945
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   170
                # add options to processed, they may be necessary during migration
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   171
                processed.update(confdict)
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   172
        return processed
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   173
12155
4ba8fd2cd41a [sources] Turn classmethod into standard method
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12153
diff changeset
   174
    def check_config(self, source_entity):
12150
6260c88e0ff5 [sources] Enhance prototype of check_conf_dict
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12149
diff changeset
   175
        """Check configuration of source entity, raise ValidationError if some
6260c88e0ff5 [sources] Enhance prototype of check_conf_dict
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12149
diff changeset
   176
        errors are detected.
6260c88e0ff5 [sources] Enhance prototype of check_conf_dict
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12149
diff changeset
   177
        """
12155
4ba8fd2cd41a [sources] Turn classmethod into standard method
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12153
diff changeset
   178
        return self._check_config_dict(source_entity.eid, source_entity.dictconfig)
6945
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   179
12153
0ff0aff4413d [sources] Check source's url attribute value on creation/modification
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12150
diff changeset
   180
    def check_urls(self, source_entity):
0ff0aff4413d [sources] Check source's url attribute value on creation/modification
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12150
diff changeset
   181
        """Check URL of source entity: `urls` is a string that may contain one
0ff0aff4413d [sources] Check source's url attribute value on creation/modification
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12150
diff changeset
   182
        URL per line), and return a list of at least one validated URL.
0ff0aff4413d [sources] Check source's url attribute value on creation/modification
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12150
diff changeset
   183
        """
0ff0aff4413d [sources] Check source's url attribute value on creation/modification
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12150
diff changeset
   184
        urls = source_entity.url if source_entity.url else ''
0ff0aff4413d [sources] Check source's url attribute value on creation/modification
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12150
diff changeset
   185
        urls = [url.strip() for url in urls.splitlines() if url.strip()]
0ff0aff4413d [sources] Check source's url attribute value on creation/modification
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12150
diff changeset
   186
        if not urls:
0ff0aff4413d [sources] Check source's url attribute value on creation/modification
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12150
diff changeset
   187
            msg = _('specifying an URL is mandatory')
0ff0aff4413d [sources] Check source's url attribute value on creation/modification
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12150
diff changeset
   188
            raise ValidationError(source_entity.eid, {role_name('url', 'subject'): msg})
0ff0aff4413d [sources] Check source's url attribute value on creation/modification
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12150
diff changeset
   189
        return urls
0ff0aff4413d [sources] Check source's url attribute value on creation/modification
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12150
diff changeset
   190
6943
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   191
    # source initialization / finalization #####################################
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   192
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   193
    def set_schema(self, schema):
2476
1294a6bdf3bf application -> instance where it makes sense
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1977
diff changeset
   194
        """set the instance'schema"""
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   195
        self.schema = schema
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   196
6943
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   197
    def init_creating(self):
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   198
        """method called by the repository once ready to create a new instance"""
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   199
        pass
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   200
12148
79160d54662e [sources] Simplify source's init method
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12144
diff changeset
   201
    def init(self, source_entity):
6943
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   202
        """method called by the repository once ready to handle request.
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   203
        `activated` is a boolean flag telling if the source is activated or not.
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   204
        """
12144
f54286c1cef5 [server] Inline _entity_update method into init method of AbstractSource
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12143
diff changeset
   205
        source_entity.complete()
f54286c1cef5 [server] Inline _entity_update method into init method of AbstractSource
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12143
diff changeset
   206
        if source_entity.url:
12153
0ff0aff4413d [sources] Check source's url attribute value on creation/modification
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12150
diff changeset
   207
            self.urls = self.check_urls(source_entity)
12144
f54286c1cef5 [server] Inline _entity_update method into init method of AbstractSource
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12143
diff changeset
   208
        else:
f54286c1cef5 [server] Inline _entity_update method into init method of AbstractSource
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12143
diff changeset
   209
            self.urls = []
6943
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   210
7552
82dde8276a5b [datafeed, entities] url for entities from a datafeed source should be on their origin site. Closes #1769391
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7514
diff changeset
   211
    PUBLIC_KEYS = ('type', 'uri', 'use-cwuri-as-url')
12156
0d1d22a3a48b Fix some flake8 errors
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12155
diff changeset
   212
6943
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   213
    def remove_sensitive_information(self, sourcedef):
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   214
        """remove sensitive information such as login / password from source
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   215
        definition
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   216
        """
8696
0bb18407c053 [toward py3k] rewrite dict.keys() and dict.values() (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8695
diff changeset
   217
        for key in list(sourcedef):
12156
0d1d22a3a48b Fix some flake8 errors
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12155
diff changeset
   218
            if key not in self.PUBLIC_KEYS:
6943
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   219
                sourcedef.pop(key)
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   220
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   221
    # connections handling #####################################################
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   222
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   223
    def get_connection(self):
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   224
        """open and return a connection to the source"""
8134
7f93da785e3a add instance to not implemented error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7922
diff changeset
   225
        raise NotImplementedError(self)
6943
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   226
7398
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   227
    def close_source_connections(self):
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   228
        for cnxset in self.repo.cnxsets:
9463
d62e13eba033 [multi-sources-removal] Simplify ConnectionsSet internal structures and public methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9452
diff changeset
   229
            cnxset.cu = None
d62e13eba033 [multi-sources-removal] Simplify ConnectionsSet internal structures and public methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9452
diff changeset
   230
            cnxset.cnx.close()
6943
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   231
7398
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   232
    def open_source_connections(self):
26695dd703d8 [repository api] definitly kill usage of word 'pool' to refer to connections set used by a session
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7342
diff changeset
   233
        for cnxset in self.repo.cnxsets:
9463
d62e13eba033 [multi-sources-removal] Simplify ConnectionsSet internal structures and public methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9452
diff changeset
   234
            cnxset.cnx = self.get_connection()
d62e13eba033 [multi-sources-removal] Simplify ConnectionsSet internal structures and public methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9452
diff changeset
   235
            cnxset.cu = cnxset.cnx.cursor()
6943
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   236
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   237
    # user authentication api ##################################################
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   238
9512
88dc96fc9fc1 [server] use a connection instead of a session for user authentication
Julien Cristau <julien.cristau@logilab.fr>
parents: 9469
diff changeset
   239
    def authenticate(self, cnx, login, **kwargs):
6152
6824f8b61098 use is_instance in a number of places (esp. documentation) rather than the deprecated implements
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6057
diff changeset
   240
        """if the source support CWUser entity type, it should implement
1398
5fe84a5f7035 rename internal entity types to have CW prefix instead of E
sylvain.thenault@logilab.fr
parents: 1263
diff changeset
   241
        this method which should return CWUser eid for the given login/password
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   242
        if this account is defined in this source and valid login / password is
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   243
        given. Else raise `AuthenticationError`
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   244
        """
8134
7f93da785e3a add instance to not implemented error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7922
diff changeset
   245
        raise NotImplementedError(self)
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   246
6943
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   247
    # RQL query api ############################################################
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   248
9512
88dc96fc9fc1 [server] use a connection instead of a session for user authentication
Julien Cristau <julien.cristau@logilab.fr>
parents: 9469
diff changeset
   249
    def syntax_tree_search(self, cnx, union,
11237
f32134dd0067 [repository] drop remanescence of old multi-sources code
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11057
diff changeset
   250
                           args=None, cachekey=None, debug=0):
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   251
        """return result from this source for a rql query (actually from a rql
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   252
        syntax tree and a solution dictionary mapping each used variable to a
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   253
        possible type). If cachekey is given, the query necessary to fetch the
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   254
        results (but not the results themselves) may be cached using this key.
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   255
        """
8134
7f93da785e3a add instance to not implemented error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7922
diff changeset
   256
        raise NotImplementedError(self)
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   257
6943
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6931
diff changeset
   258
    # write modification api ###################################################
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   259
    # read-only sources don't have to implement methods below
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   260
10086
98bc2ca1a816 [source/native] session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9822
diff changeset
   261
    def add_entity(self, cnx, entity):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   262
        """add a new entity to the source"""
8134
7f93da785e3a add instance to not implemented error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7922
diff changeset
   263
        raise NotImplementedError(self)
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   264
10086
98bc2ca1a816 [source/native] session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9822
diff changeset
   265
    def update_entity(self, cnx, entity):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   266
        """update an entity in the source"""
8134
7f93da785e3a add instance to not implemented error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7922
diff changeset
   267
        raise NotImplementedError(self)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   268
10086
98bc2ca1a816 [source/native] session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9822
diff changeset
   269
    def delete_entities(self, cnx, entities):
6889
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6722
diff changeset
   270
        """delete several entities from the source"""
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6722
diff changeset
   271
        for entity in entities:
10086
98bc2ca1a816 [source/native] session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9822
diff changeset
   272
            self.delete_entity(cnx, entity)
6889
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6722
diff changeset
   273
10086
98bc2ca1a816 [source/native] session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9822
diff changeset
   274
    def delete_entity(self, cnx, entity):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   275
        """delete an entity from the source"""
8134
7f93da785e3a add instance to not implemented error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7922
diff changeset
   276
        raise NotImplementedError(self)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   277
10086
98bc2ca1a816 [source/native] session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9822
diff changeset
   278
    def add_relation(self, cnx, subject, rtype, object):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   279
        """add a relation to the source"""
8134
7f93da785e3a add instance to not implemented error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7922
diff changeset
   280
        raise NotImplementedError(self)
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   281
12156
0d1d22a3a48b Fix some flake8 errors
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 12155
diff changeset
   282
    def add_relations(self, cnx, rtype, subj_obj_list):
7237
9f619715665b [server] improve the speed of setting relations between entities (closes #1625257)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7118
diff changeset
   283
        """add a relations to the source"""
9f619715665b [server] improve the speed of setting relations between entities (closes #1625257)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7118
diff changeset
   284
        # override in derived classes if you feel you can
9f619715665b [server] improve the speed of setting relations between entities (closes #1625257)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7118
diff changeset
   285
        # optimize
9f619715665b [server] improve the speed of setting relations between entities (closes #1625257)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7118
diff changeset
   286
        for subject, object in subj_obj_list:
10086
98bc2ca1a816 [source/native] session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9822
diff changeset
   287
            self.add_relation(cnx, subject, rtype, object)
7237
9f619715665b [server] improve the speed of setting relations between entities (closes #1625257)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 7118
diff changeset
   288
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   289
    def delete_relation(self, session, subject, rtype, object):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   290
        """delete a relation from the source"""
8134
7f93da785e3a add instance to not implemented error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7922
diff changeset
   291
        raise NotImplementedError(self)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   292
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   293
    # system source interface #################################################
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   294
11774
51c160677afe [repository] Drop the entities.extid column and associated cache
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11767
diff changeset
   295
    def eid_type(self, cnx, eid):
51c160677afe [repository] Drop the entities.extid column and associated cache
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11767
diff changeset
   296
        """Return the type of entity `eid`."""
8134
7f93da785e3a add instance to not implemented error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7922
diff changeset
   297
        raise NotImplementedError(self)
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   298
10086
98bc2ca1a816 [source/native] session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9822
diff changeset
   299
    def create_eid(self, cnx):
8134
7f93da785e3a add instance to not implemented error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7922
diff changeset
   300
        raise NotImplementedError(self)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   301
11774
51c160677afe [repository] Drop the entities.extid column and associated cache
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11767
diff changeset
   302
    def add_info(self, cnx, entity, source):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   303
        """add type and source info for an eid into the system table"""
8134
7f93da785e3a add instance to not implemented error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7922
diff changeset
   304
        raise NotImplementedError(self)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   305
10086
98bc2ca1a816 [source/native] session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9822
diff changeset
   306
    def update_info(self, cnx, entity, need_fti_update):
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4807
diff changeset
   307
        """mark entity as being modified, fulltext reindex if needed"""
8134
7f93da785e3a add instance to not implemented error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7922
diff changeset
   308
        raise NotImplementedError(self)
4913
083b4d454192 server/web api for accessing to deleted_entites
Katia Saurfelt <katia.saurfelt@logilab.fr>
parents: 4807
diff changeset
   309
10086
98bc2ca1a816 [source/native] session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9822
diff changeset
   310
    def index_entity(self, cnx, entity):
4807
5642bfa43236 [cleanup] add index_entity to abstract source, add docstring
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4252
diff changeset
   311
        """create an operation to [re]index textual content of the given entity
5642bfa43236 [cleanup] add index_entity to abstract source, add docstring
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4252
diff changeset
   312
        on commit
5642bfa43236 [cleanup] add index_entity to abstract source, add docstring
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4252
diff changeset
   313
        """
8134
7f93da785e3a add instance to not implemented error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7922
diff changeset
   314
        raise NotImplementedError(self)
4807
5642bfa43236 [cleanup] add index_entity to abstract source, add docstring
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4252
diff changeset
   315
10086
98bc2ca1a816 [source/native] session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9822
diff changeset
   316
    def fti_unindex_entities(self, cnx, entities):
6889
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6722
diff changeset
   317
        """remove text content for entities from the full text index
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   318
        """
8134
7f93da785e3a add instance to not implemented error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7922
diff changeset
   319
        raise NotImplementedError(self)
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   320
10086
98bc2ca1a816 [source/native] session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9822
diff changeset
   321
    def fti_index_entities(self, cnx, entities):
6889
37668bf302f5 improve massive deletion performance
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6722
diff changeset
   322
        """add text content of created/modified entities to the full text index
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   323
        """
8134
7f93da785e3a add instance to not implemented error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7922
diff changeset
   324
        raise NotImplementedError(self)
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   325
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   326
    # sql system source interface #############################################
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   327
10086
98bc2ca1a816 [source/native] session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9822
diff changeset
   328
    def sqlexec(self, cnx, sql, args=None):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   329
        """execute the query and return its result"""
8134
7f93da785e3a add instance to not implemented error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7922
diff changeset
   330
        raise NotImplementedError(self)
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   331
10086
98bc2ca1a816 [source/native] session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9822
diff changeset
   332
    def create_index(self, cnx, table, column, unique=False):
8134
7f93da785e3a add instance to not implemented error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7922
diff changeset
   333
        raise NotImplementedError(self)
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   334
10086
98bc2ca1a816 [source/native] session -> cnx
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9822
diff changeset
   335
    def drop_index(self, cnx, table, column, unique=False):
8134
7f93da785e3a add instance to not implemented error
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7922
diff changeset
   336
        raise NotImplementedError(self)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   337
7697
ef50074a0314 [repo api] deprecates source.extid2eid, only the system source should implement it,
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7552
diff changeset
   338
6427
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6162
diff changeset
   339
def source_adapter(source_type):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   340
    try:
6427
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6162
diff changeset
   341
        return SOURCE_TYPES[source_type]
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   342
    except KeyError:
6427
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6162
diff changeset
   343
        raise RuntimeError('Unknown source type %r' % source_type)
1792
9eadf34fd860 delete-trailing-spaces
sylvain.thenault@logilab.fr
parents: 1398
diff changeset
   344
11767
432f87a63057 flake8 and all
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11763
diff changeset
   345
6945
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   346
def get_source(type, source_config, repo, eid):
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   347
    """return a source adapter according to the adapter field in the source's
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   348
    configuration
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   349
    """
6945
28bf94d062a9 [sources] refactor source creation and options handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   350
    return source_adapter(type)(repo, source_config, eid)