author | Sylvain Thénault <sylvain.thenault@logilab.fr> |
Fri, 16 Sep 2011 14:43:41 +0200 | |
branch | stable |
changeset 7804 | 814fb7f379a3 |
parent 7771 | 59ddc2dbe7e0 |
child 7815 | 2a164a9cf81c |
permissions | -rw-r--r-- |
6943
406a41c25e13
[sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6794
diff
changeset
|
1 |
# copyright 2003-2011 LOGILAB S.A. (Paris, FRANCE), all rights reserved. |
5421
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5420
diff
changeset
|
2 |
# contact http://www.logilab.fr/ -- mailto:contact@logilab.fr |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5420
diff
changeset
|
3 |
# |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5420
diff
changeset
|
4 |
# This file is part of CubicWeb. |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5420
diff
changeset
|
5 |
# |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5420
diff
changeset
|
6 |
# CubicWeb is free software: you can redistribute it and/or modify it under the |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5420
diff
changeset
|
7 |
# terms of the GNU Lesser General Public License as published by the Free |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5420
diff
changeset
|
8 |
# Software Foundation, either version 2.1 of the License, or (at your option) |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5420
diff
changeset
|
9 |
# any later version. |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5420
diff
changeset
|
10 |
# |
5424
8ecbcbff9777
replace logilab-common by CubicWeb in disclaimer
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5421
diff
changeset
|
11 |
# CubicWeb is distributed in the hope that it will be useful, but WITHOUT |
5421
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5420
diff
changeset
|
12 |
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5420
diff
changeset
|
13 |
# FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5420
diff
changeset
|
14 |
# details. |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5420
diff
changeset
|
15 |
# |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5420
diff
changeset
|
16 |
# You should have received a copy of the GNU Lesser General Public License along |
8167de96c523
proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5420
diff
changeset
|
17 |
# with CubicWeb. If not, see <http://www.gnu.org/licenses/>. |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
18 |
"""plan execution of rql queries on multiple sources |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
19 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
20 |
the best way to understand what are we trying to acheive here is to read the |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
21 |
unit-tests in unittest_msplanner.py |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
22 |
|
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
23 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
24 |
What you need to know |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
25 |
~~~~~~~~~~~~~~~~~~~~~ |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
26 |
1. The system source is expected to support every entity and relation types |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
27 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
28 |
2. Given "X relation Y": |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
29 |
|
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
30 |
* if relation, X and Y types are supported by the external source, we suppose |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
31 |
by default that X and Y should both come from the same source as the |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
32 |
relation. You can specify otherwise by adding relation into the |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
33 |
"cross_relations" set in the source's mapping file and it that case, we'll |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
34 |
consider that we can also find in the system source some relation between |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
35 |
X and Y coming from different sources. |
1785 | 36 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
37 |
* if "relation" isn't supported by the external source but X or Y |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
38 |
types (or both) are, we suppose by default that can find in the system |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
39 |
source some relation where X and/or Y come from the external source. You |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
40 |
can specify otherwise by adding relation into the "dont_cross_relations" |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
41 |
set in the source's mapping file and it that case, we'll consider that we |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
42 |
can only find in the system source some relation between X and Y coming |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
43 |
the system source. |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
44 |
|
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
45 |
|
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
46 |
Implementation |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
47 |
~~~~~~~~~~~~~~ |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
48 |
XXX explain algorithm |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
49 |
|
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
50 |
|
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
51 |
Exemples of multi-sources query execution |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
52 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
1398
5fe84a5f7035
rename internal entity types to have CW prefix instead of E
sylvain.thenault@logilab.fr
parents:
1263
diff
changeset
|
53 |
For a system source and a ldap user source (only CWUser and its attributes |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
54 |
is supported, no group or such): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
55 |
|
1398
5fe84a5f7035
rename internal entity types to have CW prefix instead of E
sylvain.thenault@logilab.fr
parents:
1263
diff
changeset
|
56 |
:CWUser X: |
5fe84a5f7035
rename internal entity types to have CW prefix instead of E
sylvain.thenault@logilab.fr
parents:
1263
diff
changeset
|
57 |
1. fetch CWUser X from both sources and return concatenation of results |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
58 |
|
1398
5fe84a5f7035
rename internal entity types to have CW prefix instead of E
sylvain.thenault@logilab.fr
parents:
1263
diff
changeset
|
59 |
:CWUser X WHERE X in_group G, G name 'users': |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
60 |
* catch 1 |
1398
5fe84a5f7035
rename internal entity types to have CW prefix instead of E
sylvain.thenault@logilab.fr
parents:
1263
diff
changeset
|
61 |
1. fetch CWUser X from both sources, store concatenation of results into a |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
62 |
temporary table |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
63 |
2. return the result of TMP X WHERE X in_group G, G name 'users' from the |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
64 |
system source |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
65 |
* catch 2 |
1398
5fe84a5f7035
rename internal entity types to have CW prefix instead of E
sylvain.thenault@logilab.fr
parents:
1263
diff
changeset
|
66 |
1. return the result of CWUser X WHERE X in_group G, G name 'users' from system |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
67 |
source, that's enough (optimization of the sql querier will avoid join on |
1398
5fe84a5f7035
rename internal entity types to have CW prefix instead of E
sylvain.thenault@logilab.fr
parents:
1263
diff
changeset
|
68 |
CWUser, so we will directly get local eids) |
1785 | 69 |
|
1398
5fe84a5f7035
rename internal entity types to have CW prefix instead of E
sylvain.thenault@logilab.fr
parents:
1263
diff
changeset
|
70 |
:CWUser X,L WHERE X in_group G, X login L, G name 'users': |
5fe84a5f7035
rename internal entity types to have CW prefix instead of E
sylvain.thenault@logilab.fr
parents:
1263
diff
changeset
|
71 |
1. fetch Any X,L WHERE X is CWUser, X login L from both sources, store |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
72 |
concatenation of results into a temporary table |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
73 |
2. return the result of Any X, L WHERE X is TMP, X login LX in_group G, |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
74 |
G name 'users' from the system source |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
75 |
|
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
76 |
|
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
77 |
:Any X WHERE X owned_by Y: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
78 |
* catch 1 |
1398
5fe84a5f7035
rename internal entity types to have CW prefix instead of E
sylvain.thenault@logilab.fr
parents:
1263
diff
changeset
|
79 |
1. fetch CWUser X from both sources, store concatenation of results into a |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
80 |
temporary table |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
81 |
2. return the result of Any X WHERE X owned_by Y, Y is TMP from the system |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
82 |
source |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
83 |
* catch 2 |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
84 |
1. return the result of Any X WHERE X owned_by Y from system source, that's |
1398
5fe84a5f7035
rename internal entity types to have CW prefix instead of E
sylvain.thenault@logilab.fr
parents:
1263
diff
changeset
|
85 |
enough (optimization of the sql querier will avoid join on CWUser, so we |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
86 |
will directly get local eids) |
6427
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
87 |
""" |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
88 |
|
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
89 |
__docformat__ = "restructuredtext en" |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
90 |
|
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
91 |
from itertools import imap, ifilterfalse |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
92 |
|
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
93 |
from logilab.common.compat import any |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
94 |
from logilab.common.decorators import cached |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
95 |
|
6427
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
96 |
from rql import BadRQLQuery |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
97 |
from rql.stmts import Union, Select |
5582
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
98 |
from rql.nodes import (VariableRef, Comparison, Relation, Constant, Variable, |
5768
1e73a466aa69
[fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5582
diff
changeset
|
99 |
Not, Exists, SortTerm, Function) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
100 |
|
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
101 |
from cubicweb import server |
940
15dcdc863965
fix imports : common.utils -> utils
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
442
diff
changeset
|
102 |
from cubicweb.utils import make_uid |
6128 | 103 |
from cubicweb.rqlrewrite import add_types_restriction |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
104 |
from cubicweb.server.utils import cleanup_solutions |
6128 | 105 |
from cubicweb.server.ssplanner import SSPlanner, OneFetchStep |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
106 |
from cubicweb.server.mssteps import * |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
107 |
|
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
108 |
Variable._ms_table_key = lambda x: x.name |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
109 |
Relation._ms_table_key = lambda x: x.r_type |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
110 |
# str() Constant.value to ensure generated table name won't be unicode |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
111 |
Constant._ms_table_key = lambda x: str(x.value) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
112 |
|
6131
087c5a168010
[ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6129
diff
changeset
|
113 |
Variable._ms_may_be_processed = lambda x, terms, linkedterms: any( |
087c5a168010
[ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6129
diff
changeset
|
114 |
t for t in terms if t in linkedterms.get(x, ())) |
087c5a168010
[ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6129
diff
changeset
|
115 |
Relation._ms_may_be_processed = lambda x, terms, linkedterms: all( |
087c5a168010
[ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6129
diff
changeset
|
116 |
getattr(hs, 'variable', hs) in terms for hs in x.get_variable_parts()) |
087c5a168010
[ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6129
diff
changeset
|
117 |
|
5582
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
118 |
def ms_scope(term): |
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
119 |
rel = None |
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
120 |
scope = term.scope |
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
121 |
if isinstance(term, Variable) and len(term.stinfo['relations']) == 1: |
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
122 |
rel = iter(term.stinfo['relations']).next().relation() |
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
123 |
elif isinstance(term, Constant): |
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
124 |
rel = term.relation() |
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
125 |
elif isinstance(term, Relation): |
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
126 |
rel = term |
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
127 |
if rel is not None and ( |
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
128 |
rel.r_type != 'identity' and rel.scope is scope |
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
129 |
and isinstance(rel.parent, Exists) and rel.parent.neged(strict=True)): |
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
130 |
return scope.parent.scope |
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
131 |
return scope |
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
132 |
|
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
133 |
def need_intersect(select, getrschema): |
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
134 |
for rel in select.iget_nodes(Relation): |
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
135 |
if isinstance(rel.parent, Exists) and rel.parent.neged(strict=True) and not rel.is_types_restriction(): |
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
136 |
rschema = getrschema(rel.r_type) |
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
137 |
if not rschema.final: |
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
138 |
# if one of the relation's variable is ambiguous but not |
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
139 |
# invariant, an intersection will be necessary |
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
140 |
for vref in rel.get_nodes(VariableRef): |
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
141 |
var = vref.variable |
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
142 |
if (var.valuable_references() == 1 |
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
143 |
and len(var.stinfo['possibletypes']) > 1): |
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
144 |
return True |
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
145 |
return False |
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
146 |
|
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
147 |
def neged_relation(rel): |
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
148 |
parent = rel.parent |
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
149 |
return isinstance(parent, Not) or (isinstance(parent, Exists) and |
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
150 |
isinstance(parent.parent, Not)) |
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
151 |
|
2688
afa99d53401c
[multi-source] test and fix msplanner regression
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2170
diff
changeset
|
152 |
def need_source_access_relation(vargraph): |
afa99d53401c
[multi-source] test and fix msplanner regression
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2170
diff
changeset
|
153 |
if not vargraph: |
afa99d53401c
[multi-source] test and fix msplanner regression
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2170
diff
changeset
|
154 |
return False |
afa99d53401c
[multi-source] test and fix msplanner regression
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2170
diff
changeset
|
155 |
# check vargraph contains some other relation than the identity relation |
afa99d53401c
[multi-source] test and fix msplanner regression
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2170
diff
changeset
|
156 |
# test of key nature since it may be a variable name (don't care about that) |
afa99d53401c
[multi-source] test and fix msplanner regression
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2170
diff
changeset
|
157 |
# or a 2-uple (var1, var2) associated to the relation to traverse to go from |
afa99d53401c
[multi-source] test and fix msplanner regression
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2170
diff
changeset
|
158 |
# var1 to var2 |
afa99d53401c
[multi-source] test and fix msplanner regression
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2170
diff
changeset
|
159 |
return any(key for key, val in vargraph.iteritems() |
afa99d53401c
[multi-source] test and fix msplanner regression
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2170
diff
changeset
|
160 |
if isinstance(key, tuple) and val != 'identity') |
afa99d53401c
[multi-source] test and fix msplanner regression
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2170
diff
changeset
|
161 |
|
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
162 |
def need_aggr_step(select, sources, stepdefs=None): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
163 |
"""return True if a temporary table is necessary to store some partial |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
164 |
results to execute the given query |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
165 |
""" |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
166 |
if len(sources) == 1: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
167 |
# can do everything at once with a single source |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
168 |
return False |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
169 |
if select.orderby or select.groupby or select.has_aggregat: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
170 |
# if more than one source, we need a temp table to deal with sort / |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
171 |
# groups / aggregat if : |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
172 |
# * the rqlst won't be splitted (in the other case the last query |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
173 |
# using partial temporary table can do sort/groups/aggregat without |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
174 |
# the need for a later AggrStep) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
175 |
# * the rqlst is splitted in multiple steps and there are more than one |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
176 |
# final step |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
177 |
if stepdefs is None: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
178 |
return True |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
179 |
has_one_final = False |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
180 |
fstepsolindices = set() |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
181 |
for stepdef in stepdefs: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
182 |
if stepdef[-1]: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
183 |
if has_one_final or frozenset(stepdef[2]) != fstepsolindices: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
184 |
return True |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
185 |
has_one_final = True |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
186 |
else: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
187 |
fstepsolindices.update(stepdef[2]) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
188 |
return False |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
189 |
|
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
190 |
def select_group_sort(select): # XXX something similar done in rql2sql |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
191 |
# add variables used in groups and sort terms to the selection |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
192 |
# if necessary |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
193 |
if select.groupby: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
194 |
for vref in select.groupby: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
195 |
if not vref in select.selection: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
196 |
select.append_selected(vref.copy(select)) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
197 |
for sortterm in select.orderby: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
198 |
for vref in sortterm.iget_nodes(VariableRef): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
199 |
if not vref in select.get_selected_variables(): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
200 |
# we can't directly insert sortterm.term because it references |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
201 |
# a variable of the select before the copy. |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
202 |
# XXX if constant term are used to define sort, their value |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
203 |
# may necessite a decay |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
204 |
select.append_selected(vref.copy(select)) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
205 |
if select.groupby and not vref in select.groupby: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
206 |
select.add_group_var(vref.copy(select)) |
1120
1c24cde4bf72
fix ms planning w/ relation constraint propagation
sylvain.thenault@logilab.fr
parents:
442
diff
changeset
|
207 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
208 |
def allequals(solutions): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
209 |
"""return true if all solutions are identical""" |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
210 |
sol = solutions.next() |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
211 |
noconstsol = None |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
212 |
for sol_ in solutions: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
213 |
if sol_ != sol: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
214 |
return False |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
215 |
return True |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
216 |
|
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
217 |
# XXX move functions below to rql ############################################## |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
218 |
|
1120
1c24cde4bf72
fix ms planning w/ relation constraint propagation
sylvain.thenault@logilab.fr
parents:
442
diff
changeset
|
219 |
def is_ancestor(n1, n2): |
2075
933ccfce6e91
fix bug in multisource planner, w/ inlined relation
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1977
diff
changeset
|
220 |
"""return True if n2 is a parent scope of n1""" |
1120
1c24cde4bf72
fix ms planning w/ relation constraint propagation
sylvain.thenault@logilab.fr
parents:
442
diff
changeset
|
221 |
p = n1.parent |
1c24cde4bf72
fix ms planning w/ relation constraint propagation
sylvain.thenault@logilab.fr
parents:
442
diff
changeset
|
222 |
while p is not None: |
1c24cde4bf72
fix ms planning w/ relation constraint propagation
sylvain.thenault@logilab.fr
parents:
442
diff
changeset
|
223 |
if p is n2: |
1c24cde4bf72
fix ms planning w/ relation constraint propagation
sylvain.thenault@logilab.fr
parents:
442
diff
changeset
|
224 |
return True |
1c24cde4bf72
fix ms planning w/ relation constraint propagation
sylvain.thenault@logilab.fr
parents:
442
diff
changeset
|
225 |
p = p.parent |
1c24cde4bf72
fix ms planning w/ relation constraint propagation
sylvain.thenault@logilab.fr
parents:
442
diff
changeset
|
226 |
return False |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
227 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
228 |
def copy_node(newroot, node, subparts=()): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
229 |
newnode = node.__class__(*node.initargs(newroot)) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
230 |
for part in subparts: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
231 |
newnode.append(part) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
232 |
return newnode |
1785 | 233 |
|
2075
933ccfce6e91
fix bug in multisource planner, w/ inlined relation
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1977
diff
changeset
|
234 |
def used_in_outer_scope(var, scope): |
933ccfce6e91
fix bug in multisource planner, w/ inlined relation
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1977
diff
changeset
|
235 |
"""return true if the variable is used in an outer scope of the given scope |
933ccfce6e91
fix bug in multisource planner, w/ inlined relation
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1977
diff
changeset
|
236 |
""" |
933ccfce6e91
fix bug in multisource planner, w/ inlined relation
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1977
diff
changeset
|
237 |
for rel in var.stinfo['relations']: |
5582
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
238 |
rscope = ms_scope(rel) |
2075
933ccfce6e91
fix bug in multisource planner, w/ inlined relation
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1977
diff
changeset
|
239 |
if not rscope is scope and is_ancestor(scope, rscope): |
933ccfce6e91
fix bug in multisource planner, w/ inlined relation
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1977
diff
changeset
|
240 |
return True |
933ccfce6e91
fix bug in multisource planner, w/ inlined relation
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1977
diff
changeset
|
241 |
return False |
1785 | 242 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
243 |
################################################################################ |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
244 |
|
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
245 |
class PartPlanInformation(object): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
246 |
"""regroups necessary information to execute some part of a "global" rql |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
247 |
query ("global" means as received by the querier, which may result in |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
248 |
several internal queries, e.g. parts, due to security insertions). Actually |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
249 |
a PPI is created for each subquery and for each query in a union. |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
250 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
251 |
It exposes as well some methods helping in executing this part on a |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
252 |
multi-sources repository, modifying its internal structure during the |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
253 |
process. |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
254 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
255 |
:attr plan: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
256 |
the execution plan |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
257 |
:attr rqlst: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
258 |
the original rql syntax tree handled by this part |
1785 | 259 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
260 |
:attr needsplit: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
261 |
bool telling if the query has to be split into multiple steps for |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
262 |
execution or if it can be executed at once |
1785 | 263 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
264 |
:attr temptable: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
265 |
a SQL temporary table name or None, if necessary to handle aggregate / |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
266 |
sorting for this part of the query |
1785 | 267 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
268 |
:attr finaltable: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
269 |
a SQL table name or None, if results for this part of the query should be |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
270 |
written into a temporary table (usually shared by multiple PPI) |
1785 | 271 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
272 |
:attr sourcesterms: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
273 |
a dictionary {source : {term: set([solution index, ])}} telling for each |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
274 |
source which terms are supported for which solutions. A "term" may be |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
275 |
either a rql Variable, Constant or Relation node. |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
276 |
""" |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
277 |
def __init__(self, plan, rqlst, rqlhelper=None): |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
278 |
self.plan = plan |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
279 |
self.rqlst = rqlst |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
280 |
self.needsplit = False |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
281 |
self.temptable = None |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
282 |
self.finaltable = None |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
283 |
# shortcuts |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
284 |
self._schema = plan.schema |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
285 |
self._session = plan.session |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
286 |
self._repo = self._session.repo |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
287 |
self._solutions = rqlst.solutions |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
288 |
self._solindices = range(len(self._solutions)) |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
289 |
self.system_source = self._repo.system_source |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
290 |
# source : {term: [solution index, ]} |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
291 |
self.sourcesterms = self._sourcesterms = {} |
393 | 292 |
# source : {relation: set(child variable and constant)} |
293 |
self._crossrelations = {} |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
294 |
# dictionary of variables and constants which are linked to each other |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
295 |
# using a non final relation supported by multiple sources (crossed or |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
296 |
# not). |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
297 |
self._linkedterms = {} |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
298 |
# processing |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
299 |
termssources = self._compute_sourcesterms() |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
300 |
self._remove_invalid_sources(termssources) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
301 |
self._compute_needsplit() |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
302 |
# after initialisation, .sourcesterms contains the same thing as |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
303 |
# ._sourcesterms though during plan construction, ._sourcesterms will |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
304 |
# be modified while .sourcesterms will be kept unmodified |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
305 |
self.sourcesterms = {} |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
306 |
for k, v in self._sourcesterms.iteritems(): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
307 |
self.sourcesterms[k] = {} |
393 | 308 |
for k2, v2 in v.iteritems(): |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
309 |
self.sourcesterms[k][k2] = v2.copy() |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
310 |
# cleanup linked var |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
311 |
for var, linkedrelsinfo in self._linkedterms.iteritems(): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
312 |
self._linkedterms[var] = frozenset(x[0] for x in linkedrelsinfo) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
313 |
# map output of a step to input of a following step |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
314 |
self._inputmaps = {} |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
315 |
# record input map conflicts to resolve them on final step generation |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
316 |
self._conflicts = [] |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
317 |
if rqlhelper is not None: # else test |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
318 |
self._insert_identity_variable = rqlhelper._annotator.rewrite_shared_optional |
2766
07e47b84acc2
DBG_MS flag
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2688
diff
changeset
|
319 |
if server.DEBUG & server.DBG_MS: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
320 |
print 'sourcesterms:' |
1786
eccd1885d42e
one more msplanning bug fixed...
sylvain.thenault@logilab.fr
parents:
1785
diff
changeset
|
321 |
self._debug_sourcesterms() |
eccd1885d42e
one more msplanning bug fixed...
sylvain.thenault@logilab.fr
parents:
1785
diff
changeset
|
322 |
|
eccd1885d42e
one more msplanning bug fixed...
sylvain.thenault@logilab.fr
parents:
1785
diff
changeset
|
323 |
def _debug_sourcesterms(self): |
eccd1885d42e
one more msplanning bug fixed...
sylvain.thenault@logilab.fr
parents:
1785
diff
changeset
|
324 |
for source in self._sourcesterms: |
eccd1885d42e
one more msplanning bug fixed...
sylvain.thenault@logilab.fr
parents:
1785
diff
changeset
|
325 |
print '-', source |
eccd1885d42e
one more msplanning bug fixed...
sylvain.thenault@logilab.fr
parents:
1785
diff
changeset
|
326 |
for term, sols in self._sourcesterms[source].items(): |
4721
8f63691ccb7f
pylint style fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4719
diff
changeset
|
327 |
print ' -', term, id(term), ':', sols |
1785 | 328 |
|
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
329 |
def copy_solutions(self, solindices): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
330 |
return [self._solutions[solidx].copy() for solidx in solindices] |
1785 | 331 |
|
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
332 |
@property |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
333 |
@cached |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
334 |
def part_sources(self): |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
335 |
if self._sourcesterms: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
336 |
return tuple(sorted(self._sourcesterms)) |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
337 |
return (self.system_source,) |
1785 | 338 |
|
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
339 |
@property |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
340 |
@cached |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
341 |
def _sys_source_set(self): |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
342 |
return frozenset((self.system_source, solindex) |
1785 | 343 |
for solindex in self._solindices) |
344 |
||
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
345 |
@cached |
393 | 346 |
def _norel_support_set(self, relation): |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
347 |
"""return a set of (source, solindex) where source doesn't support the |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
348 |
relation |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
349 |
""" |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
350 |
return frozenset((source, solidx) for source in self._repo.sources |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
351 |
for solidx in self._solindices |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
352 |
if not ((source.support_relation(relation.r_type)) |
393 | 353 |
or relation.r_type in source.dont_cross_relations)) |
354 |
||
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
355 |
def _compute_sourcesterms(self): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
356 |
"""compute for each term (variable, rewritten constant, relation) and |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
357 |
for each solution in the rqlst which sources support them |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
358 |
""" |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
359 |
repo = self._repo |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
360 |
eschema = self._schema.eschema |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
361 |
sourcesterms = self._sourcesterms |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
362 |
# find for each source which variable/solution are supported |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
363 |
for varname, varobj in self.rqlst.defined_vars.items(): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
364 |
# if variable has an eid specified, we can get its source directly |
5004
4cc020ee70e2
le patch rql26 a été importé
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4956
diff
changeset
|
365 |
# NOTE: use uidrel and not constnode to deal with "X eid IN(1,2,3,4)" |
4cc020ee70e2
le patch rql26 a été importé
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4956
diff
changeset
|
366 |
if varobj.stinfo['uidrel'] is not None: |
4cc020ee70e2
le patch rql26 a été importé
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4956
diff
changeset
|
367 |
rel = varobj.stinfo['uidrel'] |
5161
a19f22bacedc
[ms] fix planning bug introduced with recent rql refactoring for queries with only eids (fix unittest_multisources.test_in_eid)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5047
diff
changeset
|
368 |
hasrel = len(varobj.stinfo['relations']) > 1 |
5004
4cc020ee70e2
le patch rql26 a été importé
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4956
diff
changeset
|
369 |
for const in rel.children[1].get_nodes(Constant): |
4cc020ee70e2
le patch rql26 a été importé
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4956
diff
changeset
|
370 |
eid = const.eval(self.plan.args) |
4cc020ee70e2
le patch rql26 a été importé
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4956
diff
changeset
|
371 |
source = self._session.source_from_eid(eid) |
5161
a19f22bacedc
[ms] fix planning bug introduced with recent rql refactoring for queries with only eids (fix unittest_multisources.test_in_eid)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5047
diff
changeset
|
372 |
if (source is self.system_source |
7651
7c0af7ef3325
[repo, ms] fix planning of some queries where variable 'non-invarianess' should be considered, leading for instance to junk in entities table on source deletion
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7306
diff
changeset
|
373 |
or (hasrel and varobj._q_invariant and |
5161
a19f22bacedc
[ms] fix planning bug introduced with recent rql refactoring for queries with only eids (fix unittest_multisources.test_in_eid)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5047
diff
changeset
|
374 |
not any(source.support_relation(r.r_type) |
a19f22bacedc
[ms] fix planning bug introduced with recent rql refactoring for queries with only eids (fix unittest_multisources.test_in_eid)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5047
diff
changeset
|
375 |
for r in varobj.stinfo['relations'] |
a19f22bacedc
[ms] fix planning bug introduced with recent rql refactoring for queries with only eids (fix unittest_multisources.test_in_eid)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5047
diff
changeset
|
376 |
if not r is rel))): |
5004
4cc020ee70e2
le patch rql26 a été importé
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4956
diff
changeset
|
377 |
self._set_source_for_term(self.system_source, varobj) |
4cc020ee70e2
le patch rql26 a été importé
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4956
diff
changeset
|
378 |
else: |
4cc020ee70e2
le patch rql26 a été importé
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4956
diff
changeset
|
379 |
self._set_source_for_term(source, varobj) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
380 |
continue |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
381 |
rels = varobj.stinfo['relations'] |
5004
4cc020ee70e2
le patch rql26 a été importé
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4956
diff
changeset
|
382 |
if not rels and varobj.stinfo['typerel'] is None: |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
383 |
# (rare) case where the variable has no type specified nor |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
384 |
# relation accessed ex. "Any MAX(X)" |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
385 |
self._set_source_for_term(self.system_source, varobj) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
386 |
continue |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
387 |
for i, sol in enumerate(self._solutions): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
388 |
vartype = sol[varname] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
389 |
# skip final variable |
3689
deb13e88e037
follow yams 0.25 api changes to improve performance
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3624
diff
changeset
|
390 |
if eschema(vartype).final: |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
391 |
break |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
392 |
for source in repo.sources: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
393 |
if source.support_entity(vartype): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
394 |
# the source support the entity type, though we will |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
395 |
# actually have to fetch from it only if |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
396 |
# * the variable isn't invariant |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
397 |
# * at least one supported relation specified |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
398 |
if not varobj._q_invariant or \ |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
399 |
any(imap(source.support_relation, |
3239
1ceac4cd4fb7
[ms] fix planning w/ query using identity
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3040
diff
changeset
|
400 |
(r.r_type for r in rels if r.r_type not in ('identity', 'eid')))): |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
401 |
sourcesterms.setdefault(source, {}).setdefault(varobj, set()).add(i) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
402 |
# if variable is not invariant and is used by a relation |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
403 |
# not supported by this source, we'll have to split the |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
404 |
# query |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
405 |
if not varobj._q_invariant and any(ifilterfalse( |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
406 |
source.support_relation, (r.r_type for r in rels))): |
1785 | 407 |
self.needsplit = True |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
408 |
# add source for rewritten constants to sourcesterms |
5420
09b50d7e5321
[ms] fix planning bug with rewritten constants: when removing some source for a const due to relation constraints, we should propagate this change to other constants coming from the same original variable
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5168
diff
changeset
|
409 |
self._const_vars = {} |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
410 |
for vconsts in self.rqlst.stinfo['rewritten'].itervalues(): |
5420
09b50d7e5321
[ms] fix planning bug with rewritten constants: when removing some source for a const due to relation constraints, we should propagate this change to other constants coming from the same original variable
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5168
diff
changeset
|
411 |
# remember those consts come from the same variable |
09b50d7e5321
[ms] fix planning bug with rewritten constants: when removing some source for a const due to relation constraints, we should propagate this change to other constants coming from the same original variable
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5168
diff
changeset
|
412 |
for const in vconsts: |
09b50d7e5321
[ms] fix planning bug with rewritten constants: when removing some source for a const due to relation constraints, we should propagate this change to other constants coming from the same original variable
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5168
diff
changeset
|
413 |
self._const_vars[const] = vconsts |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
414 |
source = self._session.source_from_eid(const.eval(self.plan.args)) |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
415 |
if source is self.system_source: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
416 |
for const in vconsts: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
417 |
self._set_source_for_term(source, const) |
2108
b3f6fa7ad59f
add test and fix for queries where all variables are simplified and come from an external source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2075
diff
changeset
|
418 |
elif not self._sourcesterms: |
6131
087c5a168010
[ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6129
diff
changeset
|
419 |
for const in vconsts: |
087c5a168010
[ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6129
diff
changeset
|
420 |
self._set_source_for_term(source, const) |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
421 |
elif source in self._sourcesterms: |
5582
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
422 |
source_scopes = frozenset(ms_scope(t) for t in self._sourcesterms[source]) |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
423 |
for const in vconsts: |
5582
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
424 |
if ms_scope(const) in source_scopes: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
425 |
self._set_source_for_term(source, const) |
1230
232e16835fff
fix constant handling in sourcesterms + minor cleanup
sylvain.thenault@logilab.fr
parents:
1229
diff
changeset
|
426 |
# if system source is used, add every rewritten constant |
232e16835fff
fix constant handling in sourcesterms + minor cleanup
sylvain.thenault@logilab.fr
parents:
1229
diff
changeset
|
427 |
# to its supported terms even when associated entity |
6057
cb5c8852cbda
[ms] cleanups
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6056
diff
changeset
|
428 |
# doesn't actually come from it so we get a changes that |
cb5c8852cbda
[ms] cleanups
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6056
diff
changeset
|
429 |
# allequals will return True as expected when computing |
cb5c8852cbda
[ms] cleanups
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6056
diff
changeset
|
430 |
# needsplit |
1786
eccd1885d42e
one more msplanning bug fixed...
sylvain.thenault@logilab.fr
parents:
1785
diff
changeset
|
431 |
# check const is used in a relation restriction |
eccd1885d42e
one more msplanning bug fixed...
sylvain.thenault@logilab.fr
parents:
1785
diff
changeset
|
432 |
if const.relation() and self.system_source in sourcesterms: |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
433 |
self._set_source_for_term(self.system_source, const) |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
434 |
# add source for relations |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
435 |
rschema = self._schema.rschema |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
436 |
termssources = {} |
6427
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
437 |
sourcerels = [] |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
438 |
for rel in self.rqlst.iget_nodes(Relation): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
439 |
# process non final relations only |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
440 |
# note: don't try to get schema for 'is' relation (not available |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
441 |
# during bootstrap) |
6427
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
442 |
if rel.r_type == 'cw_source': |
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
443 |
sourcerels.append(rel) |
6649
29f8e5c35392
[ms planning] fix/enhance cw_source consideration in ms planning
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6633
diff
changeset
|
444 |
if not (rel.is_types_restriction() or rschema(rel.r_type).final): |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
445 |
# nothing to do if relation is not supported by multiple sources |
393 | 446 |
# or if some source has it listed in its cross_relations |
447 |
# attribute |
|
448 |
# |
|
449 |
# XXX code below don't deal if some source allow relation |
|
450 |
# crossing but not another one |
|
6129
fea746b60093
[ms] fix two planner bugs: one occuring query such as X created_by U where X in a external source and U may come from an ldap source. The other being that when we've to merge input maps, we were modifying the same tree/solutions while a copy were needed. Also, ensure we add type restrictions, necessary for pyro source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6128
diff
changeset
|
451 |
relsources = [s for s in repo.rel_type_sources(rel.r_type) |
fea746b60093
[ms] fix two planner bugs: one occuring query such as X created_by U where X in a external source and U may come from an ldap source. The other being that when we've to merge input maps, we were modifying the same tree/solutions while a copy were needed. Also, ensure we add type restrictions, necessary for pyro source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6128
diff
changeset
|
452 |
if s is self.system_source |
fea746b60093
[ms] fix two planner bugs: one occuring query such as X created_by U where X in a external source and U may come from an ldap source. The other being that when we've to merge input maps, we were modifying the same tree/solutions while a copy were needed. Also, ensure we add type restrictions, necessary for pyro source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6128
diff
changeset
|
453 |
or s in self._sourcesterms] |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
454 |
if len(relsources) < 2: |
393 | 455 |
# filter out sources being there because they have this |
456 |
# relation in their dont_cross_relations attribute |
|
457 |
relsources = [source for source in relsources |
|
458 |
if source.support_relation(rel.r_type)] |
|
341
0a426be2f3a2
fixes for some cases of neged inlined relations, may have to use (new) IntersectStep. XXX: write IntersectFetchStep
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
257
diff
changeset
|
459 |
if relsources: |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
460 |
# this means the relation is using a variable inlined as |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
461 |
# a constant and another unsupported variable, in which |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
462 |
# case we put the relation in sourcesterms |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
463 |
self._sourcesterms.setdefault(relsources[0], {})[rel] = set(self._solindices) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
464 |
continue |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
465 |
lhs, rhs = rel.get_variable_parts() |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
466 |
lhsv, rhsv = getattr(lhs, 'variable', lhs), getattr(rhs, 'variable', rhs) |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
467 |
# update dictionary of sources supporting lhs and rhs vars |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
468 |
if not lhsv in termssources: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
469 |
termssources[lhsv] = self._term_sources(lhs) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
470 |
if not rhsv in termssources: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
471 |
termssources[rhsv] = self._term_sources(rhs) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
472 |
self._handle_cross_relation(rel, relsources, termssources) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
473 |
self._linkedterms.setdefault(lhsv, set()).add((rhsv, rel)) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
474 |
self._linkedterms.setdefault(rhsv, set()).add((lhsv, rel)) |
6427
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
475 |
# extract information from cw_source relation |
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
476 |
for srel in sourcerels: |
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
477 |
vref = srel.children[1].children[0] |
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
478 |
sourceeids, sourcenames = [], [] |
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
479 |
if isinstance(vref, Constant): |
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
480 |
# simplified variable |
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
481 |
sourceeids = None, (vref.eval(self.plan.args),) |
6649
29f8e5c35392
[ms planning] fix/enhance cw_source consideration in ms planning
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6633
diff
changeset
|
482 |
var = vref |
6427
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
483 |
else: |
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
484 |
var = vref.variable |
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
485 |
for rel in var.stinfo['relations'] - var.stinfo['rhsrelations']: |
7306
a21bafe9fe35
[msplanner] fix regression and test failure introduced in 7292:258c92805066
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7292
diff
changeset
|
486 |
# skip neged eid relation since it's the kind of query |
a21bafe9fe35
[msplanner] fix regression and test failure introduced in 7292:258c92805066
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7292
diff
changeset
|
487 |
# generated when clearing old value of '?1" relation, |
a21bafe9fe35
[msplanner] fix regression and test failure introduced in 7292:258c92805066
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7292
diff
changeset
|
488 |
# cw_source included. See |
a21bafe9fe35
[msplanner] fix regression and test failure introduced in 7292:258c92805066
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7292
diff
changeset
|
489 |
# unittest_ldapuser.test_copy_to_system_source |
a21bafe9fe35
[msplanner] fix regression and test failure introduced in 7292:258c92805066
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7292
diff
changeset
|
490 |
if rel.r_type == 'name' or \ |
a21bafe9fe35
[msplanner] fix regression and test failure introduced in 7292:258c92805066
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7292
diff
changeset
|
491 |
(rel.r_type == 'eid' and not rel.neged(strict=True)): |
6427
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
492 |
if rel.r_type == 'eid': |
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
493 |
slist = sourceeids |
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
494 |
else: |
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
495 |
slist = sourcenames |
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
496 |
sources = [cst.eval(self.plan.args) |
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
497 |
for cst in rel.children[1].get_nodes(Constant)] |
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
498 |
if sources: |
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
499 |
if slist: |
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
500 |
# don't attempt to do anything |
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
501 |
sourcenames = sourceeids = None |
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
502 |
break |
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
503 |
slist[:] = (rel, sources) |
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
504 |
if sourceeids: |
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
505 |
rel, values = sourceeids |
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
506 |
sourcesdict = self._repo.sources_by_eid |
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
507 |
elif sourcenames: |
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
508 |
rel, values = sourcenames |
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
509 |
sourcesdict = self._repo.sources_by_uri |
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
510 |
else: |
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
511 |
sourcesdict = None |
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
512 |
if sourcesdict is not None: |
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
513 |
lhs = srel.children[0] |
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
514 |
try: |
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
515 |
sources = [sourcesdict[key] for key in values] |
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
516 |
except KeyError: |
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
517 |
raise BadRQLQuery('source conflict for term %s' % lhs.as_string()) |
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
518 |
if isinstance(lhs, Constant): |
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
519 |
source = self._session.source_from_eid(lhs.eval(self.plan.args)) |
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
520 |
if not source in sources: |
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
521 |
raise BadRQLQuery('source conflict for term %s' % lhs.as_string()) |
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
522 |
else: |
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
523 |
lhs = getattr(lhs, 'variable', lhs) |
6649
29f8e5c35392
[ms planning] fix/enhance cw_source consideration in ms planning
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6633
diff
changeset
|
524 |
invariant = getattr(lhs, '_q_invariant', False) |
6427
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
525 |
# XXX NOT NOT |
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
526 |
neged = srel.neged(traverse_scope=True) or (rel and rel.neged(strict=True)) |
6957
ffda12be2e9f
[repository] #1460066: backport datafeed cube as cubicweb source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6943
diff
changeset
|
527 |
has_copy_based_source = False |
ffda12be2e9f
[repository] #1460066: backport datafeed cube as cubicweb source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6943
diff
changeset
|
528 |
sources_ = [] |
ffda12be2e9f
[repository] #1460066: backport datafeed cube as cubicweb source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6943
diff
changeset
|
529 |
for source in sources: |
ffda12be2e9f
[repository] #1460066: backport datafeed cube as cubicweb source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6943
diff
changeset
|
530 |
if source.copy_based_source: |
ffda12be2e9f
[repository] #1460066: backport datafeed cube as cubicweb source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6943
diff
changeset
|
531 |
has_copy_based_source = True |
ffda12be2e9f
[repository] #1460066: backport datafeed cube as cubicweb source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6943
diff
changeset
|
532 |
if not self.system_source in sources_: |
ffda12be2e9f
[repository] #1460066: backport datafeed cube as cubicweb source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6943
diff
changeset
|
533 |
sources_.append(self.system_source) |
ffda12be2e9f
[repository] #1460066: backport datafeed cube as cubicweb source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6943
diff
changeset
|
534 |
else: |
ffda12be2e9f
[repository] #1460066: backport datafeed cube as cubicweb source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6943
diff
changeset
|
535 |
sources_.append(source) |
ffda12be2e9f
[repository] #1460066: backport datafeed cube as cubicweb source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6943
diff
changeset
|
536 |
sources = sources_ |
6427
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
537 |
if neged: |
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
538 |
for source in sources: |
6649
29f8e5c35392
[ms planning] fix/enhance cw_source consideration in ms planning
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6633
diff
changeset
|
539 |
if invariant and source is self.system_source: |
29f8e5c35392
[ms planning] fix/enhance cw_source consideration in ms planning
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6633
diff
changeset
|
540 |
continue |
6633
7baea108d326
[ms] simplify cw_source handling : don't try evil optmization that may hurt in some cases
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6598
diff
changeset
|
541 |
self._remove_source_term(source, lhs) |
6649
29f8e5c35392
[ms planning] fix/enhance cw_source consideration in ms planning
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6633
diff
changeset
|
542 |
usesys = self.system_source not in sources |
6427
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
543 |
else: |
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
544 |
for source, terms in sourcesterms.items(): |
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
545 |
if lhs in terms and not source in sources: |
6649
29f8e5c35392
[ms planning] fix/enhance cw_source consideration in ms planning
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6633
diff
changeset
|
546 |
if invariant and source is self.system_source: |
29f8e5c35392
[ms planning] fix/enhance cw_source consideration in ms planning
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6633
diff
changeset
|
547 |
continue |
6633
7baea108d326
[ms] simplify cw_source handling : don't try evil optmization that may hurt in some cases
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6598
diff
changeset
|
548 |
self._remove_source_term(source, lhs) |
6649
29f8e5c35392
[ms planning] fix/enhance cw_source consideration in ms planning
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6633
diff
changeset
|
549 |
usesys = self.system_source in sources |
29f8e5c35392
[ms planning] fix/enhance cw_source consideration in ms planning
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6633
diff
changeset
|
550 |
if rel is None or (len(var.stinfo['relations']) == 2 and |
29f8e5c35392
[ms planning] fix/enhance cw_source consideration in ms planning
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6633
diff
changeset
|
551 |
not var.stinfo['selected']): |
6427
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
552 |
self._remove_source_term(self.system_source, var) |
6957
ffda12be2e9f
[repository] #1460066: backport datafeed cube as cubicweb source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6943
diff
changeset
|
553 |
if not (has_copy_based_source or len(sources) > 1 |
ffda12be2e9f
[repository] #1460066: backport datafeed cube as cubicweb source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6943
diff
changeset
|
554 |
or usesys or invariant): |
6649
29f8e5c35392
[ms planning] fix/enhance cw_source consideration in ms planning
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6633
diff
changeset
|
555 |
if rel is None: |
29f8e5c35392
[ms planning] fix/enhance cw_source consideration in ms planning
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6633
diff
changeset
|
556 |
srel.parent.remove(srel) |
29f8e5c35392
[ms planning] fix/enhance cw_source consideration in ms planning
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6633
diff
changeset
|
557 |
else: |
29f8e5c35392
[ms planning] fix/enhance cw_source consideration in ms planning
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6633
diff
changeset
|
558 |
self.rqlst.undefine_variable(var) |
29f8e5c35392
[ms planning] fix/enhance cw_source consideration in ms planning
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6633
diff
changeset
|
559 |
self._remove_source_term(self.system_source, srel) |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
560 |
return termssources |
1785 | 561 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
562 |
def _handle_cross_relation(self, rel, relsources, termssources): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
563 |
for source in relsources: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
564 |
if rel.r_type in source.cross_relations: |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
565 |
ssource = self.system_source |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
566 |
crossvars = set(x.variable for x in rel.get_nodes(VariableRef)) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
567 |
for const in rel.get_nodes(Constant): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
568 |
if source.uri != 'system' and not const in self._sourcesterms.get(source, ()): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
569 |
continue |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
570 |
crossvars.add(const) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
571 |
self._crossrelations.setdefault(source, {})[rel] = crossvars |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
572 |
if len(crossvars) < 2: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
573 |
# this means there is a constant in the relation which is |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
574 |
# not supported by the source, so we can stop here |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
575 |
continue |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
576 |
self._sourcesterms.setdefault(ssource, {})[rel] = set(self._solindices) |
6131
087c5a168010
[ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6129
diff
changeset
|
577 |
solindices = None |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
578 |
for term in crossvars: |
1229
dd9bdcfc03b6
cleanup _handle_cross_relation
sylvain.thenault@logilab.fr
parents:
1228
diff
changeset
|
579 |
if len(termssources[term]) == 1 and iter(termssources[term]).next()[0].uri == 'system': |
dd9bdcfc03b6
cleanup _handle_cross_relation
sylvain.thenault@logilab.fr
parents:
1228
diff
changeset
|
580 |
for ov in crossvars: |
dd9bdcfc03b6
cleanup _handle_cross_relation
sylvain.thenault@logilab.fr
parents:
1228
diff
changeset
|
581 |
if ov is not term and (isinstance(ov, Constant) or ov._q_invariant): |
dd9bdcfc03b6
cleanup _handle_cross_relation
sylvain.thenault@logilab.fr
parents:
1228
diff
changeset
|
582 |
ssset = frozenset((ssource,)) |
dd9bdcfc03b6
cleanup _handle_cross_relation
sylvain.thenault@logilab.fr
parents:
1228
diff
changeset
|
583 |
self._remove_sources(ov, termssources[ov] - ssset) |
dd9bdcfc03b6
cleanup _handle_cross_relation
sylvain.thenault@logilab.fr
parents:
1228
diff
changeset
|
584 |
break |
6131
087c5a168010
[ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6129
diff
changeset
|
585 |
if solindices is None: |
087c5a168010
[ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6129
diff
changeset
|
586 |
solindices = set(sol for s, sol in termssources[term] |
087c5a168010
[ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6129
diff
changeset
|
587 |
if s is source) |
087c5a168010
[ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6129
diff
changeset
|
588 |
else: |
087c5a168010
[ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6129
diff
changeset
|
589 |
solindices &= set(sol for s, sol in termssources[term] |
087c5a168010
[ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6129
diff
changeset
|
590 |
if s is source) |
1229
dd9bdcfc03b6
cleanup _handle_cross_relation
sylvain.thenault@logilab.fr
parents:
1228
diff
changeset
|
591 |
else: |
6131
087c5a168010
[ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6129
diff
changeset
|
592 |
self._sourcesterms.setdefault(source, {})[rel] = solindices |
1785 | 593 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
594 |
def _remove_invalid_sources(self, termssources): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
595 |
"""removes invalid sources from `sourcesterms` member according to |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
596 |
traversed relations and their properties (which sources support them, |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
597 |
can they cross sources, etc...) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
598 |
""" |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
599 |
for term in self._linkedterms: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
600 |
self._remove_sources_until_stable(term, termssources) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
601 |
if len(self._sourcesterms) > 1 and hasattr(self.plan.rqlst, 'main_relations'): |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
602 |
# the querier doesn't annotate write queries, need to do it here |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
603 |
self.plan.annotate_rqlst() |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
604 |
# insert/update/delete queries, we may get extra information from |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
605 |
# the main relation (eg relations to the left of the WHERE |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
606 |
if self.plan.rqlst.TYPE == 'insert': |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
607 |
inserted = dict((vref.variable, etype) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
608 |
for etype, vref in self.plan.rqlst.main_variables) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
609 |
else: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
610 |
inserted = {} |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
611 |
repo = self._repo |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
612 |
rschema = self._schema.rschema |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
613 |
for rel in self.plan.rqlst.main_relations: |
3689
deb13e88e037
follow yams 0.25 api changes to improve performance
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3624
diff
changeset
|
614 |
if not rschema(rel.r_type).final: |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
615 |
# nothing to do if relation is not supported by multiple sources |
393 | 616 |
if len(repo.rel_type_sources(rel.r_type)) < 2: |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
617 |
continue |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
618 |
lhs, rhs = rel.get_variable_parts() |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
619 |
try: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
620 |
lhsv = self._extern_term(lhs, termssources, inserted) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
621 |
rhsv = self._extern_term(rhs, termssources, inserted) |
4719
aaed3f813ef8
kill dead/useless code as suggested by pylint
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4716
diff
changeset
|
622 |
except KeyError: |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
623 |
continue |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
624 |
self._remove_term_sources(lhsv, rel, rhsv, termssources) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
625 |
self._remove_term_sources(rhsv, rel, lhsv, termssources) |
1785 | 626 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
627 |
def _extern_term(self, term, termssources, inserted): |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
628 |
var = term.variable |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
629 |
if var.stinfo['constnode']: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
630 |
termv = var.stinfo['constnode'] |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
631 |
termssources[termv] = self._term_sources(termv) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
632 |
elif var in inserted: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
633 |
termv = var |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
634 |
source = self._repo.locate_etype_source(inserted[var]) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
635 |
termssources[termv] = set((source, solindex) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
636 |
for solindex in self._solindices) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
637 |
else: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
638 |
termv = self.rqlst.defined_vars[var.name] |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
639 |
if not termv in termssources: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
640 |
termssources[termv] = self._term_sources(termv) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
641 |
return termv |
1785 | 642 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
643 |
def _remove_sources_until_stable(self, term, termssources): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
644 |
sourcesterms = self._sourcesterms |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
645 |
for oterm, rel in self._linkedterms.get(term, ()): |
5582
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
646 |
tscope = ms_scope(term) |
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
647 |
otscope = ms_scope(oterm) |
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
648 |
rscope = ms_scope(rel) |
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
649 |
if not tscope is otscope and rscope.neged(strict=True): |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
650 |
# can't get information from relation inside a NOT exists |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
651 |
# where terms don't belong to the same scope |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
652 |
continue |
1120
1c24cde4bf72
fix ms planning w/ relation constraint propagation
sylvain.thenault@logilab.fr
parents:
442
diff
changeset
|
653 |
need_ancestor_scope = False |
5582
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
654 |
if not (tscope is rscope and otscope is rscope): |
1120
1c24cde4bf72
fix ms planning w/ relation constraint propagation
sylvain.thenault@logilab.fr
parents:
442
diff
changeset
|
655 |
if rel.ored(): |
1c24cde4bf72
fix ms planning w/ relation constraint propagation
sylvain.thenault@logilab.fr
parents:
442
diff
changeset
|
656 |
continue |
1c24cde4bf72
fix ms planning w/ relation constraint propagation
sylvain.thenault@logilab.fr
parents:
442
diff
changeset
|
657 |
if rel.ored(traverse_scope=True): |
1c24cde4bf72
fix ms planning w/ relation constraint propagation
sylvain.thenault@logilab.fr
parents:
442
diff
changeset
|
658 |
# if relation has some OR as parent, constraints should only |
1c24cde4bf72
fix ms planning w/ relation constraint propagation
sylvain.thenault@logilab.fr
parents:
442
diff
changeset
|
659 |
# propagate from parent scope to child scope, nothing else |
1c24cde4bf72
fix ms planning w/ relation constraint propagation
sylvain.thenault@logilab.fr
parents:
442
diff
changeset
|
660 |
need_ancestor_scope = True |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
661 |
relsources = self._repo.rel_type_sources(rel.r_type) |
5582
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
662 |
if neged_relation(rel) and ( |
341
0a426be2f3a2
fixes for some cases of neged inlined relations, may have to use (new) IntersectStep. XXX: write IntersectFetchStep
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
257
diff
changeset
|
663 |
len(relsources) < 2 |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
664 |
or not isinstance(oterm, Variable) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
665 |
or oterm.valuable_references() != 1 |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
666 |
or any(sourcesterms[source][term] != sourcesterms[source][oterm] |
341
0a426be2f3a2
fixes for some cases of neged inlined relations, may have to use (new) IntersectStep. XXX: write IntersectFetchStep
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
257
diff
changeset
|
667 |
for source in relsources |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
668 |
if term in sourcesterms.get(source, ()) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
669 |
and oterm in sourcesterms.get(source, ()))): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
670 |
# neged relation doesn't allow to infer term sources unless |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
671 |
# we're on a multisource relation for a term only used by this |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
672 |
# relation (eg "Any X WHERE NOT X multisource_rel Y" and over is |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
673 |
# Y) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
674 |
continue |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
675 |
# compute invalid sources for terms and remove them |
5582
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
676 |
if not need_ancestor_scope or is_ancestor(tscope, otscope): |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
677 |
self._remove_term_sources(term, rel, oterm, termssources) |
5582
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
678 |
if not need_ancestor_scope or is_ancestor(otscope, tscope): |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
679 |
self._remove_term_sources(oterm, rel, term, termssources) |
1785 | 680 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
681 |
def _remove_term_sources(self, term, rel, oterm, termssources): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
682 |
"""remove invalid sources for term according to oterm's sources and the |
1785 | 683 |
relation between those two terms. |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
684 |
""" |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
685 |
norelsup = self._norel_support_set(rel) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
686 |
termsources = termssources[term] |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
687 |
invalid_sources = termsources - (termssources[oterm] | norelsup) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
688 |
if invalid_sources and self._repo.can_cross_relation(rel.r_type): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
689 |
invalid_sources -= self._sys_source_set |
1230
232e16835fff
fix constant handling in sourcesterms + minor cleanup
sylvain.thenault@logilab.fr
parents:
1229
diff
changeset
|
690 |
if invalid_sources and isinstance(term, Variable) \ |
232e16835fff
fix constant handling in sourcesterms + minor cleanup
sylvain.thenault@logilab.fr
parents:
1229
diff
changeset
|
691 |
and self._need_ext_source_access(term, rel): |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
692 |
# if the term is a not invariant variable, we should filter out |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
693 |
# source where the relation is a cross relation from invalid |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
694 |
# sources |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
695 |
invalid_sources = frozenset((s, solidx) for s, solidx in invalid_sources |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
696 |
if not (s in self._crossrelations and |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
697 |
rel in self._crossrelations[s])) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
698 |
if invalid_sources: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
699 |
self._remove_sources(term, invalid_sources) |
5420
09b50d7e5321
[ms] fix planning bug with rewritten constants: when removing some source for a const due to relation constraints, we should propagate this change to other constants coming from the same original variable
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5168
diff
changeset
|
700 |
# if term is a rewritten const, we can apply the same changes to |
09b50d7e5321
[ms] fix planning bug with rewritten constants: when removing some source for a const due to relation constraints, we should propagate this change to other constants coming from the same original variable
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5168
diff
changeset
|
701 |
# all other consts inserted from the same original variable |
09b50d7e5321
[ms] fix planning bug with rewritten constants: when removing some source for a const due to relation constraints, we should propagate this change to other constants coming from the same original variable
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5168
diff
changeset
|
702 |
for const in self._const_vars.get(term, ()): |
09b50d7e5321
[ms] fix planning bug with rewritten constants: when removing some source for a const due to relation constraints, we should propagate this change to other constants coming from the same original variable
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5168
diff
changeset
|
703 |
if const is not term: |
09b50d7e5321
[ms] fix planning bug with rewritten constants: when removing some source for a const due to relation constraints, we should propagate this change to other constants coming from the same original variable
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5168
diff
changeset
|
704 |
self._remove_sources(const, invalid_sources) |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
705 |
termsources -= invalid_sources |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
706 |
self._remove_sources_until_stable(term, termssources) |
1785 | 707 |
if isinstance(oterm, Constant): |
708 |
self._remove_sources(oterm, invalid_sources) |
|
709 |
||
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
710 |
def _compute_needsplit(self): |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
711 |
"""tell according to sourcesterms if the rqlst has to be splitted for |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
712 |
execution among multiple sources |
1785 | 713 |
|
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
714 |
the execution has to be split if |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
715 |
* a source support an entity (non invariant) but doesn't support a |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
716 |
relation on it |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
717 |
* a source support an entity which is accessed by an optional relation |
1785 | 718 |
* there is more than one source and either all sources'supported |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
719 |
variable/solutions are not equivalent or multiple variables have to |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
720 |
be fetched from some source |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
721 |
""" |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
722 |
# NOTE: < 2 since may be 0 on queries such as Any X WHERE X eid 2 |
1785 | 723 |
if len(self._sourcesterms) < 2: |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
724 |
self.needsplit = False |
2165
dd40c375044e
#344177: ms planning bug with "Any X WHERE X eid 123" and 123 coming from an external source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2114
diff
changeset
|
725 |
# if this is not the system source but we have only constant terms |
dd40c375044e
#344177: ms planning bug with "Any X WHERE X eid 123" and 123 coming from an external source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2114
diff
changeset
|
726 |
# and no relation (other than eid), apply query on the system source |
dd40c375044e
#344177: ms planning bug with "Any X WHERE X eid 123" and 123 coming from an external source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2114
diff
changeset
|
727 |
# |
dd40c375044e
#344177: ms planning bug with "Any X WHERE X eid 123" and 123 coming from an external source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2114
diff
changeset
|
728 |
# testing for rqlst with nothing in vargraph nor defined_vars is the |
dd40c375044e
#344177: ms planning bug with "Any X WHERE X eid 123" and 123 coming from an external source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2114
diff
changeset
|
729 |
# simplest way the check the condition explained below |
dd40c375044e
#344177: ms planning bug with "Any X WHERE X eid 123" and 123 coming from an external source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2114
diff
changeset
|
730 |
if not self.system_source in self._sourcesterms and \ |
2688
afa99d53401c
[multi-source] test and fix msplanner regression
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2170
diff
changeset
|
731 |
not self.rqlst.defined_vars and \ |
afa99d53401c
[multi-source] test and fix msplanner regression
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2170
diff
changeset
|
732 |
not need_source_access_relation(self.rqlst.vargraph): |
2165
dd40c375044e
#344177: ms planning bug with "Any X WHERE X eid 123" and 123 coming from an external source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2114
diff
changeset
|
733 |
self._sourcesterms = {self.system_source: {}} |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
734 |
elif not self.needsplit: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
735 |
if not allequals(self._sourcesterms.itervalues()): |
2170
6adbc965c51b
fix jpl/ticket/34862: msplanner bug with query generated to check if a user from ldap is owned_by himself
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2165
diff
changeset
|
736 |
for source, terms in self._sourcesterms.iteritems(): |
6adbc965c51b
fix jpl/ticket/34862: msplanner bug with query generated to check if a user from ldap is owned_by himself
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2165
diff
changeset
|
737 |
if source is self.system_source: |
6adbc965c51b
fix jpl/ticket/34862: msplanner bug with query generated to check if a user from ldap is owned_by himself
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2165
diff
changeset
|
738 |
continue |
2114
862f27d94af4
one more ms planning bug
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
2110
diff
changeset
|
739 |
if any(x for x in terms if not isinstance(x, Constant)): |
862f27d94af4
one more ms planning bug
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
2110
diff
changeset
|
740 |
self.needsplit = True |
862f27d94af4
one more ms planning bug
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
2110
diff
changeset
|
741 |
return |
862f27d94af4
one more ms planning bug
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
2110
diff
changeset
|
742 |
self._sourcesterms = {self.system_source: {}} |
862f27d94af4
one more ms planning bug
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
2110
diff
changeset
|
743 |
self.needsplit = False |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
744 |
else: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
745 |
sample = self._sourcesterms.itervalues().next() |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
746 |
if len(sample) > 1: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
747 |
for term in sample: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
748 |
# need split if unlinked variable |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
749 |
if isinstance(term, Variable) and not term in self._linkedterms: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
750 |
self.needsplit = True |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
751 |
break |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
752 |
else: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
753 |
# need split if there are some cross relation on non |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
754 |
# invariant variable or if the variable is used in |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
755 |
# multi-sources relation |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
756 |
if self._crossrelations: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
757 |
for reldict in self._crossrelations.itervalues(): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
758 |
for rel, terms in reldict.iteritems(): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
759 |
for term in terms: |
1230
232e16835fff
fix constant handling in sourcesterms + minor cleanup
sylvain.thenault@logilab.fr
parents:
1229
diff
changeset
|
760 |
if isinstance(term, Variable) \ |
232e16835fff
fix constant handling in sourcesterms + minor cleanup
sylvain.thenault@logilab.fr
parents:
1229
diff
changeset
|
761 |
and self._need_ext_source_access(term, rel): |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
762 |
self.needsplit = True |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
763 |
return |
6673
5eb2f1f2c277
[ms planning] test and fix bad plannification of a query used on cascading delete of a deleted (external) entity relations
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6654
diff
changeset
|
764 |
else: |
5eb2f1f2c277
[ms planning] test and fix bad plannification of a query used on cascading delete of a deleted (external) entity relations
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6654
diff
changeset
|
765 |
# remove sources only accessing to constant nodes |
5eb2f1f2c277
[ms planning] test and fix bad plannification of a query used on cascading delete of a deleted (external) entity relations
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6654
diff
changeset
|
766 |
for source, terms in self._sourcesterms.items(): |
5eb2f1f2c277
[ms planning] test and fix bad plannification of a query used on cascading delete of a deleted (external) entity relations
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6654
diff
changeset
|
767 |
if source is self.system_source: |
5eb2f1f2c277
[ms planning] test and fix bad plannification of a query used on cascading delete of a deleted (external) entity relations
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6654
diff
changeset
|
768 |
continue |
5eb2f1f2c277
[ms planning] test and fix bad plannification of a query used on cascading delete of a deleted (external) entity relations
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6654
diff
changeset
|
769 |
if not any(x for x in terms if not isinstance(x, Constant)): |
5eb2f1f2c277
[ms planning] test and fix bad plannification of a query used on cascading delete of a deleted (external) entity relations
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6654
diff
changeset
|
770 |
del self._sourcesterms[source] |
5eb2f1f2c277
[ms planning] test and fix bad plannification of a query used on cascading delete of a deleted (external) entity relations
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6654
diff
changeset
|
771 |
if len(self._sourcesterms) < 2: |
5eb2f1f2c277
[ms planning] test and fix bad plannification of a query used on cascading delete of a deleted (external) entity relations
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6654
diff
changeset
|
772 |
self.needsplit = False |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
773 |
|
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
774 |
@cached |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
775 |
def _need_ext_source_access(self, var, rel): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
776 |
if not var._q_invariant: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
777 |
return True |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
778 |
if any(r for x, r in self._linkedterms[var] |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
779 |
if not r is rel and self._repo.is_multi_sources_relation(r.r_type)): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
780 |
return True |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
781 |
return False |
1785 | 782 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
783 |
def _set_source_for_term(self, source, term): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
784 |
self._sourcesterms.setdefault(source, {})[term] = set(self._solindices) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
785 |
|
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
786 |
def _term_sources(self, term): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
787 |
"""returns possible sources for terms `term`""" |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
788 |
if isinstance(term, Constant): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
789 |
source = self._session.source_from_eid(term.eval(self.plan.args)) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
790 |
return set((source, solindex) for solindex in self._solindices) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
791 |
else: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
792 |
var = getattr(term, 'variable', term) |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
793 |
sources = [source for source, varobjs in self.sourcesterms.iteritems() |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
794 |
if var in varobjs] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
795 |
return set((source, solindex) for source in sources |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
796 |
for solindex in self.sourcesterms[source][var]) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
797 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
798 |
def _remove_sources(self, term, sources): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
799 |
"""removes invalid sources (`sources`) from `sourcesterms` |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
800 |
|
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
801 |
:param sources: the list of sources to remove |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
802 |
:param term: the analyzed term |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
803 |
""" |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
804 |
sourcesterms = self._sourcesterms |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
805 |
for source, solindex in sources: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
806 |
try: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
807 |
sourcesterms[source][term].remove(solindex) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
808 |
except KeyError: |
1786
eccd1885d42e
one more msplanning bug fixed...
sylvain.thenault@logilab.fr
parents:
1785
diff
changeset
|
809 |
import rql.base as rqlb |
3624
05932c6f7db2
fix buggy assertion
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3239
diff
changeset
|
810 |
assert isinstance(term, (rqlb.BaseNode, Variable)), repr(term) |
1785 | 811 |
continue # may occur with subquery column alias |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
812 |
if not sourcesterms[source][term]: |
6427
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
813 |
self._remove_source_term(source, term) |
c8a5ac2d1eaa
[schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6131
diff
changeset
|
814 |
|
6633
7baea108d326
[ms] simplify cw_source handling : don't try evil optmization that may hurt in some cases
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6598
diff
changeset
|
815 |
def _remove_source_term(self, source, term): |
7baea108d326
[ms] simplify cw_source handling : don't try evil optmization that may hurt in some cases
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6598
diff
changeset
|
816 |
try: |
7baea108d326
[ms] simplify cw_source handling : don't try evil optmization that may hurt in some cases
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6598
diff
changeset
|
817 |
poped = self._sourcesterms[source].pop(term, None) |
7baea108d326
[ms] simplify cw_source handling : don't try evil optmization that may hurt in some cases
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6598
diff
changeset
|
818 |
except KeyError: |
7baea108d326
[ms] simplify cw_source handling : don't try evil optmization that may hurt in some cases
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6598
diff
changeset
|
819 |
pass |
7baea108d326
[ms] simplify cw_source handling : don't try evil optmization that may hurt in some cases
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6598
diff
changeset
|
820 |
else: |
7baea108d326
[ms] simplify cw_source handling : don't try evil optmization that may hurt in some cases
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6598
diff
changeset
|
821 |
if not self._sourcesterms[source]: |
7baea108d326
[ms] simplify cw_source handling : don't try evil optmization that may hurt in some cases
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6598
diff
changeset
|
822 |
del self._sourcesterms[source] |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
823 |
|
393 | 824 |
def crossed_relation(self, source, relation): |
825 |
return relation in self._crossrelations.get(source, ()) |
|
1785 | 826 |
|
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
827 |
def part_steps(self): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
828 |
"""precompute necessary part steps before generating actual rql for |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
829 |
each step. This is necessary to know if an aggregate step will be |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
830 |
necessary or not. |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
831 |
""" |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
832 |
steps = [] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
833 |
select = self.rqlst |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
834 |
rschema = self._schema.rschema |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
835 |
for source in self.part_sources: |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
836 |
try: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
837 |
sourceterms = self._sourcesterms[source] |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
838 |
except KeyError: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
839 |
continue # already proceed |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
840 |
while sourceterms: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
841 |
# take a term randomly, and all terms supporting the |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
842 |
# same solutions |
6794
140d42b41b31
[multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6759
diff
changeset
|
843 |
term, solindices = self._choose_term(source, sourceterms) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
844 |
if source.uri == 'system': |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
845 |
# ensure all variables are available for the latest step |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
846 |
# (missing one will be available from temporary tables |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
847 |
# of previous steps) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
848 |
scope = select |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
849 |
terms = scope.defined_vars.values() + scope.aliases.values() |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
850 |
sourceterms.clear() |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
851 |
sources = [source] |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
852 |
else: |
5582
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
853 |
scope = ms_scope(term) |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
854 |
# find which sources support the same term and solutions |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
855 |
sources = self._expand_sources(source, term, solindices) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
856 |
# no try to get as much terms as possible |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
857 |
terms = self._expand_terms(term, sources, sourceterms, |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
858 |
scope, solindices) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
859 |
if len(terms) == 1 and isinstance(terms[0], Constant): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
860 |
# we can't generate anything interesting with a single |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
861 |
# constant term (will generate an empty "Any" query), |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
862 |
# go to the next iteration directly! |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
863 |
continue |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
864 |
if not sourceterms: |
4721
8f63691ccb7f
pylint style fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4719
diff
changeset
|
865 |
try: |
8f63691ccb7f
pylint style fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4719
diff
changeset
|
866 |
del self._sourcesterms[source] |
8f63691ccb7f
pylint style fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4719
diff
changeset
|
867 |
except KeyError: |
8f63691ccb7f
pylint style fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4719
diff
changeset
|
868 |
# XXX already cleaned |
8f63691ccb7f
pylint style fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4719
diff
changeset
|
869 |
pass |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
870 |
# set of terms which should be additionaly selected when |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
871 |
# possible |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
872 |
needsel = set() |
6794
140d42b41b31
[multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6759
diff
changeset
|
873 |
if not self._sourcesterms and scope is select: |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
874 |
terms += scope.defined_vars.values() + scope.aliases.values() |
6758
28b11ecf319b
[ms] #1382452: incorrect results with multi-sources
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6673
diff
changeset
|
875 |
if isinstance(term, Relation) and len(sources) > 1: |
28b11ecf319b
[ms] #1382452: incorrect results with multi-sources
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6673
diff
changeset
|
876 |
variants = set() |
28b11ecf319b
[ms] #1382452: incorrect results with multi-sources
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6673
diff
changeset
|
877 |
partterms = [term] |
28b11ecf319b
[ms] #1382452: incorrect results with multi-sources
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6673
diff
changeset
|
878 |
for vref in term.get_nodes(VariableRef): |
28b11ecf319b
[ms] #1382452: incorrect results with multi-sources
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6673
diff
changeset
|
879 |
if not vref.variable._q_invariant: |
28b11ecf319b
[ms] #1382452: incorrect results with multi-sources
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6673
diff
changeset
|
880 |
variants.add(vref.name) |
28b11ecf319b
[ms] #1382452: incorrect results with multi-sources
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6673
diff
changeset
|
881 |
if len(variants) == 2: |
28b11ecf319b
[ms] #1382452: incorrect results with multi-sources
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6673
diff
changeset
|
882 |
# we need an extra-step to fetch relations from each source |
28b11ecf319b
[ms] #1382452: incorrect results with multi-sources
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6673
diff
changeset
|
883 |
# before a join with prefetched inputs |
28b11ecf319b
[ms] #1382452: incorrect results with multi-sources
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6673
diff
changeset
|
884 |
# (see test_crossed_relation_noeid_needattr in |
28b11ecf319b
[ms] #1382452: incorrect results with multi-sources
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6673
diff
changeset
|
885 |
# unittest_msplanner / unittest_multisources) |
28b11ecf319b
[ms] #1382452: incorrect results with multi-sources
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6673
diff
changeset
|
886 |
lhs, rhs = term.get_variable_parts() |
28b11ecf319b
[ms] #1382452: incorrect results with multi-sources
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6673
diff
changeset
|
887 |
steps.append( (sources, [term, getattr(lhs, 'variable', lhs), |
28b11ecf319b
[ms] #1382452: incorrect results with multi-sources
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6673
diff
changeset
|
888 |
getattr(rhs, 'variable', rhs)], |
6794
140d42b41b31
[multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6759
diff
changeset
|
889 |
solindices, scope, variants, False) ) |
6758
28b11ecf319b
[ms] #1382452: incorrect results with multi-sources
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6673
diff
changeset
|
890 |
sources = [self.system_source] |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
891 |
final = True |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
892 |
else: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
893 |
# suppose this is a final step until the contrary is proven |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
894 |
final = scope is select |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
895 |
# add attribute variables and mark variables which should be |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
896 |
# additionaly selected when possible |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
897 |
for var in select.defined_vars.itervalues(): |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
898 |
if not var in terms: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
899 |
stinfo = var.stinfo |
5004
4cc020ee70e2
le patch rql26 a été importé
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4956
diff
changeset
|
900 |
for ovar, rtype in stinfo.get('attrvars', ()): |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
901 |
if ovar in terms: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
902 |
needsel.add(var.name) |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
903 |
terms.append(var) |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
904 |
break |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
905 |
else: |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
906 |
needsel.add(var.name) |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
907 |
final = False |
6649
29f8e5c35392
[ms planning] fix/enhance cw_source consideration in ms planning
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6633
diff
changeset
|
908 |
# check all relations are supported by the sources |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
909 |
for rel in scope.iget_nodes(Relation): |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
910 |
if rel.is_types_restriction(): |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
911 |
continue |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
912 |
# take care not overwriting the existing "source" identifier |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
913 |
for _source in sources: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
914 |
if not _source.support_relation(rel.r_type) or ( |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
915 |
self.crossed_relation(_source, rel) and not rel in terms): |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
916 |
for vref in rel.iget_nodes(VariableRef): |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
917 |
needsel.add(vref.name) |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
918 |
final = False |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
919 |
break |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
920 |
else: |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
921 |
if not scope is select: |
6794
140d42b41b31
[multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6759
diff
changeset
|
922 |
self._exists_relation(rel, terms, needsel, source) |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
923 |
# if relation is supported by all sources and some of |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
924 |
# its lhs/rhs variable isn't in "terms", and the |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
925 |
# other end *is* in "terms", mark it have to be |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
926 |
# selected |
3689
deb13e88e037
follow yams 0.25 api changes to improve performance
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3624
diff
changeset
|
927 |
if source.uri != 'system' and not rschema(rel.r_type).final: |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
928 |
lhs, rhs = rel.get_variable_parts() |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
929 |
try: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
930 |
lhsvar = lhs.variable |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
931 |
except AttributeError: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
932 |
lhsvar = lhs |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
933 |
try: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
934 |
rhsvar = rhs.variable |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
935 |
except AttributeError: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
936 |
rhsvar = rhs |
6131
087c5a168010
[ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6129
diff
changeset
|
937 |
try: |
087c5a168010
[ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6129
diff
changeset
|
938 |
if lhsvar in terms and not rhsvar in terms: |
087c5a168010
[ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6129
diff
changeset
|
939 |
needsel.add(lhsvar.name) |
087c5a168010
[ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6129
diff
changeset
|
940 |
elif rhsvar in terms and not lhsvar in terms: |
087c5a168010
[ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6129
diff
changeset
|
941 |
needsel.add(rhsvar.name) |
087c5a168010
[ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6129
diff
changeset
|
942 |
except AttributeError: |
087c5a168010
[ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6129
diff
changeset
|
943 |
continue # not an attribute, no selection needed |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
944 |
if final and source.uri != 'system': |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
945 |
# check rewritten constants |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
946 |
for vconsts in select.stinfo['rewritten'].itervalues(): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
947 |
const = vconsts[0] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
948 |
eid = const.eval(self.plan.args) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
949 |
_source = self._session.source_from_eid(eid) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
950 |
if len(sources) > 1 or not _source in sources: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
951 |
# if there is some rewriten constant used by a not |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
952 |
# neged relation while there are some source not |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
953 |
# supporting the associated entity, this step can't |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
954 |
# be final (unless the relation is explicitly in |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
955 |
# `terms`, eg cross relations) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
956 |
for c in vconsts: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
957 |
rel = c.relation() |
5582
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
958 |
if rel is None or not (rel in terms or neged_relation(rel)): |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
959 |
final = False |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
960 |
break |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
961 |
break |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
962 |
if final: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
963 |
self._cleanup_sourcesterms(sources, solindices) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
964 |
steps.append((sources, terms, solindices, scope, needsel, final) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
965 |
) |
6794
140d42b41b31
[multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6759
diff
changeset
|
966 |
if not steps[-1][-1]: |
140d42b41b31
[multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6759
diff
changeset
|
967 |
# add a final step |
140d42b41b31
[multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6759
diff
changeset
|
968 |
terms = select.defined_vars.values() + select.aliases.values() |
140d42b41b31
[multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6759
diff
changeset
|
969 |
steps.append( ([self.system_source], terms, set(self._solindices), |
140d42b41b31
[multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6759
diff
changeset
|
970 |
select, set(), True) ) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
971 |
return steps |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
972 |
|
6794
140d42b41b31
[multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6759
diff
changeset
|
973 |
def _exists_relation(self, rel, terms, needsel, source): |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
974 |
rschema = self._schema.rschema(rel.r_type) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
975 |
lhs, rhs = rel.get_variable_parts() |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
976 |
try: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
977 |
lhsvar, rhsvar = lhs.variable, rhs.variable |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
978 |
except AttributeError: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
979 |
pass |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
980 |
else: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
981 |
# supported relation with at least one end supported, check the |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
982 |
# other end is in as well. If not this usually means the |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
983 |
# variable is refed by an outer scope and should be substituted |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
984 |
# using an 'identity' relation (else we'll get a conflict of |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
985 |
# temporary tables) |
6794
140d42b41b31
[multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6759
diff
changeset
|
986 |
relscope = ms_scope(rel) |
140d42b41b31
[multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6759
diff
changeset
|
987 |
lhsscope = ms_scope(lhsvar) |
140d42b41b31
[multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6759
diff
changeset
|
988 |
rhsscope = ms_scope(rhsvar) |
140d42b41b31
[multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6759
diff
changeset
|
989 |
if rhsvar in terms and not lhsvar in terms and lhsscope is lhsvar.stmt: |
140d42b41b31
[multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6759
diff
changeset
|
990 |
self._identity_substitute(rel, lhsvar, terms, needsel, relscope) |
140d42b41b31
[multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6759
diff
changeset
|
991 |
elif lhsvar in terms and not rhsvar in terms and rhsscope is rhsvar.stmt: |
140d42b41b31
[multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6759
diff
changeset
|
992 |
self._identity_substitute(rel, rhsvar, terms, needsel, relscope) |
140d42b41b31
[multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6759
diff
changeset
|
993 |
elif self.crossed_relation(source, rel): |
140d42b41b31
[multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6759
diff
changeset
|
994 |
if lhsscope is not relscope: |
140d42b41b31
[multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6759
diff
changeset
|
995 |
self._identity_substitute(rel, lhsvar, terms, needsel, |
140d42b41b31
[multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6759
diff
changeset
|
996 |
relscope, lhsscope) |
140d42b41b31
[multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6759
diff
changeset
|
997 |
if rhsscope is not relscope: |
140d42b41b31
[multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6759
diff
changeset
|
998 |
self._identity_substitute(rel, rhsvar, terms, needsel, |
140d42b41b31
[multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6759
diff
changeset
|
999 |
relscope, rhsscope) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1000 |
|
6794
140d42b41b31
[multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6759
diff
changeset
|
1001 |
def _identity_substitute(self, relation, var, terms, needsel, exist, |
140d42b41b31
[multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6759
diff
changeset
|
1002 |
idrelscope=None): |
140d42b41b31
[multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6759
diff
changeset
|
1003 |
newvar = self._insert_identity_variable(exist, var, idrelscope) |
5004
4cc020ee70e2
le patch rql26 a été importé
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4956
diff
changeset
|
1004 |
# ensure relation is using '=' operator, else we rely on a |
4cc020ee70e2
le patch rql26 a été importé
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4956
diff
changeset
|
1005 |
# sqlgenerator side effect (it won't insert an inequality operator |
4cc020ee70e2
le patch rql26 a été importé
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4956
diff
changeset
|
1006 |
# in this case) |
4cc020ee70e2
le patch rql26 a été importé
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4956
diff
changeset
|
1007 |
relation.children[1].operator = '=' |
4cc020ee70e2
le patch rql26 a été importé
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4956
diff
changeset
|
1008 |
terms.append(newvar) |
4cc020ee70e2
le patch rql26 a été importé
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4956
diff
changeset
|
1009 |
needsel.add(newvar.name) |
1785 | 1010 |
|
6794
140d42b41b31
[multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6759
diff
changeset
|
1011 |
def _choose_term(self, source, sourceterms): |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1012 |
"""pick one term among terms supported by a source, which will be used |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1013 |
as a base to generate an execution step |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1014 |
""" |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1015 |
secondchoice = None |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1016 |
if len(self._sourcesterms) > 1: |
6794
140d42b41b31
[multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6759
diff
changeset
|
1017 |
# first, return non invariant variable of crossed relation, then the |
140d42b41b31
[multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6759
diff
changeset
|
1018 |
# crossed relation itself |
140d42b41b31
[multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6759
diff
changeset
|
1019 |
for term in sourceterms: |
140d42b41b31
[multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6759
diff
changeset
|
1020 |
if (isinstance(term, Relation) |
140d42b41b31
[multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6759
diff
changeset
|
1021 |
and self.crossed_relation(source, term) |
140d42b41b31
[multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6759
diff
changeset
|
1022 |
and not ms_scope(term) is self.rqlst): |
140d42b41b31
[multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6759
diff
changeset
|
1023 |
for vref in term.get_variable_parts(): |
140d42b41b31
[multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6759
diff
changeset
|
1024 |
try: |
140d42b41b31
[multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6759
diff
changeset
|
1025 |
var = vref.variable |
140d42b41b31
[multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6759
diff
changeset
|
1026 |
except AttributeError: |
140d42b41b31
[multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6759
diff
changeset
|
1027 |
# Constant |
140d42b41b31
[multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6759
diff
changeset
|
1028 |
continue |
140d42b41b31
[multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6759
diff
changeset
|
1029 |
if ((len(var.stinfo['relations']) > 1 or var.stinfo['selected']) |
140d42b41b31
[multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6759
diff
changeset
|
1030 |
and var in sourceterms): |
140d42b41b31
[multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6759
diff
changeset
|
1031 |
return var, sourceterms.pop(var) |
140d42b41b31
[multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6759
diff
changeset
|
1032 |
return term, sourceterms.pop(term) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1033 |
# priority to variable from subscopes |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1034 |
for term in sourceterms: |
5582
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
1035 |
if not ms_scope(term) is self.rqlst: |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1036 |
if isinstance(term, Variable): |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1037 |
return term, sourceterms.pop(term) |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1038 |
secondchoice = term |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1039 |
else: |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1040 |
# priority to variable from outer scope |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1041 |
for term in sourceterms: |
5582
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
1042 |
if ms_scope(term) is self.rqlst: |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1043 |
if isinstance(term, Variable): |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1044 |
return term, sourceterms.pop(term) |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1045 |
secondchoice = term |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1046 |
if secondchoice is not None: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1047 |
return secondchoice, sourceterms.pop(secondchoice) |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1048 |
# priority to variable with the less solutions supported and with the |
1237 | 1049 |
# most valuable refs. Add variable name for test predictability |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1050 |
variables = sorted([(var, sols) for (var, sols) in sourceterms.items() |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1051 |
if isinstance(var, Variable)], |
1237 | 1052 |
key=lambda (v, s): (len(s), -v.valuable_references(), v.name)) |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1053 |
if variables: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1054 |
var = variables[0][0] |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1055 |
return var, sourceterms.pop(var) |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1056 |
# priority to constant |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1057 |
for term in sourceterms: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1058 |
if isinstance(term, Constant): |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1059 |
return term, sourceterms.pop(term) |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1060 |
# whatever (relation) |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1061 |
term = iter(sourceterms).next() |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1062 |
return term, sourceterms.pop(term) |
1785 | 1063 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1064 |
def _expand_sources(self, selected_source, term, solindices): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1065 |
"""return all sources supporting given term / solindices""" |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1066 |
sources = [selected_source] |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1067 |
sourcesterms = self._sourcesterms |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1068 |
for source in sourcesterms.keys(): |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1069 |
if source is selected_source: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1070 |
continue |
1785 | 1071 |
if not (term in sourcesterms[source] and |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1072 |
solindices.issubset(sourcesterms[source][term])): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1073 |
continue |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1074 |
sources.append(source) |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1075 |
if source.uri != 'system' or not (isinstance(term, Variable) and not term in self._linkedterms): |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1076 |
termsolindices = sourcesterms[source][term] |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1077 |
termsolindices -= solindices |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1078 |
if not termsolindices: |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1079 |
del sourcesterms[source][term] |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1080 |
if not sourcesterms[source]: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1081 |
del sourcesterms[source] |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1082 |
return sources |
1785 | 1083 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1084 |
def _expand_terms(self, term, sources, sourceterms, scope, solindices): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1085 |
terms = [term] |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1086 |
sources = sorted(sources) |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1087 |
sourcesterms = self._sourcesterms |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1088 |
linkedterms = self._linkedterms |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1089 |
# term has to belong to the same scope if there is more |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1090 |
# than the system source remaining |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1091 |
if len(sourcesterms) > 1 and not scope is self.rqlst: |
5582
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
1092 |
candidates = (t for t in sourceterms.keys() if scope is ms_scope(t)) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1093 |
else: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1094 |
candidates = sourceterms #.iterkeys() |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1095 |
# we only want one unlinked term in each generated query |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1096 |
candidates = [t for t in candidates |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1097 |
if isinstance(t, (Constant, Relation)) or |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1098 |
(solindices.issubset(sourceterms[t]) and t in linkedterms)] |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1099 |
cross_rels = {} |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1100 |
for source in sources: |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1101 |
cross_rels.update(self._crossrelations.get(source, {})) |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1102 |
exclude = {} |
4719
aaed3f813ef8
kill dead/useless code as suggested by pylint
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4716
diff
changeset
|
1103 |
for crossvars in cross_rels.itervalues(): |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1104 |
vars = [t for t in crossvars if isinstance(t, Variable)] |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1105 |
try: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1106 |
exclude[vars[0]] = vars[1] |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1107 |
exclude[vars[1]] = vars[0] |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1108 |
except IndexError: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1109 |
pass |
6131
087c5a168010
[ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6129
diff
changeset
|
1110 |
accept_term = lambda x: (not any(s for s in sources |
087c5a168010
[ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6129
diff
changeset
|
1111 |
if not x in sourcesterms.get(s, ())) |
087c5a168010
[ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6129
diff
changeset
|
1112 |
and x._ms_may_be_processed(terms, linkedterms) |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1113 |
and not exclude.get(x) in terms) |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1114 |
if isinstance(term, Relation) and term in cross_rels: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1115 |
cross_terms = cross_rels.pop(term) |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1116 |
base_accept_term = accept_term |
6598
78eaccfbd2b7
[ms] fix an infinite loop in ms planner and another crash
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6427
diff
changeset
|
1117 |
accept_term = lambda x: (base_accept_term(x) or x in cross_terms) |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1118 |
for refed in cross_terms: |
393 | 1119 |
if not refed in candidates: |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1120 |
terms.append(refed) |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1121 |
# repeat until no term can't be added, since addition of a new |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1122 |
# term may permit to another one to be added |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1123 |
modified = True |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1124 |
while modified and candidates: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1125 |
modified = False |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1126 |
for term in candidates[:]: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1127 |
if isinstance(term, Constant): |
6131
087c5a168010
[ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6129
diff
changeset
|
1128 |
termsources = set(x[0] for x in self._term_sources(term)) |
087c5a168010
[ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6129
diff
changeset
|
1129 |
# ensure system source is there for constant |
087c5a168010
[ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6129
diff
changeset
|
1130 |
if self.system_source in sources: |
087c5a168010
[ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6129
diff
changeset
|
1131 |
termsources.add(self.system_source) |
087c5a168010
[ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6129
diff
changeset
|
1132 |
if sorted(termsources) != sources: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1133 |
continue |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1134 |
terms.append(term) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1135 |
candidates.remove(term) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1136 |
modified = True |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1137 |
del sourceterms[term] |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1138 |
elif accept_term(term): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1139 |
terms.append(term) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1140 |
candidates.remove(term) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1141 |
modified = True |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1142 |
self._cleanup_sourcesterms(sources, solindices, term) |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1143 |
return terms |
1785 | 1144 |
|
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1145 |
def _cleanup_sourcesterms(self, sources, solindices, term=None): |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1146 |
"""remove solutions so we know they are already processed""" |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1147 |
for source in sources: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1148 |
try: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1149 |
sourceterms = self._sourcesterms[source] |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1150 |
except KeyError: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1151 |
continue |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1152 |
if term is None: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1153 |
for term, termsolindices in sourceterms.items(): |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1154 |
if isinstance(term, Relation) and self.crossed_relation(source, term): |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1155 |
continue |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1156 |
termsolindices -= solindices |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1157 |
if not termsolindices: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1158 |
del sourceterms[term] |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1159 |
else: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1160 |
try: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1161 |
sourceterms[term] -= solindices |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1162 |
if not sourceterms[term]: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1163 |
del sourceterms[term] |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1164 |
except KeyError: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1165 |
pass |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1166 |
#assert term in cross_terms |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1167 |
if not sourceterms: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1168 |
del self._sourcesterms[source] |
1785 | 1169 |
|
6654
18d159a2d1ba
[ms planner] fix changeset 6650:72f2fd93a622: we should sometime complete input map anyway
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6651
diff
changeset
|
1170 |
def merge_input_maps(self, allsolindices, complete=True): |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1171 |
"""inputmaps is a dictionary with tuple of solution indices as key with |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1172 |
an associated input map as value. This function compute for each |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1173 |
solution its necessary input map and return them grouped |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1174 |
|
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1175 |
ex: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1176 |
inputmaps = {(0, 1, 2): {'A': 't1.login1', 'U': 't1.C0', 'U.login': 't1.login1'}, |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1177 |
(1,): {'X': 't2.C0', 'T': 't2.C1'}} |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1178 |
return : [([1], {'A': 't1.login1', 'U': 't1.C0', 'U.login': 't1.login1', |
1785 | 1179 |
'X': 't2.C0', 'T': 't2.C1'}), |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1180 |
([0,2], {'A': 't1.login1', 'U': 't1.C0', 'U.login': 't1.login1'})] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1181 |
""" |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1182 |
if not self._inputmaps: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1183 |
return [(allsolindices, None)] |
6650
72f2fd93a622
[ms planning] fix bug in merge_input_map causing bad plans computed
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6649
diff
changeset
|
1184 |
_allsolindices = allsolindices.copy() |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1185 |
mapbysol = {} |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1186 |
# compute a single map for each solution |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1187 |
for solindices, basemap in self._inputmaps.iteritems(): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1188 |
for solindex in solindices: |
6654
18d159a2d1ba
[ms planner] fix changeset 6650:72f2fd93a622: we should sometime complete input map anyway
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6651
diff
changeset
|
1189 |
if not (complete or solindex in allsolindices): |
6650
72f2fd93a622
[ms planning] fix bug in merge_input_map causing bad plans computed
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6649
diff
changeset
|
1190 |
continue |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1191 |
solmap = mapbysol.setdefault(solindex, {}) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1192 |
solmap.update(basemap) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1193 |
try: |
6650
72f2fd93a622
[ms planning] fix bug in merge_input_map causing bad plans computed
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6649
diff
changeset
|
1194 |
_allsolindices.remove(solindex) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1195 |
except KeyError: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1196 |
continue # already removed |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1197 |
# group results by identical input map |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1198 |
result = [] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1199 |
for solindex, solmap in mapbysol.iteritems(): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1200 |
for solindices, commonmap in result: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1201 |
if commonmap == solmap: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1202 |
solindices.append(solindex) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1203 |
break |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1204 |
else: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1205 |
result.append( ([solindex], solmap) ) |
6650
72f2fd93a622
[ms planning] fix bug in merge_input_map causing bad plans computed
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6649
diff
changeset
|
1206 |
if _allsolindices: |
72f2fd93a622
[ms planning] fix bug in merge_input_map causing bad plans computed
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6649
diff
changeset
|
1207 |
result.append( (list(_allsolindices), None) ) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1208 |
return result |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1209 |
|
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1210 |
def build_final_part(self, select, solindices, inputmap, sources, |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1211 |
insertedvars): |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1212 |
solutions = [self._solutions[i] for i in solindices] |
6598
78eaccfbd2b7
[ms] fix an infinite loop in ms planner and another crash
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6427
diff
changeset
|
1213 |
if self._conflicts and inputmap: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1214 |
for varname, mappedto in self._conflicts: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1215 |
var = select.defined_vars[varname] |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1216 |
newvar = select.make_variable() |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1217 |
# XXX should use var.scope but scope hasn't been computed yet |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1218 |
select.add_relation(var, 'identity', newvar) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1219 |
for sol in solutions: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1220 |
sol[newvar.name] = sol[varname] |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1221 |
inputmap[newvar.name] = mappedto |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1222 |
rqlst = self.plan.finalize(select, solutions, insertedvars) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1223 |
if self.temptable is None and self.finaltable is None: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1224 |
return OneFetchStep(self.plan, rqlst, sources, inputmap=inputmap) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1225 |
table = self.temptable or self.finaltable |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1226 |
return FetchStep(self.plan, rqlst, sources, table, True, inputmap) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1227 |
|
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1228 |
def build_non_final_part(self, select, solindices, sources, insertedvars, |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1229 |
table): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1230 |
"""non final step, will have to store results in a temporary table""" |
7188
b2c45b7396fb
[ms planner] use an index to avoid doing the same step twice
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6957
diff
changeset
|
1231 |
inputmapkey = tuple(sorted(solindices)) |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1232 |
solutions = [self._solutions[i] for i in solindices] |
7188
b2c45b7396fb
[ms planner] use an index to avoid doing the same step twice
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6957
diff
changeset
|
1233 |
# XXX be smarter vs rql comparison |
b2c45b7396fb
[ms planner] use an index to avoid doing the same step twice
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6957
diff
changeset
|
1234 |
idx_key = (select.as_string(), inputmapkey, |
b2c45b7396fb
[ms planner] use an index to avoid doing the same step twice
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6957
diff
changeset
|
1235 |
tuple(sorted(sources)), tuple(sorted(insertedvars))) |
b2c45b7396fb
[ms planner] use an index to avoid doing the same step twice
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6957
diff
changeset
|
1236 |
try: |
b2c45b7396fb
[ms planner] use an index to avoid doing the same step twice
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6957
diff
changeset
|
1237 |
# if a similar step has already been process, simply backport its |
b2c45b7396fb
[ms planner] use an index to avoid doing the same step twice
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6957
diff
changeset
|
1238 |
# input map |
b2c45b7396fb
[ms planner] use an index to avoid doing the same step twice
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6957
diff
changeset
|
1239 |
step = self.plan.ms_steps_idx[idx_key] |
b2c45b7396fb
[ms planner] use an index to avoid doing the same step twice
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6957
diff
changeset
|
1240 |
except KeyError: |
b2c45b7396fb
[ms planner] use an index to avoid doing the same step twice
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6957
diff
changeset
|
1241 |
# processing needed |
b2c45b7396fb
[ms planner] use an index to avoid doing the same step twice
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6957
diff
changeset
|
1242 |
rqlst = self.plan.finalize(select, solutions, insertedvars) |
b2c45b7396fb
[ms planner] use an index to avoid doing the same step twice
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6957
diff
changeset
|
1243 |
step = FetchStep(self.plan, rqlst, sources, table, False) |
b2c45b7396fb
[ms planner] use an index to avoid doing the same step twice
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6957
diff
changeset
|
1244 |
self.plan.ms_steps_idx[idx_key] = step |
b2c45b7396fb
[ms planner] use an index to avoid doing the same step twice
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6957
diff
changeset
|
1245 |
self.plan.add_step(step) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1246 |
# update input map for following steps, according to processed solutions |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1247 |
inputmap = self._inputmaps.setdefault(inputmapkey, {}) |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1248 |
for varname, mapping in step.outputmap.iteritems(): |
6651
4a0283b3ce0c
[ms planning] fix potential key error due to additional entries in input maps
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6650
diff
changeset
|
1249 |
if varname in inputmap and not '.' in varname and \ |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1250 |
not (mapping == inputmap[varname] or |
3689
deb13e88e037
follow yams 0.25 api changes to improve performance
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3624
diff
changeset
|
1251 |
self._schema.eschema(solutions[0][varname]).final): |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1252 |
self._conflicts.append((varname, inputmap[varname])) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1253 |
inputmap.update(step.outputmap) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1254 |
|
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1255 |
|
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1256 |
class MSPlanner(SSPlanner): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1257 |
"""MultiSourcesPlanner: build execution plan for rql queries |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1258 |
|
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1259 |
decompose the RQL query according to sources'schema |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1260 |
""" |
1785 | 1261 |
|
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1262 |
def build_select_plan(self, plan, rqlst): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1263 |
"""build execution plan for a SELECT RQL query |
1785 | 1264 |
|
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1265 |
the rqlst should not be tagged at this point |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1266 |
""" |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1267 |
# preprocess deals with security insertion and returns a new syntax tree |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1268 |
# which have to be executed to fulfill the query: according |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1269 |
# to permissions for variable's type, different rql queries may have to |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1270 |
# be executed |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1271 |
plan.preprocess(rqlst) |
6027
f91b72def783
[ms] log after preprocessing, else we don't see the query which is actually planned (may have been modified by security insertion)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5989
diff
changeset
|
1272 |
if server.DEBUG & server.DBG_MS: |
f91b72def783
[ms] log after preprocessing, else we don't see the query which is actually planned (may have been modified by security insertion)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5989
diff
changeset
|
1273 |
print '-'*80 |
f91b72def783
[ms] log after preprocessing, else we don't see the query which is actually planned (may have been modified by security insertion)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5989
diff
changeset
|
1274 |
print 'PLANNING', rqlst |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1275 |
ppis = [PartPlanInformation(plan, select, self.rqlhelper) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1276 |
for select in rqlst.children] |
7188
b2c45b7396fb
[ms planner] use an index to avoid doing the same step twice
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6957
diff
changeset
|
1277 |
plan.ms_steps_idx = {} |
4722
9c13d5db03d9
pylint suggested refactorings
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4721
diff
changeset
|
1278 |
steps = self._union_plan(plan, ppis) |
2766
07e47b84acc2
DBG_MS flag
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2688
diff
changeset
|
1279 |
if server.DEBUG & server.DBG_MS: |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1280 |
from pprint import pprint |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1281 |
for step in plan.steps: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1282 |
pprint(step.test_repr()) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1283 |
pprint(steps[0].test_repr()) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1284 |
return steps |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1285 |
|
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1286 |
def _ppi_subqueries(self, ppi): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1287 |
# part plan info for subqueries |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1288 |
plan = ppi.plan |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1289 |
inputmap = {} |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1290 |
for subquery in ppi.rqlst.with_[:]: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1291 |
sppis = [PartPlanInformation(plan, select) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1292 |
for select in subquery.query.children] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1293 |
for sppi in sppis: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1294 |
if sppi.needsplit or sppi.part_sources != ppi.part_sources: |
5168
1ab032df5ca3
SQL Server port: temporary table handling
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
4956
diff
changeset
|
1295 |
temptable = plan.make_temp_table_name('T%s' % make_uid(id(subquery))) |
4722
9c13d5db03d9
pylint suggested refactorings
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4721
diff
changeset
|
1296 |
sstep = self._union_plan(plan, sppis, temptable)[0] |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1297 |
break |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1298 |
else: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1299 |
sstep = None |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1300 |
if sstep is not None: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1301 |
ppi.rqlst.with_.remove(subquery) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1302 |
for i, colalias in enumerate(subquery.aliases): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1303 |
inputmap[colalias.name] = '%s.C%s' % (temptable, i) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1304 |
ppi.plan.add_step(sstep) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1305 |
return inputmap |
1785 | 1306 |
|
4722
9c13d5db03d9
pylint suggested refactorings
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4721
diff
changeset
|
1307 |
def _union_plan(self, plan, ppis, temptable=None): |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1308 |
tosplit, cango, allsources = [], {}, set() |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1309 |
for planinfo in ppis: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1310 |
if planinfo.needsplit: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1311 |
tosplit.append(planinfo) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1312 |
else: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1313 |
cango.setdefault(planinfo.part_sources, []).append(planinfo) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1314 |
for source in planinfo.part_sources: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1315 |
allsources.add(source) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1316 |
# first add steps for query parts which doesn't need to splitted |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1317 |
steps = [] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1318 |
for sources, cppis in cango.iteritems(): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1319 |
byinputmap = {} |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1320 |
for ppi in cppis: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1321 |
select = ppi.rqlst |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1322 |
if sources != (ppi.system_source,): |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1323 |
add_types_restriction(self.schema, select) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1324 |
# part plan info for subqueries |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1325 |
inputmap = self._ppi_subqueries(ppi) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1326 |
aggrstep = need_aggr_step(select, sources) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1327 |
if aggrstep: |
5168
1ab032df5ca3
SQL Server port: temporary table handling
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
4956
diff
changeset
|
1328 |
atemptable = plan.make_temp_table_name('T%s' % make_uid(id(select))) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1329 |
sunion = Union() |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1330 |
sunion.append(select) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1331 |
selected = select.selection[:] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1332 |
select_group_sort(select) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1333 |
step = AggrStep(plan, selected, select, atemptable, temptable) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1334 |
step.set_limit_offset(select.limit, select.offset) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1335 |
select.limit = None |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1336 |
select.offset = 0 |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1337 |
fstep = FetchStep(plan, sunion, sources, atemptable, True, inputmap) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1338 |
step.children.append(fstep) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1339 |
steps.append(step) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1340 |
else: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1341 |
byinputmap.setdefault(tuple(inputmap.iteritems()), []).append( (select) ) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1342 |
for inputmap, queries in byinputmap.iteritems(): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1343 |
inputmap = dict(inputmap) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1344 |
sunion = Union() |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1345 |
for select in queries: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1346 |
sunion.append(select) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1347 |
if temptable: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1348 |
steps.append(FetchStep(plan, sunion, sources, temptable, True, inputmap)) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1349 |
else: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1350 |
steps.append(OneFetchStep(plan, sunion, sources, inputmap)) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1351 |
# then add steps for splitted query parts |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1352 |
for planinfo in tosplit: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1353 |
steps.append(self.split_part(planinfo, temptable)) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1354 |
if len(steps) > 1: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1355 |
if temptable: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1356 |
step = UnionFetchStep(plan) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1357 |
else: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1358 |
step = UnionStep(plan) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1359 |
step.children = steps |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1360 |
return (step,) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1361 |
return steps |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1362 |
|
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1363 |
# internal methods for multisources decomposition ######################### |
1785 | 1364 |
|
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1365 |
def split_part(self, ppi, temptable): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1366 |
ppi.finaltable = temptable |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1367 |
plan = ppi.plan |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1368 |
select = ppi.rqlst |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1369 |
subinputmap = self._ppi_subqueries(ppi) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1370 |
stepdefs = ppi.part_steps() |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1371 |
if need_aggr_step(select, ppi.part_sources, stepdefs): |
5168
1ab032df5ca3
SQL Server port: temporary table handling
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents:
4956
diff
changeset
|
1372 |
atemptable = plan.make_temp_table_name('T%s' % make_uid(id(select))) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1373 |
selection = select.selection[:] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1374 |
select_group_sort(select) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1375 |
else: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1376 |
atemptable = None |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1377 |
selection = select.selection |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1378 |
ppi.temptable = atemptable |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1379 |
vfilter = TermsFiltererVisitor(self.schema, ppi) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1380 |
steps = [] |
6654
18d159a2d1ba
[ms planner] fix changeset 6650:72f2fd93a622: we should sometime complete input map anyway
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6651
diff
changeset
|
1381 |
multifinal = len([x for x in stepdefs if x[-1]]) >= 2 |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1382 |
for sources, terms, solindices, scope, needsel, final in stepdefs: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1383 |
# extract an executable query using only the specified terms |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1384 |
if sources[0].uri == 'system': |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1385 |
# in this case we have to merge input maps before call to |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1386 |
# filter so already processed restriction are correctly |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1387 |
# removed |
6759
5d016d5bacca
[ms] fix ms planner crash on some ambiguous queries, as demonstrated in a comment on http://www.cubicweb.org/ticket/1382452
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6758
diff
changeset
|
1388 |
solsinputmaps = ppi.merge_input_maps( |
5d016d5bacca
[ms] fix ms planner crash on some ambiguous queries, as demonstrated in a comment on http://www.cubicweb.org/ticket/1382452
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6758
diff
changeset
|
1389 |
solindices, complete=not (final and multifinal)) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1390 |
for solindices, inputmap in solsinputmaps: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1391 |
minrqlst, insertedvars = vfilter.filter( |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1392 |
sources, terms, scope, set(solindices), needsel, final) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1393 |
if inputmap is None: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1394 |
inputmap = subinputmap |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1395 |
else: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1396 |
inputmap.update(subinputmap) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1397 |
steps.append(ppi.build_final_part(minrqlst, solindices, inputmap, |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1398 |
sources, insertedvars)) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1399 |
else: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1400 |
# this is a final part (i.e. retreiving results for the |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1401 |
# original query part) if all term / sources have been |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1402 |
# treated or if this is the last shot for used solutions |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1403 |
minrqlst, insertedvars = vfilter.filter( |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1404 |
sources, terms, scope, solindices, needsel, final) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1405 |
if final: |
6759
5d016d5bacca
[ms] fix ms planner crash on some ambiguous queries, as demonstrated in a comment on http://www.cubicweb.org/ticket/1382452
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6758
diff
changeset
|
1406 |
solsinputmaps = ppi.merge_input_maps( |
5d016d5bacca
[ms] fix ms planner crash on some ambiguous queries, as demonstrated in a comment on http://www.cubicweb.org/ticket/1382452
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6758
diff
changeset
|
1407 |
solindices, complete=not (final and multifinal)) |
6129
fea746b60093
[ms] fix two planner bugs: one occuring query such as X created_by U where X in a external source and U may come from an ldap source. The other being that when we've to merge input maps, we were modifying the same tree/solutions while a copy were needed. Also, ensure we add type restrictions, necessary for pyro source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6128
diff
changeset
|
1408 |
if len(solsinputmaps) > 1: |
fea746b60093
[ms] fix two planner bugs: one occuring query such as X created_by U where X in a external source and U may come from an ldap source. The other being that when we've to merge input maps, we were modifying the same tree/solutions while a copy were needed. Also, ensure we add type restrictions, necessary for pyro source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6128
diff
changeset
|
1409 |
refrqlst = minrqlst |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1410 |
for solindices, inputmap in solsinputmaps: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1411 |
if inputmap is None: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1412 |
inputmap = subinputmap |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1413 |
else: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1414 |
inputmap.update(subinputmap) |
6129
fea746b60093
[ms] fix two planner bugs: one occuring query such as X created_by U where X in a external source and U may come from an ldap source. The other being that when we've to merge input maps, we were modifying the same tree/solutions while a copy were needed. Also, ensure we add type restrictions, necessary for pyro source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6128
diff
changeset
|
1415 |
if len(solsinputmaps) > 1: |
fea746b60093
[ms] fix two planner bugs: one occuring query such as X created_by U where X in a external source and U may come from an ldap source. The other being that when we've to merge input maps, we were modifying the same tree/solutions while a copy were needed. Also, ensure we add type restrictions, necessary for pyro source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6128
diff
changeset
|
1416 |
minrqlst = refrqlst.copy() |
fea746b60093
[ms] fix two planner bugs: one occuring query such as X created_by U where X in a external source and U may come from an ldap source. The other being that when we've to merge input maps, we were modifying the same tree/solutions while a copy were needed. Also, ensure we add type restrictions, necessary for pyro source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6128
diff
changeset
|
1417 |
sources = sources[:] |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1418 |
if inputmap and len(sources) > 1: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1419 |
sources.remove(ppi.system_source) |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1420 |
steps.append(ppi.build_final_part(minrqlst, solindices, None, |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1421 |
sources, insertedvars)) |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1422 |
steps.append(ppi.build_final_part(minrqlst, solindices, inputmap, |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1423 |
[ppi.system_source], insertedvars)) |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1424 |
else: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1425 |
steps.append(ppi.build_final_part(minrqlst, solindices, inputmap, |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1426 |
sources, insertedvars)) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1427 |
else: |
7191
1e2d8e4d2c7b
[ms planner] fix buggy temp table name generation, may generate duplicated unique id with subqueries/union
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7188
diff
changeset
|
1428 |
table = plan.make_temp_table_name('T%s' % make_uid(id(select))) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1429 |
ppi.build_non_final_part(minrqlst, solindices, sources, |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1430 |
insertedvars, table) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1431 |
# finally: join parts, deal with aggregat/group/sorts if necessary |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1432 |
if atemptable is not None: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1433 |
step = AggrStep(plan, selection, select, atemptable, temptable) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1434 |
step.children = steps |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1435 |
elif len(steps) > 1: |
5582
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
1436 |
getrschema = self.schema.rschema |
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
1437 |
if need_intersect(select, getrschema) or any(need_intersect(select, getrschema) |
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
1438 |
for step in steps |
3e133b29a1a4
[rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5426
diff
changeset
|
1439 |
for select in step.union.children): |
341
0a426be2f3a2
fixes for some cases of neged inlined relations, may have to use (new) IntersectStep. XXX: write IntersectFetchStep
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
257
diff
changeset
|
1440 |
if temptable: |
4716
55b6a3262071
fix some pylint detected errors
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4212
diff
changeset
|
1441 |
step = IntersectFetchStep(plan) # XXX not implemented |
341
0a426be2f3a2
fixes for some cases of neged inlined relations, may have to use (new) IntersectStep. XXX: write IntersectFetchStep
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
257
diff
changeset
|
1442 |
else: |
0a426be2f3a2
fixes for some cases of neged inlined relations, may have to use (new) IntersectStep. XXX: write IntersectFetchStep
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
257
diff
changeset
|
1443 |
step = IntersectStep(plan) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1444 |
else: |
341
0a426be2f3a2
fixes for some cases of neged inlined relations, may have to use (new) IntersectStep. XXX: write IntersectFetchStep
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
257
diff
changeset
|
1445 |
if temptable: |
0a426be2f3a2
fixes for some cases of neged inlined relations, may have to use (new) IntersectStep. XXX: write IntersectFetchStep
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
257
diff
changeset
|
1446 |
step = UnionFetchStep(plan) |
0a426be2f3a2
fixes for some cases of neged inlined relations, may have to use (new) IntersectStep. XXX: write IntersectFetchStep
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
257
diff
changeset
|
1447 |
else: |
0a426be2f3a2
fixes for some cases of neged inlined relations, may have to use (new) IntersectStep. XXX: write IntersectFetchStep
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
257
diff
changeset
|
1448 |
step = UnionStep(plan) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1449 |
step.children = steps |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1450 |
else: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1451 |
step = steps[0] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1452 |
if select.limit is not None or select.offset: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1453 |
step.set_limit_offset(select.limit, select.offset) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1454 |
return step |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1455 |
|
1785 | 1456 |
|
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1457 |
class UnsupportedBranch(Exception): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1458 |
pass |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1459 |
|
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1460 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1461 |
class TermsFiltererVisitor(object): |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1462 |
def __init__(self, schema, ppi): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1463 |
self.schema = schema |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1464 |
self.ppi = ppi |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1465 |
self.skip = {} |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1466 |
self.hasaggrstep = self.ppi.temptable |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1467 |
self.extneedsel = frozenset(vref.name for sortterm in ppi.rqlst.orderby |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1468 |
for vref in sortterm.iget_nodes(VariableRef)) |
1785 | 1469 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1470 |
def _rqlst_accept(self, rqlst, node, newroot, terms, setfunc=None): |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1471 |
try: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1472 |
newrestr, node_ = node.accept(self, newroot, terms[:]) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1473 |
except UnsupportedBranch: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1474 |
return rqlst |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1475 |
if setfunc is not None and newrestr is not None: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1476 |
setfunc(newrestr) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1477 |
if not node_ is node: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1478 |
rqlst = node.parent |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1479 |
return rqlst |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1480 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1481 |
def filter(self, sources, terms, rqlst, solindices, needsel, final): |
2766
07e47b84acc2
DBG_MS flag
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2688
diff
changeset
|
1482 |
if server.DEBUG & server.DBG_MS: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1483 |
print 'filter', final and 'final' or '', sources, terms, rqlst, solindices, needsel |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1484 |
newroot = Select() |
393 | 1485 |
self.sources = sorted(sources) |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1486 |
self.terms = terms |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1487 |
self.solindices = solindices |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1488 |
self.final = final |
2109
f469368df530
when a relation isn't supported by a ms query parts, dereference introduced variable refs
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2108
diff
changeset
|
1489 |
self._pending_vrefs = [] |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1490 |
# terms which appear in unsupported branches |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1491 |
needsel |= self.extneedsel |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1492 |
self.needsel = needsel |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1493 |
# terms which appear in supported branches |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1494 |
self.mayneedsel = set() |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1495 |
# new inserted variables |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1496 |
self.insertedvars = [] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1497 |
# other structures (XXX document) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1498 |
self.mayneedvar, self.hasvar = {}, {} |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1499 |
self.use_only_defined = False |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1500 |
self.scopes = {rqlst: newroot} |
2075
933ccfce6e91
fix bug in multisource planner, w/ inlined relation
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1977
diff
changeset
|
1501 |
self.current_scope = rqlst |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1502 |
if rqlst.where: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1503 |
rqlst = self._rqlst_accept(rqlst, rqlst.where, newroot, terms, |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1504 |
newroot.set_where) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1505 |
if isinstance(rqlst, Select): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1506 |
self.use_only_defined = True |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1507 |
if rqlst.groupby: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1508 |
groupby = [] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1509 |
for node in rqlst.groupby: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1510 |
rqlst = self._rqlst_accept(rqlst, node, newroot, terms, |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1511 |
groupby.append) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1512 |
if groupby: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1513 |
newroot.set_groupby(groupby) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1514 |
if rqlst.having: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1515 |
having = [] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1516 |
for node in rqlst.having: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1517 |
rqlst = self._rqlst_accept(rqlst, node, newroot, terms, |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1518 |
having.append) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1519 |
if having: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1520 |
newroot.set_having(having) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1521 |
if final and rqlst.orderby and not self.hasaggrstep: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1522 |
orderby = [] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1523 |
for node in rqlst.orderby: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1524 |
rqlst = self._rqlst_accept(rqlst, node, newroot, terms, |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1525 |
orderby.append) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1526 |
if orderby: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1527 |
newroot.set_orderby(orderby) |
5768
1e73a466aa69
[fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5582
diff
changeset
|
1528 |
elif rqlst.orderby: |
1e73a466aa69
[fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5582
diff
changeset
|
1529 |
for sortterm in rqlst.orderby: |
1e73a466aa69
[fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5582
diff
changeset
|
1530 |
if any(f for f in sortterm.iget_nodes(Function) if f.name == 'FTIRANK'): |
1e73a466aa69
[fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5582
diff
changeset
|
1531 |
newnode, oldnode = sortterm.accept(self, newroot, terms) |
1e73a466aa69
[fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5582
diff
changeset
|
1532 |
if newnode is not None: |
1e73a466aa69
[fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5582
diff
changeset
|
1533 |
newroot.add_sort_term(newnode) |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1534 |
self.process_selection(newroot, terms, rqlst) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1535 |
elif not newroot.where: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1536 |
# no restrictions have been copied, just select terms and add |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1537 |
# type restriction (done later by add_types_restriction) |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1538 |
for v in terms: |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1539 |
if not isinstance(v, Variable): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1540 |
continue |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1541 |
newroot.append_selected(VariableRef(newroot.get_variable(v.name))) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1542 |
solutions = self.ppi.copy_solutions(solindices) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1543 |
cleanup_solutions(newroot, solutions) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1544 |
newroot.set_possible_types(solutions) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1545 |
if final: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1546 |
if self.hasaggrstep: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1547 |
self.add_necessary_selection(newroot, self.mayneedsel & self.extneedsel) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1548 |
newroot.distinct = rqlst.distinct |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1549 |
else: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1550 |
self.add_necessary_selection(newroot, self.mayneedsel & self.needsel) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1551 |
# insert vars to fetch constant values when needed |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1552 |
for (varname, rschema), reldefs in self.mayneedvar.iteritems(): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1553 |
for rel, ored in reldefs: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1554 |
if not (varname, rschema) in self.hasvar: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1555 |
self.hasvar[(varname, rschema)] = None # just to avoid further insertion |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1556 |
cvar = newroot.make_variable() |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1557 |
for sol in newroot.solutions: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1558 |
sol[cvar.name] = rschema.objects(sol[varname])[0] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1559 |
# if the current restriction is not used in a OR branch, |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1560 |
# we can keep it, else we have to drop the constant |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1561 |
# restriction (or we may miss some results) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1562 |
if not ored: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1563 |
rel = rel.copy(newroot) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1564 |
newroot.add_restriction(rel) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1565 |
# add a relation to link the variable |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1566 |
newroot.remove_node(rel.children[1]) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1567 |
cmp = Comparison('=') |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1568 |
rel.append(cmp) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1569 |
cmp.append(VariableRef(cvar)) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1570 |
self.insertedvars.append((varname, rschema, cvar.name)) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1571 |
newroot.append_selected(VariableRef(newroot.get_variable(cvar.name))) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1572 |
# NOTE: even if the restriction is done by this query, we have |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1573 |
# to let it in the original rqlst so that it appears anyway in |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1574 |
# the "final" query, else we may change the meaning of the query |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1575 |
# if there are NOT somewhere : |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1576 |
# 'NOT X relation Y, Y name "toto"' means X WHERE X isn't related |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1577 |
# to Y whose name is toto while |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1578 |
# 'NOT X relation Y' means X WHERE X has no 'relation' (whatever Y) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1579 |
elif ored: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1580 |
newroot.remove_node(rel) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1581 |
add_types_restriction(self.schema, rqlst, newroot, solutions) |
2766
07e47b84acc2
DBG_MS flag
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2688
diff
changeset
|
1582 |
if server.DEBUG & server.DBG_MS: |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1583 |
print '--->', newroot |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1584 |
return newroot, self.insertedvars |
1785 | 1585 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1586 |
def visit_and(self, node, newroot, terms): |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1587 |
subparts = [] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1588 |
for i in xrange(len(node.children)): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1589 |
child = node.children[i] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1590 |
try: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1591 |
newchild, child_ = child.accept(self, newroot, terms) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1592 |
if not child_ is child: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1593 |
node = child_.parent |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1594 |
if newchild is None: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1595 |
continue |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1596 |
subparts.append(newchild) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1597 |
except UnsupportedBranch: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1598 |
continue |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1599 |
if not subparts: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1600 |
return None, node |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1601 |
if len(subparts) == 1: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1602 |
return subparts[0], node |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1603 |
return copy_node(newroot, node, subparts), node |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1604 |
|
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1605 |
visit_or = visit_and |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1606 |
|
393 | 1607 |
def _relation_supported(self, relation): |
1608 |
rtype = relation.r_type |
|
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1609 |
for source in self.sources: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1610 |
if not source.support_relation(rtype) or ( |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1611 |
rtype in source.cross_relations and not relation in self.terms): |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1612 |
return False |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1613 |
if not self.final and not relation in self.terms: |
393 | 1614 |
rschema = self.schema.rschema(relation.r_type) |
3689
deb13e88e037
follow yams 0.25 api changes to improve performance
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3624
diff
changeset
|
1615 |
if not rschema.final: |
393 | 1616 |
for term in relation.get_nodes((VariableRef, Constant)): |
1617 |
term = getattr(term, 'variable', term) |
|
1618 |
termsources = sorted(set(x[0] for x in self.ppi._term_sources(term))) |
|
1619 |
if termsources and termsources != self.sources: |
|
1620 |
return False |
|
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1621 |
return True |
1785 | 1622 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1623 |
def visit_relation(self, node, newroot, terms): |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1624 |
if not node.is_types_restriction(): |
6758
28b11ecf319b
[ms] #1382452: incorrect results with multi-sources
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6673
diff
changeset
|
1625 |
if not node in terms and node in self.skip and self.solindices.issubset(self.skip[node]): |
7771
59ddc2dbe7e0
[multi-sources]Â closes #1918120: drop behaviour leading to wrong result as demonstrated by the ticket
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
7651
diff
changeset
|
1626 |
return None, node |
393 | 1627 |
if not self._relation_supported(node): |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1628 |
raise UnsupportedBranch() |
6056
37a37e5ba330
[ms] fix a ms plannification bug with NOT is
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6027
diff
changeset
|
1629 |
# don't copy type restriction unless this is the only supported relation |
37a37e5ba330
[ms] fix a ms plannification bug with NOT is
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6027
diff
changeset
|
1630 |
# for the lhs variable, else they'll be reinserted later as needed (in |
37a37e5ba330
[ms] fix a ms plannification bug with NOT is
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6027
diff
changeset
|
1631 |
# other cases we may copy a type restriction while the variable is not |
37a37e5ba330
[ms] fix a ms plannification bug with NOT is
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6027
diff
changeset
|
1632 |
# actually used) |
37a37e5ba330
[ms] fix a ms plannification bug with NOT is
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6027
diff
changeset
|
1633 |
elif not (node.neged(strict=True) or |
37a37e5ba330
[ms] fix a ms plannification bug with NOT is
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6027
diff
changeset
|
1634 |
any(self._relation_supported(rel) |
37a37e5ba330
[ms] fix a ms plannification bug with NOT is
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6027
diff
changeset
|
1635 |
for rel in node.children[0].variable.stinfo['relations'])): |
37a37e5ba330
[ms] fix a ms plannification bug with NOT is
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6027
diff
changeset
|
1636 |
return self.visit_default(node, newroot, terms) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1637 |
else: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1638 |
raise UnsupportedBranch() |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1639 |
rschema = self.schema.rschema(node.r_type) |
2109
f469368df530
when a relation isn't supported by a ms query parts, dereference introduced variable refs
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2108
diff
changeset
|
1640 |
self._pending_vrefs = [] |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1641 |
try: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1642 |
res = self.visit_default(node, newroot, terms)[0] |
2109
f469368df530
when a relation isn't supported by a ms query parts, dereference introduced variable refs
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2108
diff
changeset
|
1643 |
except: |
2110
d7773fb1b892
add comment explaining the pb
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2109
diff
changeset
|
1644 |
# when a relation isn't supported, we should dereference potentially |
d7773fb1b892
add comment explaining the pb
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2109
diff
changeset
|
1645 |
# introduced variable refs |
2109
f469368df530
when a relation isn't supported by a ms query parts, dereference introduced variable refs
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2108
diff
changeset
|
1646 |
for vref in self._pending_vrefs: |
f469368df530
when a relation isn't supported by a ms query parts, dereference introduced variable refs
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2108
diff
changeset
|
1647 |
vref.unregister_reference() |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1648 |
raise |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1649 |
ored = node.ored() |
3689
deb13e88e037
follow yams 0.25 api changes to improve performance
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3624
diff
changeset
|
1650 |
if rschema.final or rschema.inlined: |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1651 |
vrefs = node.children[1].get_nodes(VariableRef) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1652 |
if not vrefs: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1653 |
if not ored: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1654 |
self.skip.setdefault(node, set()).update(self.solindices) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1655 |
else: |
1785 | 1656 |
self.mayneedvar.setdefault((node.children[0].name, rschema), []).append( (res, ored) ) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1657 |
else: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1658 |
assert len(vrefs) == 1 |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1659 |
vref = vrefs[0] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1660 |
# XXX check operator ? |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1661 |
self.hasvar[(node.children[0].name, rschema)] = vref |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1662 |
if self._may_skip_attr_rel(rschema, node, vref, ored, terms, res): |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1663 |
self.skip.setdefault(node, set()).update(self.solindices) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1664 |
elif not ored: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1665 |
self.skip.setdefault(node, set()).update(self.solindices) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1666 |
return res, node |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1667 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1668 |
def _may_skip_attr_rel(self, rschema, rel, vref, ored, terms, res): |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1669 |
var = vref.variable |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1670 |
if ored: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1671 |
return False |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1672 |
if var.name in self.extneedsel or var.stinfo['selected']: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1673 |
return False |
2075
933ccfce6e91
fix bug in multisource planner, w/ inlined relation
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1977
diff
changeset
|
1674 |
if not var in terms or used_in_outer_scope(var, self.current_scope): |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1675 |
return False |
5004
4cc020ee70e2
le patch rql26 a été importé
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
4956
diff
changeset
|
1676 |
if any(v for v, _ in var.stinfo.get('attrvars', ()) if not v in terms): |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1677 |
return False |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1678 |
return True |
1785 | 1679 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1680 |
def visit_exists(self, node, newroot, terms): |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1681 |
newexists = node.__class__() |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1682 |
self.scopes = {node: newexists} |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1683 |
subparts, node = self._visit_children(node, newroot, terms) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1684 |
if not subparts: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1685 |
return None, node |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1686 |
newexists.set_where(subparts[0]) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1687 |
return newexists, node |
1785 | 1688 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1689 |
def visit_not(self, node, newroot, terms): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1690 |
subparts, node = self._visit_children(node, newroot, terms) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1691 |
if not subparts: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1692 |
return None, node |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1693 |
return copy_node(newroot, node, subparts), node |
1785 | 1694 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1695 |
def visit_group(self, node, newroot, terms): |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1696 |
if not self.final: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1697 |
return None, node |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1698 |
return self.visit_default(node, newroot, terms) |
1785 | 1699 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1700 |
def visit_variableref(self, node, newroot, terms): |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1701 |
if self.use_only_defined: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1702 |
if not node.variable.name in newroot.defined_vars: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1703 |
raise UnsupportedBranch(node.name) |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1704 |
elif not node.variable in terms: |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1705 |
raise UnsupportedBranch(node.name) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1706 |
self.mayneedsel.add(node.name) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1707 |
# set scope so we can insert types restriction properly |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1708 |
newvar = newroot.get_variable(node.name) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1709 |
newvar.stinfo['scope'] = self.scopes.get(node.variable.scope, newroot) |
2109
f469368df530
when a relation isn't supported by a ms query parts, dereference introduced variable refs
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2108
diff
changeset
|
1710 |
vref = VariableRef(newvar) |
f469368df530
when a relation isn't supported by a ms query parts, dereference introduced variable refs
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2108
diff
changeset
|
1711 |
self._pending_vrefs.append(vref) |
f469368df530
when a relation isn't supported by a ms query parts, dereference introduced variable refs
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2108
diff
changeset
|
1712 |
return vref, node |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1713 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1714 |
def visit_constant(self, node, newroot, terms): |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1715 |
return copy_node(newroot, node), node |
1785 | 1716 |
|
3815
50b87f759b5d
test and fix http://www.logilab.org/ticket/499838
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3689
diff
changeset
|
1717 |
def visit_comparison(self, node, newroot, terms): |
50b87f759b5d
test and fix http://www.logilab.org/ticket/499838
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3689
diff
changeset
|
1718 |
subparts, node = self._visit_children(node, newroot, terms) |
50b87f759b5d
test and fix http://www.logilab.org/ticket/499838
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3689
diff
changeset
|
1719 |
copy = copy_node(newroot, node, subparts) |
50b87f759b5d
test and fix http://www.logilab.org/ticket/499838
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3689
diff
changeset
|
1720 |
# ignore comparison operator when fetching non final query |
50b87f759b5d
test and fix http://www.logilab.org/ticket/499838
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3689
diff
changeset
|
1721 |
if not self.final and isinstance(node.children[0], VariableRef): |
50b87f759b5d
test and fix http://www.logilab.org/ticket/499838
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3689
diff
changeset
|
1722 |
copy.operator = '=' |
50b87f759b5d
test and fix http://www.logilab.org/ticket/499838
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3689
diff
changeset
|
1723 |
return copy, node |
50b87f759b5d
test and fix http://www.logilab.org/ticket/499838
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3689
diff
changeset
|
1724 |
|
5768
1e73a466aa69
[fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5582
diff
changeset
|
1725 |
def visit_function(self, node, newroot, terms): |
1e73a466aa69
[fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5582
diff
changeset
|
1726 |
if node.name == 'FTIRANK': |
1e73a466aa69
[fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5582
diff
changeset
|
1727 |
# FTIRANK is somewhat special... Rank function should be included in |
1e73a466aa69
[fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5582
diff
changeset
|
1728 |
# the same query has the has_text relation, potentially added to |
1e73a466aa69
[fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5582
diff
changeset
|
1729 |
# selection for latter usage |
1e73a466aa69
[fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5582
diff
changeset
|
1730 |
if not self.hasaggrstep and self.final and node not in self.skip: |
1e73a466aa69
[fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5582
diff
changeset
|
1731 |
return self.visit_default(node, newroot, terms) |
1e73a466aa69
[fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5582
diff
changeset
|
1732 |
elif any(s for s in self.sources if s.uri != 'system'): |
1e73a466aa69
[fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5582
diff
changeset
|
1733 |
return None, node |
1e73a466aa69
[fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5582
diff
changeset
|
1734 |
# p = node.parent |
1e73a466aa69
[fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5582
diff
changeset
|
1735 |
# while p is not None and not isinstance(p, SortTerm): |
1e73a466aa69
[fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5582
diff
changeset
|
1736 |
# p = p.parent |
1e73a466aa69
[fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5582
diff
changeset
|
1737 |
# if isinstance(p, SortTerm): |
1e73a466aa69
[fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5582
diff
changeset
|
1738 |
if not self.hasaggrstep and self.final and node in self.skip: |
1e73a466aa69
[fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5582
diff
changeset
|
1739 |
return Constant(self.skip[node], 'Int'), node |
1e73a466aa69
[fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5582
diff
changeset
|
1740 |
# XXX only if not yet selected |
1e73a466aa69
[fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5582
diff
changeset
|
1741 |
newroot.append_selected(node.copy(newroot)) |
1e73a466aa69
[fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5582
diff
changeset
|
1742 |
self.skip[node] = len(newroot.selection) |
1e73a466aa69
[fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5582
diff
changeset
|
1743 |
return None, node |
1e73a466aa69
[fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5582
diff
changeset
|
1744 |
return self.visit_default(node, newroot, terms) |
1e73a466aa69
[fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5582
diff
changeset
|
1745 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1746 |
def visit_default(self, node, newroot, terms): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1747 |
subparts, node = self._visit_children(node, newroot, terms) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1748 |
return copy_node(newroot, node, subparts), node |
1785 | 1749 |
|
5768
1e73a466aa69
[fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5582
diff
changeset
|
1750 |
visit_mathexpression = visit_constant = visit_default |
1e73a466aa69
[fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5582
diff
changeset
|
1751 |
|
1e73a466aa69
[fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5582
diff
changeset
|
1752 |
def visit_sortterm(self, node, newroot, terms): |
1e73a466aa69
[fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5582
diff
changeset
|
1753 |
subparts, node = self._visit_children(node, newroot, terms) |
1e73a466aa69
[fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5582
diff
changeset
|
1754 |
if not subparts: |
1e73a466aa69
[fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5582
diff
changeset
|
1755 |
return None, node |
1e73a466aa69
[fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
5582
diff
changeset
|
1756 |
return copy_node(newroot, node, subparts), node |
1785 | 1757 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1758 |
def _visit_children(self, node, newroot, terms): |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1759 |
subparts = [] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1760 |
for i in xrange(len(node.children)): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1761 |
child = node.children[i] |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1762 |
newchild, child_ = child.accept(self, newroot, terms) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1763 |
if not child is child_: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1764 |
node = child_.parent |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1765 |
if newchild is not None: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1766 |
subparts.append(newchild) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1767 |
return subparts, node |
1785 | 1768 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1769 |
def process_selection(self, newroot, terms, rqlst): |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1770 |
if self.final: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1771 |
for term in rqlst.selection: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1772 |
newroot.append_selected(term.copy(newroot)) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1773 |
for vref in term.get_nodes(VariableRef): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1774 |
self.needsel.add(vref.name) |
1785 | 1775 |
return |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1776 |
for term in rqlst.selection: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1777 |
vrefs = term.get_nodes(VariableRef) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1778 |
if vrefs: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1779 |
supportedvars = [] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1780 |
for vref in vrefs: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1781 |
var = vref.variable |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1782 |
if var in terms: |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1783 |
supportedvars.append(vref) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1784 |
continue |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1785 |
else: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1786 |
self.needsel.add(vref.name) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1787 |
break |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1788 |
else: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1789 |
for vref in vrefs: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1790 |
newroot.append_selected(vref.copy(newroot)) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1791 |
supportedvars = [] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1792 |
for vref in supportedvars: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1793 |
if not vref in newroot.get_selected_variables(): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1794 |
newroot.append_selected(VariableRef(newroot.get_variable(vref.name))) |
6131
087c5a168010
[ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6129
diff
changeset
|
1795 |
elif term in self.terms: |
087c5a168010
[ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
6129
diff
changeset
|
1796 |
newroot.append_selected(term.copy(newroot)) |
1785 | 1797 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1798 |
def add_necessary_selection(self, newroot, terms): |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1799 |
selected = tuple(newroot.get_selected_variables()) |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1800 |
for varname in terms: |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1801 |
var = newroot.defined_vars[varname] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1802 |
for vref in var.references(): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1803 |
rel = vref.relation() |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1804 |
if rel is None and vref in selected: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1805 |
# already selected |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1806 |
break |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1807 |
else: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1808 |
selvref = VariableRef(var) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1809 |
newroot.append_selected(selvref) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1810 |
if newroot.groupby: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1811 |
newroot.add_group_var(VariableRef(selvref.variable, noautoref=1)) |