author | Sylvain Thénault <sylvain.thenault@logilab.fr> |
Wed, 27 Jan 2010 10:17:27 +0100 | |
changeset 4397 | fbac86ca7af1 |
parent 4212 | ab6573088b4a |
child 4716 | 55b6a3262071 |
permissions | -rw-r--r-- |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1 |
"""plan execution of rql queries on multiple sources |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
2 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
3 |
the best way to understand what are we trying to acheive here is to read the |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
4 |
unit-tests in unittest_msplanner.py |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
5 |
|
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
6 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
7 |
What you need to know |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
8 |
~~~~~~~~~~~~~~~~~~~~~ |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
9 |
1. The system source is expected to support every entity and relation types |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
10 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
11 |
2. Given "X relation Y": |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
12 |
|
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
13 |
* if relation, X and Y types are supported by the external source, we suppose |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
14 |
by default that X and Y should both come from the same source as the |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
15 |
relation. You can specify otherwise by adding relation into the |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
16 |
"cross_relations" set in the source's mapping file and it that case, we'll |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
17 |
consider that we can also find in the system source some relation between |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
18 |
X and Y coming from different sources. |
1785 | 19 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
20 |
* if "relation" isn't supported by the external source but X or Y |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
21 |
types (or both) are, we suppose by default that can find in the system |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
22 |
source some relation where X and/or Y come from the external source. You |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
23 |
can specify otherwise by adding relation into the "dont_cross_relations" |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
24 |
set in the source's mapping file and it that case, we'll consider that we |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
25 |
can only find in the system source some relation between X and Y coming |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
26 |
the system source. |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
27 |
|
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
28 |
|
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
29 |
Implementation |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
30 |
~~~~~~~~~~~~~~ |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
31 |
XXX explain algorithm |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
32 |
|
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
33 |
|
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
34 |
Exemples of multi-sources query execution |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
35 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
1398
5fe84a5f7035
rename internal entity types to have CW prefix instead of E
sylvain.thenault@logilab.fr
parents:
1263
diff
changeset
|
36 |
For a system source and a ldap user source (only CWUser and its attributes |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
37 |
is supported, no group or such): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
38 |
|
1398
5fe84a5f7035
rename internal entity types to have CW prefix instead of E
sylvain.thenault@logilab.fr
parents:
1263
diff
changeset
|
39 |
:CWUser X: |
5fe84a5f7035
rename internal entity types to have CW prefix instead of E
sylvain.thenault@logilab.fr
parents:
1263
diff
changeset
|
40 |
1. fetch CWUser X from both sources and return concatenation of results |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
41 |
|
1398
5fe84a5f7035
rename internal entity types to have CW prefix instead of E
sylvain.thenault@logilab.fr
parents:
1263
diff
changeset
|
42 |
:CWUser X WHERE X in_group G, G name 'users': |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
43 |
* catch 1 |
1398
5fe84a5f7035
rename internal entity types to have CW prefix instead of E
sylvain.thenault@logilab.fr
parents:
1263
diff
changeset
|
44 |
1. fetch CWUser X from both sources, store concatenation of results into a |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
45 |
temporary table |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
46 |
2. return the result of TMP X WHERE X in_group G, G name 'users' from the |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
47 |
system source |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
48 |
* catch 2 |
1398
5fe84a5f7035
rename internal entity types to have CW prefix instead of E
sylvain.thenault@logilab.fr
parents:
1263
diff
changeset
|
49 |
1. return the result of CWUser X WHERE X in_group G, G name 'users' from system |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
50 |
source, that's enough (optimization of the sql querier will avoid join on |
1398
5fe84a5f7035
rename internal entity types to have CW prefix instead of E
sylvain.thenault@logilab.fr
parents:
1263
diff
changeset
|
51 |
CWUser, so we will directly get local eids) |
1785 | 52 |
|
1398
5fe84a5f7035
rename internal entity types to have CW prefix instead of E
sylvain.thenault@logilab.fr
parents:
1263
diff
changeset
|
53 |
:CWUser X,L WHERE X in_group G, X login L, G name 'users': |
5fe84a5f7035
rename internal entity types to have CW prefix instead of E
sylvain.thenault@logilab.fr
parents:
1263
diff
changeset
|
54 |
1. fetch Any X,L WHERE X is CWUser, X login L from both sources, store |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
55 |
concatenation of results into a temporary table |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
56 |
2. return the result of Any X, L WHERE X is TMP, X login LX in_group G, |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
57 |
G name 'users' from the system source |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
58 |
|
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
59 |
|
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
60 |
:Any X WHERE X owned_by Y: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
61 |
* catch 1 |
1398
5fe84a5f7035
rename internal entity types to have CW prefix instead of E
sylvain.thenault@logilab.fr
parents:
1263
diff
changeset
|
62 |
1. fetch CWUser X from both sources, store concatenation of results into a |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
63 |
temporary table |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
64 |
2. return the result of Any X WHERE X owned_by Y, Y is TMP from the system |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
65 |
source |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
66 |
* catch 2 |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
67 |
1. return the result of Any X WHERE X owned_by Y from system source, that's |
1398
5fe84a5f7035
rename internal entity types to have CW prefix instead of E
sylvain.thenault@logilab.fr
parents:
1263
diff
changeset
|
68 |
enough (optimization of the sql querier will avoid join on CWUser, so we |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
69 |
will directly get local eids) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
70 |
|
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
71 |
|
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
72 |
:organization: Logilab |
4212
ab6573088b4a
update copyright: welcome 2010
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
3815
diff
changeset
|
73 |
:copyright: 2003-2010 LOGILAB S.A. (Paris, FRANCE), license is LGPL v2. |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
74 |
:contact: http://www.logilab.fr/ -- mailto:contact@logilab.fr |
1977
606923dff11b
big bunch of copyright / docstring update
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
1786
diff
changeset
|
75 |
:license: GNU Lesser General Public License, v2.1 - http://www.gnu.org/licenses |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
76 |
""" |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
77 |
__docformat__ = "restructuredtext en" |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
78 |
|
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
79 |
from itertools import imap, ifilterfalse |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
80 |
|
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
81 |
from logilab.common.compat import any |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
82 |
from logilab.common.decorators import cached |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
83 |
|
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
84 |
from rql.stmts import Union, Select |
1132 | 85 |
from rql.nodes import VariableRef, Comparison, Relation, Constant, Variable |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
86 |
|
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
87 |
from cubicweb import server |
940
15dcdc863965
fix imports : common.utils -> utils
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
442
diff
changeset
|
88 |
from cubicweb.utils import make_uid |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
89 |
from cubicweb.server.utils import cleanup_solutions |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
90 |
from cubicweb.server.ssplanner import (SSPlanner, OneFetchStep, |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
91 |
add_types_restriction) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
92 |
from cubicweb.server.mssteps import * |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
93 |
from cubicweb.server.sources import AbstractSource |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
94 |
|
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
95 |
Variable._ms_table_key = lambda x: x.name |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
96 |
Relation._ms_table_key = lambda x: x.r_type |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
97 |
# str() Constant.value to ensure generated table name won't be unicode |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
98 |
Constant._ms_table_key = lambda x: str(x.value) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
99 |
|
2688
afa99d53401c
[multi-source] test and fix msplanner regression
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2170
diff
changeset
|
100 |
def need_source_access_relation(vargraph): |
afa99d53401c
[multi-source] test and fix msplanner regression
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2170
diff
changeset
|
101 |
if not vargraph: |
afa99d53401c
[multi-source] test and fix msplanner regression
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2170
diff
changeset
|
102 |
return False |
afa99d53401c
[multi-source] test and fix msplanner regression
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2170
diff
changeset
|
103 |
# check vargraph contains some other relation than the identity relation |
afa99d53401c
[multi-source] test and fix msplanner regression
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2170
diff
changeset
|
104 |
# test of key nature since it may be a variable name (don't care about that) |
afa99d53401c
[multi-source] test and fix msplanner regression
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2170
diff
changeset
|
105 |
# or a 2-uple (var1, var2) associated to the relation to traverse to go from |
afa99d53401c
[multi-source] test and fix msplanner regression
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2170
diff
changeset
|
106 |
# var1 to var2 |
afa99d53401c
[multi-source] test and fix msplanner regression
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2170
diff
changeset
|
107 |
return any(key for key, val in vargraph.iteritems() |
afa99d53401c
[multi-source] test and fix msplanner regression
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2170
diff
changeset
|
108 |
if isinstance(key, tuple) and val != 'identity') |
afa99d53401c
[multi-source] test and fix msplanner regression
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2170
diff
changeset
|
109 |
|
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
110 |
def need_aggr_step(select, sources, stepdefs=None): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
111 |
"""return True if a temporary table is necessary to store some partial |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
112 |
results to execute the given query |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
113 |
""" |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
114 |
if len(sources) == 1: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
115 |
# can do everything at once with a single source |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
116 |
return False |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
117 |
if select.orderby or select.groupby or select.has_aggregat: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
118 |
# if more than one source, we need a temp table to deal with sort / |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
119 |
# groups / aggregat if : |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
120 |
# * the rqlst won't be splitted (in the other case the last query |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
121 |
# using partial temporary table can do sort/groups/aggregat without |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
122 |
# the need for a later AggrStep) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
123 |
# * the rqlst is splitted in multiple steps and there are more than one |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
124 |
# final step |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
125 |
if stepdefs is None: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
126 |
return True |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
127 |
has_one_final = False |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
128 |
fstepsolindices = set() |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
129 |
for stepdef in stepdefs: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
130 |
if stepdef[-1]: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
131 |
if has_one_final or frozenset(stepdef[2]) != fstepsolindices: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
132 |
return True |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
133 |
has_one_final = True |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
134 |
else: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
135 |
fstepsolindices.update(stepdef[2]) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
136 |
return False |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
137 |
|
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
138 |
def select_group_sort(select): # XXX something similar done in rql2sql |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
139 |
# add variables used in groups and sort terms to the selection |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
140 |
# if necessary |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
141 |
if select.groupby: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
142 |
for vref in select.groupby: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
143 |
if not vref in select.selection: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
144 |
select.append_selected(vref.copy(select)) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
145 |
for sortterm in select.orderby: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
146 |
for vref in sortterm.iget_nodes(VariableRef): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
147 |
if not vref in select.get_selected_variables(): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
148 |
# we can't directly insert sortterm.term because it references |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
149 |
# a variable of the select before the copy. |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
150 |
# XXX if constant term are used to define sort, their value |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
151 |
# may necessite a decay |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
152 |
select.append_selected(vref.copy(select)) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
153 |
if select.groupby and not vref in select.groupby: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
154 |
select.add_group_var(vref.copy(select)) |
1120
1c24cde4bf72
fix ms planning w/ relation constraint propagation
sylvain.thenault@logilab.fr
parents:
442
diff
changeset
|
155 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
156 |
def allequals(solutions): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
157 |
"""return true if all solutions are identical""" |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
158 |
sol = solutions.next() |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
159 |
noconstsol = None |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
160 |
for sol_ in solutions: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
161 |
if sol_ != sol: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
162 |
return False |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
163 |
return True |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
164 |
|
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
165 |
# XXX move functions below to rql ############################################## |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
166 |
|
1120
1c24cde4bf72
fix ms planning w/ relation constraint propagation
sylvain.thenault@logilab.fr
parents:
442
diff
changeset
|
167 |
def is_ancestor(n1, n2): |
2075
933ccfce6e91
fix bug in multisource planner, w/ inlined relation
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1977
diff
changeset
|
168 |
"""return True if n2 is a parent scope of n1""" |
1120
1c24cde4bf72
fix ms planning w/ relation constraint propagation
sylvain.thenault@logilab.fr
parents:
442
diff
changeset
|
169 |
p = n1.parent |
1c24cde4bf72
fix ms planning w/ relation constraint propagation
sylvain.thenault@logilab.fr
parents:
442
diff
changeset
|
170 |
while p is not None: |
1c24cde4bf72
fix ms planning w/ relation constraint propagation
sylvain.thenault@logilab.fr
parents:
442
diff
changeset
|
171 |
if p is n2: |
1c24cde4bf72
fix ms planning w/ relation constraint propagation
sylvain.thenault@logilab.fr
parents:
442
diff
changeset
|
172 |
return True |
1c24cde4bf72
fix ms planning w/ relation constraint propagation
sylvain.thenault@logilab.fr
parents:
442
diff
changeset
|
173 |
p = p.parent |
1c24cde4bf72
fix ms planning w/ relation constraint propagation
sylvain.thenault@logilab.fr
parents:
442
diff
changeset
|
174 |
return False |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
175 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
176 |
def copy_node(newroot, node, subparts=()): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
177 |
newnode = node.__class__(*node.initargs(newroot)) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
178 |
for part in subparts: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
179 |
newnode.append(part) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
180 |
return newnode |
1785 | 181 |
|
2075
933ccfce6e91
fix bug in multisource planner, w/ inlined relation
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1977
diff
changeset
|
182 |
def used_in_outer_scope(var, scope): |
933ccfce6e91
fix bug in multisource planner, w/ inlined relation
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1977
diff
changeset
|
183 |
"""return true if the variable is used in an outer scope of the given scope |
933ccfce6e91
fix bug in multisource planner, w/ inlined relation
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1977
diff
changeset
|
184 |
""" |
933ccfce6e91
fix bug in multisource planner, w/ inlined relation
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1977
diff
changeset
|
185 |
for rel in var.stinfo['relations']: |
933ccfce6e91
fix bug in multisource planner, w/ inlined relation
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1977
diff
changeset
|
186 |
rscope = rel.scope |
933ccfce6e91
fix bug in multisource planner, w/ inlined relation
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1977
diff
changeset
|
187 |
if not rscope is scope and is_ancestor(scope, rscope): |
933ccfce6e91
fix bug in multisource planner, w/ inlined relation
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1977
diff
changeset
|
188 |
return True |
933ccfce6e91
fix bug in multisource planner, w/ inlined relation
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1977
diff
changeset
|
189 |
return False |
1785 | 190 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
191 |
################################################################################ |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
192 |
|
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
193 |
class PartPlanInformation(object): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
194 |
"""regroups necessary information to execute some part of a "global" rql |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
195 |
query ("global" means as received by the querier, which may result in |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
196 |
several internal queries, e.g. parts, due to security insertions). Actually |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
197 |
a PPI is created for each subquery and for each query in a union. |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
198 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
199 |
It exposes as well some methods helping in executing this part on a |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
200 |
multi-sources repository, modifying its internal structure during the |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
201 |
process. |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
202 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
203 |
:attr plan: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
204 |
the execution plan |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
205 |
:attr rqlst: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
206 |
the original rql syntax tree handled by this part |
1785 | 207 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
208 |
:attr needsplit: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
209 |
bool telling if the query has to be split into multiple steps for |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
210 |
execution or if it can be executed at once |
1785 | 211 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
212 |
:attr temptable: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
213 |
a SQL temporary table name or None, if necessary to handle aggregate / |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
214 |
sorting for this part of the query |
1785 | 215 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
216 |
:attr finaltable: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
217 |
a SQL table name or None, if results for this part of the query should be |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
218 |
written into a temporary table (usually shared by multiple PPI) |
1785 | 219 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
220 |
:attr sourcesterms: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
221 |
a dictionary {source : {term: set([solution index, ])}} telling for each |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
222 |
source which terms are supported for which solutions. A "term" may be |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
223 |
either a rql Variable, Constant or Relation node. |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
224 |
""" |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
225 |
def __init__(self, plan, rqlst, rqlhelper=None): |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
226 |
self.plan = plan |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
227 |
self.rqlst = rqlst |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
228 |
self.needsplit = False |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
229 |
self.temptable = None |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
230 |
self.finaltable = None |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
231 |
# shortcuts |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
232 |
self._schema = plan.schema |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
233 |
self._session = plan.session |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
234 |
self._repo = self._session.repo |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
235 |
self._solutions = rqlst.solutions |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
236 |
self._solindices = range(len(self._solutions)) |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
237 |
self.system_source = self._repo.system_source |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
238 |
# source : {term: [solution index, ]} |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
239 |
self.sourcesterms = self._sourcesterms = {} |
393 | 240 |
# source : {relation: set(child variable and constant)} |
241 |
self._crossrelations = {} |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
242 |
# dictionary of variables and constants which are linked to each other |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
243 |
# using a non final relation supported by multiple sources (crossed or |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
244 |
# not). |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
245 |
self._linkedterms = {} |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
246 |
# processing |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
247 |
termssources = self._compute_sourcesterms() |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
248 |
self._remove_invalid_sources(termssources) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
249 |
self._compute_needsplit() |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
250 |
# after initialisation, .sourcesterms contains the same thing as |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
251 |
# ._sourcesterms though during plan construction, ._sourcesterms will |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
252 |
# be modified while .sourcesterms will be kept unmodified |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
253 |
self.sourcesterms = {} |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
254 |
for k, v in self._sourcesterms.iteritems(): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
255 |
self.sourcesterms[k] = {} |
393 | 256 |
for k2, v2 in v.iteritems(): |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
257 |
self.sourcesterms[k][k2] = v2.copy() |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
258 |
# cleanup linked var |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
259 |
for var, linkedrelsinfo in self._linkedterms.iteritems(): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
260 |
self._linkedterms[var] = frozenset(x[0] for x in linkedrelsinfo) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
261 |
# map output of a step to input of a following step |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
262 |
self._inputmaps = {} |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
263 |
# record input map conflicts to resolve them on final step generation |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
264 |
self._conflicts = [] |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
265 |
if rqlhelper is not None: # else test |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
266 |
self._insert_identity_variable = rqlhelper._annotator.rewrite_shared_optional |
2766
07e47b84acc2
DBG_MS flag
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2688
diff
changeset
|
267 |
if server.DEBUG & server.DBG_MS: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
268 |
print 'sourcesterms:' |
1786
eccd1885d42e
one more msplanning bug fixed...
sylvain.thenault@logilab.fr
parents:
1785
diff
changeset
|
269 |
self._debug_sourcesterms() |
eccd1885d42e
one more msplanning bug fixed...
sylvain.thenault@logilab.fr
parents:
1785
diff
changeset
|
270 |
|
eccd1885d42e
one more msplanning bug fixed...
sylvain.thenault@logilab.fr
parents:
1785
diff
changeset
|
271 |
def _debug_sourcesterms(self): |
eccd1885d42e
one more msplanning bug fixed...
sylvain.thenault@logilab.fr
parents:
1785
diff
changeset
|
272 |
for source in self._sourcesterms: |
eccd1885d42e
one more msplanning bug fixed...
sylvain.thenault@logilab.fr
parents:
1785
diff
changeset
|
273 |
print '-', source |
eccd1885d42e
one more msplanning bug fixed...
sylvain.thenault@logilab.fr
parents:
1785
diff
changeset
|
274 |
for term, sols in self._sourcesterms[source].items(): |
eccd1885d42e
one more msplanning bug fixed...
sylvain.thenault@logilab.fr
parents:
1785
diff
changeset
|
275 |
print ' -', term, id(term), ':' ,sols |
1785 | 276 |
|
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
277 |
def copy_solutions(self, solindices): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
278 |
return [self._solutions[solidx].copy() for solidx in solindices] |
1785 | 279 |
|
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
280 |
@property |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
281 |
@cached |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
282 |
def part_sources(self): |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
283 |
if self._sourcesterms: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
284 |
return tuple(sorted(self._sourcesterms)) |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
285 |
return (self.system_source,) |
1785 | 286 |
|
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
287 |
@property |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
288 |
@cached |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
289 |
def _sys_source_set(self): |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
290 |
return frozenset((self.system_source, solindex) |
1785 | 291 |
for solindex in self._solindices) |
292 |
||
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
293 |
@cached |
393 | 294 |
def _norel_support_set(self, relation): |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
295 |
"""return a set of (source, solindex) where source doesn't support the |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
296 |
relation |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
297 |
""" |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
298 |
return frozenset((source, solidx) for source in self._repo.sources |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
299 |
for solidx in self._solindices |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
300 |
if not ((source.support_relation(relation.r_type)) |
393 | 301 |
or relation.r_type in source.dont_cross_relations)) |
302 |
||
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
303 |
def _compute_sourcesterms(self): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
304 |
"""compute for each term (variable, rewritten constant, relation) and |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
305 |
for each solution in the rqlst which sources support them |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
306 |
""" |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
307 |
repo = self._repo |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
308 |
eschema = self._schema.eschema |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
309 |
sourcesterms = self._sourcesterms |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
310 |
# find for each source which variable/solution are supported |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
311 |
for varname, varobj in self.rqlst.defined_vars.items(): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
312 |
# if variable has an eid specified, we can get its source directly |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
313 |
# NOTE: use uidrels and not constnode to deal with "X eid IN(1,2,3,4)" |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
314 |
if varobj.stinfo['uidrels']: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
315 |
vrels = varobj.stinfo['relations'] - varobj.stinfo['uidrels'] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
316 |
for rel in varobj.stinfo['uidrels']: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
317 |
if rel.neged(strict=True) or rel.operator() != '=': |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
318 |
continue |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
319 |
for const in rel.children[1].get_nodes(Constant): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
320 |
eid = const.eval(self.plan.args) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
321 |
source = self._session.source_from_eid(eid) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
322 |
if vrels and not any(source.support_relation(r.r_type) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
323 |
for r in vrels): |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
324 |
self._set_source_for_term(self.system_source, varobj) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
325 |
else: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
326 |
self._set_source_for_term(source, varobj) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
327 |
continue |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
328 |
rels = varobj.stinfo['relations'] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
329 |
if not rels and not varobj.stinfo['typerels']: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
330 |
# (rare) case where the variable has no type specified nor |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
331 |
# relation accessed ex. "Any MAX(X)" |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
332 |
self._set_source_for_term(self.system_source, varobj) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
333 |
continue |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
334 |
for i, sol in enumerate(self._solutions): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
335 |
vartype = sol[varname] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
336 |
# skip final variable |
3689
deb13e88e037
follow yams 0.25 api changes to improve performance
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3624
diff
changeset
|
337 |
if eschema(vartype).final: |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
338 |
break |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
339 |
for source in repo.sources: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
340 |
if source.support_entity(vartype): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
341 |
# the source support the entity type, though we will |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
342 |
# actually have to fetch from it only if |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
343 |
# * the variable isn't invariant |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
344 |
# * at least one supported relation specified |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
345 |
if not varobj._q_invariant or \ |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
346 |
any(imap(source.support_relation, |
3239
1ceac4cd4fb7
[ms] fix planning w/ query using identity
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3040
diff
changeset
|
347 |
(r.r_type for r in rels if r.r_type not in ('identity', 'eid')))): |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
348 |
sourcesterms.setdefault(source, {}).setdefault(varobj, set()).add(i) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
349 |
# if variable is not invariant and is used by a relation |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
350 |
# not supported by this source, we'll have to split the |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
351 |
# query |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
352 |
if not varobj._q_invariant and any(ifilterfalse( |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
353 |
source.support_relation, (r.r_type for r in rels))): |
1785 | 354 |
self.needsplit = True |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
355 |
# add source for rewritten constants to sourcesterms |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
356 |
for vconsts in self.rqlst.stinfo['rewritten'].itervalues(): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
357 |
const = vconsts[0] |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
358 |
source = self._session.source_from_eid(const.eval(self.plan.args)) |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
359 |
if source is self.system_source: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
360 |
for const in vconsts: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
361 |
self._set_source_for_term(source, const) |
2108
b3f6fa7ad59f
add test and fix for queries where all variables are simplified and come from an external source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2075
diff
changeset
|
362 |
elif not self._sourcesterms: |
b3f6fa7ad59f
add test and fix for queries where all variables are simplified and come from an external source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2075
diff
changeset
|
363 |
self._set_source_for_term(source, const) |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
364 |
elif source in self._sourcesterms: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
365 |
source_scopes = frozenset(t.scope for t in self._sourcesterms[source]) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
366 |
for const in vconsts: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
367 |
if const.scope in source_scopes: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
368 |
self._set_source_for_term(source, const) |
1230
232e16835fff
fix constant handling in sourcesterms + minor cleanup
sylvain.thenault@logilab.fr
parents:
1229
diff
changeset
|
369 |
# if system source is used, add every rewritten constant |
232e16835fff
fix constant handling in sourcesterms + minor cleanup
sylvain.thenault@logilab.fr
parents:
1229
diff
changeset
|
370 |
# to its supported terms even when associated entity |
2114
862f27d94af4
one more ms planning bug
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
2110
diff
changeset
|
371 |
# doesn't actually come from it so we get a changes |
1230
232e16835fff
fix constant handling in sourcesterms + minor cleanup
sylvain.thenault@logilab.fr
parents:
1229
diff
changeset
|
372 |
# that allequals will return True as expected when |
232e16835fff
fix constant handling in sourcesterms + minor cleanup
sylvain.thenault@logilab.fr
parents:
1229
diff
changeset
|
373 |
# computing needsplit |
1786
eccd1885d42e
one more msplanning bug fixed...
sylvain.thenault@logilab.fr
parents:
1785
diff
changeset
|
374 |
# check const is used in a relation restriction |
eccd1885d42e
one more msplanning bug fixed...
sylvain.thenault@logilab.fr
parents:
1785
diff
changeset
|
375 |
if const.relation() and self.system_source in sourcesterms: |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
376 |
self._set_source_for_term(self.system_source, const) |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
377 |
# add source for relations |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
378 |
rschema = self._schema.rschema |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
379 |
termssources = {} |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
380 |
for rel in self.rqlst.iget_nodes(Relation): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
381 |
# process non final relations only |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
382 |
# note: don't try to get schema for 'is' relation (not available |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
383 |
# during bootstrap) |
3689
deb13e88e037
follow yams 0.25 api changes to improve performance
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3624
diff
changeset
|
384 |
if not rel.is_types_restriction() and not rschema(rel.r_type).final: |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
385 |
# nothing to do if relation is not supported by multiple sources |
393 | 386 |
# or if some source has it listed in its cross_relations |
387 |
# attribute |
|
388 |
# |
|
389 |
# XXX code below don't deal if some source allow relation |
|
390 |
# crossing but not another one |
|
341
0a426be2f3a2
fixes for some cases of neged inlined relations, may have to use (new) IntersectStep. XXX: write IntersectFetchStep
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
257
diff
changeset
|
391 |
relsources = repo.rel_type_sources(rel.r_type) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
392 |
if len(relsources) < 2: |
393 | 393 |
# filter out sources being there because they have this |
394 |
# relation in their dont_cross_relations attribute |
|
395 |
relsources = [source for source in relsources |
|
396 |
if source.support_relation(rel.r_type)] |
|
341
0a426be2f3a2
fixes for some cases of neged inlined relations, may have to use (new) IntersectStep. XXX: write IntersectFetchStep
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
257
diff
changeset
|
397 |
if relsources: |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
398 |
# this means the relation is using a variable inlined as |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
399 |
# a constant and another unsupported variable, in which |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
400 |
# case we put the relation in sourcesterms |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
401 |
self._sourcesterms.setdefault(relsources[0], {})[rel] = set(self._solindices) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
402 |
continue |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
403 |
lhs, rhs = rel.get_variable_parts() |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
404 |
lhsv, rhsv = getattr(lhs, 'variable', lhs), getattr(rhs, 'variable', rhs) |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
405 |
# update dictionary of sources supporting lhs and rhs vars |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
406 |
if not lhsv in termssources: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
407 |
termssources[lhsv] = self._term_sources(lhs) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
408 |
if not rhsv in termssources: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
409 |
termssources[rhsv] = self._term_sources(rhs) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
410 |
self._handle_cross_relation(rel, relsources, termssources) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
411 |
self._linkedterms.setdefault(lhsv, set()).add((rhsv, rel)) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
412 |
self._linkedterms.setdefault(rhsv, set()).add((lhsv, rel)) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
413 |
return termssources |
1785 | 414 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
415 |
def _handle_cross_relation(self, rel, relsources, termssources): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
416 |
for source in relsources: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
417 |
if rel.r_type in source.cross_relations: |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
418 |
ssource = self.system_source |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
419 |
crossvars = set(x.variable for x in rel.get_nodes(VariableRef)) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
420 |
for const in rel.get_nodes(Constant): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
421 |
if source.uri != 'system' and not const in self._sourcesterms.get(source, ()): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
422 |
continue |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
423 |
crossvars.add(const) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
424 |
self._crossrelations.setdefault(source, {})[rel] = crossvars |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
425 |
if len(crossvars) < 2: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
426 |
# this means there is a constant in the relation which is |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
427 |
# not supported by the source, so we can stop here |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
428 |
continue |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
429 |
self._sourcesterms.setdefault(ssource, {})[rel] = set(self._solindices) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
430 |
for term in crossvars: |
1229
dd9bdcfc03b6
cleanup _handle_cross_relation
sylvain.thenault@logilab.fr
parents:
1228
diff
changeset
|
431 |
if len(termssources[term]) == 1 and iter(termssources[term]).next()[0].uri == 'system': |
dd9bdcfc03b6
cleanup _handle_cross_relation
sylvain.thenault@logilab.fr
parents:
1228
diff
changeset
|
432 |
for ov in crossvars: |
dd9bdcfc03b6
cleanup _handle_cross_relation
sylvain.thenault@logilab.fr
parents:
1228
diff
changeset
|
433 |
if ov is not term and (isinstance(ov, Constant) or ov._q_invariant): |
dd9bdcfc03b6
cleanup _handle_cross_relation
sylvain.thenault@logilab.fr
parents:
1228
diff
changeset
|
434 |
ssset = frozenset((ssource,)) |
dd9bdcfc03b6
cleanup _handle_cross_relation
sylvain.thenault@logilab.fr
parents:
1228
diff
changeset
|
435 |
self._remove_sources(ov, termssources[ov] - ssset) |
dd9bdcfc03b6
cleanup _handle_cross_relation
sylvain.thenault@logilab.fr
parents:
1228
diff
changeset
|
436 |
break |
dd9bdcfc03b6
cleanup _handle_cross_relation
sylvain.thenault@logilab.fr
parents:
1228
diff
changeset
|
437 |
else: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
438 |
self._sourcesterms.setdefault(source, {})[rel] = set(self._solindices) |
1785 | 439 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
440 |
def _remove_invalid_sources(self, termssources): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
441 |
"""removes invalid sources from `sourcesterms` member according to |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
442 |
traversed relations and their properties (which sources support them, |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
443 |
can they cross sources, etc...) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
444 |
""" |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
445 |
for term in self._linkedterms: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
446 |
self._remove_sources_until_stable(term, termssources) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
447 |
if len(self._sourcesterms) > 1 and hasattr(self.plan.rqlst, 'main_relations'): |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
448 |
# the querier doesn't annotate write queries, need to do it here |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
449 |
self.plan.annotate_rqlst() |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
450 |
# insert/update/delete queries, we may get extra information from |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
451 |
# the main relation (eg relations to the left of the WHERE |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
452 |
if self.plan.rqlst.TYPE == 'insert': |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
453 |
inserted = dict((vref.variable, etype) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
454 |
for etype, vref in self.plan.rqlst.main_variables) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
455 |
else: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
456 |
inserted = {} |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
457 |
repo = self._repo |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
458 |
rschema = self._schema.rschema |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
459 |
for rel in self.plan.rqlst.main_relations: |
3689
deb13e88e037
follow yams 0.25 api changes to improve performance
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3624
diff
changeset
|
460 |
if not rschema(rel.r_type).final: |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
461 |
# nothing to do if relation is not supported by multiple sources |
393 | 462 |
if len(repo.rel_type_sources(rel.r_type)) < 2: |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
463 |
continue |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
464 |
lhs, rhs = rel.get_variable_parts() |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
465 |
try: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
466 |
lhsv = self._extern_term(lhs, termssources, inserted) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
467 |
rhsv = self._extern_term(rhs, termssources, inserted) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
468 |
except KeyError, ex: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
469 |
continue |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
470 |
self._remove_term_sources(lhsv, rel, rhsv, termssources) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
471 |
self._remove_term_sources(rhsv, rel, lhsv, termssources) |
1785 | 472 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
473 |
def _extern_term(self, term, termssources, inserted): |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
474 |
var = term.variable |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
475 |
if var.stinfo['constnode']: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
476 |
termv = var.stinfo['constnode'] |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
477 |
termssources[termv] = self._term_sources(termv) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
478 |
elif var in inserted: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
479 |
termv = var |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
480 |
source = self._repo.locate_etype_source(inserted[var]) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
481 |
termssources[termv] = set((source, solindex) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
482 |
for solindex in self._solindices) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
483 |
else: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
484 |
termv = self.rqlst.defined_vars[var.name] |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
485 |
if not termv in termssources: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
486 |
termssources[termv] = self._term_sources(termv) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
487 |
return termv |
1785 | 488 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
489 |
def _remove_sources_until_stable(self, term, termssources): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
490 |
sourcesterms = self._sourcesterms |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
491 |
for oterm, rel in self._linkedterms.get(term, ()): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
492 |
if not term.scope is oterm.scope and rel.scope.neged(strict=True): |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
493 |
# can't get information from relation inside a NOT exists |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
494 |
# where terms don't belong to the same scope |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
495 |
continue |
1120
1c24cde4bf72
fix ms planning w/ relation constraint propagation
sylvain.thenault@logilab.fr
parents:
442
diff
changeset
|
496 |
need_ancestor_scope = False |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
497 |
if not (term.scope is rel.scope and oterm.scope is rel.scope): |
1120
1c24cde4bf72
fix ms planning w/ relation constraint propagation
sylvain.thenault@logilab.fr
parents:
442
diff
changeset
|
498 |
if rel.ored(): |
1c24cde4bf72
fix ms planning w/ relation constraint propagation
sylvain.thenault@logilab.fr
parents:
442
diff
changeset
|
499 |
continue |
1c24cde4bf72
fix ms planning w/ relation constraint propagation
sylvain.thenault@logilab.fr
parents:
442
diff
changeset
|
500 |
if rel.ored(traverse_scope=True): |
1c24cde4bf72
fix ms planning w/ relation constraint propagation
sylvain.thenault@logilab.fr
parents:
442
diff
changeset
|
501 |
# if relation has some OR as parent, constraints should only |
1c24cde4bf72
fix ms planning w/ relation constraint propagation
sylvain.thenault@logilab.fr
parents:
442
diff
changeset
|
502 |
# propagate from parent scope to child scope, nothing else |
1c24cde4bf72
fix ms planning w/ relation constraint propagation
sylvain.thenault@logilab.fr
parents:
442
diff
changeset
|
503 |
need_ancestor_scope = True |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
504 |
relsources = self._repo.rel_type_sources(rel.r_type) |
341
0a426be2f3a2
fixes for some cases of neged inlined relations, may have to use (new) IntersectStep. XXX: write IntersectFetchStep
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
257
diff
changeset
|
505 |
if rel.neged(strict=True) and ( |
0a426be2f3a2
fixes for some cases of neged inlined relations, may have to use (new) IntersectStep. XXX: write IntersectFetchStep
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
257
diff
changeset
|
506 |
len(relsources) < 2 |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
507 |
or not isinstance(oterm, Variable) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
508 |
or oterm.valuable_references() != 1 |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
509 |
or any(sourcesterms[source][term] != sourcesterms[source][oterm] |
341
0a426be2f3a2
fixes for some cases of neged inlined relations, may have to use (new) IntersectStep. XXX: write IntersectFetchStep
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
257
diff
changeset
|
510 |
for source in relsources |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
511 |
if term in sourcesterms.get(source, ()) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
512 |
and oterm in sourcesterms.get(source, ()))): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
513 |
# neged relation doesn't allow to infer term sources unless |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
514 |
# we're on a multisource relation for a term only used by this |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
515 |
# relation (eg "Any X WHERE NOT X multisource_rel Y" and over is |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
516 |
# Y) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
517 |
continue |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
518 |
# compute invalid sources for terms and remove them |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
519 |
if not need_ancestor_scope or is_ancestor(term.scope, oterm.scope): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
520 |
self._remove_term_sources(term, rel, oterm, termssources) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
521 |
if not need_ancestor_scope or is_ancestor(oterm.scope, term.scope): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
522 |
self._remove_term_sources(oterm, rel, term, termssources) |
1785 | 523 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
524 |
def _remove_term_sources(self, term, rel, oterm, termssources): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
525 |
"""remove invalid sources for term according to oterm's sources and the |
1785 | 526 |
relation between those two terms. |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
527 |
""" |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
528 |
norelsup = self._norel_support_set(rel) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
529 |
termsources = termssources[term] |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
530 |
invalid_sources = termsources - (termssources[oterm] | norelsup) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
531 |
if invalid_sources and self._repo.can_cross_relation(rel.r_type): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
532 |
invalid_sources -= self._sys_source_set |
1230
232e16835fff
fix constant handling in sourcesterms + minor cleanup
sylvain.thenault@logilab.fr
parents:
1229
diff
changeset
|
533 |
if invalid_sources and isinstance(term, Variable) \ |
232e16835fff
fix constant handling in sourcesterms + minor cleanup
sylvain.thenault@logilab.fr
parents:
1229
diff
changeset
|
534 |
and self._need_ext_source_access(term, rel): |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
535 |
# if the term is a not invariant variable, we should filter out |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
536 |
# source where the relation is a cross relation from invalid |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
537 |
# sources |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
538 |
invalid_sources = frozenset((s, solidx) for s, solidx in invalid_sources |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
539 |
if not (s in self._crossrelations and |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
540 |
rel in self._crossrelations[s])) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
541 |
if invalid_sources: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
542 |
self._remove_sources(term, invalid_sources) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
543 |
termsources -= invalid_sources |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
544 |
self._remove_sources_until_stable(term, termssources) |
1785 | 545 |
if isinstance(oterm, Constant): |
546 |
self._remove_sources(oterm, invalid_sources) |
|
547 |
||
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
548 |
def _compute_needsplit(self): |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
549 |
"""tell according to sourcesterms if the rqlst has to be splitted for |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
550 |
execution among multiple sources |
1785 | 551 |
|
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
552 |
the execution has to be split if |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
553 |
* a source support an entity (non invariant) but doesn't support a |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
554 |
relation on it |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
555 |
* a source support an entity which is accessed by an optional relation |
1785 | 556 |
* there is more than one source and either all sources'supported |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
557 |
variable/solutions are not equivalent or multiple variables have to |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
558 |
be fetched from some source |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
559 |
""" |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
560 |
# NOTE: < 2 since may be 0 on queries such as Any X WHERE X eid 2 |
1785 | 561 |
if len(self._sourcesterms) < 2: |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
562 |
self.needsplit = False |
2165
dd40c375044e
#344177: ms planning bug with "Any X WHERE X eid 123" and 123 coming from an external source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2114
diff
changeset
|
563 |
# if this is not the system source but we have only constant terms |
dd40c375044e
#344177: ms planning bug with "Any X WHERE X eid 123" and 123 coming from an external source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2114
diff
changeset
|
564 |
# and no relation (other than eid), apply query on the system source |
dd40c375044e
#344177: ms planning bug with "Any X WHERE X eid 123" and 123 coming from an external source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2114
diff
changeset
|
565 |
# |
dd40c375044e
#344177: ms planning bug with "Any X WHERE X eid 123" and 123 coming from an external source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2114
diff
changeset
|
566 |
# testing for rqlst with nothing in vargraph nor defined_vars is the |
dd40c375044e
#344177: ms planning bug with "Any X WHERE X eid 123" and 123 coming from an external source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2114
diff
changeset
|
567 |
# simplest way the check the condition explained below |
dd40c375044e
#344177: ms planning bug with "Any X WHERE X eid 123" and 123 coming from an external source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2114
diff
changeset
|
568 |
if not self.system_source in self._sourcesterms and \ |
2688
afa99d53401c
[multi-source] test and fix msplanner regression
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2170
diff
changeset
|
569 |
not self.rqlst.defined_vars and \ |
afa99d53401c
[multi-source] test and fix msplanner regression
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2170
diff
changeset
|
570 |
not need_source_access_relation(self.rqlst.vargraph): |
2165
dd40c375044e
#344177: ms planning bug with "Any X WHERE X eid 123" and 123 coming from an external source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2114
diff
changeset
|
571 |
self._sourcesterms = {self.system_source: {}} |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
572 |
elif not self.needsplit: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
573 |
if not allequals(self._sourcesterms.itervalues()): |
2170
6adbc965c51b
fix jpl/ticket/34862: msplanner bug with query generated to check if a user from ldap is owned_by himself
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2165
diff
changeset
|
574 |
for source, terms in self._sourcesterms.iteritems(): |
6adbc965c51b
fix jpl/ticket/34862: msplanner bug with query generated to check if a user from ldap is owned_by himself
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2165
diff
changeset
|
575 |
if source is self.system_source: |
6adbc965c51b
fix jpl/ticket/34862: msplanner bug with query generated to check if a user from ldap is owned_by himself
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2165
diff
changeset
|
576 |
continue |
2114
862f27d94af4
one more ms planning bug
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
2110
diff
changeset
|
577 |
if any(x for x in terms if not isinstance(x, Constant)): |
862f27d94af4
one more ms planning bug
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
2110
diff
changeset
|
578 |
self.needsplit = True |
862f27d94af4
one more ms planning bug
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
2110
diff
changeset
|
579 |
return |
862f27d94af4
one more ms planning bug
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
2110
diff
changeset
|
580 |
self._sourcesterms = {self.system_source: {}} |
862f27d94af4
one more ms planning bug
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
2110
diff
changeset
|
581 |
self.needsplit = False |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
582 |
else: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
583 |
sample = self._sourcesterms.itervalues().next() |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
584 |
if len(sample) > 1: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
585 |
for term in sample: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
586 |
# need split if unlinked variable |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
587 |
if isinstance(term, Variable) and not term in self._linkedterms: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
588 |
self.needsplit = True |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
589 |
break |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
590 |
else: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
591 |
# need split if there are some cross relation on non |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
592 |
# invariant variable or if the variable is used in |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
593 |
# multi-sources relation |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
594 |
if self._crossrelations: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
595 |
for reldict in self._crossrelations.itervalues(): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
596 |
for rel, terms in reldict.iteritems(): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
597 |
for term in terms: |
1230
232e16835fff
fix constant handling in sourcesterms + minor cleanup
sylvain.thenault@logilab.fr
parents:
1229
diff
changeset
|
598 |
if isinstance(term, Variable) \ |
232e16835fff
fix constant handling in sourcesterms + minor cleanup
sylvain.thenault@logilab.fr
parents:
1229
diff
changeset
|
599 |
and self._need_ext_source_access(term, rel): |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
600 |
self.needsplit = True |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
601 |
return |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
602 |
|
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
603 |
@cached |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
604 |
def _need_ext_source_access(self, var, rel): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
605 |
if not var._q_invariant: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
606 |
return True |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
607 |
if any(r for x, r in self._linkedterms[var] |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
608 |
if not r is rel and self._repo.is_multi_sources_relation(r.r_type)): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
609 |
return True |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
610 |
return False |
1785 | 611 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
612 |
def _set_source_for_term(self, source, term): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
613 |
self._sourcesterms.setdefault(source, {})[term] = set(self._solindices) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
614 |
|
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
615 |
def _term_sources(self, term): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
616 |
"""returns possible sources for terms `term`""" |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
617 |
if isinstance(term, Constant): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
618 |
source = self._session.source_from_eid(term.eval(self.plan.args)) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
619 |
return set((source, solindex) for solindex in self._solindices) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
620 |
else: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
621 |
var = getattr(term, 'variable', term) |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
622 |
sources = [source for source, varobjs in self.sourcesterms.iteritems() |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
623 |
if var in varobjs] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
624 |
return set((source, solindex) for source in sources |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
625 |
for solindex in self.sourcesterms[source][var]) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
626 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
627 |
def _remove_sources(self, term, sources): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
628 |
"""removes invalid sources (`sources`) from `sourcesterms` |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
629 |
|
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
630 |
:param sources: the list of sources to remove |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
631 |
:param term: the analyzed term |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
632 |
""" |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
633 |
sourcesterms = self._sourcesterms |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
634 |
for source, solindex in sources: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
635 |
try: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
636 |
sourcesterms[source][term].remove(solindex) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
637 |
except KeyError: |
1786
eccd1885d42e
one more msplanning bug fixed...
sylvain.thenault@logilab.fr
parents:
1785
diff
changeset
|
638 |
import rql.base as rqlb |
3624
05932c6f7db2
fix buggy assertion
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3239
diff
changeset
|
639 |
assert isinstance(term, (rqlb.BaseNode, Variable)), repr(term) |
1785 | 640 |
continue # may occur with subquery column alias |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
641 |
if not sourcesterms[source][term]: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
642 |
del sourcesterms[source][term] |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
643 |
if not sourcesterms[source]: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
644 |
del sourcesterms[source] |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
645 |
|
393 | 646 |
def crossed_relation(self, source, relation): |
647 |
return relation in self._crossrelations.get(source, ()) |
|
1785 | 648 |
|
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
649 |
def part_steps(self): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
650 |
"""precompute necessary part steps before generating actual rql for |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
651 |
each step. This is necessary to know if an aggregate step will be |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
652 |
necessary or not. |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
653 |
""" |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
654 |
steps = [] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
655 |
select = self.rqlst |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
656 |
rschema = self._schema.rschema |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
657 |
for source in self.part_sources: |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
658 |
try: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
659 |
sourceterms = self._sourcesterms[source] |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
660 |
except KeyError: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
661 |
continue # already proceed |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
662 |
while sourceterms: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
663 |
# take a term randomly, and all terms supporting the |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
664 |
# same solutions |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
665 |
term, solindices = self._choose_term(sourceterms) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
666 |
if source.uri == 'system': |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
667 |
# ensure all variables are available for the latest step |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
668 |
# (missing one will be available from temporary tables |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
669 |
# of previous steps) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
670 |
scope = select |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
671 |
terms = scope.defined_vars.values() + scope.aliases.values() |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
672 |
sourceterms.clear() |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
673 |
sources = [source] |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
674 |
else: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
675 |
scope = term.scope |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
676 |
# find which sources support the same term and solutions |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
677 |
sources = self._expand_sources(source, term, solindices) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
678 |
# no try to get as much terms as possible |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
679 |
terms = self._expand_terms(term, sources, sourceterms, |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
680 |
scope, solindices) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
681 |
if len(terms) == 1 and isinstance(terms[0], Constant): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
682 |
# we can't generate anything interesting with a single |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
683 |
# constant term (will generate an empty "Any" query), |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
684 |
# go to the next iteration directly! |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
685 |
continue |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
686 |
if not sourceterms: |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
687 |
try: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
688 |
del self._sourcesterms[source] |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
689 |
except KeyError: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
690 |
# XXX already cleaned |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
691 |
pass |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
692 |
# set of terms which should be additionaly selected when |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
693 |
# possible |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
694 |
needsel = set() |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
695 |
if not self._sourcesterms: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
696 |
terms += scope.defined_vars.values() + scope.aliases.values() |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
697 |
final = True |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
698 |
else: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
699 |
# suppose this is a final step until the contrary is proven |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
700 |
final = scope is select |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
701 |
# add attribute variables and mark variables which should be |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
702 |
# additionaly selected when possible |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
703 |
for var in select.defined_vars.itervalues(): |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
704 |
if not var in terms: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
705 |
stinfo = var.stinfo |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
706 |
for ovar, rtype in stinfo['attrvars']: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
707 |
if ovar in terms: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
708 |
needsel.add(var.name) |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
709 |
terms.append(var) |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
710 |
break |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
711 |
else: |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
712 |
needsel.add(var.name) |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
713 |
final = False |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
714 |
# check where all relations are supported by the sources |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
715 |
for rel in scope.iget_nodes(Relation): |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
716 |
if rel.is_types_restriction(): |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
717 |
continue |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
718 |
# take care not overwriting the existing "source" identifier |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
719 |
for _source in sources: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
720 |
if not _source.support_relation(rel.r_type) or ( |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
721 |
self.crossed_relation(_source, rel) and not rel in terms): |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
722 |
for vref in rel.iget_nodes(VariableRef): |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
723 |
needsel.add(vref.name) |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
724 |
final = False |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
725 |
break |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
726 |
else: |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
727 |
if not scope is select: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
728 |
self._exists_relation(rel, terms, needsel) |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
729 |
# if relation is supported by all sources and some of |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
730 |
# its lhs/rhs variable isn't in "terms", and the |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
731 |
# other end *is* in "terms", mark it have to be |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
732 |
# selected |
3689
deb13e88e037
follow yams 0.25 api changes to improve performance
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3624
diff
changeset
|
733 |
if source.uri != 'system' and not rschema(rel.r_type).final: |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
734 |
lhs, rhs = rel.get_variable_parts() |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
735 |
try: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
736 |
lhsvar = lhs.variable |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
737 |
except AttributeError: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
738 |
lhsvar = lhs |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
739 |
try: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
740 |
rhsvar = rhs.variable |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
741 |
except AttributeError: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
742 |
rhsvar = rhs |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
743 |
if lhsvar in terms and not rhsvar in terms: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
744 |
needsel.add(lhsvar.name) |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
745 |
elif rhsvar in terms and not lhsvar in terms: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
746 |
needsel.add(rhsvar.name) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
747 |
if final and source.uri != 'system': |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
748 |
# check rewritten constants |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
749 |
for vconsts in select.stinfo['rewritten'].itervalues(): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
750 |
const = vconsts[0] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
751 |
eid = const.eval(self.plan.args) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
752 |
_source = self._session.source_from_eid(eid) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
753 |
if len(sources) > 1 or not _source in sources: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
754 |
# if there is some rewriten constant used by a not |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
755 |
# neged relation while there are some source not |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
756 |
# supporting the associated entity, this step can't |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
757 |
# be final (unless the relation is explicitly in |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
758 |
# `terms`, eg cross relations) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
759 |
for c in vconsts: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
760 |
rel = c.relation() |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
761 |
if rel is None or not (rel in terms or rel.neged(strict=True)): |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
762 |
final = False |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
763 |
break |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
764 |
break |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
765 |
if final: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
766 |
self._cleanup_sourcesterms(sources, solindices) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
767 |
steps.append((sources, terms, solindices, scope, needsel, final) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
768 |
) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
769 |
return steps |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
770 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
771 |
def _exists_relation(self, rel, terms, needsel): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
772 |
rschema = self._schema.rschema(rel.r_type) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
773 |
lhs, rhs = rel.get_variable_parts() |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
774 |
try: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
775 |
lhsvar, rhsvar = lhs.variable, rhs.variable |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
776 |
except AttributeError: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
777 |
pass |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
778 |
else: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
779 |
# supported relation with at least one end supported, check the |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
780 |
# other end is in as well. If not this usually means the |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
781 |
# variable is refed by an outer scope and should be substituted |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
782 |
# using an 'identity' relation (else we'll get a conflict of |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
783 |
# temporary tables) |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
784 |
if rhsvar in terms and not lhsvar in terms: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
785 |
self._identity_substitute(rel, lhsvar, terms, needsel) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
786 |
elif lhsvar in terms and not rhsvar in terms: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
787 |
self._identity_substitute(rel, rhsvar, terms, needsel) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
788 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
789 |
def _identity_substitute(self, relation, var, terms, needsel): |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
790 |
newvar = self._insert_identity_variable(relation.scope, var) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
791 |
if newvar is not None: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
792 |
# ensure relation is using '=' operator, else we rely on a |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
793 |
# sqlgenerator side effect (it won't insert an inequality operator |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
794 |
# in this case) |
1785 | 795 |
relation.children[1].operator = '=' |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
796 |
terms.append(newvar) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
797 |
needsel.add(newvar.name) |
1785 | 798 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
799 |
def _choose_term(self, sourceterms): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
800 |
"""pick one term among terms supported by a source, which will be used |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
801 |
as a base to generate an execution step |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
802 |
""" |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
803 |
secondchoice = None |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
804 |
if len(self._sourcesterms) > 1: |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
805 |
# priority to variable from subscopes |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
806 |
for term in sourceterms: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
807 |
if not term.scope is self.rqlst: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
808 |
if isinstance(term, Variable): |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
809 |
return term, sourceterms.pop(term) |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
810 |
secondchoice = term |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
811 |
else: |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
812 |
# priority to variable from outer scope |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
813 |
for term in sourceterms: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
814 |
if term.scope is self.rqlst: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
815 |
if isinstance(term, Variable): |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
816 |
return term, sourceterms.pop(term) |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
817 |
secondchoice = term |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
818 |
if secondchoice is not None: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
819 |
return secondchoice, sourceterms.pop(secondchoice) |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
820 |
# priority to variable with the less solutions supported and with the |
1237 | 821 |
# most valuable refs. Add variable name for test predictability |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
822 |
variables = sorted([(var, sols) for (var, sols) in sourceterms.items() |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
823 |
if isinstance(var, Variable)], |
1237 | 824 |
key=lambda (v, s): (len(s), -v.valuable_references(), v.name)) |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
825 |
if variables: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
826 |
var = variables[0][0] |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
827 |
return var, sourceterms.pop(var) |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
828 |
# priority to constant |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
829 |
for term in sourceterms: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
830 |
if isinstance(term, Constant): |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
831 |
return term, sourceterms.pop(term) |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
832 |
# whatever (relation) |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
833 |
term = iter(sourceterms).next() |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
834 |
return term, sourceterms.pop(term) |
1785 | 835 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
836 |
def _expand_sources(self, selected_source, term, solindices): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
837 |
"""return all sources supporting given term / solindices""" |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
838 |
sources = [selected_source] |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
839 |
sourcesterms = self._sourcesterms |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
840 |
for source in sourcesterms.keys(): |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
841 |
if source is selected_source: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
842 |
continue |
1785 | 843 |
if not (term in sourcesterms[source] and |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
844 |
solindices.issubset(sourcesterms[source][term])): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
845 |
continue |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
846 |
sources.append(source) |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
847 |
if source.uri != 'system' or not (isinstance(term, Variable) and not term in self._linkedterms): |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
848 |
termsolindices = sourcesterms[source][term] |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
849 |
termsolindices -= solindices |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
850 |
if not termsolindices: |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
851 |
del sourcesterms[source][term] |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
852 |
if not sourcesterms[source]: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
853 |
del sourcesterms[source] |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
854 |
return sources |
1785 | 855 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
856 |
def _expand_terms(self, term, sources, sourceterms, scope, solindices): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
857 |
terms = [term] |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
858 |
sources = sorted(sources) |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
859 |
sourcesterms = self._sourcesterms |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
860 |
nbunlinked = 1 |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
861 |
linkedterms = self._linkedterms |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
862 |
# term has to belong to the same scope if there is more |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
863 |
# than the system source remaining |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
864 |
if len(sourcesterms) > 1 and not scope is self.rqlst: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
865 |
candidates = (t for t in sourceterms.keys() if scope is t.scope) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
866 |
else: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
867 |
candidates = sourceterms #.iterkeys() |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
868 |
# we only want one unlinked term in each generated query |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
869 |
candidates = [t for t in candidates |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
870 |
if isinstance(t, (Constant, Relation)) or |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
871 |
(solindices.issubset(sourceterms[t]) and t in linkedterms)] |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
872 |
cross_rels = {} |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
873 |
for source in sources: |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
874 |
cross_rels.update(self._crossrelations.get(source, {})) |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
875 |
exclude = {} |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
876 |
for rel, crossvars in cross_rels.iteritems(): |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
877 |
vars = [t for t in crossvars if isinstance(t, Variable)] |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
878 |
try: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
879 |
exclude[vars[0]] = vars[1] |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
880 |
exclude[vars[1]] = vars[0] |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
881 |
except IndexError: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
882 |
pass |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
883 |
accept_term = lambda x: (not any(s for s in sources if not x in sourcesterms.get(s, ())) |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
884 |
and any(t for t in terms if t in linkedterms.get(x, ())) |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
885 |
and not exclude.get(x) in terms) |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
886 |
if isinstance(term, Relation) and term in cross_rels: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
887 |
cross_terms = cross_rels.pop(term) |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
888 |
base_accept_term = accept_term |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
889 |
accept_term = lambda x: (base_accept_term(x) or x in cross_terms) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
890 |
for refed in cross_terms: |
393 | 891 |
if not refed in candidates: |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
892 |
terms.append(refed) |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
893 |
# repeat until no term can't be added, since addition of a new |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
894 |
# term may permit to another one to be added |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
895 |
modified = True |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
896 |
while modified and candidates: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
897 |
modified = False |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
898 |
for term in candidates[:]: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
899 |
if isinstance(term, Constant): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
900 |
relation = term.relation() |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
901 |
if sorted(set(x[0] for x in self._term_sources(term))) != sources: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
902 |
continue |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
903 |
terms.append(term) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
904 |
candidates.remove(term) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
905 |
modified = True |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
906 |
del sourceterms[term] |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
907 |
elif accept_term(term): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
908 |
terms.append(term) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
909 |
candidates.remove(term) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
910 |
modified = True |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
911 |
self._cleanup_sourcesterms(sources, solindices, term) |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
912 |
return terms |
1785 | 913 |
|
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
914 |
def _cleanup_sourcesterms(self, sources, solindices, term=None): |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
915 |
"""remove solutions so we know they are already processed""" |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
916 |
for source in sources: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
917 |
try: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
918 |
sourceterms = self._sourcesterms[source] |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
919 |
except KeyError: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
920 |
continue |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
921 |
if term is None: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
922 |
for term, termsolindices in sourceterms.items(): |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
923 |
if isinstance(term, Relation) and self.crossed_relation(source, term): |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
924 |
continue |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
925 |
termsolindices -= solindices |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
926 |
if not termsolindices: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
927 |
del sourceterms[term] |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
928 |
else: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
929 |
try: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
930 |
sourceterms[term] -= solindices |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
931 |
if not sourceterms[term]: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
932 |
del sourceterms[term] |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
933 |
except KeyError: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
934 |
pass |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
935 |
#assert term in cross_terms |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
936 |
if not sourceterms: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
937 |
del self._sourcesterms[source] |
1785 | 938 |
|
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
939 |
def merge_input_maps(self, allsolindices): |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
940 |
"""inputmaps is a dictionary with tuple of solution indices as key with |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
941 |
an associated input map as value. This function compute for each |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
942 |
solution its necessary input map and return them grouped |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
943 |
|
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
944 |
ex: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
945 |
inputmaps = {(0, 1, 2): {'A': 't1.login1', 'U': 't1.C0', 'U.login': 't1.login1'}, |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
946 |
(1,): {'X': 't2.C0', 'T': 't2.C1'}} |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
947 |
return : [([1], {'A': 't1.login1', 'U': 't1.C0', 'U.login': 't1.login1', |
1785 | 948 |
'X': 't2.C0', 'T': 't2.C1'}), |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
949 |
([0,2], {'A': 't1.login1', 'U': 't1.C0', 'U.login': 't1.login1'})] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
950 |
""" |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
951 |
if not self._inputmaps: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
952 |
return [(allsolindices, None)] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
953 |
mapbysol = {} |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
954 |
# compute a single map for each solution |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
955 |
for solindices, basemap in self._inputmaps.iteritems(): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
956 |
for solindex in solindices: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
957 |
solmap = mapbysol.setdefault(solindex, {}) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
958 |
solmap.update(basemap) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
959 |
try: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
960 |
allsolindices.remove(solindex) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
961 |
except KeyError: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
962 |
continue # already removed |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
963 |
# group results by identical input map |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
964 |
result = [] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
965 |
for solindex, solmap in mapbysol.iteritems(): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
966 |
for solindices, commonmap in result: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
967 |
if commonmap == solmap: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
968 |
solindices.append(solindex) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
969 |
break |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
970 |
else: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
971 |
result.append( ([solindex], solmap) ) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
972 |
if allsolindices: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
973 |
result.append( (list(allsolindices), None) ) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
974 |
return result |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
975 |
|
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
976 |
def build_final_part(self, select, solindices, inputmap, sources, |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
977 |
insertedvars): |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
978 |
solutions = [self._solutions[i] for i in solindices] |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
979 |
if self._conflicts: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
980 |
for varname, mappedto in self._conflicts: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
981 |
var = select.defined_vars[varname] |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
982 |
newvar = select.make_variable() |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
983 |
# XXX should use var.scope but scope hasn't been computed yet |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
984 |
select.add_relation(var, 'identity', newvar) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
985 |
for sol in solutions: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
986 |
sol[newvar.name] = sol[varname] |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
987 |
inputmap[newvar.name] = mappedto |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
988 |
rqlst = self.plan.finalize(select, solutions, insertedvars) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
989 |
if self.temptable is None and self.finaltable is None: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
990 |
return OneFetchStep(self.plan, rqlst, sources, inputmap=inputmap) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
991 |
table = self.temptable or self.finaltable |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
992 |
return FetchStep(self.plan, rqlst, sources, table, True, inputmap) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
993 |
|
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
994 |
def build_non_final_part(self, select, solindices, sources, insertedvars, |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
995 |
table): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
996 |
"""non final step, will have to store results in a temporary table""" |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
997 |
solutions = [self._solutions[i] for i in solindices] |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
998 |
rqlst = self.plan.finalize(select, solutions, insertedvars) |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
999 |
step = FetchStep(self.plan, rqlst, sources, table, False) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1000 |
# update input map for following steps, according to processed solutions |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1001 |
inputmapkey = tuple(sorted(solindices)) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1002 |
inputmap = self._inputmaps.setdefault(inputmapkey, {}) |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1003 |
for varname, mapping in step.outputmap.iteritems(): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1004 |
if varname in inputmap and \ |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1005 |
not (mapping == inputmap[varname] or |
3689
deb13e88e037
follow yams 0.25 api changes to improve performance
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3624
diff
changeset
|
1006 |
self._schema.eschema(solutions[0][varname]).final): |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1007 |
self._conflicts.append((varname, inputmap[varname])) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1008 |
inputmap.update(step.outputmap) |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1009 |
self.plan.add_step(step) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1010 |
|
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1011 |
|
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1012 |
class MSPlanner(SSPlanner): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1013 |
"""MultiSourcesPlanner: build execution plan for rql queries |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1014 |
|
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1015 |
decompose the RQL query according to sources'schema |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1016 |
""" |
1785 | 1017 |
|
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1018 |
def build_select_plan(self, plan, rqlst): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1019 |
"""build execution plan for a SELECT RQL query |
1785 | 1020 |
|
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1021 |
the rqlst should not be tagged at this point |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1022 |
""" |
2766
07e47b84acc2
DBG_MS flag
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2688
diff
changeset
|
1023 |
if server.DEBUG & server.DBG_MS: |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1024 |
print '-'*80 |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1025 |
print 'PLANNING', rqlst |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1026 |
for select in rqlst.children: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1027 |
if len(select.solutions) > 1: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1028 |
hasmultiplesols = True |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1029 |
break |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1030 |
else: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1031 |
hasmultiplesols = False |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1032 |
# preprocess deals with security insertion and returns a new syntax tree |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1033 |
# which have to be executed to fulfill the query: according |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1034 |
# to permissions for variable's type, different rql queries may have to |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1035 |
# be executed |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1036 |
plan.preprocess(rqlst) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1037 |
ppis = [PartPlanInformation(plan, select, self.rqlhelper) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1038 |
for select in rqlst.children] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1039 |
steps = self._union_plan(plan, rqlst, ppis) |
2766
07e47b84acc2
DBG_MS flag
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2688
diff
changeset
|
1040 |
if server.DEBUG & server.DBG_MS: |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1041 |
from pprint import pprint |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1042 |
for step in plan.steps: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1043 |
pprint(step.test_repr()) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1044 |
pprint(steps[0].test_repr()) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1045 |
return steps |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1046 |
|
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1047 |
def _ppi_subqueries(self, ppi): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1048 |
# part plan info for subqueries |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1049 |
plan = ppi.plan |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1050 |
inputmap = {} |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1051 |
for subquery in ppi.rqlst.with_[:]: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1052 |
sppis = [PartPlanInformation(plan, select) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1053 |
for select in subquery.query.children] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1054 |
for sppi in sppis: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1055 |
if sppi.needsplit or sppi.part_sources != ppi.part_sources: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1056 |
temptable = 'T%s' % make_uid(id(subquery)) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1057 |
sstep = self._union_plan(plan, subquery.query, sppis, temptable)[0] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1058 |
break |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1059 |
else: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1060 |
sstep = None |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1061 |
if sstep is not None: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1062 |
ppi.rqlst.with_.remove(subquery) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1063 |
for i, colalias in enumerate(subquery.aliases): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1064 |
inputmap[colalias.name] = '%s.C%s' % (temptable, i) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1065 |
ppi.plan.add_step(sstep) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1066 |
return inputmap |
1785 | 1067 |
|
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1068 |
def _union_plan(self, plan, union, ppis, temptable=None): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1069 |
tosplit, cango, allsources = [], {}, set() |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1070 |
for planinfo in ppis: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1071 |
if planinfo.needsplit: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1072 |
tosplit.append(planinfo) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1073 |
else: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1074 |
cango.setdefault(planinfo.part_sources, []).append(planinfo) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1075 |
for source in planinfo.part_sources: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1076 |
allsources.add(source) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1077 |
# first add steps for query parts which doesn't need to splitted |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1078 |
steps = [] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1079 |
for sources, cppis in cango.iteritems(): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1080 |
byinputmap = {} |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1081 |
for ppi in cppis: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1082 |
select = ppi.rqlst |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1083 |
if sources != (ppi.system_source,): |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1084 |
add_types_restriction(self.schema, select) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1085 |
# part plan info for subqueries |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1086 |
inputmap = self._ppi_subqueries(ppi) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1087 |
aggrstep = need_aggr_step(select, sources) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1088 |
if aggrstep: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1089 |
atemptable = 'T%s' % make_uid(id(select)) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1090 |
sunion = Union() |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1091 |
sunion.append(select) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1092 |
selected = select.selection[:] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1093 |
select_group_sort(select) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1094 |
step = AggrStep(plan, selected, select, atemptable, temptable) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1095 |
step.set_limit_offset(select.limit, select.offset) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1096 |
select.limit = None |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1097 |
select.offset = 0 |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1098 |
fstep = FetchStep(plan, sunion, sources, atemptable, True, inputmap) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1099 |
step.children.append(fstep) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1100 |
steps.append(step) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1101 |
else: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1102 |
byinputmap.setdefault(tuple(inputmap.iteritems()), []).append( (select) ) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1103 |
for inputmap, queries in byinputmap.iteritems(): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1104 |
inputmap = dict(inputmap) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1105 |
sunion = Union() |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1106 |
for select in queries: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1107 |
sunion.append(select) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1108 |
if temptable: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1109 |
steps.append(FetchStep(plan, sunion, sources, temptable, True, inputmap)) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1110 |
else: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1111 |
steps.append(OneFetchStep(plan, sunion, sources, inputmap)) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1112 |
# then add steps for splitted query parts |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1113 |
for planinfo in tosplit: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1114 |
steps.append(self.split_part(planinfo, temptable)) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1115 |
if len(steps) > 1: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1116 |
if temptable: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1117 |
step = UnionFetchStep(plan) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1118 |
else: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1119 |
step = UnionStep(plan) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1120 |
step.children = steps |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1121 |
return (step,) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1122 |
return steps |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1123 |
|
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1124 |
# internal methods for multisources decomposition ######################### |
1785 | 1125 |
|
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1126 |
def split_part(self, ppi, temptable): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1127 |
ppi.finaltable = temptable |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1128 |
plan = ppi.plan |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1129 |
select = ppi.rqlst |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1130 |
subinputmap = self._ppi_subqueries(ppi) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1131 |
stepdefs = ppi.part_steps() |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1132 |
if need_aggr_step(select, ppi.part_sources, stepdefs): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1133 |
atemptable = 'T%s' % make_uid(id(select)) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1134 |
selection = select.selection[:] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1135 |
select_group_sort(select) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1136 |
else: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1137 |
atemptable = None |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1138 |
selection = select.selection |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1139 |
ppi.temptable = atemptable |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1140 |
vfilter = TermsFiltererVisitor(self.schema, ppi) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1141 |
steps = [] |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1142 |
for sources, terms, solindices, scope, needsel, final in stepdefs: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1143 |
# extract an executable query using only the specified terms |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1144 |
if sources[0].uri == 'system': |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1145 |
# in this case we have to merge input maps before call to |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1146 |
# filter so already processed restriction are correctly |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1147 |
# removed |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1148 |
solsinputmaps = ppi.merge_input_maps(solindices) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1149 |
for solindices, inputmap in solsinputmaps: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1150 |
minrqlst, insertedvars = vfilter.filter( |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1151 |
sources, terms, scope, set(solindices), needsel, final) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1152 |
if inputmap is None: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1153 |
inputmap = subinputmap |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1154 |
else: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1155 |
inputmap.update(subinputmap) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1156 |
steps.append(ppi.build_final_part(minrqlst, solindices, inputmap, |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1157 |
sources, insertedvars)) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1158 |
else: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1159 |
# this is a final part (i.e. retreiving results for the |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1160 |
# original query part) if all term / sources have been |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1161 |
# treated or if this is the last shot for used solutions |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1162 |
minrqlst, insertedvars = vfilter.filter( |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1163 |
sources, terms, scope, solindices, needsel, final) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1164 |
if final: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1165 |
solsinputmaps = ppi.merge_input_maps(solindices) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1166 |
for solindices, inputmap in solsinputmaps: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1167 |
if inputmap is None: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1168 |
inputmap = subinputmap |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1169 |
else: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1170 |
inputmap.update(subinputmap) |
1231
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1171 |
if inputmap and len(sources) > 1: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1172 |
sources.remove(ppi.system_source) |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1173 |
steps.append(ppi.build_final_part(minrqlst, solindices, None, |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1174 |
sources, insertedvars)) |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1175 |
steps.append(ppi.build_final_part(minrqlst, solindices, inputmap, |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1176 |
[ppi.system_source], insertedvars)) |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1177 |
else: |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1178 |
steps.append(ppi.build_final_part(minrqlst, solindices, inputmap, |
1457a545af03
more fix of cross relation handling
sylvain.thenault@logilab.fr
parents:
1230
diff
changeset
|
1179 |
sources, insertedvars)) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1180 |
else: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1181 |
table = '_T%s%s' % (''.join(sorted(v._ms_table_key() for v in terms)), |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1182 |
''.join(sorted(str(i) for i in solindices))) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1183 |
ppi.build_non_final_part(minrqlst, solindices, sources, |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1184 |
insertedvars, table) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1185 |
# finally: join parts, deal with aggregat/group/sorts if necessary |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1186 |
if atemptable is not None: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1187 |
step = AggrStep(plan, selection, select, atemptable, temptable) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1188 |
step.children = steps |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1189 |
elif len(steps) > 1: |
442 | 1190 |
if select.need_intersect or any(select.need_intersect |
1191 |
for step in steps |
|
1192 |
for select in step.union.children): |
|
341
0a426be2f3a2
fixes for some cases of neged inlined relations, may have to use (new) IntersectStep. XXX: write IntersectFetchStep
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
257
diff
changeset
|
1193 |
if temptable: |
0a426be2f3a2
fixes for some cases of neged inlined relations, may have to use (new) IntersectStep. XXX: write IntersectFetchStep
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
257
diff
changeset
|
1194 |
step = IntersectFetchStep(plan) |
0a426be2f3a2
fixes for some cases of neged inlined relations, may have to use (new) IntersectStep. XXX: write IntersectFetchStep
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
257
diff
changeset
|
1195 |
else: |
0a426be2f3a2
fixes for some cases of neged inlined relations, may have to use (new) IntersectStep. XXX: write IntersectFetchStep
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
257
diff
changeset
|
1196 |
step = IntersectStep(plan) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1197 |
else: |
341
0a426be2f3a2
fixes for some cases of neged inlined relations, may have to use (new) IntersectStep. XXX: write IntersectFetchStep
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
257
diff
changeset
|
1198 |
if temptable: |
0a426be2f3a2
fixes for some cases of neged inlined relations, may have to use (new) IntersectStep. XXX: write IntersectFetchStep
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
257
diff
changeset
|
1199 |
step = UnionFetchStep(plan) |
0a426be2f3a2
fixes for some cases of neged inlined relations, may have to use (new) IntersectStep. XXX: write IntersectFetchStep
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
257
diff
changeset
|
1200 |
else: |
0a426be2f3a2
fixes for some cases of neged inlined relations, may have to use (new) IntersectStep. XXX: write IntersectFetchStep
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
257
diff
changeset
|
1201 |
step = UnionStep(plan) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1202 |
step.children = steps |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1203 |
else: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1204 |
step = steps[0] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1205 |
if select.limit is not None or select.offset: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1206 |
step.set_limit_offset(select.limit, select.offset) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1207 |
return step |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1208 |
|
1785 | 1209 |
|
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1210 |
class UnsupportedBranch(Exception): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1211 |
pass |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1212 |
|
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1213 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1214 |
class TermsFiltererVisitor(object): |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1215 |
def __init__(self, schema, ppi): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1216 |
self.schema = schema |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1217 |
self.ppi = ppi |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1218 |
self.skip = {} |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1219 |
self.hasaggrstep = self.ppi.temptable |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1220 |
self.extneedsel = frozenset(vref.name for sortterm in ppi.rqlst.orderby |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1221 |
for vref in sortterm.iget_nodes(VariableRef)) |
1785 | 1222 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1223 |
def _rqlst_accept(self, rqlst, node, newroot, terms, setfunc=None): |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1224 |
try: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1225 |
newrestr, node_ = node.accept(self, newroot, terms[:]) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1226 |
except UnsupportedBranch: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1227 |
return rqlst |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1228 |
if setfunc is not None and newrestr is not None: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1229 |
setfunc(newrestr) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1230 |
if not node_ is node: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1231 |
rqlst = node.parent |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1232 |
return rqlst |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1233 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1234 |
def filter(self, sources, terms, rqlst, solindices, needsel, final): |
2766
07e47b84acc2
DBG_MS flag
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2688
diff
changeset
|
1235 |
if server.DEBUG & server.DBG_MS: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1236 |
print 'filter', final and 'final' or '', sources, terms, rqlst, solindices, needsel |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1237 |
newroot = Select() |
393 | 1238 |
self.sources = sorted(sources) |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1239 |
self.terms = terms |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1240 |
self.solindices = solindices |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1241 |
self.final = final |
2109
f469368df530
when a relation isn't supported by a ms query parts, dereference introduced variable refs
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2108
diff
changeset
|
1242 |
self._pending_vrefs = [] |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1243 |
# terms which appear in unsupported branches |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1244 |
needsel |= self.extneedsel |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1245 |
self.needsel = needsel |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1246 |
# terms which appear in supported branches |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1247 |
self.mayneedsel = set() |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1248 |
# new inserted variables |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1249 |
self.insertedvars = [] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1250 |
# other structures (XXX document) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1251 |
self.mayneedvar, self.hasvar = {}, {} |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1252 |
self.use_only_defined = False |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1253 |
self.scopes = {rqlst: newroot} |
2075
933ccfce6e91
fix bug in multisource planner, w/ inlined relation
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1977
diff
changeset
|
1254 |
self.current_scope = rqlst |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1255 |
if rqlst.where: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1256 |
rqlst = self._rqlst_accept(rqlst, rqlst.where, newroot, terms, |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1257 |
newroot.set_where) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1258 |
if isinstance(rqlst, Select): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1259 |
self.use_only_defined = True |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1260 |
if rqlst.groupby: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1261 |
groupby = [] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1262 |
for node in rqlst.groupby: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1263 |
rqlst = self._rqlst_accept(rqlst, node, newroot, terms, |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1264 |
groupby.append) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1265 |
if groupby: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1266 |
newroot.set_groupby(groupby) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1267 |
if rqlst.having: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1268 |
having = [] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1269 |
for node in rqlst.having: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1270 |
rqlst = self._rqlst_accept(rqlst, node, newroot, terms, |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1271 |
having.append) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1272 |
if having: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1273 |
newroot.set_having(having) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1274 |
if final and rqlst.orderby and not self.hasaggrstep: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1275 |
orderby = [] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1276 |
for node in rqlst.orderby: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1277 |
rqlst = self._rqlst_accept(rqlst, node, newroot, terms, |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1278 |
orderby.append) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1279 |
if orderby: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1280 |
newroot.set_orderby(orderby) |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1281 |
self.process_selection(newroot, terms, rqlst) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1282 |
elif not newroot.where: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1283 |
# no restrictions have been copied, just select terms and add |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1284 |
# type restriction (done later by add_types_restriction) |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1285 |
for v in terms: |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1286 |
if not isinstance(v, Variable): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1287 |
continue |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1288 |
newroot.append_selected(VariableRef(newroot.get_variable(v.name))) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1289 |
solutions = self.ppi.copy_solutions(solindices) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1290 |
cleanup_solutions(newroot, solutions) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1291 |
newroot.set_possible_types(solutions) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1292 |
if final: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1293 |
if self.hasaggrstep: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1294 |
self.add_necessary_selection(newroot, self.mayneedsel & self.extneedsel) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1295 |
newroot.distinct = rqlst.distinct |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1296 |
else: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1297 |
self.add_necessary_selection(newroot, self.mayneedsel & self.needsel) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1298 |
# insert vars to fetch constant values when needed |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1299 |
for (varname, rschema), reldefs in self.mayneedvar.iteritems(): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1300 |
for rel, ored in reldefs: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1301 |
if not (varname, rschema) in self.hasvar: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1302 |
self.hasvar[(varname, rschema)] = None # just to avoid further insertion |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1303 |
cvar = newroot.make_variable() |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1304 |
for sol in newroot.solutions: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1305 |
sol[cvar.name] = rschema.objects(sol[varname])[0] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1306 |
# if the current restriction is not used in a OR branch, |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1307 |
# we can keep it, else we have to drop the constant |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1308 |
# restriction (or we may miss some results) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1309 |
if not ored: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1310 |
rel = rel.copy(newroot) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1311 |
newroot.add_restriction(rel) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1312 |
# add a relation to link the variable |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1313 |
newroot.remove_node(rel.children[1]) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1314 |
cmp = Comparison('=') |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1315 |
rel.append(cmp) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1316 |
cmp.append(VariableRef(cvar)) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1317 |
self.insertedvars.append((varname, rschema, cvar.name)) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1318 |
newroot.append_selected(VariableRef(newroot.get_variable(cvar.name))) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1319 |
# NOTE: even if the restriction is done by this query, we have |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1320 |
# to let it in the original rqlst so that it appears anyway in |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1321 |
# the "final" query, else we may change the meaning of the query |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1322 |
# if there are NOT somewhere : |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1323 |
# 'NOT X relation Y, Y name "toto"' means X WHERE X isn't related |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1324 |
# to Y whose name is toto while |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1325 |
# 'NOT X relation Y' means X WHERE X has no 'relation' (whatever Y) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1326 |
elif ored: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1327 |
newroot.remove_node(rel) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1328 |
add_types_restriction(self.schema, rqlst, newroot, solutions) |
2766
07e47b84acc2
DBG_MS flag
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2688
diff
changeset
|
1329 |
if server.DEBUG & server.DBG_MS: |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1330 |
print '--->', newroot |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1331 |
return newroot, self.insertedvars |
1785 | 1332 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1333 |
def visit_and(self, node, newroot, terms): |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1334 |
subparts = [] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1335 |
for i in xrange(len(node.children)): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1336 |
child = node.children[i] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1337 |
try: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1338 |
newchild, child_ = child.accept(self, newroot, terms) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1339 |
if not child_ is child: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1340 |
node = child_.parent |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1341 |
if newchild is None: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1342 |
continue |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1343 |
subparts.append(newchild) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1344 |
except UnsupportedBranch: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1345 |
continue |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1346 |
if not subparts: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1347 |
return None, node |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1348 |
if len(subparts) == 1: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1349 |
return subparts[0], node |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1350 |
return copy_node(newroot, node, subparts), node |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1351 |
|
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1352 |
visit_or = visit_and |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1353 |
|
393 | 1354 |
def _relation_supported(self, relation): |
1355 |
rtype = relation.r_type |
|
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1356 |
for source in self.sources: |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1357 |
if not source.support_relation(rtype) or ( |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1358 |
rtype in source.cross_relations and not relation in self.terms): |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1359 |
return False |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1360 |
if not self.final and not relation in self.terms: |
393 | 1361 |
rschema = self.schema.rschema(relation.r_type) |
3689
deb13e88e037
follow yams 0.25 api changes to improve performance
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3624
diff
changeset
|
1362 |
if not rschema.final: |
393 | 1363 |
for term in relation.get_nodes((VariableRef, Constant)): |
1364 |
term = getattr(term, 'variable', term) |
|
1365 |
termsources = sorted(set(x[0] for x in self.ppi._term_sources(term))) |
|
1366 |
if termsources and termsources != self.sources: |
|
1367 |
return False |
|
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1368 |
return True |
1785 | 1369 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1370 |
def visit_relation(self, node, newroot, terms): |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1371 |
if not node.is_types_restriction(): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1372 |
if node in self.skip and self.solindices.issubset(self.skip[node]): |
3689
deb13e88e037
follow yams 0.25 api changes to improve performance
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3624
diff
changeset
|
1373 |
if not self.schema.rschema(node.r_type).final: |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1374 |
# can't really skip the relation if one variable is selected and only |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1375 |
# referenced by this relation |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1376 |
for vref in node.iget_nodes(VariableRef): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1377 |
stinfo = vref.variable.stinfo |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1378 |
if stinfo['selected'] and len(stinfo['relations']) == 1: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1379 |
break |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1380 |
else: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1381 |
return None, node |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1382 |
else: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1383 |
return None, node |
393 | 1384 |
if not self._relation_supported(node): |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1385 |
raise UnsupportedBranch() |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1386 |
# don't copy type restriction unless this is the only relation for the |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1387 |
# rhs variable, else they'll be reinserted later as needed (else we may |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1388 |
# copy a type restriction while the variable is not actually used) |
393 | 1389 |
elif not any(self._relation_supported(rel) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1390 |
for rel in node.children[0].variable.stinfo['relations']): |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1391 |
rel, node = self.visit_default(node, newroot, terms) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1392 |
return rel, node |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1393 |
else: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1394 |
raise UnsupportedBranch() |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1395 |
rschema = self.schema.rschema(node.r_type) |
2109
f469368df530
when a relation isn't supported by a ms query parts, dereference introduced variable refs
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2108
diff
changeset
|
1396 |
self._pending_vrefs = [] |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1397 |
try: |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1398 |
res = self.visit_default(node, newroot, terms)[0] |
2109
f469368df530
when a relation isn't supported by a ms query parts, dereference introduced variable refs
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2108
diff
changeset
|
1399 |
except: |
2110
d7773fb1b892
add comment explaining the pb
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2109
diff
changeset
|
1400 |
# when a relation isn't supported, we should dereference potentially |
d7773fb1b892
add comment explaining the pb
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2109
diff
changeset
|
1401 |
# introduced variable refs |
2109
f469368df530
when a relation isn't supported by a ms query parts, dereference introduced variable refs
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2108
diff
changeset
|
1402 |
for vref in self._pending_vrefs: |
f469368df530
when a relation isn't supported by a ms query parts, dereference introduced variable refs
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2108
diff
changeset
|
1403 |
vref.unregister_reference() |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1404 |
raise |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1405 |
ored = node.ored() |
3689
deb13e88e037
follow yams 0.25 api changes to improve performance
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3624
diff
changeset
|
1406 |
if rschema.final or rschema.inlined: |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1407 |
vrefs = node.children[1].get_nodes(VariableRef) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1408 |
if not vrefs: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1409 |
if not ored: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1410 |
self.skip.setdefault(node, set()).update(self.solindices) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1411 |
else: |
1785 | 1412 |
self.mayneedvar.setdefault((node.children[0].name, rschema), []).append( (res, ored) ) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1413 |
else: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1414 |
assert len(vrefs) == 1 |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1415 |
vref = vrefs[0] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1416 |
# XXX check operator ? |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1417 |
self.hasvar[(node.children[0].name, rschema)] = vref |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1418 |
if self._may_skip_attr_rel(rschema, node, vref, ored, terms, res): |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1419 |
self.skip.setdefault(node, set()).update(self.solindices) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1420 |
elif not ored: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1421 |
self.skip.setdefault(node, set()).update(self.solindices) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1422 |
return res, node |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1423 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1424 |
def _may_skip_attr_rel(self, rschema, rel, vref, ored, terms, res): |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1425 |
var = vref.variable |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1426 |
if ored: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1427 |
return False |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1428 |
if var.name in self.extneedsel or var.stinfo['selected']: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1429 |
return False |
2075
933ccfce6e91
fix bug in multisource planner, w/ inlined relation
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
1977
diff
changeset
|
1430 |
if not var in terms or used_in_outer_scope(var, self.current_scope): |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1431 |
return False |
1409
f4dee84a618f
sql attributes bugfix
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents:
1398
diff
changeset
|
1432 |
if any(v for v, _ in var.stinfo['attrvars'] if not v in terms): |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1433 |
return False |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1434 |
return True |
1785 | 1435 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1436 |
def visit_exists(self, node, newroot, terms): |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1437 |
newexists = node.__class__() |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1438 |
self.scopes = {node: newexists} |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1439 |
subparts, node = self._visit_children(node, newroot, terms) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1440 |
if not subparts: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1441 |
return None, node |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1442 |
newexists.set_where(subparts[0]) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1443 |
return newexists, node |
1785 | 1444 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1445 |
def visit_not(self, node, newroot, terms): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1446 |
subparts, node = self._visit_children(node, newroot, terms) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1447 |
if not subparts: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1448 |
return None, node |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1449 |
return copy_node(newroot, node, subparts), node |
1785 | 1450 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1451 |
def visit_group(self, node, newroot, terms): |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1452 |
if not self.final: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1453 |
return None, node |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1454 |
return self.visit_default(node, newroot, terms) |
1785 | 1455 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1456 |
def visit_variableref(self, node, newroot, terms): |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1457 |
if self.use_only_defined: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1458 |
if not node.variable.name in newroot.defined_vars: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1459 |
raise UnsupportedBranch(node.name) |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1460 |
elif not node.variable in terms: |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1461 |
raise UnsupportedBranch(node.name) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1462 |
self.mayneedsel.add(node.name) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1463 |
# set scope so we can insert types restriction properly |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1464 |
newvar = newroot.get_variable(node.name) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1465 |
newvar.stinfo['scope'] = self.scopes.get(node.variable.scope, newroot) |
2109
f469368df530
when a relation isn't supported by a ms query parts, dereference introduced variable refs
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2108
diff
changeset
|
1466 |
vref = VariableRef(newvar) |
f469368df530
when a relation isn't supported by a ms query parts, dereference introduced variable refs
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2108
diff
changeset
|
1467 |
self._pending_vrefs.append(vref) |
f469368df530
when a relation isn't supported by a ms query parts, dereference introduced variable refs
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
2108
diff
changeset
|
1468 |
return vref, node |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1469 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1470 |
def visit_constant(self, node, newroot, terms): |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1471 |
return copy_node(newroot, node), node |
1785 | 1472 |
|
3815
50b87f759b5d
test and fix http://www.logilab.org/ticket/499838
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3689
diff
changeset
|
1473 |
def visit_comparison(self, node, newroot, terms): |
50b87f759b5d
test and fix http://www.logilab.org/ticket/499838
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3689
diff
changeset
|
1474 |
subparts, node = self._visit_children(node, newroot, terms) |
50b87f759b5d
test and fix http://www.logilab.org/ticket/499838
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3689
diff
changeset
|
1475 |
copy = copy_node(newroot, node, subparts) |
50b87f759b5d
test and fix http://www.logilab.org/ticket/499838
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3689
diff
changeset
|
1476 |
# ignore comparison operator when fetching non final query |
50b87f759b5d
test and fix http://www.logilab.org/ticket/499838
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3689
diff
changeset
|
1477 |
if not self.final and isinstance(node.children[0], VariableRef): |
50b87f759b5d
test and fix http://www.logilab.org/ticket/499838
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3689
diff
changeset
|
1478 |
copy.operator = '=' |
50b87f759b5d
test and fix http://www.logilab.org/ticket/499838
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3689
diff
changeset
|
1479 |
return copy, node |
50b87f759b5d
test and fix http://www.logilab.org/ticket/499838
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3689
diff
changeset
|
1480 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1481 |
def visit_default(self, node, newroot, terms): |
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1482 |
subparts, node = self._visit_children(node, newroot, terms) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1483 |
return copy_node(newroot, node, subparts), node |
1785 | 1484 |
|
3815
50b87f759b5d
test and fix http://www.logilab.org/ticket/499838
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
3689
diff
changeset
|
1485 |
visit_mathexpression = visit_constant = visit_function = visit_default |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1486 |
visit_sort = visit_sortterm = visit_default |
1785 | 1487 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1488 |
def _visit_children(self, node, newroot, terms): |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1489 |
subparts = [] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1490 |
for i in xrange(len(node.children)): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1491 |
child = node.children[i] |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1492 |
newchild, child_ = child.accept(self, newroot, terms) |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1493 |
if not child is child_: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1494 |
node = child_.parent |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1495 |
if newchild is not None: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1496 |
subparts.append(newchild) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1497 |
return subparts, node |
1785 | 1498 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1499 |
def process_selection(self, newroot, terms, rqlst): |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1500 |
if self.final: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1501 |
for term in rqlst.selection: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1502 |
newroot.append_selected(term.copy(newroot)) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1503 |
for vref in term.get_nodes(VariableRef): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1504 |
self.needsel.add(vref.name) |
1785 | 1505 |
return |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1506 |
for term in rqlst.selection: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1507 |
vrefs = term.get_nodes(VariableRef) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1508 |
if vrefs: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1509 |
supportedvars = [] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1510 |
for vref in vrefs: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1511 |
var = vref.variable |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1512 |
if var in terms: |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1513 |
supportedvars.append(vref) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1514 |
continue |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1515 |
else: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1516 |
self.needsel.add(vref.name) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1517 |
break |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1518 |
else: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1519 |
for vref in vrefs: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1520 |
newroot.append_selected(vref.copy(newroot)) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1521 |
supportedvars = [] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1522 |
for vref in supportedvars: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1523 |
if not vref in newroot.get_selected_variables(): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1524 |
newroot.append_selected(VariableRef(newroot.get_variable(vref.name))) |
1785 | 1525 |
|
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1526 |
def add_necessary_selection(self, newroot, terms): |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1527 |
selected = tuple(newroot.get_selected_variables()) |
1228
91ae10ffb611
* refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents:
1124
diff
changeset
|
1528 |
for varname in terms: |
257
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1529 |
var = newroot.defined_vars[varname] |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1530 |
for vref in var.references(): |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1531 |
rel = vref.relation() |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1532 |
if rel is None and vref in selected: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1533 |
# already selected |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1534 |
break |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1535 |
else: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1536 |
selvref = VariableRef(var) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1537 |
newroot.append_selected(selvref) |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1538 |
if newroot.groupby: |
4c7d3af7e94d
restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff
changeset
|
1539 |
newroot.add_group_var(VariableRef(selvref.variable, noautoref=1)) |