server/msplanner.py
author Sylvain Thénault <sylvain.thenault@logilab.fr>
Wed, 09 Feb 2011 18:06:17 +0100
changeset 6957 ffda12be2e9f
parent 6943 406a41c25e13
child 7188 b2c45b7396fb
permissions -rw-r--r--
[repository] #1460066: backport datafeed cube as cubicweb source * add some attributes to CWSource to handle this kind of source (not natural to put everything in 'config' string). Adding a CWSource subclass has been attempted then rollbacked because it adds pain to handle multi-sources planning and it introduce an ambiguity on a generic relation (cw_source), which may be a penalty in multiple case * data feed sources are a new kind of source, namely 'copy based', which have no effect on the query planner * a data feed source is associated to a list of url and a parser (appobjects in the 'parsers' registry * entities imported by a data feed have cwuri set to their url on the distant site, their cw_source relation point to the data feed source, though their source stored in the entities table (returned by cw_metainformation) is their physical source, hence 'system'
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
6943
406a41c25e13 [sources] reorganize abstract source code, drop no more used cleanup_entities_info and _cleanup_system_relations methods
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6794
diff changeset
     1
# copyright 2003-2011 LOGILAB S.A. (Paris, FRANCE), all rights reserved.
5421
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5420
diff changeset
     2
# contact http://www.logilab.fr/ -- mailto:contact@logilab.fr
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5420
diff changeset
     3
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5420
diff changeset
     4
# This file is part of CubicWeb.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5420
diff changeset
     5
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5420
diff changeset
     6
# CubicWeb is free software: you can redistribute it and/or modify it under the
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5420
diff changeset
     7
# terms of the GNU Lesser General Public License as published by the Free
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5420
diff changeset
     8
# Software Foundation, either version 2.1 of the License, or (at your option)
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5420
diff changeset
     9
# any later version.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5420
diff changeset
    10
#
5424
8ecbcbff9777 replace logilab-common by CubicWeb in disclaimer
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5421
diff changeset
    11
# CubicWeb is distributed in the hope that it will be useful, but WITHOUT
5421
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5420
diff changeset
    12
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5420
diff changeset
    13
# FOR A PARTICULAR PURPOSE.  See the GNU Lesser General Public License for more
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5420
diff changeset
    14
# details.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5420
diff changeset
    15
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5420
diff changeset
    16
# You should have received a copy of the GNU Lesser General Public License along
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5420
diff changeset
    17
# with CubicWeb.  If not, see <http://www.gnu.org/licenses/>.
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    18
"""plan execution of rql queries on multiple sources
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    19
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
    20
the best way to understand what are we trying to acheive here is to read the
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
    21
unit-tests in unittest_msplanner.py
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    22
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    23
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
    24
What you need to know
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
    25
~~~~~~~~~~~~~~~~~~~~~
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
    26
1. The system source is expected  to support every entity and relation types
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    27
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
    28
2. Given "X relation Y":
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
    29
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
    30
   * if relation, X and Y types are supported by the external source, we suppose
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
    31
     by default that X and Y should both come from the same source as the
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
    32
     relation. You can specify otherwise by adding relation into the
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
    33
     "cross_relations" set in the source's mapping file and it that case, we'll
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
    34
     consider that we can also find in the system source some relation between
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
    35
     X and Y coming from different sources.
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
    36
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
    37
   * if "relation" isn't supported by the external source but X or Y
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
    38
     types (or both) are, we suppose by default that can find in the system
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
    39
     source some relation where X and/or Y come from the external source. You
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
    40
     can specify otherwise by adding relation into the "dont_cross_relations"
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
    41
     set in the source's mapping file and it that case, we'll consider that we
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
    42
     can only find in the system source some relation between X and Y coming
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
    43
     the system source.
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
    44
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
    45
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
    46
Implementation
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
    47
~~~~~~~~~~~~~~
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
    48
XXX explain algorithm
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
    49
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
    50
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
    51
Exemples of multi-sources query execution
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
    52
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1398
5fe84a5f7035 rename internal entity types to have CW prefix instead of E
sylvain.thenault@logilab.fr
parents: 1263
diff changeset
    53
For a system source and a ldap user source (only CWUser and its attributes
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    54
is supported, no group or such):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    55
1398
5fe84a5f7035 rename internal entity types to have CW prefix instead of E
sylvain.thenault@logilab.fr
parents: 1263
diff changeset
    56
:CWUser X:
5fe84a5f7035 rename internal entity types to have CW prefix instead of E
sylvain.thenault@logilab.fr
parents: 1263
diff changeset
    57
1. fetch CWUser X from both sources and return concatenation of results
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    58
1398
5fe84a5f7035 rename internal entity types to have CW prefix instead of E
sylvain.thenault@logilab.fr
parents: 1263
diff changeset
    59
:CWUser X WHERE X in_group G, G name 'users':
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    60
* catch 1
1398
5fe84a5f7035 rename internal entity types to have CW prefix instead of E
sylvain.thenault@logilab.fr
parents: 1263
diff changeset
    61
  1. fetch CWUser X from both sources, store concatenation of results into a
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
    62
     temporary table
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
    63
  2. return the result of TMP X WHERE X in_group G, G name 'users' from the
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
    64
     system source
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    65
* catch 2
1398
5fe84a5f7035 rename internal entity types to have CW prefix instead of E
sylvain.thenault@logilab.fr
parents: 1263
diff changeset
    66
  1. return the result of CWUser X WHERE X in_group G, G name 'users' from system
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
    67
     source, that's enough (optimization of the sql querier will avoid join on
1398
5fe84a5f7035 rename internal entity types to have CW prefix instead of E
sylvain.thenault@logilab.fr
parents: 1263
diff changeset
    68
     CWUser, so we will directly get local eids)
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
    69
1398
5fe84a5f7035 rename internal entity types to have CW prefix instead of E
sylvain.thenault@logilab.fr
parents: 1263
diff changeset
    70
:CWUser X,L WHERE X in_group G, X login L, G name 'users':
5fe84a5f7035 rename internal entity types to have CW prefix instead of E
sylvain.thenault@logilab.fr
parents: 1263
diff changeset
    71
1. fetch Any X,L WHERE X is CWUser, X login L from both sources, store
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    72
   concatenation of results into a temporary table
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    73
2. return the result of Any X, L WHERE X is TMP, X login LX in_group G,
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    74
   G name 'users' from the system source
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    75
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    76
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    77
:Any X WHERE X owned_by Y:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    78
* catch 1
1398
5fe84a5f7035 rename internal entity types to have CW prefix instead of E
sylvain.thenault@logilab.fr
parents: 1263
diff changeset
    79
  1. fetch CWUser X from both sources, store concatenation of results into a
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
    80
     temporary table
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
    81
  2. return the result of Any X WHERE X owned_by Y, Y is TMP from the system
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
    82
     source
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    83
* catch 2
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
    84
  1. return the result of Any X WHERE X owned_by Y from system source, that's
1398
5fe84a5f7035 rename internal entity types to have CW prefix instead of E
sylvain.thenault@logilab.fr
parents: 1263
diff changeset
    85
     enough (optimization of the sql querier will avoid join on CWUser, so we
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
    86
     will directly get local eids)
6427
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
    87
"""
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    88
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    89
__docformat__ = "restructuredtext en"
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    90
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    91
from itertools import imap, ifilterfalse
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    92
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    93
from logilab.common.compat import any
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    94
from logilab.common.decorators import cached
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    95
6427
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
    96
from rql import BadRQLQuery
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    97
from rql.stmts import Union, Select
5582
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
    98
from rql.nodes import (VariableRef, Comparison, Relation, Constant, Variable,
5768
1e73a466aa69 [fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5582
diff changeset
    99
                       Not, Exists, SortTerm, Function)
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   100
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   101
from cubicweb import server
940
15dcdc863965 fix imports : common.utils -> utils
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 442
diff changeset
   102
from cubicweb.utils import make_uid
6128
fbb8398f80dc cleanups
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6057
diff changeset
   103
from cubicweb.rqlrewrite import add_types_restriction
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   104
from cubicweb.server.utils import cleanup_solutions
6128
fbb8398f80dc cleanups
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6057
diff changeset
   105
from cubicweb.server.ssplanner import SSPlanner, OneFetchStep
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   106
from cubicweb.server.mssteps import *
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   107
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   108
Variable._ms_table_key = lambda x: x.name
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   109
Relation._ms_table_key = lambda x: x.r_type
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   110
# str() Constant.value to ensure generated table name won't be unicode
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   111
Constant._ms_table_key = lambda x: str(x.value)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   112
6131
087c5a168010 [ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6129
diff changeset
   113
Variable._ms_may_be_processed = lambda x, terms, linkedterms: any(
087c5a168010 [ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6129
diff changeset
   114
    t for t in terms if t in linkedterms.get(x, ()))
087c5a168010 [ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6129
diff changeset
   115
Relation._ms_may_be_processed = lambda x, terms, linkedterms: all(
087c5a168010 [ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6129
diff changeset
   116
    getattr(hs, 'variable', hs) in terms for hs in x.get_variable_parts())
087c5a168010 [ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6129
diff changeset
   117
5582
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   118
def ms_scope(term):
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   119
    rel = None
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   120
    scope = term.scope
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   121
    if isinstance(term, Variable) and len(term.stinfo['relations']) == 1:
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   122
        rel = iter(term.stinfo['relations']).next().relation()
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   123
    elif isinstance(term, Constant):
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   124
        rel = term.relation()
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   125
    elif isinstance(term, Relation):
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   126
        rel = term
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   127
    if rel is not None and (
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   128
        rel.r_type != 'identity' and rel.scope is scope
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   129
        and isinstance(rel.parent, Exists) and rel.parent.neged(strict=True)):
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   130
        return scope.parent.scope
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   131
    return scope
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   132
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   133
def need_intersect(select, getrschema):
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   134
    for rel in select.iget_nodes(Relation):
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   135
        if isinstance(rel.parent, Exists) and rel.parent.neged(strict=True) and not rel.is_types_restriction():
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   136
            rschema = getrschema(rel.r_type)
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   137
            if not rschema.final:
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   138
                # if one of the relation's variable is ambiguous but not
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   139
                # invariant, an intersection will be necessary
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   140
                for vref in rel.get_nodes(VariableRef):
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   141
                    var = vref.variable
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   142
                    if (var.valuable_references() == 1
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   143
                        and len(var.stinfo['possibletypes']) > 1):
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   144
                        return True
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   145
    return False
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   146
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   147
def neged_relation(rel):
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   148
    parent = rel.parent
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   149
    return isinstance(parent, Not) or (isinstance(parent, Exists) and
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   150
                                       isinstance(parent.parent, Not))
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   151
2688
afa99d53401c [multi-source] test and fix msplanner regression
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2170
diff changeset
   152
def need_source_access_relation(vargraph):
afa99d53401c [multi-source] test and fix msplanner regression
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2170
diff changeset
   153
    if not vargraph:
afa99d53401c [multi-source] test and fix msplanner regression
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2170
diff changeset
   154
        return False
afa99d53401c [multi-source] test and fix msplanner regression
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2170
diff changeset
   155
    # check vargraph contains some other relation than the identity relation
afa99d53401c [multi-source] test and fix msplanner regression
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2170
diff changeset
   156
    # test of key nature since it may be a variable name (don't care about that)
afa99d53401c [multi-source] test and fix msplanner regression
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2170
diff changeset
   157
    # or a 2-uple (var1, var2) associated to the relation to traverse to go from
afa99d53401c [multi-source] test and fix msplanner regression
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2170
diff changeset
   158
    # var1 to var2
afa99d53401c [multi-source] test and fix msplanner regression
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2170
diff changeset
   159
    return any(key for key, val in vargraph.iteritems()
afa99d53401c [multi-source] test and fix msplanner regression
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2170
diff changeset
   160
               if isinstance(key, tuple) and val != 'identity')
afa99d53401c [multi-source] test and fix msplanner regression
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2170
diff changeset
   161
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   162
def need_aggr_step(select, sources, stepdefs=None):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   163
    """return True if a temporary table is necessary to store some partial
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   164
    results to execute the given query
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   165
    """
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   166
    if len(sources) == 1:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   167
        # can do everything at once with a single source
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   168
        return False
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   169
    if select.orderby or select.groupby or select.has_aggregat:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   170
        # if more than one source, we need a temp table to deal with sort /
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   171
        # groups / aggregat if :
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   172
        # * the rqlst won't be splitted (in the other case the last query
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   173
        #   using partial temporary table can do sort/groups/aggregat without
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   174
        #   the need for a later AggrStep)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   175
        # * the rqlst is splitted in multiple steps and there are more than one
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   176
        #   final step
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   177
        if stepdefs is None:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   178
            return True
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   179
        has_one_final = False
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   180
        fstepsolindices = set()
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   181
        for stepdef in stepdefs:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   182
            if stepdef[-1]:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   183
                if has_one_final or frozenset(stepdef[2]) != fstepsolindices:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   184
                    return True
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   185
                has_one_final = True
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   186
            else:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   187
                fstepsolindices.update(stepdef[2])
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   188
    return False
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   189
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   190
def select_group_sort(select): # XXX something similar done in rql2sql
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   191
    # add variables used in groups and sort terms to the selection
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   192
    # if necessary
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   193
    if select.groupby:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   194
        for vref in select.groupby:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   195
            if not vref in select.selection:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   196
                select.append_selected(vref.copy(select))
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   197
    for sortterm in select.orderby:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   198
        for vref in sortterm.iget_nodes(VariableRef):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   199
            if not vref in select.get_selected_variables():
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   200
                # we can't directly insert sortterm.term because it references
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   201
                # a variable of the select before the copy.
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   202
                # XXX if constant term are used to define sort, their value
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   203
                # may necessite a decay
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   204
                select.append_selected(vref.copy(select))
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   205
                if select.groupby and not vref in select.groupby:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   206
                    select.add_group_var(vref.copy(select))
1120
1c24cde4bf72 fix ms planning w/ relation constraint propagation
sylvain.thenault@logilab.fr
parents: 442
diff changeset
   207
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   208
def allequals(solutions):
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   209
    """return true if all solutions are identical"""
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   210
    sol = solutions.next()
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   211
    noconstsol = None
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   212
    for sol_ in solutions:
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   213
        if sol_ != sol:
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   214
            return False
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   215
    return True
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   216
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   217
# XXX move functions below to rql ##############################################
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   218
1120
1c24cde4bf72 fix ms planning w/ relation constraint propagation
sylvain.thenault@logilab.fr
parents: 442
diff changeset
   219
def is_ancestor(n1, n2):
2075
933ccfce6e91 fix bug in multisource planner, w/ inlined relation
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1977
diff changeset
   220
    """return True if n2 is a parent scope of n1"""
1120
1c24cde4bf72 fix ms planning w/ relation constraint propagation
sylvain.thenault@logilab.fr
parents: 442
diff changeset
   221
    p = n1.parent
1c24cde4bf72 fix ms planning w/ relation constraint propagation
sylvain.thenault@logilab.fr
parents: 442
diff changeset
   222
    while p is not None:
1c24cde4bf72 fix ms planning w/ relation constraint propagation
sylvain.thenault@logilab.fr
parents: 442
diff changeset
   223
        if p is n2:
1c24cde4bf72 fix ms planning w/ relation constraint propagation
sylvain.thenault@logilab.fr
parents: 442
diff changeset
   224
            return True
1c24cde4bf72 fix ms planning w/ relation constraint propagation
sylvain.thenault@logilab.fr
parents: 442
diff changeset
   225
        p = p.parent
1c24cde4bf72 fix ms planning w/ relation constraint propagation
sylvain.thenault@logilab.fr
parents: 442
diff changeset
   226
    return False
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   227
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   228
def copy_node(newroot, node, subparts=()):
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   229
    newnode = node.__class__(*node.initargs(newroot))
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   230
    for part in subparts:
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   231
        newnode.append(part)
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   232
    return newnode
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
   233
2075
933ccfce6e91 fix bug in multisource planner, w/ inlined relation
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1977
diff changeset
   234
def used_in_outer_scope(var, scope):
933ccfce6e91 fix bug in multisource planner, w/ inlined relation
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1977
diff changeset
   235
    """return true if the variable is used in an outer scope of the given scope
933ccfce6e91 fix bug in multisource planner, w/ inlined relation
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1977
diff changeset
   236
    """
933ccfce6e91 fix bug in multisource planner, w/ inlined relation
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1977
diff changeset
   237
    for rel in var.stinfo['relations']:
5582
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   238
        rscope = ms_scope(rel)
2075
933ccfce6e91 fix bug in multisource planner, w/ inlined relation
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1977
diff changeset
   239
        if not rscope is scope and is_ancestor(scope, rscope):
933ccfce6e91 fix bug in multisource planner, w/ inlined relation
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1977
diff changeset
   240
            return True
933ccfce6e91 fix bug in multisource planner, w/ inlined relation
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1977
diff changeset
   241
    return False
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
   242
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   243
################################################################################
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   244
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   245
class PartPlanInformation(object):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   246
    """regroups necessary information to execute some part of a "global" rql
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   247
    query ("global" means as received by the querier, which may result in
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   248
    several internal queries, e.g. parts, due to security insertions). Actually
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   249
    a PPI is created for each subquery and for each query in a union.
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   250
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   251
    It exposes as well some methods helping in executing this part on a
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   252
    multi-sources repository, modifying its internal structure during the
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   253
    process.
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   254
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   255
    :attr plan:
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   256
      the execution plan
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   257
    :attr rqlst:
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   258
      the original rql syntax tree handled by this part
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
   259
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   260
    :attr needsplit:
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   261
      bool telling if the query has to be split into multiple steps for
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   262
      execution or if it can be executed at once
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
   263
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   264
    :attr temptable:
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   265
      a SQL temporary table name or None, if necessary to handle aggregate /
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   266
      sorting for this part of the query
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
   267
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   268
    :attr finaltable:
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   269
      a SQL table name or None, if results for this part of the query should be
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   270
      written into a temporary table (usually shared by multiple PPI)
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
   271
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   272
    :attr sourcesterms:
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   273
      a dictionary {source : {term: set([solution index, ])}} telling for each
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   274
      source which terms are supported for which solutions. A "term" may be
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   275
      either a rql Variable, Constant or Relation node.
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   276
    """
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   277
    def __init__(self, plan, rqlst, rqlhelper=None):
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   278
        self.plan = plan
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   279
        self.rqlst = rqlst
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   280
        self.needsplit = False
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   281
        self.temptable = None
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   282
        self.finaltable = None
1231
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   283
        # shortcuts
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   284
        self._schema = plan.schema
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   285
        self._session = plan.session
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   286
        self._repo = self._session.repo
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   287
        self._solutions = rqlst.solutions
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   288
        self._solindices = range(len(self._solutions))
1231
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   289
        self.system_source = self._repo.system_source
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   290
        # source : {term: [solution index, ]}
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   291
        self.sourcesterms = self._sourcesterms = {}
393
45a675515427 cross_relations support
sylvain.thenault@logilab.fr
parents: 392
diff changeset
   292
        # source : {relation: set(child variable and constant)}
45a675515427 cross_relations support
sylvain.thenault@logilab.fr
parents: 392
diff changeset
   293
        self._crossrelations = {}
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   294
        # dictionary of variables and constants which are linked to each other
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   295
        # using a non final relation supported by multiple sources (crossed or
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   296
        # not).
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   297
        self._linkedterms = {}
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   298
        # processing
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   299
        termssources = self._compute_sourcesterms()
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   300
        self._remove_invalid_sources(termssources)
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   301
        self._compute_needsplit()
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   302
        # after initialisation, .sourcesterms contains the same thing as
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   303
        # ._sourcesterms though during plan construction, ._sourcesterms will
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   304
        # be modified while .sourcesterms will be kept unmodified
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   305
        self.sourcesterms = {}
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   306
        for k, v in self._sourcesterms.iteritems():
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   307
            self.sourcesterms[k] = {}
393
45a675515427 cross_relations support
sylvain.thenault@logilab.fr
parents: 392
diff changeset
   308
            for k2, v2 in v.iteritems():
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   309
                self.sourcesterms[k][k2] = v2.copy()
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   310
        # cleanup linked var
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   311
        for var, linkedrelsinfo in self._linkedterms.iteritems():
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   312
            self._linkedterms[var] = frozenset(x[0] for x in linkedrelsinfo)
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   313
        # map output of a step to input of a following step
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   314
        self._inputmaps = {}
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   315
        # record input map conflicts to resolve them on final step generation
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   316
        self._conflicts = []
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   317
        if rqlhelper is not None: # else test
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   318
            self._insert_identity_variable = rqlhelper._annotator.rewrite_shared_optional
2766
07e47b84acc2 DBG_MS flag
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2688
diff changeset
   319
        if server.DEBUG & server.DBG_MS:
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   320
            print 'sourcesterms:'
1786
eccd1885d42e one more msplanning bug fixed...
sylvain.thenault@logilab.fr
parents: 1785
diff changeset
   321
            self._debug_sourcesterms()
eccd1885d42e one more msplanning bug fixed...
sylvain.thenault@logilab.fr
parents: 1785
diff changeset
   322
eccd1885d42e one more msplanning bug fixed...
sylvain.thenault@logilab.fr
parents: 1785
diff changeset
   323
    def _debug_sourcesterms(self):
eccd1885d42e one more msplanning bug fixed...
sylvain.thenault@logilab.fr
parents: 1785
diff changeset
   324
        for source in self._sourcesterms:
eccd1885d42e one more msplanning bug fixed...
sylvain.thenault@logilab.fr
parents: 1785
diff changeset
   325
            print '-', source
eccd1885d42e one more msplanning bug fixed...
sylvain.thenault@logilab.fr
parents: 1785
diff changeset
   326
            for term, sols in self._sourcesterms[source].items():
4721
8f63691ccb7f pylint style fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
   327
                print '  -', term, id(term), ':', sols
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
   328
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   329
    def copy_solutions(self, solindices):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   330
        return [self._solutions[solidx].copy() for solidx in solindices]
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
   331
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   332
    @property
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   333
    @cached
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   334
    def part_sources(self):
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   335
        if self._sourcesterms:
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   336
            return tuple(sorted(self._sourcesterms))
1231
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   337
        return (self.system_source,)
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
   338
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   339
    @property
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   340
    @cached
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   341
    def _sys_source_set(self):
1231
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   342
        return frozenset((self.system_source, solindex)
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
   343
                         for solindex in self._solindices)
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
   344
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   345
    @cached
393
45a675515427 cross_relations support
sylvain.thenault@logilab.fr
parents: 392
diff changeset
   346
    def _norel_support_set(self, relation):
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   347
        """return a set of (source, solindex) where source doesn't support the
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   348
        relation
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   349
        """
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   350
        return frozenset((source, solidx) for source in self._repo.sources
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   351
                         for solidx in self._solindices
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   352
                         if not ((source.support_relation(relation.r_type))
393
45a675515427 cross_relations support
sylvain.thenault@logilab.fr
parents: 392
diff changeset
   353
                                 or relation.r_type in source.dont_cross_relations))
45a675515427 cross_relations support
sylvain.thenault@logilab.fr
parents: 392
diff changeset
   354
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   355
    def _compute_sourcesterms(self):
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   356
        """compute for each term (variable, rewritten constant, relation) and
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   357
        for each solution in the rqlst which sources support them
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   358
        """
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   359
        repo = self._repo
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   360
        eschema = self._schema.eschema
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   361
        sourcesterms = self._sourcesterms
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   362
        # find for each source which variable/solution are supported
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   363
        for varname, varobj in self.rqlst.defined_vars.items():
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   364
            # if variable has an eid specified, we can get its source directly
5004
4cc020ee70e2 le patch rql26 a été importé
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4956
diff changeset
   365
            # NOTE: use uidrel and not constnode to deal with "X eid IN(1,2,3,4)"
4cc020ee70e2 le patch rql26 a été importé
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4956
diff changeset
   366
            if varobj.stinfo['uidrel'] is not None:
4cc020ee70e2 le patch rql26 a été importé
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4956
diff changeset
   367
                rel = varobj.stinfo['uidrel']
5161
a19f22bacedc [ms] fix planning bug introduced with recent rql refactoring for queries with only eids (fix unittest_multisources.test_in_eid)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5047
diff changeset
   368
                hasrel = len(varobj.stinfo['relations']) > 1
5004
4cc020ee70e2 le patch rql26 a été importé
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4956
diff changeset
   369
                for const in rel.children[1].get_nodes(Constant):
4cc020ee70e2 le patch rql26 a été importé
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4956
diff changeset
   370
                    eid = const.eval(self.plan.args)
4cc020ee70e2 le patch rql26 a été importé
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4956
diff changeset
   371
                    source = self._session.source_from_eid(eid)
5161
a19f22bacedc [ms] fix planning bug introduced with recent rql refactoring for queries with only eids (fix unittest_multisources.test_in_eid)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5047
diff changeset
   372
                    if (source is self.system_source
a19f22bacedc [ms] fix planning bug introduced with recent rql refactoring for queries with only eids (fix unittest_multisources.test_in_eid)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5047
diff changeset
   373
                        or (hasrel and
a19f22bacedc [ms] fix planning bug introduced with recent rql refactoring for queries with only eids (fix unittest_multisources.test_in_eid)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5047
diff changeset
   374
                            not any(source.support_relation(r.r_type)
a19f22bacedc [ms] fix planning bug introduced with recent rql refactoring for queries with only eids (fix unittest_multisources.test_in_eid)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5047
diff changeset
   375
                                    for r in varobj.stinfo['relations']
a19f22bacedc [ms] fix planning bug introduced with recent rql refactoring for queries with only eids (fix unittest_multisources.test_in_eid)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5047
diff changeset
   376
                                    if not r is rel))):
5004
4cc020ee70e2 le patch rql26 a été importé
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4956
diff changeset
   377
                        self._set_source_for_term(self.system_source, varobj)
4cc020ee70e2 le patch rql26 a été importé
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4956
diff changeset
   378
                    else:
4cc020ee70e2 le patch rql26 a été importé
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4956
diff changeset
   379
                        self._set_source_for_term(source, varobj)
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   380
                continue
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   381
            rels = varobj.stinfo['relations']
5004
4cc020ee70e2 le patch rql26 a été importé
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4956
diff changeset
   382
            if not rels and varobj.stinfo['typerel'] is None:
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   383
                # (rare) case where the variable has no type specified nor
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   384
                # relation accessed ex. "Any MAX(X)"
1231
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   385
                self._set_source_for_term(self.system_source, varobj)
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   386
                continue
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   387
            for i, sol in enumerate(self._solutions):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   388
                vartype = sol[varname]
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   389
                # skip final variable
3689
deb13e88e037 follow yams 0.25 api changes to improve performance
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3624
diff changeset
   390
                if eschema(vartype).final:
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   391
                    break
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   392
                for source in repo.sources:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   393
                    if source.support_entity(vartype):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   394
                        # the source support the entity type, though we will
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   395
                        # actually have to fetch from it only if
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   396
                        # * the variable isn't invariant
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   397
                        # * at least one supported relation specified
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   398
                        if not varobj._q_invariant or \
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   399
                               any(imap(source.support_relation,
3239
1ceac4cd4fb7 [ms] fix planning w/ query using identity
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3040
diff changeset
   400
                                        (r.r_type for r in rels if r.r_type not in ('identity', 'eid')))):
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   401
                            sourcesterms.setdefault(source, {}).setdefault(varobj, set()).add(i)
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   402
                        # if variable is not invariant and is used by a relation
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   403
                        # not supported by this source, we'll have to split the
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   404
                        # query
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   405
                        if not varobj._q_invariant and any(ifilterfalse(
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   406
                            source.support_relation, (r.r_type for r in rels))):
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
   407
                            self.needsplit = True
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   408
        # add source for rewritten constants to sourcesterms
5420
09b50d7e5321 [ms] fix planning bug with rewritten constants: when removing some source for a const due to relation constraints, we should propagate this change to other constants coming from the same original variable
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5168
diff changeset
   409
        self._const_vars = {}
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   410
        for vconsts in self.rqlst.stinfo['rewritten'].itervalues():
5420
09b50d7e5321 [ms] fix planning bug with rewritten constants: when removing some source for a const due to relation constraints, we should propagate this change to other constants coming from the same original variable
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5168
diff changeset
   411
            # remember those consts come from the same variable
09b50d7e5321 [ms] fix planning bug with rewritten constants: when removing some source for a const due to relation constraints, we should propagate this change to other constants coming from the same original variable
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5168
diff changeset
   412
            for const in vconsts:
09b50d7e5321 [ms] fix planning bug with rewritten constants: when removing some source for a const due to relation constraints, we should propagate this change to other constants coming from the same original variable
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5168
diff changeset
   413
                self._const_vars[const] = vconsts
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   414
            source = self._session.source_from_eid(const.eval(self.plan.args))
1231
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   415
            if source is self.system_source:
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   416
                for const in vconsts:
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   417
                    self._set_source_for_term(source, const)
2108
b3f6fa7ad59f add test and fix for queries where all variables are simplified and come from an external source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2075
diff changeset
   418
            elif not self._sourcesterms:
6131
087c5a168010 [ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6129
diff changeset
   419
                for const in vconsts:
087c5a168010 [ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6129
diff changeset
   420
                    self._set_source_for_term(source, const)
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   421
            elif source in self._sourcesterms:
5582
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   422
                source_scopes = frozenset(ms_scope(t) for t in self._sourcesterms[source])
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   423
                for const in vconsts:
5582
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   424
                    if ms_scope(const) in source_scopes:
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   425
                        self._set_source_for_term(source, const)
1230
232e16835fff fix constant handling in sourcesterms + minor cleanup
sylvain.thenault@logilab.fr
parents: 1229
diff changeset
   426
                        # if system source is used, add every rewritten constant
232e16835fff fix constant handling in sourcesterms + minor cleanup
sylvain.thenault@logilab.fr
parents: 1229
diff changeset
   427
                        # to its supported terms even when associated entity
6057
cb5c8852cbda [ms] cleanups
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6056
diff changeset
   428
                        # doesn't actually come from it so we get a changes that
cb5c8852cbda [ms] cleanups
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6056
diff changeset
   429
                        # allequals will return True as expected when computing
cb5c8852cbda [ms] cleanups
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6056
diff changeset
   430
                        # needsplit
1786
eccd1885d42e one more msplanning bug fixed...
sylvain.thenault@logilab.fr
parents: 1785
diff changeset
   431
                        # check const is used in a relation restriction
eccd1885d42e one more msplanning bug fixed...
sylvain.thenault@logilab.fr
parents: 1785
diff changeset
   432
                        if const.relation() and self.system_source in sourcesterms:
1231
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   433
                            self._set_source_for_term(self.system_source, const)
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   434
        # add source for relations
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   435
        rschema = self._schema.rschema
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   436
        termssources = {}
6427
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   437
        sourcerels = []
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   438
        for rel in self.rqlst.iget_nodes(Relation):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   439
            # process non final relations only
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   440
            # note: don't try to get schema for 'is' relation (not available
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   441
            # during bootstrap)
6427
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   442
            if rel.r_type == 'cw_source':
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   443
                sourcerels.append(rel)
6649
29f8e5c35392 [ms planning] fix/enhance cw_source consideration in ms planning
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6633
diff changeset
   444
            if not (rel.is_types_restriction() or rschema(rel.r_type).final):
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   445
                # nothing to do if relation is not supported by multiple sources
393
45a675515427 cross_relations support
sylvain.thenault@logilab.fr
parents: 392
diff changeset
   446
                # or if some source has it listed in its cross_relations
45a675515427 cross_relations support
sylvain.thenault@logilab.fr
parents: 392
diff changeset
   447
                # attribute
45a675515427 cross_relations support
sylvain.thenault@logilab.fr
parents: 392
diff changeset
   448
                #
45a675515427 cross_relations support
sylvain.thenault@logilab.fr
parents: 392
diff changeset
   449
                # XXX code below don't deal if some source allow relation
45a675515427 cross_relations support
sylvain.thenault@logilab.fr
parents: 392
diff changeset
   450
                #     crossing but not another one
6129
fea746b60093 [ms] fix two planner bugs: one occuring query such as X created_by U where X in a external source and U may come from an ldap source. The other being that when we've to merge input maps, we were modifying the same tree/solutions while a copy were needed. Also, ensure we add type restrictions, necessary for pyro source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6128
diff changeset
   451
                relsources = [s for s in repo.rel_type_sources(rel.r_type)
fea746b60093 [ms] fix two planner bugs: one occuring query such as X created_by U where X in a external source and U may come from an ldap source. The other being that when we've to merge input maps, we were modifying the same tree/solutions while a copy were needed. Also, ensure we add type restrictions, necessary for pyro source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6128
diff changeset
   452
                               if s is self.system_source
fea746b60093 [ms] fix two planner bugs: one occuring query such as X created_by U where X in a external source and U may come from an ldap source. The other being that when we've to merge input maps, we were modifying the same tree/solutions while a copy were needed. Also, ensure we add type restrictions, necessary for pyro source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6128
diff changeset
   453
                               or s in self._sourcesterms]
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   454
                if len(relsources) < 2:
393
45a675515427 cross_relations support
sylvain.thenault@logilab.fr
parents: 392
diff changeset
   455
                    # filter out sources being there because they have this
45a675515427 cross_relations support
sylvain.thenault@logilab.fr
parents: 392
diff changeset
   456
                    # relation in their dont_cross_relations attribute
45a675515427 cross_relations support
sylvain.thenault@logilab.fr
parents: 392
diff changeset
   457
                    relsources = [source for source in relsources
45a675515427 cross_relations support
sylvain.thenault@logilab.fr
parents: 392
diff changeset
   458
                                  if source.support_relation(rel.r_type)]
341
0a426be2f3a2 fixes for some cases of neged inlined relations, may have to use (new) IntersectStep. XXX: write IntersectFetchStep
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents: 257
diff changeset
   459
                    if relsources:
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   460
                        # this means the relation is using a variable inlined as
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   461
                        # a constant and another unsupported variable, in which
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   462
                        # case we put the relation in sourcesterms
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   463
                        self._sourcesterms.setdefault(relsources[0], {})[rel] = set(self._solindices)
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   464
                    continue
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   465
                lhs, rhs = rel.get_variable_parts()
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   466
                lhsv, rhsv = getattr(lhs, 'variable', lhs), getattr(rhs, 'variable', rhs)
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   467
                # update dictionary of sources supporting lhs and rhs vars
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   468
                if not lhsv in termssources:
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   469
                    termssources[lhsv] = self._term_sources(lhs)
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   470
                if not rhsv in termssources:
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   471
                    termssources[rhsv] = self._term_sources(rhs)
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   472
                self._handle_cross_relation(rel, relsources, termssources)
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   473
                self._linkedterms.setdefault(lhsv, set()).add((rhsv, rel))
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   474
                self._linkedterms.setdefault(rhsv, set()).add((lhsv, rel))
6427
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   475
        # extract information from cw_source relation
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   476
        for srel in sourcerels:
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   477
            vref = srel.children[1].children[0]
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   478
            sourceeids, sourcenames = [], []
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   479
            if isinstance(vref, Constant):
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   480
                # simplified variable
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   481
                sourceeids = None, (vref.eval(self.plan.args),)
6649
29f8e5c35392 [ms planning] fix/enhance cw_source consideration in ms planning
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6633
diff changeset
   482
                var = vref
6427
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   483
            else:
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   484
                var = vref.variable
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   485
                for rel in var.stinfo['relations'] - var.stinfo['rhsrelations']:
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   486
                    if rel.r_type in ('eid', 'name'):
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   487
                        if rel.r_type == 'eid':
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   488
                            slist = sourceeids
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   489
                        else:
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   490
                            slist = sourcenames
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   491
                        sources = [cst.eval(self.plan.args)
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   492
                                   for cst in rel.children[1].get_nodes(Constant)]
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   493
                        if sources:
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   494
                            if slist:
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   495
                                # don't attempt to do anything
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   496
                                sourcenames = sourceeids = None
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   497
                                break
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   498
                            slist[:] = (rel, sources)
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   499
            if sourceeids:
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   500
                rel, values = sourceeids
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   501
                sourcesdict = self._repo.sources_by_eid
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   502
            elif sourcenames:
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   503
                rel, values = sourcenames
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   504
                sourcesdict = self._repo.sources_by_uri
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   505
            else:
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   506
                sourcesdict = None
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   507
            if sourcesdict is not None:
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   508
                lhs = srel.children[0]
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   509
                try:
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   510
                    sources = [sourcesdict[key] for key in values]
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   511
                except KeyError:
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   512
                    raise BadRQLQuery('source conflict for term %s' % lhs.as_string())
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   513
                if isinstance(lhs, Constant):
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   514
                    source = self._session.source_from_eid(lhs.eval(self.plan.args))
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   515
                    if not source in sources:
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   516
                        raise BadRQLQuery('source conflict for term %s' % lhs.as_string())
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   517
                else:
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   518
                    lhs = getattr(lhs, 'variable', lhs)
6649
29f8e5c35392 [ms planning] fix/enhance cw_source consideration in ms planning
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6633
diff changeset
   519
                invariant = getattr(lhs, '_q_invariant', False)
6427
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   520
                # XXX NOT NOT
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   521
                neged = srel.neged(traverse_scope=True) or (rel and rel.neged(strict=True))
6957
ffda12be2e9f [repository] #1460066: backport datafeed cube as cubicweb source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6943
diff changeset
   522
                has_copy_based_source = False
ffda12be2e9f [repository] #1460066: backport datafeed cube as cubicweb source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6943
diff changeset
   523
                sources_ = []
ffda12be2e9f [repository] #1460066: backport datafeed cube as cubicweb source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6943
diff changeset
   524
                for source in sources:
ffda12be2e9f [repository] #1460066: backport datafeed cube as cubicweb source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6943
diff changeset
   525
                    if source.copy_based_source:
ffda12be2e9f [repository] #1460066: backport datafeed cube as cubicweb source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6943
diff changeset
   526
                        has_copy_based_source = True
ffda12be2e9f [repository] #1460066: backport datafeed cube as cubicweb source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6943
diff changeset
   527
                        if not self.system_source in sources_:
ffda12be2e9f [repository] #1460066: backport datafeed cube as cubicweb source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6943
diff changeset
   528
                            sources_.append(self.system_source)
ffda12be2e9f [repository] #1460066: backport datafeed cube as cubicweb source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6943
diff changeset
   529
                    else:
ffda12be2e9f [repository] #1460066: backport datafeed cube as cubicweb source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6943
diff changeset
   530
                        sources_.append(source)
ffda12be2e9f [repository] #1460066: backport datafeed cube as cubicweb source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6943
diff changeset
   531
                sources = sources_
6427
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   532
                if neged:
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   533
                    for source in sources:
6649
29f8e5c35392 [ms planning] fix/enhance cw_source consideration in ms planning
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6633
diff changeset
   534
                        if invariant and source is self.system_source:
29f8e5c35392 [ms planning] fix/enhance cw_source consideration in ms planning
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6633
diff changeset
   535
                            continue
6633
7baea108d326 [ms] simplify cw_source handling : don't try evil optmization that may hurt in some cases
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6598
diff changeset
   536
                        self._remove_source_term(source, lhs)
6649
29f8e5c35392 [ms planning] fix/enhance cw_source consideration in ms planning
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6633
diff changeset
   537
                    usesys = self.system_source not in sources
6427
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   538
                else:
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   539
                    for source, terms in sourcesterms.items():
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   540
                        if lhs in terms and not source in sources:
6649
29f8e5c35392 [ms planning] fix/enhance cw_source consideration in ms planning
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6633
diff changeset
   541
                            if invariant and source is self.system_source:
29f8e5c35392 [ms planning] fix/enhance cw_source consideration in ms planning
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6633
diff changeset
   542
                                continue
6633
7baea108d326 [ms] simplify cw_source handling : don't try evil optmization that may hurt in some cases
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6598
diff changeset
   543
                            self._remove_source_term(source, lhs)
6649
29f8e5c35392 [ms planning] fix/enhance cw_source consideration in ms planning
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6633
diff changeset
   544
                    usesys = self.system_source in sources
29f8e5c35392 [ms planning] fix/enhance cw_source consideration in ms planning
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6633
diff changeset
   545
                if rel is None or (len(var.stinfo['relations']) == 2 and
29f8e5c35392 [ms planning] fix/enhance cw_source consideration in ms planning
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6633
diff changeset
   546
                                   not var.stinfo['selected']):
6427
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   547
                    self._remove_source_term(self.system_source, var)
6957
ffda12be2e9f [repository] #1460066: backport datafeed cube as cubicweb source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6943
diff changeset
   548
                    if not (has_copy_based_source or len(sources) > 1
ffda12be2e9f [repository] #1460066: backport datafeed cube as cubicweb source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6943
diff changeset
   549
                            or usesys or invariant):
6649
29f8e5c35392 [ms planning] fix/enhance cw_source consideration in ms planning
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6633
diff changeset
   550
                        if rel is None:
29f8e5c35392 [ms planning] fix/enhance cw_source consideration in ms planning
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6633
diff changeset
   551
                            srel.parent.remove(srel)
29f8e5c35392 [ms planning] fix/enhance cw_source consideration in ms planning
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6633
diff changeset
   552
                        else:
29f8e5c35392 [ms planning] fix/enhance cw_source consideration in ms planning
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6633
diff changeset
   553
                            self.rqlst.undefine_variable(var)
29f8e5c35392 [ms planning] fix/enhance cw_source consideration in ms planning
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6633
diff changeset
   554
                        self._remove_source_term(self.system_source, srel)
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   555
        return termssources
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
   556
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   557
    def _handle_cross_relation(self, rel, relsources, termssources):
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   558
        for source in relsources:
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   559
            if rel.r_type in source.cross_relations:
1231
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   560
                ssource = self.system_source
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   561
                crossvars = set(x.variable for x in rel.get_nodes(VariableRef))
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   562
                for const in rel.get_nodes(Constant):
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   563
                    if source.uri != 'system' and not const in self._sourcesterms.get(source, ()):
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   564
                        continue
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   565
                    crossvars.add(const)
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   566
                self._crossrelations.setdefault(source, {})[rel] = crossvars
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   567
                if len(crossvars) < 2:
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   568
                    # this means there is a constant in the relation which is
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   569
                    # not supported by the source, so we can stop here
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   570
                    continue
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   571
                self._sourcesterms.setdefault(ssource, {})[rel] = set(self._solindices)
6131
087c5a168010 [ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6129
diff changeset
   572
                solindices = None
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   573
                for term in crossvars:
1229
dd9bdcfc03b6 cleanup _handle_cross_relation
sylvain.thenault@logilab.fr
parents: 1228
diff changeset
   574
                    if len(termssources[term]) == 1 and iter(termssources[term]).next()[0].uri == 'system':
dd9bdcfc03b6 cleanup _handle_cross_relation
sylvain.thenault@logilab.fr
parents: 1228
diff changeset
   575
                        for ov in crossvars:
dd9bdcfc03b6 cleanup _handle_cross_relation
sylvain.thenault@logilab.fr
parents: 1228
diff changeset
   576
                            if ov is not term and (isinstance(ov, Constant) or ov._q_invariant):
dd9bdcfc03b6 cleanup _handle_cross_relation
sylvain.thenault@logilab.fr
parents: 1228
diff changeset
   577
                                ssset = frozenset((ssource,))
dd9bdcfc03b6 cleanup _handle_cross_relation
sylvain.thenault@logilab.fr
parents: 1228
diff changeset
   578
                                self._remove_sources(ov, termssources[ov] - ssset)
dd9bdcfc03b6 cleanup _handle_cross_relation
sylvain.thenault@logilab.fr
parents: 1228
diff changeset
   579
                        break
6131
087c5a168010 [ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6129
diff changeset
   580
                    if solindices is None:
087c5a168010 [ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6129
diff changeset
   581
                        solindices = set(sol for s, sol in termssources[term]
087c5a168010 [ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6129
diff changeset
   582
                                         if s is source)
087c5a168010 [ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6129
diff changeset
   583
                    else:
087c5a168010 [ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6129
diff changeset
   584
                        solindices &= set(sol for s, sol in termssources[term]
087c5a168010 [ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6129
diff changeset
   585
                                          if s is source)
1229
dd9bdcfc03b6 cleanup _handle_cross_relation
sylvain.thenault@logilab.fr
parents: 1228
diff changeset
   586
                else:
6131
087c5a168010 [ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6129
diff changeset
   587
                    self._sourcesterms.setdefault(source, {})[rel] = solindices
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
   588
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   589
    def _remove_invalid_sources(self, termssources):
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   590
        """removes invalid sources from `sourcesterms` member according to
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   591
        traversed relations and their properties (which sources support them,
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   592
        can they cross sources, etc...)
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   593
        """
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   594
        for term in self._linkedterms:
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   595
            self._remove_sources_until_stable(term, termssources)
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   596
        if len(self._sourcesterms) > 1 and hasattr(self.plan.rqlst, 'main_relations'):
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   597
            # the querier doesn't annotate write queries, need to do it here
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   598
            self.plan.annotate_rqlst()
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   599
            # insert/update/delete queries, we may get extra information from
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   600
            # the main relation (eg relations to the left of the WHERE
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   601
            if self.plan.rqlst.TYPE == 'insert':
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   602
                inserted = dict((vref.variable, etype)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   603
                                for etype, vref in self.plan.rqlst.main_variables)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   604
            else:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   605
                inserted = {}
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   606
            repo = self._repo
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   607
            rschema = self._schema.rschema
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   608
            for rel in self.plan.rqlst.main_relations:
3689
deb13e88e037 follow yams 0.25 api changes to improve performance
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3624
diff changeset
   609
                if not rschema(rel.r_type).final:
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   610
                    # nothing to do if relation is not supported by multiple sources
393
45a675515427 cross_relations support
sylvain.thenault@logilab.fr
parents: 392
diff changeset
   611
                    if len(repo.rel_type_sources(rel.r_type)) < 2:
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   612
                        continue
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   613
                    lhs, rhs = rel.get_variable_parts()
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   614
                    try:
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   615
                        lhsv = self._extern_term(lhs, termssources, inserted)
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   616
                        rhsv = self._extern_term(rhs, termssources, inserted)
4719
aaed3f813ef8 kill dead/useless code as suggested by pylint
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4716
diff changeset
   617
                    except KeyError:
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   618
                        continue
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   619
                    self._remove_term_sources(lhsv, rel, rhsv, termssources)
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   620
                    self._remove_term_sources(rhsv, rel, lhsv, termssources)
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
   621
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   622
    def _extern_term(self, term, termssources, inserted):
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   623
        var = term.variable
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   624
        if var.stinfo['constnode']:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   625
            termv = var.stinfo['constnode']
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   626
            termssources[termv] = self._term_sources(termv)
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   627
        elif var in inserted:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   628
            termv = var
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   629
            source = self._repo.locate_etype_source(inserted[var])
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   630
            termssources[termv] = set((source, solindex)
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   631
                                      for solindex in self._solindices)
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   632
        else:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   633
            termv = self.rqlst.defined_vars[var.name]
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   634
            if not termv in termssources:
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   635
                termssources[termv] = self._term_sources(termv)
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   636
        return termv
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
   637
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   638
    def _remove_sources_until_stable(self, term, termssources):
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   639
        sourcesterms = self._sourcesterms
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   640
        for oterm, rel in self._linkedterms.get(term, ()):
5582
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   641
            tscope = ms_scope(term)
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   642
            otscope = ms_scope(oterm)
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   643
            rscope = ms_scope(rel)
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   644
            if not tscope is otscope and rscope.neged(strict=True):
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   645
                # can't get information from relation inside a NOT exists
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   646
                # where terms don't belong to the same scope
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   647
                continue
1120
1c24cde4bf72 fix ms planning w/ relation constraint propagation
sylvain.thenault@logilab.fr
parents: 442
diff changeset
   648
            need_ancestor_scope = False
5582
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   649
            if not (tscope is rscope and otscope is rscope):
1120
1c24cde4bf72 fix ms planning w/ relation constraint propagation
sylvain.thenault@logilab.fr
parents: 442
diff changeset
   650
                if rel.ored():
1c24cde4bf72 fix ms planning w/ relation constraint propagation
sylvain.thenault@logilab.fr
parents: 442
diff changeset
   651
                    continue
1c24cde4bf72 fix ms planning w/ relation constraint propagation
sylvain.thenault@logilab.fr
parents: 442
diff changeset
   652
                if rel.ored(traverse_scope=True):
1c24cde4bf72 fix ms planning w/ relation constraint propagation
sylvain.thenault@logilab.fr
parents: 442
diff changeset
   653
                    # if relation has some OR as parent, constraints should only
1c24cde4bf72 fix ms planning w/ relation constraint propagation
sylvain.thenault@logilab.fr
parents: 442
diff changeset
   654
                    # propagate from parent scope to child scope, nothing else
1c24cde4bf72 fix ms planning w/ relation constraint propagation
sylvain.thenault@logilab.fr
parents: 442
diff changeset
   655
                    need_ancestor_scope = True
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   656
            relsources = self._repo.rel_type_sources(rel.r_type)
5582
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   657
            if neged_relation(rel) and (
341
0a426be2f3a2 fixes for some cases of neged inlined relations, may have to use (new) IntersectStep. XXX: write IntersectFetchStep
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents: 257
diff changeset
   658
                len(relsources) < 2
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   659
                or not isinstance(oterm, Variable)
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   660
                or oterm.valuable_references() != 1
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   661
                or any(sourcesterms[source][term] != sourcesterms[source][oterm]
341
0a426be2f3a2 fixes for some cases of neged inlined relations, may have to use (new) IntersectStep. XXX: write IntersectFetchStep
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents: 257
diff changeset
   662
                       for source in relsources
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   663
                       if term in sourcesterms.get(source, ())
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   664
                       and oterm in sourcesterms.get(source, ()))):
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   665
                # neged relation doesn't allow to infer term sources unless
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   666
                # we're on a multisource relation for a term only used by this
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   667
                # relation (eg "Any X WHERE NOT X multisource_rel Y" and over is
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   668
                # Y)
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   669
                continue
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   670
            # compute invalid sources for terms and remove them
5582
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   671
            if not need_ancestor_scope or is_ancestor(tscope, otscope):
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   672
                self._remove_term_sources(term, rel, oterm, termssources)
5582
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   673
            if not need_ancestor_scope or is_ancestor(otscope, tscope):
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   674
                self._remove_term_sources(oterm, rel, term, termssources)
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
   675
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   676
    def _remove_term_sources(self, term, rel, oterm, termssources):
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   677
        """remove invalid sources for term according to oterm's sources and the
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
   678
        relation between those two terms.
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   679
        """
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   680
        norelsup = self._norel_support_set(rel)
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   681
        termsources = termssources[term]
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   682
        invalid_sources = termsources - (termssources[oterm] | norelsup)
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   683
        if invalid_sources and self._repo.can_cross_relation(rel.r_type):
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   684
            invalid_sources -= self._sys_source_set
1230
232e16835fff fix constant handling in sourcesterms + minor cleanup
sylvain.thenault@logilab.fr
parents: 1229
diff changeset
   685
            if invalid_sources and isinstance(term, Variable) \
232e16835fff fix constant handling in sourcesterms + minor cleanup
sylvain.thenault@logilab.fr
parents: 1229
diff changeset
   686
                   and self._need_ext_source_access(term, rel):
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   687
                # if the term is a not invariant variable, we should filter out
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   688
                # source where the relation is a cross relation from invalid
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   689
                # sources
1231
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   690
                invalid_sources = frozenset((s, solidx) for s, solidx in invalid_sources
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   691
                                            if not (s in self._crossrelations and
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   692
                                                    rel in self._crossrelations[s]))
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   693
        if invalid_sources:
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   694
            self._remove_sources(term, invalid_sources)
5420
09b50d7e5321 [ms] fix planning bug with rewritten constants: when removing some source for a const due to relation constraints, we should propagate this change to other constants coming from the same original variable
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5168
diff changeset
   695
            # if term is a rewritten const, we can apply the same changes to
09b50d7e5321 [ms] fix planning bug with rewritten constants: when removing some source for a const due to relation constraints, we should propagate this change to other constants coming from the same original variable
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5168
diff changeset
   696
            # all other consts inserted from the same original variable
09b50d7e5321 [ms] fix planning bug with rewritten constants: when removing some source for a const due to relation constraints, we should propagate this change to other constants coming from the same original variable
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5168
diff changeset
   697
            for const in self._const_vars.get(term, ()):
09b50d7e5321 [ms] fix planning bug with rewritten constants: when removing some source for a const due to relation constraints, we should propagate this change to other constants coming from the same original variable
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5168
diff changeset
   698
                if const is not term:
09b50d7e5321 [ms] fix planning bug with rewritten constants: when removing some source for a const due to relation constraints, we should propagate this change to other constants coming from the same original variable
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5168
diff changeset
   699
                    self._remove_sources(const, invalid_sources)
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   700
            termsources -= invalid_sources
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   701
            self._remove_sources_until_stable(term, termssources)
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
   702
            if isinstance(oterm, Constant):
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
   703
                self._remove_sources(oterm, invalid_sources)
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
   704
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   705
    def _compute_needsplit(self):
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   706
        """tell according to sourcesterms if the rqlst has to be splitted for
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   707
        execution among multiple sources
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
   708
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   709
        the execution has to be split if
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   710
        * a source support an entity (non invariant) but doesn't support a
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   711
          relation on it
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   712
        * a source support an entity which is accessed by an optional relation
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
   713
        * there is more than one source and either all sources'supported
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   714
          variable/solutions are not equivalent or multiple variables have to
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   715
          be fetched from some source
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   716
        """
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   717
        # NOTE: < 2 since may be 0 on queries such as Any X WHERE X eid 2
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
   718
        if len(self._sourcesterms) < 2:
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   719
            self.needsplit = False
2165
dd40c375044e #344177: ms planning bug with "Any X WHERE X eid 123" and 123 coming from an external source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2114
diff changeset
   720
            # if this is not the system source but we have only constant terms
dd40c375044e #344177: ms planning bug with "Any X WHERE X eid 123" and 123 coming from an external source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2114
diff changeset
   721
            # and no relation (other than eid), apply query on the system source
dd40c375044e #344177: ms planning bug with "Any X WHERE X eid 123" and 123 coming from an external source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2114
diff changeset
   722
            #
dd40c375044e #344177: ms planning bug with "Any X WHERE X eid 123" and 123 coming from an external source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2114
diff changeset
   723
            # testing for rqlst with nothing in vargraph nor defined_vars is the
dd40c375044e #344177: ms planning bug with "Any X WHERE X eid 123" and 123 coming from an external source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2114
diff changeset
   724
            # simplest way the check the condition explained below
dd40c375044e #344177: ms planning bug with "Any X WHERE X eid 123" and 123 coming from an external source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2114
diff changeset
   725
            if not self.system_source in self._sourcesterms and \
2688
afa99d53401c [multi-source] test and fix msplanner regression
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2170
diff changeset
   726
                   not self.rqlst.defined_vars and \
afa99d53401c [multi-source] test and fix msplanner regression
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2170
diff changeset
   727
                   not need_source_access_relation(self.rqlst.vargraph):
2165
dd40c375044e #344177: ms planning bug with "Any X WHERE X eid 123" and 123 coming from an external source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2114
diff changeset
   728
                self._sourcesterms = {self.system_source: {}}
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   729
        elif not self.needsplit:
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   730
            if not allequals(self._sourcesterms.itervalues()):
2170
6adbc965c51b fix jpl/ticket/34862: msplanner bug with query generated to check if a user from ldap is owned_by himself
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2165
diff changeset
   731
                for source, terms in self._sourcesterms.iteritems():
6adbc965c51b fix jpl/ticket/34862: msplanner bug with query generated to check if a user from ldap is owned_by himself
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2165
diff changeset
   732
                    if source is self.system_source:
6adbc965c51b fix jpl/ticket/34862: msplanner bug with query generated to check if a user from ldap is owned_by himself
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2165
diff changeset
   733
                        continue
2114
862f27d94af4 one more ms planning bug
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 2110
diff changeset
   734
                    if any(x for x in terms if not isinstance(x, Constant)):
862f27d94af4 one more ms planning bug
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 2110
diff changeset
   735
                        self.needsplit = True
862f27d94af4 one more ms planning bug
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 2110
diff changeset
   736
                        return
862f27d94af4 one more ms planning bug
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 2110
diff changeset
   737
                self._sourcesterms = {self.system_source: {}}
862f27d94af4 one more ms planning bug
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 2110
diff changeset
   738
                self.needsplit = False
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   739
            else:
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   740
                sample = self._sourcesterms.itervalues().next()
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   741
                if len(sample) > 1:
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   742
                    for term in sample:
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   743
                        # need split if unlinked variable
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   744
                        if isinstance(term, Variable) and not term in self._linkedterms:
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   745
                            self.needsplit = True
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   746
                            break
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   747
                    else:
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   748
                        # need split if there are some cross relation on non
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   749
                        # invariant variable or if the variable is used in
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   750
                        # multi-sources relation
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   751
                        if self._crossrelations:
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   752
                            for reldict in self._crossrelations.itervalues():
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   753
                                for rel, terms in reldict.iteritems():
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   754
                                    for term in terms:
1230
232e16835fff fix constant handling in sourcesterms + minor cleanup
sylvain.thenault@logilab.fr
parents: 1229
diff changeset
   755
                                        if isinstance(term, Variable) \
232e16835fff fix constant handling in sourcesterms + minor cleanup
sylvain.thenault@logilab.fr
parents: 1229
diff changeset
   756
                                               and self._need_ext_source_access(term, rel):
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   757
                                            self.needsplit = True
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   758
                                            return
6673
5eb2f1f2c277 [ms planning] test and fix bad plannification of a query used on cascading delete of a deleted (external) entity relations
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6654
diff changeset
   759
        else:
5eb2f1f2c277 [ms planning] test and fix bad plannification of a query used on cascading delete of a deleted (external) entity relations
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6654
diff changeset
   760
            # remove sources only accessing to constant nodes
5eb2f1f2c277 [ms planning] test and fix bad plannification of a query used on cascading delete of a deleted (external) entity relations
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6654
diff changeset
   761
            for source, terms in self._sourcesterms.items():
5eb2f1f2c277 [ms planning] test and fix bad plannification of a query used on cascading delete of a deleted (external) entity relations
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6654
diff changeset
   762
                if source is self.system_source:
5eb2f1f2c277 [ms planning] test and fix bad plannification of a query used on cascading delete of a deleted (external) entity relations
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6654
diff changeset
   763
                    continue
5eb2f1f2c277 [ms planning] test and fix bad plannification of a query used on cascading delete of a deleted (external) entity relations
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6654
diff changeset
   764
                if not any(x for x in terms if not isinstance(x, Constant)):
5eb2f1f2c277 [ms planning] test and fix bad plannification of a query used on cascading delete of a deleted (external) entity relations
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6654
diff changeset
   765
                    del self._sourcesterms[source]
5eb2f1f2c277 [ms planning] test and fix bad plannification of a query used on cascading delete of a deleted (external) entity relations
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6654
diff changeset
   766
            if len(self._sourcesterms) < 2:
5eb2f1f2c277 [ms planning] test and fix bad plannification of a query used on cascading delete of a deleted (external) entity relations
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6654
diff changeset
   767
                self.needsplit = False
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   768
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   769
    @cached
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   770
    def _need_ext_source_access(self, var, rel):
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   771
        if not var._q_invariant:
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   772
            return True
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   773
        if  any(r for x, r in self._linkedterms[var]
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   774
                if not r is rel and self._repo.is_multi_sources_relation(r.r_type)):
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   775
            return True
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   776
        return False
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
   777
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   778
    def _set_source_for_term(self, source, term):
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   779
        self._sourcesterms.setdefault(source, {})[term] = set(self._solindices)
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   780
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   781
    def _term_sources(self, term):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   782
        """returns possible sources for terms `term`"""
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   783
        if isinstance(term, Constant):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   784
            source = self._session.source_from_eid(term.eval(self.plan.args))
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   785
            return set((source, solindex) for solindex in self._solindices)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   786
        else:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   787
            var = getattr(term, 'variable', term)
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   788
            sources = [source for source, varobjs in self.sourcesterms.iteritems()
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   789
                       if var in varobjs]
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   790
            return set((source, solindex) for source in sources
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   791
                       for solindex in self.sourcesterms[source][var])
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   792
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   793
    def _remove_sources(self, term, sources):
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   794
        """removes invalid sources (`sources`) from `sourcesterms`
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   795
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   796
        :param sources: the list of sources to remove
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   797
        :param term: the analyzed term
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   798
        """
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   799
        sourcesterms = self._sourcesterms
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   800
        for source, solindex in sources:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   801
            try:
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   802
                sourcesterms[source][term].remove(solindex)
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   803
            except KeyError:
1786
eccd1885d42e one more msplanning bug fixed...
sylvain.thenault@logilab.fr
parents: 1785
diff changeset
   804
                import rql.base as rqlb
3624
05932c6f7db2 fix buggy assertion
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3239
diff changeset
   805
                assert isinstance(term, (rqlb.BaseNode, Variable)), repr(term)
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
   806
                continue # may occur with subquery column alias
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   807
            if not sourcesterms[source][term]:
6427
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   808
                self._remove_source_term(source, term)
c8a5ac2d1eaa [schema / sources] store data sources as cubicweb entities
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6131
diff changeset
   809
6633
7baea108d326 [ms] simplify cw_source handling : don't try evil optmization that may hurt in some cases
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6598
diff changeset
   810
    def _remove_source_term(self, source, term):
7baea108d326 [ms] simplify cw_source handling : don't try evil optmization that may hurt in some cases
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6598
diff changeset
   811
        try:
7baea108d326 [ms] simplify cw_source handling : don't try evil optmization that may hurt in some cases
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6598
diff changeset
   812
            poped = self._sourcesterms[source].pop(term, None)
7baea108d326 [ms] simplify cw_source handling : don't try evil optmization that may hurt in some cases
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6598
diff changeset
   813
        except KeyError:
7baea108d326 [ms] simplify cw_source handling : don't try evil optmization that may hurt in some cases
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6598
diff changeset
   814
            pass
7baea108d326 [ms] simplify cw_source handling : don't try evil optmization that may hurt in some cases
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6598
diff changeset
   815
        else:
7baea108d326 [ms] simplify cw_source handling : don't try evil optmization that may hurt in some cases
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6598
diff changeset
   816
            if not self._sourcesterms[source]:
7baea108d326 [ms] simplify cw_source handling : don't try evil optmization that may hurt in some cases
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6598
diff changeset
   817
                del self._sourcesterms[source]
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   818
393
45a675515427 cross_relations support
sylvain.thenault@logilab.fr
parents: 392
diff changeset
   819
    def crossed_relation(self, source, relation):
45a675515427 cross_relations support
sylvain.thenault@logilab.fr
parents: 392
diff changeset
   820
        return relation in self._crossrelations.get(source, ())
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
   821
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   822
    def part_steps(self):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   823
        """precompute necessary part steps before generating actual rql for
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   824
        each step. This is necessary to know if an aggregate step will be
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   825
        necessary or not.
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   826
        """
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   827
        steps = []
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   828
        select = self.rqlst
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   829
        rschema = self._schema.rschema
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   830
        for source in self.part_sources:
1231
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   831
            try:
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   832
                sourceterms = self._sourcesterms[source]
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   833
            except KeyError:
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   834
                continue # already proceed
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   835
            while sourceterms:
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   836
                # take a term randomly, and all terms supporting the
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   837
                # same solutions
6794
140d42b41b31 [multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6759
diff changeset
   838
                term, solindices = self._choose_term(source, sourceterms)
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   839
                if source.uri == 'system':
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   840
                    # ensure all variables are available for the latest step
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   841
                    # (missing one will be available from temporary tables
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   842
                    # of previous steps)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   843
                    scope = select
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   844
                    terms = scope.defined_vars.values() + scope.aliases.values()
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   845
                    sourceterms.clear()
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   846
                    sources = [source]
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   847
                else:
5582
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   848
                    scope = ms_scope(term)
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   849
                    # find which sources support the same term and solutions
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   850
                    sources = self._expand_sources(source, term, solindices)
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   851
                    # no try to get as much terms as possible
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   852
                    terms = self._expand_terms(term, sources, sourceterms,
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   853
                                               scope, solindices)
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   854
                    if len(terms) == 1 and isinstance(terms[0], Constant):
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   855
                        # we can't generate anything interesting with a single
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   856
                        # constant term (will generate an empty "Any" query),
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   857
                        # go to the next iteration directly!
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   858
                        continue
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   859
                    if not sourceterms:
4721
8f63691ccb7f pylint style fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
   860
                        try:
8f63691ccb7f pylint style fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
   861
                            del self._sourcesterms[source]
8f63691ccb7f pylint style fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
   862
                        except KeyError:
8f63691ccb7f pylint style fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
   863
                            # XXX already cleaned
8f63691ccb7f pylint style fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
   864
                            pass
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   865
                # set of terms which should be additionaly selected when
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   866
                # possible
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   867
                needsel = set()
6794
140d42b41b31 [multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6759
diff changeset
   868
                if not self._sourcesterms and scope is select:
1231
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   869
                    terms += scope.defined_vars.values() + scope.aliases.values()
6758
28b11ecf319b [ms] #1382452: incorrect results with multi-sources
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6673
diff changeset
   870
                    if isinstance(term, Relation) and len(sources) > 1:
28b11ecf319b [ms] #1382452: incorrect results with multi-sources
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6673
diff changeset
   871
                        variants = set()
28b11ecf319b [ms] #1382452: incorrect results with multi-sources
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6673
diff changeset
   872
                        partterms = [term]
28b11ecf319b [ms] #1382452: incorrect results with multi-sources
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6673
diff changeset
   873
                        for vref in term.get_nodes(VariableRef):
28b11ecf319b [ms] #1382452: incorrect results with multi-sources
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6673
diff changeset
   874
                            if not vref.variable._q_invariant:
28b11ecf319b [ms] #1382452: incorrect results with multi-sources
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6673
diff changeset
   875
                                variants.add(vref.name)
28b11ecf319b [ms] #1382452: incorrect results with multi-sources
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6673
diff changeset
   876
                        if len(variants) == 2:
28b11ecf319b [ms] #1382452: incorrect results with multi-sources
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6673
diff changeset
   877
                            # we need an extra-step to fetch relations from each source
28b11ecf319b [ms] #1382452: incorrect results with multi-sources
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6673
diff changeset
   878
                            # before a join with prefetched inputs
28b11ecf319b [ms] #1382452: incorrect results with multi-sources
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6673
diff changeset
   879
                            # (see test_crossed_relation_noeid_needattr in
28b11ecf319b [ms] #1382452: incorrect results with multi-sources
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6673
diff changeset
   880
                            #  unittest_msplanner / unittest_multisources)
28b11ecf319b [ms] #1382452: incorrect results with multi-sources
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6673
diff changeset
   881
                            lhs, rhs = term.get_variable_parts()
28b11ecf319b [ms] #1382452: incorrect results with multi-sources
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6673
diff changeset
   882
                            steps.append( (sources, [term, getattr(lhs, 'variable', lhs),
28b11ecf319b [ms] #1382452: incorrect results with multi-sources
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6673
diff changeset
   883
                                                     getattr(rhs, 'variable', rhs)],
6794
140d42b41b31 [multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6759
diff changeset
   884
                                           solindices, scope, variants, False) )
6758
28b11ecf319b [ms] #1382452: incorrect results with multi-sources
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6673
diff changeset
   885
                            sources = [self.system_source]
1231
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   886
                    final = True
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   887
                else:
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   888
                    # suppose this is a final step until the contrary is proven
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   889
                    final = scope is select
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   890
                    # add attribute variables and mark variables which should be
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   891
                    # additionaly selected when possible
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   892
                    for var in select.defined_vars.itervalues():
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   893
                        if not var in terms:
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   894
                            stinfo = var.stinfo
5004
4cc020ee70e2 le patch rql26 a été importé
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4956
diff changeset
   895
                            for ovar, rtype in stinfo.get('attrvars', ()):
1231
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   896
                                if ovar in terms:
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   897
                                    needsel.add(var.name)
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   898
                                    terms.append(var)
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   899
                                    break
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   900
                            else:
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   901
                                needsel.add(var.name)
1231
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   902
                                final = False
6649
29f8e5c35392 [ms planning] fix/enhance cw_source consideration in ms planning
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6633
diff changeset
   903
                    # check all relations are supported by the sources
1231
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   904
                    for rel in scope.iget_nodes(Relation):
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   905
                        if rel.is_types_restriction():
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   906
                            continue
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   907
                        # take care not overwriting the existing "source" identifier
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   908
                        for _source in sources:
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   909
                            if not _source.support_relation(rel.r_type) or (
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   910
                                self.crossed_relation(_source, rel) and not rel in terms):
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   911
                                for vref in rel.iget_nodes(VariableRef):
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   912
                                    needsel.add(vref.name)
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   913
                                final = False
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   914
                                break
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   915
                        else:
1231
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   916
                            if not scope is select:
6794
140d42b41b31 [multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6759
diff changeset
   917
                                self._exists_relation(rel, terms, needsel, source)
1231
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   918
                            # if relation is supported by all sources and some of
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   919
                            # its lhs/rhs variable isn't in "terms", and the
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   920
                            # other end *is* in "terms", mark it have to be
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   921
                            # selected
3689
deb13e88e037 follow yams 0.25 api changes to improve performance
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3624
diff changeset
   922
                            if source.uri != 'system' and not rschema(rel.r_type).final:
1231
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   923
                                lhs, rhs = rel.get_variable_parts()
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   924
                                try:
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   925
                                    lhsvar = lhs.variable
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   926
                                except AttributeError:
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   927
                                    lhsvar = lhs
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   928
                                try:
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   929
                                    rhsvar = rhs.variable
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   930
                                except AttributeError:
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
   931
                                    rhsvar = rhs
6131
087c5a168010 [ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6129
diff changeset
   932
                                try:
087c5a168010 [ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6129
diff changeset
   933
                                    if lhsvar in terms and not rhsvar in terms:
087c5a168010 [ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6129
diff changeset
   934
                                        needsel.add(lhsvar.name)
087c5a168010 [ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6129
diff changeset
   935
                                    elif rhsvar in terms and not lhsvar in terms:
087c5a168010 [ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6129
diff changeset
   936
                                        needsel.add(rhsvar.name)
087c5a168010 [ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6129
diff changeset
   937
                                except AttributeError:
087c5a168010 [ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6129
diff changeset
   938
                                    continue # not an attribute, no selection needed
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   939
                if final and source.uri != 'system':
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   940
                    # check rewritten constants
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   941
                    for vconsts in select.stinfo['rewritten'].itervalues():
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   942
                        const = vconsts[0]
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   943
                        eid = const.eval(self.plan.args)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   944
                        _source = self._session.source_from_eid(eid)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   945
                        if len(sources) > 1 or not _source in sources:
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   946
                            # if there is some rewriten constant used by a not
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   947
                            # neged relation while there are some source not
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   948
                            # supporting the associated entity, this step can't
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   949
                            # be final (unless the relation is explicitly in
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   950
                            # `terms`, eg cross relations)
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   951
                            for c in vconsts:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   952
                                rel = c.relation()
5582
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   953
                                if rel is None or not (rel in terms or neged_relation(rel)):
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   954
                                    final = False
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   955
                                    break
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   956
                            break
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   957
                if final:
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   958
                    self._cleanup_sourcesterms(sources, solindices)
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   959
                steps.append((sources, terms, solindices, scope, needsel, final)
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   960
                             )
6794
140d42b41b31 [multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6759
diff changeset
   961
        if not steps[-1][-1]:
140d42b41b31 [multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6759
diff changeset
   962
            # add a final step
140d42b41b31 [multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6759
diff changeset
   963
            terms = select.defined_vars.values() + select.aliases.values()
140d42b41b31 [multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6759
diff changeset
   964
            steps.append( ([self.system_source], terms, set(self._solindices),
140d42b41b31 [multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6759
diff changeset
   965
                           select, set(), True) )
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   966
        return steps
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   967
6794
140d42b41b31 [multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6759
diff changeset
   968
    def _exists_relation(self, rel, terms, needsel, source):
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
   969
        rschema = self._schema.rschema(rel.r_type)
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   970
        lhs, rhs = rel.get_variable_parts()
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   971
        try:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   972
            lhsvar, rhsvar = lhs.variable, rhs.variable
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   973
        except AttributeError:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   974
            pass
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   975
        else:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   976
            # supported relation with at least one end supported, check the
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   977
            # other end is in as well. If not this usually means the
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   978
            # variable is refed by an outer scope and should be substituted
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   979
            # using an 'identity' relation (else we'll get a conflict of
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   980
            # temporary tables)
6794
140d42b41b31 [multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6759
diff changeset
   981
            relscope = ms_scope(rel)
140d42b41b31 [multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6759
diff changeset
   982
            lhsscope = ms_scope(lhsvar)
140d42b41b31 [multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6759
diff changeset
   983
            rhsscope = ms_scope(rhsvar)
140d42b41b31 [multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6759
diff changeset
   984
            if rhsvar in terms and not lhsvar in terms and lhsscope is lhsvar.stmt:
140d42b41b31 [multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6759
diff changeset
   985
                self._identity_substitute(rel, lhsvar, terms, needsel, relscope)
140d42b41b31 [multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6759
diff changeset
   986
            elif lhsvar in terms and not rhsvar in terms and rhsscope is rhsvar.stmt:
140d42b41b31 [multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6759
diff changeset
   987
                self._identity_substitute(rel, rhsvar, terms, needsel, relscope)
140d42b41b31 [multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6759
diff changeset
   988
            elif self.crossed_relation(source, rel):
140d42b41b31 [multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6759
diff changeset
   989
                if lhsscope is not relscope:
140d42b41b31 [multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6759
diff changeset
   990
                    self._identity_substitute(rel, lhsvar, terms, needsel,
140d42b41b31 [multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6759
diff changeset
   991
                                              relscope, lhsscope)
140d42b41b31 [multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6759
diff changeset
   992
                if rhsscope is not relscope:
140d42b41b31 [multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6759
diff changeset
   993
                    self._identity_substitute(rel, rhsvar, terms, needsel,
140d42b41b31 [multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6759
diff changeset
   994
                                              relscope, rhsscope)
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   995
6794
140d42b41b31 [multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6759
diff changeset
   996
    def _identity_substitute(self, relation, var, terms, needsel, exist,
140d42b41b31 [multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6759
diff changeset
   997
                             idrelscope=None):
140d42b41b31 [multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6759
diff changeset
   998
        newvar = self._insert_identity_variable(exist, var, idrelscope)
5004
4cc020ee70e2 le patch rql26 a été importé
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4956
diff changeset
   999
        # ensure relation is using '=' operator, else we rely on a
4cc020ee70e2 le patch rql26 a été importé
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4956
diff changeset
  1000
        # sqlgenerator side effect (it won't insert an inequality operator
4cc020ee70e2 le patch rql26 a été importé
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4956
diff changeset
  1001
        # in this case)
4cc020ee70e2 le patch rql26 a été importé
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4956
diff changeset
  1002
        relation.children[1].operator = '='
4cc020ee70e2 le patch rql26 a été importé
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4956
diff changeset
  1003
        terms.append(newvar)
4cc020ee70e2 le patch rql26 a été importé
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4956
diff changeset
  1004
        needsel.add(newvar.name)
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
  1005
6794
140d42b41b31 [multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6759
diff changeset
  1006
    def _choose_term(self, source, sourceterms):
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1007
        """pick one term among terms supported by a source, which will be used
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1008
        as a base to generate an execution step
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1009
        """
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1010
        secondchoice = None
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1011
        if len(self._sourcesterms) > 1:
6794
140d42b41b31 [multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6759
diff changeset
  1012
            # first, return non invariant variable of crossed relation, then the
140d42b41b31 [multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6759
diff changeset
  1013
            # crossed relation itself
140d42b41b31 [multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6759
diff changeset
  1014
            for term in sourceterms:
140d42b41b31 [multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6759
diff changeset
  1015
                if (isinstance(term, Relation)
140d42b41b31 [multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6759
diff changeset
  1016
                    and self.crossed_relation(source, term)
140d42b41b31 [multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6759
diff changeset
  1017
                    and not ms_scope(term) is self.rqlst):
140d42b41b31 [multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6759
diff changeset
  1018
                    for vref in term.get_variable_parts():
140d42b41b31 [multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6759
diff changeset
  1019
                        try:
140d42b41b31 [multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6759
diff changeset
  1020
                            var = vref.variable
140d42b41b31 [multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6759
diff changeset
  1021
                        except AttributeError:
140d42b41b31 [multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6759
diff changeset
  1022
                            # Constant
140d42b41b31 [multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6759
diff changeset
  1023
                            continue
140d42b41b31 [multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6759
diff changeset
  1024
                        if ((len(var.stinfo['relations']) > 1 or var.stinfo['selected'])
140d42b41b31 [multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6759
diff changeset
  1025
                            and var in sourceterms):
140d42b41b31 [multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6759
diff changeset
  1026
                            return var, sourceterms.pop(var)
140d42b41b31 [multi-sources] fix planning of crossed relation w/ some complex queries
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6759
diff changeset
  1027
                    return term, sourceterms.pop(term)
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1028
            # priority to variable from subscopes
1231
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1029
            for term in sourceterms:
5582
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
  1030
                if not ms_scope(term) is self.rqlst:
1231
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1031
                    if isinstance(term, Variable):
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1032
                        return term, sourceterms.pop(term)
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1033
                    secondchoice = term
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1034
        else:
1231
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1035
            # priority to variable from outer scope
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1036
            for term in sourceterms:
5582
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
  1037
                if ms_scope(term) is self.rqlst:
1231
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1038
                    if isinstance(term, Variable):
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1039
                        return term, sourceterms.pop(term)
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1040
                    secondchoice = term
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1041
        if secondchoice is not None:
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1042
            return secondchoice, sourceterms.pop(secondchoice)
1231
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1043
        # priority to variable with the less solutions supported and with the
1237
c836bdb3b17b more test predictability
Sylvain <syt@logilab.fr>
parents: 1231
diff changeset
  1044
        # most valuable refs. Add variable name for test predictability
1231
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1045
        variables = sorted([(var, sols) for (var, sols) in sourceterms.items()
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1046
                            if isinstance(var, Variable)],
1237
c836bdb3b17b more test predictability
Sylvain <syt@logilab.fr>
parents: 1231
diff changeset
  1047
                           key=lambda (v, s): (len(s), -v.valuable_references(), v.name))
1231
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1048
        if variables:
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1049
            var = variables[0][0]
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1050
            return var, sourceterms.pop(var)
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1051
        # priority to constant
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1052
        for term in sourceterms:
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1053
            if isinstance(term, Constant):
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1054
                return term, sourceterms.pop(term)
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1055
        # whatever (relation)
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1056
        term = iter(sourceterms).next()
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1057
        return term, sourceterms.pop(term)
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
  1058
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1059
    def _expand_sources(self, selected_source, term, solindices):
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1060
        """return all sources supporting given term / solindices"""
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1061
        sources = [selected_source]
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1062
        sourcesterms = self._sourcesterms
1231
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1063
        for source in sourcesterms.keys():
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1064
            if source is selected_source:
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1065
                continue
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
  1066
            if not (term in sourcesterms[source] and
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1067
                    solindices.issubset(sourcesterms[source][term])):
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1068
                continue
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1069
            sources.append(source)
1231
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1070
            if source.uri != 'system' or not (isinstance(term, Variable) and not term in self._linkedterms):
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1071
                termsolindices = sourcesterms[source][term]
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1072
                termsolindices -= solindices
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1073
                if not termsolindices:
1231
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1074
                    del sourcesterms[source][term]
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1075
                    if not sourcesterms[source]:
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1076
                        del sourcesterms[source]
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1077
        return sources
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
  1078
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1079
    def _expand_terms(self, term, sources, sourceterms, scope, solindices):
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1080
        terms = [term]
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1081
        sources = sorted(sources)
1231
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1082
        sourcesterms = self._sourcesterms
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1083
        linkedterms = self._linkedterms
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1084
        # term has to belong to the same scope if there is more
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1085
        # than the system source remaining
1231
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1086
        if len(sourcesterms) > 1 and not scope is self.rqlst:
5582
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
  1087
            candidates = (t for t in sourceterms.keys() if scope is ms_scope(t))
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1088
        else:
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1089
            candidates = sourceterms #.iterkeys()
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1090
        # we only want one unlinked term in each generated query
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1091
        candidates = [t for t in candidates
1231
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1092
                      if isinstance(t, (Constant, Relation)) or
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1093
                      (solindices.issubset(sourceterms[t]) and t in linkedterms)]
1231
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1094
        cross_rels = {}
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1095
        for source in sources:
1231
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1096
            cross_rels.update(self._crossrelations.get(source, {}))
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1097
        exclude = {}
4719
aaed3f813ef8 kill dead/useless code as suggested by pylint
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4716
diff changeset
  1098
        for crossvars in cross_rels.itervalues():
1231
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1099
            vars = [t for t in crossvars if isinstance(t, Variable)]
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1100
            try:
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1101
                exclude[vars[0]] = vars[1]
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1102
                exclude[vars[1]] = vars[0]
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1103
            except IndexError:
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1104
                pass
6131
087c5a168010 [ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6129
diff changeset
  1105
        accept_term = lambda x: (not any(s for s in sources
087c5a168010 [ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6129
diff changeset
  1106
                                         if not x in sourcesterms.get(s, ()))
087c5a168010 [ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6129
diff changeset
  1107
                                 and x._ms_may_be_processed(terms, linkedterms)
1231
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1108
                                 and not exclude.get(x) in terms)
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1109
        if isinstance(term, Relation) and term in cross_rels:
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1110
            cross_terms = cross_rels.pop(term)
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1111
            base_accept_term = accept_term
6598
78eaccfbd2b7 [ms] fix an infinite loop in ms planner and another crash
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6427
diff changeset
  1112
            accept_term = lambda x: (base_accept_term(x) or x in cross_terms)
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1113
            for refed in cross_terms:
393
45a675515427 cross_relations support
sylvain.thenault@logilab.fr
parents: 392
diff changeset
  1114
                if not refed in candidates:
1231
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1115
                    terms.append(refed)
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1116
        # repeat until no term can't be added, since addition of a new
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1117
        # term may permit to another one to be added
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1118
        modified = True
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1119
        while modified and candidates:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1120
            modified = False
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1121
            for term in candidates[:]:
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1122
                if isinstance(term, Constant):
6131
087c5a168010 [ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6129
diff changeset
  1123
                    termsources = set(x[0] for x in self._term_sources(term))
087c5a168010 [ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6129
diff changeset
  1124
                    # ensure system source is there for constant
087c5a168010 [ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6129
diff changeset
  1125
                    if self.system_source in sources:
087c5a168010 [ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6129
diff changeset
  1126
                        termsources.add(self.system_source)
087c5a168010 [ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6129
diff changeset
  1127
                    if sorted(termsources) != sources:
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1128
                        continue
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1129
                    terms.append(term)
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1130
                    candidates.remove(term)
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1131
                    modified = True
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1132
                    del sourceterms[term]
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1133
                elif accept_term(term):
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1134
                    terms.append(term)
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1135
                    candidates.remove(term)
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1136
                    modified = True
1231
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1137
                    self._cleanup_sourcesterms(sources, solindices, term)
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1138
        return terms
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
  1139
1231
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1140
    def _cleanup_sourcesterms(self, sources, solindices, term=None):
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1141
        """remove solutions so we know they are already processed"""
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1142
        for source in sources:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1143
            try:
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1144
                sourceterms = self._sourcesterms[source]
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1145
            except KeyError:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1146
                continue
1231
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1147
            if term is None:
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1148
                for term, termsolindices in sourceterms.items():
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1149
                    if isinstance(term, Relation) and self.crossed_relation(source, term):
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1150
                        continue
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1151
                    termsolindices -= solindices
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1152
                    if not termsolindices:
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1153
                        del sourceterms[term]
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1154
            else:
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1155
                try:
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1156
                    sourceterms[term] -= solindices
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1157
                    if not sourceterms[term]:
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1158
                        del sourceterms[term]
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1159
                except KeyError:
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1160
                    pass
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1161
                    #assert term in cross_terms
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1162
            if not sourceterms:
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1163
                del self._sourcesterms[source]
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
  1164
6654
18d159a2d1ba [ms planner] fix changeset 6650:72f2fd93a622: we should sometime complete input map anyway
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6651
diff changeset
  1165
    def merge_input_maps(self, allsolindices, complete=True):
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1166
        """inputmaps is a dictionary with tuple of solution indices as key with
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1167
        an associated input map as value. This function compute for each
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1168
        solution its necessary input map and return them grouped
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1169
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1170
        ex:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1171
        inputmaps = {(0, 1, 2): {'A': 't1.login1', 'U': 't1.C0', 'U.login': 't1.login1'},
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1172
                     (1,): {'X': 't2.C0', 'T': 't2.C1'}}
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1173
        return : [([1],  {'A': 't1.login1', 'U': 't1.C0', 'U.login': 't1.login1',
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
  1174
                           'X': 't2.C0', 'T': 't2.C1'}),
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1175
                  ([0,2], {'A': 't1.login1', 'U': 't1.C0', 'U.login': 't1.login1'})]
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1176
        """
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1177
        if not self._inputmaps:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1178
            return [(allsolindices, None)]
6650
72f2fd93a622 [ms planning] fix bug in merge_input_map causing bad plans computed
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6649
diff changeset
  1179
        _allsolindices = allsolindices.copy()
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1180
        mapbysol = {}
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1181
        # compute a single map for each solution
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1182
        for solindices, basemap in self._inputmaps.iteritems():
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1183
            for solindex in solindices:
6654
18d159a2d1ba [ms planner] fix changeset 6650:72f2fd93a622: we should sometime complete input map anyway
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6651
diff changeset
  1184
                if not (complete or solindex in allsolindices):
6650
72f2fd93a622 [ms planning] fix bug in merge_input_map causing bad plans computed
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6649
diff changeset
  1185
                    continue
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1186
                solmap = mapbysol.setdefault(solindex, {})
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1187
                solmap.update(basemap)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1188
                try:
6650
72f2fd93a622 [ms planning] fix bug in merge_input_map causing bad plans computed
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6649
diff changeset
  1189
                    _allsolindices.remove(solindex)
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1190
                except KeyError:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1191
                    continue # already removed
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1192
        # group results by identical input map
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1193
        result = []
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1194
        for solindex, solmap in mapbysol.iteritems():
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1195
            for solindices, commonmap in result:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1196
                if commonmap == solmap:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1197
                    solindices.append(solindex)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1198
                    break
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1199
            else:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1200
                result.append( ([solindex], solmap) )
6650
72f2fd93a622 [ms planning] fix bug in merge_input_map causing bad plans computed
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6649
diff changeset
  1201
        if _allsolindices:
72f2fd93a622 [ms planning] fix bug in merge_input_map causing bad plans computed
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6649
diff changeset
  1202
            result.append( (list(_allsolindices), None) )
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1203
        return result
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1204
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1205
    def build_final_part(self, select, solindices, inputmap,  sources,
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1206
                         insertedvars):
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1207
        solutions = [self._solutions[i] for i in solindices]
6598
78eaccfbd2b7 [ms] fix an infinite loop in ms planner and another crash
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6427
diff changeset
  1208
        if self._conflicts and inputmap:
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1209
            for varname, mappedto in self._conflicts:
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1210
                var = select.defined_vars[varname]
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1211
                newvar = select.make_variable()
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1212
                # XXX should use var.scope but scope hasn't been computed yet
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1213
                select.add_relation(var, 'identity', newvar)
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1214
                for sol in solutions:
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1215
                    sol[newvar.name] = sol[varname]
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1216
                inputmap[newvar.name] = mappedto
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1217
        rqlst = self.plan.finalize(select, solutions, insertedvars)
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1218
        if self.temptable is None and self.finaltable is None:
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1219
            return OneFetchStep(self.plan, rqlst, sources, inputmap=inputmap)
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1220
        table = self.temptable or self.finaltable
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1221
        return FetchStep(self.plan, rqlst, sources, table, True, inputmap)
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1222
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1223
    def build_non_final_part(self, select, solindices, sources, insertedvars,
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1224
                             table):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1225
        """non final step, will have to store results in a temporary table"""
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1226
        solutions = [self._solutions[i] for i in solindices]
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1227
        rqlst = self.plan.finalize(select, solutions, insertedvars)
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1228
        step = FetchStep(self.plan, rqlst, sources, table, False)
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1229
        # update input map for following steps, according to processed solutions
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1230
        inputmapkey = tuple(sorted(solindices))
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1231
        inputmap = self._inputmaps.setdefault(inputmapkey, {})
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1232
        for varname, mapping in step.outputmap.iteritems():
6651
4a0283b3ce0c [ms planning] fix potential key error due to additional entries in input maps
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6650
diff changeset
  1233
            if varname in inputmap and not '.' in varname and  \
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1234
                   not (mapping == inputmap[varname] or
3689
deb13e88e037 follow yams 0.25 api changes to improve performance
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3624
diff changeset
  1235
                        self._schema.eschema(solutions[0][varname]).final):
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1236
                self._conflicts.append((varname, inputmap[varname]))
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1237
        inputmap.update(step.outputmap)
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1238
        self.plan.add_step(step)
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1239
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1240
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1241
class MSPlanner(SSPlanner):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1242
    """MultiSourcesPlanner: build execution plan for rql queries
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1243
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1244
    decompose the RQL query according to sources'schema
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1245
    """
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
  1246
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1247
    def build_select_plan(self, plan, rqlst):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1248
        """build execution plan for a SELECT RQL query
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
  1249
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1250
        the rqlst should not be tagged at this point
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1251
        """
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1252
        # preprocess deals with security insertion and returns a new syntax tree
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1253
        # which have to be executed to fulfill the query: according
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1254
        # to permissions for variable's type, different rql queries may have to
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1255
        # be executed
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1256
        plan.preprocess(rqlst)
6027
f91b72def783 [ms] log after preprocessing, else we don't see the query which is actually planned (may have been modified by security insertion)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5989
diff changeset
  1257
        if server.DEBUG & server.DBG_MS:
f91b72def783 [ms] log after preprocessing, else we don't see the query which is actually planned (may have been modified by security insertion)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5989
diff changeset
  1258
            print '-'*80
f91b72def783 [ms] log after preprocessing, else we don't see the query which is actually planned (may have been modified by security insertion)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5989
diff changeset
  1259
            print 'PLANNING', rqlst
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1260
        ppis = [PartPlanInformation(plan, select, self.rqlhelper)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1261
                for select in rqlst.children]
4722
9c13d5db03d9 pylint suggested refactorings
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4721
diff changeset
  1262
        steps = self._union_plan(plan, ppis)
2766
07e47b84acc2 DBG_MS flag
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2688
diff changeset
  1263
        if server.DEBUG & server.DBG_MS:
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1264
            from pprint import pprint
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1265
            for step in plan.steps:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1266
                pprint(step.test_repr())
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1267
            pprint(steps[0].test_repr())
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1268
        return steps
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1269
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1270
    def _ppi_subqueries(self, ppi):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1271
        # part plan info for subqueries
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1272
        plan = ppi.plan
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1273
        inputmap = {}
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1274
        for subquery in ppi.rqlst.with_[:]:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1275
            sppis = [PartPlanInformation(plan, select)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1276
                     for select in subquery.query.children]
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1277
            for sppi in sppis:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1278
                if sppi.needsplit or sppi.part_sources != ppi.part_sources:
5168
1ab032df5ca3 SQL Server port: temporary table handling
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 4956
diff changeset
  1279
                    temptable = plan.make_temp_table_name('T%s' % make_uid(id(subquery)))
4722
9c13d5db03d9 pylint suggested refactorings
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4721
diff changeset
  1280
                    sstep = self._union_plan(plan, sppis, temptable)[0]
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1281
                    break
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1282
            else:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1283
                sstep = None
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1284
            if sstep is not None:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1285
                ppi.rqlst.with_.remove(subquery)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1286
                for i, colalias in enumerate(subquery.aliases):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1287
                    inputmap[colalias.name] = '%s.C%s' % (temptable, i)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1288
                ppi.plan.add_step(sstep)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1289
        return inputmap
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
  1290
4722
9c13d5db03d9 pylint suggested refactorings
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4721
diff changeset
  1291
    def _union_plan(self, plan, ppis, temptable=None):
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1292
        tosplit, cango, allsources = [], {}, set()
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1293
        for planinfo in ppis:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1294
            if planinfo.needsplit:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1295
                tosplit.append(planinfo)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1296
            else:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1297
                cango.setdefault(planinfo.part_sources, []).append(planinfo)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1298
            for source in planinfo.part_sources:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1299
                allsources.add(source)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1300
        # first add steps for query parts which doesn't need to splitted
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1301
        steps = []
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1302
        for sources, cppis in cango.iteritems():
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1303
            byinputmap = {}
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1304
            for ppi in cppis:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1305
                select = ppi.rqlst
1231
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1306
                if sources != (ppi.system_source,):
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1307
                    add_types_restriction(self.schema, select)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1308
                # part plan info for subqueries
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1309
                inputmap = self._ppi_subqueries(ppi)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1310
                aggrstep = need_aggr_step(select, sources)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1311
                if aggrstep:
5168
1ab032df5ca3 SQL Server port: temporary table handling
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 4956
diff changeset
  1312
                    atemptable = plan.make_temp_table_name('T%s' % make_uid(id(select)))
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1313
                    sunion = Union()
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1314
                    sunion.append(select)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1315
                    selected = select.selection[:]
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1316
                    select_group_sort(select)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1317
                    step = AggrStep(plan, selected, select, atemptable, temptable)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1318
                    step.set_limit_offset(select.limit, select.offset)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1319
                    select.limit = None
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1320
                    select.offset = 0
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1321
                    fstep = FetchStep(plan, sunion, sources, atemptable, True, inputmap)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1322
                    step.children.append(fstep)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1323
                    steps.append(step)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1324
                else:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1325
                    byinputmap.setdefault(tuple(inputmap.iteritems()), []).append( (select) )
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1326
            for inputmap, queries in byinputmap.iteritems():
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1327
                inputmap = dict(inputmap)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1328
                sunion = Union()
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1329
                for select in queries:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1330
                    sunion.append(select)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1331
                if temptable:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1332
                    steps.append(FetchStep(plan, sunion, sources, temptable, True, inputmap))
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1333
                else:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1334
                    steps.append(OneFetchStep(plan, sunion, sources, inputmap))
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1335
        # then add steps for splitted query parts
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1336
        for planinfo in tosplit:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1337
            steps.append(self.split_part(planinfo, temptable))
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1338
        if len(steps) > 1:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1339
            if temptable:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1340
                step = UnionFetchStep(plan)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1341
            else:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1342
                step = UnionStep(plan)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1343
            step.children = steps
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1344
            return (step,)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1345
        return steps
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1346
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1347
    # internal methods for multisources decomposition #########################
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
  1348
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1349
    def split_part(self, ppi, temptable):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1350
        ppi.finaltable = temptable
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1351
        plan = ppi.plan
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1352
        select = ppi.rqlst
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1353
        subinputmap = self._ppi_subqueries(ppi)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1354
        stepdefs = ppi.part_steps()
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1355
        if need_aggr_step(select, ppi.part_sources, stepdefs):
5168
1ab032df5ca3 SQL Server port: temporary table handling
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 4956
diff changeset
  1356
            atemptable = plan.make_temp_table_name('T%s' % make_uid(id(select)))
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1357
            selection = select.selection[:]
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1358
            select_group_sort(select)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1359
        else:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1360
            atemptable = None
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1361
            selection = select.selection
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1362
        ppi.temptable = atemptable
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1363
        vfilter = TermsFiltererVisitor(self.schema, ppi)
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1364
        steps = []
6654
18d159a2d1ba [ms planner] fix changeset 6650:72f2fd93a622: we should sometime complete input map anyway
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6651
diff changeset
  1365
        multifinal = len([x for x in stepdefs if x[-1]]) >= 2
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1366
        for sources, terms, solindices, scope, needsel, final in stepdefs:
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1367
            # extract an executable query using only the specified terms
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1368
            if sources[0].uri == 'system':
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1369
                # in this case we have to merge input maps before call to
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1370
                # filter so already processed restriction are correctly
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1371
                # removed
6759
5d016d5bacca [ms] fix ms planner crash on some ambiguous queries, as demonstrated in a comment on http://www.cubicweb.org/ticket/1382452
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6758
diff changeset
  1372
                solsinputmaps = ppi.merge_input_maps(
5d016d5bacca [ms] fix ms planner crash on some ambiguous queries, as demonstrated in a comment on http://www.cubicweb.org/ticket/1382452
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6758
diff changeset
  1373
                    solindices, complete=not (final and multifinal))
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1374
                for solindices, inputmap in solsinputmaps:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1375
                    minrqlst, insertedvars = vfilter.filter(
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1376
                        sources, terms, scope, set(solindices), needsel, final)
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1377
                    if inputmap is None:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1378
                        inputmap = subinputmap
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1379
                    else:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1380
                        inputmap.update(subinputmap)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1381
                    steps.append(ppi.build_final_part(minrqlst, solindices, inputmap,
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1382
                                                      sources, insertedvars))
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1383
            else:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1384
                # this is a final part (i.e. retreiving results for the
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1385
                # original query part) if all term / sources have been
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1386
                # treated or if this is the last shot for used solutions
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1387
                minrqlst, insertedvars = vfilter.filter(
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1388
                    sources, terms, scope, solindices, needsel, final)
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1389
                if final:
6759
5d016d5bacca [ms] fix ms planner crash on some ambiguous queries, as demonstrated in a comment on http://www.cubicweb.org/ticket/1382452
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6758
diff changeset
  1390
                    solsinputmaps = ppi.merge_input_maps(
5d016d5bacca [ms] fix ms planner crash on some ambiguous queries, as demonstrated in a comment on http://www.cubicweb.org/ticket/1382452
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6758
diff changeset
  1391
                        solindices, complete=not (final and multifinal))
6129
fea746b60093 [ms] fix two planner bugs: one occuring query such as X created_by U where X in a external source and U may come from an ldap source. The other being that when we've to merge input maps, we were modifying the same tree/solutions while a copy were needed. Also, ensure we add type restrictions, necessary for pyro source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6128
diff changeset
  1392
                    if len(solsinputmaps) > 1:
fea746b60093 [ms] fix two planner bugs: one occuring query such as X created_by U where X in a external source and U may come from an ldap source. The other being that when we've to merge input maps, we were modifying the same tree/solutions while a copy were needed. Also, ensure we add type restrictions, necessary for pyro source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6128
diff changeset
  1393
                        refrqlst = minrqlst
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1394
                    for solindices, inputmap in solsinputmaps:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1395
                        if inputmap is None:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1396
                            inputmap = subinputmap
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1397
                        else:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1398
                            inputmap.update(subinputmap)
6129
fea746b60093 [ms] fix two planner bugs: one occuring query such as X created_by U where X in a external source and U may come from an ldap source. The other being that when we've to merge input maps, we were modifying the same tree/solutions while a copy were needed. Also, ensure we add type restrictions, necessary for pyro source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6128
diff changeset
  1399
                        if len(solsinputmaps) > 1:
fea746b60093 [ms] fix two planner bugs: one occuring query such as X created_by U where X in a external source and U may come from an ldap source. The other being that when we've to merge input maps, we were modifying the same tree/solutions while a copy were needed. Also, ensure we add type restrictions, necessary for pyro source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6128
diff changeset
  1400
                            minrqlst = refrqlst.copy()
fea746b60093 [ms] fix two planner bugs: one occuring query such as X created_by U where X in a external source and U may come from an ldap source. The other being that when we've to merge input maps, we were modifying the same tree/solutions while a copy were needed. Also, ensure we add type restrictions, necessary for pyro source
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6128
diff changeset
  1401
                            sources = sources[:]
1231
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1402
                        if inputmap and len(sources) > 1:
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1403
                            sources.remove(ppi.system_source)
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1404
                            steps.append(ppi.build_final_part(minrqlst, solindices, None,
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1405
                                                              sources, insertedvars))
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1406
                            steps.append(ppi.build_final_part(minrqlst, solindices, inputmap,
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1407
                                                              [ppi.system_source], insertedvars))
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1408
                        else:
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1409
                            steps.append(ppi.build_final_part(minrqlst, solindices, inputmap,
1457a545af03 more fix of cross relation handling
sylvain.thenault@logilab.fr
parents: 1230
diff changeset
  1410
                                                              sources, insertedvars))
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1411
                else:
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1412
                    table = '_T%s%s' % (''.join(sorted(v._ms_table_key() for v in terms)),
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1413
                                        ''.join(sorted(str(i) for i in solindices)))
5168
1ab032df5ca3 SQL Server port: temporary table handling
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 4956
diff changeset
  1414
                    table = plan.make_temp_table_name(table)
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1415
                    ppi.build_non_final_part(minrqlst, solindices, sources,
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1416
                                             insertedvars, table)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1417
        # finally: join parts, deal with aggregat/group/sorts if necessary
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1418
        if atemptable is not None:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1419
            step = AggrStep(plan, selection, select, atemptable, temptable)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1420
            step.children = steps
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1421
        elif len(steps) > 1:
5582
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
  1422
            getrschema = self.schema.rschema
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
  1423
            if need_intersect(select, getrschema) or any(need_intersect(select, getrschema)
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
  1424
                                                         for step in steps
3e133b29a1a4 [rql2sql] follow rql 0.26.1 changes: NOT nodes normalization, allowing simplification of sql generation, and fix #XXX
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
  1425
                                                         for select in step.union.children):
341
0a426be2f3a2 fixes for some cases of neged inlined relations, may have to use (new) IntersectStep. XXX: write IntersectFetchStep
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents: 257
diff changeset
  1426
                if temptable:
4716
55b6a3262071 fix some pylint detected errors
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4212
diff changeset
  1427
                    step = IntersectFetchStep(plan) # XXX not implemented
341
0a426be2f3a2 fixes for some cases of neged inlined relations, may have to use (new) IntersectStep. XXX: write IntersectFetchStep
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents: 257
diff changeset
  1428
                else:
0a426be2f3a2 fixes for some cases of neged inlined relations, may have to use (new) IntersectStep. XXX: write IntersectFetchStep
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents: 257
diff changeset
  1429
                    step = IntersectStep(plan)
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1430
            else:
341
0a426be2f3a2 fixes for some cases of neged inlined relations, may have to use (new) IntersectStep. XXX: write IntersectFetchStep
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents: 257
diff changeset
  1431
                if temptable:
0a426be2f3a2 fixes for some cases of neged inlined relations, may have to use (new) IntersectStep. XXX: write IntersectFetchStep
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents: 257
diff changeset
  1432
                    step = UnionFetchStep(plan)
0a426be2f3a2 fixes for some cases of neged inlined relations, may have to use (new) IntersectStep. XXX: write IntersectFetchStep
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents: 257
diff changeset
  1433
                else:
0a426be2f3a2 fixes for some cases of neged inlined relations, may have to use (new) IntersectStep. XXX: write IntersectFetchStep
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents: 257
diff changeset
  1434
                    step = UnionStep(plan)
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1435
            step.children = steps
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1436
        else:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1437
            step = steps[0]
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1438
        if select.limit is not None or select.offset:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1439
            step.set_limit_offset(select.limit, select.offset)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1440
        return step
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1441
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
  1442
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1443
class UnsupportedBranch(Exception):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1444
    pass
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1445
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1446
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1447
class TermsFiltererVisitor(object):
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1448
    def __init__(self, schema, ppi):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1449
        self.schema = schema
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1450
        self.ppi = ppi
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1451
        self.skip = {}
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1452
        self.hasaggrstep = self.ppi.temptable
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1453
        self.extneedsel = frozenset(vref.name for sortterm in ppi.rqlst.orderby
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1454
                                    for vref in sortterm.iget_nodes(VariableRef))
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
  1455
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1456
    def _rqlst_accept(self, rqlst, node, newroot, terms, setfunc=None):
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1457
        try:
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1458
            newrestr, node_ = node.accept(self, newroot, terms[:])
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1459
        except UnsupportedBranch:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1460
            return rqlst
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1461
        if setfunc is not None and newrestr is not None:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1462
            setfunc(newrestr)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1463
        if not node_ is node:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1464
            rqlst = node.parent
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1465
        return rqlst
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1466
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1467
    def filter(self, sources, terms, rqlst, solindices, needsel, final):
2766
07e47b84acc2 DBG_MS flag
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2688
diff changeset
  1468
        if server.DEBUG & server.DBG_MS:
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1469
            print 'filter', final and 'final' or '', sources, terms, rqlst, solindices, needsel
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1470
        newroot = Select()
393
45a675515427 cross_relations support
sylvain.thenault@logilab.fr
parents: 392
diff changeset
  1471
        self.sources = sorted(sources)
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1472
        self.terms = terms
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1473
        self.solindices = solindices
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1474
        self.final = final
2109
f469368df530 when a relation isn't supported by a ms query parts, dereference introduced variable refs
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2108
diff changeset
  1475
        self._pending_vrefs = []
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1476
        # terms which appear in unsupported branches
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1477
        needsel |= self.extneedsel
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1478
        self.needsel = needsel
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1479
        # terms which appear in supported branches
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1480
        self.mayneedsel = set()
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1481
        # new inserted variables
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1482
        self.insertedvars = []
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1483
        # other structures (XXX document)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1484
        self.mayneedvar, self.hasvar = {}, {}
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1485
        self.use_only_defined = False
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1486
        self.scopes = {rqlst: newroot}
2075
933ccfce6e91 fix bug in multisource planner, w/ inlined relation
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1977
diff changeset
  1487
        self.current_scope = rqlst
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1488
        if rqlst.where:
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1489
            rqlst = self._rqlst_accept(rqlst, rqlst.where, newroot, terms,
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1490
                                       newroot.set_where)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1491
        if isinstance(rqlst, Select):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1492
            self.use_only_defined = True
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1493
            if rqlst.groupby:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1494
                groupby = []
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1495
                for node in rqlst.groupby:
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1496
                    rqlst = self._rqlst_accept(rqlst, node, newroot, terms,
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1497
                                               groupby.append)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1498
                if groupby:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1499
                    newroot.set_groupby(groupby)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1500
            if rqlst.having:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1501
                having = []
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1502
                for node in rqlst.having:
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1503
                    rqlst = self._rqlst_accept(rqlst, node, newroot, terms,
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1504
                                               having.append)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1505
                if having:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1506
                    newroot.set_having(having)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1507
            if final and rqlst.orderby and not self.hasaggrstep:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1508
                orderby = []
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1509
                for node in rqlst.orderby:
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1510
                    rqlst = self._rqlst_accept(rqlst, node, newroot, terms,
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1511
                                               orderby.append)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1512
                if orderby:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1513
                    newroot.set_orderby(orderby)
5768
1e73a466aa69 [fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5582
diff changeset
  1514
            elif rqlst.orderby:
1e73a466aa69 [fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5582
diff changeset
  1515
                for sortterm in rqlst.orderby:
1e73a466aa69 [fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5582
diff changeset
  1516
                    if any(f for f in sortterm.iget_nodes(Function) if f.name == 'FTIRANK'):
1e73a466aa69 [fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5582
diff changeset
  1517
                        newnode, oldnode = sortterm.accept(self, newroot, terms)
1e73a466aa69 [fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5582
diff changeset
  1518
                        if newnode is not None:
1e73a466aa69 [fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5582
diff changeset
  1519
                            newroot.add_sort_term(newnode)
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1520
            self.process_selection(newroot, terms, rqlst)
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1521
        elif not newroot.where:
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1522
            # no restrictions have been copied, just select terms and add
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1523
            # type restriction (done later by add_types_restriction)
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1524
            for v in terms:
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1525
                if not isinstance(v, Variable):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1526
                    continue
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1527
                newroot.append_selected(VariableRef(newroot.get_variable(v.name)))
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1528
        solutions = self.ppi.copy_solutions(solindices)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1529
        cleanup_solutions(newroot, solutions)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1530
        newroot.set_possible_types(solutions)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1531
        if final:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1532
            if self.hasaggrstep:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1533
                self.add_necessary_selection(newroot, self.mayneedsel & self.extneedsel)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1534
            newroot.distinct = rqlst.distinct
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1535
        else:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1536
            self.add_necessary_selection(newroot, self.mayneedsel & self.needsel)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1537
            # insert vars to fetch constant values when needed
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1538
            for (varname, rschema), reldefs in self.mayneedvar.iteritems():
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1539
                for rel, ored in reldefs:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1540
                    if not (varname, rschema) in self.hasvar:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1541
                        self.hasvar[(varname, rschema)] = None # just to avoid further insertion
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1542
                        cvar = newroot.make_variable()
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1543
                        for sol in newroot.solutions:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1544
                            sol[cvar.name] = rschema.objects(sol[varname])[0]
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1545
                        # if the current restriction is not used in a OR branch,
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1546
                        # we can keep it, else we have to drop the constant
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1547
                        # restriction (or we may miss some results)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1548
                        if not ored:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1549
                            rel = rel.copy(newroot)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1550
                            newroot.add_restriction(rel)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1551
                        # add a relation to link the variable
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1552
                        newroot.remove_node(rel.children[1])
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1553
                        cmp = Comparison('=')
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1554
                        rel.append(cmp)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1555
                        cmp.append(VariableRef(cvar))
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1556
                        self.insertedvars.append((varname, rschema, cvar.name))
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1557
                        newroot.append_selected(VariableRef(newroot.get_variable(cvar.name)))
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1558
                        # NOTE: even if the restriction is done by this query, we have
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1559
                        # to let it in the original rqlst so that it appears anyway in
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1560
                        # the "final" query, else we may change the meaning of the query
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1561
                        # if there are NOT somewhere :
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1562
                        # 'NOT X relation Y, Y name "toto"' means X WHERE X isn't related
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1563
                        # to Y whose name is toto while
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1564
                        # 'NOT X relation Y' means X WHERE X has no 'relation' (whatever Y)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1565
                    elif ored:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1566
                        newroot.remove_node(rel)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1567
        add_types_restriction(self.schema, rqlst, newroot, solutions)
2766
07e47b84acc2 DBG_MS flag
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2688
diff changeset
  1568
        if server.DEBUG & server.DBG_MS:
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1569
            print '--->', newroot
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1570
        return newroot, self.insertedvars
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
  1571
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1572
    def visit_and(self, node, newroot, terms):
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1573
        subparts = []
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1574
        for i in xrange(len(node.children)):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1575
            child = node.children[i]
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1576
            try:
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1577
                newchild, child_ = child.accept(self, newroot, terms)
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1578
                if not child_ is child:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1579
                    node = child_.parent
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1580
                if newchild is None:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1581
                    continue
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1582
                subparts.append(newchild)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1583
            except UnsupportedBranch:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1584
                continue
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1585
        if not subparts:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1586
            return None, node
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1587
        if len(subparts) == 1:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1588
            return subparts[0], node
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1589
        return copy_node(newroot, node, subparts), node
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1590
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1591
    visit_or = visit_and
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1592
393
45a675515427 cross_relations support
sylvain.thenault@logilab.fr
parents: 392
diff changeset
  1593
    def _relation_supported(self, relation):
45a675515427 cross_relations support
sylvain.thenault@logilab.fr
parents: 392
diff changeset
  1594
        rtype = relation.r_type
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1595
        for source in self.sources:
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1596
            if not source.support_relation(rtype) or (
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1597
                rtype in source.cross_relations and not relation in self.terms):
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1598
                return False
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1599
        if not self.final and not relation in self.terms:
393
45a675515427 cross_relations support
sylvain.thenault@logilab.fr
parents: 392
diff changeset
  1600
            rschema = self.schema.rschema(relation.r_type)
3689
deb13e88e037 follow yams 0.25 api changes to improve performance
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3624
diff changeset
  1601
            if not rschema.final:
393
45a675515427 cross_relations support
sylvain.thenault@logilab.fr
parents: 392
diff changeset
  1602
                for term in relation.get_nodes((VariableRef, Constant)):
45a675515427 cross_relations support
sylvain.thenault@logilab.fr
parents: 392
diff changeset
  1603
                    term = getattr(term, 'variable', term)
45a675515427 cross_relations support
sylvain.thenault@logilab.fr
parents: 392
diff changeset
  1604
                    termsources = sorted(set(x[0] for x in self.ppi._term_sources(term)))
45a675515427 cross_relations support
sylvain.thenault@logilab.fr
parents: 392
diff changeset
  1605
                    if termsources and termsources != self.sources:
45a675515427 cross_relations support
sylvain.thenault@logilab.fr
parents: 392
diff changeset
  1606
                        return False
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1607
        return True
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
  1608
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1609
    def visit_relation(self, node, newroot, terms):
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1610
        if not node.is_types_restriction():
6758
28b11ecf319b [ms] #1382452: incorrect results with multi-sources
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6673
diff changeset
  1611
            if not node in terms and node in self.skip and self.solindices.issubset(self.skip[node]):
3689
deb13e88e037 follow yams 0.25 api changes to improve performance
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3624
diff changeset
  1612
                if not self.schema.rschema(node.r_type).final:
6057
cb5c8852cbda [ms] cleanups
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6056
diff changeset
  1613
                    # can't really skip the relation if one variable is selected
cb5c8852cbda [ms] cleanups
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6056
diff changeset
  1614
                    # and only referenced by this relation
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1615
                    for vref in node.iget_nodes(VariableRef):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1616
                        stinfo = vref.variable.stinfo
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1617
                        if stinfo['selected'] and len(stinfo['relations']) == 1:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1618
                            break
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1619
                    else:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1620
                        return None, node
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1621
                else:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1622
                    return None, node
393
45a675515427 cross_relations support
sylvain.thenault@logilab.fr
parents: 392
diff changeset
  1623
            if not self._relation_supported(node):
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1624
                raise UnsupportedBranch()
6056
37a37e5ba330 [ms] fix a ms plannification bug with NOT is
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6027
diff changeset
  1625
        # don't copy type restriction unless this is the only supported relation
37a37e5ba330 [ms] fix a ms plannification bug with NOT is
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6027
diff changeset
  1626
        # for the lhs variable, else they'll be reinserted later as needed (in
37a37e5ba330 [ms] fix a ms plannification bug with NOT is
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6027
diff changeset
  1627
        # other cases we may copy a type restriction while the variable is not
37a37e5ba330 [ms] fix a ms plannification bug with NOT is
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6027
diff changeset
  1628
        # actually used)
37a37e5ba330 [ms] fix a ms plannification bug with NOT is
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6027
diff changeset
  1629
        elif not (node.neged(strict=True) or
37a37e5ba330 [ms] fix a ms plannification bug with NOT is
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6027
diff changeset
  1630
                  any(self._relation_supported(rel)
37a37e5ba330 [ms] fix a ms plannification bug with NOT is
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6027
diff changeset
  1631
                      for rel in node.children[0].variable.stinfo['relations'])):
37a37e5ba330 [ms] fix a ms plannification bug with NOT is
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6027
diff changeset
  1632
            return self.visit_default(node, newroot, terms)
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1633
        else:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1634
            raise UnsupportedBranch()
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1635
        rschema = self.schema.rschema(node.r_type)
2109
f469368df530 when a relation isn't supported by a ms query parts, dereference introduced variable refs
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2108
diff changeset
  1636
        self._pending_vrefs = []
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1637
        try:
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1638
            res = self.visit_default(node, newroot, terms)[0]
2109
f469368df530 when a relation isn't supported by a ms query parts, dereference introduced variable refs
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2108
diff changeset
  1639
        except:
2110
d7773fb1b892 add comment explaining the pb
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2109
diff changeset
  1640
            # when a relation isn't supported, we should dereference potentially
d7773fb1b892 add comment explaining the pb
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2109
diff changeset
  1641
            # introduced variable refs
2109
f469368df530 when a relation isn't supported by a ms query parts, dereference introduced variable refs
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2108
diff changeset
  1642
            for vref in self._pending_vrefs:
f469368df530 when a relation isn't supported by a ms query parts, dereference introduced variable refs
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2108
diff changeset
  1643
                vref.unregister_reference()
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1644
            raise
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1645
        ored = node.ored()
3689
deb13e88e037 follow yams 0.25 api changes to improve performance
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3624
diff changeset
  1646
        if rschema.final or rschema.inlined:
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1647
            vrefs = node.children[1].get_nodes(VariableRef)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1648
            if not vrefs:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1649
                if not ored:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1650
                    self.skip.setdefault(node, set()).update(self.solindices)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1651
                else:
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
  1652
                    self.mayneedvar.setdefault((node.children[0].name, rschema), []).append( (res, ored) )
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1653
            else:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1654
                assert len(vrefs) == 1
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1655
                vref = vrefs[0]
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1656
                # XXX check operator ?
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1657
                self.hasvar[(node.children[0].name, rschema)] = vref
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1658
                if self._may_skip_attr_rel(rschema, node, vref, ored, terms, res):
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1659
                    self.skip.setdefault(node, set()).update(self.solindices)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1660
        elif not ored:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1661
            self.skip.setdefault(node, set()).update(self.solindices)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1662
        return res, node
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1663
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1664
    def _may_skip_attr_rel(self, rschema, rel, vref, ored, terms, res):
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1665
        var = vref.variable
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1666
        if ored:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1667
            return False
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1668
        if var.name in self.extneedsel or var.stinfo['selected']:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1669
            return False
2075
933ccfce6e91 fix bug in multisource planner, w/ inlined relation
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1977
diff changeset
  1670
        if not var in terms or used_in_outer_scope(var, self.current_scope):
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1671
            return False
5004
4cc020ee70e2 le patch rql26 a été importé
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4956
diff changeset
  1672
        if any(v for v, _ in var.stinfo.get('attrvars', ()) if not v in terms):
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1673
            return False
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1674
        return True
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
  1675
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1676
    def visit_exists(self, node, newroot, terms):
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1677
        newexists = node.__class__()
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1678
        self.scopes = {node: newexists}
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1679
        subparts, node = self._visit_children(node, newroot, terms)
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1680
        if not subparts:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1681
            return None, node
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1682
        newexists.set_where(subparts[0])
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1683
        return newexists, node
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
  1684
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1685
    def visit_not(self, node, newroot, terms):
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1686
        subparts, node = self._visit_children(node, newroot, terms)
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1687
        if not subparts:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1688
            return None, node
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1689
        return copy_node(newroot, node, subparts), node
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
  1690
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1691
    def visit_group(self, node, newroot, terms):
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1692
        if not self.final:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1693
            return None, node
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1694
        return self.visit_default(node, newroot, terms)
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
  1695
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1696
    def visit_variableref(self, node, newroot, terms):
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1697
        if self.use_only_defined:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1698
            if not node.variable.name in newroot.defined_vars:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1699
                raise UnsupportedBranch(node.name)
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1700
        elif not node.variable in terms:
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1701
            raise UnsupportedBranch(node.name)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1702
        self.mayneedsel.add(node.name)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1703
        # set scope so we can insert types restriction properly
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1704
        newvar = newroot.get_variable(node.name)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1705
        newvar.stinfo['scope'] = self.scopes.get(node.variable.scope, newroot)
2109
f469368df530 when a relation isn't supported by a ms query parts, dereference introduced variable refs
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2108
diff changeset
  1706
        vref = VariableRef(newvar)
f469368df530 when a relation isn't supported by a ms query parts, dereference introduced variable refs
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2108
diff changeset
  1707
        self._pending_vrefs.append(vref)
f469368df530 when a relation isn't supported by a ms query parts, dereference introduced variable refs
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2108
diff changeset
  1708
        return vref, node
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1709
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1710
    def visit_constant(self, node, newroot, terms):
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1711
        return copy_node(newroot, node), node
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
  1712
3815
50b87f759b5d test and fix http://www.logilab.org/ticket/499838
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3689
diff changeset
  1713
    def visit_comparison(self, node, newroot, terms):
50b87f759b5d test and fix http://www.logilab.org/ticket/499838
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3689
diff changeset
  1714
        subparts, node = self._visit_children(node, newroot, terms)
50b87f759b5d test and fix http://www.logilab.org/ticket/499838
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3689
diff changeset
  1715
        copy = copy_node(newroot, node, subparts)
50b87f759b5d test and fix http://www.logilab.org/ticket/499838
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3689
diff changeset
  1716
        # ignore comparison operator when fetching non final query
50b87f759b5d test and fix http://www.logilab.org/ticket/499838
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3689
diff changeset
  1717
        if not self.final and isinstance(node.children[0], VariableRef):
50b87f759b5d test and fix http://www.logilab.org/ticket/499838
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3689
diff changeset
  1718
            copy.operator = '='
50b87f759b5d test and fix http://www.logilab.org/ticket/499838
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3689
diff changeset
  1719
        return copy, node
50b87f759b5d test and fix http://www.logilab.org/ticket/499838
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3689
diff changeset
  1720
5768
1e73a466aa69 [fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5582
diff changeset
  1721
    def visit_function(self, node, newroot, terms):
1e73a466aa69 [fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5582
diff changeset
  1722
        if node.name == 'FTIRANK':
1e73a466aa69 [fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5582
diff changeset
  1723
            # FTIRANK is somewhat special... Rank function should be included in
1e73a466aa69 [fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5582
diff changeset
  1724
            # the same query has the has_text relation, potentially added to
1e73a466aa69 [fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5582
diff changeset
  1725
            # selection for latter usage
1e73a466aa69 [fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5582
diff changeset
  1726
            if not self.hasaggrstep and self.final and node not in self.skip:
1e73a466aa69 [fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5582
diff changeset
  1727
                return self.visit_default(node, newroot, terms)
1e73a466aa69 [fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5582
diff changeset
  1728
            elif any(s for s in self.sources if s.uri != 'system'):
1e73a466aa69 [fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5582
diff changeset
  1729
                return None, node
1e73a466aa69 [fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5582
diff changeset
  1730
            # p = node.parent
1e73a466aa69 [fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5582
diff changeset
  1731
            # while p is not None and not isinstance(p, SortTerm):
1e73a466aa69 [fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5582
diff changeset
  1732
            #     p = p.parent
1e73a466aa69 [fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5582
diff changeset
  1733
            # if isinstance(p, SortTerm):
1e73a466aa69 [fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5582
diff changeset
  1734
            if not self.hasaggrstep and self.final and node in self.skip:
1e73a466aa69 [fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5582
diff changeset
  1735
                return Constant(self.skip[node], 'Int'), node
1e73a466aa69 [fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5582
diff changeset
  1736
            # XXX only if not yet selected
1e73a466aa69 [fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5582
diff changeset
  1737
            newroot.append_selected(node.copy(newroot))
1e73a466aa69 [fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5582
diff changeset
  1738
            self.skip[node] = len(newroot.selection)
1e73a466aa69 [fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5582
diff changeset
  1739
            return None, node
1e73a466aa69 [fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5582
diff changeset
  1740
        return self.visit_default(node, newroot, terms)
1e73a466aa69 [fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5582
diff changeset
  1741
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1742
    def visit_default(self, node, newroot, terms):
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1743
        subparts, node = self._visit_children(node, newroot, terms)
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1744
        return copy_node(newroot, node, subparts), node
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
  1745
5768
1e73a466aa69 [fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5582
diff changeset
  1746
    visit_mathexpression = visit_constant = visit_default
1e73a466aa69 [fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5582
diff changeset
  1747
1e73a466aa69 [fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5582
diff changeset
  1748
    def visit_sortterm(self, node, newroot, terms):
1e73a466aa69 [fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5582
diff changeset
  1749
        subparts, node = self._visit_children(node, newroot, terms)
1e73a466aa69 [fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5582
diff changeset
  1750
        if not subparts:
1e73a466aa69 [fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5582
diff changeset
  1751
            return None, node
1e73a466aa69 [fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5582
diff changeset
  1752
        return copy_node(newroot, node, subparts), node
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
  1753
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1754
    def _visit_children(self, node, newroot, terms):
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1755
        subparts = []
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1756
        for i in xrange(len(node.children)):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1757
            child = node.children[i]
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1758
            newchild, child_ = child.accept(self, newroot, terms)
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1759
            if not child is child_:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1760
                node = child_.parent
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1761
            if newchild is not None:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1762
                subparts.append(newchild)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1763
        return subparts, node
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
  1764
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1765
    def process_selection(self, newroot, terms, rqlst):
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1766
        if self.final:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1767
            for term in rqlst.selection:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1768
                newroot.append_selected(term.copy(newroot))
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1769
                for vref in term.get_nodes(VariableRef):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1770
                    self.needsel.add(vref.name)
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
  1771
            return
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1772
        for term in rqlst.selection:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1773
            vrefs = term.get_nodes(VariableRef)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1774
            if vrefs:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1775
                supportedvars = []
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1776
                for vref in vrefs:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1777
                    var = vref.variable
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1778
                    if var in terms:
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1779
                        supportedvars.append(vref)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1780
                        continue
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1781
                    else:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1782
                        self.needsel.add(vref.name)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1783
                        break
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1784
                else:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1785
                    for vref in vrefs:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1786
                        newroot.append_selected(vref.copy(newroot))
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1787
                    supportedvars = []
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1788
                for vref in supportedvars:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1789
                    if not vref in newroot.get_selected_variables():
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1790
                        newroot.append_selected(VariableRef(newroot.get_variable(vref.name)))
6131
087c5a168010 [ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6129
diff changeset
  1791
            elif term in self.terms:
087c5a168010 [ms] more planning bug fixes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6129
diff changeset
  1792
                newroot.append_selected(term.copy(newroot))
1785
01245e2a777d ms planning fix
sylvain.thenault@logilab.fr
parents: 1409
diff changeset
  1793
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1794
    def add_necessary_selection(self, newroot, terms):
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1795
        selected = tuple(newroot.get_selected_variables())
1228
91ae10ffb611 * refactor ms planner (renaming, reorganization)
sylvain.thenault@logilab.fr
parents: 1124
diff changeset
  1796
        for varname in terms:
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1797
            var = newroot.defined_vars[varname]
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1798
            for vref in var.references():
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1799
                rel = vref.relation()
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1800
                if rel is None and vref in selected:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1801
                    # already selected
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1802
                    break
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1803
            else:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1804
                selvref = VariableRef(var)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1805
                newroot.append_selected(selvref)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1806
                if newroot.groupby:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
  1807
                    newroot.add_group_var(VariableRef(selvref.variable, noautoref=1))