server/mssteps.py
author Sylvain Thénault <sylvain.thenault@logilab.fr>
Tue, 11 Sep 2012 22:32:01 +0200
changeset 8554 6e5d8512b07d
parent 8151 8e3fa37cd8aa
child 8694 d901c36bcfce
permissions -rw-r--r--
[repo cleanup] drop code moved to querier by 7e264ce34cd4
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
7521
a5a5eea29c1b [querier] turn remove_clauses/restore_clauses into a context manager
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7108
diff changeset
     1
# copyright 2003-2011 LOGILAB S.A. (Paris, FRANCE), all rights reserved.
5421
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4212
diff changeset
     2
# contact http://www.logilab.fr/ -- mailto:contact@logilab.fr
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4212
diff changeset
     3
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4212
diff changeset
     4
# This file is part of CubicWeb.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4212
diff changeset
     5
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4212
diff changeset
     6
# CubicWeb is free software: you can redistribute it and/or modify it under the
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4212
diff changeset
     7
# terms of the GNU Lesser General Public License as published by the Free
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4212
diff changeset
     8
# Software Foundation, either version 2.1 of the License, or (at your option)
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4212
diff changeset
     9
# any later version.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4212
diff changeset
    10
#
5424
8ecbcbff9777 replace logilab-common by CubicWeb in disclaimer
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5421
diff changeset
    11
# CubicWeb is distributed in the hope that it will be useful, but WITHOUT
5421
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4212
diff changeset
    12
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4212
diff changeset
    13
# FOR A PARTICULAR PURPOSE.  See the GNU Lesser General Public License for more
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4212
diff changeset
    14
# details.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4212
diff changeset
    15
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4212
diff changeset
    16
# You should have received a copy of the GNU Lesser General Public License along
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4212
diff changeset
    17
# with CubicWeb.  If not, see <http://www.gnu.org/licenses/>.
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    18
"""Defines the diferent querier steps usable in plans.
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    19
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    20
FIXME : this code needs refactoring. Some problems :
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    21
* get data from the parent plan, the latest step, temporary table...
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    22
* each step has is own members (this is not necessarily bad, but a bit messy
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    23
  for now)
5784
8b2264e485bc [ms] orderby/groupby/having attributes of Select node should be resetted to empty tuple, not None
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
    24
"""
7521
a5a5eea29c1b [querier] turn remove_clauses/restore_clauses into a context manager
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7108
diff changeset
    25
from __future__ import with_statement
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    26
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    27
__docformat__ = "restructuredtext en"
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    28
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    29
from rql.nodes import VariableRef, Variable, Function
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    30
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    31
from cubicweb.server.ssplanner import (LimitOffsetMixIn, Step, OneFetchStep,
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    32
                                    varmap_test_repr, offset_result)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    33
1802
d628defebc17 delete-trailing-whitespace + some copyright update
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 470
diff changeset
    34
AGGR_TRANSFORMS = {'COUNT':'SUM', 'MIN':'MIN', 'MAX':'MAX', 'SUM': 'SUM'}
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    35
7521
a5a5eea29c1b [querier] turn remove_clauses/restore_clauses into a context manager
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7108
diff changeset
    36
class remove_and_restore_clauses(object):
a5a5eea29c1b [querier] turn remove_clauses/restore_clauses into a context manager
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7108
diff changeset
    37
    def __init__(self, union, keepgroup):
a5a5eea29c1b [querier] turn remove_clauses/restore_clauses into a context manager
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7108
diff changeset
    38
        self.union = union
a5a5eea29c1b [querier] turn remove_clauses/restore_clauses into a context manager
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7108
diff changeset
    39
        self.keepgroup = keepgroup
a5a5eea29c1b [querier] turn remove_clauses/restore_clauses into a context manager
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7108
diff changeset
    40
        self.clauses = None
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    41
7521
a5a5eea29c1b [querier] turn remove_clauses/restore_clauses into a context manager
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7108
diff changeset
    42
    def __enter__(self):
a5a5eea29c1b [querier] turn remove_clauses/restore_clauses into a context manager
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7108
diff changeset
    43
        self.clauses = clauses = []
a5a5eea29c1b [querier] turn remove_clauses/restore_clauses into a context manager
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7108
diff changeset
    44
        for select in self.union.children:
a5a5eea29c1b [querier] turn remove_clauses/restore_clauses into a context manager
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7108
diff changeset
    45
            if self.keepgroup:
a5a5eea29c1b [querier] turn remove_clauses/restore_clauses into a context manager
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7108
diff changeset
    46
                having, orderby = select.having, select.orderby
a5a5eea29c1b [querier] turn remove_clauses/restore_clauses into a context manager
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7108
diff changeset
    47
                select.having, select.orderby = (), ()
a5a5eea29c1b [querier] turn remove_clauses/restore_clauses into a context manager
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7108
diff changeset
    48
                clauses.append( (having, orderby) )
a5a5eea29c1b [querier] turn remove_clauses/restore_clauses into a context manager
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7108
diff changeset
    49
            else:
a5a5eea29c1b [querier] turn remove_clauses/restore_clauses into a context manager
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7108
diff changeset
    50
                groupby, having, orderby = select.groupby, select.having, select.orderby
a5a5eea29c1b [querier] turn remove_clauses/restore_clauses into a context manager
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7108
diff changeset
    51
                select.groupby, select.having, select.orderby = (), (), ()
a5a5eea29c1b [querier] turn remove_clauses/restore_clauses into a context manager
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7108
diff changeset
    52
                clauses.append( (groupby, having, orderby) )
a5a5eea29c1b [querier] turn remove_clauses/restore_clauses into a context manager
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7108
diff changeset
    53
a5a5eea29c1b [querier] turn remove_clauses/restore_clauses into a context manager
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7108
diff changeset
    54
    def __exit__(self, exctype, exc, traceback):
a5a5eea29c1b [querier] turn remove_clauses/restore_clauses into a context manager
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7108
diff changeset
    55
        for i, select in enumerate(self.union.children):
a5a5eea29c1b [querier] turn remove_clauses/restore_clauses into a context manager
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7108
diff changeset
    56
            if self.keepgroup:
a5a5eea29c1b [querier] turn remove_clauses/restore_clauses into a context manager
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7108
diff changeset
    57
                select.having, select.orderby = self.clauses[i]
a5a5eea29c1b [querier] turn remove_clauses/restore_clauses into a context manager
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7108
diff changeset
    58
            else:
a5a5eea29c1b [querier] turn remove_clauses/restore_clauses into a context manager
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7108
diff changeset
    59
                select.groupby, select.having, select.orderby = self.clauses[i]
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    60
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    61
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    62
class FetchStep(OneFetchStep):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    63
    """step consisting in fetching data from sources, and storing result in
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    64
    a temporary table
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    65
    """
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    66
    def __init__(self, plan, union, sources, table, keepgroup, inputmap=None):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    67
        OneFetchStep.__init__(self, plan, union, sources)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    68
        # temporary table to store step result
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    69
        self.table = table
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    70
        # should groupby clause be kept or not
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    71
        self.keepgroup = keepgroup
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    72
        # variables mapping to use as input
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    73
        self.inputmap = inputmap
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    74
        # output variable mapping
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    75
        srqlst = union.children[0] # sample select node
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    76
        # add additional information to the output mapping
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    77
        self.outputmap = plan.init_temp_table(table, srqlst.selection,
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    78
                                              srqlst.solutions[0])
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    79
        for vref in srqlst.selection:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    80
            if not isinstance(vref, VariableRef):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    81
                continue
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    82
            var = vref.variable
5004
4cc020ee70e2 le patch rql26 a été importé
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4212
diff changeset
    83
            if var.stinfo.get('attrvars'):
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    84
                for lhsvar, rtype in var.stinfo['attrvars']:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    85
                    if lhsvar.name in srqlst.defined_vars:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    86
                        key = '%s.%s' % (lhsvar.name, rtype)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    87
                        self.outputmap[key] = self.outputmap[var.name]
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    88
            else:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    89
                rschema = self.plan.schema.rschema
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    90
                for rel in var.stinfo['rhsrelations']:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    91
                    if rschema(rel.r_type).inlined:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    92
                        lhsvar = rel.children[0]
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    93
                        if lhsvar.name in srqlst.defined_vars:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    94
                            key = '%s.%s' % (lhsvar.name, rel.r_type)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    95
                            self.outputmap[key] = self.outputmap[var.name]
1802
d628defebc17 delete-trailing-whitespace + some copyright update
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 470
diff changeset
    96
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    97
    def execute(self):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    98
        """execute this step"""
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    99
        self.execute_children()
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   100
        plan = self.plan
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   101
        plan.create_temp_table(self.table)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   102
        union = self.union
7521
a5a5eea29c1b [querier] turn remove_clauses/restore_clauses into a context manager
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7108
diff changeset
   103
        with remove_and_restore_clauses(union, self.keepgroup):
a5a5eea29c1b [querier] turn remove_clauses/restore_clauses into a context manager
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7108
diff changeset
   104
            for source in self.sources:
a5a5eea29c1b [querier] turn remove_clauses/restore_clauses into a context manager
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7108
diff changeset
   105
                source.flying_insert(self.table, plan.session, union, plan.args,
a5a5eea29c1b [querier] turn remove_clauses/restore_clauses into a context manager
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7108
diff changeset
   106
                                     self.inputmap)
1802
d628defebc17 delete-trailing-whitespace + some copyright update
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 470
diff changeset
   107
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   108
    def mytest_repr(self):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   109
        """return a representation of this step suitable for test"""
7521
a5a5eea29c1b [querier] turn remove_clauses/restore_clauses into a context manager
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7108
diff changeset
   110
        with remove_and_restore_clauses(self.union, self.keepgroup):
a5a5eea29c1b [querier] turn remove_clauses/restore_clauses into a context manager
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7108
diff changeset
   111
            try:
a5a5eea29c1b [querier] turn remove_clauses/restore_clauses into a context manager
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7108
diff changeset
   112
                inputmap = varmap_test_repr(self.inputmap, self.plan.tablesinorder)
a5a5eea29c1b [querier] turn remove_clauses/restore_clauses into a context manager
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7108
diff changeset
   113
                outputmap = varmap_test_repr(self.outputmap, self.plan.tablesinorder)
a5a5eea29c1b [querier] turn remove_clauses/restore_clauses into a context manager
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7108
diff changeset
   114
            except AttributeError:
a5a5eea29c1b [querier] turn remove_clauses/restore_clauses into a context manager
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7108
diff changeset
   115
                inputmap = self.inputmap
a5a5eea29c1b [querier] turn remove_clauses/restore_clauses into a context manager
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7108
diff changeset
   116
                outputmap = self.outputmap
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   117
            return (self.__class__.__name__,
7521
a5a5eea29c1b [querier] turn remove_clauses/restore_clauses into a context manager
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7108
diff changeset
   118
                    sorted((r.as_string(kwargs=self.plan.args), r.solutions)
a5a5eea29c1b [querier] turn remove_clauses/restore_clauses into a context manager
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7108
diff changeset
   119
                           for r in self.union.children),
a5a5eea29c1b [querier] turn remove_clauses/restore_clauses into a context manager
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7108
diff changeset
   120
                    sorted(self.sources), inputmap, outputmap)
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   121
1802
d628defebc17 delete-trailing-whitespace + some copyright update
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 470
diff changeset
   122
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   123
class AggrStep(LimitOffsetMixIn, Step):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   124
    """step consisting in making aggregat from temporary data in the system
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   125
    source
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   126
    """
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   127
    def __init__(self, plan, selection, select, table, outputtable=None):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   128
        Step.__init__(self, plan)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   129
        # original selection
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   130
        self.selection = selection
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   131
        # original Select RQL tree
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   132
        self.select = select
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   133
        # table where are located temporary results
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   134
        self.table = table
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   135
        # optional table where to write results
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   136
        self.outputtable = outputtable
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   137
        if outputtable is not None:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   138
            plan.init_temp_table(outputtable, selection, select.solutions[0])
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   139
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   140
        #self.inputmap = inputmap
1802
d628defebc17 delete-trailing-whitespace + some copyright update
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 470
diff changeset
   141
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   142
    def mytest_repr(self):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   143
        """return a representation of this step suitable for test"""
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   144
        try:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   145
            # rely on a monkey patch (cf unittest_querier)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   146
            table = self.plan.tablesinorder[self.table]
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   147
            outputtable = self.outputtable and self.plan.tablesinorder[self.outputtable]
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   148
        except AttributeError:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   149
            # not monkey patched
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   150
            table = self.table
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   151
            outputtable = self.outputtable
5768
1e73a466aa69 [fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   152
        sql = self.get_sql().replace(self.table, table)
1e73a466aa69 [fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   153
        return (self.__class__.__name__, sql, outputtable)
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   154
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   155
    def execute(self):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   156
        """execute this step"""
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   157
        self.execute_children()
5768
1e73a466aa69 [fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   158
        sql = self.get_sql()
1e73a466aa69 [fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   159
        if self.outputtable:
1e73a466aa69 [fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   160
            self.plan.create_temp_table(self.outputtable)
1e73a466aa69 [fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   161
            sql = 'INSERT INTO %s %s' % (self.outputtable, sql)
8151
8e3fa37cd8aa [multi-sources] call doexec rather than sqlexec since we do not want to call process_result (as there aren't any and this breaks with lgdb default)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7521
diff changeset
   162
            self.plan.syssource.doexec(self.plan.session, sql, self.plan.args)
8e3fa37cd8aa [multi-sources] call doexec rather than sqlexec since we do not want to call process_result (as there aren't any and this breaks with lgdb default)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7521
diff changeset
   163
        else:
8e3fa37cd8aa [multi-sources] call doexec rather than sqlexec since we do not want to call process_result (as there aren't any and this breaks with lgdb default)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7521
diff changeset
   164
            return self.plan.sqlexec(sql, self.plan.args)
5768
1e73a466aa69 [fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   165
1e73a466aa69 [fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   166
    def get_sql(self):
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   167
        self.inputmap = inputmap = self.children[-1].outputmap
7108
bcdf22734059 Abstract the support for ORDER BY and LIMIT/OFFSET SQL generation
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5811
diff changeset
   168
        dbhelper=self.plan.syssource.dbhelper
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   169
        # get the select clause
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   170
        clause = []
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   171
        for i, term in enumerate(self.selection):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   172
            try:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   173
                var_name = inputmap[term.as_string()]
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   174
            except KeyError:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   175
                var_name = 'C%s' % i
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   176
            if isinstance(term, Function):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   177
                # we have to translate some aggregat function
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   178
                # (for instance COUNT -> SUM)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   179
                orig_name = term.name
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   180
                try:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   181
                    term.name = AGGR_TRANSFORMS[term.name]
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   182
                    # backup and reduce children
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   183
                    orig_children = term.children
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   184
                    term.children = [VariableRef(Variable(var_name))]
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   185
                    clause.append(term.accept(self))
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   186
                    # restaure the tree XXX necessary?
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   187
                    term.name = orig_name
1802
d628defebc17 delete-trailing-whitespace + some copyright update
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 470
diff changeset
   188
                    term.children = orig_children
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   189
                except KeyError:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   190
                    clause.append(var_name)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   191
            else:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   192
                clause.append(var_name)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   193
                for vref in term.iget_nodes(VariableRef):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   194
                    inputmap[vref.name] = var_name
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   195
        # XXX handle distinct with non selected sort term
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   196
        if self.select.distinct:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   197
            sql = ['SELECT DISTINCT %s' % ', '.join(clause)]
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   198
        else:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   199
            sql = ['SELECT %s' % ', '.join(clause)]
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   200
        sql.append("FROM %s" % self.table)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   201
        # get the group/having clauses
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   202
        if self.select.groupby:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   203
            clause = [inputmap[var.name] for var in self.select.groupby]
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   204
            grouped = set(var.name for var in self.select.groupby)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   205
            sql.append('GROUP BY %s' % ', '.join(clause))
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   206
        else:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   207
            grouped = None
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   208
        if self.select.having:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   209
            clause = [term.accept(self) for term in self.select.having]
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   210
            sql.append('HAVING %s' % ', '.join(clause))
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   211
        # get the orderby clause
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   212
        if self.select.orderby:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   213
            clause = []
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   214
            for sortterm in self.select.orderby:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   215
                sqlterm = sortterm.term.accept(self)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   216
                if sortterm.asc:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   217
                    clause.append(sqlterm)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   218
                else:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   219
                    clause.append('%s DESC' % sqlterm)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   220
                if grouped is not None:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   221
                    for vref in sortterm.iget_nodes(VariableRef):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   222
                        if not vref.name in grouped:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   223
                            sql[-1] += ', ' + self.inputmap[vref.name]
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   224
                            grouped.add(vref.name)
7108
bcdf22734059 Abstract the support for ORDER BY and LIMIT/OFFSET SQL generation
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5811
diff changeset
   225
            sql = dbhelper.sql_add_order_by(' '.join(sql),
bcdf22734059 Abstract the support for ORDER BY and LIMIT/OFFSET SQL generation
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5811
diff changeset
   226
                                            clause,
bcdf22734059 Abstract the support for ORDER BY and LIMIT/OFFSET SQL generation
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5811
diff changeset
   227
                                            None, False,
bcdf22734059 Abstract the support for ORDER BY and LIMIT/OFFSET SQL generation
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5811
diff changeset
   228
                                            self.limit or self.offset)
bcdf22734059 Abstract the support for ORDER BY and LIMIT/OFFSET SQL generation
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5811
diff changeset
   229
        else:
bcdf22734059 Abstract the support for ORDER BY and LIMIT/OFFSET SQL generation
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5811
diff changeset
   230
            sql = ' '.join(sql)
bcdf22734059 Abstract the support for ORDER BY and LIMIT/OFFSET SQL generation
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5811
diff changeset
   231
            clause = None
bcdf22734059 Abstract the support for ORDER BY and LIMIT/OFFSET SQL generation
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5811
diff changeset
   232
bcdf22734059 Abstract the support for ORDER BY and LIMIT/OFFSET SQL generation
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5811
diff changeset
   233
        sql = dbhelper.sql_add_limit_offset(sql, self.limit, self.offset, clause)
bcdf22734059 Abstract the support for ORDER BY and LIMIT/OFFSET SQL generation
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5811
diff changeset
   234
        return sql
1802
d628defebc17 delete-trailing-whitespace + some copyright update
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 470
diff changeset
   235
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   236
    def visit_function(self, function):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   237
        """generate SQL name for a function"""
5768
1e73a466aa69 [fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   238
        try:
1e73a466aa69 [fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   239
            return self.children[0].outputmap[str(function)]
1e73a466aa69 [fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   240
        except KeyError:
1e73a466aa69 [fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   241
            return '%s(%s)' % (function.name,
1e73a466aa69 [fti] support for fti ranking: has_text query results sorted by relevance, and provides a way to control weight per entity / entity's attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5426
diff changeset
   242
                               ','.join(c.accept(self) for c in function.children))
1802
d628defebc17 delete-trailing-whitespace + some copyright update
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 470
diff changeset
   243
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   244
    def visit_variableref(self, variableref):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   245
        """get the sql name for a variable reference"""
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   246
        try:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   247
            return self.inputmap[variableref.name]
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   248
        except KeyError: # XXX duh? explain
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   249
            return variableref.variable.name
1802
d628defebc17 delete-trailing-whitespace + some copyright update
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 470
diff changeset
   250
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   251
    def visit_constant(self, constant):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   252
        """generate SQL name for a constant"""
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   253
        assert constant.type == 'Int'
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   254
        return str(constant.value)
1802
d628defebc17 delete-trailing-whitespace + some copyright update
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 470
diff changeset
   255
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   256
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   257
class UnionStep(LimitOffsetMixIn, Step):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   258
    """union results of child in-memory steps (e.g. OneFetchStep / AggrStep)"""
1802
d628defebc17 delete-trailing-whitespace + some copyright update
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 470
diff changeset
   259
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   260
    def execute(self):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   261
        """execute this step"""
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   262
        result = []
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   263
        limit = olimit = self.limit
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   264
        offset = self.offset
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   265
        assert offset != 0
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   266
        if offset is not None:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   267
            limit = limit + offset
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   268
        for step in self.children:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   269
            if limit is not None:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   270
                if offset is None:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   271
                    limit = olimit - len(result)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   272
                step.set_limit_offset(limit, None)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   273
            result_ = step.execute()
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   274
            if offset is not None:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   275
                offset, result_ = offset_result(offset, result_)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   276
            result += result_
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   277
            if limit is not None:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   278
                if len(result) >= olimit:
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   279
                    return result[:olimit]
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   280
        return result
1802
d628defebc17 delete-trailing-whitespace + some copyright update
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 470
diff changeset
   281
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   282
    def mytest_repr(self):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   283
        """return a representation of this step suitable for test"""
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   284
        return (self.__class__.__name__, self.limit, self.offset)
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   285
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   286
341
0a426be2f3a2 fixes for some cases of neged inlined relations, may have to use (new) IntersectStep. XXX: write IntersectFetchStep
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents: 257
diff changeset
   287
class IntersectStep(UnionStep):
0a426be2f3a2 fixes for some cases of neged inlined relations, may have to use (new) IntersectStep. XXX: write IntersectFetchStep
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents: 257
diff changeset
   288
    """return intersection of results of child in-memory steps (e.g. OneFetchStep / AggrStep)"""
1802
d628defebc17 delete-trailing-whitespace + some copyright update
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 470
diff changeset
   289
341
0a426be2f3a2 fixes for some cases of neged inlined relations, may have to use (new) IntersectStep. XXX: write IntersectFetchStep
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents: 257
diff changeset
   290
    def execute(self):
0a426be2f3a2 fixes for some cases of neged inlined relations, may have to use (new) IntersectStep. XXX: write IntersectFetchStep
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents: 257
diff changeset
   291
        """execute this step"""
0a426be2f3a2 fixes for some cases of neged inlined relations, may have to use (new) IntersectStep. XXX: write IntersectFetchStep
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents: 257
diff changeset
   292
        result = set()
0a426be2f3a2 fixes for some cases of neged inlined relations, may have to use (new) IntersectStep. XXX: write IntersectFetchStep
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents: 257
diff changeset
   293
        for step in self.children:
0a426be2f3a2 fixes for some cases of neged inlined relations, may have to use (new) IntersectStep. XXX: write IntersectFetchStep
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents: 257
diff changeset
   294
            result &= frozenset(step.execute())
0a426be2f3a2 fixes for some cases of neged inlined relations, may have to use (new) IntersectStep. XXX: write IntersectFetchStep
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents: 257
diff changeset
   295
        result = list(result)
0a426be2f3a2 fixes for some cases of neged inlined relations, may have to use (new) IntersectStep. XXX: write IntersectFetchStep
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents: 257
diff changeset
   296
        if self.offset:
448
ce91d0f23ed2 pylint fixes
sylvain.thenault@logilab.fr
parents: 341
diff changeset
   297
            result = result[self.offset:]
341
0a426be2f3a2 fixes for some cases of neged inlined relations, may have to use (new) IntersectStep. XXX: write IntersectFetchStep
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents: 257
diff changeset
   298
        if self.limit:
470
sylvain.thenault@logilab.fr
parents: 449
diff changeset
   299
            result = result[:self.limit]
341
0a426be2f3a2 fixes for some cases of neged inlined relations, may have to use (new) IntersectStep. XXX: write IntersectFetchStep
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents: 257
diff changeset
   300
        return result
0a426be2f3a2 fixes for some cases of neged inlined relations, may have to use (new) IntersectStep. XXX: write IntersectFetchStep
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents: 257
diff changeset
   301
0a426be2f3a2 fixes for some cases of neged inlined relations, may have to use (new) IntersectStep. XXX: write IntersectFetchStep
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents: 257
diff changeset
   302
257
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   303
class UnionFetchStep(Step):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   304
    """union results of child steps using temporary tables (e.g. FetchStep)"""
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   305
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   306
    def execute(self):
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   307
        """execute this step"""
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   308
        self.execute_children()
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   309
4c7d3af7e94d restore multi-sources capabilities
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   310
341
0a426be2f3a2 fixes for some cases of neged inlined relations, may have to use (new) IntersectStep. XXX: write IntersectFetchStep
Sylvain Thenault <sylvain.thenault@logilab.fr>
parents: 257
diff changeset
   311
__all__ = ('FetchStep', 'AggrStep', 'UnionStep', 'UnionFetchStep', 'IntersectStep')