web/views/cwsources.py
author Julien Cristau <julien.cristau@logilab.fr>
Fri, 21 Oct 2011 14:32:37 +0200
changeset 7995 9a9f35ef418c
parent 7994 af3fb709c061
child 8136 273d8a03700c
permissions -rw-r--r--
Record a log of datafeed source imports (closes #2026097) Formatting, css and js stolen from narval.
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
6944
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
     1
# copyright 2010-2011 LOGILAB S.A. (Paris, FRANCE), all rights reserved.
6724
24bf6f181d0e [pyro source] store pyro source mapping file into the database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
     2
# contact http://www.logilab.fr/ -- mailto:contact@logilab.fr
24bf6f181d0e [pyro source] store pyro source mapping file into the database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
     3
#
24bf6f181d0e [pyro source] store pyro source mapping file into the database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
     4
# This file is part of CubicWeb.
24bf6f181d0e [pyro source] store pyro source mapping file into the database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
     5
#
24bf6f181d0e [pyro source] store pyro source mapping file into the database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
     6
# CubicWeb is free software: you can redistribute it and/or modify it under the
24bf6f181d0e [pyro source] store pyro source mapping file into the database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
     7
# terms of the GNU Lesser General Public License as published by the Free
24bf6f181d0e [pyro source] store pyro source mapping file into the database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
     8
# Software Foundation, either version 2.1 of the License, or (at your option)
24bf6f181d0e [pyro source] store pyro source mapping file into the database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
     9
# any later version.
24bf6f181d0e [pyro source] store pyro source mapping file into the database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    10
#
24bf6f181d0e [pyro source] store pyro source mapping file into the database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    11
# CubicWeb is distributed in the hope that it will be useful, but WITHOUT
24bf6f181d0e [pyro source] store pyro source mapping file into the database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    12
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
24bf6f181d0e [pyro source] store pyro source mapping file into the database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    13
# FOR A PARTICULAR PURPOSE.  See the GNU Lesser General Public License for more
24bf6f181d0e [pyro source] store pyro source mapping file into the database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    14
# details.
24bf6f181d0e [pyro source] store pyro source mapping file into the database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    15
#
24bf6f181d0e [pyro source] store pyro source mapping file into the database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    16
# You should have received a copy of the GNU Lesser General Public License along
24bf6f181d0e [pyro source] store pyro source mapping file into the database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    17
# with CubicWeb.  If not, see <http://www.gnu.org/licenses/>.
6944
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
    18
"""Specific views for data sources and related entities (eg CWSource,
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
    19
CWSourceHostConfig, CWSourceSchemaConfig).
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
    20
"""
6724
24bf6f181d0e [pyro source] store pyro source mapping file into the database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    21
24bf6f181d0e [pyro source] store pyro source mapping file into the database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    22
__docformat__ = "restructuredtext en"
24bf6f181d0e [pyro source] store pyro source mapping file into the database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    23
_ = unicode
24bf6f181d0e [pyro source] store pyro source mapping file into the database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    24
7995
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
    25
import logging
6724
24bf6f181d0e [pyro source] store pyro source mapping file into the database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    26
from itertools import repeat, chain
7995
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
    27
from logilab.mtconverter import xml_escape
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
    28
from logilab.common.decorators import cachedproperty
6724
24bf6f181d0e [pyro source] store pyro source mapping file into the database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    29
7995
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
    30
from cubicweb import Unauthorized, tags
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
    31
from cubicweb.utils import make_uid
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
    32
from cubicweb.selectors import (is_instance, score_entity, has_related_entities,
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
    33
                                match_user_groups, match_kwargs, match_view)
6962
220e32f058be [management ui] add views to manage sources similarly to users
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
    34
from cubicweb.view import EntityView, StartupView
6724
24bf6f181d0e [pyro source] store pyro source mapping file into the database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    35
from cubicweb.schema import META_RTYPES, VIRTUAL_RTYPES, display_name
7995
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
    36
from cubicweb.web import uicfg, formwidgets as wdgs, facet
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
    37
from cubicweb.web.views import add_etype_button
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
    38
from cubicweb.web.views import (tabs, actions, ibreadcrumbs, navigation,
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
    39
                                tableview, pyviews)
6962
220e32f058be [management ui] add views to manage sources similarly to users
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
    40
7019
46413742f9b5 [source ui] uicfg tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6962
diff changeset
    41
46413742f9b5 [source ui] uicfg tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6962
diff changeset
    42
_abaa = uicfg.actionbox_appearsin_addmenu
7832
953f224357af [sources management] nicer source management view and breadcrumbs (closes #1946329)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7806
diff changeset
    43
# there are explicit 'add' buttons for those
7019
46413742f9b5 [source ui] uicfg tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6962
diff changeset
    44
_abaa.tag_object_of(('CWSourceSchemaConfig', 'cw_schema', '*'), False)
46413742f9b5 [source ui] uicfg tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6962
diff changeset
    45
_abaa.tag_object_of(('CWSourceSchemaConfig', 'cw_for_source', '*'), False)
7832
953f224357af [sources management] nicer source management view and breadcrumbs (closes #1946329)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7806
diff changeset
    46
_abaa.tag_object_of(('CWSourceSchemaConfig', 'cw_host_config_of', '*'), False)
7995
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
    47
_abaa.tag_object_of(('CWDataImport', 'cw_import_of', '*'), False)
7019
46413742f9b5 [source ui] uicfg tweaks
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6962
diff changeset
    48
7524
c019c3426049 [ui, sources] configure source form: hide synchronizing and mapping entities, use simple text input for parser
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7375
diff changeset
    49
_afs = uicfg.autoform_section
c019c3426049 [ui, sources] configure source form: hide synchronizing and mapping entities, use simple text input for parser
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7375
diff changeset
    50
_afs.tag_object_of(('*', 'cw_for_source', 'CWSource'), 'main', 'hidden')
7832
953f224357af [sources management] nicer source management view and breadcrumbs (closes #1946329)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7806
diff changeset
    51
7524
c019c3426049 [ui, sources] configure source form: hide synchronizing and mapping entities, use simple text input for parser
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7375
diff changeset
    52
_affk = uicfg.autoform_field_kwargs
c019c3426049 [ui, sources] configure source form: hide synchronizing and mapping entities, use simple text input for parser
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7375
diff changeset
    53
_affk.tag_attribute(('CWSource', 'parser'), {'widget': wdgs.TextInput})
c019c3426049 [ui, sources] configure source form: hide synchronizing and mapping entities, use simple text input for parser
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7375
diff changeset
    54
6962
220e32f058be [management ui] add views to manage sources similarly to users
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
    55
# source primary views #########################################################
6724
24bf6f181d0e [pyro source] store pyro source mapping file into the database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    56
6944
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
    57
_pvs = uicfg.primaryview_section
7832
953f224357af [sources management] nicer source management view and breadcrumbs (closes #1946329)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7806
diff changeset
    58
_pvs.tag_attribute(('CWSource', 'name'), 'hidden')
6944
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
    59
_pvs.tag_object_of(('*', 'cw_for_source', 'CWSource'), 'hidden')
7832
953f224357af [sources management] nicer source management view and breadcrumbs (closes #1946329)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7806
diff changeset
    60
_pvs.tag_object_of(('*', 'cw_host_config_of', 'CWSource'), 'hidden')
953f224357af [sources management] nicer source management view and breadcrumbs (closes #1946329)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7806
diff changeset
    61
953f224357af [sources management] nicer source management view and breadcrumbs (closes #1946329)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7806
diff changeset
    62
_pvdc = uicfg.primaryview_display_ctrl
953f224357af [sources management] nicer source management view and breadcrumbs (closes #1946329)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7806
diff changeset
    63
_pvdc.tag_attribute(('CWSource', 'type'), {'vid': 'attribute'})# disable reledit
953f224357af [sources management] nicer source management view and breadcrumbs (closes #1946329)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7806
diff changeset
    64
953f224357af [sources management] nicer source management view and breadcrumbs (closes #1946329)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7806
diff changeset
    65
_rc = uicfg.reledit_ctrl
7846
dd3f0871d8b7 [primary view] add 'verbatim' attribute view (closes #1947475)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7832
diff changeset
    66
_rc.tag_attribute(('CWSource', 'config'), {'rvid': 'verbatimattr'})
dd3f0871d8b7 [primary view] add 'verbatim' attribute view (closes #1947475)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7832
diff changeset
    67
_rc.tag_attribute(('CWSourceHostConfig', 'config'), {'rvid': 'verbatimattr'})
dd3f0871d8b7 [primary view] add 'verbatim' attribute view (closes #1947475)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7832
diff changeset
    68
_rc.tag_attribute(('CWSourceSchemaConfig', 'options'), {'rvid': 'verbatimattr'})
6944
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
    69
6724
24bf6f181d0e [pyro source] store pyro source mapping file into the database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    70
24bf6f181d0e [pyro source] store pyro source mapping file into the database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    71
class CWSourcePrimaryView(tabs.TabbedPrimaryView):
24bf6f181d0e [pyro source] store pyro source mapping file into the database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    72
    __select__ = is_instance('CWSource')
7995
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
    73
    tabs = [_('cwsource-main'), _('cwsource-mapping'), _('cwsource-imports')]
6724
24bf6f181d0e [pyro source] store pyro source mapping file into the database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    74
    default_tab = 'cwsource-main'
24bf6f181d0e [pyro source] store pyro source mapping file into the database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    75
24bf6f181d0e [pyro source] store pyro source mapping file into the database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    76
24bf6f181d0e [pyro source] store pyro source mapping file into the database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    77
class CWSourceMainTab(tabs.PrimaryTab):
24bf6f181d0e [pyro source] store pyro source mapping file into the database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    78
    __regid__ = 'cwsource-main'
7995
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
    79
    __select__ = is_instance('CWSource')
6724
24bf6f181d0e [pyro source] store pyro source mapping file into the database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    80
7832
953f224357af [sources management] nicer source management view and breadcrumbs (closes #1946329)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7806
diff changeset
    81
    def render_entity_attributes(self, entity):
953f224357af [sources management] nicer source management view and breadcrumbs (closes #1946329)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7806
diff changeset
    82
        super(CWSourceMainTab, self).render_entity_attributes(entity)
953f224357af [sources management] nicer source management view and breadcrumbs (closes #1946329)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7806
diff changeset
    83
        self.w(add_etype_button(self._cw, 'CWSourceHostConfig',
953f224357af [sources management] nicer source management view and breadcrumbs (closes #1946329)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7806
diff changeset
    84
                                __linkto='cw_host_config_of:%s:subject' % entity.eid,
953f224357af [sources management] nicer source management view and breadcrumbs (closes #1946329)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7806
diff changeset
    85
                                __redirectpath=entity.rest_path()))
953f224357af [sources management] nicer source management view and breadcrumbs (closes #1946329)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7806
diff changeset
    86
        try:
953f224357af [sources management] nicer source management view and breadcrumbs (closes #1946329)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7806
diff changeset
    87
            hostconfig = self._cw.execute(
953f224357af [sources management] nicer source management view and breadcrumbs (closes #1946329)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7806
diff changeset
    88
                'Any X, XC, XH WHERE X cw_host_config_of S, S eid %(s)s, '
953f224357af [sources management] nicer source management view and breadcrumbs (closes #1946329)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7806
diff changeset
    89
                'X config XC, X match_host XH', {'s': entity.eid})
953f224357af [sources management] nicer source management view and breadcrumbs (closes #1946329)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7806
diff changeset
    90
        except Unauthorized:
953f224357af [sources management] nicer source management view and breadcrumbs (closes #1946329)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7806
diff changeset
    91
            pass
953f224357af [sources management] nicer source management view and breadcrumbs (closes #1946329)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7806
diff changeset
    92
        else:
953f224357af [sources management] nicer source management view and breadcrumbs (closes #1946329)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7806
diff changeset
    93
            if hostconfig:
953f224357af [sources management] nicer source management view and breadcrumbs (closes #1946329)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7806
diff changeset
    94
                self.w(u'<h3>%s</h3>' % self._cw._('CWSourceHostConfig_plural'))
7992
4ff9f25cb06e [table views] closes #1986413: refactor TableView, EntityAttributesTableView, PyValTableView
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7935
diff changeset
    95
                self._cw.view('table', hostconfig, w=self.w,
4ff9f25cb06e [table views] closes #1986413: refactor TableView, EntityAttributesTableView, PyValTableView
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7935
diff changeset
    96
                              displaycols=range(2),
4ff9f25cb06e [table views] closes #1986413: refactor TableView, EntityAttributesTableView, PyValTableView
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7935
diff changeset
    97
                              cellvids={1: 'editable-final'})
7832
953f224357af [sources management] nicer source management view and breadcrumbs (closes #1946329)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7806
diff changeset
    98
6724
24bf6f181d0e [pyro source] store pyro source mapping file into the database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
    99
6944
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   100
MAPPED_SOURCE_TYPES = set( ('pyrorql', 'datafeed') )
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   101
6724
24bf6f181d0e [pyro source] store pyro source mapping file into the database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   102
class CWSourceMappingTab(EntityView):
24bf6f181d0e [pyro source] store pyro source mapping file into the database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   103
    __regid__ = 'cwsource-mapping'
7995
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   104
    __select__ = (is_instance('CWSource')
6944
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   105
                  & match_user_groups('managers')
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   106
                  & score_entity(lambda x:x.type in MAPPED_SOURCE_TYPES))
6724
24bf6f181d0e [pyro source] store pyro source mapping file into the database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   107
24bf6f181d0e [pyro source] store pyro source mapping file into the database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   108
    def entity_call(self, entity):
24bf6f181d0e [pyro source] store pyro source mapping file into the database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   109
        _ = self._cw._
6944
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   110
        self.w('<h3>%s</h3>' % _('Entity and relation supported by this source'))
7806
aa30c665bd06 [refactoring] introduce add_etype_button function to properly create button to add a new entity
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7804
diff changeset
   111
        self.w(add_etype_button(self._cw, 'CWSourceSchemaConfig',
aa30c665bd06 [refactoring] introduce add_etype_button function to properly create button to add a new entity
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7804
diff changeset
   112
                                __linkto='cw_for_source:%s:subject' % entity.eid))
aa30c665bd06 [refactoring] introduce add_etype_button function to properly create button to add a new entity
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7804
diff changeset
   113
        self.w(u'<div class="clear"></div>')
6944
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   114
        rset = self._cw.execute(
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   115
            'Any X, SCH, XO ORDERBY ET WHERE X options XO, X cw_for_source S, S eid %(s)s, '
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   116
            'X cw_schema SCH, SCH is ET', {'s': entity.eid})
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   117
        self.wview('table', rset, 'noresult')
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   118
        # self.w('<h3>%s</h3>' % _('Relations that should not be crossed'))
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   119
        # self.w('<p>%s</p>' % _(
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   120
        #     'By default, when a relation is not supported by a source, it is '
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   121
        #     'supposed that a local relation may point to an entity from the '
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   122
        #     'external source. Relations listed here won\'t have this '
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   123
        #     '"crossing" behaviour.'))
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   124
        # self.wview('list', entity.related('cw_dont_cross'), 'noresult')
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   125
        # self.w('<h3>%s</h3>' % _('Relations that can be crossed'))
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   126
        # self.w('<p>%s</p>' % _(
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   127
        #     'By default, when a relation is supported by a source, it is '
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   128
        #     'supposed that a local relation can\'t point to an entity from the '
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   129
        #     'external source. Relations listed here may have this '
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   130
        #     '"crossing" behaviour anyway.'))
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   131
        # self.wview('list', entity.related('cw_may_cross'), 'noresult')
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   132
        checker = MAPPING_CHECKERS.get(entity.type, MappingChecker)(entity)
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   133
        checker.check()
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   134
        if (checker.errors or checker.warnings or checker.infos):
7806
aa30c665bd06 [refactoring] introduce add_etype_button function to properly create button to add a new entity
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7804
diff changeset
   135
            self.w('<h2>%s</h2>' % _('Detected problems'))
aa30c665bd06 [refactoring] introduce add_etype_button function to properly create button to add a new entity
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7804
diff changeset
   136
            errors = zip(repeat(_('error')), checker.errors)
aa30c665bd06 [refactoring] introduce add_etype_button function to properly create button to add a new entity
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7804
diff changeset
   137
            warnings = zip(repeat(_('warning')), checker.warnings)
aa30c665bd06 [refactoring] introduce add_etype_button function to properly create button to add a new entity
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7804
diff changeset
   138
            infos = zip(repeat(_('warning')), checker.infos)
aa30c665bd06 [refactoring] introduce add_etype_button function to properly create button to add a new entity
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7804
diff changeset
   139
            self.wview('pyvaltable', pyvalue=chain(errors, warnings, infos))
6724
24bf6f181d0e [pyro source] store pyro source mapping file into the database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   140
6944
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   141
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   142
class MappingChecker(object):
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   143
    def __init__(self, cwsource):
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   144
        self.cwsource = cwsource
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   145
        self.errors = []
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   146
        self.warnings = []
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   147
        self.infos = []
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   148
        self.schema = cwsource._cw.vreg.schema
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   149
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   150
    def init(self):
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   151
        # supported entity types
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   152
        self.sentities = set()
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   153
        # supported relations
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   154
        self.srelations = {}
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   155
        # avoid duplicated messages
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   156
        self.seen = set()
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   157
        # first get mapping as dict/sets
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   158
        for schemacfg in self.cwsource.reverse_cw_for_source:
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   159
            self.init_schemacfg(schemacfg)
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   160
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   161
    def init_schemacfg(self, schemacfg):
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   162
        cwerschema = schemacfg.schema
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   163
        if cwerschema.__regid__ == 'CWEType':
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   164
            self.sentities.add(cwerschema.name)
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   165
        elif cwerschema.__regid__ == 'CWRType':
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   166
            assert not cwerschema.name in self.srelations
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   167
            self.srelations[cwerschema.name] = None
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   168
        else: # CWAttribute/CWRelation
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   169
            self.srelations.setdefault(cwerschema.rtype.name, []).append(
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   170
                (cwerschema.stype.name, cwerschema.otype.name) )
7375
c8a8fb32733b [cw source ui] fix mapping checking for datafeed source (avoid false positive error, but doesn't check that much things)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7019
diff changeset
   171
            self.sentities.add(cwerschema.stype.name)
c8a8fb32733b [cw source ui] fix mapping checking for datafeed source (avoid false positive error, but doesn't check that much things)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7019
diff changeset
   172
            self.sentities.add(cwerschema.otype.name)
6944
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   173
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   174
    def check(self):
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   175
        self.init()
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   176
        error = self.errors.append
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   177
        warning = self.warnings.append
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   178
        info = self.infos.append
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   179
        for etype in self.sentities:
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   180
            eschema = self.schema[etype]
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   181
            for rschema, ttypes, role in eschema.relation_definitions():
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   182
                if rschema in META_RTYPES:
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   183
                    continue
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   184
                ttypes = [ttype for ttype in ttypes if ttype in self.sentities]
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   185
                if not rschema in self.srelations:
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   186
                    for ttype in ttypes:
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   187
                        rdef = rschema.role_rdef(etype, ttype, role)
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   188
                        self.seen.add(rdef)
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   189
                        if rdef.role_cardinality(role) in '1+':
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   190
                            error(_('relation %(type)s with %(etype)s as %(role)s '
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   191
                                    'and target type %(target)s is mandatory but '
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   192
                                    'not supported') %
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   193
                                  {'rtype': rschema, 'etype': etype, 'role': role,
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   194
                                   'target': ttype})
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   195
                        elif ttype in self.sentities:
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   196
                            warning(_('%s could be supported') % rdef)
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   197
                elif not ttypes:
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   198
                    warning(_('relation %(rtype)s with %(etype)s as %(role)s is '
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   199
                              'supported but no target type supported') %
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   200
                            {'rtype': rschema, 'role': role, 'etype': etype})
7375
c8a8fb32733b [cw source ui] fix mapping checking for datafeed source (avoid false positive error, but doesn't check that much things)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7019
diff changeset
   201
        for rtype, rdefs in self.srelations.iteritems():
c8a8fb32733b [cw source ui] fix mapping checking for datafeed source (avoid false positive error, but doesn't check that much things)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7019
diff changeset
   202
            if rdefs is None:
c8a8fb32733b [cw source ui] fix mapping checking for datafeed source (avoid false positive error, but doesn't check that much things)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7019
diff changeset
   203
                rschema = self.schema[rtype]
c8a8fb32733b [cw source ui] fix mapping checking for datafeed source (avoid false positive error, but doesn't check that much things)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7019
diff changeset
   204
                for subj, obj in rschema.rdefs:
c8a8fb32733b [cw source ui] fix mapping checking for datafeed source (avoid false positive error, but doesn't check that much things)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7019
diff changeset
   205
                    if subj in self.sentities and obj in self.sentities:
c8a8fb32733b [cw source ui] fix mapping checking for datafeed source (avoid false positive error, but doesn't check that much things)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7019
diff changeset
   206
                        break
c8a8fb32733b [cw source ui] fix mapping checking for datafeed source (avoid false positive error, but doesn't check that much things)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7019
diff changeset
   207
                else:
c8a8fb32733b [cw source ui] fix mapping checking for datafeed source (avoid false positive error, but doesn't check that much things)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7019
diff changeset
   208
                    error(_('relation %s is supported but none of its definitions '
c8a8fb32733b [cw source ui] fix mapping checking for datafeed source (avoid false positive error, but doesn't check that much things)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7019
diff changeset
   209
                            'matches supported entities') % rtype)
6944
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   210
        self.custom_check()
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   211
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   212
    def custom_check(self):
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   213
        pass
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   214
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   215
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   216
class PyroRQLMappingChecker(MappingChecker):
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   217
    """pyrorql source mapping checker"""
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   218
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   219
    def init(self):
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   220
        self.dontcross = set()
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   221
        self.maycross = set()
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   222
        super(PyroRQLMappingChecker, self).init()
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   223
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   224
    def init_schemacfg(self, schemacfg):
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   225
        options = schemacfg.options or ()
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   226
        if 'dontcross' in options:
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   227
            self.dontcross.add(schemacfg.schema.name)
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   228
        else:
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   229
            super(PyroRQLMappingChecker, self).init_schemacfg(schemacfg)
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   230
            if 'maycross' in options:
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   231
                self.maycross.add(schemacfg.schema.name)
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   232
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   233
    def custom_check(self):
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   234
        error = self.errors.append
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   235
        info = self.infos.append
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   236
        for etype in self.sentities:
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   237
            eschema = self.schema[etype]
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   238
            for rschema, ttypes, role in eschema.relation_definitions():
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   239
                if rschema in META_RTYPES:
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   240
                    continue
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   241
                if not rschema in self.srelations:
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   242
                    if rschema not in self.dontcross:
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   243
                        if role == 'subject' and rschema.inlined:
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   244
                            error(_('inlined relation %(rtype)s of %(etype)s '
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   245
                                    'should be supported') %
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   246
                                  {'rtype': rschema, 'etype': etype})
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   247
                        elif (rschema not in self.seen and rschema not in self.maycross):
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   248
                            info(_('you may want to specify something for %s') %
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   249
                                 rschema)
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   250
                            self.seen.add(rschema)
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   251
                elif rschema in self.maycross and rschema.inlined:
6724
24bf6f181d0e [pyro source] store pyro source mapping file into the database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   252
                    error(_('you should un-inline relation %s which is '
24bf6f181d0e [pyro source] store pyro source mapping file into the database
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents:
diff changeset
   253
                            'supported and may be crossed ') % rschema)
6944
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   254
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   255
MAPPING_CHECKERS = {
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   256
    'pyrorql': PyroRQLMappingChecker,
0cf10429ad39 [sources] rewrite the way pyrorql mapping are stored in the database so it can be reused for other sources (eg datafeed+cwxml)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6724
diff changeset
   257
    }
6962
220e32f058be [management ui] add views to manage sources similarly to users
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   258
7995
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   259
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   260
class CWSourceImportsTab(EntityView):
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   261
    __regid__ = 'cwsource-imports'
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   262
    __select__ = (is_instance('CWSource')
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   263
                  & has_related_entities('cw_import_of', 'object'))
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   264
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   265
    def entity_call(self, entity):
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   266
        rset = self._cw.execute('Any X, XST, XET, XS ORDERBY XST DESC WHERE '
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   267
                                'X cw_import_of S, S eid %(s)s, X status XS, '
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   268
                                'X start_timestamp XST, X end_timestamp XET',
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   269
                                {'s': entity.eid})
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   270
        self._cw.view('cw.imports-table', rset, w=self.w)
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   271
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   272
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   273
class CWImportsTable(tableview.EntityTableView):
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   274
    __regid__ = 'cw.imports-table'
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   275
    __select__ = is_instance('CWDataImport')
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   276
    columns = ['import', 'start_timestamp', 'end_timestamp']
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   277
    column_renderers = {'import': tableview.MainEntityColRenderer()}
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   278
    layout_args = {'display_filter': 'top'}
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   279
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   280
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   281
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   282
6962
220e32f058be [management ui] add views to manage sources similarly to users
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   283
# sources management view ######################################################
220e32f058be [management ui] add views to manage sources similarly to users
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   284
220e32f058be [management ui] add views to manage sources similarly to users
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   285
class ManageSourcesAction(actions.ManagersAction):
220e32f058be [management ui] add views to manage sources similarly to users
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   286
    __regid__ = 'cwsource'
220e32f058be [management ui] add views to manage sources similarly to users
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   287
    title = _('data sources')
220e32f058be [management ui] add views to manage sources similarly to users
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   288
    category = 'manage'
220e32f058be [management ui] add views to manage sources similarly to users
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   289
7994
af3fb709c061 [management table] upgrade sources groups management table to new api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7992
diff changeset
   290
af3fb709c061 [management table] upgrade sources groups management table to new api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7992
diff changeset
   291
class CWSourcesManagementView(StartupView):
af3fb709c061 [management table] upgrade sources groups management table to new api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7992
diff changeset
   292
    __regid__ = 'cw.sources-management'
af3fb709c061 [management table] upgrade sources groups management table to new api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7992
diff changeset
   293
    rql = ('Any S,ST,SP,SD,SN ORDERBY SN WHERE S is CWSource, S name SN, S type ST, '
7455
694b21f0fc62 [views] add columns parser and latest_retrieval to cwsource admin view
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7375
diff changeset
   294
           'S latest_retrieval SD, S parser SP')
6962
220e32f058be [management ui] add views to manage sources similarly to users
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   295
    title = _('data sources management')
220e32f058be [management ui] add views to manage sources similarly to users
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   296
220e32f058be [management ui] add views to manage sources similarly to users
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   297
    def call(self, **kwargs):
220e32f058be [management ui] add views to manage sources similarly to users
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6944
diff changeset
   298
        self.w('<h1>%s</h1>' % self._cw._(self.title))
7852
cb3f700399b2 [schema, cleanup] one more case for add_etype_button
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7846
diff changeset
   299
        self.w(add_etype_button(self._cw, 'CWSource'))
cb3f700399b2 [schema, cleanup] one more case for add_etype_button
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7846
diff changeset
   300
        self.w(u'<div class="clear"></div>')
7994
af3fb709c061 [management table] upgrade sources groups management table to new api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7992
diff changeset
   301
        self.wview('cw.sources-table', self._cw.execute(self.rql))
af3fb709c061 [management table] upgrade sources groups management table to new api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7992
diff changeset
   302
af3fb709c061 [management table] upgrade sources groups management table to new api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7992
diff changeset
   303
af3fb709c061 [management table] upgrade sources groups management table to new api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7992
diff changeset
   304
class CWSourcesTable(tableview.EntityTableView):
af3fb709c061 [management table] upgrade sources groups management table to new api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7992
diff changeset
   305
    __regid__ = 'cw.sources-table'
af3fb709c061 [management table] upgrade sources groups management table to new api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7992
diff changeset
   306
    __select__ = is_instance('CWSource')
7995
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   307
    columns = ['source', 'type', 'parser', 'latest_retrieval', 'latest_import']
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   308
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   309
    class LatestImportColRenderer(tableview.EntityTableColRenderer):
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   310
        def render_cell(self, w, rownum):
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   311
            entity = self.entity(rownum)
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   312
            rset = self._cw.execute('Any X,XS,XST ORDERBY XST DESC LIMIT 1 WHERE '
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   313
                                    'X cw_import_of S, S eid %(s)s, X status XS, '
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   314
                                    'X start_timestamp XST', {'s': entity.eid})
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   315
            if rset:
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   316
                self._cw.view('incontext', rset, row=0, w=w)
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   317
            else:
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   318
                w(self.empty_cell_content)
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   319
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   320
    column_renderers = {
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   321
        'source': tableview.MainEntityColRenderer(),
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   322
        'latest_import': LatestImportColRenderer(header=_('latest import'),
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   323
                                                 sortable=False)
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   324
        }
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   325
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   326
# datafeed source import #######################################################
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   327
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   328
REVERSE_SEVERITIES = {
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   329
    logging.DEBUG :   _('DEBUG'),
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   330
    logging.INFO :    _('INFO'),
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   331
    logging.WARNING : _('WARNING'),
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   332
    logging.ERROR :   _('ERROR'),
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   333
    logging.FATAL :   _('FATAL')
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   334
}
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   335
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   336
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   337
def log_to_table(req, rawdata):
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   338
    data = []
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   339
    for msg_idx, msg in enumerate(rawdata.split('<br/>')):
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   340
        record = msg.strip()
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   341
        if not record:
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   342
            continue
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   343
        try:
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   344
            severity, url, line, msg = record.split('\t', 3)
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   345
        except ValueError:
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   346
            req.warning('badly formated log %s' % record)
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   347
            url = line = u''
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   348
            severity = logging.DEBUG
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   349
            msg = record
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   350
        data.append( (severity, url, line, msg) )
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   351
    return data
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   352
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   353
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   354
class LogTableLayout(tableview.TableLayout):
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   355
    __select__ = match_view('cw.log.table')
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   356
    needs_js = tableview.TableLayout.needs_js + ('cubicweb.log.js',)
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   357
    needs_css = tableview.TableLayout.needs_css + ('cubicweb.log.css',)
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   358
    columns_css = {
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   359
        0: 'logSeverity',
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   360
        1: 'logPath',
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   361
        2: 'logLine',
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   362
        3: 'logMsg',
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   363
        }
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   364
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   365
    def render_table(self, w, actions, paginate):
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   366
        default_level = self.view.cw_extra_kwargs['default_level']
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   367
        if default_level != 'Debug':
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   368
            self._cw.add_onload('$("select.logFilter").val("%s").change();'
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   369
                           % self._cw.form.get('logLevel', default_level))
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   370
        w(u'\n<form action="#"><fieldset>')
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   371
        w(u'<label>%s</label>' % self._cw._(u'Message threshold'))
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   372
        w(u'<select class="log_filter" onchange="filterLog(\'%s\', this.options[this.selectedIndex].value)">'
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   373
          % self.view.domid)
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   374
        for level in ('Debug', 'Info', 'Warning', 'Error', 'Fatal'):
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   375
            w('<option value="%s">%s</option>' % (level, self._cw._(level)))
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   376
        w(u'</select>')
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   377
        w(u'</fieldset></form>')
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   378
        super(LogTableLayout, self).render_table(w, actions, paginate)
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   379
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   380
    def table_attributes(self):
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   381
        attrs = super(LogTableLayout, self).table_attributes()
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   382
        attrs['id'] = 'table'+self.view.domid
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   383
        return attrs
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   384
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   385
    def row_attributes(self, rownum):
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   386
        attrs = super(LogTableLayout, self).row_attributes(rownum)
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   387
        attrs['id'] = 'log_msg_%i' % rownum
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   388
        severityname = REVERSE_SEVERITIES[int(self.view.pyvalue[rownum][0])]
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   389
        attrs['class'] = 'log%s' % severityname.capitalize()
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   390
        return attrs
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   391
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   392
    def cell_attributes(self, rownum, colnum, colid):
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   393
        attrs = super(LogTableLayout, self).cell_attributes(rownum, colnum, colid)
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   394
        attrs['class'] = self.columns_css[colnum]
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   395
        return attrs
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   396
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   397
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   398
class LogTable(pyviews.PyValTableView):
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   399
    __regid__ = 'cw.log.table'
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   400
    headers = [_('severity'), _('url'), _('line'), _('message')]
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   401
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   402
    @cachedproperty
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   403
    def domid(self):
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   404
        return make_uid('logTable')
7994
af3fb709c061 [management table] upgrade sources groups management table to new api
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7992
diff changeset
   405
7995
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   406
    class SeverityRenderer(pyviews.PyValTableColRenderer):
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   407
        def render_cell(self, w, rownum):
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   408
            severity = self.data[rownum][0]
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   409
            w(u'<a class="internallink" href="javascript:;" title="%(title)s" '
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   410
              u'''onclick="document.location.hash='%(msg_id)s';">&#182;</a>'''
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   411
              u'&#160;%(severity)s' % {
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   412
                'severity': self._cw._(REVERSE_SEVERITIES[int(severity)]),
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   413
                'title': self._cw._('permalink to this message'),
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   414
                'msg_id': 'log_msg_%i' % rownum,
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   415
            })
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   416
        def sortvalue(self, rownum):
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   417
            return int(self.data[rownum][0])
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   418
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   419
    class URLRenderer(pyviews.PyValTableColRenderer):
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   420
        def render_cell(self, w, rownum):
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   421
            url = self.data[rownum][1]
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   422
            w(url and tags.a(url, href=url) or u'&#160;')
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   423
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   424
    class LineRenderer(pyviews.PyValTableColRenderer):
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   425
        def render_cell(self, w, rownum):
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   426
            line = self.data[rownum][2]
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   427
            w(line or u'&#160;')
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   428
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   429
    class MessageRenderer(pyviews.PyValTableColRenderer):
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   430
        snip_over = 7
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   431
        def render_cell(self, w, rownum):
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   432
            msg = self.data[rownum][3]
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   433
            lines = msg.splitlines()
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   434
            if len(lines) <= self.snip_over:
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   435
                w(u'<pre class="rawtext">%s</pre>' % msg)
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   436
            else:
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   437
                # The make_uid argument has no specific meaning here.
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   438
                div_snip_id = make_uid(u'log_snip_')
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   439
                div_full_id = make_uid(u'log_full_')
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   440
                divs_id = (div_snip_id, div_full_id)
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   441
                snip = u'\n'.join((lines[0], lines[1],
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   442
                                   u'  ...',
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   443
                                   u'    %i more lines [double click to expand]' % (len(lines)-4),
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   444
                                   u'  ...',
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   445
                                   lines[-2], lines[-1]))
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   446
                divs = (
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   447
                        (div_snip_id, snip, u'expand', "class='collapsed'"),
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   448
                        (div_full_id, msg,  u'collapse', "class='hidden'")
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   449
                )
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   450
                for div_id, content, button, h_class in divs:
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   451
                    text = self._cw._(button)
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   452
                    js = u"toggleVisibility('%s'); toggleVisibility('%s');" % divs_id
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   453
                    w(u'<div id="%s" %s>' % (div_id, h_class))
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   454
                    w(u'<pre class="raw_test" ondblclick="javascript: %s" '
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   455
                      u'title="%s" style="display: block;">' % (js, text))
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   456
                    w(content)
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   457
                    w(u'</pre>')
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   458
                    w(u'</div>')
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   459
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   460
    column_renderers = {0: SeverityRenderer(),
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   461
                        1: URLRenderer(),
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   462
                        2: LineRenderer(),
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   463
                        3: MessageRenderer(),
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   464
                        }
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   465
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   466
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   467
class DataFeedSourceDataImport(EntityView):
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   468
    __select__ = EntityView.__select__ & match_kwargs('rtype')
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   469
    __regid__ = 'cw.formated_log'
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   470
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   471
    def cell_call(self, row, col, rtype, loglevel='Info', **kwargs):
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   472
        if 'dispctrl' in self.cw_extra_kwargs:
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   473
            loglevel = self.cw_extra_kwargs['dispctrl'].get('loglevel', loglevel)
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   474
        entity = self.cw_rset.get_entity(row, col)
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   475
        value = getattr(entity, rtype)
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   476
        if value:
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   477
            self._cw.view('cw.log.table', pyvalue=log_to_table(self._cw, value),
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   478
                          default_level=loglevel, w=self.w)
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   479
        else:
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   480
            self.w(self._cw._('no log to display'))
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   481
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   482
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   483
_pvs.tag_attribute(('CWDataImport', 'log'), 'relations')
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   484
_pvdc.tag_attribute(('CWDataImport', 'log'), {'vid': 'cw.formated_log'})
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   485
_pvs.tag_subject_of(('CWDataImport', 'cw_import_of', '*'), 'hidden') # in breadcrumbs
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   486
_pvs.tag_object_of(('*', 'cw_import_of', 'CWSource'), 'hidden') # in dedicated tab
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   487
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   488
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   489
class CWDataImportIPrevNextAdapter(navigation.IPrevNextAdapter):
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   490
    __select__ = is_instance('CWDataImport')
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   491
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   492
    def next_entity(self):
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   493
        if self.entity.start_timestamp is not None:
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   494
            # add NOT X eid %(e)s because > may not be enough
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   495
            rset = self._cw.execute(
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   496
                'Any X,XSTS ORDERBY 2 LIMIT 1 WHERE X is CWDataImport, '
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   497
                'X cw_import_of S, S eid %(s)s, NOT X eid %(e)s, '
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   498
                'X start_timestamp XSTS, X start_timestamp > %(sts)s',
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   499
                {'sts': self.entity.start_timestamp,
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   500
                 'e': self.entity.eid,
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   501
                 's': self.entity.cwsource.eid})
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   502
            if rset:
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   503
                return rset.get_entity(0, 0)
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   504
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   505
    def previous_entity(self):
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   506
        if self.entity.start_timestamp is not None:
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   507
            # add NOT X eid %(e)s because < may not be enough
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   508
            rset = self._cw.execute(
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   509
                'Any X,XSTS ORDERBY 2 DESC LIMIT 1 WHERE X is CWDataImport, '
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   510
                'X cw_import_of S, S eid %(s)s, NOT X eid %(e)s, '
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   511
                'X start_timestamp XSTS, X start_timestamp < %(sts)s',
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   512
                {'sts': self.entity.start_timestamp,
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   513
                 'e': self.entity.eid,
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   514
                 's': self.entity.cwsource.eid})
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   515
            if rset:
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   516
                return rset.get_entity(0, 0)
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   517
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   518
class CWDataImportStatusFacet(facet.AttributeFacet):
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   519
    __regid__ = 'datafeed.dataimport.status'
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   520
    __select__ = is_instance('CWDataImport')
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   521
    rtype = 'status'
7832
953f224357af [sources management] nicer source management view and breadcrumbs (closes #1946329)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7806
diff changeset
   522
953f224357af [sources management] nicer source management view and breadcrumbs (closes #1946329)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7806
diff changeset
   523
953f224357af [sources management] nicer source management view and breadcrumbs (closes #1946329)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7806
diff changeset
   524
# breadcrumbs configuration ####################################################
953f224357af [sources management] nicer source management view and breadcrumbs (closes #1946329)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7806
diff changeset
   525
953f224357af [sources management] nicer source management view and breadcrumbs (closes #1946329)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7806
diff changeset
   526
class CWsourceConfigIBreadCrumbsAdapter(ibreadcrumbs.IBreadCrumbsAdapter):
953f224357af [sources management] nicer source management view and breadcrumbs (closes #1946329)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7806
diff changeset
   527
    __select__ = is_instance('CWSourceHostConfig', 'CWSourceSchemaConfig')
953f224357af [sources management] nicer source management view and breadcrumbs (closes #1946329)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7806
diff changeset
   528
    def parent_entity(self):
953f224357af [sources management] nicer source management view and breadcrumbs (closes #1946329)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7806
diff changeset
   529
        return self.entity.cwsource
7995
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   530
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   531
class CWDataImportIBreadCrumbsAdapter(ibreadcrumbs.IBreadCrumbsAdapter):
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   532
    __select__ = is_instance('CWDataImport')
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   533
    def parent_entity(self):
9a9f35ef418c Record a log of datafeed source imports (closes #2026097)
Julien Cristau <julien.cristau@logilab.fr>
parents: 7994
diff changeset
   534
        return self.entity.cw_import_of[0]