server/checkintegrity.py
author Rémi Cardona <remi.cardona@logilab.fr>
Mon, 21 Dec 2015 09:59:45 +0100
changeset 11014 9c9f5e913f9c
parent 10847 ce5403611cbe
child 11033 63d860a14a17
permissions -rw-r--r--
[devtools] Move attribute docstring to correct location
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
9797
4e640ab62f51 [tests/checkintegrity] use the new connection api
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9578
diff changeset
     1
# copyright 2003-2014 LOGILAB S.A. (Paris, FRANCE), all rights reserved.
5421
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5341
diff changeset
     2
# contact http://www.logilab.fr/ -- mailto:contact@logilab.fr
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5341
diff changeset
     3
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5341
diff changeset
     4
# This file is part of CubicWeb.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5341
diff changeset
     5
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5341
diff changeset
     6
# CubicWeb is free software: you can redistribute it and/or modify it under the
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5341
diff changeset
     7
# terms of the GNU Lesser General Public License as published by the Free
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5341
diff changeset
     8
# Software Foundation, either version 2.1 of the License, or (at your option)
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5341
diff changeset
     9
# any later version.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5341
diff changeset
    10
#
5424
8ecbcbff9777 replace logilab-common by CubicWeb in disclaimer
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5421
diff changeset
    11
# CubicWeb is distributed in the hope that it will be useful, but WITHOUT
5421
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5341
diff changeset
    12
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5341
diff changeset
    13
# FOR A PARTICULAR PURPOSE.  See the GNU Lesser General Public License for more
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5341
diff changeset
    14
# details.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5341
diff changeset
    15
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5341
diff changeset
    16
# You should have received a copy of the GNU Lesser General Public License along
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5341
diff changeset
    17
# with CubicWeb.  If not, see <http://www.gnu.org/licenses/>.
6127
747e423093fc [ms, c-c] new command checking for consistency / potentian flaws and enhancements of mapping file of a multi-sources instance
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6112
diff changeset
    18
"""Integrity checking tool for instances:
747e423093fc [ms, c-c] new command checking for consistency / potentian flaws and enhancements of mapping file of a multi-sources instance
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6112
diff changeset
    19
747e423093fc [ms, c-c] new command checking for consistency / potentian flaws and enhancements of mapping file of a multi-sources instance
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6112
diff changeset
    20
* integrity of a CubicWeb repository. Hum actually only the system database is
747e423093fc [ms, c-c] new command checking for consistency / potentian flaws and enhancements of mapping file of a multi-sources instance
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6112
diff changeset
    21
  checked.
5999
eaf8219f8b7d [migration] fix rename_entity_type to avoid to loose some relations on the way
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5954
diff changeset
    22
"""
10589
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 10365
diff changeset
    23
from __future__ import print_function
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 10365
diff changeset
    24
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    25
__docformat__ = "restructuredtext en"
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    26
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    27
import sys
1016
26387b836099 use datetime instead of mx.DateTime
sylvain.thenault@logilab.fr
parents: 713
diff changeset
    28
from datetime import datetime
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    29
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    30
from logilab.common.shellutils import ProgressBar
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    31
10813
ab626726a70a [syncschema] don't arbitrarily remove constraints from the in-memory schema
Julien Cristau <julien.cristau@logilab.fr>
parents: 10663
diff changeset
    32
from cubicweb.schema import PURE_VIRTUAL_RTYPES, VIRTUAL_RTYPES, UNIQUE_CONSTRAINTS
1251
af40e615dc89 introduce a 'cw_' prefix on entity table and column names so we don't conflict with sql or DBMS specific keywords
sylvain.thenault@logilab.fr
parents: 1161
diff changeset
    33
from cubicweb.server.sqlutils import SQL_PREFIX
af40e615dc89 introduce a 'cw_' prefix on entity table and column names so we don't conflict with sql or DBMS specific keywords
sylvain.thenault@logilab.fr
parents: 1161
diff changeset
    34
7035
8d2cf36bd79d [c-c db-check] factorize code by introducing notify_fixed dumb function
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6889
diff changeset
    35
def notify_fixed(fix):
8d2cf36bd79d [c-c db-check] factorize code by introducing notify_fixed dumb function
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6889
diff changeset
    36
    if fix:
7896
4c954e1e73ef [lint] remove uses of "print >> sys.stderr" (closes #1908571)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7815
diff changeset
    37
        sys.stderr.write(' [FIXED]')
4c954e1e73ef [lint] remove uses of "print >> sys.stderr" (closes #1908571)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7815
diff changeset
    38
    sys.stderr.write('\n')
7035
8d2cf36bd79d [c-c db-check] factorize code by introducing notify_fixed dumb function
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6889
diff changeset
    39
9577
c52441e4a3d7 [serverctl] use repoapi for db-check, add-source, rebuild-fti commands
Julien Cristau <julien.cristau@logilab.fr>
parents: 9574
diff changeset
    40
def has_eid(cnx, sqlcursor, eid, eids):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    41
    """return true if the eid is a valid eid"""
5341
0de53140bd29 [db-check] cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5340
diff changeset
    42
    if eid in eids:
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    43
        return eids[eid]
9469
032825bbacab [multi-sources-removal] Drop entities.source column
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9468
diff changeset
    44
    sqlcursor.execute('SELECT type FROM entities WHERE eid=%s' % eid)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    45
    try:
9469
032825bbacab [multi-sources-removal] Drop entities.source column
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 9468
diff changeset
    46
        etype = sqlcursor.fetchone()[0]
7815
2a164a9cf81c [exceptions] stop catching any exception in various places (closes #1942716)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7687
diff changeset
    47
    except Exception:
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    48
        eids[eid] = False
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    49
        return False
9577
c52441e4a3d7 [serverctl] use repoapi for db-check, add-source, rebuild-fti commands
Julien Cristau <julien.cristau@logilab.fr>
parents: 9574
diff changeset
    50
    if etype not in cnx.vreg.schema:
9171
be9596750678 Fix two crashes in db-check (closes #3024964)
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 8900
diff changeset
    51
        eids[eid] = False
be9596750678 Fix two crashes in db-check (closes #3024964)
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 8900
diff changeset
    52
        return False
1251
af40e615dc89 introduce a 'cw_' prefix on entity table and column names so we don't conflict with sql or DBMS specific keywords
sylvain.thenault@logilab.fr
parents: 1161
diff changeset
    53
    sqlcursor.execute('SELECT * FROM %s%s WHERE %seid=%s' % (SQL_PREFIX, etype,
af40e615dc89 introduce a 'cw_' prefix on entity table and column names so we don't conflict with sql or DBMS specific keywords
sylvain.thenault@logilab.fr
parents: 1161
diff changeset
    54
                                                             SQL_PREFIX, eid))
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    55
    result = sqlcursor.fetchall()
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    56
    if len(result) == 0:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    57
        eids[eid] = False
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    58
        return False
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    59
    elif len(result) > 1:
9492
c7fc56eecd1a English typography
Dimitri Papadopoulos <dimitri.papadopoulos@cea.fr>
parents: 9171
diff changeset
    60
        msg = ('  More than one entity with eid %s exists in source!\n'
c7fc56eecd1a English typography
Dimitri Papadopoulos <dimitri.papadopoulos@cea.fr>
parents: 9171
diff changeset
    61
               '  WARNING : Unable to fix this, do it yourself!\n')
7896
4c954e1e73ef [lint] remove uses of "print >> sys.stderr" (closes #1908571)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7815
diff changeset
    62
        sys.stderr.write(msg % eid)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    63
    eids[eid] = True
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    64
    return True
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    65
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    66
# XXX move to yams?
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    67
def etype_fti_containers(eschema, _done=None):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    68
    if _done is None:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    69
        _done = set()
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    70
    _done.add(eschema)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    71
    containers = tuple(eschema.fulltext_containers())
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    72
    if containers:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    73
        for rschema, target in containers:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    74
            if target == 'object':
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    75
                targets = rschema.objects(eschema)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    76
            else:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    77
                targets = rschema.subjects(eschema)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    78
            for targeteschema in targets:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    79
                if targeteschema in _done:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    80
                    continue
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    81
                _done.add(targeteschema)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    82
                for container in etype_fti_containers(targeteschema, _done):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    83
                    yield container
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    84
    else:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    85
        yield eschema
1802
d628defebc17 delete-trailing-whitespace + some copyright update
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 1398
diff changeset
    86
9574
2d4c4842bd04 [server] some s/session/cnx/
Julien Cristau <julien.cristau@logilab.fr>
parents: 9543
diff changeset
    87
def reindex_entities(schema, cnx, withpb=True, etypes=None):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    88
    """reindex all entities in the repository"""
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    89
    # deactivate modification_date hook since we don't want them
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    90
    # to be updated due to the reindexation
9574
2d4c4842bd04 [server] some s/session/cnx/
Julien Cristau <julien.cristau@logilab.fr>
parents: 9543
diff changeset
    91
    repo = cnx.repo
2d4c4842bd04 [server] some s/session/cnx/
Julien Cristau <julien.cristau@logilab.fr>
parents: 9543
diff changeset
    92
    dbhelper = repo.system_source.dbhelper
10365
21461f80f348 [connection] remove ensure_cnx_set context manager uses
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9797
diff changeset
    93
    cursor = cnx.cnxset.cu
21461f80f348 [connection] remove ensure_cnx_set context manager uses
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9797
diff changeset
    94
    if not dbhelper.has_fti_table(cursor):
10589
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 10365
diff changeset
    95
        print('no text index table')
10365
21461f80f348 [connection] remove ensure_cnx_set context manager uses
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9797
diff changeset
    96
        dbhelper.init_fti(cursor)
4806
4f12f59b1a13 [fti] refactor and fix full text indexation handling
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4691
diff changeset
    97
    repo.system_source.do_fti = True  # ensure full-text indexation is activated
5850
fabff2813ee4 [migration] schema should be accessed through .repo
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
    98
    if etypes is None:
10589
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 10365
diff changeset
    99
        print('Reindexing entities')
5850
fabff2813ee4 [migration] schema should be accessed through .repo
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   100
        etypes = set()
fabff2813ee4 [migration] schema should be accessed through .repo
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   101
        for eschema in schema.entities():
fabff2813ee4 [migration] schema should be accessed through .repo
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   102
            if eschema.final:
fabff2813ee4 [migration] schema should be accessed through .repo
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   103
                continue
fabff2813ee4 [migration] schema should be accessed through .repo
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   104
            indexable_attrs = tuple(eschema.indexable_attributes()) # generator
fabff2813ee4 [migration] schema should be accessed through .repo
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   105
            if not indexable_attrs:
fabff2813ee4 [migration] schema should be accessed through .repo
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   106
                continue
fabff2813ee4 [migration] schema should be accessed through .repo
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   107
            for container in etype_fti_containers(eschema):
fabff2813ee4 [migration] schema should be accessed through .repo
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5693
diff changeset
   108
                etypes.add(container)
5954
987086484876 [fti migration] test and fix reindexation of some specific entity types
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5850
diff changeset
   109
        # clear fti table first
9574
2d4c4842bd04 [server] some s/session/cnx/
Julien Cristau <julien.cristau@logilab.fr>
parents: 9543
diff changeset
   110
        cnx.system_sql('DELETE FROM %s' % dbhelper.fti_table)
5954
987086484876 [fti migration] test and fix reindexation of some specific entity types
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5850
diff changeset
   111
    else:
10589
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 10365
diff changeset
   112
        print('Reindexing entities of type %s' % \
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 10365
diff changeset
   113
              ', '.join(sorted(str(e) for e in etypes)))
5954
987086484876 [fti migration] test and fix reindexation of some specific entity types
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5850
diff changeset
   114
        # clear fti table first. Use subquery for sql compatibility
9574
2d4c4842bd04 [server] some s/session/cnx/
Julien Cristau <julien.cristau@logilab.fr>
parents: 9543
diff changeset
   115
        cnx.system_sql("DELETE FROM %s WHERE EXISTS(SELECT 1 FROM ENTITIES "
2d4c4842bd04 [server] some s/session/cnx/
Julien Cristau <julien.cristau@logilab.fr>
parents: 9543
diff changeset
   116
                       "WHERE eid=%s AND type IN (%s))" % (
2d4c4842bd04 [server] some s/session/cnx/
Julien Cristau <julien.cristau@logilab.fr>
parents: 9543
diff changeset
   117
                           dbhelper.fti_table, dbhelper.fti_uid_attr,
2d4c4842bd04 [server] some s/session/cnx/
Julien Cristau <julien.cristau@logilab.fr>
parents: 9543
diff changeset
   118
                           ','.join("'%s'" % etype for etype in etypes)))
4675
9233a8350420 [test] don't display progress bar when testing checkintegrity
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4252
diff changeset
   119
    if withpb:
6112
913979c79244 [db-fti-index] simple fix fpr progressbar-related crash when etypes is None in reindex_entities()
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 5999
diff changeset
   120
        pb = ProgressBar(len(etypes) + 1)
4675
9233a8350420 [test] don't display progress bar when testing checkintegrity
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4252
diff changeset
   121
        pb.update()
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   122
    # reindex entities by generating rql queries which set all indexable
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   123
    # attribute to their current value
4816
c02583cb80a9 repair stuff broken by fti handling changes
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4806
diff changeset
   124
    source = repo.system_source
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   125
    for eschema in etypes:
9574
2d4c4842bd04 [server] some s/session/cnx/
Julien Cristau <julien.cristau@logilab.fr>
parents: 9543
diff changeset
   126
        etype_class = cnx.vreg['etypes'].etype_class(str(eschema))
10847
ce5403611cbe [fti] add cw_fti_index_rql_limit method
Julien Cristau <julien.cristau@logilab.fr>
parents: 10846
diff changeset
   127
        for rset in etype_class.cw_fti_index_rql_limit(cnx):
9574
2d4c4842bd04 [server] some s/session/cnx/
Julien Cristau <julien.cristau@logilab.fr>
parents: 9543
diff changeset
   128
            source.fti_index_entities(cnx, rset.entities())
8450
11063635c4e4 [fti] allow usage of custom RQL to fetch entities to index (closes #2410509)
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 8447
diff changeset
   129
            # clear entity cache to avoid high memory consumption on big tables
9574
2d4c4842bd04 [server] some s/session/cnx/
Julien Cristau <julien.cristau@logilab.fr>
parents: 9543
diff changeset
   130
            cnx.drop_entity_cache()
4675
9233a8350420 [test] don't display progress bar when testing checkintegrity
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4252
diff changeset
   131
        if withpb:
9233a8350420 [test] don't display progress bar when testing checkintegrity
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4252
diff changeset
   132
            pb.update()
9578
68049d53426c [fti] properly close the ProgressBar
Julien Cristau <julien.cristau@logilab.fr>
parents: 9577
diff changeset
   133
    if withpb:
68049d53426c [fti] properly close the ProgressBar
Julien Cristau <julien.cristau@logilab.fr>
parents: 9577
diff changeset
   134
        pb.finish()
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   135
1802
d628defebc17 delete-trailing-whitespace + some copyright update
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 1398
diff changeset
   136
9577
c52441e4a3d7 [serverctl] use repoapi for db-check, add-source, rebuild-fti commands
Julien Cristau <julien.cristau@logilab.fr>
parents: 9574
diff changeset
   137
def check_schema(schema, cnx, eids, fix=1):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   138
    """check serialized schema"""
10589
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 10365
diff changeset
   139
    print('Checking serialized schema')
5338
3e5a256d17ba [db-check] fix duplicated schema constraint detection
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4835
diff changeset
   140
    rql = ('Any COUNT(X),RN,SN,ON,CTN GROUPBY RN,SN,ON,CTN ORDERBY 1 '
1398
5fe84a5f7035 rename internal entity types to have CW prefix instead of E
sylvain.thenault@logilab.fr
parents: 1263
diff changeset
   141
           'WHERE X is CWConstraint, R constrained_by X, '
5338
3e5a256d17ba [db-check] fix duplicated schema constraint detection
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4835
diff changeset
   142
           'R relation_type RT, RT name RN, R from_entity ST, ST name SN, '
3e5a256d17ba [db-check] fix duplicated schema constraint detection
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4835
diff changeset
   143
           'R to_entity OT, OT name ON, X cstrtype CT, CT name CTN')
9577
c52441e4a3d7 [serverctl] use repoapi for db-check, add-source, rebuild-fti commands
Julien Cristau <julien.cristau@logilab.fr>
parents: 9574
diff changeset
   144
    for count, rn, sn, on, cstrname in cnx.execute(rql):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   145
        if count == 1:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   146
            continue
10813
ab626726a70a [syncschema] don't arbitrarily remove constraints from the in-memory schema
Julien Cristau <julien.cristau@logilab.fr>
parents: 10663
diff changeset
   147
        if cstrname in UNIQUE_CONSTRAINTS:
10589
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 10365
diff changeset
   148
            print("ERROR: got %s %r constraints on relation %s.%s.%s" % (
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 10365
diff changeset
   149
                count, cstrname, sn, rn, on))
5523
4bf975c049a6 [db-check] RQLConstraint is not a 'unique' constraint
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5424
diff changeset
   150
            if fix:
10589
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 10365
diff changeset
   151
                print('dunno how to fix, do it yourself')
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   152
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   153
1802
d628defebc17 delete-trailing-whitespace + some copyright update
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 1398
diff changeset
   154
9577
c52441e4a3d7 [serverctl] use repoapi for db-check, add-source, rebuild-fti commands
Julien Cristau <julien.cristau@logilab.fr>
parents: 9574
diff changeset
   155
def check_text_index(schema, cnx, eids, fix=1):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   156
    """check all entities registered in the text index"""
10589
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 10365
diff changeset
   157
    print('Checking text index')
7965
d45c0eb39e72 [server] fix integrity checks that destroy data (closes #1972842)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7896
diff changeset
   158
    msg = '  Entity with eid %s exists in the text index but in no source (autofix will remove from text index)'
9577
c52441e4a3d7 [serverctl] use repoapi for db-check, add-source, rebuild-fti commands
Julien Cristau <julien.cristau@logilab.fr>
parents: 9574
diff changeset
   159
    cursor = cnx.system_sql('SELECT uid FROM appears;')
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   160
    for row in cursor.fetchall():
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   161
        eid = row[0]
9577
c52441e4a3d7 [serverctl] use repoapi for db-check, add-source, rebuild-fti commands
Julien Cristau <julien.cristau@logilab.fr>
parents: 9574
diff changeset
   162
        if not has_eid(cnx, cursor, eid, eids):
7896
4c954e1e73ef [lint] remove uses of "print >> sys.stderr" (closes #1908571)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7815
diff changeset
   163
            sys.stderr.write(msg % eid)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   164
            if fix:
9577
c52441e4a3d7 [serverctl] use repoapi for db-check, add-source, rebuild-fti commands
Julien Cristau <julien.cristau@logilab.fr>
parents: 9574
diff changeset
   165
                cnx.system_sql('DELETE FROM appears WHERE uid=%s;' % eid)
7035
8d2cf36bd79d [c-c db-check] factorize code by introducing notify_fixed dumb function
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6889
diff changeset
   166
            notify_fixed(fix)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   167
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   168
9577
c52441e4a3d7 [serverctl] use repoapi for db-check, add-source, rebuild-fti commands
Julien Cristau <julien.cristau@logilab.fr>
parents: 9574
diff changeset
   169
def check_entities(schema, cnx, eids, fix=1):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   170
    """check all entities registered in the repo system table"""
10589
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 10365
diff changeset
   171
    print('Checking entities system table')
7965
d45c0eb39e72 [server] fix integrity checks that destroy data (closes #1972842)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7896
diff changeset
   172
    # system table but no source
9171
be9596750678 Fix two crashes in db-check (closes #3024964)
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 8900
diff changeset
   173
    msg = '  Entity %s with eid %s exists in the system table but in no source (autofix will delete the entity)'
9577
c52441e4a3d7 [serverctl] use repoapi for db-check, add-source, rebuild-fti commands
Julien Cristau <julien.cristau@logilab.fr>
parents: 9574
diff changeset
   174
    cursor = cnx.system_sql('SELECT eid,type FROM entities;')
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   175
    for row in cursor.fetchall():
9171
be9596750678 Fix two crashes in db-check (closes #3024964)
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 8900
diff changeset
   176
        eid, etype = row
9577
c52441e4a3d7 [serverctl] use repoapi for db-check, add-source, rebuild-fti commands
Julien Cristau <julien.cristau@logilab.fr>
parents: 9574
diff changeset
   177
        if not has_eid(cnx, cursor, eid, eids):
9171
be9596750678 Fix two crashes in db-check (closes #3024964)
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 8900
diff changeset
   178
            sys.stderr.write(msg % (etype, eid))
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   179
            if fix:
9577
c52441e4a3d7 [serverctl] use repoapi for db-check, add-source, rebuild-fti commands
Julien Cristau <julien.cristau@logilab.fr>
parents: 9574
diff changeset
   180
                cnx.system_sql('DELETE FROM entities WHERE eid=%s;' % eid)
7035
8d2cf36bd79d [c-c db-check] factorize code by introducing notify_fixed dumb function
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6889
diff changeset
   181
            notify_fixed(fix)
7965
d45c0eb39e72 [server] fix integrity checks that destroy data (closes #1972842)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7896
diff changeset
   182
    # source in entities, but no relation cw_source
9577
c52441e4a3d7 [serverctl] use repoapi for db-check, add-source, rebuild-fti commands
Julien Cristau <julien.cristau@logilab.fr>
parents: 9574
diff changeset
   183
    # XXX this (get_versions) requires a second connection to the db when we already have one open
c52441e4a3d7 [serverctl] use repoapi for db-check, add-source, rebuild-fti commands
Julien Cristau <julien.cristau@logilab.fr>
parents: 9574
diff changeset
   184
    applcwversion = cnx.repo.get_versions().get('cubicweb')
9468
39b7a91a3f4c [repo] pylint cleanup, mainly of imports, with a bit of style
Julien Cristau <julien.cristau@logilab.fr>
parents: 9463
diff changeset
   185
    if applcwversion >= (3, 13, 1): # entities.asource appeared in 3.13.1
9577
c52441e4a3d7 [serverctl] use repoapi for db-check, add-source, rebuild-fti commands
Julien Cristau <julien.cristau@logilab.fr>
parents: 9574
diff changeset
   186
        cursor = cnx.system_sql('SELECT e.eid FROM entities as e, cw_CWSource as s '
7965
d45c0eb39e72 [server] fix integrity checks that destroy data (closes #1972842)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7896
diff changeset
   187
                                    'WHERE s.cw_name=e.asource AND '
d45c0eb39e72 [server] fix integrity checks that destroy data (closes #1972842)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7896
diff changeset
   188
                                    'NOT EXISTS(SELECT 1 FROM cw_source_relation as cs '
d45c0eb39e72 [server] fix integrity checks that destroy data (closes #1972842)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7896
diff changeset
   189
                                    '  WHERE cs.eid_from=e.eid AND cs.eid_to=s.cw_eid) '
d45c0eb39e72 [server] fix integrity checks that destroy data (closes #1972842)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7896
diff changeset
   190
                                    'ORDER BY e.eid')
d45c0eb39e72 [server] fix integrity checks that destroy data (closes #1972842)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7896
diff changeset
   191
        msg = ('  Entity with eid %s refers to source in entities table, '
d45c0eb39e72 [server] fix integrity checks that destroy data (closes #1972842)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7896
diff changeset
   192
               'but is missing relation cw_source (autofix will create the relation)\n')
d45c0eb39e72 [server] fix integrity checks that destroy data (closes #1972842)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7896
diff changeset
   193
        for row in cursor.fetchall():
d45c0eb39e72 [server] fix integrity checks that destroy data (closes #1972842)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7896
diff changeset
   194
            sys.stderr.write(msg % row[0])
d45c0eb39e72 [server] fix integrity checks that destroy data (closes #1972842)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7896
diff changeset
   195
        if fix:
9577
c52441e4a3d7 [serverctl] use repoapi for db-check, add-source, rebuild-fti commands
Julien Cristau <julien.cristau@logilab.fr>
parents: 9574
diff changeset
   196
            cnx.system_sql('INSERT INTO cw_source_relation (eid_from, eid_to) '
7965
d45c0eb39e72 [server] fix integrity checks that destroy data (closes #1972842)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7896
diff changeset
   197
                               'SELECT e.eid, s.cw_eid FROM entities as e, cw_CWSource as s '
d45c0eb39e72 [server] fix integrity checks that destroy data (closes #1972842)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7896
diff changeset
   198
                               'WHERE s.cw_name=e.asource AND NOT EXISTS(SELECT 1 FROM cw_source_relation as cs '
d45c0eb39e72 [server] fix integrity checks that destroy data (closes #1972842)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7896
diff changeset
   199
                               '  WHERE cs.eid_from=e.eid AND cs.eid_to=s.cw_eid)')
d45c0eb39e72 [server] fix integrity checks that destroy data (closes #1972842)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7896
diff changeset
   200
            notify_fixed(True)
d45c0eb39e72 [server] fix integrity checks that destroy data (closes #1972842)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7896
diff changeset
   201
    # inconsistencies for 'is'
d45c0eb39e72 [server] fix integrity checks that destroy data (closes #1972842)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7896
diff changeset
   202
    msg = '  %s #%s is missing relation "is" (autofix will create the relation)\n'
9577
c52441e4a3d7 [serverctl] use repoapi for db-check, add-source, rebuild-fti commands
Julien Cristau <julien.cristau@logilab.fr>
parents: 9574
diff changeset
   203
    cursor = cnx.system_sql('SELECT e.type, e.eid FROM entities as e, cw_CWEType as s '
7965
d45c0eb39e72 [server] fix integrity checks that destroy data (closes #1972842)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7896
diff changeset
   204
                                'WHERE s.cw_name=e.type AND NOT EXISTS(SELECT 1 FROM is_relation as cs '
d45c0eb39e72 [server] fix integrity checks that destroy data (closes #1972842)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7896
diff changeset
   205
                                '  WHERE cs.eid_from=e.eid AND cs.eid_to=s.cw_eid) '
d45c0eb39e72 [server] fix integrity checks that destroy data (closes #1972842)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7896
diff changeset
   206
                                'ORDER BY e.eid')
d45c0eb39e72 [server] fix integrity checks that destroy data (closes #1972842)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7896
diff changeset
   207
    for row in cursor.fetchall():
10625
684dbf6722ef [server] fix checkintegrity with pyodbc
Julien Cristau <julien.cristau@logilab.fr>
parents: 10365
diff changeset
   208
        sys.stderr.write(msg % tuple(row))
7965
d45c0eb39e72 [server] fix integrity checks that destroy data (closes #1972842)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7896
diff changeset
   209
    if fix:
9577
c52441e4a3d7 [serverctl] use repoapi for db-check, add-source, rebuild-fti commands
Julien Cristau <julien.cristau@logilab.fr>
parents: 9574
diff changeset
   210
        cnx.system_sql('INSERT INTO is_relation (eid_from, eid_to) '
7965
d45c0eb39e72 [server] fix integrity checks that destroy data (closes #1972842)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7896
diff changeset
   211
                           'SELECT e.eid, s.cw_eid FROM entities as e, cw_CWEType as s '
d45c0eb39e72 [server] fix integrity checks that destroy data (closes #1972842)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7896
diff changeset
   212
                           'WHERE s.cw_name=e.type AND NOT EXISTS(SELECT 1 FROM is_relation as cs '
d45c0eb39e72 [server] fix integrity checks that destroy data (closes #1972842)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7896
diff changeset
   213
                           '  WHERE cs.eid_from=e.eid AND cs.eid_to=s.cw_eid)')
d45c0eb39e72 [server] fix integrity checks that destroy data (closes #1972842)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7896
diff changeset
   214
        notify_fixed(True)
d45c0eb39e72 [server] fix integrity checks that destroy data (closes #1972842)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7896
diff changeset
   215
    # inconsistencies for 'is_instance_of'
d45c0eb39e72 [server] fix integrity checks that destroy data (closes #1972842)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7896
diff changeset
   216
    msg = '  %s #%s is missing relation "is_instance_of" (autofix will create the relation)\n'
9577
c52441e4a3d7 [serverctl] use repoapi for db-check, add-source, rebuild-fti commands
Julien Cristau <julien.cristau@logilab.fr>
parents: 9574
diff changeset
   217
    cursor = cnx.system_sql('SELECT e.type, e.eid FROM entities as e, cw_CWEType as s '
7965
d45c0eb39e72 [server] fix integrity checks that destroy data (closes #1972842)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7896
diff changeset
   218
                                'WHERE s.cw_name=e.type AND NOT EXISTS(SELECT 1 FROM is_instance_of_relation as cs '
d45c0eb39e72 [server] fix integrity checks that destroy data (closes #1972842)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7896
diff changeset
   219
                                '  WHERE cs.eid_from=e.eid AND cs.eid_to=s.cw_eid) '
d45c0eb39e72 [server] fix integrity checks that destroy data (closes #1972842)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7896
diff changeset
   220
                                'ORDER BY e.eid')
d45c0eb39e72 [server] fix integrity checks that destroy data (closes #1972842)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7896
diff changeset
   221
    for row in cursor.fetchall():
10625
684dbf6722ef [server] fix checkintegrity with pyodbc
Julien Cristau <julien.cristau@logilab.fr>
parents: 10365
diff changeset
   222
        sys.stderr.write(msg % tuple(row))
7965
d45c0eb39e72 [server] fix integrity checks that destroy data (closes #1972842)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7896
diff changeset
   223
    if fix:
9577
c52441e4a3d7 [serverctl] use repoapi for db-check, add-source, rebuild-fti commands
Julien Cristau <julien.cristau@logilab.fr>
parents: 9574
diff changeset
   224
        cnx.system_sql('INSERT INTO is_instance_of_relation (eid_from, eid_to) '
7965
d45c0eb39e72 [server] fix integrity checks that destroy data (closes #1972842)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7896
diff changeset
   225
                           'SELECT e.eid, s.cw_eid FROM entities as e, cw_CWEType as s '
d45c0eb39e72 [server] fix integrity checks that destroy data (closes #1972842)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7896
diff changeset
   226
                           'WHERE s.cw_name=e.type AND NOT EXISTS(SELECT 1 FROM is_instance_of_relation as cs '
d45c0eb39e72 [server] fix integrity checks that destroy data (closes #1972842)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7896
diff changeset
   227
                           '  WHERE cs.eid_from=e.eid AND cs.eid_to=s.cw_eid)')
d45c0eb39e72 [server] fix integrity checks that destroy data (closes #1972842)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7896
diff changeset
   228
        notify_fixed(True)
10589
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 10365
diff changeset
   229
    print('Checking entities tables')
7965
d45c0eb39e72 [server] fix integrity checks that destroy data (closes #1972842)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7896
diff changeset
   230
    msg = '  Entity with eid %s exists in the %s table but not in the system table (autofix will delete the entity)'
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   231
    for eschema in schema.entities():
3689
deb13e88e037 follow yams 0.25 api changes to improve performance
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3374
diff changeset
   232
        if eschema.final:
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   233
            continue
1251
af40e615dc89 introduce a 'cw_' prefix on entity table and column names so we don't conflict with sql or DBMS specific keywords
sylvain.thenault@logilab.fr
parents: 1161
diff changeset
   234
        table = SQL_PREFIX + eschema.type
af40e615dc89 introduce a 'cw_' prefix on entity table and column names so we don't conflict with sql or DBMS specific keywords
sylvain.thenault@logilab.fr
parents: 1161
diff changeset
   235
        column = SQL_PREFIX +  'eid'
9577
c52441e4a3d7 [serverctl] use repoapi for db-check, add-source, rebuild-fti commands
Julien Cristau <julien.cristau@logilab.fr>
parents: 9574
diff changeset
   236
        cursor = cnx.system_sql('SELECT %s FROM %s;' % (column, table))
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   237
        for row in cursor.fetchall():
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   238
            eid = row[0]
5341
0de53140bd29 [db-check] cleanup
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5340
diff changeset
   239
            # eids is full since we have fetched everything from the entities table,
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   240
            # no need to call has_eid
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   241
            if not eid in eids or not eids[eid]:
7896
4c954e1e73ef [lint] remove uses of "print >> sys.stderr" (closes #1908571)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7815
diff changeset
   242
                sys.stderr.write(msg % (eid, eschema.type))
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   243
                if fix:
9577
c52441e4a3d7 [serverctl] use repoapi for db-check, add-source, rebuild-fti commands
Julien Cristau <julien.cristau@logilab.fr>
parents: 9574
diff changeset
   244
                    cnx.system_sql('DELETE FROM %s WHERE %s=%s;' % (table, column, eid))
7035
8d2cf36bd79d [c-c db-check] factorize code by introducing notify_fixed dumb function
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6889
diff changeset
   245
                notify_fixed(fix)
1802
d628defebc17 delete-trailing-whitespace + some copyright update
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 1398
diff changeset
   246
d628defebc17 delete-trailing-whitespace + some copyright update
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 1398
diff changeset
   247
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   248
def bad_related_msg(rtype, target, eid, fix):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   249
    msg = '  A relation %s with %s eid %s exists but no such entity in sources'
7896
4c954e1e73ef [lint] remove uses of "print >> sys.stderr" (closes #1908571)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7815
diff changeset
   250
    sys.stderr.write(msg % (rtype, target, eid))
7035
8d2cf36bd79d [c-c db-check] factorize code by introducing notify_fixed dumb function
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6889
diff changeset
   251
    notify_fixed(fix)
1802
d628defebc17 delete-trailing-whitespace + some copyright update
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 1398
diff changeset
   252
9171
be9596750678 Fix two crashes in db-check (closes #3024964)
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 8900
diff changeset
   253
def bad_inlined_msg(rtype, parent_eid, eid, fix):
be9596750678 Fix two crashes in db-check (closes #3024964)
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 8900
diff changeset
   254
    msg = ('  An inlined relation %s from %s to %s exists but the latter '
be9596750678 Fix two crashes in db-check (closes #3024964)
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 8900
diff changeset
   255
           'entity does not exist')
be9596750678 Fix two crashes in db-check (closes #3024964)
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 8900
diff changeset
   256
    sys.stderr.write(msg % (rtype, parent_eid, eid))
be9596750678 Fix two crashes in db-check (closes #3024964)
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 8900
diff changeset
   257
    notify_fixed(fix)
be9596750678 Fix two crashes in db-check (closes #3024964)
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 8900
diff changeset
   258
1802
d628defebc17 delete-trailing-whitespace + some copyright update
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 1398
diff changeset
   259
9577
c52441e4a3d7 [serverctl] use repoapi for db-check, add-source, rebuild-fti commands
Julien Cristau <julien.cristau@logilab.fr>
parents: 9574
diff changeset
   260
def check_relations(schema, cnx, eids, fix=1):
7035
8d2cf36bd79d [c-c db-check] factorize code by introducing notify_fixed dumb function
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6889
diff changeset
   261
    """check that eids referenced by relations are registered in the repo system
8d2cf36bd79d [c-c db-check] factorize code by introducing notify_fixed dumb function
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6889
diff changeset
   262
    table
8d2cf36bd79d [c-c db-check] factorize code by introducing notify_fixed dumb function
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6889
diff changeset
   263
    """
10589
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 10365
diff changeset
   264
    print('Checking relations')
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   265
    for rschema in schema.relations():
7965
d45c0eb39e72 [server] fix integrity checks that destroy data (closes #1972842)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7896
diff changeset
   266
        if rschema.final or rschema.type in PURE_VIRTUAL_RTYPES:
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   267
            continue
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   268
        if rschema.inlined:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   269
            for subjtype in rschema.subjects():
1251
af40e615dc89 introduce a 'cw_' prefix on entity table and column names so we don't conflict with sql or DBMS specific keywords
sylvain.thenault@logilab.fr
parents: 1161
diff changeset
   270
                table = SQL_PREFIX + str(subjtype)
af40e615dc89 introduce a 'cw_' prefix on entity table and column names so we don't conflict with sql or DBMS specific keywords
sylvain.thenault@logilab.fr
parents: 1161
diff changeset
   271
                column = SQL_PREFIX +  str(rschema)
9171
be9596750678 Fix two crashes in db-check (closes #3024964)
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 8900
diff changeset
   272
                sql = 'SELECT cw_eid,%s FROM %s WHERE %s IS NOT NULL;' % (
1251
af40e615dc89 introduce a 'cw_' prefix on entity table and column names so we don't conflict with sql or DBMS specific keywords
sylvain.thenault@logilab.fr
parents: 1161
diff changeset
   273
                    column, table, column)
9577
c52441e4a3d7 [serverctl] use repoapi for db-check, add-source, rebuild-fti commands
Julien Cristau <julien.cristau@logilab.fr>
parents: 9574
diff changeset
   274
                cursor = cnx.system_sql(sql)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   275
                for row in cursor.fetchall():
9171
be9596750678 Fix two crashes in db-check (closes #3024964)
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 8900
diff changeset
   276
                    parent_eid, eid = row
9577
c52441e4a3d7 [serverctl] use repoapi for db-check, add-source, rebuild-fti commands
Julien Cristau <julien.cristau@logilab.fr>
parents: 9574
diff changeset
   277
                    if not has_eid(cnx, cursor, eid, eids):
9171
be9596750678 Fix two crashes in db-check (closes #3024964)
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 8900
diff changeset
   278
                        bad_inlined_msg(rschema, parent_eid, eid, fix)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   279
                        if fix:
3374
d5bd1b659ce8 [db-check] fix sql to fix bad eid referenced by inlined relation
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2596
diff changeset
   280
                            sql = 'UPDATE %s SET %s=NULL WHERE %s=%s;' % (
d5bd1b659ce8 [db-check] fix sql to fix bad eid referenced by inlined relation
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2596
diff changeset
   281
                                table, column, column, eid)
9577
c52441e4a3d7 [serverctl] use repoapi for db-check, add-source, rebuild-fti commands
Julien Cristau <julien.cristau@logilab.fr>
parents: 9574
diff changeset
   282
                            cnx.system_sql(sql)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   283
            continue
6185
229006accd26 [c-c db-check] skip error while checking relation, useful when analyzing really broken database (after a migration failure for instance
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6132
diff changeset
   284
        try:
9577
c52441e4a3d7 [serverctl] use repoapi for db-check, add-source, rebuild-fti commands
Julien Cristau <julien.cristau@logilab.fr>
parents: 9574
diff changeset
   285
            cursor = cnx.system_sql('SELECT eid_from FROM %s_relation;' % rschema)
8695
358d8bed9626 [toward-py3k] rewrite to "except AnException as exc:" (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8694
diff changeset
   286
        except Exception as ex:
6185
229006accd26 [c-c db-check] skip error while checking relation, useful when analyzing really broken database (after a migration failure for instance
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6132
diff changeset
   287
            # usually because table doesn't exist
10589
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 10365
diff changeset
   288
            print('ERROR', ex)
6185
229006accd26 [c-c db-check] skip error while checking relation, useful when analyzing really broken database (after a migration failure for instance
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6132
diff changeset
   289
            continue
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   290
        for row in cursor.fetchall():
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   291
            eid = row[0]
9577
c52441e4a3d7 [serverctl] use repoapi for db-check, add-source, rebuild-fti commands
Julien Cristau <julien.cristau@logilab.fr>
parents: 9574
diff changeset
   292
            if not has_eid(cnx, cursor, eid, eids):
1251
af40e615dc89 introduce a 'cw_' prefix on entity table and column names so we don't conflict with sql or DBMS specific keywords
sylvain.thenault@logilab.fr
parents: 1161
diff changeset
   293
                bad_related_msg(rschema, 'subject', eid, fix)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   294
                if fix:
380
06e7f2932afe db-check fixes
sylvain.thenault@logilab.fr
parents: 0
diff changeset
   295
                    sql = 'DELETE FROM %s_relation WHERE eid_from=%s;' % (
1251
af40e615dc89 introduce a 'cw_' prefix on entity table and column names so we don't conflict with sql or DBMS specific keywords
sylvain.thenault@logilab.fr
parents: 1161
diff changeset
   296
                        rschema, eid)
9577
c52441e4a3d7 [serverctl] use repoapi for db-check, add-source, rebuild-fti commands
Julien Cristau <julien.cristau@logilab.fr>
parents: 9574
diff changeset
   297
                    cnx.system_sql(sql)
c52441e4a3d7 [serverctl] use repoapi for db-check, add-source, rebuild-fti commands
Julien Cristau <julien.cristau@logilab.fr>
parents: 9574
diff changeset
   298
        cursor = cnx.system_sql('SELECT eid_to FROM %s_relation;' % rschema)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   299
        for row in cursor.fetchall():
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   300
            eid = row[0]
9577
c52441e4a3d7 [serverctl] use repoapi for db-check, add-source, rebuild-fti commands
Julien Cristau <julien.cristau@logilab.fr>
parents: 9574
diff changeset
   301
            if not has_eid(cnx, cursor, eid, eids):
1251
af40e615dc89 introduce a 'cw_' prefix on entity table and column names so we don't conflict with sql or DBMS specific keywords
sylvain.thenault@logilab.fr
parents: 1161
diff changeset
   302
                bad_related_msg(rschema, 'object', eid, fix)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   303
                if fix:
380
06e7f2932afe db-check fixes
sylvain.thenault@logilab.fr
parents: 0
diff changeset
   304
                    sql = 'DELETE FROM %s_relation WHERE eid_to=%s;' % (
1251
af40e615dc89 introduce a 'cw_' prefix on entity table and column names so we don't conflict with sql or DBMS specific keywords
sylvain.thenault@logilab.fr
parents: 1161
diff changeset
   305
                        rschema, eid)
9577
c52441e4a3d7 [serverctl] use repoapi for db-check, add-source, rebuild-fti commands
Julien Cristau <julien.cristau@logilab.fr>
parents: 9574
diff changeset
   306
                    cnx.system_sql(sql)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   307
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   308
9577
c52441e4a3d7 [serverctl] use repoapi for db-check, add-source, rebuild-fti commands
Julien Cristau <julien.cristau@logilab.fr>
parents: 9574
diff changeset
   309
def check_mandatory_relations(schema, cnx, eids, fix=1):
7036
63386b35ec69 [c-c db-check] new checks for entities missing a mandatory relation/attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7035
diff changeset
   310
    """check entities missing some mandatory relation"""
10589
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 10365
diff changeset
   311
    print('Checking mandatory relations')
7965
d45c0eb39e72 [server] fix integrity checks that destroy data (closes #1972842)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7896
diff changeset
   312
    msg = '%s #%s is missing mandatory %s relation %s (autofix will delete the entity)'
7036
63386b35ec69 [c-c db-check] new checks for entities missing a mandatory relation/attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7035
diff changeset
   313
    for rschema in schema.relations():
8527
6151b0199bc1 [c-c db-check] don't skip is/is_instance_of mandatory relation to avoid telling two opposite messages when the relation is missing. Closes #2465659
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 8450
diff changeset
   314
        if rschema.final or rschema in PURE_VIRTUAL_RTYPES or rschema in ('is', 'is_instance_of'):
7036
63386b35ec69 [c-c db-check] new checks for entities missing a mandatory relation/attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7035
diff changeset
   315
            continue
63386b35ec69 [c-c db-check] new checks for entities missing a mandatory relation/attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7035
diff changeset
   316
        smandatory = set()
63386b35ec69 [c-c db-check] new checks for entities missing a mandatory relation/attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7035
diff changeset
   317
        omandatory = set()
10663
54b8a1f249fb [py3k] dict.itervalues → dict.values
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10651
diff changeset
   318
        for rdef in rschema.rdefs.values():
7036
63386b35ec69 [c-c db-check] new checks for entities missing a mandatory relation/attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7035
diff changeset
   319
            if rdef.cardinality[0] in '1+':
63386b35ec69 [c-c db-check] new checks for entities missing a mandatory relation/attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7035
diff changeset
   320
                smandatory.add(rdef.subject)
63386b35ec69 [c-c db-check] new checks for entities missing a mandatory relation/attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7035
diff changeset
   321
            if rdef.cardinality[1] in '1+':
63386b35ec69 [c-c db-check] new checks for entities missing a mandatory relation/attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7035
diff changeset
   322
                omandatory.add(rdef.object)
63386b35ec69 [c-c db-check] new checks for entities missing a mandatory relation/attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7035
diff changeset
   323
        for role, etypes in (('subject', smandatory), ('object', omandatory)):
63386b35ec69 [c-c db-check] new checks for entities missing a mandatory relation/attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7035
diff changeset
   324
            for etype in etypes:
63386b35ec69 [c-c db-check] new checks for entities missing a mandatory relation/attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7035
diff changeset
   325
                if role == 'subject':
63386b35ec69 [c-c db-check] new checks for entities missing a mandatory relation/attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7035
diff changeset
   326
                    rql = 'Any X WHERE NOT X %s Y, X is %s' % (rschema, etype)
63386b35ec69 [c-c db-check] new checks for entities missing a mandatory relation/attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7035
diff changeset
   327
                else:
63386b35ec69 [c-c db-check] new checks for entities missing a mandatory relation/attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7035
diff changeset
   328
                    rql = 'Any X WHERE NOT Y %s X, X is %s' % (rschema, etype)
9577
c52441e4a3d7 [serverctl] use repoapi for db-check, add-source, rebuild-fti commands
Julien Cristau <julien.cristau@logilab.fr>
parents: 9574
diff changeset
   329
                for entity in cnx.execute(rql).entities():
8900
010a59e12d89 use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8696
diff changeset
   330
                    sys.stderr.write(msg % (entity.cw_etype, entity.eid, role, rschema))
7036
63386b35ec69 [c-c db-check] new checks for entities missing a mandatory relation/attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7035
diff changeset
   331
                    if fix:
63386b35ec69 [c-c db-check] new checks for entities missing a mandatory relation/attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7035
diff changeset
   332
                        #if entity.cw_describe()['source']['uri'] == 'system': XXX
7965
d45c0eb39e72 [server] fix integrity checks that destroy data (closes #1972842)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7896
diff changeset
   333
                        entity.cw_delete() # XXX this is BRUTAL!
7036
63386b35ec69 [c-c db-check] new checks for entities missing a mandatory relation/attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7035
diff changeset
   334
                    notify_fixed(fix)
63386b35ec69 [c-c db-check] new checks for entities missing a mandatory relation/attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7035
diff changeset
   335
63386b35ec69 [c-c db-check] new checks for entities missing a mandatory relation/attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7035
diff changeset
   336
9577
c52441e4a3d7 [serverctl] use repoapi for db-check, add-source, rebuild-fti commands
Julien Cristau <julien.cristau@logilab.fr>
parents: 9574
diff changeset
   337
def check_mandatory_attributes(schema, cnx, eids, fix=1):
7036
63386b35ec69 [c-c db-check] new checks for entities missing a mandatory relation/attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7035
diff changeset
   338
    """check for entities stored in the system source missing some mandatory
63386b35ec69 [c-c db-check] new checks for entities missing a mandatory relation/attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7035
diff changeset
   339
    attribute
63386b35ec69 [c-c db-check] new checks for entities missing a mandatory relation/attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7035
diff changeset
   340
    """
10589
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 10365
diff changeset
   341
    print('Checking mandatory attributes')
7965
d45c0eb39e72 [server] fix integrity checks that destroy data (closes #1972842)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7896
diff changeset
   342
    msg = '%s #%s is missing mandatory attribute %s (autofix will delete the entity)'
7036
63386b35ec69 [c-c db-check] new checks for entities missing a mandatory relation/attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7035
diff changeset
   343
    for rschema in schema.relations():
63386b35ec69 [c-c db-check] new checks for entities missing a mandatory relation/attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7035
diff changeset
   344
        if not rschema.final or rschema in VIRTUAL_RTYPES:
63386b35ec69 [c-c db-check] new checks for entities missing a mandatory relation/attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7035
diff changeset
   345
            continue
10663
54b8a1f249fb [py3k] dict.itervalues → dict.values
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10651
diff changeset
   346
        for rdef in rschema.rdefs.values():
7036
63386b35ec69 [c-c db-check] new checks for entities missing a mandatory relation/attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7035
diff changeset
   347
            if rdef.cardinality[0] in '1+':
63386b35ec69 [c-c db-check] new checks for entities missing a mandatory relation/attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7035
diff changeset
   348
                rql = 'Any X WHERE X %s NULL, X is %s, X cw_source S, S name "system"' % (
63386b35ec69 [c-c db-check] new checks for entities missing a mandatory relation/attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7035
diff changeset
   349
                    rschema, rdef.subject)
9577
c52441e4a3d7 [serverctl] use repoapi for db-check, add-source, rebuild-fti commands
Julien Cristau <julien.cristau@logilab.fr>
parents: 9574
diff changeset
   350
                for entity in cnx.execute(rql).entities():
8900
010a59e12d89 use cw_etype instead of __regid__
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 8696
diff changeset
   351
                    sys.stderr.write(msg % (entity.cw_etype, entity.eid, rschema))
7036
63386b35ec69 [c-c db-check] new checks for entities missing a mandatory relation/attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7035
diff changeset
   352
                    if fix:
7682
07b592a62be3 [c-c db-check] drop some deprecation warning
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7473
diff changeset
   353
                        entity.cw_delete()
7036
63386b35ec69 [c-c db-check] new checks for entities missing a mandatory relation/attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7035
diff changeset
   354
                    notify_fixed(fix)
63386b35ec69 [c-c db-check] new checks for entities missing a mandatory relation/attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7035
diff changeset
   355
63386b35ec69 [c-c db-check] new checks for entities missing a mandatory relation/attribute
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7035
diff changeset
   356
9577
c52441e4a3d7 [serverctl] use repoapi for db-check, add-source, rebuild-fti commands
Julien Cristau <julien.cristau@logilab.fr>
parents: 9574
diff changeset
   357
def check_metadata(schema, cnx, eids, fix=1):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   358
    """check entities has required metadata
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   359
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   360
    FIXME: rewrite using RQL queries ?
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   361
    """
10589
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 10365
diff changeset
   362
    print('Checking metadata')
9577
c52441e4a3d7 [serverctl] use repoapi for db-check, add-source, rebuild-fti commands
Julien Cristau <julien.cristau@logilab.fr>
parents: 9574
diff changeset
   363
    cursor = cnx.system_sql("SELECT DISTINCT type FROM entities;")
1251
af40e615dc89 introduce a 'cw_' prefix on entity table and column names so we don't conflict with sql or DBMS specific keywords
sylvain.thenault@logilab.fr
parents: 1161
diff changeset
   364
    eidcolumn = SQL_PREFIX + 'eid'
7965
d45c0eb39e72 [server] fix integrity checks that destroy data (closes #1972842)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7896
diff changeset
   365
    msg = '  %s with eid %s has no %s (autofix will set it to now)'
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   366
    for etype, in cursor.fetchall():
9577
c52441e4a3d7 [serverctl] use repoapi for db-check, add-source, rebuild-fti commands
Julien Cristau <julien.cristau@logilab.fr>
parents: 9574
diff changeset
   367
        if etype not in cnx.vreg.schema:
9171
be9596750678 Fix two crashes in db-check (closes #3024964)
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 8900
diff changeset
   368
            sys.stderr.write('entities table references unknown type %s\n' %
be9596750678 Fix two crashes in db-check (closes #3024964)
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 8900
diff changeset
   369
                             etype)
be9596750678 Fix two crashes in db-check (closes #3024964)
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 8900
diff changeset
   370
            if fix:
9577
c52441e4a3d7 [serverctl] use repoapi for db-check, add-source, rebuild-fti commands
Julien Cristau <julien.cristau@logilab.fr>
parents: 9574
diff changeset
   371
                cnx.system_sql("DELETE FROM entities WHERE type = %(type)s",
9171
be9596750678 Fix two crashes in db-check (closes #3024964)
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 8900
diff changeset
   372
                                   {'type': etype})
be9596750678 Fix two crashes in db-check (closes #3024964)
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 8900
diff changeset
   373
            continue
1251
af40e615dc89 introduce a 'cw_' prefix on entity table and column names so we don't conflict with sql or DBMS specific keywords
sylvain.thenault@logilab.fr
parents: 1161
diff changeset
   374
        table = SQL_PREFIX + etype
1016
26387b836099 use datetime instead of mx.DateTime
sylvain.thenault@logilab.fr
parents: 713
diff changeset
   375
        for rel, default in ( ('creation_date', datetime.now()),
26387b836099 use datetime instead of mx.DateTime
sylvain.thenault@logilab.fr
parents: 713
diff changeset
   376
                              ('modification_date', datetime.now()), ):
1251
af40e615dc89 introduce a 'cw_' prefix on entity table and column names so we don't conflict with sql or DBMS specific keywords
sylvain.thenault@logilab.fr
parents: 1161
diff changeset
   377
            column = SQL_PREFIX + rel
9577
c52441e4a3d7 [serverctl] use repoapi for db-check, add-source, rebuild-fti commands
Julien Cristau <julien.cristau@logilab.fr>
parents: 9574
diff changeset
   378
            cursor = cnx.system_sql("SELECT %s FROM %s WHERE %s is NULL"
1251
af40e615dc89 introduce a 'cw_' prefix on entity table and column names so we don't conflict with sql or DBMS specific keywords
sylvain.thenault@logilab.fr
parents: 1161
diff changeset
   379
                                        % (eidcolumn, table, column))
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   380
            for eid, in cursor.fetchall():
7896
4c954e1e73ef [lint] remove uses of "print >> sys.stderr" (closes #1908571)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7815
diff changeset
   381
                sys.stderr.write(msg % (etype, eid, rel))
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   382
                if fix:
9577
c52441e4a3d7 [serverctl] use repoapi for db-check, add-source, rebuild-fti commands
Julien Cristau <julien.cristau@logilab.fr>
parents: 9574
diff changeset
   383
                    cnx.system_sql("UPDATE %s SET %s=%%(v)s WHERE %s=%s ;"
1251
af40e615dc89 introduce a 'cw_' prefix on entity table and column names so we don't conflict with sql or DBMS specific keywords
sylvain.thenault@logilab.fr
parents: 1161
diff changeset
   384
                                       % (table, column, eidcolumn, eid),
af40e615dc89 introduce a 'cw_' prefix on entity table and column names so we don't conflict with sql or DBMS specific keywords
sylvain.thenault@logilab.fr
parents: 1161
diff changeset
   385
                                       {'v': default})
7035
8d2cf36bd79d [c-c db-check] factorize code by introducing notify_fixed dumb function
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 6889
diff changeset
   386
                notify_fixed(fix)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   387
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   388
4675
9233a8350420 [test] don't display progress bar when testing checkintegrity
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4252
diff changeset
   389
def check(repo, cnx, checks, reindex, fix, withpb=True):
2476
1294a6bdf3bf application -> instance where it makes sense
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2248
diff changeset
   390
    """check integrity of instance's repository,
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   391
    using given user and password to locally connect to the repository
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   392
    (no running cubicweb server needed)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   393
    """
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   394
    # yo, launch checks
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   395
    if checks:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   396
        eids_cache = {}
9797
4e640ab62f51 [tests/checkintegrity] use the new connection api
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9578
diff changeset
   397
        with cnx.security_enabled(read=False, write=False): # ensure no read security
4835
13b0b96d7982 [repo] enhanced security handling: deprecates unsafe_execute, in favor of explicit read/write security control using the `enabled_security` context manager. Also code executed on the repository side is now unsafe by default.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4834
diff changeset
   398
            for check in checks:
13b0b96d7982 [repo] enhanced security handling: deprecates unsafe_execute, in favor of explicit read/write security control using the `enabled_security` context manager. Also code executed on the repository side is now unsafe by default.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4834
diff changeset
   399
                check_func = globals()['check_%s' % check]
10365
21461f80f348 [connection] remove ensure_cnx_set context manager uses
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9797
diff changeset
   400
                check_func(repo.schema, cnx, eids_cache, fix=fix)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   401
        if fix:
9797
4e640ab62f51 [tests/checkintegrity] use the new connection api
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9578
diff changeset
   402
            cnx.commit()
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   403
        else:
10589
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 10365
diff changeset
   404
            print()
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   405
        if not fix:
10589
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 10365
diff changeset
   406
            print('WARNING: Diagnostic run, nothing has been corrected')
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   407
    if reindex:
9797
4e640ab62f51 [tests/checkintegrity] use the new connection api
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9578
diff changeset
   408
        cnx.rollback()
10365
21461f80f348 [connection] remove ensure_cnx_set context manager uses
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9797
diff changeset
   409
        reindex_entities(repo.schema, cnx, withpb=withpb)
9797
4e640ab62f51 [tests/checkintegrity] use the new connection api
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 9578
diff changeset
   410
        cnx.commit()