cubicweb/server/utils.py
author Denis Laxalde <denis.laxalde@logilab.fr>
Thu, 14 Mar 2019 12:08:37 +0100
changeset 12508 a8c1ea390400
parent 12280 577e8d3896b4
child 12567 26744ad37953
permissions -rw-r--r--
Drop most of deprecated code As in previous changesets, most code has been deprecated since a very long time. This changeset concerns "core" cubicweb modules (e.g. "server", "appobjects") for which it's not easy to handle change atomically. In cubicweb/server/querier.py, we adjust empty_rset() function as a result of "rqlst" argument of ResultSet being dropped. (There was no use of the keyword argument anyways.)
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
11091
29aebc1edd29 [repository] drop usage of no more necessary eschema_eid function
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 10907
diff changeset
     1
# copyright 2003-2016 LOGILAB S.A. (Paris, FRANCE), all rights reserved.
5421
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5376
diff changeset
     2
# contact http://www.logilab.fr/ -- mailto:contact@logilab.fr
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5376
diff changeset
     3
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5376
diff changeset
     4
# This file is part of CubicWeb.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5376
diff changeset
     5
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5376
diff changeset
     6
# CubicWeb is free software: you can redistribute it and/or modify it under the
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5376
diff changeset
     7
# terms of the GNU Lesser General Public License as published by the Free
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5376
diff changeset
     8
# Software Foundation, either version 2.1 of the License, or (at your option)
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5376
diff changeset
     9
# any later version.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5376
diff changeset
    10
#
5424
8ecbcbff9777 replace logilab-common by CubicWeb in disclaimer
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5421
diff changeset
    11
# CubicWeb is distributed in the hope that it will be useful, but WITHOUT
5421
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5376
diff changeset
    12
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5376
diff changeset
    13
# FOR A PARTICULAR PURPOSE.  See the GNU Lesser General Public License for more
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5376
diff changeset
    14
# details.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5376
diff changeset
    15
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5376
diff changeset
    16
# You should have received a copy of the GNU Lesser General Public License along
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5376
diff changeset
    17
# with CubicWeb.  If not, see <http://www.gnu.org/licenses/>.
6128
fbb8398f80dc cleanups
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5606
diff changeset
    18
"""Some utilities for the CubicWeb server."""
10589
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 10095
diff changeset
    19
from __future__ import print_function
7815
2a164a9cf81c [exceptions] stop catching any exception in various places (closes #1942716)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7573
diff changeset
    20
11767
432f87a63057 flake8 and all
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11129
diff changeset
    21
12011
d2888fee6031 [server] introduce a scheduler class to run repository "looping tasks"
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11880
diff changeset
    22
from functools import wraps
d2888fee6031 [server] introduce a scheduler class to run repository "looping tasks"
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11880
diff changeset
    23
import sched
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    24
import sys
6765
b922e3a817e9 fix ticket #1382716 (problem was actually more subtle than I originally thought)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6427
diff changeset
    25
import logging
12013
7b975655d0ae [server] Drop utils's LoopTask and TasksManager classes not used anymore
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12011
diff changeset
    26
from threading import Thread
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    27
from getpass import getpass
2105
92ea410806fe refactor sources configuration, add source to sources when using a cube defining
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1977
diff changeset
    28
12280
577e8d3896b4 [server] unicode → text_type in utils
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12193
diff changeset
    29
from six import PY2, text_type
10678
77333ec71fab [server/utils] convert to py3k using six
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10589
diff changeset
    30
from six.moves import input
77333ec71fab [server/utils] convert to py3k using six
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10589
diff changeset
    31
8317
9c59258e7798 [security] use a stronger encryption algorythm for password, keeping bw compat
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7823
diff changeset
    32
from passlib.utils import handlers as uh, to_hash_str
9c59258e7798 [security] use a stronger encryption algorythm for password, keeping bw compat
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7823
diff changeset
    33
from passlib.context import CryptContext
9c59258e7798 [security] use a stronger encryption algorythm for password, keeping bw compat
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7823
diff changeset
    34
9c59258e7798 [security] use a stronger encryption algorythm for password, keeping bw compat
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7823
diff changeset
    35
from cubicweb.md5crypt import crypt as md5crypt
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    36
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    37
8317
9c59258e7798 [security] use a stronger encryption algorythm for password, keeping bw compat
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7823
diff changeset
    38
class CustomMD5Crypt(uh.HasSalt, uh.GenericHandler):
8398
a9fe30c953be [server/utils] passlib 1.6 is now less tolerant wrt handler names (closes #2349330)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 8317
diff changeset
    39
    name = 'cubicwebmd5crypt'
a9fe30c953be [server/utils] passlib 1.6 is now less tolerant wrt handler names (closes #2349330)
Aurelien Campeas <aurelien.campeas@logilab.fr>
parents: 8317
diff changeset
    40
    setting_kwds = ('salt',)
8317
9c59258e7798 [security] use a stronger encryption algorythm for password, keeping bw compat
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7823
diff changeset
    41
    min_salt_size = 0
9c59258e7798 [security] use a stronger encryption algorythm for password, keeping bw compat
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7823
diff changeset
    42
    max_salt_size = 8
9c59258e7798 [security] use a stronger encryption algorythm for password, keeping bw compat
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7823
diff changeset
    43
    salt_chars = uh.H64_CHARS
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    44
8317
9c59258e7798 [security] use a stronger encryption algorythm for password, keeping bw compat
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7823
diff changeset
    45
    @classmethod
9c59258e7798 [security] use a stronger encryption algorythm for password, keeping bw compat
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7823
diff changeset
    46
    def from_string(cls, hash):
8414
e7243ed7bb0a Fixes for compatibility with passlib 1.6 (closes #2356393)
Julien Cristau <julien.cristau@logilab.fr>
parents: 8398
diff changeset
    47
        salt, chk = uh.parse_mc2(hash, u'')
e7243ed7bb0a Fixes for compatibility with passlib 1.6 (closes #2356393)
Julien Cristau <julien.cristau@logilab.fr>
parents: 8398
diff changeset
    48
        if chk is None:
e7243ed7bb0a Fixes for compatibility with passlib 1.6 (closes #2356393)
Julien Cristau <julien.cristau@logilab.fr>
parents: 8398
diff changeset
    49
            raise ValueError('missing checksum')
e7243ed7bb0a Fixes for compatibility with passlib 1.6 (closes #2356393)
Julien Cristau <julien.cristau@logilab.fr>
parents: 8398
diff changeset
    50
        return cls(salt=salt, checksum=chk)
8317
9c59258e7798 [security] use a stronger encryption algorythm for password, keeping bw compat
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7823
diff changeset
    51
9c59258e7798 [security] use a stronger encryption algorythm for password, keeping bw compat
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7823
diff changeset
    52
    def to_string(self):
9c59258e7798 [security] use a stronger encryption algorythm for password, keeping bw compat
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7823
diff changeset
    53
        return to_hash_str(u'%s$%s' % (self.salt, self.checksum or u''))
9c59258e7798 [security] use a stronger encryption algorythm for password, keeping bw compat
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7823
diff changeset
    54
12193
e4ee04d442be [server/utils] update passlib API to 1.7 (closes #17054805)
David Douard <david.douard@logilab.fr>
parents: 12014
diff changeset
    55
    def _calc_checksum(self, secret):
8414
e7243ed7bb0a Fixes for compatibility with passlib 1.6 (closes #2356393)
Julien Cristau <julien.cristau@logilab.fr>
parents: 8398
diff changeset
    56
        return md5crypt(secret, self.salt.encode('ascii')).decode('utf-8')
8317
9c59258e7798 [security] use a stronger encryption algorythm for password, keeping bw compat
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7823
diff changeset
    57
12014
116005b5fce9 [server] Make cubicweb/server/utils.py flake8-clean
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12013
diff changeset
    58
8546
3d2038d6f20d [sources/native] automatically update passwords using deprecated hashes on login
Julien Cristau <julien.cristau@logilab.fr>
parents: 8446
diff changeset
    59
_CRYPTO_CTX = CryptContext(['sha512_crypt', CustomMD5Crypt, 'des_crypt', 'ldap_salted_sha1'],
3d2038d6f20d [sources/native] automatically update passwords using deprecated hashes on login
Julien Cristau <julien.cristau@logilab.fr>
parents: 8446
diff changeset
    60
                           deprecated=['cubicwebmd5crypt', 'des_crypt'])
12193
e4ee04d442be [server/utils] update passlib API to 1.7 (closes #17054805)
David Douard <david.douard@logilab.fr>
parents: 12014
diff changeset
    61
# for bw compat with passlib < 1.7
e4ee04d442be [server/utils] update passlib API to 1.7 (closes #17054805)
David Douard <david.douard@logilab.fr>
parents: 12014
diff changeset
    62
if not hasattr(_CRYPTO_CTX, 'hash'):
e4ee04d442be [server/utils] update passlib API to 1.7 (closes #17054805)
David Douard <david.douard@logilab.fr>
parents: 12014
diff changeset
    63
    _CRYPTO_CTX.hash = _CRYPTO_CTX.encrypt
8546
3d2038d6f20d [sources/native] automatically update passwords using deprecated hashes on login
Julien Cristau <julien.cristau@logilab.fr>
parents: 8446
diff changeset
    64
verify_and_update = _CRYPTO_CTX.verify_and_update
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    65
12014
116005b5fce9 [server] Make cubicweb/server/utils.py flake8-clean
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12013
diff changeset
    66
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    67
def crypt_password(passwd, salt=None):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    68
    """return the encrypted password using the given salt or a generated one
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    69
    """
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    70
    if salt is None:
11880
bc9d901cb9e6 [server] Use CryptContext's hash method instead of deprecated encrypt method
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11767
diff changeset
    71
        return _CRYPTO_CTX.hash(passwd).encode('ascii')
8317
9c59258e7798 [security] use a stronger encryption algorythm for password, keeping bw compat
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7823
diff changeset
    72
    # empty hash, accept any password for backwards compat
9c59258e7798 [security] use a stronger encryption algorythm for password, keeping bw compat
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7823
diff changeset
    73
    if salt == '':
9c59258e7798 [security] use a stronger encryption algorythm for password, keeping bw compat
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7823
diff changeset
    74
        return salt
8550
63260486de89 [server/utils] catch ValueError from password verification
Julien Cristau <julien.cristau@logilab.fr>
parents: 8546
diff changeset
    75
    try:
63260486de89 [server/utils] catch ValueError from password verification
Julien Cristau <julien.cristau@logilab.fr>
parents: 8546
diff changeset
    76
        if _CRYPTO_CTX.verify(passwd, salt):
63260486de89 [server/utils] catch ValueError from password verification
Julien Cristau <julien.cristau@logilab.fr>
parents: 8546
diff changeset
    77
            return salt
12014
116005b5fce9 [server] Make cubicweb/server/utils.py flake8-clean
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12013
diff changeset
    78
    except ValueError:  # e.g. couldn't identify hash
8550
63260486de89 [server/utils] catch ValueError from password verification
Julien Cristau <julien.cristau@logilab.fr>
parents: 8546
diff changeset
    79
        pass
8317
9c59258e7798 [security] use a stronger encryption algorythm for password, keeping bw compat
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7823
diff changeset
    80
    # wrong password
10681
4383f5a30504 [server/utils] crypt_password should always return bytes
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10678
diff changeset
    81
    return b''
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    82
12014
116005b5fce9 [server] Make cubicweb/server/utils.py flake8-clean
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12013
diff changeset
    83
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    84
DEFAULT_MSG = 'we need a manager connection on the repository \
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    85
(the server doesn\'t have to run, even should better not)'
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    86
12014
116005b5fce9 [server] Make cubicweb/server/utils.py flake8-clean
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12013
diff changeset
    87
1910
864aa3ea0db5 [server] refactor server.utils.manager_userpasswd
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 1802
diff changeset
    88
def manager_userpasswd(user=None, msg=DEFAULT_MSG, confirm=False,
864aa3ea0db5 [server] refactor server.utils.manager_userpasswd
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 1802
diff changeset
    89
                       passwdmsg='password'):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    90
    if not user:
3701
104b7c326172 check we've some message to display
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 3585
diff changeset
    91
        if msg:
10589
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 10095
diff changeset
    92
            print(msg)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    93
        while not user:
10678
77333ec71fab [server/utils] convert to py3k using six
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10589
diff changeset
    94
            user = input('login: ')
77333ec71fab [server/utils] convert to py3k using six
Rémi Cardona <remi.cardona@logilab.fr>
parents: 10589
diff changeset
    95
        if PY2:
12280
577e8d3896b4 [server] unicode → text_type in utils
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12193
diff changeset
    96
            user = text_type(user, sys.stdin.encoding)
1910
864aa3ea0db5 [server] refactor server.utils.manager_userpasswd
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 1802
diff changeset
    97
    passwd = getpass('%s: ' % passwdmsg)
864aa3ea0db5 [server] refactor server.utils.manager_userpasswd
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 1802
diff changeset
    98
    if confirm:
864aa3ea0db5 [server] refactor server.utils.manager_userpasswd
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 1802
diff changeset
    99
        while True:
864aa3ea0db5 [server] refactor server.utils.manager_userpasswd
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 1802
diff changeset
   100
            passwd2 = getpass('confirm password: ')
864aa3ea0db5 [server] refactor server.utils.manager_userpasswd
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 1802
diff changeset
   101
            if passwd == passwd2:
864aa3ea0db5 [server] refactor server.utils.manager_userpasswd
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 1802
diff changeset
   102
                break
10589
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 10095
diff changeset
   103
            print('password doesn\'t match')
1910
864aa3ea0db5 [server] refactor server.utils.manager_userpasswd
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 1802
diff changeset
   104
            passwd = getpass('password: ')
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   105
    # XXX decode password using stdin encoding then encode it using appl'encoding
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   106
    return user, passwd
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   107
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   108
12011
d2888fee6031 [server] introduce a scheduler class to run repository "looping tasks"
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11880
diff changeset
   109
if PY2:
d2888fee6031 [server] introduce a scheduler class to run repository "looping tasks"
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11880
diff changeset
   110
    import time  # noqa
d2888fee6031 [server] introduce a scheduler class to run repository "looping tasks"
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11880
diff changeset
   111
d2888fee6031 [server] introduce a scheduler class to run repository "looping tasks"
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11880
diff changeset
   112
    class scheduler(sched.scheduler):
d2888fee6031 [server] introduce a scheduler class to run repository "looping tasks"
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11880
diff changeset
   113
        """Python2 version of sched.scheduler that matches Python3 API."""
d2888fee6031 [server] introduce a scheduler class to run repository "looping tasks"
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11880
diff changeset
   114
d2888fee6031 [server] introduce a scheduler class to run repository "looping tasks"
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11880
diff changeset
   115
        def __init__(self, **kwargs):
d2888fee6031 [server] introduce a scheduler class to run repository "looping tasks"
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11880
diff changeset
   116
            kwargs.setdefault('timefunc', time.time)
d2888fee6031 [server] introduce a scheduler class to run repository "looping tasks"
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11880
diff changeset
   117
            kwargs.setdefault('delayfunc', time.sleep)
d2888fee6031 [server] introduce a scheduler class to run repository "looping tasks"
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11880
diff changeset
   118
            # sched.scheduler is an old-style class.
d2888fee6031 [server] introduce a scheduler class to run repository "looping tasks"
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11880
diff changeset
   119
            sched.scheduler.__init__(self, **kwargs)
d2888fee6031 [server] introduce a scheduler class to run repository "looping tasks"
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11880
diff changeset
   120
d2888fee6031 [server] introduce a scheduler class to run repository "looping tasks"
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11880
diff changeset
   121
else:
d2888fee6031 [server] introduce a scheduler class to run repository "looping tasks"
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11880
diff changeset
   122
    scheduler = sched.scheduler
d2888fee6031 [server] introduce a scheduler class to run repository "looping tasks"
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11880
diff changeset
   123
d2888fee6031 [server] introduce a scheduler class to run repository "looping tasks"
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11880
diff changeset
   124
d2888fee6031 [server] introduce a scheduler class to run repository "looping tasks"
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11880
diff changeset
   125
def schedule_periodic_task(scheduler, interval, func, *args):
d2888fee6031 [server] introduce a scheduler class to run repository "looping tasks"
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11880
diff changeset
   126
    """Enter a task with `func(*args)` as a periodic event in `scheduler`
d2888fee6031 [server] introduce a scheduler class to run repository "looping tasks"
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11880
diff changeset
   127
    executing at `interval` seconds. Once executed, the task would re-schedule
d2888fee6031 [server] introduce a scheduler class to run repository "looping tasks"
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11880
diff changeset
   128
    itself unless a BaseException got raised.
d2888fee6031 [server] introduce a scheduler class to run repository "looping tasks"
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11880
diff changeset
   129
    """
d2888fee6031 [server] introduce a scheduler class to run repository "looping tasks"
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11880
diff changeset
   130
    @wraps(func)
d2888fee6031 [server] introduce a scheduler class to run repository "looping tasks"
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11880
diff changeset
   131
    def task(*args):
d2888fee6031 [server] introduce a scheduler class to run repository "looping tasks"
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11880
diff changeset
   132
        restart = True
d2888fee6031 [server] introduce a scheduler class to run repository "looping tasks"
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11880
diff changeset
   133
        try:
d2888fee6031 [server] introduce a scheduler class to run repository "looping tasks"
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11880
diff changeset
   134
            func(*args)
d2888fee6031 [server] introduce a scheduler class to run repository "looping tasks"
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11880
diff changeset
   135
        except Exception:
d2888fee6031 [server] introduce a scheduler class to run repository "looping tasks"
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11880
diff changeset
   136
            logger = logging.getLogger('cubicweb.scheduler')
d2888fee6031 [server] introduce a scheduler class to run repository "looping tasks"
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11880
diff changeset
   137
            logger.exception('Unhandled exception in periodic task "%s"',
d2888fee6031 [server] introduce a scheduler class to run repository "looping tasks"
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11880
diff changeset
   138
                             func.__name__)
d2888fee6031 [server] introduce a scheduler class to run repository "looping tasks"
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11880
diff changeset
   139
        except BaseException as exc:
d2888fee6031 [server] introduce a scheduler class to run repository "looping tasks"
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11880
diff changeset
   140
            logger = logging.getLogger('cubicweb.scheduler')
d2888fee6031 [server] introduce a scheduler class to run repository "looping tasks"
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11880
diff changeset
   141
            logger.error('periodic task "%s" not re-scheduled due to %r',
d2888fee6031 [server] introduce a scheduler class to run repository "looping tasks"
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11880
diff changeset
   142
                         func.__name__, exc)
d2888fee6031 [server] introduce a scheduler class to run repository "looping tasks"
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11880
diff changeset
   143
            restart = False
d2888fee6031 [server] introduce a scheduler class to run repository "looping tasks"
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11880
diff changeset
   144
        finally:
d2888fee6031 [server] introduce a scheduler class to run repository "looping tasks"
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11880
diff changeset
   145
            if restart:
d2888fee6031 [server] introduce a scheduler class to run repository "looping tasks"
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11880
diff changeset
   146
                scheduler.enter(interval, 1, task, argument=args)
d2888fee6031 [server] introduce a scheduler class to run repository "looping tasks"
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11880
diff changeset
   147
d2888fee6031 [server] introduce a scheduler class to run repository "looping tasks"
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11880
diff changeset
   148
    return scheduler.enter(interval, 1, task, argument=args)
d2888fee6031 [server] introduce a scheduler class to run repository "looping tasks"
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11880
diff changeset
   149
d2888fee6031 [server] introduce a scheduler class to run repository "looping tasks"
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 11880
diff changeset
   150
9468
39b7a91a3f4c [repo] pylint cleanup, mainly of imports, with a bit of style
Julien Cristau <julien.cristau@logilab.fr>
parents: 9467
diff changeset
   151
_MARKER = object()
12014
116005b5fce9 [server] Make cubicweb/server/utils.py flake8-clean
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12013
diff changeset
   152
116005b5fce9 [server] Make cubicweb/server/utils.py flake8-clean
Denis Laxalde <denis.laxalde@logilab.fr>
parents: 12013
diff changeset
   153
6381
c9eed5037223 [repo threads] Add several safety when looking for a callable name.
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 6128
diff changeset
   154
def func_name(func):
c9eed5037223 [repo threads] Add several safety when looking for a callable name.
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 6128
diff changeset
   155
    name = getattr(func, '__name__', _MARKER)
c9eed5037223 [repo threads] Add several safety when looking for a callable name.
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 6128
diff changeset
   156
    if name is _MARKER:
c9eed5037223 [repo threads] Add several safety when looking for a callable name.
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 6128
diff changeset
   157
        name = getattr(func, 'func_name', _MARKER)
c9eed5037223 [repo threads] Add several safety when looking for a callable name.
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 6128
diff changeset
   158
    if name is _MARKER:
c9eed5037223 [repo threads] Add several safety when looking for a callable name.
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 6128
diff changeset
   159
        name = repr(func)
c9eed5037223 [repo threads] Add several safety when looking for a callable name.
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 6128
diff changeset
   160
    return name
2105
92ea410806fe refactor sources configuration, add source to sources when using a cube defining
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 1977
diff changeset
   161
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   162
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   163
class RepoThread(Thread):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   164
    """subclass of thread so it auto remove itself from a given list once
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   165
    executed
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   166
    """
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   167
    def __init__(self, target, running_threads):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   168
        def auto_remove_func(self=self, func=target):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   169
            try:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   170
                func()
7815
2a164a9cf81c [exceptions] stop catching any exception in various places (closes #1942716)
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 7573
diff changeset
   171
            except Exception:
6765
b922e3a817e9 fix ticket #1382716 (problem was actually more subtle than I originally thought)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6427
diff changeset
   172
                logger = logging.getLogger('cubicweb.repository')
b922e3a817e9 fix ticket #1382716 (problem was actually more subtle than I originally thought)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6427
diff changeset
   173
                logger.exception('Unhandled exception in RepoThread %s', self._name)
b922e3a817e9 fix ticket #1382716 (problem was actually more subtle than I originally thought)
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 6427
diff changeset
   174
                raise
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   175
            finally:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   176
                self.running_threads.remove(self)
1138
22f634977c95 make pylint happy, fix some bugs on the way
sylvain.thenault@logilab.fr
parents: 1134
diff changeset
   177
        Thread.__init__(self, target=auto_remove_func)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   178
        self.running_threads = running_threads
6381
c9eed5037223 [repo threads] Add several safety when looking for a callable name.
Pierre-Yves David <pierre-yves.david@logilab.fr>
parents: 6128
diff changeset
   179
        self._name = func_name(target)
1802
d628defebc17 delete-trailing-whitespace + some copyright update
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 1138
diff changeset
   180
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   181
    def start(self):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   182
        self.running_threads.append(self)
3585
cd437d24aa65 use daemon thread
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2708
diff changeset
   183
        self.daemon = True
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   184
        Thread.start(self)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   185
5376
2c3f14bc2590 [python2.6] don't add a name property on Thread
Alexandre Fayolle <alexandre.fayolle@logilab.fr>
parents: 5066
diff changeset
   186
    def getName(self):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   187
        return '%s(%s)' % (self._name, Thread.getName(self))