cubicweb/statsd_logger.py
author Philippe Pepiot <philippe.pepiot@logilab.fr>
Thu, 19 Jan 2017 15:27:39 +0100
changeset 11899 bf6106b91633
parent 11767 432f87a63057
child 12291 b640ef6ad633
permissions -rw-r--r--
[schema] load schema from modules names instead of directories Introspect cubicweb, cubes and apphome using pkgutil to generate the full list of modules names for loading the schema. Keep historical behavior and check if source .py file exists if a module is found using python bytecode file (.pyc and .pyo) Loading schema from apphome require apphome to be present in sys.path and that "schema" module resolve to a file located in apphome. Update migraction tests to explicitely update sys.path when loading schema from different apps, use a contextmanager for this so it's more readable. Require updated logilab-common and yams
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
10477
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
     1
# copyright 2015 LOGILAB S.A. (Paris, FRANCE), all rights reserved.
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
     2
# contact http://www.logilab.fr/ -- mailto:contact@logilab.fr
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
     3
#
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
     4
# This file is part of CubicWeb.
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
     5
#
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
     6
# CubicWeb is free software: you can redistribute it and/or modify it under the
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
     7
# terms of the GNU Lesser General Public License as published by the Free
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
     8
# Software Foundation, either version 2.1 of the License, or (at your option)
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
     9
# any later version.
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    10
#
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    11
# CubicWeb is distributed in the hope that it will be useful, but WITHOUT
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    12
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    13
# FOR A PARTICULAR PURPOSE.  See the GNU Lesser General Public License for more
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    14
# details.
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    15
#
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    16
# You should have received a copy of the GNU Lesser General Public License along
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    17
# with CubicWeb.  If not, see <http://www.gnu.org/licenses/>.
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    18
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    19
"""Simple statsd_ logger for cubicweb.
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    20
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    21
This module is meant to be configured by setting a couple of global variables:
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    22
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    23
- ``bucket`` global variable will be used as statsd bucket in every
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    24
statsd_ UDP sent packet.
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    25
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    26
`- `address`` is a pair (IP, port) specifying the address of the
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    27
statsd_ server
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    28
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    29
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    30
There are 3 kinds of statds_ message::
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    31
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    32
- ``statsd_c(context, n)`` is a simple function to send statsd_
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    33
  counter-type of messages like::
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    34
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    35
    <bucket>.<context>:<n>|c\n
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    36
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    37
- ``statsd_g(context, value)`` to send statsd_ gauge-type of messages
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    38
  like::
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    39
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    40
    <bucket>.<context>:<n>|g\n
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    41
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    42
- ``statsd_t(context, ms)`` to send statsd_ time-type of messages
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    43
  like::
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    44
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    45
    <bucket>.<context>:<ms>|ms\n
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    46
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    47
There is also a decorator (``statsd_timeit``) that may be used to
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    48
measure and send to the statsd_ server the time passed in a function
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    49
or a method and the number of calls. It will send a message like::
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    50
   
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    51
    <bucket>.<funcname>:<ms>|ms\n<bucket>.<funcname>:1|c\n
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    52
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    53
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    54
.. _statsd: https://github.com/etsy/statsd
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    55
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    56
"""
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    57
11767
432f87a63057 flake8 and all
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 11057
diff changeset
    58
10477
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    59
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    60
import time
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    61
import socket
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    62
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    63
_bucket = 'cubicweb'
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    64
_address = None
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    65
_socket = None
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    66
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    67
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    68
def setup(bucket, address):
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    69
    """Configure the statsd endpoint
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    70
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    71
    :param bucket: the name of the statsd bucket that will be used to
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    72
                   build messages.
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    73
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    74
    :param address: the UDP endpoint of the statsd server. Must a
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    75
                    couple (ip, port).
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    76
    """
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    77
    global _bucket, _address, _socket
10478
026b4ee032fb [statsd_logger] handle ipv6 addresses
Julien Cristau <julien.cristau@logilab.fr>
parents: 10477
diff changeset
    78
    packed = None
026b4ee032fb [statsd_logger] handle ipv6 addresses
Julien Cristau <julien.cristau@logilab.fr>
parents: 10477
diff changeset
    79
    for family in (socket.AF_INET6, socket.AF_INET):
026b4ee032fb [statsd_logger] handle ipv6 addresses
Julien Cristau <julien.cristau@logilab.fr>
parents: 10477
diff changeset
    80
        try:
10650
28b3d39bcbc6 [statsd] fix the statsd logger (closes #7558147)
David Douard <david.douard@logilab.fr>
parents: 10478
diff changeset
    81
            packed = socket.inet_pton(family, address[0])
10478
026b4ee032fb [statsd_logger] handle ipv6 addresses
Julien Cristau <julien.cristau@logilab.fr>
parents: 10477
diff changeset
    82
            break
026b4ee032fb [statsd_logger] handle ipv6 addresses
Julien Cristau <julien.cristau@logilab.fr>
parents: 10477
diff changeset
    83
        except socket.error:
026b4ee032fb [statsd_logger] handle ipv6 addresses
Julien Cristau <julien.cristau@logilab.fr>
parents: 10477
diff changeset
    84
            continue
026b4ee032fb [statsd_logger] handle ipv6 addresses
Julien Cristau <julien.cristau@logilab.fr>
parents: 10477
diff changeset
    85
    if packed is None:
026b4ee032fb [statsd_logger] handle ipv6 addresses
Julien Cristau <julien.cristau@logilab.fr>
parents: 10477
diff changeset
    86
        return
10477
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    87
    _bucket, _address = bucket, address
10478
026b4ee032fb [statsd_logger] handle ipv6 addresses
Julien Cristau <julien.cristau@logilab.fr>
parents: 10477
diff changeset
    88
    _socket = socket.socket(family, socket.SOCK_DGRAM)
10477
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    89
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    90
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    91
def statsd_c(context, n=1):
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    92
    if _address is not None:
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    93
        _socket.sendto('{0}.{1}:{2}|c\n'.format(_bucket, context, n), _address)
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    94
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    95
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    96
def statsd_g(context, value):
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    97
    if _address is not None:
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    98
        _socket.sendto('{0}.{1}:{2}|g\n'.format(_bucket, context, value), _address)
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
    99
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
   100
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
   101
def statsd_t(context, value):
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
   102
    if _address is not None:
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
   103
        _socket.sendto('{0}.{1}:{2:.4f}|ms\n'.format(_bucket, context, value), _address)
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
   104
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
   105
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
   106
class statsd_timeit(object):
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
   107
    __slots__ = ('callable',)
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
   108
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
   109
    def __init__(self, callableobj):
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
   110
        self.callable = callableobj
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
   111
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
   112
    @property
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
   113
    def __doc__(self):
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
   114
        return self.callable.__doc__
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
   115
    @property
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
   116
    def __name__(self):
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
   117
        return self.callable.__name__
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
   118
    
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
   119
    def __call__(self, *args, **kw):
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
   120
        if _address is None:
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
   121
            return self.callable(*args, **kw)
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
   122
        t0 = time.time()
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
   123
        try:
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
   124
            return self.callable(*args, **kw)
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
   125
        finally:
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
   126
            dt = 1000*(time.time()-t0)
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
   127
            msg = '{0}.{1}:{2:.4f}|ms\n{0}.{1}:1|c\n'.format(_bucket, self.__name__, dt)
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
   128
            _socket.sendto(msg, _address)
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
   129
                
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
   130
    def __get__(self, obj, objtype):
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
   131
        """Support instance methods."""
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
   132
        if obj is None: # class method or some already wrapped method
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
   133
            return self
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
   134
        import functools
ee21c559f94f implement a simple statsd logger (closes #5488711)
David Douard <david.douard@logilab.fr>
parents:
diff changeset
   135
        return functools.partial(self.__call__, obj)