1 # -*- coding: utf-8 -*-

     2 # copyright 2003-2012 LOGILAB S.A. (Paris, FRANCE), all rights reserved.

     3 # contact http://www.logilab.fr/ -- mailto:contact@logilab.fr

     4 #

     5 # This file is part of CubicWeb.

     6 #

     7 # CubicWeb is free software: you can redistribute it and/or modify it under the

     8 # terms of the GNU Lesser General Public License as published by the Free

     9 # Software Foundation, either version 2.1 of the License, or (at your option)

    10 # any later version.

    11 #

    12 # CubicWeb is distributed in the hope that it will be useful, but WITHOUT

    13 # ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS

    14 # FOR A PARTICULAR PURPOSE.  See the GNU Lesser General Public License for more

    15 # details.

    16 #

    17 # You should have received a copy of the GNU Lesser General Public License along

    18 # with CubicWeb.  If not, see <http://www.gnu.org/licenses/>.

    19 """

    20 .. _ResourceMode:

    21 

    22 Resource mode

    23 -------------

    24 

    25 Standard resource mode

    26 ```````````````````````````

    27 

    28 A resource *mode* is a predefined set of settings for various resources

    29 directories, such as cubes, instances, etc. to ease development with the

    30 framework. There are two running modes with *CubicWeb*:

    31 

    32 * **system**: resources are searched / created in the system directories (eg

    33   usually requiring root access):

    34 

    35   - instances are stored in :file:`<INSTALL_PREFIX>/etc/cubicweb.d`

    36   - temporary files (such as pid file) in :file:`<INSTALL_PREFIX>/var/run/cubicweb`

    37 

    38   where `<INSTALL_PREFIX>` is the detected installation prefix ('/usr/local' for

    39   instance).

    40 

    41 * **user**: resources are searched / created in the user home directory:

    42 

    43   - instances are stored in :file:`~/etc/cubicweb.d`

    44   - temporary files (such as pid file) in :file:`/tmp`

    45 

    46 

    47 

    48 

    49 .. _CubicwebWithinVirtualEnv:

    50 

    51 Within virtual environment

    52 ```````````````````````````

    53 

    54 If you are not administrator of you machine or if you need to play with some

    55 specific version of |cubicweb| you can use `virtualenv`_ a tool to create

    56 isolated Python environments.

    57 

    58 - instances are stored in :file:`<VIRTUAL_ENV>/etc/cubicweb.d`

    59 - temporary files (such as pid file) in :file:`<VIRTUAL_ENV>/var/run/cubicweb`

    60 

    61 .. _`virtualenv`: http://pypi.python.org/pypi/virtualenv

    62 

    63 Custom resource location

    64 ````````````````````````````````

    65 

    66 Notice that each resource path may be explicitly set using an environment

    67 variable if the default doesn't suit your needs. Here are the default resource

    68 directories that are affected according to mode:

    69 

    70 * **system**: ::

    71 

    72         CW_INSTANCES_DIR = <INSTALL_PREFIX>/etc/cubicweb.d/

    73         CW_INSTANCES_DATA_DIR = <INSTALL_PREFIX>/var/lib/cubicweb/instances/

    74         CW_RUNTIME_DIR = <INSTALL_PREFIX>/var/run/cubicweb/

    75 

    76 * **user**: ::

    77 

    78         CW_INSTANCES_DIR = ~/etc/cubicweb.d/

    79         CW_INSTANCES_DATA_DIR = ~/etc/cubicweb.d/

    80         CW_RUNTIME_DIR = /tmp

    81 

    82 Cubes search path is also affected, see the :ref:`Cube` section.

    83 

    84 Setting Cubicweb Mode

    85 `````````````````````

    86 

    87 By default, the mode is set to 'system' for standard installation. The mode is

    88 set to 'user' if `cubicweb is used from a mercurial repository`_. You can force

    89 this by setting the :envvar:`CW_MODE` environment variable to either 'user' or

    90 'system' so you can easily:

    91 

    92 * use system wide installation but user specific instances and all, without root

    93   privileges on the system (`export CW_MODE=user`)

    94 

    95 * use local checkout of cubicweb on system wide instances (requires root

    96   privileges on the system (`export CW_MODE=system`)

    97 

    98 If you've a doubt about the mode you're currently running, check the first line

    99 outputed by the :command:`cubicweb-ctl list` command.

   100 

   101 .. _`cubicweb is used from a mercurial repository`: CubicwebDevelopmentMod_

   102 

   103 .. _CubicwebDevelopmentMod:

   104 

   105 Development Mode

   106 `````````````````````

   107 If :file:`.hg` directory is found into the cubicweb package, there are specific resource rules.

   108 

   109 `<CW_SOFTWARE_ROOT>` is the source checkout's ``cubicweb`` directory:

   110 

   111 * main cubes directory is `<CW_SOFTWARE_ROOT>/../../cubes`. You can specify

   112   another one with :envvar:`CW_INSTANCES_DIR` environment variable or simply

   113   add some other directories by using :envvar:`CW_CUBES_PATH`

   114 

   115 * cubicweb migration files are searched in `<CW_SOFTWARE_ROOT>/misc/migration`

   116   instead of `<INSTALL_PREFIX>/share/cubicweb/migration/`.

   117 

   118 

   119 .. _ConfigurationEnv:

   120 

   121 Environment configuration

   122 -------------------------

   123 

   124 Python

   125 ``````

   126 

   127 If you installed *CubicWeb* by cloning the Mercurial shell repository or from source

   128 distribution, then you will need to update the environment variable PYTHONPATH by

   129 adding the path to `cubicweb`:

   130 

   131 Add the following lines to either :file:`.bashrc` or :file:`.bash_profile` to

   132 configure your development environment ::

   133 

   134     export PYTHONPATH=/full/path/to/grshell-cubicweb

   135 

   136 If you installed *CubicWeb* with packages, no configuration is required and your

   137 new cubes will be placed in `/usr/share/cubicweb/cubes` and your instances will

   138 be placed in `/etc/cubicweb.d`.

   139 

   140 

   141 CubicWeb

   142 ````````

   143 

   144 Here are all environment variables that may be used to configure *CubicWeb*:

   145 

   146 .. envvar:: CW_MODE

   147 

   148    Resource mode: user or system, as explained in :ref:`ResourceMode`.

   149 

   150 .. envvar:: CW_CUBES_PATH

   151 

   152    Augments the default search path for cubes. You may specify several

   153    directories using ':' as separator (';' under windows environment).

   154 

   155 .. envvar:: CW_INSTANCES_DIR

   156 

   157    Directory where cubicweb instances will be found.

   158 

   159 .. envvar:: CW_INSTANCES_DATA_DIR

   160 

   161    Directory where cubicweb instances data will be written (backup file...)

   162 

   163 .. envvar:: CW_RUNTIME_DIR

   164 

   165    Directory where pid files will be written

   166 """

   167 from __future__ import print_function

   168 

   169 __docformat__ = "restructuredtext en"

   170 

   171 import sys

   172 import os

   173 import stat

   174 import logging

   175 import logging.config

   176 from smtplib import SMTP

   177 from threading import Lock

   178 from os.path import (exists, join, expanduser, abspath, normpath,

   179                      basename, isdir, dirname, splitext)

   180 from warnings import warn, filterwarnings

   181 

   182 from six import text_type

   183 

   184 from logilab.common.decorators import cached, classproperty

   185 from logilab.common.deprecation import deprecated

   186 from logilab.common.logging_ext import set_log_methods, init_log

   187 from logilab.common.configuration import (Configuration, Method,

   188                                           ConfigurationMixIn, merge_options)

   189 

   190 from cubicweb import (CW_SOFTWARE_ROOT, CW_MIGRATION_MAP,

   191                       ConfigurationError, Binary, _)

   192 from cubicweb.toolsutils import create_dir

   193 

   194 CONFIGURATIONS = []

   195 

   196 SMTP_LOCK = Lock()

   197 

   198 

   199 def configuration_cls(name):

   200     """return the configuration class registered with the given name"""

   201     try:

   202         return [c for c in CONFIGURATIONS if c.name == name][0]

   203     except IndexError:

   204         raise ConfigurationError('no such config %r (check it exists with "cubicweb-ctl list")' % name)

   205 

   206 def possible_configurations(directory):

   207     """return a list of installed configurations in a directory

   208     according to \*-ctl files

   209     """

   210     return [name for name in ('repository', 'all-in-one')

   211             if exists(join(directory, '%s.conf' % name))]

   212 

   213 def guess_configuration(directory):

   214     """try to guess the configuration to use for a directory. If multiple

   215     configurations are found, ConfigurationError is raised

   216     """

   217     modes = possible_configurations(directory)

   218     if len(modes) != 1:

   219         raise ConfigurationError('unable to guess configuration from %r %s'

   220                                  % (directory, modes))

   221     return modes[0]

   222 

   223 def _find_prefix(start_path=CW_SOFTWARE_ROOT):

   224     """Runs along the parent directories of *start_path* (default to cubicweb source directory)

   225     looking for one containing a 'share/cubicweb' directory.

   226     The first matching directory is assumed as the prefix installation of cubicweb

   227 

   228     Returns the matching prefix or None.

   229     """

   230     prefix = start_path

   231     old_prefix = None

   232     if not isdir(start_path):

   233         prefix = dirname(start_path)

   234     while (not isdir(join(prefix, 'share', 'cubicweb'))

   235           or prefix.endswith('.egg')) and prefix != old_prefix:

   236         old_prefix = prefix

   237         prefix = dirname(prefix)

   238     if isdir(join(prefix, 'share', 'cubicweb')):

   239         return prefix

   240     return sys.prefix

   241 

   242 # persistent options definition

   243 PERSISTENT_OPTIONS = (

   244     ('encoding',

   245      {'type' : 'string',

   246       'default': 'UTF-8',

   247       'help': _('user interface encoding'),

   248       'group': 'ui', 'sitewide': True,

   249       }),

   250     ('language',

   251      {'type' : 'string',

   252       'default': 'en',

   253       'vocabulary': Method('available_languages'),

   254       'help': _('language of the user interface'),

   255       'group': 'ui',

   256       }),

   257     ('date-format',

   258      {'type' : 'string',

   259       'default': '%Y/%m/%d',

   260       'help': _('how to format date in the ui (see <a href="http://docs.python.org/library/datetime.html#strftime-strptime-behavior">this page</a> for format description)'),

   261       'group': 'ui',

   262       }),

   263     ('datetime-format',

   264      {'type' : 'string',

   265       'default': '%Y/%m/%d %H:%M',

   266       'help': _('how to format date and time in the ui (see <a href="http://docs.python.org/library/datetime.html#strftime-strptime-behavior">this page</a> for format description)'),

   267       'group': 'ui',

   268       }),

   269     ('time-format',

   270      {'type' : 'string',

   271       'default': '%H:%M',

   272       'help': _('how to format time in the ui (see <a href="http://docs.python.org/library/datetime.html#strftime-strptime-behavior">this page</a> for format description)'),

   273       'group': 'ui',

   274       }),

   275     ('float-format',

   276      {'type' : 'string',

   277       'default': '%.3f',

   278       'help': _('how to format float numbers in the ui'),

   279       'group': 'ui',

   280       }),

   281     ('default-text-format',

   282      {'type' : 'choice',

   283       'choices': ('text/plain', 'text/rest', 'text/html', 'text/markdown'),

   284       'default': 'text/plain',

   285       'help': _('default text format for rich text fields.'),

   286       'group': 'ui',

   287       }),

   288     ('short-line-size',

   289      {'type' : 'int',

   290       'default': 80,

   291       'help': _('maximum number of characters in short description'),

   292       'group': 'navigation',

   293       }),

   294     )

   295 

   296 def register_persistent_options(options):

   297     global PERSISTENT_OPTIONS

   298     PERSISTENT_OPTIONS = merge_options(PERSISTENT_OPTIONS + options)

   299 

   300 CFGTYPE2ETYPE_MAP = {

   301     'string': 'String',

   302     'choice': 'String',

   303     'yn':     'Boolean',

   304     'int':    'Int',

   305     'float' : 'Float',

   306     }

   307 

   308 _forced_mode = os.environ.get('CW_MODE')

   309 assert _forced_mode in (None, 'system', 'user')

   310 

   311 # CWDEV tells whether directories such as i18n/, web/data/, etc. (ie containing

   312 # some other resources than python libraries) are located with the python code

   313 # or as a 'shared' cube

   314 CWDEV = exists(join(CW_SOFTWARE_ROOT, 'i18n'))

   315 

   316 try:

   317     _INSTALL_PREFIX = os.environ['CW_INSTALL_PREFIX']

   318 except KeyError:

   319     _INSTALL_PREFIX = _find_prefix()

   320 _USR_INSTALL = _INSTALL_PREFIX == '/usr'

   321 

   322 class CubicWebNoAppConfiguration(ConfigurationMixIn):

   323     """base class for cubicweb configuration without a specific instance directory

   324     """

   325     # to set in concrete configuration

   326     name = None

   327     # log messages format (see logging module documentation for available keys)

   328     log_format = '%(asctime)s - (%(name)s) %(levelname)s: %(message)s'

   329     # the format below can be useful to debug multi thread issues:

   330     # log_format = '%(asctime)s - [%(threadName)s] (%(name)s) %(levelname)s: %(message)s'

   331     # nor remove appobjects based on unused interface [???]

   332     cleanup_unused_appobjects = True

   333 

   334     quick_start = False

   335 

   336     if (CWDEV and _forced_mode != 'system'):

   337         mode = 'user'

   338         _CUBES_DIR = join(CW_SOFTWARE_ROOT, '../../cubes')

   339     else:

   340         mode = _forced_mode or 'system'

   341         _CUBES_DIR = join(_INSTALL_PREFIX, 'share', 'cubicweb', 'cubes')

   342 

   343     CUBES_DIR = abspath(os.environ.get('CW_CUBES_DIR', _CUBES_DIR))

   344     CUBES_PATH = os.environ.get('CW_CUBES_PATH', '').split(os.pathsep)

   345 

   346     options = (

   347        ('log-threshold',

   348          {'type' : 'string', # XXX use a dedicated type?

   349           'default': 'WARNING',

   350           'help': 'server\'s log level',

   351           'group': 'main', 'level': 1,

   352           }),

   353         ('umask',

   354          {'type' : 'int',

   355           'default': 0o077,

   356           'help': 'permission umask for files created by the server',

   357           'group': 'main', 'level': 2,

   358           }),

   359         # common configuration options which are potentially required as soon as

   360         # you're using "base" application objects (ie to really server/web

   361         # specific)

   362         ('base-url',

   363          {'type' : 'string',

   364           'default': None,

   365           'help': 'web server root url',

   366           'group': 'main', 'level': 1,

   367           }),

   368         ('allow-email-login',

   369          {'type' : 'yn',

   370           'default': False,

   371           'help': 'allow users to login with their primary email if set',

   372           'group': 'main', 'level': 2,

   373           }),

   374         ('mangle-emails',

   375          {'type' : 'yn',

   376           'default': False,

   377           'help': "don't display actual email addresses but mangle them if \

   378 this option is set to yes",

   379           'group': 'email', 'level': 3,

   380           }),

   381         )

   382     # static and class methods used to get instance independant resources ##

   383     @staticmethod

   384     def cubicweb_version():

   385         """return installed cubicweb version"""

   386         from logilab.common.changelog import Version

   387         from cubicweb import __pkginfo__

   388         version = __pkginfo__.numversion

   389         assert len(version) == 3, version

   390         return Version(version)

   391 

   392     @staticmethod

   393     def persistent_options_configuration():

   394         return Configuration(options=PERSISTENT_OPTIONS)

   395 

   396     @classmethod

   397     def shared_dir(cls):

   398         """return the shared data directory (i.e. directory where standard

   399         library views and data may be found)

   400         """

   401         if CWDEV:

   402             return join(CW_SOFTWARE_ROOT, 'web')

   403         return cls.cube_dir('shared')

   404 

   405     @classmethod

   406     def i18n_lib_dir(cls):

   407         """return instance's i18n directory"""

   408         if CWDEV:

   409             return join(CW_SOFTWARE_ROOT, 'i18n')

   410         return join(cls.shared_dir(), 'i18n')

   411 

   412     @classmethod

   413     def cw_languages(cls):

   414         for fname in os.listdir(join(cls.i18n_lib_dir())):

   415             if fname.endswith('.po'):

   416                 yield splitext(fname)[0]

   417 

   418 

   419     @classmethod

   420     def available_cubes(cls):

   421         import re

   422         cubes = set()

   423         for directory in cls.cubes_search_path():

   424             if not exists(directory):

   425                 cls.error('unexistant directory in cubes search path: %s'

   426                           % directory)

   427                 continue

   428             for cube in os.listdir(directory):

   429                 if cube == 'shared':

   430                     continue

   431                 if not re.match('[_A-Za-z][_A-Za-z0-9]*$', cube):

   432                     continue # skip invalid python package name

   433                 cubedir = join(directory, cube)

   434                 if isdir(cubedir) and exists(join(cubedir, '__init__.py')):

   435                     cubes.add(cube)

   436         return sorted(cubes)

   437 

   438     @classmethod

   439     def cubes_search_path(cls):

   440         """return the path of directories where cubes should be searched"""

   441         path = [abspath(normpath(directory)) for directory in cls.CUBES_PATH

   442                 if directory.strip() and exists(directory.strip())]

   443         if not cls.CUBES_DIR in path and exists(cls.CUBES_DIR):

   444             path.append(cls.CUBES_DIR)

   445         return path

   446 

   447     @classproperty

   448     def extrapath(cls):

   449         extrapath = {}

   450         for cubesdir in cls.cubes_search_path():

   451             if cubesdir != cls.CUBES_DIR:

   452                 extrapath[cubesdir] = 'cubes'

   453         return extrapath

   454 

   455     @classmethod

   456     def cube_dir(cls, cube):

   457         """return the cube directory for the given cube id, raise

   458         `ConfigurationError` if it doesn't exist

   459         """

   460         for directory in cls.cubes_search_path():

   461             cubedir = join(directory, cube)

   462             if exists(cubedir):

   463                 return cubedir

   464         raise ConfigurationError('no cube %r in %s' % (

   465             cube, cls.cubes_search_path()))

   466 

   467     @classmethod

   468     def cube_migration_scripts_dir(cls, cube):

   469         """cube migration scripts directory"""

   470         return join(cls.cube_dir(cube), 'migration')

   471 

   472     @classmethod

   473     def cube_pkginfo(cls, cube):

   474         """return the information module for the given cube"""

   475         cube = CW_MIGRATION_MAP.get(cube, cube)

   476         try:

   477             parent = __import__('cubes.%s.__pkginfo__' % cube)

   478             return getattr(parent, cube).__pkginfo__

   479         except Exception as ex:

   480             raise ConfigurationError(

   481                 'unable to find packaging information for cube %s (%s: %s)'

   482                 % (cube, ex.__class__.__name__, ex))

   483 

   484     @classmethod

   485     def cube_version(cls, cube):

   486         """return the version of the cube located in the given directory

   487         """

   488         from logilab.common.changelog import Version

   489         version = cls.cube_pkginfo(cube).numversion

   490         assert len(version) == 3, version

   491         return Version(version)

   492 

   493     @classmethod

   494     def _cube_deps(cls, cube, key, oldkey):

   495         """return cubicweb cubes used by the given cube"""

   496         pkginfo = cls.cube_pkginfo(cube)

   497         try:

   498             # explicit __xxx_cubes__ attribute

   499             deps = getattr(pkginfo, key)

   500         except AttributeError:

   501             # deduce cubes from generic __xxx__ attribute

   502             try:

   503                 gendeps = getattr(pkginfo, key.replace('_cubes', ''))

   504             except AttributeError:

   505                 deps = {}

   506             else:

   507                 deps = dict( (x[len('cubicweb-'):], v)

   508                              for x, v in gendeps.items()

   509                              if x.startswith('cubicweb-'))

   510         for depcube in deps:

   511             try:

   512                 newname = CW_MIGRATION_MAP[depcube]

   513             except KeyError:

   514                 pass

   515             else:

   516                 deps[newname] = deps.pop(depcube)

   517         return deps

   518 

   519     @classmethod

   520     def cube_depends_cubicweb_version(cls, cube):

   521         # XXX no backward compat (see _cube_deps above)

   522         try:

   523             pkginfo = cls.cube_pkginfo(cube)

   524             deps = getattr(pkginfo, '__depends__')

   525             return deps.get('cubicweb')

   526         except AttributeError:

   527             return None

   528 

   529     @classmethod

   530     def cube_dependencies(cls, cube):

   531         """return cubicweb cubes used by the given cube"""

   532         return cls._cube_deps(cube, '__depends_cubes__', '__use__')

   533 

   534     @classmethod

   535     def cube_recommends(cls, cube):

   536         """return cubicweb cubes recommended by the given cube"""

   537         return cls._cube_deps(cube, '__recommends_cubes__', '__recommend__')

   538 

   539     @classmethod

   540     def expand_cubes(cls, cubes, with_recommends=False):

   541         """expand the given list of top level cubes used by adding recursivly

   542         each cube dependencies

   543         """

   544         cubes = list(cubes)

   545         todo = cubes[:]

   546         if with_recommends:

   547             available = set(cls.available_cubes())

   548         while todo:

   549             cube = todo.pop(0)

   550             for depcube in cls.cube_dependencies(cube):

   551                 if depcube not in cubes:

   552                     cubes.append(depcube)

   553                     todo.append(depcube)

   554             if with_recommends:

   555                 for depcube in cls.cube_recommends(cube):

   556                     if depcube not in cubes and depcube in available:

   557                         cubes.append(depcube)

   558                         todo.append(depcube)

   559         return cubes

   560 

   561     @classmethod

   562     def reorder_cubes(cls, cubes):

   563         """reorder cubes from the top level cubes to inner dependencies

   564         cubes

   565         """

   566         from logilab.common.graph import ordered_nodes, UnorderableGraph

   567         graph = {}

   568         for cube in cubes:

   569             cube = CW_MIGRATION_MAP.get(cube, cube)

   570             graph[cube] = set(dep for dep in cls.cube_dependencies(cube)

   571                               if dep in cubes)

   572             graph[cube] |= set(dep for dep in cls.cube_recommends(cube)

   573                                if dep in cubes)

   574         try:

   575             return ordered_nodes(graph)

   576         except UnorderableGraph as ex:

   577             raise ConfigurationError(ex)

   578 

   579     @classmethod

   580     def cls_adjust_sys_path(cls):

   581         """update python path if necessary"""

   582         cubes_parent_dir = normpath(join(cls.CUBES_DIR, '..'))

   583         if not cubes_parent_dir in sys.path:

   584             sys.path.insert(0, cubes_parent_dir)

   585         try:

   586             import cubes

   587             cubes.__path__ = cls.cubes_search_path()

   588         except ImportError:

   589             return # cubes dir doesn't exists

   590 

   591     @classmethod

   592     def load_available_configs(cls):

   593         from logilab.common.modutils import load_module_from_file

   594         for conffile in ('web/webconfig.py',  'etwist/twconfig.py',

   595                         'server/serverconfig.py',):

   596             if exists(join(CW_SOFTWARE_ROOT, conffile)):

   597                 load_module_from_file(join(CW_SOFTWARE_ROOT, conffile))

   598 

   599     @classmethod

   600     def load_cwctl_plugins(cls):

   601         from logilab.common.modutils import load_module_from_file

   602         cls.cls_adjust_sys_path()

   603         for ctlfile in ('web/webctl.py',  'etwist/twctl.py',

   604                         'server/serverctl.py',

   605                         'devtools/devctl.py', 'goa/goactl.py'):

   606             if exists(join(CW_SOFTWARE_ROOT, ctlfile)):

   607                 try:

   608                     load_module_from_file(join(CW_SOFTWARE_ROOT, ctlfile))

   609                 except ImportError as err:

   610                     cls.error('could not import the command provider %s: %s',

   611                               ctlfile, err)

   612                 cls.info('loaded cubicweb-ctl plugin %s', ctlfile)

   613         for cube in cls.available_cubes():

   614             pluginfile = join(cls.cube_dir(cube), 'ccplugin.py')

   615             initfile = join(cls.cube_dir(cube), '__init__.py')

   616             if exists(pluginfile):

   617                 try:

   618                     __import__('cubes.%s.ccplugin' % cube)

   619                     cls.info('loaded cubicweb-ctl plugin from %s', cube)

   620                 except Exception:

   621                     cls.exception('while loading plugin %s', pluginfile)

   622             elif exists(initfile):

   623                 try:

   624                     __import__('cubes.%s' % cube)

   625                 except Exception:

   626                     cls.exception('while loading cube %s', cube)

   627             else:

   628                 cls.warning('no __init__ file in cube %s', cube)

   629 

   630     @classmethod

   631     def init_available_cubes(cls):

   632         """cubes may register some sources (svnfile for instance) in their

   633         __init__ file, so they should be loaded early in the startup process

   634         """

   635         for cube in cls.available_cubes():

   636             try:

   637                 __import__('cubes.%s' % cube)

   638             except Exception as ex:

   639                 cls.warning("can't init cube %s: %s", cube, ex)

   640 

   641     cubicweb_appobject_path = set(['entities'])

   642     cube_appobject_path = set(['entities'])

   643 

   644     def __init__(self, debugmode=False):

   645         if debugmode:

   646             # in python 2.7, DeprecationWarning are not shown anymore by default

   647             filterwarnings('default', category=DeprecationWarning)

   648         register_stored_procedures()

   649         self._cubes = None

   650         super(CubicWebNoAppConfiguration, self).__init__()

   651         self.debugmode = debugmode

   652         self.adjust_sys_path()

   653         self.load_defaults()

   654         # will be properly initialized later by _gettext_init

   655         self.translations = {'en': (text_type, lambda ctx, msgid: text_type(msgid) )}

   656         self._site_loaded = set()

   657         # don't register ReStructured Text directives by simple import, avoid pb

   658         # with eg sphinx.

   659         # XXX should be done properly with a function from cw.uicfg

   660         try:

   661             from cubicweb.ext.rest import cw_rest_init

   662         except ImportError:

   663             pass

   664         else:

   665             cw_rest_init()

   666 

   667     def adjust_sys_path(self):

   668         # overriden in CubicWebConfiguration

   669         self.cls_adjust_sys_path()

   670 

   671     def init_log(self, logthreshold=None, logfile=None, syslog=False):

   672         """init the log service"""

   673         if logthreshold is None:

   674             if self.debugmode:

   675                 logthreshold = 'DEBUG'

   676             else:

   677                 logthreshold = self['log-threshold']

   678         if sys.platform == 'win32':

   679             # no logrotate on win32, so use logging rotation facilities

   680             # for now, hard code weekly rotation every sunday, and 52 weeks kept

   681             # idea: make this configurable?

   682             init_log(self.debugmode, syslog, logthreshold, logfile, self.log_format,

   683                      rotation_parameters={'when': 'W6', # every sunday

   684                                           'interval': 1,

   685                                           'backupCount': 52})

   686         else:

   687             init_log(self.debugmode, syslog, logthreshold, logfile, self.log_format)

   688         # configure simpleTal logger

   689         logging.getLogger('simpleTAL').setLevel(logging.ERROR)

   690 

   691     def appobjects_path(self):

   692         """return a list of files or directories where the registry will look

   693         for application objects. By default return nothing in NoApp config.

   694         """

   695         return []

   696 

   697     def build_appobjects_path(self, templpath, evobjpath=None, tvobjpath=None):

   698         """given a list of directories, return a list of sub files and

   699         directories that should be loaded by the instance objects registry.

   700 

   701         :param evobjpath:

   702           optional list of sub-directories (or files without the .py ext) of

   703           the cubicweb library that should be tested and added to the output list

   704           if they exists. If not give, default to `cubicweb_appobject_path` class

   705           attribute.

   706         :param tvobjpath:

   707           optional list of sub-directories (or files without the .py ext) of

   708           directories given in `templpath` that should be tested and added to

   709           the output list if they exists. If not give, default to

   710           `cube_appobject_path` class attribute.

   711         """

   712         vregpath = self.build_appobjects_cubicweb_path(evobjpath)

   713         vregpath += self.build_appobjects_cube_path(templpath, tvobjpath)

   714         return vregpath

   715 

   716     def build_appobjects_cubicweb_path(self, evobjpath=None):

   717         vregpath = []

   718         if evobjpath is None:

   719             evobjpath = self.cubicweb_appobject_path

   720         # NOTE: for the order, see http://www.cubicweb.org/ticket/2330799

   721         #       it is clearly a workaround

   722         for subdir in sorted(evobjpath, key=lambda x:x != 'entities'):

   723             path = join(CW_SOFTWARE_ROOT, subdir)

   724             if exists(path):

   725                 vregpath.append(path)

   726         return vregpath

   727 

   728     def build_appobjects_cube_path(self, templpath, tvobjpath=None):

   729         vregpath = []

   730         if tvobjpath is None:

   731             tvobjpath = self.cube_appobject_path

   732         for directory in templpath:

   733             # NOTE: for the order, see http://www.cubicweb.org/ticket/2330799

   734             for subdir in sorted(tvobjpath, key=lambda x:x != 'entities'):

   735                 path = join(directory, subdir)

   736                 if exists(path):

   737                     vregpath.append(path)

   738                 elif exists(path + '.py'):

   739                     vregpath.append(path + '.py')

   740         return vregpath

   741 

   742     apphome = None

   743 

   744     def load_site_cubicweb(self, paths=None):

   745         """load instance's specific site_cubicweb file"""

   746         if paths is None:

   747             paths = self.cubes_path()

   748             if self.apphome is not None:

   749                 paths = [self.apphome] + paths

   750         for path in reversed(paths):

   751             sitefile = join(path, 'site_cubicweb.py')

   752             if exists(sitefile) and not sitefile in self._site_loaded:

   753                 self._load_site_cubicweb(sitefile)

   754                 self._site_loaded.add(sitefile)

   755 

   756     def _load_site_cubicweb(self, sitefile):

   757         # XXX extrapath argument to load_module_from_file only in lgc > 0.50.2

   758         from logilab.common.modutils import load_module_from_modpath, modpath_from_file

   759         module = load_module_from_modpath(modpath_from_file(sitefile, self.extrapath))

   760         self.debug('%s loaded', sitefile)

   761         return module

   762 

   763     def cwproperty_definitions(self):

   764         cfg = self.persistent_options_configuration()

   765         for section, options in cfg.options_by_section():

   766             section = section.lower()

   767             for optname, optdict, value in options:

   768                 key = '%s.%s' % (section, optname)

   769                 type, vocab = self.map_option(optdict)

   770                 default = cfg.option_default(optname, optdict)

   771                 pdef = {'type': type, 'vocabulary': vocab, 'default': default,

   772                         'help': optdict['help'],

   773                         'sitewide': optdict.get('sitewide', False)}

   774                 yield key, pdef

   775 

   776     def map_option(self, optdict):

   777         try:

   778             vocab = optdict['choices']

   779         except KeyError:

   780             vocab = optdict.get('vocabulary')

   781             if isinstance(vocab, Method):

   782                 vocab = getattr(self, vocab.method, ())

   783         return CFGTYPE2ETYPE_MAP[optdict['type']], vocab

   784 

   785     def default_instance_id(self):

   786         """return the instance identifier, useful for option which need this

   787         as default value

   788         """

   789         return None

   790 

   791     _cubes = None

   792 

   793     def init_cubes(self, cubes):

   794         self._cubes = self.reorder_cubes(cubes)

   795         # load cubes'__init__.py file first

   796         for cube in cubes:

   797             __import__('cubes.%s' % cube)

   798         self.load_site_cubicweb()

   799 

   800     def cubes(self):

   801         """return the list of cubes used by this instance

   802 

   803         result is ordered from the top level cubes to inner dependencies

   804         cubes

   805         """

   806         assert self._cubes is not None, 'cubes not initialized'

   807         return self._cubes

   808 

   809     def cubes_path(self):

   810         """return the list of path to cubes used by this instance, from outer

   811         most to inner most cubes

   812         """

   813         return [self.cube_dir(p) for p in self.cubes()]

   814 

   815     # these are overridden by set_log_methods below

   816     # only defining here to prevent pylint from complaining

   817     @classmethod

   818     def debug(cls, msg, *a, **kw):

   819         pass

   820     info = warning = error = critical = exception = debug

   821 

   822 

   823 class CubicWebConfiguration(CubicWebNoAppConfiguration):

   824     """base class for cubicweb server and web configurations"""

   825 

   826     if CubicWebNoAppConfiguration.mode == 'user':

   827         _INSTANCES_DIR = expanduser('~/etc/cubicweb.d/')

   828     #mode == system'

   829     elif _USR_INSTALL:

   830         _INSTANCES_DIR = '/etc/cubicweb.d/'

   831     else:

   832         _INSTANCES_DIR = join(_INSTALL_PREFIX, 'etc', 'cubicweb.d')

   833 

   834     # set to true during repair (shell, migration) to allow some things which

   835     # wouldn't be possible otherwise

   836     repairing = False

   837 

   838     # set by upgrade command

   839     verbosity = 0

   840     cmdline_options = None

   841     options = CubicWebNoAppConfiguration.options + (

   842         ('log-file',

   843          {'type' : 'string',

   844           'default': Method('default_log_file'),

   845           'help': 'file where output logs should be written',

   846           'group': 'main', 'level': 2,

   847           }),

   848         ('statsd-endpoint',

   849          {'type' : 'string',

   850           'default': '',

   851           'help': 'UDP address of the statsd endpoint; it must be formatted'

   852                   'like <ip>:<port>; disabled is unset.',

   853           'group': 'main', 'level': 2,

   854           }),

   855         # email configuration

   856         ('smtp-host',

   857          {'type' : 'string',

   858           'default': 'mail',

   859           'help': 'hostname of the SMTP mail server',

   860           'group': 'email', 'level': 1,

   861           }),

   862         ('smtp-port',

   863          {'type' : 'int',

   864           'default': 25,

   865           'help': 'listening port of the SMTP mail server',

   866           'group': 'email', 'level': 1,

   867           }),

   868         ('sender-name',

   869          {'type' : 'string',

   870           'default': Method('default_instance_id'),

   871           'help': 'name used as HELO name for outgoing emails from the \

   872 repository.',

   873           'group': 'email', 'level': 2,

   874           }),

   875         ('sender-addr',

   876          {'type' : 'string',

   877           'default': 'cubicweb@mydomain.com',

   878           'help': 'email address used as HELO address for outgoing emails from \

   879 the repository',

   880           'group': 'email', 'level': 1,

   881           }),

   882         ('logstat-interval',

   883          {'type' : 'int',

   884           'default': 0,

   885           'help': 'interval (in seconds) at which stats are dumped in the logstat file; set 0 to disable',

   886           'group': 'main', 'level': 2,

   887           }),

   888         ('logstat-file',

   889          {'type' : 'string',

   890           'default': Method('default_stats_file'),

   891           'help': 'file where stats for the instance should be written',

   892           'group': 'main', 'level': 2,

   893           }),

   894         )

   895 

   896     @classmethod

   897     def instances_dir(cls):

   898         """return the control directory"""

   899         return abspath(os.environ.get('CW_INSTANCES_DIR', cls._INSTANCES_DIR))

   900 

   901     @classmethod

   902     def migration_scripts_dir(cls):

   903         """cubicweb migration scripts directory"""

   904         if CWDEV:

   905             return join(CW_SOFTWARE_ROOT, 'misc', 'migration')

   906         mdir = join(_INSTALL_PREFIX, 'share', 'cubicweb', 'migration')

   907         if not exists(mdir):

   908             raise ConfigurationError('migration path %s doesn\'t exist' % mdir)

   909         return mdir

   910 

   911     @classmethod

   912     def config_for(cls, appid, config=None, debugmode=False, creating=False):

   913         """return a configuration instance for the given instance identifier

   914         """

   915         cls.load_available_configs()

   916         config = config or guess_configuration(cls.instance_home(appid))

   917         configcls = configuration_cls(config)

   918         return configcls(appid, debugmode, creating)

   919 

   920     @classmethod

   921     def possible_configurations(cls, appid):

   922         """return the name of possible configurations for the given

   923         instance id

   924         """

   925         home = cls.instance_home(appid)

   926         return possible_configurations(home)

   927 

   928     @classmethod

   929     def instance_home(cls, appid):

   930         """return the home directory of the instance with the given

   931         instance id

   932         """

   933         home = join(cls.instances_dir(), appid)

   934         if not exists(home):

   935             raise ConfigurationError('no such instance %s (check it exists with'

   936                                      ' "cubicweb-ctl list")' % appid)

   937         return home

   938 

   939     MODES = ('common', 'repository', 'Any')

   940     MCOMPAT = {'all-in-one': MODES,

   941                'repository': ('common', 'repository', 'Any')}

   942     @classmethod

   943     def accept_mode(cls, mode):

   944         #assert mode in cls.MODES, mode

   945         return mode in cls.MCOMPAT[cls.name]

   946 

   947     # default configuration methods ###########################################

   948 

   949     def default_instance_id(self):

   950         """return the instance identifier, useful for option which need this

   951         as default value

   952         """

   953         return self.appid

   954 

   955     def default_log_file(self):

   956         """return default path to the log file of the instance'server"""

   957         if self.mode == 'user':

   958             import tempfile

   959             basepath = join(tempfile.gettempdir(), '%s-%s' % (

   960                 basename(self.appid), self.name))

   961             path = basepath + '.log'

   962             i = 1

   963             while exists(path) and i < 100: # arbitrary limit to avoid infinite loop

   964                 try:

   965                     open(path, 'a')

   966                     break

   967                 except IOError:

   968                     path = '%s-%s.log' % (basepath, i)

   969                     i += 1

   970             return path

   971         if _USR_INSTALL:

   972             return '/var/log/cubicweb/%s-%s.log' % (self.appid, self.name)

   973         else:

   974             log_path = os.path.join(_INSTALL_PREFIX, 'var', 'log', 'cubicweb', '%s-%s.log')

   975             return log_path % (self.appid, self.name)

   976 

   977     def default_stats_file(self):

   978         """return default path to the stats file of the instance'server"""

   979         logfile = self.default_log_file()

   980         if logfile.endswith('.log'):

   981             logfile = logfile[:-4]

   982         return logfile + '.stats'

   983 

   984     def default_pid_file(self):

   985         """return default path to the pid file of the instance'server"""

   986         if self.mode == 'system':

   987             if _USR_INSTALL:

   988                 default = '/var/run/cubicweb/'

   989             else:

   990                 default = os.path.join(_INSTALL_PREFIX, 'var', 'run', 'cubicweb')

   991         else:

   992             import tempfile

   993             default = tempfile.gettempdir()

   994         # runtime directory created on startup if necessary, don't check it

   995         # exists

   996         rtdir = abspath(os.environ.get('CW_RUNTIME_DIR', default))

   997         return join(rtdir, '%s-%s.pid' % (self.appid, self.name))

   998 

   999     # config -> repository

  1000 

  1001     def repository(self, vreg=None):

  1002         from cubicweb.server.repository import Repository

  1003         from cubicweb.server.utils import TasksManager

  1004         return Repository(self, TasksManager(), vreg=vreg)

  1005 

  1006     # instance methods used to get instance specific resources #############

  1007 

  1008     def __init__(self, appid, debugmode=False, creating=False):

  1009         self.appid = appid

  1010         # set to true while creating an instance

  1011         self.creating = creating

  1012         super(CubicWebConfiguration, self).__init__(debugmode)

  1013         fake_gettext = (text_type, lambda ctx, msgid: text_type(msgid))

  1014         for lang in self.available_languages():

  1015             self.translations[lang] = fake_gettext

  1016         self._cubes = None

  1017         self.load_file_configuration(self.main_config_file())

  1018 

  1019     def adjust_sys_path(self):

  1020         super(CubicWebConfiguration, self).adjust_sys_path()

  1021         # adding apphome to python path is not usually necessary in production

  1022         # environments, but necessary for tests

  1023         if self.apphome and self.apphome not in sys.path:

  1024             sys.path.insert(0, self.apphome)

  1025 

  1026     @property

  1027     def apphome(self):

  1028         return join(self.instances_dir(), self.appid)

  1029 

  1030     @property

  1031     def appdatahome(self):

  1032         if self.mode == 'system':

  1033             if _USR_INSTALL:

  1034                 iddir = os.path.join('/var','lib', 'cubicweb', 'instances')

  1035             else:

  1036                 iddir = os.path.join(_INSTALL_PREFIX, 'var', 'lib', 'cubicweb', 'instances')

  1037         else:

  1038             iddir = self.instances_dir()

  1039         iddir = abspath(os.environ.get('CW_INSTANCES_DATA_DIR', iddir))

  1040         return join(iddir, self.appid)

  1041 

  1042     def init_cubes(self, cubes):

  1043         super(CubicWebConfiguration, self).init_cubes(cubes)

  1044         # reload config file in cases options are defined in cubes __init__

  1045         # or site_cubicweb files

  1046         self.load_file_configuration(self.main_config_file())

  1047         # configuration initialization hook

  1048         self.load_configuration(**(self.cmdline_options or {}))

  1049 

  1050     def add_cubes(self, cubes):

  1051         """add given cubes to the list of used cubes"""

  1052         if not isinstance(cubes, list):

  1053             cubes = list(cubes)

  1054         self._cubes = self.reorder_cubes(list(self._cubes) + cubes)

  1055         self.load_site_cubicweb([self.cube_dir(cube) for cube in cubes])

  1056 

  1057     def main_config_file(self):

  1058         """return instance's control configuration file"""

  1059         return join(self.apphome, '%s.conf' % self.name)

  1060 

  1061     def save(self):

  1062         """write down current configuration"""

  1063         with open(self.main_config_file(), 'w') as fobj:

  1064             self.generate_config(fobj)

  1065 

  1066     def check_writeable_uid_directory(self, path):

  1067         """check given directory path exists, belongs to the user running the

  1068         server process and is writeable.

  1069 

  1070         If not, try to fix this, letting exception propagate when not possible.

  1071         """

  1072         if not exists(path):

  1073             self.info('creating %s directory', path)

  1074             try:

  1075                 os.makedirs(path)

  1076             except OSError as ex:

  1077                 self.warning('error while creating %s directory: %s', path, ex)

  1078                 return

  1079         if self['uid']:

  1080             try:

  1081                 uid = int(self['uid'])

  1082             except ValueError:

  1083                 from pwd import getpwnam

  1084                 uid = getpwnam(self['uid']).pw_uid

  1085         else:

  1086             try:

  1087                 uid = os.getuid()

  1088             except AttributeError: # we are on windows

  1089                 return

  1090         fstat = os.stat(path)

  1091         if fstat.st_uid != uid:

  1092             self.info('giving ownership of %s directory to %s', path, self['uid'])

  1093             try:

  1094                 os.chown(path, uid, os.getgid())

  1095             except OSError as ex:

  1096                 self.warning('error while giving ownership of %s directory to %s: %s',

  1097                              path, self['uid'], ex)

  1098         if not (fstat.st_mode & stat.S_IWUSR):

  1099             self.info('forcing write permission on directory %s', path)

  1100             try:

  1101                 os.chmod(path, fstat.st_mode | stat.S_IWUSR)

  1102             except OSError as ex:

  1103                 self.warning('error while forcing write permission on directory %s: %s',

  1104                              path, ex)

  1105                 return

  1106 

  1107     @cached

  1108     def instance_md5_version(self):

  1109         from hashlib import md5 # pylint: disable=E0611

  1110         infos = []

  1111         for pkg in sorted(self.cubes()):

  1112             version = self.cube_version(pkg)

  1113             infos.append('%s-%s' % (pkg, version))

  1114         infos.append('cubicweb-%s' % str(self.cubicweb_version()))

  1115         return md5((';'.join(infos)).encode('ascii')).hexdigest()

  1116 

  1117     def load_configuration(self, **kw):

  1118         """load instance's configuration files"""

  1119         super(CubicWebConfiguration, self).load_configuration(**kw)

  1120         if self.apphome and not self.creating:

  1121             # init gettext

  1122             self._gettext_init()

  1123 

  1124     def _load_site_cubicweb(self, sitefile):

  1125         # overridden to register cube specific options

  1126         mod = super(CubicWebConfiguration, self)._load_site_cubicweb(sitefile)

  1127         if getattr(mod, 'options', None):

  1128             self.register_options(mod.options)

  1129             self.load_defaults()

  1130 

  1131     def init_log(self, logthreshold=None, force=False):

  1132         """init the log service"""

  1133         if not force and hasattr(self, '_logging_initialized'):

  1134             return