i18n.py
author Pierre-Yves David <pierre-yves.david@logilab.fr>
Wed, 27 Mar 2013 19:54:36 +0100
changeset 8809 9ee4d0c65ad2
parent 8695 358d8bed9626
child 10589 7c23b7de2b8d
permissions -rw-r--r--
[session] use a dedicated class to track cnxset We introduce a new CnxSetTracker to track `cnxset` used by Transaction and allows to wait for them. This new class does not use Thread ID not thread joining to work. This allows to use multiple transaction per thread and a transaction in multiple thread. The class itself is totally threadsafe by the Transaction is still not thread safe. The old _threads_in_transaction attribute is dropped in favor of a new logic based on this object. The registration of cnxset used is not done by the Transaction itself. tx.cnset is a property handling the Consistency of its value with the CnxSetTracker instance. Note: The CnxSetTracker instance only track transaction id, not transaction itself, So not reference cycle are created.
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
5421
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
     1
# copyright 2003-2010 LOGILAB S.A. (Paris, FRANCE), all rights reserved.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
     2
# contact http://www.logilab.fr/ -- mailto:contact@logilab.fr
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
     3
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
     4
# This file is part of CubicWeb.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
     5
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
     6
# CubicWeb is free software: you can redistribute it and/or modify it under the
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
     7
# terms of the GNU Lesser General Public License as published by the Free
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
     8
# Software Foundation, either version 2.1 of the License, or (at your option)
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
     9
# any later version.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
    10
#
5424
8ecbcbff9777 replace logilab-common by CubicWeb in disclaimer
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5421
diff changeset
    11
# CubicWeb is distributed in the hope that it will be useful, but WITHOUT
5421
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
    12
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
    13
# FOR A PARTICULAR PURPOSE.  See the GNU Lesser General Public License for more
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
    14
# details.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
    15
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
    16
# You should have received a copy of the GNU Lesser General Public License along
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
    17
# with CubicWeb.  If not, see <http://www.gnu.org/licenses/>.
6356
e89f71a41e20 [c-c i18n] namespace/line wrap cleanup + dynamically compute available language + make it works in installed mode
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5424
diff changeset
    18
"""Some i18n/gettext utilities."""
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    19
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    20
__docformat__ = "restructuredtext en"
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    21
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    22
import re
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    23
import os
1132
96752791c2b6 pylint cleanup
sylvain.thenault@logilab.fr
parents: 0
diff changeset
    24
from os.path import join, basename, splitext, exists
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    25
from glob import glob
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    26
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    27
from cubicweb.toolsutils import create_dir
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    28
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    29
def extract_from_tal(files, output_file):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    30
    """extract i18n strings from tal and write them into the given output file
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    31
    using standard python gettext marker (_)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    32
    """
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    33
    output = open(output_file, 'w')
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    34
    for filepath in files:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    35
        for match in re.finditer('i18n:(content|replace)="([^"]+)"', open(filepath).read()):
7896
4c954e1e73ef [lint] remove uses of "print >> sys.stderr" (closes #1908571)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7155
diff changeset
    36
            output.write('_("%s")' % match.group(2))
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    37
    output.close()
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    38
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    39
3275
5247789df541 [gettext] provide GNU contexts to avoid translations ambiguities
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 3202
diff changeset
    40
def add_msg(w, msgid, msgctx=None):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    41
    """write an empty pot msgid definition"""
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    42
    if isinstance(msgid, unicode):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    43
        msgid = msgid.encode('utf-8')
3275
5247789df541 [gettext] provide GNU contexts to avoid translations ambiguities
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 3202
diff changeset
    44
    if msgctx:
5247789df541 [gettext] provide GNU contexts to avoid translations ambiguities
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 3202
diff changeset
    45
        if isinstance(msgctx, unicode):
5247789df541 [gettext] provide GNU contexts to avoid translations ambiguities
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 3202
diff changeset
    46
            msgctx = msgctx.encode('utf-8')
5247789df541 [gettext] provide GNU contexts to avoid translations ambiguities
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 3202
diff changeset
    47
        w('msgctxt "%s"\n' % msgctx)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    48
    msgid = msgid.replace('"', r'\"').splitlines()
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    49
    if len(msgid) > 1:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    50
        w('msgid ""\n')
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    51
        for line in msgid:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    52
            w('"%s"' % line.replace('"', r'\"'))
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    53
    else:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    54
        w('msgid "%s"\n' % msgid[0])
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    55
    w('msgstr ""\n\n')
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    56
8620
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    57
def execute2(args):
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    58
    # XXX replace this with check_output in Python 2.7
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    59
    from subprocess import Popen, PIPE, CalledProcessError
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    60
    p = Popen(args, stdout=PIPE, stderr=PIPE)
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    61
    out, err = p.communicate()
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    62
    if p.returncode != 0:
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    63
        exc = CalledProcessError(p.returncode, args[0])
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    64
        exc.cmd = args
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    65
        exc.data = (out, err)
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    66
        raise exc
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    67
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    68
def available_catalogs(i18ndir=None):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    69
    if i18ndir is None:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    70
        wildcard = '*.po'
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    71
    else:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    72
        wildcard = join(i18ndir, '*.po')
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    73
    for popath in glob(wildcard):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    74
        lang = splitext(basename(popath))[0]
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    75
        yield lang, popath
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    76
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    77
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    78
def compile_i18n_catalogs(sourcedirs, destdir, langs):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    79
    """generate .mo files for a set of languages into the `destdir` i18n directory
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    80
    """
8620
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    81
    from subprocess import CalledProcessError
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    82
    from logilab.common.fileutils import ensure_fs_mode
7155
4bab50b02927 [cwctl] refactor ui messages aiming for consistency and simplicity
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 6356
diff changeset
    83
    print '-> compiling message catalogs to %s' % destdir
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    84
    errors = []
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    85
    for lang in langs:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    86
        langdir = join(destdir, lang, 'LC_MESSAGES')
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    87
        if not exists(langdir):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    88
            create_dir(langdir)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    89
        pofiles = [join(path, '%s.po' % lang) for path in sourcedirs]
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    90
        pofiles = [pof for pof in pofiles if exists(pof)]
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    91
        mergedpo = join(destdir, '%s_merged.po' % lang)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    92
        try:
2476
1294a6bdf3bf application -> instance where it makes sense
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2395
diff changeset
    93
            # merge instance/cubes messages catalogs with the stdlib's one
8620
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    94
            cmd = ['msgcat', '--use-first', '--sort-output', '--strict',
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    95
                   '-o', mergedpo] + pofiles
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    96
            execute2(cmd)
3118
9e7a155bc4e5 more i18n commands fixes :
Aurélien Campéas
parents: 3117
diff changeset
    97
            # make sure the .mo file is writeable and compiles with *msgfmt*
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    98
            applmo = join(destdir, lang, 'LC_MESSAGES', 'cubicweb.mo')
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    99
            try:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   100
                ensure_fs_mode(applmo)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   101
            except OSError:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   102
                pass # suppose not exists
8620
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
   103
            execute2(['msgfmt', mergedpo, '-o', applmo])
8695
358d8bed9626 [toward-py3k] rewrite to "except AnException as exc:" (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8620
diff changeset
   104
        except CalledProcessError as exc:
8620
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
   105
            errors.append(u'while handling language %s:\ncmd:\n%s\nstdout:\n%s\nstderr:\n%s\n' %
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
   106
                          (lang, exc.cmd, repr(exc.data[0]), repr(exc.data[1])))
8695
358d8bed9626 [toward-py3k] rewrite to "except AnException as exc:" (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8620
diff changeset
   107
        except Exception as exc:
8620
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
   108
            errors.append(u'while handling language %s: %s' % (lang, exc))
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   109
        try:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   110
            # clean everything
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   111
            os.unlink(mergedpo)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   112
        except Exception:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   113
            continue
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   114
    return errors