i18n.py
author Julien Cristau <julien.cristau@logilab.fr>
Tue, 08 Sep 2015 16:26:56 +0200
changeset 10653 42c5bd7286b7
parent 10589 7c23b7de2b8d
child 10776 b1834143fec8
permissions -rw-r--r--
[schema] improve normalization of RQLExpressions Parse and print back the expression instead of manipulating the string. Among other benefits, it means we don't mangle embedded string constants that contain commas or multiple spaces. Closes #6694426
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
5421
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
     1
# copyright 2003-2010 LOGILAB S.A. (Paris, FRANCE), all rights reserved.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
     2
# contact http://www.logilab.fr/ -- mailto:contact@logilab.fr
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
     3
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
     4
# This file is part of CubicWeb.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
     5
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
     6
# CubicWeb is free software: you can redistribute it and/or modify it under the
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
     7
# terms of the GNU Lesser General Public License as published by the Free
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
     8
# Software Foundation, either version 2.1 of the License, or (at your option)
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
     9
# any later version.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
    10
#
5424
8ecbcbff9777 replace logilab-common by CubicWeb in disclaimer
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5421
diff changeset
    11
# CubicWeb is distributed in the hope that it will be useful, but WITHOUT
5421
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
    12
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
    13
# FOR A PARTICULAR PURPOSE.  See the GNU Lesser General Public License for more
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
    14
# details.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
    15
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
    16
# You should have received a copy of the GNU Lesser General Public License along
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
    17
# with CubicWeb.  If not, see <http://www.gnu.org/licenses/>.
6356
e89f71a41e20 [c-c i18n] namespace/line wrap cleanup + dynamically compute available language + make it works in installed mode
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5424
diff changeset
    18
"""Some i18n/gettext utilities."""
10589
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 8695
diff changeset
    19
from __future__ import print_function
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    20
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    21
__docformat__ = "restructuredtext en"
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    22
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    23
import re
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    24
import os
1132
96752791c2b6 pylint cleanup
sylvain.thenault@logilab.fr
parents: 0
diff changeset
    25
from os.path import join, basename, splitext, exists
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    26
from glob import glob
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    27
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    28
from cubicweb.toolsutils import create_dir
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    29
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    30
def extract_from_tal(files, output_file):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    31
    """extract i18n strings from tal and write them into the given output file
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    32
    using standard python gettext marker (_)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    33
    """
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    34
    output = open(output_file, 'w')
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    35
    for filepath in files:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    36
        for match in re.finditer('i18n:(content|replace)="([^"]+)"', open(filepath).read()):
7896
4c954e1e73ef [lint] remove uses of "print >> sys.stderr" (closes #1908571)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7155
diff changeset
    37
            output.write('_("%s")' % match.group(2))
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    38
    output.close()
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    39
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    40
3275
5247789df541 [gettext] provide GNU contexts to avoid translations ambiguities
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 3202
diff changeset
    41
def add_msg(w, msgid, msgctx=None):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    42
    """write an empty pot msgid definition"""
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    43
    if isinstance(msgid, unicode):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    44
        msgid = msgid.encode('utf-8')
3275
5247789df541 [gettext] provide GNU contexts to avoid translations ambiguities
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 3202
diff changeset
    45
    if msgctx:
5247789df541 [gettext] provide GNU contexts to avoid translations ambiguities
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 3202
diff changeset
    46
        if isinstance(msgctx, unicode):
5247789df541 [gettext] provide GNU contexts to avoid translations ambiguities
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 3202
diff changeset
    47
            msgctx = msgctx.encode('utf-8')
5247789df541 [gettext] provide GNU contexts to avoid translations ambiguities
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 3202
diff changeset
    48
        w('msgctxt "%s"\n' % msgctx)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    49
    msgid = msgid.replace('"', r'\"').splitlines()
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    50
    if len(msgid) > 1:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    51
        w('msgid ""\n')
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    52
        for line in msgid:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    53
            w('"%s"' % line.replace('"', r'\"'))
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    54
    else:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    55
        w('msgid "%s"\n' % msgid[0])
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    56
    w('msgstr ""\n\n')
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    57
8620
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    58
def execute2(args):
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    59
    # XXX replace this with check_output in Python 2.7
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    60
    from subprocess import Popen, PIPE, CalledProcessError
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    61
    p = Popen(args, stdout=PIPE, stderr=PIPE)
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    62
    out, err = p.communicate()
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    63
    if p.returncode != 0:
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    64
        exc = CalledProcessError(p.returncode, args[0])
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    65
        exc.cmd = args
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    66
        exc.data = (out, err)
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    67
        raise exc
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    68
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    69
def available_catalogs(i18ndir=None):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    70
    if i18ndir is None:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    71
        wildcard = '*.po'
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    72
    else:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    73
        wildcard = join(i18ndir, '*.po')
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    74
    for popath in glob(wildcard):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    75
        lang = splitext(basename(popath))[0]
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    76
        yield lang, popath
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    77
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    78
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    79
def compile_i18n_catalogs(sourcedirs, destdir, langs):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    80
    """generate .mo files for a set of languages into the `destdir` i18n directory
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    81
    """
8620
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    82
    from subprocess import CalledProcessError
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    83
    from logilab.common.fileutils import ensure_fs_mode
10589
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 8695
diff changeset
    84
    print('-> compiling message catalogs to %s' % destdir)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    85
    errors = []
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    86
    for lang in langs:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    87
        langdir = join(destdir, lang, 'LC_MESSAGES')
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    88
        if not exists(langdir):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    89
            create_dir(langdir)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    90
        pofiles = [join(path, '%s.po' % lang) for path in sourcedirs]
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    91
        pofiles = [pof for pof in pofiles if exists(pof)]
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    92
        mergedpo = join(destdir, '%s_merged.po' % lang)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    93
        try:
2476
1294a6bdf3bf application -> instance where it makes sense
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2395
diff changeset
    94
            # merge instance/cubes messages catalogs with the stdlib's one
8620
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    95
            cmd = ['msgcat', '--use-first', '--sort-output', '--strict',
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    96
                   '-o', mergedpo] + pofiles
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    97
            execute2(cmd)
3118
9e7a155bc4e5 more i18n commands fixes :
Aurélien Campéas
parents: 3117
diff changeset
    98
            # make sure the .mo file is writeable and compiles with *msgfmt*
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    99
            applmo = join(destdir, lang, 'LC_MESSAGES', 'cubicweb.mo')
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   100
            try:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   101
                ensure_fs_mode(applmo)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   102
            except OSError:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   103
                pass # suppose not exists
8620
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
   104
            execute2(['msgfmt', mergedpo, '-o', applmo])
8695
358d8bed9626 [toward-py3k] rewrite to "except AnException as exc:" (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8620
diff changeset
   105
        except CalledProcessError as exc:
8620
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
   106
            errors.append(u'while handling language %s:\ncmd:\n%s\nstdout:\n%s\nstderr:\n%s\n' %
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
   107
                          (lang, exc.cmd, repr(exc.data[0]), repr(exc.data[1])))
8695
358d8bed9626 [toward-py3k] rewrite to "except AnException as exc:" (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8620
diff changeset
   108
        except Exception as exc:
8620
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
   109
            errors.append(u'while handling language %s: %s' % (lang, exc))
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   110
        try:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   111
            # clean everything
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   112
            os.unlink(mergedpo)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   113
        except Exception:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   114
            continue
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   115
    return errors