i18n.py
author Julien Cristau <julien.cristau@logilab.fr>
Mon, 09 Nov 2015 15:22:43 +0100
changeset 10869 575982c948a9
parent 10776 b1834143fec8
permissions -rw-r--r--
[dataimport] remove drop_index parameter from massive store "drop_index=False" also implied not dropping any constraints, in particular foreign keys, which meant any attempt to import entities would fail, because we only add metadata (the entities table) after the entity's insertion.
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
5421
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
     1
# copyright 2003-2010 LOGILAB S.A. (Paris, FRANCE), all rights reserved.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
     2
# contact http://www.logilab.fr/ -- mailto:contact@logilab.fr
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
     3
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
     4
# This file is part of CubicWeb.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
     5
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
     6
# CubicWeb is free software: you can redistribute it and/or modify it under the
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
     7
# terms of the GNU Lesser General Public License as published by the Free
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
     8
# Software Foundation, either version 2.1 of the License, or (at your option)
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
     9
# any later version.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
    10
#
5424
8ecbcbff9777 replace logilab-common by CubicWeb in disclaimer
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5421
diff changeset
    11
# CubicWeb is distributed in the hope that it will be useful, but WITHOUT
5421
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
    12
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
    13
# FOR A PARTICULAR PURPOSE.  See the GNU Lesser General Public License for more
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
    14
# details.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
    15
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
    16
# You should have received a copy of the GNU Lesser General Public License along
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
    17
# with CubicWeb.  If not, see <http://www.gnu.org/licenses/>.
6356
e89f71a41e20 [c-c i18n] namespace/line wrap cleanup + dynamically compute available language + make it works in installed mode
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5424
diff changeset
    18
"""Some i18n/gettext utilities."""
10589
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 8695
diff changeset
    19
from __future__ import print_function
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    20
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    21
__docformat__ = "restructuredtext en"
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    22
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    23
import re
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    24
import os
1132
96752791c2b6 pylint cleanup
sylvain.thenault@logilab.fr
parents: 0
diff changeset
    25
from os.path import join, basename, splitext, exists
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    26
from glob import glob
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    27
10776
b1834143fec8 [py3k] fix i18n
Julien Cristau <julien.cristau@logilab.fr>
parents: 10589
diff changeset
    28
from six import PY2
b1834143fec8 [py3k] fix i18n
Julien Cristau <julien.cristau@logilab.fr>
parents: 10589
diff changeset
    29
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    30
from cubicweb.toolsutils import create_dir
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    31
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    32
def extract_from_tal(files, output_file):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    33
    """extract i18n strings from tal and write them into the given output file
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    34
    using standard python gettext marker (_)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    35
    """
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    36
    output = open(output_file, 'w')
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    37
    for filepath in files:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    38
        for match in re.finditer('i18n:(content|replace)="([^"]+)"', open(filepath).read()):
7896
4c954e1e73ef [lint] remove uses of "print >> sys.stderr" (closes #1908571)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7155
diff changeset
    39
            output.write('_("%s")' % match.group(2))
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    40
    output.close()
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    41
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    42
3275
5247789df541 [gettext] provide GNU contexts to avoid translations ambiguities
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 3202
diff changeset
    43
def add_msg(w, msgid, msgctx=None):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    44
    """write an empty pot msgid definition"""
10776
b1834143fec8 [py3k] fix i18n
Julien Cristau <julien.cristau@logilab.fr>
parents: 10589
diff changeset
    45
    if PY2 and isinstance(msgid, unicode):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    46
        msgid = msgid.encode('utf-8')
3275
5247789df541 [gettext] provide GNU contexts to avoid translations ambiguities
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 3202
diff changeset
    47
    if msgctx:
10776
b1834143fec8 [py3k] fix i18n
Julien Cristau <julien.cristau@logilab.fr>
parents: 10589
diff changeset
    48
        if PY2 and isinstance(msgctx, unicode):
3275
5247789df541 [gettext] provide GNU contexts to avoid translations ambiguities
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 3202
diff changeset
    49
            msgctx = msgctx.encode('utf-8')
5247789df541 [gettext] provide GNU contexts to avoid translations ambiguities
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 3202
diff changeset
    50
        w('msgctxt "%s"\n' % msgctx)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    51
    msgid = msgid.replace('"', r'\"').splitlines()
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    52
    if len(msgid) > 1:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    53
        w('msgid ""\n')
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    54
        for line in msgid:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    55
            w('"%s"' % line.replace('"', r'\"'))
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    56
    else:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    57
        w('msgid "%s"\n' % msgid[0])
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    58
    w('msgstr ""\n\n')
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    59
8620
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    60
def execute2(args):
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    61
    # XXX replace this with check_output in Python 2.7
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    62
    from subprocess import Popen, PIPE, CalledProcessError
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    63
    p = Popen(args, stdout=PIPE, stderr=PIPE)
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    64
    out, err = p.communicate()
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    65
    if p.returncode != 0:
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    66
        exc = CalledProcessError(p.returncode, args[0])
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    67
        exc.cmd = args
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    68
        exc.data = (out, err)
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    69
        raise exc
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    70
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    71
def available_catalogs(i18ndir=None):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    72
    if i18ndir is None:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    73
        wildcard = '*.po'
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    74
    else:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    75
        wildcard = join(i18ndir, '*.po')
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    76
    for popath in glob(wildcard):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    77
        lang = splitext(basename(popath))[0]
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    78
        yield lang, popath
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    79
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    80
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    81
def compile_i18n_catalogs(sourcedirs, destdir, langs):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    82
    """generate .mo files for a set of languages into the `destdir` i18n directory
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    83
    """
8620
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    84
    from subprocess import CalledProcessError
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    85
    from logilab.common.fileutils import ensure_fs_mode
10589
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 8695
diff changeset
    86
    print('-> compiling message catalogs to %s' % destdir)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    87
    errors = []
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    88
    for lang in langs:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    89
        langdir = join(destdir, lang, 'LC_MESSAGES')
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    90
        if not exists(langdir):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    91
            create_dir(langdir)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    92
        pofiles = [join(path, '%s.po' % lang) for path in sourcedirs]
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    93
        pofiles = [pof for pof in pofiles if exists(pof)]
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    94
        mergedpo = join(destdir, '%s_merged.po' % lang)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    95
        try:
2476
1294a6bdf3bf application -> instance where it makes sense
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2395
diff changeset
    96
            # merge instance/cubes messages catalogs with the stdlib's one
8620
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    97
            cmd = ['msgcat', '--use-first', '--sort-output', '--strict',
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    98
                   '-o', mergedpo] + pofiles
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    99
            execute2(cmd)
3118
9e7a155bc4e5 more i18n commands fixes :
Aurélien Campéas
parents: 3117
diff changeset
   100
            # make sure the .mo file is writeable and compiles with *msgfmt*
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   101
            applmo = join(destdir, lang, 'LC_MESSAGES', 'cubicweb.mo')
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   102
            try:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   103
                ensure_fs_mode(applmo)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   104
            except OSError:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   105
                pass # suppose not exists
8620
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
   106
            execute2(['msgfmt', mergedpo, '-o', applmo])
8695
358d8bed9626 [toward-py3k] rewrite to "except AnException as exc:" (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8620
diff changeset
   107
        except CalledProcessError as exc:
8620
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
   108
            errors.append(u'while handling language %s:\ncmd:\n%s\nstdout:\n%s\nstderr:\n%s\n' %
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
   109
                          (lang, exc.cmd, repr(exc.data[0]), repr(exc.data[1])))
8695
358d8bed9626 [toward-py3k] rewrite to "except AnException as exc:" (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8620
diff changeset
   110
        except Exception as exc:
8620
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
   111
            errors.append(u'while handling language %s: %s' % (lang, exc))
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   112
        try:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   113
            # clean everything
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   114
            os.unlink(mergedpo)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   115
        except Exception:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   116
            continue
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   117
    return errors