i18n.py
author David Douard <david.douard@logilab.fr>
Fri, 19 Feb 2016 11:03:32 +0100
changeset 11147 ddd5dbf52b72
parent 10776 b1834143fec8
permissions -rw-r--r--
[datafeed] do NOT call commit in DataFeedXMLParser.process() (closes #10928592) and so, remove more useless code. Doing a commit in DataFeedXMLParser.process() is a bad idea because this method is called for every imported/updated entity, before handling deletions. So if we commit here, odds are quite high that we're violating some integrity constraint and end up with a ValidationError.
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
5421
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
     1
# copyright 2003-2010 LOGILAB S.A. (Paris, FRANCE), all rights reserved.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
     2
# contact http://www.logilab.fr/ -- mailto:contact@logilab.fr
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
     3
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
     4
# This file is part of CubicWeb.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
     5
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
     6
# CubicWeb is free software: you can redistribute it and/or modify it under the
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
     7
# terms of the GNU Lesser General Public License as published by the Free
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
     8
# Software Foundation, either version 2.1 of the License, or (at your option)
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
     9
# any later version.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
    10
#
5424
8ecbcbff9777 replace logilab-common by CubicWeb in disclaimer
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5421
diff changeset
    11
# CubicWeb is distributed in the hope that it will be useful, but WITHOUT
5421
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
    12
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
    13
# FOR A PARTICULAR PURPOSE.  See the GNU Lesser General Public License for more
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
    14
# details.
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
    15
#
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
    16
# You should have received a copy of the GNU Lesser General Public License along
8167de96c523 proper licensing information (LGPL-2.1). Hope I get it right this time.
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 4719
diff changeset
    17
# with CubicWeb.  If not, see <http://www.gnu.org/licenses/>.
6356
e89f71a41e20 [c-c i18n] namespace/line wrap cleanup + dynamically compute available language + make it works in installed mode
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 5424
diff changeset
    18
"""Some i18n/gettext utilities."""
10589
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 8695
diff changeset
    19
from __future__ import print_function
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    20
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    21
__docformat__ = "restructuredtext en"
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    22
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    23
import re
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    24
import os
1132
96752791c2b6 pylint cleanup
sylvain.thenault@logilab.fr
parents: 0
diff changeset
    25
from os.path import join, basename, splitext, exists
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    26
from glob import glob
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    27
10776
b1834143fec8 [py3k] fix i18n
Julien Cristau <julien.cristau@logilab.fr>
parents: 10589
diff changeset
    28
from six import PY2
b1834143fec8 [py3k] fix i18n
Julien Cristau <julien.cristau@logilab.fr>
parents: 10589
diff changeset
    29
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    30
from cubicweb.toolsutils import create_dir
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    31
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    32
def extract_from_tal(files, output_file):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    33
    """extract i18n strings from tal and write them into the given output file
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    34
    using standard python gettext marker (_)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    35
    """
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    36
    output = open(output_file, 'w')
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    37
    for filepath in files:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    38
        for match in re.finditer('i18n:(content|replace)="([^"]+)"', open(filepath).read()):
7896
4c954e1e73ef [lint] remove uses of "print >> sys.stderr" (closes #1908571)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 7155
diff changeset
    39
            output.write('_("%s")' % match.group(2))
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    40
    output.close()
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    41
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    42
3275
5247789df541 [gettext] provide GNU contexts to avoid translations ambiguities
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 3202
diff changeset
    43
def add_msg(w, msgid, msgctx=None):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    44
    """write an empty pot msgid definition"""
10776
b1834143fec8 [py3k] fix i18n
Julien Cristau <julien.cristau@logilab.fr>
parents: 10589
diff changeset
    45
    if PY2 and isinstance(msgid, unicode):
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    46
        msgid = msgid.encode('utf-8')
3275
5247789df541 [gettext] provide GNU contexts to avoid translations ambiguities
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 3202
diff changeset
    47
    if msgctx:
10776
b1834143fec8 [py3k] fix i18n
Julien Cristau <julien.cristau@logilab.fr>
parents: 10589
diff changeset
    48
        if PY2 and isinstance(msgctx, unicode):
3275
5247789df541 [gettext] provide GNU contexts to avoid translations ambiguities
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 3202
diff changeset
    49
            msgctx = msgctx.encode('utf-8')
5247789df541 [gettext] provide GNU contexts to avoid translations ambiguities
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents: 3202
diff changeset
    50
        w('msgctxt "%s"\n' % msgctx)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    51
    msgid = msgid.replace('"', r'\"').splitlines()
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    52
    if len(msgid) > 1:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    53
        w('msgid ""\n')
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    54
        for line in msgid:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    55
            w('"%s"' % line.replace('"', r'\"'))
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    56
    else:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    57
        w('msgid "%s"\n' % msgid[0])
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    58
    w('msgstr ""\n\n')
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    59
8620
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    60
def execute2(args):
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    61
    # XXX replace this with check_output in Python 2.7
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    62
    from subprocess import Popen, PIPE, CalledProcessError
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    63
    p = Popen(args, stdout=PIPE, stderr=PIPE)
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    64
    out, err = p.communicate()
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    65
    if p.returncode != 0:
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    66
        exc = CalledProcessError(p.returncode, args[0])
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    67
        exc.cmd = args
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    68
        exc.data = (out, err)
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    69
        raise exc
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    70
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    71
def available_catalogs(i18ndir=None):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    72
    if i18ndir is None:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    73
        wildcard = '*.po'
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    74
    else:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    75
        wildcard = join(i18ndir, '*.po')
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    76
    for popath in glob(wildcard):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    77
        lang = splitext(basename(popath))[0]
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    78
        yield lang, popath
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    79
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    80
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    81
def compile_i18n_catalogs(sourcedirs, destdir, langs):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    82
    """generate .mo files for a set of languages into the `destdir` i18n directory
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    83
    """
8620
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    84
    from subprocess import CalledProcessError
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    85
    from logilab.common.fileutils import ensure_fs_mode
10589
7c23b7de2b8d [py3k] print function
Samuel Trégouët <samuel.tregouet@logilab.fr>
parents: 8695
diff changeset
    86
    print('-> compiling message catalogs to %s' % destdir)
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    87
    errors = []
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    88
    for lang in langs:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    89
        langdir = join(destdir, lang, 'LC_MESSAGES')
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    90
        if not exists(langdir):
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    91
            create_dir(langdir)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    92
        pofiles = [join(path, '%s.po' % lang) for path in sourcedirs]
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    93
        pofiles = [pof for pof in pofiles if exists(pof)]
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    94
        mergedpo = join(destdir, '%s_merged.po' % lang)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
    95
        try:
2476
1294a6bdf3bf application -> instance where it makes sense
Sylvain Thénault <sylvain.thenault@logilab.fr>
parents: 2395
diff changeset
    96
            # merge instance/cubes messages catalogs with the stdlib's one
8620
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    97
            cmd = ['msgcat', '--use-first', '--sort-output', '--strict',
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    98
                   '-o', mergedpo] + pofiles
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
    99
            execute2(cmd)
3118
9e7a155bc4e5 more i18n commands fixes :
Aurélien Campéas
parents: 3117
diff changeset
   100
            # make sure the .mo file is writeable and compiles with *msgfmt*
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   101
            applmo = join(destdir, lang, 'LC_MESSAGES', 'cubicweb.mo')
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   102
            try:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   103
                ensure_fs_mode(applmo)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   104
            except OSError:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   105
                pass # suppose not exists
8620
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
   106
            execute2(['msgfmt', mergedpo, '-o', applmo])
8695
358d8bed9626 [toward-py3k] rewrite to "except AnException as exc:" (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8620
diff changeset
   107
        except CalledProcessError as exc:
8620
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
   108
            errors.append(u'while handling language %s:\ncmd:\n%s\nstdout:\n%s\nstderr:\n%s\n' %
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
   109
                          (lang, exc.cmd, repr(exc.data[0]), repr(exc.data[1])))
8695
358d8bed9626 [toward-py3k] rewrite to "except AnException as exc:" (part of #2711624)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8620
diff changeset
   110
        except Exception as exc:
8620
61c4bdd70dd8 [cw-ctl] silence msgcat and msgfmt (closes #2527594)
Nicolas Chauvat <nicolas.chauvat@logilab.fr>
parents: 8225
diff changeset
   111
            errors.append(u'while handling language %s: %s' % (lang, exc))
0
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   112
        try:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   113
            # clean everything
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   114
            os.unlink(mergedpo)
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   115
        except Exception:
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   116
            continue
b97547f5f1fa Showtime !
Adrien Di Mascio <Adrien.DiMascio@logilab.fr>
parents:
diff changeset
   117
    return errors