.. -*- coding: utf-8 -*-

.. _internationalization:

Internationalization

---------------------

Cubicweb fully supports the internalization of its content and interface.

Cubicweb's interface internationalization is based on the translation project `GNU gettext`_.

.. _`GNU gettext`: http://www.gnu.org/software/gettext/

Cubicweb' internalization involves two steps:

* in your Python code and cubicweb-tal templates : mark translatable strings

* in your instance : handle the translation catalog, edit translations

String internationalization

~~~~~~~~~~~~~~~~~~~~~~~~~~~

User defined string

```````````````````

In the Python code and cubicweb-tal templates translatable strings can be

marked in one of the following ways :

 * by using the *built-in* function `_`:

   .. sourcecode:: python

     class PrimaryView(EntityView):

         """the full view of an non final entity"""

         __regid__ = 'primary'

         title = _('primary')

  OR

 * by using the equivalent request's method:

   .. sourcecode:: python

     class NoResultView(View):

         """default view when no result has been found"""

         __regid__ = 'noresult'

         def call(self, **kwargs):

             self.w(u'<div class="searchMessage"><strong>%s</strong></div>\n'

                 % self._cw._('No result matching query'))

The goal of the *built-in* function `_` is only **to mark the

translatable strings**, it will only return the string to translate

itself, but not its translation (it's actually another name for the

`unicode` builtin).

In the other hand the request's method `self._cw._` is also meant to

retrieve the proper translation of translation strings in the

requested language.

Finally you can also use the `__` attribute of request object to get a

translation for a string *which should not itself added to the catalog*,

usually in case where the actual msgid is created by string interpolation ::

  self._cw.__('This %s' % etype)

In this example ._cw.__` is used instead of ._cw._` so we don't have 'This %s' in

messages catalogs.

Translations in cubicweb-tal template can also be done with TAL tags

`i18n:content` and `i18n:replace`.

If you need to add messages on top of those that can be found in the source,

you can create a file named `i18n/static-messages.pot`.

You could put there messages not found in the python sources or

overrides for some messages of used cubes.

Generated string

````````````````

We do not need to mark the translation strings of entities/relations used by a

particular instance's schema as they are generated automatically. String for

various actions are also generated.

For exemple the following schema:

.. sourcecode:: python

  class EntityA(EntityType):

      relation_a2b = SubjectRelation('EntityB')

  class EntityB(EntityType):

      pass

May generate the following message ::

  add EntityA relation_a2b EntityB subject

This message will be used in views of ``EntityA`` for creation of a new

``EntityB`` with a preset relation ``relation_a2b`` between the current

``EntityA`` and the new ``EntityB``. The opposite message ::

  add EntityA relation_a2b EntityB object

Is used for similar creation of an ``EntityA`` from a view of ``EntityB``. The

title of they respective creation form will be ::

  creating EntityB (EntityA %(linkto)s relation_a2b EntityB)

  creating EntityA (EntityA relation_a2b %(linkto)s EntityA)

In the translated string you can use ``%(linkto)s`` for reference to the source

``entity``.

Handling the translation catalog

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Once the internationalization is done in your code, you need to populate and

update the translation catalog. Cubicweb provides the following commands for this

purpose:

* `i18ncubicweb` updates Cubicweb framework's translation

  catalogs. Unless you actually work on the framework itself, you

  don't need to use this command.

* `i18ncube` updates the translation catalogs of *one particular cube*

  (or of all cubes). After this command is executed you must update

  the translation files *.po* in the "i18n" directory of your

  cube. This command will of course not remove existing translations

  still in use. It will mark unused translation but not remove them.

* `i18ninstance` recompiles the translation catalogs of *one particular

  instance* (or of all instances) after the translation catalogs of

  its cubes have been updated. This command is automatically

  called every time you create or update your instance. The compiled

  catalogs (*.mo*) are stored in the i18n/<lang>/LC_MESSAGES of

  instance where `lang` is the language identifier ('en' or 'fr'

  for exemple).

Example

```````

You have added and/or modified some translation strings in your cube

(after creating a new view or modifying the cube's schema for exemple).

To update the translation catalogs you need to do:

1. `cubicweb-ctl i18ncube <cube>`

2. Edit the <cube>/i18n/xxx.po  files and add missing translations (empty `msgstr`)

3. `hg ci -m "updated i18n catalogs"`

4. `cubicweb-ctl i18ninstance <myinstance>`

Customizing the messages extraction process

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The messages extraction performed by the ``i18ncommand`` collects messages

from a few different sources:

- the schema and application definition (entity names, docstrings,

  help messages, uicfg),

- the source files:

  - ``i18n:content`` or ``i18n:replace`` directives from TAL files (with ``.pt`` extension),

  - strings prefixed by an underscore (``_``) in python files,

  - strings with double quotes prefixed by an underscore in javascript files.

The source files are collected by walking through the cube directory,

but ignoring a few directories like ``.hg``, ``.tox``, ``test`` or

``node_modules``.

If you need to customize this behaviour in your cube, you have to

extend the ``cubicweb.devtools.devctl.I18nCubeMessageExtractor``. The

example below will collect strings from ``jinja2`` files and ignore

the ``static`` directory during the messages collection phase::

  # mymodule.py

  from cubicweb.devtools import devctl

  class MyMessageExtractor(devctl.I18nCubeMessageExtractor):

      blacklist = devctl.I18nCubeMessageExtractor | {'static'}

      formats = devctl.I18nCubeMessageExtractor.formats + ['jinja2']

      def collect_jinja2(self):

          return self.find('.jinja2')

      def extract_jinja2(self, files):

          return self._xgettext(files, output='jinja.pot',

                                extraopts='-L python --from-code=utf-8')

Then, you'll have to register it with a ``cubicweb.i18ncube`` entry point

in your cube's setup.py::

  setup(

      # ...

      entry_points={

          # ...

          'cubicweb.i18ncube': [

              'mycube=cubicweb_mycube.mymodule:MyMessageExtractor',

          ],

      },

      # ...

  )

Editing po files

~~~~~~~~~~~~~~~~

Using a PO aware editor

````````````````````````

Many tools exist to help maintain .po (PO) files. Common editors or

development environment provides modes for these. One can also find

dedicated PO files editor, such as `poedit`_.

.. _`poedit`:  http://www.poedit.net/

While usage of such a tool is commendable, PO files are perfectly

editable with a (unicode aware) plain text editor. It is also useful

to know their structure for troubleshooting purposes.

Structure of a PO file

``````````````````````

In this section, we selectively quote passages of the `GNU gettext`_

manual chapter on PO files, available there::

 http://www.gnu.org/software/hello/manual/gettext/PO-Files.html

One PO file entry has the following schematic structure::

     white-space

     #  translator-comments

     #. extracted-comments

     #: reference...

     #, flag...

     #| msgid previous-untranslated-string

     msgid untranslated-string

     msgstr translated-string

A simple entry can look like this::

     #: lib/error.c:116

     msgid "Unknown system error"

     msgstr "Error desconegut del sistema"

It is also possible to have entries with a context specifier. They

look like this::

     white-space

     #  translator-comments

     #. extracted-comments

     #: reference...

     #, flag...

     #| msgctxt previous-context

     #| msgid previous-untranslated-string

     msgctxt context

     msgid untranslated-string

     msgstr translated-string

The context serves to disambiguate messages with the same

untranslated-string. It is possible to have several entries with the

same untranslated-string in a PO file, provided that they each have a

different context. Note that an empty context string and an absent

msgctxt line do not mean the same thing.

Contexts and CubicWeb

`````````````````````

CubicWeb PO files have both non-contextual and contextual msgids.

Contextual entries are automatically used in some cases. For instance,

entity.dc_type(), eschema.display_name(req) or display_name(etype,

req, form, context) methods/function calls will use them.

It is also possible to explicitly use the with _cw.pgettext(context,

msgid).

Specialize translation for an application cube

``````````````````````````````````````````````

Every cube has its own translation files. For a specific application cube

it can be useful to specialize translations of other cubes. You can either mark

those strings for translation using `_` in the python code, or add a

`static-messages.pot` file into the `i18n` directory. This file

looks like: ::

    msgid ""

    msgstr ""

    "PO-Revision-Date: YEAR-MO-DA HO:MI +ZONE\n"

    "MIME-Version: 1.0\n"

    "Content-Type: text/plain; charset=UTF-8\n"

    "Content-Transfer-Encoding: 8bit\n"

    "Generated-By: pygettext.py 1.5\n"

    "Plural-Forms: nplurals=2; plural=(n > 1);\n"

    msgig "expression to be translated"

    msgstr ""

Doing this, ``expression to be translated`` will be taken into account by

the ``i18ncube`` command and additional messages will then appear in `.po` files

of the cube.