|
|
[[_TOC_]]
|
|
|
|
|
|
# Overview
|
|
|
|
|
|
In a somewhat reverse order, the life of a translation string in CiviCRM:
|
|
|
|
|
|
* When CiviCRM does a string lookup with `ts` to fetch the translation, it must find a matching string in the gettext `mo` dictionary for the target language.
|
|
|
* The `mo` files are shipped in the localisation files (ex: `civicrm-5.x.x-l10n.tar.gz`) on the [download](https://civicrm.org/download) page.
|
|
|
* Those translation tar.gz files are generated during the release process of the various CiviCRM archives (CMS-specific + translation).
|
|
|
* The release scripts fetch the `mo` files from gcloud, where they were uploaded by a [Jenkins job](https://test.civicrm.org/view/i18n/) that runs daily. From `mo` to `po` is basically the process of indexing the plain text `po` file in a binary format efficient for fast lookups.
|
|
|
* To build the `mo` files, the Jenkins Job fetches the `po` (source / plain text) from the [gitlab translation repository](https://lab.civicrm.org/dev/translation). This repository is updated daily, as part of the same daily jenkins job. This repo is a safeguard in case Transifex disappears. The repo is also [mirrored on github](https://github.com/civicrm/l10n/).
|
|
|
* The daily fetch from Transifex basically does a `tx pull -a -t` (pull all translations) and commits them to github.
|
|
|
|
|
|
Now, for the string to exist on Transifex, we need to run the extraction from the CiviCRM source code. This process is done manually, and will be described more in detail below. Short overview:
|
|
|
|
|
|
* A member of the core team has to run the extraction scripts. They may run on multiple CiviCRM versions, depending which are supported (ex: [ESR](https://civicrm.org/esr)).
|
|
|
* Validate that the scripts ran correctly
|
|
|
* Push the new strings to transifex
|
|
|
|
|
|
Once the new strings are added to Transifex, translators can go and translate. After 24h, the translation files are updated on gcloud (from the first process above), and admin can manually update their translation files if they want to, or wait for the next CiviCRM release.
|
|
|
|
|
|
# Stack prerequisites
|
|
|
|
|
|
* Install Git, e.g. sudo apt-get install git
|
|
|
* Install [Transifex Client](https://docs.transifex.com/client/introduction/), e.g. `sudo easy_install transifex-client` or `sudo apt-get install transifex-client`
|
|
|
* Install "moreutils" for the sponge utility, e.g. `sudo apt-get install moreutils`
|
|
|
* Install the PHP command line interface (CLI) and MySQL bindings, e.g. `sudo apt-get install php-cli php-mysql`
|
|
|
* Install [civistrings](https://github.com/civicrm/civistrings) and composer, and have them available in your $PATH.
|
|
|
|
|
|
# Get a copy of the git repositories
|
|
|
|
|
|
Clone [the Git repositories](https://github.com/civicrm/) from github:
|
|
|
|
|
|
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
|
|
|
<script type="syntaxhighlighter" class="theme: Confluence; brush: java; gutter: false"><![CDATA[
|
|
|
mkdir -p ~/repositories/
|
|
|
cd ~/repositories/
|
|
|
git clone https://github.com/civicrm/l10n.git
|
|
|
git clone https://github.com/civicrm/civicrm-core.git civicrm
|
|
|
git clone https://github.com/civicrm/civicrm-drupal.git civicrm/drupal
|
|
|
git clone https://github.com/civicrm/civicrm-joomla.git civicrm/joomla
|
|
|
git clone https://github.com/civicrm/civicrm-wordpress.git civicrm/wordpress
|
|
|
git clone https://github.com/civicrm/civicrm-packages.git civicrm/packages]]></script>
|
|
|
</div></div>
|
|
|
|
|
|
<p>NB: the ~/repositories/civicrm directory is only an example. You can use your preferred location.</p><p>The l10n repository contains:</p><ul><li>"bin" : scripts for automating some of the tasks</li><li>"po/<lang>": translation files for each language</li><li>"po/pot/": .pot files are the source translation files (in English, as extracted from the CiviCRM code)</li></ul><div class='panelMacro'><table class='noteMacro'><colgroup><col width='24'><col></colgroup><tr><td valign='top'><img src="/confluence/images/icons/emoticons/warning.png" width="16" height="16" align="absmiddle" alt="" border="0"></td><td><p>The "po" files use a different file hierarchy than the default Transifex one.</p><p>With Transifex, files are structured in the form of "civicrm.<component>/<lang>.po". In the l10n git repository, we use the structure "<lang>/<component>.po". It makes it easier to extract and compile only part of the translation. The magic for making this happen is in the file: l10n/.tx/config.</p></td></tr></table></div><h1 id="PushingnewstringstoTransifex-RefreshingPOTfiles">Refreshing POT files</h1><p>The "POT" files are the source translation files that will be sent to Transifex for translation. They contain the source US-English strings.</p><p>We need to regenerate the POT files when a new release branch is created (ex: v4.2) when the new release goes beta (to allow translators a bit of time to translate before the official release). If we do not refresh the files, translators will not be able to find the string in Transifex.</p><h2 id="PushingnewstringstoTransifex-GeneratingnewPOTfiles">Generating new POT files</h2><p>NOTE: the extraction creates a lot of temporary files in /tmp. If you do not have enough space in that directory, you can set the TMPDIR environment variable to use another directory (before running the build-unified-pots.sh script).</p><p>For example:</p><div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
|
|
|
<script type="syntaxhighlighter" class="theme: Confluence; brush: java; gutter: false"><![CDATA[mkdir ~/tmp
|
|
|
export TMPDIR=~/tmp]]></script>
|
|
|
</div></div><p>To run the extraction process, run:</p><div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
|
|
|
<script type="syntaxhighlighter" class="theme: Confluence; brush: java; gutter: false"><![CDATA[./bin/build-unified-pots.sh ~/repositories/civicrm po/pot '4.2 4.3 4.4 4.5' 2>&1 | tee pots.log]]></script>
|
|
|
</div></div><p>You can also run the script on the "master" branch, but this is not recommended unless we are in beta:</p><div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
|
|
|
<script type="syntaxhighlighter" class="theme: Confluence; brush: java; gutter: false"><![CDATA[./bin/build-unified-pots.sh ~/repositories/civicrm po/pot '4.2 4.3 4.4 master' 2>&1 | tee pots.log
|
|
|
]]></script>
|
|
|
</div></div><p>Note that the releases to be covered are all provided as one, quoted, space-separated string. This will (a) make fresh copies of the relevant releases in your temp directory, (b) bootstrap them with minimal config, (c) run GenCode to create all of the DAO files and (d) create the given release’s POT files, afterwards (e) combining the per-release POT files into release-agnostic POT files and (f) putting them in <code>po/pot</code>.</p><div class='panelMacro'><table class='noteMacro'><colgroup><col width='24'><col></colgroup><tr><td valign='top'><img src="/confluence/images/icons/emoticons/warning.png" width="16" height="16" align="absmiddle" alt="" border="0"></td><td><p>Gettext will emit a warning if it detects an incorrect use of the ts() function. For example, ts("Created $foo") is a common mistake. It should be written as ts("Created !1", array(1 => $foo)). For more information, see the <a href="/confluence/display/CRMDOC/Internationalisation+for+Developers">Internationalisation for Developers</a>. If you find a mistake, correct it in the CiviCRM code, commit, then re-run the script.</p></td></tr></table></div><h2 id="PushingnewstringstoTransifex-Reviewnewtranslationstrings">Review new translation strings</h2><p>It's worth taking the time to review the new strings which have been added to CiviCRM to fix any spelling/grammar/typos mistakes in the original English strings.</p><div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
|
|
|
<script type="syntaxhighlighter" class="theme: Confluence; brush: java; gutter: false"><![CDATA[git status
|
|
|
git diff --patience po/pot/*.pot | ./bin/diff-check.php]]></script>
|
|
|
</div></div><p>In any files were added (and are still "untracked" by git), you can add them to the git repository:</p><div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
|
|
|
<script type="syntaxhighlighter" class="theme: Confluence; brush: java; gutter: false"><![CDATA[git add po/pot/something.pot
|
|
|
git commit -m "new file"]]></script>
|
|
|
</div></div><h2 id="PushingnewstringstoTransifex-PublishingPOTfilestoTransifex">Publishing POT files to Transifex</h2><p>Once the new files are generated, add them to the repository (<code>git add -p</code> if you want to take a look at what changed, <code>git add .</code> otherwise; <code>git commit</code> to commit and <code>git push</code> to push to the remote repository).</p><p>Transifex is nice enough to track the repository’s POT files, so the refreshed POT files do not need to be uploaded to Transifex explicitly. If needed, you can run: </p><div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
|
|
|
<script type="syntaxhighlighter" class="theme: Confluence; brush: java; gutter: false"><![CDATA[tx push -s]]></script>
|
|
|
</div></div><p>See the <code>.tx/config</code> file for more information.</p><p>If a new component was added to CiviCRM (i.e. you have a new something.pot that was created), it needs to be added to Transifex. For example:</p><div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
|
|
|
<script type="syntaxhighlighter" class="theme: Confluence; brush: java; gutter: false"><![CDATA[tx set --auto-local -r civicrm.pcp 'po/<lang>.po' --source-lang en --source-file po/pot/pcp.pot --execute
|
|
|
tx push -s -r civicrm.pcp
|
|
|
tx pull -a -r civicrm.pcp]]></script>
|
|
|
</div></div><p>For more information: <a href="http://help.transifex.net/features/client/index.html#auto-local" class="external-link" rel="nofollow">http://help.transifex.net/features/client/index.html#auto-local</a></p><p>You will also have to add an entry in the .tx/config file (you can copy/adapt an entry for another file).</p><h1 id="PushingnewstringstoTransifex-PullingnewtranslationsfromTransifex">Pulling new translations from Transifex</h1><p>Before a release (or whenever we want to recompile MO files) new translations need to be pulled from Transifex. This is done with the Transifex Client; the configuration is already in the repository (in the <code>.tx</code> directory), so a simple</p><div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
|
|
|
<script type="syntaxhighlighter" class="theme: Confluence; brush: java; gutter: false"><![CDATA[tx pull -a
|
|
|
]]></script>
|
|
|
</div></div><p>should suffice. After pulling the translations they should be committed to the repository (with <code>git add po; git commit -m 'pull from Transifex'; git push</code>).</p><h1 id="PushingnewstringstoTransifex-Pushfilestothegithubrepository">Push files to the github repository</h1><p><strong>Since 2013-01-18, this is done daily from sushi.c.o (on the "l10n" user account/crontab).</strong></p><p>Once you have finished adding/reviewing the .pot files, pulled the .po translations for each language, you can do a last review and push your changes to github:</p><div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
|
|
|
<script type="syntaxhighlighter" class="theme: Confluence; brush: java; gutter: false"><![CDATA[git status
|
|
|
git diff
|
|
|
./bin/commit-to-git.sh
|
|
|
git push]]></script>
|
|
|
</div></div><p>The "commit-to-git.sh" script will use the "Last-Translator" as the "author" of the git commit. Although it's not very precise, it gives some idea of who is translating what.</p><h1 id="PushingnewstringstoTransifex-CompilingMOfiles">Compiling MO files</h1><p><strong>Since 2014-04-29, this is done daily (around 9h42 PST) from sushi.c.o (on the "l10n" user account/crontab).</strong> Only languages with more than 20% are added to the official civicrm-l10n.tar.gz file. You need to request to be added <a href="http://forum.civicrm.org/index.php/board,10.0.html" class="external-link" rel="nofollow">(on the forums</a>) once you reach this milestone. The .mo files are available from download.civicrm.org, although not browsable, but you can guess the URL with the fomr:</p><div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
|
|
|
<script type="syntaxhighlighter" class="theme: Confluence; brush: java; gutter: false"><![CDATA[https://download.civicrm.org/civicrm-l10n-core/mo/XX/civicrm.mo
|
|
|
|
|
|
... where XX is the language code, such as "es", "fr", etc, and "fr_CA", "es_MX", etc, for regional translations.]]></script>
|
|
|
</div></div><div class='panelMacro'><table class='noteMacro'><colgroup><col width='24'><col></colgroup><tr><td valign='top'><img src="/confluence/images/icons/emoticons/warning.png" width="16" height="16" align="absmiddle" alt="" border="0"></td><td><p>This is a tricky, kludgy and ugly process; it should be fixed by creating a script that uses the xx → xx_YY language mappings found in the <code>xml/templates/languages.tpl</code> file (or the relevant parts of the <code>civicrm_option_value</code> table).</p></td></tr></table></div><p>To recompile MO files run the <code>bin/compile-mo-files.sh</code> script. This will create <code>civicrm.mo</code> files for every language:</p><div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
|
|
|
<script type="syntaxhighlighter" class="theme: Confluence; brush: java; gutter: false"><![CDATA[chastell@devielle:~/CiviCRM/git/l10n$ bin/compile-mo-files.sh
|
|
|
compiling af
|
|
|
compiling ar
|
|
|
…
|
|
|
compiling de
|
|
|
compiling de_CH
|
|
|
compiling el
|
|
|
…
|
|
|
compiling vi
|
|
|
compiling zh_CN
|
|
|
compiling zh_TW
|
|
|
chastell@devielle:~/CiviCRM/git/l10n$ git status
|
|
|
# On branch master
|
|
|
# Untracked files:
|
|
|
# (use "git add <file>..." to include in what will be committed)
|
|
|
#
|
|
|
# po/af/civicrm.mo
|
|
|
# po/ar/civicrm.mo
|
|
|
…
|
|
|
# po/de/civicrm.mo
|
|
|
# po/de_CH/civicrm.mo
|
|
|
…
|
|
|
# po/vi/civicrm.mo
|
|
|
# po/zh_CN/civicrm.mo
|
|
|
# po/zh_TW/civicrm.mo
|
|
|
nothing added to commit but untracked files present (use "git add" to track)
|
|
|
chastell@devielle:~/CiviCRM/git/l10n$
|
|
|
]]></script>
|
|
|
</div></div><p>The trick is to move the MO files to their relevant places in the Subversion repository (we keep the MO files in the Subversion repository, as it’s linked via <code>svn:externals</code> link from our main repository). Unfortunately, Transifex uses the shortened language names where possible (e.g., <code>pl</code> and <code>hi</code>), using the full <code>xx_YY</code> notation only where necessary (so <code>pt</code> is Portuguese and <code>pt_BR</code> is Brasilian Portuguese), while CiviCRM always uses the full <code>xx_YY} notation (i.e., {{pl_PL</code>, <code>hi_IN</code> and <code>pt_PT</code> for Polish, Hindi and Portuguese, respectively).</p><p>This means that the compiled MO files need to be carefully moved to their relevant subdirectories in the Subversion repository. A crude solution (assuming the Subversion checkout of <a href="http://svn.civicrm.org/l10n" class="external-link" rel="nofollow">http://svn.civicrm.org/l10n</a> is at <code>../../svn/l10n</code>):</p><div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
|
|
|
<script type="syntaxhighlighter" class="theme: Confluence; brush: java; gutter: false"><![CDATA[# move all xx_YY translations to same-named directories:
|
|
|
for i in po/??_??/civicrm.mo; do mv $i ../../svn/l10n/`echo $i | cut -d/ -f2`; done
|
|
|
|
|
|
# move these of xx translations which have xx_YY counterparts:
|
|
|
mv po/de/civicrm.mo ../../svn/l10n/de_DE
|
|
|
mv po/es/civicrm.mo ../../svn/l10n/es_ES
|
|
|
mv po/fr/civicrm.mo ../../svn/l10n/fr_FR
|
|
|
mv po/pt/civicrm.mo ../../svn/l10n/pt_PT
|
|
|
|
|
|
# move all the remaining xx translations to xx_*
|
|
|
# (we can’t do this for the above four, as pt would overwrite pt_BR rather than pt_PT):
|
|
|
for i in po/??/civicrm.mo; do mv $i ../../svn/l10n/`echo $i | cut -d/ -f2`*; done
|
|
|
]]></script>
|
|
|
</div></div><p>After the above the MO files should be committed to the Subversion repository.</p><h1 id="PushingnewstringstoTransifex-Syncingtranslations">Syncing translations</h1><p>When a string is not translated (or translated with a ‘fuzzy’ – i.e., tentative – flag set) it doesn’t get compiled to the MO file and the original English string is used instead. For some language pairs it makes sense to use the other language’s translation in such case – with the idea being that for, e.g., Canadian French is better to fall back to French rather than English.</p><p>This situation is handed by the <code>sync-translated.sh</code> script, which – given two locales – copies over the strings translated in one but not translated in the other (both ways). Note that there are language pairs (like de and de_CH, i.e. German and Swiss German) which explicitly asked to not do the syncing for them. We currently sync en_CA with en_GB, es with es_MX and fr with fr_CA:</p><div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
|
|
|
<script type="syntaxhighlighter" class="theme: Confluence; brush: java; gutter: false"><![CDATA[bin/sync-translated.sh en_CA en_GB
|
|
|
bin/sync-translated.sh es es_MX
|
|
|
bin/sync-translated.sh fr fr_CA
|
|
|
git add po
|
|
|
git commit -m 'sync en_CA with en_GB, es with es_MX and fr with fr_CA'
|
|
|
git push
|
|
|
]]></script>
|
|
|
</div></div><p>Such translations need also to be pushed to Transifex:</p><div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
|
|
|
<script type="syntaxhighlighter" class="theme: Confluence; brush: java; gutter: false"><![CDATA[for lang in en_CA en_GB es es_MX fr fr_CA; do tx push -t -l $lang; done
|
|
|
]]></script>
|
|
|
</div></div><h1 id="PushingnewstringstoTransifex-CaringfortheTransifexcommunity">Caring for the Transifex community</h1><p>Currently our policy is that the first person willing to translate CiviCRM into a given language creates a Transifex team and becomes the team coordinator; future translators need to be approved by the coordinator.</p><p>This makes it possible to guard from spammers and keep a given localisation coherent (e.g. with regards to how different terms, like ‘househould’, should be translated), but it also means that new translators who are not quickly accepted (or denied) by the team coordinator are in limbo and wait for the decision.</p><p>Thus we should periodically check whether new translators for existing teams are handled pronto and are not discouraged/blocked when the team coordinator is inactive (or on a leave, etc.); we should poke the coordinators, elect additional ones or, in the worst cases, accept the new translators ourselves.</p><p>http://help.transifex.net/features/client/index.html#auto-local</p>
|
|
|
|
|
|
</div>
|
|
|
</td>
|
|
|
</tr>
|
|
|
</table>
|
|
|
|
|
|
|
|
|
# Historical notes
|
|
|
|
|
|
* March 2010: [Migration from Pootle to Transifex](https://wiki.civicrm.org/confluence/display/CRM/Translation+Evaluation+Matrix)
|
|
|
* March 2011: One single translation file for multiple versions of CiviCRM (the 3 latest stable versions), instead of having version-specific translations.
|
|
|
* 2013? Extension translation
|
|
|
|