Internationalization (i18n) and localization (l10n)

[svrnm]

The purpose of this discussion is to have a central place to talk about the long-term goal to provide i18n and l10n for the OpenTelemetry Website and Documentation. Previous issues on this topic:

• Add Chinese version of website #2402
• Add banner + form for i18n support #490
• i18n process definitions + standards #376
• Website add i18n support  #352

As of today there also exist to separate repositories housing localizations (https://github.com/open-telemetry/docs-cn and https://github.com/open-telemetry/docs-ja, although docs-ja has not been updated in ~2years).

Last guidance on how we can accomplish a multi-language version of our website was the following (#2402 (comment)):

• Our website supports multiple languages already, so the translated content should live under main/content/<two_letter_code> (as @windsonsea pointed out already)
• Our point of reference is how kubernetes is doing their localization, see https://github.com/kubernetes/website
• We need at least 2 ppl owning that content, so that changes can be approved (CODEOWNERSHIP will help with that)
• We can start with the translation been hidden until we reach a point where enough material is translated to go live (with maybe some blog post & announcements around it)
• (Please anticipate that this is guidance on not a fixed set of rules)

As stated above this project requires a set of individuals that are happy to take on ownership for their language and work with SIG Comms on implementing this. So any discussion on this page "how to do it" needs to be preceded by a "I volunteer to co-own <language>"

cc @open-telemetry/docs-maintainers
cc @open-telemetry/docs-cn-maintainers
cc @ymotongpoo @katzchang @shirou (for docs-ja)

[ymotongpoo]

Thanks @svrnm for the heads up. The reasons why Japanese documentation doc has been stalled were:

1. When we actively started the translation, the spec had been actively updated and it was hard to catch up with the chages
2. We heard (I forgot from whom) that there was a plan to have i18n support at this open-telemetry/opentelemetry.io repository soon (though the real was it took a while)
3. The amount of work was quite large for three people
4. It was a burden for us to host the translated docs like what opentelemetry.io does

I am personally super happy for this main repo to support i18n and we can contribute to the single destination.

[svrnm]

Thanks @ymotongpoo, also for clarifying why the Japanese documentation stalled. As said above, I have a preference towards "doing it like kubernetes", so having all the content at one place and different owners across the languages.

If we find another person (or even 3) who is willing to step up for the Japanese documentation, we can have a discussion on the "how".

[katzchang]

I will raise my hand, of course. There is a possibility that some of my colleagues could help.

But I have other work to do, so I can't start right away. We'll have to discuss the timetable. And I would also like to know best practice on how to keep up with the heavily updated base document.

[svrnm]

Thanks @katzchang, I totally understand that this looks like (and is!) a massive undertaking with our docs being big and continuously being updated. So, as a part of the "how" we need to find a good common ground for translations (e.g. we tag some pages that are more stable and so you have to expect less fluctuation)

[ymotongpoo]

I am also happy to join for Japanese as well. When it comes to the translation of the continuously updating contents, definitely the original needs to have the way for versioning. We can talk about it including all other "how"s.

[windsonsea]

+1 support for docs-cn. Happy to see the progress 👍

[svrnm]

Thanks @windsonsea. Same is true for what I wrote above to @ymotongpoo: if you and at least another person (3 ppl in total would be perfect) step up for the Chinese documentation, we can have a discussion on the "how"

[svrnm]

@ymotongpoo, @windsonsea, @katzchang an important document for us all to review are the hugo and documentation in i18n&l10n:

• https://gohugo.io/content-management/multilingual/
• https://www.docsy.dev/docs/language/

Additionally, here is the Localization Guide from Kubernetes:

https://kubernetes.io/docs/contribute/localization/

[svrnm]

following up on this, a few points on what needs to be done:

• The current setup needs to be tweaked to accept multiple languages, I tried it yesterday but stumbled upon a few things, where I will need @chalin's support to get this working (out of office right now, so this will take a while to look into)
• We need a set of minimal pages, if anyone can go through our website&documentation and come up with a suggestion?
• With both things in place we can work towards prototypes in Chinese and Japanese and see what else needs to be done (I assume there are a few more things in our current setup that we need to rethink to make the page ready for i18n)

Thanks

[ymotongpoo]

@svrnm thanks for the update.

We need a set of minimal pages, if anyone can go through our website&documentation and come up with a suggestion?

Does the set mean the subset of the current web site? If so, I personally think, the following can be the first milestone for i18n:

• Top page
• Getting Started (both of dev and ops)
• all pages under Concepts
• all pages under Collector

[svrnm]

@svrnm thanks for the update.

We need a set of minimal pages, if anyone can go through our website&documentation and come up with a suggestion?

Does the set mean the subset of the current web site? If so, I personally think, the following can be the first milestone for i18n:

Yes, exactly. The list you suggested is a good starting point, here are a few additional ones:

• https://opentelemetry.io/
• https://opentelemetry.io/docs/
• https://opentelemetry.io/status/
• https://opentelemetry.io/docs/languages/ (only the top page!)
• probably a page under "contributing" which is language specific

for the collector pages we might skip a few ones that are very specific and detailed (especially those around building your own components)

[svrnm]

@katzchang @ymotongpoo @windsonsea we talked about localization during the SIG Comms meeting today. We want to provide a very low barrier for you to get started, so based on a proposal by @chalin here's how we want to kick this off:

• Provide us with a translation of the Landing Page (https://opentelemetry.io/ (https://github.com/open-telemetry/opentelemetry.io/blob/main/content/en/docs/_index.md) in a PR were you directly edit the English page (please do only translate that page for now!)
• After you raised that PR we will take that file as a foundation for your language (Chinese, Japanese) and modify the PR in a way the language is available
• We will add a banner to the page that this translation is experimental, and then make it available immediately
• Afterwards you are able to continuously translate the remaining pages, focusing on the critical ones we have listed above
• After a few months we will review the status and see how we continue

I hope this helps you to get started quickly. If you have any questions, let us know here or in the #otel-comms slack space. If it helps we can also schedule an additional meeting that accomodates your timezones.

Thanks,
Severin

cc @chalin @cartermp

[svrnm]

thanks for following up @ymotongpoo, @katzchang, I am happy to hear that you are still planning to tackle this. We have no time pressure here, so whenever you are ready. Note that for starting it all we need is a translation of the main index page (https://github.com/open-telemetry/opentelemetry.io/blob/main/content/en/_index.md), so by copying it into a a ja folder and translating it and pushing a PR we are set for a start. @chalin can then take that PR then and make it work similar to the zh one.

[ymotongpoo]

@svrnm I just opened up the first PR for Japanese translation of the index page. (#4580)

[svrnm]

thank you @ymotongpoo

@katzchang or @shirou would you be able to provide a review of the translation?

[svrnm]

https://opentelemetry.io/ja 🎉

Please also see open-telemetry/community#2138, we will create github teams to manage the translation. For that @katzchang and @shirou need to become member of the GitHub Org first, following this process: https://github.com/open-telemetry/community/blob/main/community-membership.md#member

[ymotongpoo]

woohoo!!! Thank you for the smooth ramp, @svrnm and @chalin.

@katzchang and @shirou, I'll create a couple of follow up PRs for more pages soon, and I appreciate your reviews on them. For that, it would be great if both of you could join OpenTelemetry org earlier.

[svrnm]

https://opentelemetry.io/zh/ just landed as a first start!

[chalin]

@open-telemetry/docs-approvers @open-telemetry/docs-zh-approvers: do we want to implement a mechanism that will help zh maintainers to measure the drift between zh pages and the corresponding en pages. If the answer is yes, the zh maintainers and page contributors will have to maintain the info that is used to measure the drift. IMHO, this might be nontrivial to setup and maintain, but it may well be worth it in the long run.

For example, given that a zh page gets created from an en page at a specific commit hash, do we want a mechanism to record the en page "permalink" or permalink equivalent (which will include the commit hash) from which the zh page was created? Or maybe the entire website commit hash?

Alternatives to the use of permatlinks could be one of the following front-matter fields:

• lastmod - the date (possibly including the time) of when the zh page was last updated. This could be used to help correlate whether the corresponding en page has been updated since. Note that this is a field recognized by Hugo
• original_date - the date/timestamp of the original used as a basis for the zh page; or maybe this should be named en_page_date?
• en_page_commit, which would hold the value of the commit hash portion of the "permalink" suggestion made earlier.

I've posted to the Hugo forum to see if anyone else has tackled this issue: https://discourse.gohugo.io/t/measuring-multilingual-page-translation-drift/50090.

[chalin]

Being able to detect page drift between en and non-en pages is a challenging problem. @cjyabraham has made me aware of solutions used by other CNCF projects. I'll be reviewing those a reporting back here.

In #4536 (comment), @my-git9 proposes to use the K8s zh localization approach of embedding en page segments in zh pages as HTML comments. I'd strongly vote against this approach because:

1. The duplication and maintenance overhead of keeping those en language HTML comments up-to-date.
2. Embedding en page segments as HTML only compounds the maintenance problem without helping us detect if there is a drift between en and zh pages.

I'll be reviewing other CNCF-project approaches to this and report back here as soon as I can.

[my-git9]

The k8s documentation community uses a script as a detection tool. The premise of using it requires the translator to standardize the commit. Perhaps we can refer to: https://github.com/kubernetes/website/blob/main/scripts/lsync.sh

[svrnm]

Thanks @my-git9 !

[svrnm]

Following up here as well for visibility: while we take a very close look at K8s and other projects and how they approach translation, we think that having the english version inlined is right now not what we want to do. The translation support is young and experimental and we want to keep things flexible for now.

[darielli]

I volunteer to co-own docs-zh! I'm a native Chinese speaker and I have been learning and using opentelemetry recently. I found that the Chinese documentation on the official website is incomplete and very glad to participate in its improvement . I'm just not sure if I need any other experience to qualify for this work.

[windsonsea]

You are welcome to provide review comments and suggestions on any Chinese PRs and issues, like current PRs with lang:zh.

[svrnm]

Thanks for offering your help @freezer712 ! All you need as a qualification is knowing something about OpenTelemetry and Chinese. As @windsonsea wrote, you can go ahead and review PRs or create your own translations. Based on your contributions we can move you towards being a member, an approver and eventually a maintainer for the translation.

[darielli]

Thank you all, I'm getting ready to do it.

[svrnm]

FYI, I created https://cloud-native.slack.com/archives/C076RUAGP37, please make sure you join that channel, we can use it for quick discussions and for helping to resolve certain issues.

[svrnm]

Note that all translation PRs will be on hold / throttled back (not only new ones), as @chalin works on resolving core issues with the i18n-support features.

We are super excited about the l10n/i18n project and everyone involved in it. This is only an intermediate situation which hopefully is not dampening anyone's enthusiasm about this!

[chalin]

Yes, thank you for your patience all. We'll get there. It was expected that it would be slower in the beginning as we work out implementation issues. It'll be smoother sailing "soon"!

[chalin]

(Edited) The following section was added to the Localization documentation: Translation guidance. Please read through this short section before submitting new translation PRs. Also, rebase existing localization PRs and ensure that you follow the guidance provided in the Localization page.

[ymotongpoo]

@svrnm @chalin Thank you for supporting each language localization! At least, the localization for Japanese could make a smooth start thanks to the great automation tools.

Now I started feeling like I need some automation tool to detect inconsistent translation and Japanese formatting to have more contributions from many people. How can I add that feature to the repo? Tricky parts are:

• Only lang:ja labeled PRs need the formatter and linter for translation
• The tool needs to have a set of mappings between correct and incorrect wordings (in some file)

What do you think about this idea?

[svrnm]

Moved this to a dedicated issue

[chalin]

Can you cross-link the issue here @svrnm?

[Andygol]

@svrnm
Could you consider streamlining the translation process and translation maintenance (https://opentelemetry.io/docs/contributing/localization/#track-changes) using CAT tools like CrowdIn or so?