Rewrite Internationalization section. #641

msporny · 2019-05-25T19:53:02Z

Attempt to address all remaining i18n considerations after talking w/ i18n folks, JSON-LD folks, and various people with strong opinions. This PR suggests a mechanism that works for JSON, JSON-LD, RDF, and HTML syntaxes/approaches while addressing all known i18n concerns raised to date.

Related to issue #436.

Preview | Diff

aphillips

@msporny Here are my comments on the PR you mentioned earlier.

@r12a You should have a look also.

index.html

aphillips

b2f3fdb looks much improved. Much depends on the outcome of our discussions, of course.

msporny · 2019-05-26T20:16:14Z

@aphillips wrote:

b2f3fdb looks much improved. Much depends on the outcome of our discussions, of course.

You might also find the more human readable version easier to read through:

https://pr-preview.s3.amazonaws.com/w3c/vc-data-model/pull/641.html#internationalization-considerations

index.html

msporny · 2019-06-09T19:31:06Z

Alright folks, after about 4 weeks of work and 5 complete rewrites, I think we're getting close to fixing all the i18n issues in the Verifiable Credentials Data Model specification! :P

A number of us from the VCWG, JSON-LD WG, I18N WG, IETF langtag community, and random friends on the Internet got together and worked through a variety of the concerns here and elsewhere. The result was a new PR against the string-meta spec:

w3c/string-meta#35

A complete rewrite of this PR (again), human-readable version here:

https://pr-preview.s3.amazonaws.com/w3c/vc-data-model/pull/641.html#internationalization-considerations

Please review and see if you agree with the latest, noting that the thing that is really time critical is the Verifiable Credentials spec (this PR). We have time to fine tune the string-meta spec (and of course, fix JSON-LD, RDF, etc.).

/cc @aphillips @r12a @dlongley @burnburn @chaals @iherman @gkellogg and @pchampin.

chaals

I think some of the explanations and examples should be changed. Examples should generally show a meaningful case, and tagging plain Arabic text as rtl isn't necessary.

chaals · 2019-05-27T12:57:14Z

index.html

      </p>

+      <pre class="example nohighlight" title="Arabic text with a base direction of right-to-left">


pure arabic text has a strong directionality already, and doesn't really need this. The sort of example that does is where the text that starts the string runs in the opposite direction to the string itself:

מאַזל-טאָוו W3C, how do we handle this now?

is a left-to-right text in english (for people whose english includes a sprinkling of yiddish written as it often is in the hebrew alphabet). But without being tagged ltr, it can end up with the yiddish text and the question mark swapping places.

בינלאומי!
bidi בינלאומי

are broken - the exclamation should be on the left, and the word "bidi" should be on the right, as they would if the entire string were tagged as RTL

My understanding (which has changed many times on this issue, so I wouldn't be surprised if it needs more updating)... is that the primary reason for adding "direction" is to handle cases where the language tag is incorrect or missing but there is direction meta data that is correct. There is apparently a common pattern where the wrong language tag gets assigned to some data but the direction is proper and should be given preference. Because of this situation, "direction" is being added here to capture this otherwise correct and dropped meta data.

For example, the text is Arabic, it is marked with an incorrect tag of "en" but the "direction" is properly "rtl".

I realize it would be odd to add an example with this erroneous information, but isn't that the point of the feature? We don't need it otherwise.

Chaals is correct to point out that the direction information is not essential for a single arabic word, such as used in the example currently.

If the word was "Revoked!" it would be necessary for the consumer to apply RTL base direction, so that the exclamation mark appears on the correct side of the arabic word. However, if the consumer is using first-strong heuristics to detect direction (which we recommend) then it can determine the direction from that, since the first letter in the arabic word is RTL.

If, however, the first letter in an arabic translation is not RTL, then the base direction needs to be indicated, since otherwise the heuristics will make the wrong decision. So an example with some initial text in latin script would be a stronger example.

In string-meta we use:
HTML و CSS: تصميم و إنشاء مواقع الويب
which means
HTML and CSS: Designing and Creating Websites

Btw, there's another issue which is common when dealing with examples of bidi text written in english: How do you show the arabic text in the example? The way a user would see if correctly displayed is with "CSS و HTML" to the right of the remaining arabic text. To achieve that in the example, you'd need to use markup under the hood. However, then it may be less clear to the casual ready of the spec that the string begins with H. It's not a problem with an easy solution. Especially in Arabic, where overriding the text to show order of characters in memory produces strange joining behaviour (which is why we use Hebrew for that).

I slightly disagree with @r12a here. The base direction is needed for other reasons besides just the bidi algorithm. For example, layout, text alignment, or bullet placement in a list might depend on the base direction. Sometimes we need base direction information even when it doesn't affect the text composition.

It's of course helpful to choose examples in which setting the direction has some observable effect, and this always means mixed direction text. The best examples start (as ours does) with a strongly "anti-directional" character and ends (as ours does not) with punctuation (? is ideal); even better if one can get a number and some enclosing punctuation (parentheses).

@aphillips, if you're referring to the first para in my comment about a single arabic word, i agree that you'd need an overall base direction to be applied by the consumer if it wasn't inserted into some other text, but the FS heuristics would get you that base direction.

Btw, a caution about ?, since Arabic text would use ؟ U+061F ARABIC QUESTION MARK, which has the bidi category of AL - Right-to-left Arabic. In short, it will end up in the right place when dropped into a LTR context if it follows arabic letters. It could be used with Hebrew examples, however, since Hebrew uses the european question mark.

chaals · 2019-06-10T11:16:56Z

index.html

-following key-value pair:
-
-      <pre>
-"nameHtml": "&lt;span dir="rtl" lang="ar">HTML و CSS: تصميم و إنشاء مواقع الويب&lt;/span>"


This text is a more meaningful example, because if you don't have the base direction it comes out wrong.

Reverted to this example in 37daadb.

chaals · 2019-06-10T11:17:58Z

index.html

-
-      <pre>
-"nameHtml": "&lt;span dir="rtl" lang="ar">HTML و CSS: تصميم و إنشاء مواقع الويب&lt;/span>"
+The next example expresses the English text "revoked" in the Arabic


instead of

expresses the English text "revoked" in the Arabic

how about something like

uses a term in Arabic

Fixed in 905fec0.

chaals · 2019-06-10T11:20:08Z

index.html

-<code>nameHtml</code> property is of type <code>rdf:HTML</code>, then software
-agents have sufficient information to deterministically identify the text
-direction of the language.
+Utilization of the design pattern above assumes that the JSON-LD


This paragraph is confusing especially if you're not automatically thinking about JSON-LD. I'll have a think about how to make it clearer...

Attempted to do this in 9956dfa.

chaals · 2019-06-10T11:29:24Z

I also agree that discouraging HTML content is generally the right thing to do... (which makes me think we should be asking the HTML and Security communities to provide some guidance around how to sanitise HTML input and why...)

dlongley · 2019-06-10T14:21:08Z

index.html

+      <pre class="example nohighlight" title="Expressing natural language text as English">
+"statusReason": {
+  "value": "<span class="highlight">Revoked</span>",
+  "lang: "<code>en</code>"


Suggested change

"lang: "<code>en</code>"

"lang": "<code>en</code>"

Fixed in 77e1d9f.

dlongley · 2019-06-10T14:31:49Z

index.html

+      "id": "did:example:c276e12ec21ebfeb1f712ebc6f1",
+      "name": [{
+        "value": "Example University",
+        "language": "en"


These examples use the alias "language" instead of "lang". Which should it be?

Replaced all "language" with "lang" in ce371e3.

TallTed · 2019-06-10T15:10:08Z

index.html

-specification and significantly diminish its value as a standard.
+There are a number of internationalization considerations implementers
+are advised to be aware of when publishing data described in this specification.
+As with any web standards or protocols implementation, ignoring


As with any web standards or protocols implementation, ignoring
-> As with implementation of any web standard or protocol, ignoring

Fixed in f0b4f50.

TallTed · 2019-06-10T15:16:12Z

index.html

+
+    <p>
+Implementers are strongly advised to read the Strings on the Web: Language and
+Direction Metadata document [[STRING-META]] published by the W3C


the Strings on the Web: Language and Direction Metadata document

->

the <em>Strings on the Web: Language and Direction Metadata</em> document

I just published an update of the Strings on the Web doc in /TR at https://www.w3.org/TR/string-meta/. You may want to point to that location, instead of the ED. I'll try to update regularly using echidna when we add new text to the ED.

@TallTed done in 34e6d2b.

@r12a done in 34e6d2b.

TallTed · 2019-06-10T15:17:42Z

index.html

    </p>

    <p>
 This section outlines general internationalization considerations to take into
-account when utilizing this data model.
+account when utilizing this data model and is intended to highlight specific
+parts of the Strings on the Web: Language and Direction Metadata document


the Strings on the Web: Language and Direction Metadata document
->
the <em>Strings on the Web: Language and Direction Metadata</em> document

Fixed in f0b4f50.

TallTed · 2019-06-10T15:24:28Z

index.html

+      </p>
+
+      <p>
+Implementers are strongly discouraged from encoding information as HTML


Implementers are
-> Despite that possibility, implementers are

Fixed in 966b348.

r12a · 2019-06-10T18:09:51Z

index.html

-internationalization.
+Implementers considering the use of HTML to encode complex language and/or
+base direction information might consider deconstructing the data into a
+format that does not require complex markup, such as an array of elements


This advice worries me, since it could lead people to think that they should break a sentence up into separate strings everywhere the direction changes. That would cause problems for translation of the string (see Working with Composite Messages), not to mention the fact that an overall base direction (for the whole string) is still necessary, in order to pull together the individual parts in the correct order.

I'm not sure that much can be done for language, without markup.

But for base direction, the answer for plain text strings is to use the Unicode formatting characters (see How to use Unicode controls for bidi text).

+1 to this.

Most "multilingual" text actually has a base language. It's true that it is possible to write a sentence that demonstrates how language identification is needed within a single string (particularly when it comes to font selection or language-specific styling). But this is a corner case and is the sort of thing that calls for markup in any event.

Done. Removed the bad advice in ea43b25.

msporny

Ok, all change requests implemented, merging.

msporny · 2019-06-30T18:25:52Z

index.html

+      <pre class="example nohighlight" title="Expressing natural language text as English">
+"statusReason": {
+  "value": "<span class="highlight">Revoked</span>",
+  "lang: "<code>en</code>"


Fixed in 77e1d9f.

msporny · 2019-06-30T18:27:29Z

index.html

-
-      <pre>
-"nameHtml": "&lt;span dir="rtl" lang="ar">HTML و CSS: تصميم و إنشاء مواقع الويب&lt;/span>"
+The next example expresses the English text "revoked" in the Arabic


Fixed in 905fec0.

msporny · 2019-06-30T18:42:45Z

index.html

-following key-value pair:
-
-      <pre>
-"nameHtml": "&lt;span dir="rtl" lang="ar">HTML و CSS: تصميم و إنشاء مواقع الويب&lt;/span>"


Reverted to this example in 37daadb.

msporny · 2019-06-30T18:46:15Z

index.html

-<code>nameHtml</code> property is of type <code>rdf:HTML</code>, then software
-agents have sufficient information to deterministically identify the text
-direction of the language.
+Utilization of the design pattern above assumes that the JSON-LD


Attempted to do this in 9956dfa.

msporny · 2019-06-30T18:48:12Z

index.html

+      </p>
+
+      <p>
+Implementers are strongly discouraged from encoding information as HTML


Fixed in 966b348.

msporny · 2019-06-30T18:53:24Z

index.html

-internationalization.
+Implementers considering the use of HTML to encode complex language and/or
+base direction information might consider deconstructing the data into a
+format that does not require complex markup, such as an array of elements


Done. Removed the bad advice in ea43b25.

msporny · 2019-06-30T18:55:16Z

I've implemented all changes requested by reviewers. This is also in line w/ what seems to be emerging between the JSON-LD WG, i18n community, IETF folks, and the VCWG.

Merging.

msporny · 2019-06-30T18:55:47Z

Merged. Thanks to all for the help over the past month to get this PR to a point that was acceptable to all communities involved! :)

Rewrite Internationalization section.

125abf7

msporny requested review from dlongley, chaals, r12a and burnburn May 25, 2019 19:55

This was referenced May 25, 2019

Many natural language text fields (such as "statusReason") have no language or direction support #436

Closed

In defense of extending language tags w3c/rdf-dir-literal#3

Open

aphillips suggested changes May 25, 2019

View reviewed changes

msporny added 5 commits May 26, 2019 11:46

Add example of scoped language context modification.

1178e24

Apply changes requested by @aphillips.

b2f3fdb

Remove use of global and internationalized.

6b54e51

Fix i18n examples throughout specification.

ba0ca29

More i18n example fixes.

32633fe

aphillips reviewed May 26, 2019

View reviewed changes

TallTed reviewed Jun 3, 2019

View reviewed changes

index.html Outdated Show resolved Hide resolved

TallTed reviewed Jun 3, 2019

View reviewed changes

index.html Outdated Show resolved Hide resolved

TallTed reviewed Jun 3, 2019

View reviewed changes

index.html Outdated Show resolved Hide resolved

r12a reviewed Jun 4, 2019

View reviewed changes

index.html Outdated Show resolved Hide resolved

msporny added 4 commits June 9, 2019 14:47

Update section to defer to string meta for cross-syntax expression.

01133d3

Rewrite section on Complex Language Markup.

8684213

Minor grammar fix to "English text".

e544fe8

Fix and/ors related to base direction.

87d5b6a

msporny requested review from r12a, brentzundel and TallTed June 9, 2019 19:31

chaals reviewed Jun 10, 2019

View reviewed changes

dlongley reviewed Jun 10, 2019

View reviewed changes

TallTed reviewed Jun 10, 2019

View reviewed changes

r12a reviewed Jun 10, 2019

View reviewed changes

msporny added 9 commits June 30, 2019 14:15

Change "language" to "lang" in examples.

ce371e3

Fix minor grammar issues with a11y section.

f0b4f50

Update reference to STRING-META.

34e6d2b

Fix "lang" syntax error.

77e1d9f

Fix grammar issue spotted by @chaals.

905fec0

Fix example text in i18n section.

37daadb

Do not assume implementers are using JSON-LD.

9956dfa

Clarify use of HTML markup per request by @TallTed.

966b348

Remove bad i18n advice per @r12a and @aphillips.

ea43b25

msporny commented Jun 30, 2019

View reviewed changes

msporny merged commit 92ba5d5 into gh-pages Jun 30, 2019

himorin mentioned this pull request Sep 12, 2019

many fields such as statusReason have no lang/dir support w3c/i18n-activity#580

Closed

msporny deleted the msporny-issue-436 branch February 20, 2021 15:39

		</p>

		<pre class="example nohighlight" title="Arabic text with a base direction of right-to-left">

Rewrite Internationalization section. #641

Rewrite Internationalization section. #641

Conversation

msporny commented May 25, 2019 • edited by pr-preview bot Loading

aphillips left a comment

Choose a reason for hiding this comment

aphillips left a comment

Choose a reason for hiding this comment

msporny commented May 26, 2019 • edited Loading

msporny commented Jun 9, 2019

chaals left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dlongley Jun 10, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chaals commented Jun 10, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

msporny Jun 30, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

msporny Jun 30, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

msporny left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

msporny commented Jun 30, 2019

msporny commented Jun 30, 2019

msporny commented May 25, 2019 •

edited by pr-preview bot

Loading

msporny commented May 26, 2019 •

edited

Loading

dlongley Jun 10, 2019 •

edited

Loading

msporny Jun 30, 2019 •

edited

Loading

msporny Jun 30, 2019 •

edited

Loading