An issue on Searching Japanese words.

[su-jang]

An issue on Searching Japanese words.

When searching for Japanese words, there is no distinction between unvoiced consonants(清音) and voiced consonants(濁音).
For example, if you search for a word "Agaru(あがる)", "Agaru(あがる)" and "Akaru(あかる)" will be searched at the same time.

Those are making the search display very complicated.

[xiaoyifang]

if you search for a word "Agaru(あがる)", "Agaru(あがる)" and "Akaru(あかる)" will be searched at the same time.

try add japanese morphology

And as a side topic, the GoldenDict-NG does not have setting options for "exact search", "forward search", and "backward search". The related issu was discussed earlier at github, #79.

#79 discuss about the page search which rely on chrome's search functionality. Chrome does not support it.

[darlopvil]

Are my messages currently being deleted? I swear this is the 3rd time i post a message here, last time i even attached a video.

This issue is getting old. Adding morphology doesn't fix the problem, and even searching for kanjinized words you get the same results...tons of unrelated entries by all dictionaries of the group of search.
Video:

goldendict_wrong_entries.mp4

[shenlebantongying]

が has two representations

As a standalone
(ga) U+304C HIRAGANA LETTER GA

Or as a combination of
(ka) か HIRAGANA LETTER KA
(ten or hw_ka) U+3099 COMBINING KATAKANA-HIRAGANA VOICED SOUND MARK (or one of https://en.wiktionary.org/wiki/%E3%82%9B)

Then we get the Unicode normalization's table8, both NFKD & NFKC will merge various combination of hiragana (?) letter and voice sound marks.

https://www.unicode.org/reports/tr15/tr15-56.html

Other references

• ja char table https://www.asahi-net.or.jp/~ax2s-kmtn/ref/unicode/u3040.html
• wikithionary HIRAGANA LETTER KA https://en.wiktionary.org/wiki/%E3%81%8B

So, the root issue appears to be the normalization process, which is considered as a feature in some other languages.

[darlopvil]

in other words, "we're not fixing this".

[xiaoyifang]

Can you provide the dictionary for testing ?

in other words, "we're not fixing this".

Maybe in the future ,this issue will be solved.

[darlopvil]

Can you provide the dictionary for testing ?

JMDict Furigana, JMDict+: https://jd4gd.com/jmdictplus.html
Jitendex: https://jitendex.org/pages/downloads.html
Edict+: https://jd4gd.com/edictplus.html

That's 4.

[xiaoyifang]

not reproduced. search word 異なる

Maybe ,try to delete the dictionary's index and try again.
or try to disable some of your other dictionaries , seems like some synonymy word search？

[shenlebantongying]

I think the issue is both か and が are showing in the results ?

[xiaoyifang]

from the video , the JMdict has show both the word 違う and 異なる （both mean different）
I wildly guess that maybe some synonymy strategy has been mixed in .

[darlopvil]

Maybe ,try to delete the dictionary's index and try again. or try to disable some of your other dictionaries , seems like some synonymy word search？

Did it and nothing changed. Also, i've deleted all these dicts from the directory and still the same issue with the rest of dicts.
Jitendex for example (made by other different author behaves the same)

[shenlebantongying]

The fix is changing Unicode normalization strategy from NormalizationForm_KD to NormalizationForm_KC

goldendict-ng/src/common/folding.cc

Line 25 in dda91a3

.normalized( QString::NormalizationForm_KD )

After the change:

---

The technical reason behind is shown in this table https://unicode.org/reports/tr15/#NFKD_And_NFKC_Applied_Table

In NormalizationForm_KD or NFKD, the ga が will be decomposed into ka か + ten ◌゙, so when searching ga, both ka and ka+ten shows up.

---

Not sure what to do. Because changing that value will probably break other languages that consider characters with these marks as the same characters (Also not sure if this is true. Does NFKC more suitable here?).

It may also require all dicts to be reindexed.

[xiaoyifang]

In NormalizationForm_KD or NFKD, the ga が will be decomposed into ka か + ten ◌゙, so when searching ga, both ka and ka+ten shows up.

This has not explained why search 異なる ,違う also show up. and can not explain why I can not reproduce it.

[xiaoyifang]

Also, i've deleted all these dicts from the directory and still the same issue with the rest of dicts.

BTW,You do not have to delete the other dictionaries , just disable them .or only enable the tested dictionary.
or create a group for your tested dictionary to get rid of other dictionaries‘ impact.

[darlopvil]

well, the issue is that after taking off those dicts, the problem still didn't get fixed.
In fact, when searching for 異なる the issue persists. So i just left the dicts as they are. It's annoying af but at least i can use them to look up words.

This is how it looks now: (only showing three dicts from a bigger list)

[xiaoyifang]

uncheck this option. Edit->preference->advanced->Extra search via synonyms