Text in csv files is shown incorrectly on GitHub

[klefo0]

The format convert right, but the string translate output is wrong badly.

[insolor]

This is a normal result. Github shows readable text only for text in utf-8 encoding and in (I guess) cp1252 (Western European Windows encoding).

Csv files are encoded in different encodings (as configured in config.yaml), so don't expect them to be shown correctly in GitHub or Excel (for example).

[klefo0]

Even I downloaded csv file and open it in notepad or notepad++, it shows the csv file encode is in ANSI not in UTF-8. Try it yourself

[insolor]

@klefo0 ANSI is not an encoding, this is rather a familiy of encodings used in Windows (see https://en.wikipedia.org/wiki/ANSI_character_set). Exact ANSI variant used in your system (and used as ANSI by Notepad++ and other editors) depends on your Windows language and configuration.

In general csv files here (in this project) are encoded in DOS encodings, with some exceptions. For Ukrainian cp1251 encoding (Windows 1251, one of so called ANSI encodings) is used, and the csv file is shown in this encoding correctly. The same for Belarusian.

For Vietnamese not standard encoding VISCII is used, and I guess it's nearly impossible to find an editor which supports it (maybe there are some specific patches for editors for it).

But in general, if you open a csv file using encoding specified in config.yaml, you'll see correct text. For example, Spanish + cp850:

Note: vscode editor (which I use) doesn't even try to guess an encoding of a file, I just manually click on UTF-8 text, click "Reopen with Encoding" and choose correct encoding.