Support Unicode encodings

[Serentty]

It would be convenient if Millfork allowed string literals for UTF-8 and UTF-16. This might seem like an odd feature to have, but I've been working on supporting rendering Unicode text on the Commander X16, especially CJK text. UTF-8 would also make sense for platforms such as the Altair 8800 where encoding is mostly a matter of agreement between the software running on the computer and the terminal being used to access it.

[KarolS]

Done. It will be available in the next release.

[Serentty]

Wow, that was fast! Thanks.

[Serentty]

I just saw that for UTF-8, it only allows characters within the BMP. What's the reasoning behind this?

[KarolS]

Honestly, the reason was me being too lazy to mess with the Char-oriented code too much, and Chars are limited to U+FFFF. But a proper fix wasn't even that hard, so I implemented it as well.

[Serentty]

Yay! And sorry to keep bringing things up on the same issue, but I feel like it would be even more spammy to keep opening new ones. Would it be possible to add a compile-time flag to disable the printable ASCII limitation in code, so that such characters can be used in identifiers? Or is there a technical aspect in the parser that prevents that? I know that Unicode equivalency is a big can of worms, but frankly I would be satisfied if it just treated identifiers blindly without worrying about normalization or anything like that.

If you're sick of Unicode by now, I would be happy to make a PR for this if you would be willing to merge it.

[KarolS]

0.3.10 has been released and contains the UTF support for string literals.

I'll leave the identifiers on the table.