Support for charsets and unify internal character storage type

[mplucinski]

This pull request introduces two changes:

• 2e6add4: in places where text characters are processed, replace char type by YY_CHAR macro. This introduces distinguishness between character as byte of file ("physical") and character as the smallest part of text ("logical"). For logical one, YY_CHAR macro should be used. For now, they represent the same value, but in future YY_CHAR may be replaced by wider type. This is essential to introduce support for extended character sets (any set with number of code points bigger than 256).

  To properly introduce YY_CHAR macro usage, some tests are changed to not use str* family of standard C functions, but instead versions that operate on YY_CHAR values. New functions are sitting in tests/strutils.h header, and test case for this file is located in tests/test-tests_strutils directory.

• f1456f9, 0d3107e: adds support for character sets. The first commit adds test case, the second actual support. This gives way to convert between "physical" bytes and "logical" characters with user-provided function. It allows input data to be encoded in different character set than the one that has been used to write input ".l" file.

  Charset support is completely optional. It is activated by %option charset. When it is active, new variables are activated: yycharset and yycharset_handler. First one should be set to currently used character set before calling yylex(). Second one should be set to function that will be called to convert incoming bytes into characters. Additionally, %option charset-source="ENCODING" should be set to the name of encoding used to write ".l" file - it will be used as internal scanner's character encoding. In case when incoming data are in the same encoding as internal one, conversion function is not used.

[mplucinski]

Forgot to mention: first commit also introduces "char header file". The idea is to put generated YY_CHAR definition into separate file, to make this macro available in places where we cannot include general scanner header, e.g. Bison files.

[westes]

Could you rebase this onto the current tip of master?

The usual pull methods are generating conflicts.

[mplucinski]

Sure, I've just submitted it as request #8.