Modelling token sets

[uuf6429]

I've recently started using Antlr to build a prototype of a language I have in mind.
One of the interesting functionalities is the ability to switch between sets of tokens.
For example, a "traditional set" would look like:

PROP_ACCESS: '.';
STMT_TERM:   ';';
MTD_L:       '(';
MTD_R:      '(';

Whereas a "natural set" could look like:

PROP_ACCESS: ' ';
STMT_TERM:   '.';
MTD_L:      ' ';
MTD_R:      ' ';

My first thought was to somehow switch between these sets based on some top-level config/pragma/whatever (is this even possible?). Or perhaps it makes more sense to actually have separate grammars?

[ericvergnaud]

You’d definitely want separate grammars, but FYI the latter is not valid i.e. 3 of the tokens have the same pattern so only 1 will ever be generated Envoyé de mon iPhoneLe 8 juil. 2023 à 12:26, Christian Sciberras ***@***.***> a écrit : I've recently started using Antlr to build a prototype of a language I have in mind. One of the interesting functionalities is the ability to switch between sets of tokens. For example, a "traditional set" would look like: PROP_ACCESS: '.'; STMT_TERM: ';'; MTD_L: '('; MTD_R: '('; Whereas a "natural set" could look like: PROP_ACCESS: ' '; STMT_TERM: '.'; MTD_L: ' '; MTD_R: ' '; My first thought was to somehow switch between these sets based on some top-level config/pragma/whatever (is this even possible?). Or perhaps it makes more sense to actually have separate grammars? —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[uuf6429]

but FYI the latter is not valid

I suspected that as well, and was wondering what would happen.

Anyway, thanks for the reply!

PS: I was able to jump into the subject and do a lot of stuff with the antlr jetbrains plugin - that's very cool and the effort well appreciated!

[KvanTTT]

You’d definitely want separate grammars, but FYI the latter is not valid i.e. 3 of the tokens have the same pattern so only 1 will ever be generated

Not correct. All three tokens will be generated but TOKEN_UNREACHABLE warnings will be reported.

[ericvergnaud]

I meant they won’t be generated at runtimeEnvoyé de mon iPhoneLe 8 juil. 2023 à 16:20, Ivan Kochurkin ***@***.***> a écrit : You’d definitely want separate grammars, but FYI the latter is not valid i.e. 3 of the tokens have the same pattern so only 1 will ever be generated Not correct. All three tokens will be generated but TOKEN_UNREACHABLE warnings will be reported. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***>

[KvanTTT]

Ok, in runtime they won't be recognized but not generated (using term generation for both generation and runtime sounds confusing).