Place island grammars into separate files

[renatahodovan]

Hi All,

It's might be a dummy question, but is it possible to organize island grammars into separate files and import them into the main grammar (or is it something equivalent solution to avoid copy-pasting island rules into the main grammar)?

E.g., If I'd like to parse an HTML file with style definitions and scripts:

HTMLParser.g4

parser grammar HTMLParser;

options { tokenVocab=HTMLLexer; }

import CSSParser, JavaScriptParser;

htmlDocument : ... ;
...
style : STYLE_OPEN stylesheet STYLE_CLOSE ;
script : SCRIPT_OPEN program SCRIPT_CLOSE ;

HTMLLexer.g4

lexer grammar HTMLLexer;

...
STYLE_OPEN : '<style' .*? '>'  ->pushMode(STYLE) ;
SCRIPT_OPEN : '<script' .*? '>' ->pushMode(SCRIPT) ;
...


mode STYLE;

import CSSLexer;

STYLE_CLOSE : '</style>' -> popMode ;


mode SCRIPT;

import JavaScriptLexer;

SCRIPT_CLOSE : '</script>' -> popMode ;

Where CSSLexer.g4 and CSSParser.g4 contain the CSS rules and JavaScriptLexer.g4 and JavaScriptParser.g4 contain the JS rules. (The snippet above is obviously wrong, since ANTLR is complaining about the lexer imports.)

[ericvergnaud]

IIRC the imports need to come first i.e.:

lexer grammar HTMLLexer;
import CSSLexer, JavaScriptLexer

Your example seems to lean more towards an include. That's not supported.

[renatahodovan]

@ericvergnaud Thanks for the reply! Indeed, imports are only allowed at the beginning of the grammar.

However, it seems that there is a solution to avoid copy-pasting complete gramamrs: if the island grammar to be imported "starts" with mode definitions, then those imported modes are respected in the main grammar as well. However, there is one drawback: grammars cannot start with mode definition, it needs at least one placeholder rule before the modes. (Is it really needed?)

My HTML grammar with CSS and JS island looks like:

HTMLParser.g4:

parser grammar HTMLParser;

// The superClass is needed to be set for the main grammar since the imported JavaScriptParser had this superClass.
options { tokenVocab=HTMLLexer; superClass=JavaScriptParserBase; }

import CSSParser, JavaScriptParser;

// HTML parser rules ...
script : SCRIPT_OPEN program SCRIPT_CLOSE;
style: STYLE_OPEN stylesheet STYLE_CLOSE;

HTMLLexer.g4:

lexer grammar HTMLLexer;
import CSSLexer, JavaScriptLexer;
// SuperClass is needed to be adapted for lexer as well.
options { superClass=JavaScriptLexerBase; }

// HTML lexer rules...

CSSLexer.g4:

lexer grammar CSSLexer;

// Can I avoid this somehow?
Dummy : 'dummy';

mode STYLE;

STYLE_CLOSE : '</style>' -> popMode ;

// Further CSS lexer rules ...

JavaScriptLexer.g4:

lexer grammar JavaScriptLexer;
// It has no effect now.
options { superClass=JavaScriptLexerBase; }

// Can I avoid this somehow?
Dummy : 'dummy';

mode SCRIPT;

SCRIPT_CLOSE : '</script>' -> popMode ;

// Further JS lexer rules ...

So my question is: Is this the right way to reuse existing grammars as islands? (It feels that I'm missing something, since imports aim to keep the grammars clean avoiding copy-pasting rules and using existing grammars as islands seems a natural use-case of this feature. My solution feels still a bit hacky because of the Dummy rules.)

Thanks in advance!