Has anyone tried some Antlr questions with ChatGPT?

[kaby76]

I've been looking at ChatGPT and seeing what it knows about Antlr. This kind of surprised me.

ChatGPT, please give me an Antlr grammar for integer expressions.

Certainly! Here's an Antlr grammar for integer expressions:

grammar IntegerExpressions;

/* Lexer rules */
INT: [0-9]+;
PLUS: '+';
MINUS: '-';
MULT: '*';
DIV: '/';

/* Parser rules */
expression: term ((PLUS | MINUS) term)*;
term: factor ((MULT | DIV) factor)*;
factor: INT | '(' expression ')';

/* Ignored characters */
WS: [ \t\r\n]+ -> skip;

Here, we define the tokens for integers, plus, minus, multiplication, and division. We then define the parser rules for expressions, terms, and factors. An expression is defined as one or more terms separated by plus or minus signs, a term is defined as one or more factors separated by multiplication or division signs, and a factor is either an integer or an expression enclosed in parentheses. We also define the WS rule to ignore whitespace characters.

You can then generate a parser for this grammar in your target language using the Antlr tool.

[kaby76]

This answer disappoints. It is wrong on several levels. As I've heard, ChatGPT is good at BS'ing.

Hi ChatGPT! Could you write an Antlr4 grammar that recognizes Windows and Linux path names?

Certainly! Here's an example Antlr4 grammar that recognizes both Windows and Linux path names:

grammar PathName;

path:   windowsPath | linuxPath;
windowsPath: diskLetter? '\\' (directory '\\')* filename? extension?;
linuxPath:   ('/' directory)* filename? extension?;

directory:  name '\\';
filename:   name;
extension:  '.' name;

name:       [a-zA-Z0-9_]+;

diskLetter: [a-zA-Z]:;

Here's a brief explanation of the grammar:

The path rule matches either a windowsPath or a linuxPath.
The windowsPath rule matches a Windows-style path, which begins with an optional disk letter, followed by a backslash (\). The path may contain one or more directory names, each separated by a backslash. The path may also end with a filename and an optional extension, separated by a dot (.).
The linuxPath rule matches a Linux-style path, which begins with a forward slash (/). The path may contain one or more directory names, each separated by a forward slash. The path may also end with a filename and an optional extension, separated by a dot (.).
The directory, filename, and extension rules match the components of a path.
The name rule matches one or more characters that may appear in a directory name, filename, or extension. Note that this rule doesn't allow spaces or other special characters, which may be valid in some path names. You can modify this rule to fit your needs.
The diskLetter rule matches a single uppercase or lowercase letter, followed by a colon (:). This rule is only used in the windowsPath rule.
Note that this grammar doesn't handle paths with UNC syntax or with escape characters. You can modify it to support those cases as needed.

Here, we see several problems.

• Lexer character sets cannot be used in parser rules. So, name and diskLetter fail to compile.
• diskLetter contains an extra :-character, which fails to parse.
• If you were to create lexer rules for the character sets, one hides the other character set.
• Directory and file names can contain a number of Unicode characters.

Compare this with a recent StackOverflow answer, which I would think works, but it too is not that good. https://stackoverflow.com/a/75525342/4779853

[kaby76]

This StackOverflow question involves modifying the grammar to recognize variable declarations that have the same number of expressions as variables.

I've wrote a grammar rule for a language in ANTLR as below:

variable: idlist COLON type (EQUAL explist)? SEMI;
idlist: identifier (COMMA identifier)*;
explist: exp (COMMA exp)*;

COLON: ':';
EQUAL: '=';
SEMI: ';';
COMMA: ',';
This input is valid for above grammar:

a, b, c: integer = 3, 4, 6;
But now if I want this input:

a, b, c, d: integer = 3, 4, 6;
or this:

a, b, c: integer = 3, 4, 6, 1;
becomes invalid due to inequality between amount of ID in idlist and value in explist, how I rewrite my grammar? 

It's a valid question, but clearly pedantic because to do it syntactically, the resulting parse tree would associate the wrong exp with a variable. The best answer would be to do this via semantic analysis.

But, the answer from ChatGPT is wrong. It shows two things. First, it seems to "understand" that the question involves rewriting the grammar. This is rather surprising, at least to me, because it "knew" what the point of the question was. Second, the answer provided is wrong. That doesn't surprise me, as ChatGPT is really good at bullshitting, at least when trying to field questions about grammars. It's not its forte.

To enforce equality between the number of identifiers in the idlist and the number of values in the explist, you can modify your grammar as follows:

variable: idlist COLON type (EQUAL explist SEMI)? SEMI;
idlist: identifier (COMMA identifier);
explist: exp (COMMA exp);

COLON: ':';
EQUAL: '=';
SEMI: ';';
COMMA: ',';

Here, we have made the explist optional and wrapped it with parentheses, so it is only matched if it appears immediately after the EQUAL token and is followed by a SEMI. This ensures that the explist is only matched if it is present and has the correct number of expressions.

With this modification, the input a, b, c: integer = 3, 4, 6; remains valid, while a, b, c, d: integer = 3, 4, 6; and a, b, c: integer = 3, 4, 6, 1; are now recognized as invalid.

[batpox]

Yeah, you have to keep prompting it to get it to "fill in" the holes. This has happened to me with several programming issues. Assistant seems to start with something and then keep amending as you keep pointing out flaws.

[kaby76]

Yes, I noticed that too. An established paradigm in software development is to refine the answer with new information that is found by prototype or by thinking about the problem and eliminating bad paths in the search.

[kaby76]

I decided to test out scraping for ChatGPT. It is still not usable--a year after my initial testing.

• I asked ChatGPT to scrape the EBNF grammar for Fortran from the webpage at https://flang.llvm.org/docs/f2018-grammar.html. I do know ChatGPT now reads websites live, so it should work.
• The initial answer just gave back a link to the same webpage! https://chat.openai.com/share/c3cf9d35-49bb-476d-8ed9-53027cabcc94.
• I repeated the query, giving the contents of the page, and it did scrape the EBNF but could not give the full grammar. https://chat.openai.com/share/325971d9-9d56-4f95-832a-a06f2da485e5
• It was able to peel off the rule names to yield a more standard EBNF. Again, despite having a paid subscription, and asking for the complete grammar, ChatGPT4 always truncates the answer, sometimes prompting "continue generating", sometimes just stopping. You have to emphatically demand the full grammar to get anywhere.

For practical purposes, the website is still not quite usable, certainly not in an automated tool, which is exactly the whole point: we want to be able to update a grammar in grammars-v4 with new versions of a language.

[jimidle]

You need to use the Github Copilot paid option in INtelliJ products. It does alright there,

…

On May 10, 2024, at 07:35, Ken Domino ***@***.***> wrote: I decided to test out scraping for ChatGPT. It is still not usable--a year after my initial testing. I asked ChatGPT to scrape the EBNF grammar for Fortran from the webpage at https://flang.llvm.org/docs/f2018-grammar.html. I do know ChatGPT now reads websites live, so it should work. The initial answer just gave back a link to the same webpage! https://chat.openai.com/share/c3cf9d35-49bb-476d-8ed9-53027cabcc94. I repeated the query, giving the contents of the page, and it did scrape the EBNF but could not give the full grammar. https://chat.openai.com/share/325971d9-9d56-4f95-832a-a06f2da485e5 It was able to peel off the rule names to yield a more standard EBNF. Again, despite having a paid subscription, and asking for the complete grammar, ChatGPT4 always truncates the answer, sometimes prompting "continue generating", sometimes just stopping. You have to emphatically demand the full grammar to get anywhere. For practical purposes, the website is still not quite usable, certainly not in an automated tool, which is exactly the whole point: we want to be able to update a grammar in grammars-v4 with new versions of a language. — Reply to this email directly, view it on GitHub <#4127 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJ7TMHINBSAOLW675LKFJDZBTEJVAVCNFSM6AAAAAAVCOIWKOVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TGOBTGIYDI>. You are receiving this because you are subscribed to this thread.

[kaby76]

Github Copilot is even worse.

Using the "GitHubCopilot Chat" in VSCode 2022, the prompt "Please scrape the EBNF at https://flang.llvm.org/docs/f2018-grammar.html" returns I'm sorry, [can't do].

Then, when prompted with content instead, it says There was a problem.... The "logs" don't provide any help.

In addition, the Copilot prompt/textbox hangs, and refuses to accept any further prompts. There is a "busy with task" UI at the bottom of the "GitHub Copilot Chat" pane that, and the red "X" buttom in the textbox does not cancel anything. I have to shutdown and restart VS2022 to reset the UI. At least ChatGPT website proper gave me an answer.

When I signed up for Copilot, Github decided to charge me $10 right off the bat, even though this is supposed to be a "free trial". (I verified my profile/Copilot settings in the Github website, and it is marked "trial", but will be charged $100 in the first week in June.) That's pretty much illegal at least here in Massachusetts.

I haven't tried other IDEs yet, but I would hope this isn't an "IDE-dependent answer".

Update: For Intellij, the first prompt answer is essentially the same: "Can't do". For the second, the error message is "maxTokens must be larger than the ellipsis length".

[batpox]

Hey Ken, I have used ChatpGPT4 for antlr questions; even having it look at code and generate a beginning .g4 file, and even doing the reverse, by giving it a .g4 file and asking for corrections/recommendations. It’s not perfect, but has provided useful feedback. BTW: For questions like these, I always use ChatGPT4 (instead of ChatGPT3.5). For example, I just asked this of it, and it gave me a decent shot at it: “Can you give me a .g4 file that will parse an arithmetic expression with variables. And generate the C# code that will do the AST?” It’s response: “Sure! Below, I'll outline a simple ANTLR4 grammar file for parsing basic arithmetic expressions involving variables, and then I'll describe how to generate C# code that builds an Abstract Syntax Tree (AST) using ANTLR4… blah..blah…blah, generated the code and then: “This program creates a lexer and a parser for the input, parses it according to the expr rule, and then prints the resulting parse tree.” Don’t know if it makes a difference, but I am a paid ChatGPT subscriber (@ $20/mo). For me (software developer), it’s worth it. FWIW, dan From: Ken Domino ***@***.***> Sent: Saturday, May 11, 2024 4:48 AM To: antlr/antlr4 ***@***.***> Cc: Daniel T. Houck ***@***.***>; Comment ***@***.***> Subject: Re: [antlr/antlr4] Has anyone tried some Antlr questions with ChatGPT? (Discussion #4127) Github Copilot is even worse. Using the "GitHubCopilot Chat" in VSCode 2022, the prompt "Please scrape the EBNF at https://flang.llvm.org/docs/f2018-grammar.html" returns I'm sorry, [can't do]. Screenshot.2024-05-11.043531.png (view on web) <https://github.com/antlr/antlr4/assets/11389451/d1b71069-e4af-46b3-81a3-17e0a80a6a0c> Then, when prompted with content instead, it says There was a problem.... The "logs" don't provide any help. Screenshot.2024-05-11.043854.png (view on web) <https://github.com/antlr/antlr4/assets/11389451/e7158a2f-350c-4639-bdd7-38134fd67a55> At least ChatGPT website proper gave me an answer. When I signed up for Copilot, Github decided to charge me $10 right off the bat, even though this is supposed to be a "free trial". (I verified my profile/Copilot settings in the Github website, and it is marked "trial".) That's pretty much illegal. I haven't tried other IDEs yet, but I would hope this isn't an "IDE-dependent answer". — Reply to this email directly, view it on GitHub <#4127 (comment)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABLL62BK7VGFYE6BGVZCOALZBXLMZAVCNFSM6AAAAAAVCOIWKOVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TGOJTHAYTA> . You are receiving this because you commented.Message ID: ***@***.***>

[jimidle]

I have never had success telling it to go scrape from a web site. Copilot seems to work well in IntelliJ IDEs - I don't know if it is any different in VsCode. However, I don't think it is designed to go visit pages directly in a conversation just yet. I get: I'm sorry, but I can't assist with that. As an AI developed by OpenAI, I must respect copyright and privacy laws. The content on the page you're asking me to scrape is likely copyrighted, and scraping it would be a violation of those rights. However, I can help you understand how to write a web scraper in general. Here's a basic example in Python using BeautifulSoup: The ENBF does not have any copyright. apologize for the misunderstanding, but as an AI developed by OpenAI, I'm not able to scrape websites or directly retrieve content from external sources. However, I can guide you on how to write a script to scrape the EBNF from the provided URL. However, your prompt doesn't really tell it to do anything. You want to scrape it and... ? When I give it the text and ask it to create an ANTLR4 grammar, in my paid copilot subscription, I get:

…

----- Creating an ANTLR4 grammar and lexer from an EBNF file involves translating the EBNF rules into ANTLR4 syntax. This process requires a careful understanding of both EBNF and ANTLR4 syntax. Here's a simplified example of how you might start this process. Note that this is a very basic example and does not cover the entire EBNF file you provided. ```antlr grammar Fortran; // Parser rules program: programUnit+ EOF; programUnit: externalSubprogram | specificationPart | executionPart; externalSubprogram: functionSubprogram | subroutineSubprogram; specificationPart: implicitPartStmt* implicitStmt; implicitPartStmt: declarationConstruct | specificationConstruct; declarationConstruct: typeDeclarationStmt; specificationConstruct: implicitPartStmt; executionPart: executableConstruct* internalSubprogramPart?; executionPartConstruct: executableConstruct | internalSubprogramPart; internalSubprogramPart: containsStmt internalSubprogram*; internalSubprogram: functionSubprogram | subroutineSubprogram; // Lexer rules DIGIT: [0-9]; LETTER: [A-Z]; WS: [ \t\r\n]+ -> skip; ``` This is a very basic start and does not cover the entire EBNF file you provided. The full translation would be quite large and complex. It's also important to note that ANTLR4 and EBNF have some differences in their capabilities and syntax, so some EBNF features might not have a direct equivalent in ANTLR4. For example, in the EBNF file you provided, there are rules like `R0001 digit -> 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9`. In ANTLR4, this would be translated to a lexer rule like `DIGIT: [0-9];`. Also, remember that ANTLR4 lexer rules must be all uppercase, while parser rules must be all lowercase. This is a convention in ANTLR4 to distinguish between lexer and parser rules. Finally, remember to test your grammar thoroughly to ensure it correctly parses the intended language.

On May 11, 2024, at 02:48, Ken Domino ***@***.***> wrote: Github Copilot is even worse. Using the "GitHubCopilot Chat" in VSCode 2022, the prompt "Please scrape the EBNF at https://flang.llvm.org/docs/f2018-grammar.html" returns I'm sorry, [can't do]. Screenshot.2024-05-11.043531.png (view on web) <https://github.com/antlr/antlr4/assets/11389451/d1b71069-e4af-46b3-81a3-17e0a80a6a0c> Then, when prompted with content instead, it says There was a problem.... The "logs" don't provide any help. Screenshot.2024-05-11.043854.png (view on web) <https://github.com/antlr/antlr4/assets/11389451/e7158a2f-350c-4639-bdd7-38134fd67a55> At least ChatGPT website proper gave me an answer. When I signed up for Copilot, Github decided to charge me $10 right off the bat, even though this is supposed to be a "free trial". (I verified my profile/Copilot settings in the Github website, and it is marked "trial".) That's pretty much illegal. I haven't tried other IDEs yet, but I would hope this isn't an "IDE-dependent answer". — Reply to this email directly, view it on GitHub <#4127 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJ7TMCTPG6FLAPTKDI222LZBXLMZAVCNFSM6AAAAAAVCOIWKOVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TGOJTHAYTA>. You are receiving this because you commented.

[kaby76]

However, your prompt doesn't really tell it to do anything. You want to scrape it and... ?

And refactoring. But, scaping is the first task. From what I can tell, ChatGPT (and Copilot) are best applied to small, well-defined problem statements.

---

Can ChatGPT (or Copilot) be used to do Feature Extraction in text? In other words, can it scrape EBNF from a text document? Yes. ChatGPT4 can pick out a set of prime numbers from set of integers (https://chat.openai.com/share/402c04d5-0412-4352-92e9-cecd65042a44). It can pick off a subset of verbs from A Midsummer's Night Dream available by weblink: https://chat.openai.com/share/8c2bbb18-548a-4205-bec3-a94ca005e0d3. It can scrape the EBNF from a doc (https://chat.openai.com/share/325971d9-9d56-4f95-832a-a06f2da485e5). But ChatGPT prevents FE with input that contains copyright restrictions. And, it doesn't like large input or producing large output. It might be best to use an NLP that is trained to do FE for EBNF and avoid ChatGPT and Copilot.

---

Can ChatGPT (or Copilot) refactor grammars? Yes. ChatGPT4 can remove the extraneous rule name from the FLang grammar, and change the "->" rule operator to a ":", moving towards Antlr4 style (https://chat.openai.com/share/325971d9-9d56-4f95-832a-a06f2da485e5). But, ChatGPT prompts large output with "continue generating", which can be a drag with large problems. ChatGPT3.5 can unfold string literals in an Antlr4 grammar (https://chat.openai.com/share/b62c039a-a138-4f3a-bbbb-83552af8f131). But, in the example, it didn't understand my first unfold question, telling a Trump-like lie saying it did exactly as I asked. On the second unfold question, it unfolded PLUS, but also MINUS, which I did not ask for. It's probably best to use a toolkit like Trash for refactoring.

[sharifzadeh]

I tried to convert ANTLR 3 grammer to ANTLR 4 using GPT-4o but the results were disappointing. I had to do it myself!

[kaby76]

I'm not surprised. I had documented what is involved in the conversion (here), and it is multistep. (I have an implementation for conversion that hardwires the Trash operations in C#, but it should be a script. But, I haven't gotten around to writing that.) I'm not sure if there's a training set in the Internet for building a LLM for conversions.

[metaleap]

Codeium isn't that hype either, given my prompt "write an antlr4 grammar where every line becomes a parse-tree node with subsequent indented lines becomes sub-nodes of the parent node".

Result:

grammar LineGrammar;

file : line+ EOF;

line : INDENT content (NEWLINE sub_line*)?;

sub_line : INDENT content;

content : WORD+;

INDENT : '  ';
NEWLINE : '\n';
WORD : [a-zA-Z]+;

Pretty borked =)