[new-parser] Merge dev-new-parser to main #5989

tkobayas · 2024-06-11T05:43:27Z

Issue:

[new-parser] Merge dev-new-parser to main #5988

Merging dev-new-parser branch to main branch.

Merging has to be done after ~~Drools 10.0 release~~ 10.0.x is branched out.
~~Until the Drools 10.0 release~~ Until 10.0.x is branched out, mark the PR [DO-NOT-MERGE] and make it draft.
This PR makes the new parser not default parser. So it shouldn't affect the default behavior.
Make sure to use "Rebase and merge" (not "Squash and merge") in the Github UI to keep commit history.
After Drools 10 release, discuss the future plan in dev at kie.apache ML.

tkobayas · 2024-06-11T05:49:52Z

[DRAFT] I'll ask for review after ~~Drools 10.0 is released~~ 10.0.x is branched out.

tkobayas · 2024-06-11T06:08:42Z

drools-drl/drools-drl-parser/src/main/java/org/drools/drl/parser/DrlParser.java

+    public static final String ANTLR4_PARSER_ENABLED_PROPERTY = "drools.drl.antlr4.parser.enabled";
+
+    // temporarily removed 'final' for testing purposes. This should be final when the feature gets stable
+    public static boolean ANTLR4_PARSER_ENABLED = Boolean.parseBoolean(System.getProperty(ANTLR4_PARSER_ENABLED_PROPERTY, "false")); // default is false


The new parser is disabled by default. So it shouldn't affect the default behavior. Also unit tests (except drools-drl-parser-tests) run with the old parser.

#5990 is for the new parser CI check.

tkobayas · 2024-10-02T08:02:30Z

Reminder:
Make sure to use "Rebase and merge" (not "Squash and merge") in the Github UI to keep commit history.

tkobayas · 2024-10-03T05:36:01Z

Hi @mariofusco @gitgabrio @baldimir @pibizza @lucamolteni ,

This is a PR to merge antlr4 based new parser. Note that this is not a default parser. So it shouldn't affect the default behavior. We would discuss the future plan in dev at kie.apache ML after 10.0.0 release.

This new parser pass all unit tests : Confirmed in #5990

Please review, thanks!

@disabled

apache#5682) * add antlr4 based drl parser * bump to drools 8.16.0.Beta * improve node position discovery * first c3 integration * Module reorganization (#2) * WIP * Ignore directories * Add debug. Minor modifications * WIP * Fix tests - add comments * manage null ast nodes and fix failing test * improve drl parser * Fix basic completion (#4) * Add Java grammar, fix DRL grammar, simplify token index calculation, remove template code * Add ignored tokens to completion Co-authored-by: livio <[email protected]> * add constraints parsing * - Added DRLParserWrapper to store errors using DRLErrorListener (#5) - Added MiscDRLParserTest which is being ported from RuleParserTest to enhance coverage. * Enhance test/grammar coverage. function import, global, exists, not (#6) * Enhance test/grammar coverage. and, or, listExpression (#7) * Enhance test/grammar coverage. matches, single quoted string, escape (#9) - Introduced drlExpression, drlPrimary, drlLiteral, DRL_STRING_LITERAL * Enhance test/grammar coverage. dialect, invalid lhs (apache#10) * Enhance test/grammar coverage. keyword collision (apache#11) - Add "DRL_" prefix for DRL keywords for clarification * Enhance test/grammar coverage. function, attributes, consequence (apache#12) * Enhance test/grammar coverage. function, attributes, consequence - better getText preserving whitespaces - resolving WARNs because of overlapping keywords * - No need to declare duplicate keywords * Enhance test/grammar coverage. OR, inline map (apache#13) * OpenRewrite applied for junit5 and assertj (apache#14) * Enhance test/grammar coverage. label, consequenceLocation, test refac… (apache#15) * Enhance test/grammar coverage. label, consequenceLocation, test refactoring, assertj best practice * - better test method names * [DROOLS-7253] Review and improve drools-lsp parser Visitor design (apache#16) * [DROOLS-7253] Review and improve drools-lsp parser Visitor design - Remove currentConstructStack. Utilize return object instead. * - Remove instance fields. Utilize return objects instead. * [DROOLS-7270] Enhance test/grammar coverage : andRestriction, orRestriction (#4) - Removed some unused cast * [DROOLS-7271] Fully port RuleParserTest as MiscDRLParserTest with @disabled and priority comment (apache#10) * [DROOLS-7285] Failed to parse and/or with parentheses in LHS (apache#15) * [DROOLS-7286] Failed to parse binding with || (apache#21) * [DROOLS-7287] Failed to parse comments in RHS (apache#23) * [DROOLS-7287] Failed to parse comments in RHS * - add rhs string check * [DROOLS-7515] Parser : Accept plain text in RHS (apache#26) * [DROOLS-7515] Parser : Accept plain text in RHS * - Handle keyword token in RHS * - Removed sysout * - Revert statementKeywords. Instead, define DRL_END with newline or EOF * - fix method name * [DROOLS-7288] Failed to parse complex parentheses (apache#27) * [DROOLS-7288] Failed to parse complex parentheses - bump to 8.43.0.Final * additional tests * [DROOLS-7289] Implement accumulate (apache#28) * [DROOLS-7296] Implement from entry-point (apache#30) * [DROOLS-7297] Implement import accumulate (apache#29) * [DROOLS-7290] Implement BigInteger literal, BigDecimal literal (apache#31) * [DROOLS-7292] Implement eval (apache#32) * [DROOLS-7293] Implement extends (apache#33) * [DROOLS-7294] Implement forall (apache#34) * [DROOLS-7298] Implement memberOf (apache#35) * [DROOLS-7300] Implement positional constraint (apache#36) * [DROOLS-7304] Implement temporal operators (apache#38) - Also covers [DROOLS-7303] Implement sliding window * [DROOLS-7301] Implement query (apache#37) - Also partially fixes [DROOLS-7302] Implement semicolon delimiter * [DROOLS-7295] Implement from collect (apache#39) * [DROOLS-7305] Implement type declaration (apache#40) - also covers [DROOLS-7291] Implement entry-point declaration - also covers window declaration * [DROOLS-7306] Implement unification (apache#41) * [DROOLS-7306] Implement unification - Also [DROOLS-7307] Parse attribute agenda-group * [DROOLS-7308] Parse attribute without value - Also [DROOLS-7309] Parse attribute with parentheses * [DROOLS-7310] parse nested parentheses (apache#42) * [DROOLS-7310] parse nested parentheses * - Adding code commentes and explanations - Adding more nested level tests * [DROOLS-7577] drools-lsp : Add comments and explanations for drools-p… (apache#43) * [DROOLS-7577] drools-lsp : Add comments and explanations for drools-parser * removed syntax examples * [DROOLS-7591] Experiment branch : migrate a new drools-lsp parser into drools (apache#44) - Add some methods to call from drools code base * Move necessary files without any additional modifications * Update DRL6Expressions.g4 to ANTLR 4 (apache#47) * Add DescrDumperTest and fix the expression parser grammar (apache#48) * Add DescrDumperTest * Fix it * [DROOLS-7591] Experiment branch : migrate a new drools-lsp parser into drools - New parser is migrated in drools-drl/drools-drl-parser with org.drools.drl10.parser package - To enable the new parser, run `mvn test -PDRL10` - At the moment, we get lots of test failures especially in drools-model/drools-model-codegen and drools-test-coverage/test-compiler-integration * - License header - Revert an inappropriate test edit - Removed unnecessary profile - Clarify unsupported methods - Removed unnecessary bom dependency --------- Co-authored-by: mariofusco <[email protected]> Co-authored-by: Gabriele Cardosi <[email protected]> Co-authored-by: Livio Benčik <[email protected]> Co-authored-by: livio <[email protected]> Co-authored-by: Jiří Locker <[email protected]>

…rser int…" (apache#5689) This reverts commit 3699d67.

* WIP * Ignore directories * Add debug. Minor modifications * WIP * Fix tests - add comments

* Add Java grammar, fix DRL grammar, simplify token index calculation, remove template code * Add ignored tokens to completion Co-authored-by: livio <[email protected]>

…not, or, and exists (apache#5914)

apache#5921) * [incubator-kie-drools-5918] [new-parser] Annotations can appear in lhsOr and lhsAnd * - infix or, and

…apache#5922) - Avoid unnecessary IllegalStateException - Avoid empty error message

…ache#5913) * [incubator-kie-drools-5908] ReteOOWaltzTest 'end' detection issue * Update drools-drl/drools-drl-parser-tests/src/test/java/org/drools/drl/parser/antlr4/MiscDRLParserTest.java Co-authored-by: Jiří Locker <[email protected]> * Update drools-drl/drools-drl-parser-tests/src/test/java/org/drools/drl/parser/antlr4/MiscDRLParserTest.java Co-authored-by: Jiří Locker <[email protected]> * Update drools-drl/drools-drl-parser-tests/src/test/java/org/drools/drl/parser/antlr4/MiscDRLParserTest.java Co-authored-by: Jiří Locker <[email protected]> * Update drools-drl/drools-drl-parser-tests/src/test/java/org/drools/drl/parser/antlr4/MiscDRLParserTest.java Co-authored-by: Jiří Locker <[email protected]> * Update drools-drl/drools-drl-parser-tests/src/test/java/org/drools/drl/parser/antlr4/MiscDRLParserTest.java Co-authored-by: Jiří Locker <[email protected]> --------- Co-authored-by: Jiří Locker <[email protected]>

…apache#5941) * [incubator-kie-drools-5924] [new-parser] Disable DRLIncompleteCodeTest * Remove editor mode tests

…side OR (apache#5950) * Tests for incubator-kie-drools-5945 * [incubator-kie-drools-5945] [new-parser] Broken named consequences inside OR --------- Co-authored-by: Jiří Locker <[email protected]>

apache#5958) * [incubator-kie-drools-5818] [new-parser] Parsing fails if a Java keyword appears in a qualified name * - Exclude 'new' from 'drlIdentifier' because of 'primary' ambiguity - Add tests

apache#5965) * [incubator-kie-drools-5909] [new-parser] Accumulate parsed incorrectly if init and action statements are empty * Update drools-drl/drools-drl-parser-tests/src/test/java/org/drools/drl/parser/antlr4/MiscDRLParserTest.java Co-authored-by: Jiří Locker <[email protected]> --------- Co-authored-by: Jiří Locker <[email protected]>

…as alternatives to infix or and and (apache#5966)

apache#5969) * [incubator-kie-drools-5938] [new-parser] Tests using wrong duration attribute format * - Use explicit test value

…nAccumulateFunction (apache#5975)

…ted (apache#5979)

…pache#5976) - Skip the test for new parser

…d for backward compatibility (apache#5977)

…#5981)

…s test (apache#5983)

apache#5984)

* [incubator-kie-drools-5792] cleanup drools-drl-parser-tests - Apply open-rewrite JUnit5BestPractices * - Apply open-rewrite SimplifyChainedAssertJAssertions * - remove 'parse_' prefix - remove unused 'throw Exception' * - code format

- new parser is disabled by default

tkobayas · 2024-10-11T01:19:28Z

Rebased to resolve conflict with #6115

pibizza

LGTM

tkobayas added the do-not-merge label Jun 11, 2024

tkobayas force-pushed the incubator-kie-drools-5988-merge branch from f6c9cf3 to 7da9750 Compare June 11, 2024 06:06

tkobayas commented Jun 11, 2024

View reviewed changes

tkobayas mentioned this pull request Jun 11, 2024

[DO-NOT-MERGE] Enable new parser test for CI check in relation to incubator-kie-drools/pull/5989 #5990

Closed

tkobayas self-assigned this Jun 11, 2024

tkobayas mentioned this pull request Jun 20, 2024

[incubator-kie-drools-5874] [new-parser] Adapt drlIdentifier to opera… #5951

Closed

tkobayas force-pushed the incubator-kie-drools-5988-merge branch from 12096bd to 1fb05d2 Compare October 2, 2024 07:48

tkobayas marked this pull request as ready for review October 2, 2024 10:01

tkobayas requested review from baldimir, mariofusco, gitgabrio, pibizza and lucamolteni October 3, 2024 05:31

tkobayas changed the title ~~[DO-NOT-MERGE] [new-parser] Merge dev-new-parser to main~~ [new-parser] Merge dev-new-parser to main Oct 3, 2024

tkobayas removed the do-not-merge label Oct 3, 2024

mariofusco approved these changes Oct 9, 2024

View reviewed changes

lucamolteni approved these changes Oct 9, 2024

View reviewed changes

tkobayas and others added 10 commits October 11, 2024 10:12

Revert "[DROOLS-7591] Experiment branch : migrate a new drools-lsp pa…

b4b6bba

…rser int…" (apache#5689) This reverts commit 3699d67.

add antlr4 based drl parser

2a88be2

bump to drools 8.16.0.Beta

c6605ba

improve node position discovery

1c3ba61

first c3 integration

eadbdec

Module reorganization (#2)

bf34cbf

* WIP * Ignore directories * Add debug. Minor modifications * WIP * Fix tests - add comments

manage null ast nodes and fix failing test

7e71947

improve drl parser

d9bd4de

Fix basic completion (#4)

ad7bf86

* Add Java grammar, fix DRL grammar, simplify token index calculation, remove template code * Add ignored tokens to completion Co-authored-by: livio <[email protected]>

tkobayas and others added 25 commits October 11, 2024 10:14

[incubator-kie-drools-5912] [new-parser] Corner-case combinations of …

e181657

…not, or, and exists (apache#5914)

[incubator-kie-drools-5918] [new-parser] Annotations can appear in lh… (

757fe84

apache#5921) * [incubator-kie-drools-5918] [new-parser] Annotations can appear in lhsOr and lhsAnd * - infix or, and

[incubator-kie-drools-5915] [new-parser] ctx.lhsPattern().size() == 0 (…

ae3d129

…apache#5922) - Avoid unnecessary IllegalStateException - Avoid empty error message

Match EOF at the end of the compilation unit (apache#5930)

3c6e364

Use InternalResource's encoding (apache#5940)

ad4e019

[incubator-kie-drools-5924] [new-parser] Remove DRLIncompleteCodeTest (…

9fb4c3d

…apache#5941) * [incubator-kie-drools-5924] [new-parser] Disable DRLIncompleteCodeTest * Remove editor mode tests

Set namespace of qualified declared types (apache#5947)

d7b152c

[incubator-kie-drools-5945] [new-parser] Broken named consequences in…

e3dc24e

…side OR (apache#5950) * Tests for incubator-kie-drools-5945 * [incubator-kie-drools-5945] [new-parser] Broken named consequences inside OR --------- Co-authored-by: Jiří Locker <[email protected]>

[incubator-kie-drools-5818] [new-parser] Parsing fails if a Java keyw… (

ec8521f

apache#5958) * [incubator-kie-drools-5818] [new-parser] Parsing fails if a Java keyword appears in a qualified name * - Exclude 'new' from 'drlIdentifier' because of 'primary' ambiguity - Add tests

Avoid IndexOutOfBoundsException in lhsUnary (apache#5956)

1a850da

[incubator-kie-drools-5936] [new-parser] || and && should be allowed …

47d0617

…as alternatives to infix or and and (apache#5966)

[incubator-kie-drools-5938] [new-parser] Tests using wrong duration a… (

75d6555

apache#5969) * [incubator-kie-drools-5938] [new-parser] Tests using wrong duration attribute format * - Use explicit test value

[incubator-kie-drools-5948] [new-parser] Broken testIncompatibleListO…

8c9183b

…nAccumulateFunction (apache#5975)

[incubator-kie-drools-5932] [new-parser] Anonymous classes not suppor…

bc324af

…ted (apache#5979)

[incubator-kie-drools-5920] [new-parser] Strict annotation parsing (a…

b7f1bcd

…pache#5976) - Skip the test for new parser

[incubator-kie-drools-5911] [new-parser] Support optional type keywor…

56a388f

…d for backward compatibility (apache#5977)

[incubator-kie-drools-5980-DSL] [new-parser] Enable DSL tests (apache…

11d3427

…#5981)

[incubator-kie-drools-5982] [new-parser] Enable alternative attribute…

05882ea

…s test (apache#5983)

[incubator-kie-drools-5933] [new-parser] PackageDescr.resource not set (

5a07673

apache#5984)

[incubator-kie-drools-5988] [new-parser] Merge dev-new-parser to main

51f3338

- new parser is disabled by default

License header for .g4

5fd3efc

License headers

32710cc

tkobayas force-pushed the incubator-kie-drools-5988-merge branch from 1fb05d2 to 32710cc Compare October 11, 2024 01:17

pibizza approved these changes Oct 11, 2024

View reviewed changes

tkobayas merged commit 8f0ca8e into apache:main Oct 11, 2024
9 checks passed

tkobayas mentioned this pull request Oct 31, 2024

[new-parser] Merge dev-new-parser to main #5988

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[new-parser] Merge dev-new-parser to main #5989

[new-parser] Merge dev-new-parser to main #5989

tkobayas commented Jun 11, 2024 •

edited

Loading

tkobayas commented Jun 11, 2024 •

edited

Loading

tkobayas Jun 11, 2024 •

edited

Loading

tkobayas commented Oct 2, 2024

tkobayas commented Oct 3, 2024 •

edited

Loading

tkobayas commented Oct 11, 2024

pibizza left a comment

[new-parser] Merge dev-new-parser to main #5989

[new-parser] Merge dev-new-parser to main #5989

Conversation

tkobayas commented Jun 11, 2024 • edited Loading

tkobayas commented Jun 11, 2024 • edited Loading

tkobayas Jun 11, 2024 • edited Loading

Choose a reason for hiding this comment

tkobayas commented Oct 2, 2024

tkobayas commented Oct 3, 2024 • edited Loading

tkobayas commented Oct 11, 2024

pibizza left a comment

Choose a reason for hiding this comment

tkobayas commented Jun 11, 2024 •

edited

Loading

tkobayas commented Jun 11, 2024 •

edited

Loading

tkobayas Jun 11, 2024 •

edited

Loading

tkobayas commented Oct 3, 2024 •

edited

Loading