Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update textmate support #7971

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

matthiasblaesing
Copy link
Contributor

It was observed, that some grammars were not accepted by the textmate grammar support. One such example was a C#-grammar. Updating textmate4eclipse, jcondings and joni fixes the issue.

Following the updates of the binaries, usesites of tm4e needed to be updated to accommodate for changed api in tm4j.

Unittests for textmate.lexer were fixed prior to update. For this the LayerBuilder needed to be fixed, as it failed for resources in the default package, which was used by the test. Slight changes to the unittests were necessary to follow the new API.

@matthiasblaesing matthiasblaesing added Java [ci] enable extra Java tests (java.completion, java.source.base, java.hints, refactoring.java, form) LSP [ci] enable Language Server Protocol tests C/C++ enterprise [ci] enable enterprise job ci:dev-build [ci] produce a dev-build zip artifact (7 days expiration, see link on workflow summary page) Markdown labels Nov 17, 2024
@matthiasblaesing matthiasblaesing added this to the NB25 milestone Nov 17, 2024
@matthiasblaesing matthiasblaesing force-pushed the update-textmate branch 2 times, most recently from 167883c to 17a6a99 Compare November 17, 2024 20:24
@Chris2011
Copy link
Contributor

@matthiasblaesing Unfortunately I dunno whether I did smth wrong here. I downloaded your build and tried it out. I got this exception.

org.joni.exception.SyntaxException: invalid pattern in look-behind
	at org.joni.ScannerSupport.newSyntaxException(ScannerSupport.java:163)
	at org.joni.Analyser.setupLookBehind(Analyser.java:1404)
	at org.joni.Analyser.setupTree(Analyser.java:2004)
	at org.joni.Analyser.setupTree(Analyser.java:1825)
	at org.joni.Analyser.setupTree(Analyser.java:1945)
	at org.joni.Analyser.compile(Analyser.java:110)
	at org.joni.Regex.<init>(Regex.java:169)
	at org.joni.Regex.<init>(Regex.java:148)
	at org.eclipse.tm4e.core.internal.oniguruma.OnigRegExp.parsePattern(OnigRegExp.java:91)
	at org.eclipse.tm4e.core.internal.oniguruma.OnigRegExp.<init>(OnigRegExp.java:73)
Caused: org.eclipse.tm4e.core.TMException: Parsing regex pattern "(?x)
  (?<=\S\s*)
  (\/\/\s*)
  (
    \[(?:
        (?:-?\d+(?:.\d)*)(?:\s*:\s*-?\d+(?:.\d)*){0,2} |
        (?:(?:[^:,]+:)?[^:,]+,)*(?:(?:[^:,]+:)?[^:,]+)
      )
    \]
  )
  [ \t]*$\n?" failed with org.joni.exception.SyntaxException: invalid pattern in look-behind
	at org.eclipse.tm4e.core.internal.oniguruma.OnigRegExp.<init>(OnigRegExp.java:78)
	at org.eclipse.tm4e.core.internal.oniguruma.OnigRegExp.<init>(OnigRegExp.java:62)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:720)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575)
	at java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
	at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616)
	at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622)
	at java.base/java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627)
	at org.eclipse.tm4e.core.internal.oniguruma.OnigSearcher.<init>(OnigSearcher.java:32)
	at org.eclipse.tm4e.core.internal.oniguruma.OnigScanner.<init>(OnigScanner.java:32)
	at org.eclipse.tm4e.core.internal.rule.CompiledRule.<init>(CompiledRule.java:37)
	at org.eclipse.tm4e.core.internal.rule.RegExpSourceList.compile(RegExpSourceList.java:76)
	at org.eclipse.tm4e.core.internal.rule.RegExpSourceList.compileAG(RegExpSourceList.java:83)
	at org.eclipse.tm4e.core.internal.rule.IncludeOnlyRule.compileAG(IncludeOnlyRule.java:57)
	at org.eclipse.tm4e.core.internal.grammar.LineTokenizer.matchRule(LineTokenizer.java:312)
	at org.eclipse.tm4e.core.internal.grammar.LineTokenizer.matchRuleOrInjections(LineTokenizer.java:326)
	at org.eclipse.tm4e.core.internal.grammar.LineTokenizer.scanNext(LineTokenizer.java:137)
	at org.eclipse.tm4e.core.internal.grammar.LineTokenizer.scan(LineTokenizer.java:128)
	at org.eclipse.tm4e.core.internal.grammar.LineTokenizer.tokenizeString(LineTokenizer.java:561)
	at org.eclipse.tm4e.core.internal.grammar.Grammar._tokenize(Grammar.java:342)
	at org.eclipse.tm4e.core.internal.grammar.Grammar.tokenizeLine(Grammar.java:259)
	at org.netbeans.modules.textmate.lexer.TextmateLexer.nextToken(TextmateLexer.java:76)
	at org.netbeans.lib.lexer.LexerInputOperation.nextToken(LexerInputOperation.java:191)
	at org.netbeans.lib.lexer.inc.IncTokenList.tokenOrEmbeddingImpl(IncTokenList.java:170)
	at org.netbeans.lib.lexer.inc.IncTokenList.tokenOrEmbedding(IncTokenList.java:165)
	at org.netbeans.api.lexer.TokenSequence.moveNext(TokenSequence.java:458)
	at org.netbeans.modules.textmate.lexer.SyntaxHighlighting$TSInfo.moveNextToken(SyntaxHighlighting.java:785)
	at org.netbeans.modules.textmate.lexer.SyntaxHighlighting$HSImpl.moveTheSequence(SyntaxHighlighting.java:482)
	at org.netbeans.modules.textmate.lexer.SyntaxHighlighting$HSImpl.moveNext(SyntaxHighlighting.java:411)
	at org.netbeans.modules.editor.lib2.highlighting.DirectMergeContainer$Wrapper.fetchNextHighlight(DirectMergeContainer.java:600)
	at org.netbeans.modules.editor.lib2.highlighting.DirectMergeContainer$HlSequence.updateMergeVars(DirectMergeContainer.java:337)
	at org.netbeans.modules.editor.lib2.highlighting.DirectMergeContainer$HlSequence.nextMerge(DirectMergeContainer.java:303)
	at org.netbeans.modules.editor.lib2.highlighting.DirectMergeContainer$HlSequence.moveNext(DirectMergeContainer.java:234)
	at org.netbeans.modules.editor.lib2.highlighting.HighlightsReader.readUntil(HighlightsReader.java:59)
	at org.netbeans.modules.editor.lib2.view.HighlightsViewFactory.restart(HighlightsViewFactory.java:168)
	at org.netbeans.modules.editor.lib2.view.ViewBuilder$FactoryState.init(ViewBuilder.java:1299)
	at org.netbeans.modules.editor.lib2.view.ViewBuilder.createViews(ViewBuilder.java:699)
	at org.netbeans.modules.editor.lib2.view.ViewBuilder.createReplaceRepaintViews(ViewBuilder.java:660)
	at org.netbeans.modules.editor.lib2.view.ViewUpdates.reinitAllViews(ViewUpdates.java:189)
	at org.netbeans.modules.editor.lib2.view.DocumentViewOp.checkViewsInited(DocumentViewOp.java:621)
	at org.netbeans.modules.editor.lib2.view.DocumentView.getPreferredSpan(DocumentView.java:226)
	at java.desktop/javax.swing.plaf.basic.BasicTextUI$RootView.getPreferredSpan(BasicTextUI.java:1457)
	at java.desktop/javax.swing.plaf.basic.BasicTextUI.getPreferredSize(BasicTextUI.java:953)
	at java.desktop/javax.swing.JComponent.getPreferredSize(JComponent.java:1725)
	at java.desktop/javax.swing.JEditorPane.getPreferredSize(JEditorPane.java:1401)
	at org.netbeans.modules.editor.NbEditorUI$LayeredEditorPane.getPreferredSize(NbEditorUI.java:454)
	at java.desktop/javax.swing.ScrollPaneLayout.layoutContainer(ScrollPaneLayout.java:799)
	at com.formdev.flatlaf.ui.FlatScrollPaneUI$FlatScrollPaneLayout.layoutContainer(FlatScrollPaneUI.java:563)
	at java.desktop/java.awt.Container.layout(Container.java:1541)
	at java.desktop/java.awt.Container.doLayout(Container.java:1530)
	at java.desktop/java.awt.Container.validateTree(Container.java:1725)
	at java.desktop/java.awt.Container.validateTree(Container.java:1734)
	at java.desktop/java.awt.Container.validateTree(Container.java:1734)
	at java.desktop/java.awt.Container.validateTree(Container.java:1734)
	at java.desktop/java.awt.Container.validateTree(Container.java:1734)
	at java.desktop/java.awt.Container.validateTree(Container.java:1734)
	at java.desktop/java.awt.Container.validateTree(Container.java:1734)
	at java.desktop/java.awt.Container.validateTree(Container.java:1734)
	at java.desktop/java.awt.Container.validateTree(Container.java:1734)
	at java.desktop/java.awt.Container.validate(Container.java:1660)
	at org.netbeans.core.windows.view.ui.DesktopImpl$LayeredLayout.layoutContainer(DesktopImpl.java:529)
	at java.desktop/java.awt.Container.layout(Container.java:1541)
	at java.desktop/java.awt.Container.doLayout(Container.java:1530)
	at java.desktop/java.awt.Container.validateTree(Container.java:1725)
	at java.desktop/java.awt.Container.validateTree(Container.java:1734)
	at java.desktop/java.awt.Container.validateTree(Container.java:1734)
	at java.desktop/java.awt.Container.validateTree(Container.java:1734)
	at java.desktop/java.awt.Container.validateTree(Container.java:1734)
	at java.desktop/java.awt.Container.validate(Container.java:1660)
	at java.desktop/javax.swing.RepaintManager$3.run(RepaintManager.java:757)
	at java.desktop/javax.swing.RepaintManager$3.run(RepaintManager.java:755)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:399)
	at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:86)
	at java.desktop/javax.swing.RepaintManager.validateInvalidComponents(RepaintManager.java:754)
	at java.desktop/javax.swing.RepaintManager$ProcessingRunnable.run(RepaintManager.java:1897)
	at java.desktop/java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:318)
	at java.desktop/java.awt.EventQueue.dispatchEventImpl(EventQueue.java:771)
	at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:722)
	at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:716)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:399)
	at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:86)
	at java.desktop/java.awt.EventQueue.dispatchEvent(EventQueue.java:741)
	at org.netbeans.core.TimableEventQueue.dispatchEvent(TimableEventQueue.java:136)
[catch] at java.desktop/java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:203)
	at java.desktop/java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:124)
	at java.desktop/java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:113)
	at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:109)
	at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
	at java.desktop/java.awt.EventDispatchThread.run(EventDispatchThread.java:90)

For this example code:

// door_stopper_simple ();

module door_stopper_simple ()
{
    height = 20  ;
    length = 125 ;
    width= 40 ;
    half_width  = width / 2;
    
    borders = 10; // 1 cm on each side

    top_cube_length = length - 4 * borders;
    top_cube_width  = width - 2 * borders;
    top_cube_height = height ;  // make it big enough
    top_z = height/2 + 4 ; // adjust manually ;)

    tip_cut_pos = length/2 - 0.5;

    difference() {
	translate ([-length/2.0, 0, 0]) {
    	    polyhedron (	
	    		points = [[0, -half_width, height], [0, half_width, height], [0, half_width, 0], [0, -half_width, 0], [length, -half_width, 0], [length, half_width, 0]],
			triangles = [[0,3,2], [0,2,1], [3,0,4], [1,2,5], [0,5,4], [0,1,5],  [5,2,4], [4,2,3], ]);
	}

	// top inset
	translate ([-1 * borders, 0, top_z]) {
	    # cube ([top_cube_length,
		     top_cube_width,
		     top_cube_height],
		    center=true) ;
	}
	// cut of the tip a bit

	translate ([tip_cut_pos, 0, 0]) {
	    # cube ([20,
		     width + 1,
		     height],
		    center=true) ;
	}
    }
}

door_stopper_simple ();

@matthiasblaesing
Copy link
Contributor Author

@Chris2011 thanks for the test. Yeah, you hit a different limitation than I did. The problem is the construct in the second line of the regexp:

(?x)
  (?<=\S\s*)
  (\/\/\s*)
  (
    \[(?:
        (?:-?\d+(?:.\d)*)(?:\s*:\s*-?\d+(?:.\d)*){0,2} |
        (?:(?:[^:,]+:)?[^:,]+,)*(?:(?:[^:,]+:)?[^:,]+)
      )
    \]
  )
  [ \t]*$\n?

It has an unlimited length, which is not yet supported by joni. Oniguruma added support for it in 2020, but it seems, that noone ported that.

The grammar can be trivially be fixed. Change line 33 to:

      "match": "(?x)\n (\\/\\/\\s*)\n  (\n    \\[(?:\n        (?:-?\\d+(?:.\\d)*)(?:\\s*:\\s*-?\\d+(?:.\\d)*){0,2} |\n        (?:(?:[^:,]+:)?[^:,]+,)*(?:(?:[^:,]+:)?[^:,]+)\n      )\n    \\]\n  )\n  [ \\t]*$\\n?",

And you should be good. At this point NetBeans does not use the extractacted values and the look-behind only ensures, that this match requires a variable before it. For the coloring it does not matter.

@Chris2011
Copy link
Contributor

I thought that the regex will a problem here. I will change the regex and will try it again and let you know. Thx.

@Chris2011
Copy link
Contributor

After changing the mentioned line, it is working now. Thx a lot.

@matthiasblaesing
Copy link
Contributor Author

I intent to merge this next week, sometime before christmas, so if you want to intercept, please do so now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C/C++ ci:dev-build [ci] produce a dev-build zip artifact (7 days expiration, see link on workflow summary page) enterprise [ci] enable enterprise job Java [ci] enable extra Java tests (java.completion, java.source.base, java.hints, refactoring.java, form) LSP [ci] enable Language Server Protocol tests Markdown
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants