Feature Request: Semantic Highlight

[rubenporras]

Hi,

I am looking at support for Semantic highlighting is an addition to syntax highlighting (more or less as described in language extensions - semantic highlight guide)). As described in that link, the idea is that the editor applies the highlighting from semantic tokens by asking a language sever on top of the highlighting from grammars.

As far as I see, there is no support for semantic highlight in TM4E and also not in LSP4E.

I would like to contribute such a feature to TM4E or/and LSP4E but I am unsure about how to start and what would be the desired architecture.

Would the best way to implement it to extend TMPresentationReconciler with an extension point to register a semantic token provider (based on a new interface, e.g. ISemanticTokensProvider), and then implement that interface in LSP4E by issuing requests to a Language Server? I think that LSP4E and TM4E do not depend on each other, so I am not sure where that interface would be.

Provided we would have that in place, would it be possible to apply first syntax highlighting and then later semantic highlight (it is to be expected that the later is slower).

I would be very happy to have some guidance with the feature request.

Thanks

[mickaelistria]

Would the best way to implement it to extend TMPresentationReconciler with an extension point to register a semantic token provider (based on a new interface, e.g. ISemanticTokensProvider), and then implement that interface in LSP4E by issuing requests to a Language Server? I think that LSP4E and TM4E do not depend on each other, so I am not sure where that interface would be.

I believe it can be a good approach. TM4E is already kind of organized that way I believe, with one part that does compute/provides token and the presentation reconcilier that turns token into styles. If we can make the grammar-based tokenizer implement such an ITMTokenProvider and be registered as 1 extension to TM4E then we could easily add additional implementations (eg LSP4E) via such extension point.

[angelozerr]

I think semantic highlight must be supported in lsp4e project since it is a part of lsp. No?

[mickaelistria]

I'd rather avoid having 2 complementary project managing things like themes and so on. So ideally, LSP4E would rely on TM4E. It would retrieve the tokens, but TM4E would apply highlighting according to other capabilities it already provides.

[angelozerr]

I think it can be a good idea that tm4e follows the same idea than vscode which provides https://code.visualstudio.com/api/language-extensions/semantic-highlight-guide#semantic-token-provider

and we can follow the same idea to map LSP semantic tokens and TextMate scope like

https://code.visualstudio.com/api/language-extensions/semantic-highlight-guide#predefined-textmate-scope-mappings

In other words, TM4E should perhaps provides an extension point DocumentSemanticTokensProvider like vscode.DocumentSemanticTokensProvider that LSP4e could implement :

• with https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#textDocument_semanticTokens to support semantic hihlighting
• https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#textDocument_documentLink to underline (as hyperlink like vscode does) the document links.

[rubenporras]

That looks like a good idea. Would this interface be defined in TM4E? I am right that then we would need a dependency from LS4E to TM4E and that would be fine?

[angelozerr]

That looks like a good idea. Would this interface be defined in TM4E?

It is just a suggestion.

I am right that then we would need a dependency from LS4E to TM4E and that would be fine?

If we follow this idea, yes.

But to be honnest with you, it is very hard for me to be sure that it will work like this. Perhaps we don't need this API?

I suggest that we go step by step and perhaps implement a new LSPPresentationReconciler which extends IPresentationReconciler which consumes documentLink and underline the links. It is just an idea...

[rubenporras]

Thanks, I would also go step by step. I am planning as well to have basic renconciler which does just very basic semantic highlight, actually what you suggest with a very simple IPresentationReconciler . I am doing a prototype to understand how that would work, since all these APIs are new to me, my reconciler colors the file green :). I hope I have some draft for next week that actually ask a language server for the tokens and does some coloring according to that.

[rubenporras]

My current work (soon to be updated) is in eclipse/lsp4e#253, we can close this issue.