[FR] improve default syntax highlighting for vignettes

[kevinushey]

When a vignette is rendered, any embedded code is also highlighted. However, it seems like the default highlighter is a bit limited in a few ways:

• Brackets (e.g. ( and )) are not tokenized, and so they cannot be styled;
• Arguments names (and a following =) are tokenized as a single token, and so cannot be styled independently. (https://invent.kde.org/frameworks/syntax-highlighting/-/blob/master/data/syntax/r.xml#L58)
• := is not tokenized as a single operator; in particular, the = following a : is tokenized as an error.

(please ignore the class names here; those are RStudio-specific experiments)

My understanding is that this highlighting / tokenization is performed by pandoc, with some extra highlight rules provided here:

https://github.com/rstudio/rmarkdown/blob/de02c926371fdadc4d92f08e1ad7b77db069be49/inst/rmarkdown/templates/html_vignette/resources/vignette.css#L187-L202

Could this be changed / improved by changing the highlight engine used by pandoc for vignettes?

[cderv]

My understanding is that this highlighting / tokenization is performed by pandoc

Yes Pandoc has a syntax highlighter which does the tokenization, and then brings some highlighting style that we can control with highlight argument of the output format function. html_vignette defaults to pygments.

some extra highlight rules provided here:

I don't think those highlight rules still applies with recent Pandoc versions. span with token class are inside other Spans inside a div to code > span.fu does not match pandoc highlighter result anymore. Currently, only the CSS bring by Pandoc is used.

Could this be changed / improved by changing the highlight engine used by pandoc for vignettes?

Changing / improving the tokenization means providing a new KDE SYNTAX XML file for Pandoc to use. Either by updating upstream or providing our own using --syntax-definition=FILE.

I've done the former already to provide support for new R pipe |> ( https://invent.kde.org/frameworks/syntax-highlighting/-/merge_requests/295). I can certainly try to do that again.
We've also done the latter in Quarto (https://github.com/quarto-dev/quarto-cli/tree/main/src/resources/pandoc/syntax-definitions) to provide custom support (especially markdown). In rmarkdown, we're still thinking about backporting the new R syntax for older pandoc (>2.15) - #2290

So overall, we can try improve the tokenization along the line of your suggestion. The KDE syntax file definition logic is not the easiest though so that is not straightforward.