(Bug?) Last letters of first keywords are sometimes unhighlighted

[TheWitheredStriker]

I'm not really sure where else to ask this, since I don't think it's a bug that, conveniently, all grammars I'm about to talk about share. Especially not since it only happens every so often.

Essentially, I've stumbled upon code files where the first keyword present (usually an import statement), or even the first few if they're on the same line, is highlighted all the way until the last letter, which is white (or another colour it shouldn't be). Some examples:

• C++ (#include)
• VB.NET (Public, Class)
• F# (module)

I would find it pretty strange if, coincidentally, C++'s, VB.NET's, and F#'s grammar all had the same issue, so I don't think it's because of the grammars, though I could be wrong. It doesn't always happen either, see this VB file and this C++ file (as well as this one from the same repo as the file where the last letter isn't highlighted).

I have no idea what causes this, so I figured here would be the best to place ask around. Sorry if this is the wrong place for this!

[lildude]

Oh yes 🧐. It looks like the highlighting of the first line is always shifted left by one char… this is most obvious with the F# example. This is most definitely not the grammars nor anything within Linguist'a control. This appear to be either a problem with the syntax highlighting engine or the presentation implementation of the results from the syntax highlighter (I suspect the latter).

Please use the "Contact GitHub" link in the bottom of one is those pages to open a support ticket. They'll be able to engage the correct engineering team. Feel free to reference this discussion too.

[TheWitheredStriker]

Thanks for the reply! I'll be sure to contact GitHub.

[jean-baka]

Hi @lildude @TheWitheredStriker ! I see a bug that could be related, but I'm not sure. I have seen this in R programming, where the first letter of a block immediately following a line of comment, without any blank line in between, lacks highlighting.

This is more or less everywhere in my code, and you have a good example here at lines 55 and 167 and 189. Note that this behaviour doesn't occur after an empty line (e.g. lines 110 and 122).

Have you ever seen that reported? Is this linked to the same as pointed at by @lildude ? Thanks!

[lildude]

Yes, this is the same issue and was asked about specifically for R last week in #6704

[jean-baka]

Alright, thanks for the speedy answer!

[lildude]

Just an update on this old issue. I've followed up this internally and it appears to be down to our handling of the byte order mark (BOM) at the beginning of all of these files:

➜ file babel.fs
babel.fs: Unicode text, UTF-8 (with BOM) text
➜ head -1 babel.fs
module Fable.Helpers.Babel
➜ head -c3 babel.fs | hexdump -C
00000000  ef bb bf                                          |...|
00000003
➜

Until the highlighting correctly handles this, you can resolve it yourself by removing the BOM using something like dos2unix:

➜ dos2unix babel.fs
dos2unix: converting file babel.fs to Unix format...
➜ file babel.fs
babel.fs: ASCII text
➜ head -1 babel.fs
module Fable.Helpers.Babel
➜ head -c3 babel.fs | hexdump -C
00000000  6d 6f 64                                          |mod|
00000003
➜