TW5-Syntax 2.0.0 Released

[joshuafontany]

TW5 Syntax v2.0.0 HAS BEEN RELEASED!!!!!

https://marketplace.visualstudio.com/items?itemName=joshua-fontany.tw5-syntax

2.0.0

• scopeName text.html.tiddlywiki5
• Full re-write to model tiddlywiki5 wikitext as a text.html.derivative style sytanx.
• Updated syntax json version to current Tiddlywiki5 v5.3.4.
• Updated known keywords/grammar concepts in json repository.
• Imported full html5 syntax from tm-grammar, added tiddlywiki style attributes, etc.
• New objects from text.html.basic renamed as htmlwidget-* for easy diff/updates.
• Added snippets for modern tiddlywiki5 wikitext.
• Added full set of automated tests to ./tests. Run npm run test to run all tests.
• Added syntax highlihgting for tests files that marks the # ^ test.commment.with.expected.scopes lines appropriately.
• Added reference required syntaxes to devDependecies in package.json
• Fixed bugs with widget attribute syntax in base html tags
• Fixed syntax bugs with macros
• Fixed sytax bugs with block quotes
• Fixed syntax bugs with image links
• Fixed syntax bugs with links
• Fixed bugs with horizontal rules
• Fixed bugs with list quotes
• Fixed bugs with codeinline, codeblock, typedblock.
• Added .tw|.tw5|.tiddlywiki|.tiddlywiki5|text/vnd.tiddlywiki to #typedblock.
• Added tw|tw5|tiddlywiki|tiddlywiki5 to #codeblock

This Text Mate Grammar compatible syntax is twice as big as the official html5 syntax, literally <5K lines of regex to text.html.basic's 2,500~ish lines of regex. 😎

I would LOVE to get this into tiddlywiki somehow, so we can have full syntax highlighting INSIDE of the "code" edit-templates. I think the best option at this point would be to try to adapt TW5-Monaco.

I have forked that here to work on the concept over the next few weeks: https://github.com/joshuafontany/tw5-monaco

[joshuafontany]

Quickstart: [Howto] Use TW5-Syntax plugin with VSCode.dev

[pmario]

@joshuafontany

This is really great stuff, but the predefined snippets are a pain. I would like to contribute. I think I do understand the snippet syntax.

But I need some help with testing -- If I make changes to the snippets.json. How can I test the changes locally? before creating a PR

[pmario]

I did find it. I only needed to install the dev-dependencies and then run the extension code with F5 - Thx for the great work.

You can expect quite some snippent PRs shortly.

[Jermolene]

I would LOVE to get this into tiddlywiki somehow, so we can have full syntax highlighting INSIDE of the "code" edit-templates. I think the best option at this point would be to try to adapt TW5-Monaco.

I'm all in favour if that is a quicker route than porting the syntax highlighter to CodeMirror. It does look like TW5-Monaco needs quite a lot of work, for example to make it work offline with a single file wiki.

[joshuafontany]

Hi. After further research, it seems that Monaco Editor is not recommended for mobile use, and cannot be feature-complete when loaded from file:// uris. That's not going to work out for my use-cases, so we pivot and see how complex recreating this for Code-Mirror will be. Anyone reading this with CodeMirror syntax resources, feel free to drop them here.

[Jermolene]

Hi @joshuafontany I had a quick look at the gory details of the TW5 textmate grammar, it's quite an undertaking. It's very disappointing about Monaco, but I'm sure a CodeMirror port of TW5-syntax would be very well received were such a thing to be possible.

[btheado]

https://gk0wk.github.io/TW5-CodeMirror-Enhanced/ - this plugin includes code which provides syntax highlighting for TW5 syntax.

[Jermolene]

Thanks @btheado, and congratulations @Gk0Wk – I did not realise your plugin includes syntax highlighting.

It would be great to get some of those enhancements into the core plugins.

[linonetwo]

Note that @Gk0Wk drop the support of TW5-Codemirror-enhanced, becuase codemirror5's syntax parser is difficult to maintain. He was looking forward for Codemirror6 plugin, which is available here https://talk.tiddlywiki.org/t/release-codemirror6-plugin/8843/2 by @oeyoews , and it also include syntax highlighting too.

Do we still need to maintain old codemirror5 plugin inside the core repo?

[oeyoews]

I saw this syntax vscode plugin which seems to support more code snippets, but it's not ideal. For me, it would be better if we could implement a tiddlywiki LSP, but it seems to be a long way off.

[oeyoews]

https://github.com/oeyoews/tiddlywiki-codemirror6/blob/main/packages/lezer-tiddlywiki/src/tiddlywiki.grammar This is a link to some tw codemirror6 plugin highlighting syntax files. In theory, it can perfectly support all existing tw syntax grammars. However, I browsed all the relevant lezer documents and only wrote some simple syntax highlighting support, and have not yet fully implemented all syntax highlighting.

[joshuafontany]

@oeyoews I just found the Codemirror6 and Lezer docs. Awesome, will start reading up.

[oeyoews]

https://github.com/codemirror/legacy-modes/blob/main/mode/tiddlywiki.js

In fact, codemirror5 also has a corresponding tiddlywiki syntax highlighting file, but no one seems to have noticed it.

[pmario]

That's a TWclassic highlighter, that I did create a long time ago, to be used with TiddlySpace, which does not exist anymore.

It has never been updated to TW5 syntax because TW5-CM-enhanced was created. I personally do not need it anymore and I wanted to wait until CM6 was mature enough to have a closer look again.

The problem with CM6 for me is it's size.

[joshuafontany]

I guess that is the main concern with upgrading. I checked just the tid tiles on disk form the tiddlywiki5 repo. Code Mirror plugins total about 500kb. The Codemirror6 plugin tid file is just under 990Kb, and that may go up by quite a bit as we incorporate more syntax highlighting.

Is that worth including in the core? Is there a way we can "externalize" a large chuck of the core code so it is not repeated in multiple wikis? This is doable in base TW "core javascript on the side" in both server and stand-alone mode (all your HTML files have to be in the same dir as the core js file). That would avoid repeating the whole thing over and over in standalone wikis.

[pmario]

Is that worth including in the core? Is there a way we can "externalize" a large chuck of the core code so it is not repeated in multiple wikis?

It is possible to externalize both the core and plugins using cdruan's plugin: https://cdruan.github.io/tw-external-js/ -- It needed to be a non-core plugin. (The full story about the PR can be found at GH - To get the whole picture you need to expand the 42 hidden items in the story-line)

I think CM6 will be a good fit for the new MWS-plugin Jeremy is developing at the moment. The plugin could be stored in a separate bag. So users can create several recipes, that show the same content, but with different plugins active, without increasing the complexity too much.

Once the core and the plugins are externalized, they can be cached by the browser for fast reload over low-bandwidth internet connections.

Also saving will be much faster, since only the "content part" of the wiki has to be saved. The core and the plugins can stay as they are

[pmario]

@Jermolene -- Could you move this issue into the GitHub Discussions section

[Jermolene]

Thanks @pmario