Teal and DSL support

[lenscas]

So, recently a feature request was made to allow a tsx like Syntax to teal. This sparked a bit of a discussion about tsx in Teal as well about DSL's in general.

However, as that issue is closed and specific about tsx I decided to make a new discussion about it so more people can weight in.

So, first question first: Should Teal have a way to write custom DSL's?
Second, what would it look like?

These are some examples that I know of:

Going the way of Rust's proc macros is probably overkill and basically kills editor support. However, being just functions that take in a TokenStream and return one, they are very powerful.

Rust's macro_rules! macros are a bit more like it. You give them some patterns and what code to generate using said patterns. However, they end up only being able to do basic transformations.

F#'s Computation expressions are sometimes used as a way to create DSL's. These are actually nice as any value can be used but they don't really create a new syntax.

C#'s Linq expressions suffer from a similar problem as they also can't create new syntax.

I am also not sure if any of these would actually fit in Teal as well, as it isn't exactly something that Lua offers. Having said that, something that I realized later in the original discussion is that it is possible to put the DSL support in Cyan rather than the Teal language.

This can be done by giving Cyan a list of file name patterns and a transform function. When building, Cyan will look for files that fit one of the given pattern and if found gives it to the corresponding transform function. This transform function then will spit out Teal code that is used instead.

This would allow people to write a version of tsx for Teal as well as for other (custom) files like Json schema, sql files, you name it. It would also keep the complexity of DSL's out of Teal itself.

Thoughts on the matter? Should Teal support DSL's? What should they look like? Any other thoughts?

[tooolbox]

Should Teal have a way to write custom DSL's?

Having thought this over, my opinion is that it seems a little out of scope for Teal itself. Teal as "typed Lua" is both sufficiently valuable and sufficiently focused.

That maybe seems contrary because I suggested JSX support in Teal, i.e. .tlx. Well, my reasoning had been that tsc supports .tsx, and I look at tsc as the spiritual godfather of tl. As someone who loves typed JSX for constructing markup, but hates to drag Node around, the concept of .tlx was very appealing. That said, Lua is not JS and there is no existing weight of JSX usage in the ecosystem, so I understand the aversion to adding something like that to Teal.

On the flipside, Teal's existence as a preprocessor in the Lua ecosystem (which otherwise has no culture of such things, that I'm aware of) presents something of an opportunity, in the way that Babel presented an opportunity for JS to do all sorts of transformations on code and related files. Somehow leveraging its existence to support wider custom transform opportunities seems interesting.

Perhaps, the simpler the DSL system, the more it would seem germane to Teal?

I am also not sure if any of these would actually fit in Teal as well, as it isn't exactly something that Lua offers. Having said that, something that I realized later in the original discussion is that it is possible to put the DSL support in Cyan rather than the Teal language.

This seems workable and uncontroversial. From what I understand, Cyan is a build system, and build systems have a history of executing arbitrary commands as part of the build process. It seems easy to have Cyan call out to your_transformer with a list of files before running the Teal typecheck & transform.

The only problem I see is editor support. Making it part of Cyan means that your Teal linter is unaware of your wacky syntax. You do cite some macro types that nuke editor support, but since Teal's mission is to add compile-time and even dev-time safety to Lua, adding a macro system that removes safety seems contrary. You also are likely to lose syntax highlighting with any slight changes.

These are some examples that I know of:

I think, for the sake of those who aren't familiar with Rust, F# and Linq, it would be helpful to give each one a concrete example in code i.e. what syntax is transformed into what syntax. And how, perhaps, would these affect Teal, and its subsequent transform to Lua? We can do our own research, but since you are familiar, it would be nice to have a summary on hand.

Also, based on your concept of these options, which ones would let me write .tlx files? 😉

Any other thoughts?

I think the priority of this should be below quality-of-life improvements such as full autocompletion support in [editor].

---

Sidebar: .tlx through Cyan

Let me just walk through this thought process:

1. Let's say I wrote a tlxc which converts .tlx to .tl.
2. Then I had Cyan run tlxc on .tlx files before passing them to Teal.
3. The building process would work great.
4. What about editors?
5. So I modify the syntax highlighting files to support my nonsense.
6. And then I made a linter which runs tlxc and then tl and underlines text in my editor based on file and line number.
7. Maybe the linter runs Cyan which knows how to turn my .tlx files into Lua. Perhaps cyan lint or something?
8. Or rather, Cyan knows how to typecheck my .tlx files, i.e. it knows there's a preprocessing step.
9. All this is working, but of course, the line/char numbers reported by Cyan are wrong.
10. Something has to know how to map positions.
    1. It could be tl, but that would add complexity to a script whose direct users would never need such a feature.
    2. It could be tlxc but then source mapping would need to be handled by every independent transformer plugin.
    3. Could it be Cyan? Thinks about it.
    4. Well, the fact is, tlxc must emit a sourcemap because it's doing the transform. No other tool has knowledge of what language constructs relate to what. But if we're doing cyan lint, Cyan could capture the sourcemap from tlxc, get the typecheck output from Teal, and remap the char/line positions before returning it to the user. Actually, that would probably be an important option even for regular building because if you have errors you want to know where the problem originated in the source code.

Fin.

[lenscas]

Don't have time for a full reply, so I will just reply to the one thing I find VERY important to make clear

What about editors?

That is a BIG problem for every DSL system. JSX got around this by becoming popular enough for tools to support natively. It using its own extension helps these tools to know when to enable this support. F#'s Computation expressions and C#'s Linq expressions solve this by not allowing new syntax to be created.

Rust's macros however suffer BIG time on this. Macro_rules macro's help the IDE out a bit by being very transparent of the kind of patterns it can process. However as it is hard for the editor support to understand what ends up happening with those patterns it can't really create suggestions.

procedural macro's are even worse as they are even less transparent in the code that they generate AND take in the tokenstream itself rather than working on patterns. So, IDE's can't even properly "see" what syntax a proc macro takes.

I think that the transformer functions system I talked about has this problem a bit less as the entire file is affected by this new syntax. That should make it easier for IDE tooling to know when to enable what and would also make it possible to reuse exciting tooling when dealing with formats that have been standardized already (For example when using a transform function to turn json schema's into Teal code. In that case it would be possible to use the already existing tooling when working with the json schema files.)

[lenscas]

Also, based on your concept of these options, which ones would let me write .tlx files? wink

Rust's proc macros would allow you to do so, but editor support would suck (look how much the current IDE tooling for Rust is struggling with it right now, despite Rust analyzer being the best language server I ever used) and they would still be normal .tl files. However, conceptually they are very easy to understand as a proc macro is just a normal function that takes in a TokenStream and returns a new one. Yew is a web framework in Rust that decided to make such a macro https://yew.rs/docs/concepts/html/introduction

A system that mimics Rust's macro_rules macro's might be able to do it. Depends on what patterns you can ask for and what code you can generate based on those patterns. I am not aware of anyone using Rust's macro_rules macros to make a jsx like macro though. Which probably means that with the current way they work in Rust that this is just not possible

F#'s Computation expression can't create new syntax. Though people have made some template systems in it: http://www.fssnip.net/hf/title/DSL-for-constructing-HTML

C#'s Linq expression system also can't create new syntax and I am not aware of anyone even trying to make a templateting system out of it. EFCore uses it as the query builder part.

Pushing the DSL part to Cyan through a system to register transformer functions is the only one that allows you to create "real" .tlx files. With that as their own extension and tooling.

The only problem I see is editor support. Making it part of Cyan means that your Teal linter is unaware of your wacky syntax. You do cite some macro types that nuke editor support, but since Teal's mission is to add compile-time and even dev-time safety to Lua, adding a macro system that removes safety seems contrary. You also are likely to lose syntax highlighting with any slight changes.

So far I am unaware of any system that neither hurts editor support and allows registering custom syntax. For the IDE to properly support the new syntax it needs to both know how this syntax works and what everything means. That is almost impossible.

F#'s Computation expressions and C# Linq expressions side step this problem by both not creating new syntax and by involving the type system.

However, in none of these cases do you lose safety as the compiler is still fully aware of the code that got generated by the DSL system and can act accordingly.

think, for the sake of those who aren't familiar with Rust, F# and Linq, it would be helpful to give each one a concrete example in code i.e. what syntax is transformed into what syntax. And how, perhaps, would these affect Teal, and its subsequent transform to Lua? We can do our own research, but since you are familiar, it would be nice to have a summary on hand.

Sure, when I have more time I will probably add some examples. However, though I am a big consumer of these systems I tend to not write them myself. The one system I have used quite a bit is Rust's macro_rules system, but even there I try to keep it pretty basic.

Something has to know how to map positions.

That one is simple. Rather than registering just a single function to Cyan to do the transformation you could register a module. This allows said transformer to do some initializing logic but would also allow it to expose more advanced functions, like one that takes the old file and the generated Teal code, as well as the span of an error message (And maybe the error message itself) and does the mapping. This keeps the sourcemap logic out of Cyan, which I expect to be rather complex if it needs to support any kind of syntax.