I wrote a TOML parser in Teal

[FourierTransformer]

Teal was an absolute joy to use for writing my TOML parser: tinytoml. For a medium scale project that needed to keep track of a lot of different variables, Teal made my life super easy whenever refactoring/renaming/removing variables or chunks of code.

I wrote the whole thing using the Teal vim plugin. It was very satisfying to see the error gutter disappear when saving after fixing an issue.

Small annoyances

• Sometimes I would declare a table in a teal record, but forget to initialize the table, and get a nil pointer error later on:

  local record MyThing
      a: {string:boolean}
  end
  
  local thing: MyThing = {}
  thing.a["word"] = false -- ends up with a nil error
  

  It would be really nice if something would indicate I didn't initialize a. Not sure if this would be a tl/linter/lsp/vim fix.

• When I was doing datetime validation, I had to take a string, tonumber it to a number, and then use that as a lookup in a table. I ended up just doing an as integer and it was fine. Not sure if tonumber could be made generic and roll with the output type (or if that is a bad idea).

• :find() captures. I would imagine accurately doing the bracket counting is not easy, but for folks who do a lot of pattern matching, it would be amazing.

  value:find("^((%d%d%d%d)%-(%d%d)%-(%d%d)[Tt ](%d%d):(%d%d):(%d%d))(.*)$")
  as (integer, integer, string, string, string, string, string, string, string, string)
  

• no syntax highlighting in the various Git hostings (I did see Syntax highligthing on github and gitlab #601) or in some text editors.

Maybe just me annoyances

• extendable enums or enum subsets
  Basically some way to indicate TomlConversionType is a subset of TomlType

  local enum TomlConversionType -- used in options to configure custom parsing, but needs to be a type of TomlType
      "datetime-local"
      "datetime"
      "date-local"
      "time-local"
  end
  
  local enum TomlType
      -- array and inline-table are also _my_ types that _shouldn't_ be extended as a TomlConversionType,
      -- this is maybe a different concept altogether
      "array"
      "inline-table"
  
      "string"
      "integer"
      "float"
      "bool"
      "datetime"
      "datetime-local"
      "date-local"
      "time-local"
  end
  

• Not a problem most folks would run into, but when writing a parser for a language that has the concepts of an array or a dictionary that both map to a Lua table, I ended up just having to choose one representation and then force the other with as a bunch. I was also dealing with dynamically created tables of either type, and I wasn't sure the best way to represent deeply nested tables. I did sort've try an interface with kind, but was a bit weary of adding extra fields to the output since I wanted the output to just be a pure Lua table without anything extra in it.

Overall

Overall, I really enjoyed using Teal on this and will definitely reach for it in the future! This was a "0-dependency" Lua project, so I didn't have to worry too much about bringing in types from teal-types. Is that handled by cyan per chance? I didn't really play with cyan since it was just the one file I was compiling.

If you guys want me to create some issue out of the above, LMK where (I feel like some are tl, some elsewhere?). I have some experience with parsers, and tried my hand at code completion a long time ago (before LSPs/tree-sitters were a thing...), and would be willing to help out a bit.

[FourierTransformer]

I think most of these things are tl related (except the syntax highlighting), so I think I'll create some issues over the next couple of days:

• nil table in record
• generic tonumber?
• find captures
• enum subsets

I might try my hand at updating the tmbundle.

[hishamhm]

Thank you for your write-up! It's super encouraging to get feedback like this!

Here are some comments on the items you brought up:

It would be really nice if something would indicate I didn't initialize a. Not sure if this would be a tl/linter/lsp/vim fix.

Anything related to nil checks is a bit tricky. Nil strictness is something I want to add to the language (that is, adding option types), but even then, there's limits. If the initialization of the table and subtables is done all at once, nil strictness would suffice. But if you have a field that's meant to be initialized later (and typed with a ?-optional type, say, a: {string:boolean}?, assuming that was added to Teal), we can't always guarantee that it is initialized at the time of usage. Some best-effort check could be done by a linter tool, possibly. But I don't think we can always catch this.

generic tonumber?

If you know your number is an integer, you can use math.tointeger instead!

find captures

Ah, this looks like a regression! In Teal 0.15 the type signature of string.find was declared as variadic (effectively string.find(string, string, ? integer, ? ,boolean): integer, integer, string... -- note the ...) — simply making this small change should already make the captures instantly typeable.

Of course, it's not exactly the same as counting the captures, but that would require some special-casing in the compiler to give this function more magical behavior (that's not out of question, but I think making the return type variadic again would already address the main annoyance!).

enum subsets

That's an interesting one. I don't know for sure how common this need is, either. Definitely worth opening as a feature request — we might get more feedback on it!

[FourierTransformer]

I did not know about math.tointeger! Since I'm targeting a wide swath of Lua versions, I ended up with something like:

local _tointeger = math.tointeger or tonumber as function(string): integer

Also, from a library perspective, having the option to have a pure Lua compatibility layer (and errors for things that just can't be compatible) would be interesting. That one feels like a bit of a tall order.

[hishamhm]

This was a "0-dependency" Lua project, so I didn't have to worry too much about bringing in types from teal-types. Is that handled by cyan per chance? I didn't really play with cyan since it was just the one file I was compiling.

Ah, that's not handled automatically yet. @euclidianAce and I were discussing the other day about where exactly would that best fit, if in cyan (being a build concern) or luarocks (being a dependency concern). I guess with modern tools like cargo and the npm ecosystem blurring the lines between build tools and package managers, that becomes a bit confusing!

[hishamhm]

I have some experience with parsers, and tried my hand at code completion a long time ago (before LSPs/tree-sitters were a thing...), and would be willing to help out a bit.

There are various projects in this space here in the teal-language GitHub org: vim-teal, vscode-teal, teal-language-server... I think it would be nice to have them eventually share a backend (which would be teal-language-server, I assume!)

Tooling is essential for a modern language, so every bit of help in that regard is super welcome!