Feedback Wanted! The state of the CLI

[euclidianAce]

Hey all, there's been quite a bit of discussion about the scope of the current cli and we'd like some feedback.

As a brief intro, I've written a good chunk of where the current cli stands as well as a little offshoot experiment here called teal-cli, and have kicked off the development of the language server here.

The cli has been growing steadily and the scope of it has come into question about if it should be its own project. I'm generally in favor of this for the reasons outlined below.

I think the end result of this discussion is basically the ideas proposed by myself in #187, where the cli gets split of into its own fully fledged module, with a nice api for external tooling like a language server to use, and this repo is left with a very minimal cli that can just type check and compile simple files, not entire projects, along with an api for the package loader tl.loader(), and external tools like the cli to use for compiling.

teal-cli was a super experimental project for implementing things that I wanted. Basically a rough draft of cool features. But the code is a little messy, was written when some of the api to Teal itself wasn't as nice, and I personally would rather start from the ground up with the hindsight of teal-cli's codebase, rather than try to salvage it. Additionally, it brought in my ltreesitter module, which is a nice module, but tree-sitter can be a little annoying to install.

So a new cli is going to be written, in its own repo and everything. So where do you, the user of Teal, come in?
Well what do you expect this tool to do? I'll start with some stuff that I would expect out of it.

• A build command, which basically functions as it does in Lazy rebuilds #362
• Some fancy colors in the output. From teal-cli it was really nice to have a visual distinction between filenames and line numbers and such.
• perhaps some very simple integration with the teal-types repo? One feature I implemented in teal-cli was in a global config, you could point to where you installed the teal-types repo and it would add the relevant folders to your path
• An api for loading a config file
• Maybe moving the config file to tlconfig.tl? Or having some sort of project.tl/build.tl/something else?

[hishamhm]

Yes, I'm in favor!

My suggestion is renaming the low-level compiler to tlc and keeping essentially the functionality of tl pre-tl build, and then the new command takes over as tl with the existing commands having the same UX for a smooth transition. The no-dependencies core compiler moves to tlc, the new tl takes over the tl entry at LuaRocks, and lists tlc as a dependency.

Open question: should tlc still have any knowledge of tlconfig.lua or only act based on flags?

[lenscas]

I don't think it should have knowledge about that file. I see little benefit to it having knowledge about it and if it does you can end up having 2 files to configure the compiler.

[lenscas]

Well what do you expect this tool to do?

As mentioned in the discussion that I started. I want this tool to be powerful enough so most if not all projects can use it directly, rather than it being driven by build tools like make.

So, I would like to see an install command, which grabs the missing dependencies from luarocks (Assuming libraries written in teal keep using luarocks), as well as runs a custom script that sets up anything else that is needed. Like a database + migrations.

Automatically runs a build script to generate code

A script that runs after compiling, for if the target needs the code to be in a special shape.

Also a new and init command to quickly create new projects, ideally with a way to load project templates.

[lenscas]

Who is saying anything about ditching luarocks? I agree that fully ditching luarocks is stupid.

As for combining a compiler and package manager. The compiler is still separate as tlc, instead the new CLI will drive tlc and luarocks. Just like how cargo drives rustc or how yarn can drive tsc (Though, in yarn's case that is because yarn can run scripts rather than it really driving it)

All the CLI will be able to do is run user defined teal script(s) with one with a special name having a bit extra meaning as it will also tell luarocks to install dependencies. I am happy if it can define multiple or just have that one. Though I also think having a script run right before compiling (if its cache is invalidated) and one that happens right after are a good thing.

[daelvn]

Sorry, I seem to have understood it wrong, judging from the many mentions of cargo and yarn (both of which are package managers), comparing a new Teal CLI to it, and when you mentioned:

unless we want to break compatibility with luarocks (unless someone wants to make a package manager specifically for teal that is backwards compatible with luarocks?)

Which looked to me more like a proposal. Fully ditching luarocks in favor of a Teal specific package manager is not a good idea, but we still need to get LuaRocks to collaborate instead of throwing features into a Teal CLI. Telling LuaRocks to install dependencies is much harder than it really looks like (there is no official/updated public LuaRocks API, so we are either forced to use io.popen, or we still go to LuaRocks to tell the team to help us out). It just seems to me that the most logic option would be getting LuaRocks to help us, since as you said, the package manger and CLI will be bound to be separate programs. I still defend that there should be no overlap between tools whatsoever.

[lenscas]

I'm mentioning cargo and yarn because those are the 2 I have the most experience with. Cargo also comes closest to just having projects work after cloning them. Yarn however gets close to this and only loses because there is no consensus on how to name those scripts.

there is no official/updated public LuaRocks API, so we are either forced to use io.popen, or we still go to LuaRocks to tell the team to help us out

Well, giving it an api even one that boils down to a glorified io.popen will help other projects that want to integrate luarocks. So, I'm all for getting that done :)

I still defend that there should be no overlap between tools whatsoever.

If the new CLI has a install command that gets luarocks to install stuff and also runs a user defined teal script, then there is no overlap though? Even if it gets something similar as what I want it still wouldn't overlap because it will either run lua scripts or shell commands like yarn. Meanwhile, the CLI is able to natively run teal scripts instead. Which (at least if you ask me) is a better feature for the teal compiler.

I'm not looking forward to writing lua again for setup scripts simply because luarocks gets the ability to run lua files as setup scripts and it was thus decided that teal's CLI doesn't need to do that. I switched to teal because I was sick and tired of dynamic typing, I would guess that most people using teal have similar feelings towards dynamic typing (though, maybe not as extreme as me?). Running commands like yarn isn't much better either. Its much more limited and also still dynamically typed.

Even if for some reason luarocks can not be integrated then I am still in favor of having the new CLI be able to understand some basic scripts. A setup script to setup the project, a build script to generate code based on files in the project and a script that runs after compiling to get the generated code in whatever (weird) shape that is required to get it to run on whatever platform is targeted. It just streamlines the process too much to be ignored.

cargo already makes a large amount of projects work out of the box by supporting build scripts. I see no reason not to take this and improve upon it, especially not because "running those scripts is the task of the package manager" when said package manager currently doesn't do that and even if it did it wouldn't do it as good as the CLI can do because they have different targets/goals.

But, ok lets say that running scripts directly is out of scope. Can we then at least bundle something with the CLI that has that goal in mind? Then we can still get every (or at least most because there will always be some outliers) projects to need the same set of commands to set it up.

That is all I want to see, having most if not all teal projects being easy to setup because there is 1 agreed upon framework with well known commands ready so everyone can just fork a teal project and without having to look it up can run the correct commands to get it setup. If people end up shelling out to make files, stay in teal or do something else doesn't bother me. For as long as the way to run those things is the same across projects. (Though, I would prefer it if it stays teal because that is 1 less thing to learn)

[daelvn]

Indeed it just boils down to a difference in opinion. I still do not find this a good thing, and it is a fully opinionated thing. I can see what cargo does and how "having everything in one place" could make it more convenient, I just don't think it is something that should be done like this. I can't shake the feeling that it would become overgrown and try to do too many things, way more than it should. I'm more of a minimal person that likes to have everything separate and modular instead of everything in one place, because it can become a mess. It's really just a principles thing. I don't understand arguments like using Teal for running scripts (they are scripts, I wouldn't want them statically typed). I don't believe in the "one tool to rule them all" philosophy. I am okay with being able to do all of these things, just not as the "Teal CLI". For now, let's see what we can get out of Alfons before moving on, I'd say?

[lenscas]

I don't understand arguments like using Teal for running scripts (they are scripts, I wouldn't want them statically typed).

Welp, that is another way that we disagree I guess :P I don't care if it is "just a script" just give me types. Been too often burned by "its just a script, don't worry about types" method of thinking for stuff that ended up being either MUCH more complicated than "just a script" or being used for WAY longer that it should have, and having plenty of duct tape "fixes" as a result.

Anyway, for as long as the CLI gets a way to run scripts (ideally teal) and have a naming scheme somewhat enforced for them (be it by special treatment or something else) then I am happy. I don't care if it has a mechanism for this custom build or comes bundled with a program or something else. I just get tired of project X needing command Y,Z and A and project B needing O and W while project C does the same thing but using commands L,E, R and Q.

I lost count of the amount of ways that people found to name the yarn command to run webpack >_>
yarn webpack
yarn fe
yarn build (I believe?)
and I am probably missing a few.

And then I am not even talking about the various code generation scripts. Please get a system in place to steer projects to 1 naming scheme so it is just a tl setup to set the project up and have tl build run a build script that can be cached. Please lets not make the same mistake as yarn with its total lack of naming scheme or worse, C/C++ where projects can't even decide on how to run build commands, let alone their naming scheme.

[daelvn]

I love the proposal! I don't think I need much out of it, but I like the ideas you proposed. There is only one thing I would like to nitpick on, if possible:

perhaps some very simple integration with the teal-types repo? One feature I implemented in teal-cli was in a global config, you could point to where you installed the teal-types repo and it would add the relevant folders to your path

I personally think LuaRocks should be the one to handle this? It's not something LuaRocks currently does, as it only handles Lua modules, but I think placing the .d.tl files beside the actual modules, for each package, sounds like the most reasonable option to me. As I said in my reply to lenscas, I think projects should just keep their own scope: LuaRocks for managing and distributing packages and modules, Teal for compiling and building Teal projects.

[euclidianAce]

Alright, over the weekend I basically hammered out the sort of basic ideas that I had for this and now have a somewhat viable product to show off. For your viewing enjoyment:

The cli bootstrapping itself from the old one:

Building my neovim config: (which is probably too big for its own good, but hey, good stress test)

So the course of action is mostly:

• I have to catch up on recent developments as I've basically been ignoring everything around here :P
• Work on getting Lazy rebuilds #362 rebased and merged, or at the very least, the api changes it makes to tl.parse_program
• Figuring out where this is going to live, ideally here in the teal-language organization, but under what name? I've been using teal-cli internally, and my first thought would be teal-language/teal-cli, but it's not hard to change and I'm open to other suggestions. And I'll probably archive my own teal-cli since this is basically just a cleaned up version of that.

Then, once this is at least somewhat stabilized, work on the language server can continue, since this provides a nice api to load a tlconfig.lua and other nice apis for working with a project

[euclidianAce]

pinging @hishamhm, any thoughts on where this should live? I have the language server loading tlconfig.lua through the api and would like to push the changes as it's vaguely functional now. But this cli also relies on the api changes in #362, so also pinging for that. And if #362 is a little too ambitious to merge I can extract out the api changes to tl.parse_program returning require arguments into its own pr since the bulk of it is reorganizing the existing tl driver script

[hishamhm]

I'm on the phone right now, but I'll give this a closer look later tonight! Splitting the PR sounds like a good idea

[hishamhm]

The code showcased in the above animations is an updated version of the one in the main branch in https://github.com/euclidianAce/teal-cli, right? I couldn't find a more recent branch there.

Anyway, I looked around the project there and I've been thinking quite a bit about this topic recently and I have some comments and questions!

I think the main question I have is about the intended scope of the tool. @euclidianAce, since you're driving this effort, I'd like to understand what's your goal — for example, I've seen that in the c-build branch you've been experimenting with building C modules, even, which overlaps with the functionality of LuaRocks's builtin build system. Do you intend this to be a full-fledged build system? Do you want to make it into an actual package manager?

(Side note: one thing that I have noticed for literally decades is that is OSS design discussions people often spend a lot of effort discussing and pushing designs feature-by-feature when in the end what they really want is something similar to something else that already exists (longwinded discussions in the GNOME and KDE worlds in the early 2000s come to mind, I should write a blog post about this!) — so it saves a lot of time if people just say "what I really want is cargo" rather than advocating feature-by-feature to something that ultimately amounts to the same (oh hello @lenscas! :) ), or "I want Haskell ADTs", or "I want Typescript optionals", etc. It's natural to want to have the things we like in other tools, and it's useful to bring experiences from other ecosystems. So my general advice is "don't be afraid to copy if that's what you want, but be blunt about it". /Side note over!)

It looks like at the very least you want to make a build system, and I have taken PRs that ended up kind of steering the tl CLI in that direction — at first I thought it would be something minimal, essentially a for-loop over files running tl check/gen. But the current design in tl-cli has steered away from that, and as you add functionality, it's clear to me that it will only grow in size (especially looking at functionality which overlaps with LuaRocks, a domain I have a long experience in, so I know where this ends :) ).

Every Lua project seems to end up inventing its own world (I've been seeing this since Kepler circa 2005), and tl-cli is going in the same direction with its own filesystem library, logging library, task scheduler, etc... in fact, Kepler old-timers would recognize that exact list of libraries :) — LuaRocks also implements some of that, and I know the story, with the fun of first building it, then the annoyance of making it all portable, then the endless maintenance (with people forever claiming that the feature set is insufficient...). I don't mean to sound like a downer, but it would not sleep well at night if I didn't let it all out and told you "do you really want to make another build system?" I've seen so many come and go in the Lua world. I can even tell what are the missing features people are going to start to ask for. As for a package manager, well, it's the same story, but I don't know if you want to dive into that.

Now, back to tl, one thing I've been constantly saying since the FOSDEM 2019 talk is that Teal is informed by the previous well-intentioned typed Lua projects that did not succeed by trying to over-reach. So that's why I put minimalism as a goal. It needed to be something small enough so that I could pull off. Lately, I've been phrasing it more like "minimalistic and pragmatic", in order to not lose practicality due to dogmatic minimalism. This reflects on a bunch of things. One of them is "a single tl.lua file to load into your projects" (I will eventually split the tl.tl compiler into multiple files, but I'll set something up that joins the generated code to still produce one tl.lua), another was the "no-dependencies CLI" that is easy to run anywhere and integrate into any project. I think one thing in my mind that diverges from some recent proposals I've seen like "tl init which creates a set of folders with a predetermined structure" is that right now I'm more interested in having people integrate Teal into their Lua ecosystems — and making sure that Teal is ecosystem-agnostic — rather than push for "the Teal way of doing things", whatever that ends up being. Having things like tlconfig.lua, with preload_modules, include_dir and such, felt like "helping people integrate Teal into their environments". The more the tool becomes a build system, or a project manager, or a package manager, that feels like moving in the other direction. I don't want people to feel like they have to buy into a particular build system in order to use Teal (yes, I know that's different from the reality of Rust and Cargo where language and build system are intertwined, but Lua has a different baggage).

Having said that, I can see that a lot of people would like to have a "Teal way of doing things". My initial goal, to be honest, was to eventually use LuaRocks for that purpose in order to not re-reinvent the wheels I've been reinventing for the past 15 years, but maybe people want something else — LuaRocks is pretty old-fashioned in many ways, and it could benefit from a serious facelift, or at least a wrapper!

So what I've been thinking most recently is that, if you want to continue pursuing the build system idea (or even a package manager), what might be best is to give it its own name (like cargo is distinct from rustc, ant from javac, etc.) so that it can grow and flourish unhindered by the "minimalistic" promises of Teal itself. It can live in the teal-language org and we can advertise it in Teal's main README.md.

The distinction of which things are bundled and which are separate tools are often arbitrary — in Go, everything is very monolithic in typical Google fashion, so you have go build, go fmt, go doc, etc; in Java javadoc is a separate tool; in Rust, cargo mixes build system and package manager, but rustdoc and rustfmt for example are separate; in Java everything is separate; etc. Perhaps the tool could be called tlbuild? (that would be the obvious-sounding-but-not-creative-option)

If we go that direction then we can merge the API changes from #362, and then later consider trimming down the build functionality from tl and directing people who want it to tlbuild (or whatever it ends up being called!).

[euclidianAce]

(I'll preface this by saying I had a much wordier response that I accidentally lost by closing my browser D:, so this is a much more condensed version of my thoughts)

The code showcased in the above animations is an updated version of the one in the main branch in https://github.com/euclidianAce/teal-cli, right? I couldn't find a more recent branch there.

No, I actually haven't pushed it anywhere yet. Basically just imagine a cleaned up and trimmed down version of teal-cli without the ltreesitter dependency.

I think the main question I have is about the intended scope of the tool.

I would say my main goal is a project manager/build system for, at the very least, pure Teal projects. The c-build branch was an experiment that I wasn't really happy with and don't think I'd like to re-implement. And I don't think I'd like to tread into package manager territory with it either.

I don't mean to sound like a downer, but it would not sleep well at night if I didn't let it all out and told you "do you really want to make another build system?"

I appreciate the honesty :). I justify this tool existing in that it is for Teal and I think that is distinct enough. And this could serve as a testing grounds for teal integration into luarocks.

So what I've been thinking most recently is that, if you want to continue pursuing the build system idea (or even a package manager), what might be best is to give it its own name

Agreed. Locally I've been using the name Charon as a placeholder and I couldn't think of a better moon pun than blue-moon, and that's already a linux bluetooth tool. (maybe teal-moon? If anyone has any good names/puns related to the color Teal and the moon hit me up! But I digress...)

But the functionality of tl vs charon would basically be

tl:

• what you would put in a Makefile
• tooling can interact through the command line and passing various arguments and flags
• has no sense of a project or even a config file
• provides an API for type checking/compiling only

charon:

• In pure teal projects, the replacement for make if desired
• tooling can interact with a project through a programmatic API
  • for example, I've been working on the language server in parallel with this and the editing experience now that it can load a config has been amazing
• uses tlconfig.lua or maybe something else to determine the root and structure of the project

Another justification for this tool would be helping to drive the development of the compiler's api since there's something with a definite use case of it.

[hishamhm]

It all makes a lot of sense. I like the contrast you made in the bulleted lists.

I wouldn't mind keeping tl minimally aware of tlconfig.lua, if only as a shorthand to the config options -- essentially, only reading the options that have equivalent as CLI arguments. In fact, the tl "namespace" of tlconfig.lua could be identical to the long-form names of the CLI arguments (which is a little different from how it is now, but hey, it's 0.x, we can break stuff). I think Busted does that in their config file, it's pretty convenient.

helping to drive the development of the compiler's api since there's something with a definite use case of it.

+1 to this!

If anyone has any good names/puns related to the color Teal and the moon hit me up!

If I can come up with a more cheerful name idea than the ferryman of the dead, I'll let you know (names are hard! it took me what, 2 years or so to settle on Teal? I'm bad at this!). Perhaps we could start running with the color theme? say, cyan? (Or some other color of your preference, but Fuchsia is taken by the Google folks 😆 )