Priorities for new language features

[titzer]

This is a thread to discuss some ideas for how to improve Virgil from a language design and productivity perspective. I'd like to hear from people who have used the language and what they might find interesting in coming versions. Some features are easier to implement than others, and given that I'm spending most of my time on Wizard and only working on Virgil in the capacity to make it a better systems language, some of these might be more aspirational than others without help.

Generators

One goal is to be able write libraries that are easy to use with terse code. Iteration over something like a Vector should look like iterating over an array or a list. E.g.

var x: Array<byte>;
for (b in x) { ... } // iterating over array is supported in "for in"
var v: Vector<byte>;
for (i < v.length( { ... v[i]; } // iteration over length and explicit access still required for vectors

With generators, a sketch:

class Vector<T> {
  def for() -> T { // the "for" keyword designates a generator method
    for (i < length) yield this[i];
  }
}
var v: Vector<int>;
for (b in v) { ... } // a user value is iterable in a "for in" if it has a "for" method defined

Function expressions, i.e. lambdas

I started sketching out a syntax for function expressions that can capture variables from the current environment; i.e. they'd be implemented as proper closures. After thinking about many alternatives, including just using the => syntax like JavaScript, which I kind of like, I settled on using the def keyword to introducing a function expression. As for the name of the feature, I find that "function expression" is very self-explanatory, including to novice programmers, while "lambda" is understood by those who have a functional PL background. I prefer the self-explanatory, approachable term. But yeah, it's a lambda.

def m(f: int -> int) -> int { // a regular top-level method taking a function parameter
  return f(99);
}
// passing a function expression that adds 1 to its argument
m(def (x: int) -> int { return x + 1; });
// a much shorter version of the above, using => to skip return type, and inferred parameter type
m(def x => x + 1); 

Double-arrow function bodies

Related to function expressions, but worthwhile in its own right, perhaps, is using => to be able to define the body of method (and implicitly, its return type) as an expression. This helps with cleaning up additional line noise and making more one liners possible.

// before
class Foo {
  def m() -> int { return 33; }
  def n() -> int { return 44; }
  def s() -> int { return m() + n(); }
}
// after
class Foo {
  def m() => 33;
  def n() => 44;
  def s() => m() + n();
}

Transparent structured data definitions

Ongoing work is adding a struct concept to Virgil. Structs are not a type in themselves; they are more like a view over memory, either a byte array (safe), or eventually, over raw, non-heap memory (unsafe). Structs are very explicit, requiring every field to have an offset and the overall struct size to be pre-declared.

struct S {
  +0 x: u32;
  +4 y: float;
  =12;
}

This will make interacting with software and hardware that has fixed memory layouts, like file formats, network packets, and external data structures, much, much easier.

[mosheduminer]

I'll think more about this and will probably have more feedback as I continue to use Virgil. But in the meantime I'll give some feedback here, based on my work on the Parser Combinator.

1. It would be convenient to have function expressions (which you already mentioned). In my work on the Parser Combinator, for every parser type, I needed to define an impl method with all required data as parameters, and then curry that the function to create the parser.

2. It would be convenient to be able to discard values when using match, right now every field in an ADT needs to be named. For example, instead of Result(target, index, result) => it would be nice to be able to do Result(target, _, _). I believe this is possible in python and rust (and probably other languages I'm not familar with).

3. Something that I thought about while implementing the Parser Combinator is the ability to define extensions on types (like here). My problem was that I wanted to define methods such as .map, errorMap, and .chain on the ParserState -> ParserState type, but this is of course impossible (you can't define methods on a function), so I needed to wrap the fn: ParserState -> ParserState into a class (I could have also made them standalone functions, but I preferred the DX of methods). There's not really a problem with this, it's just that I know classes are always allocated on the heap, and I would have preferred to avoid that. I'm not sure how generally useful this is though.

4. As I mentioned in the thread about JSON, compile time reflection (maybe also with decorators?) would be a really nice feature to have. Don't know what your plans about that are.

Other (non-language) thoughts

1. Currently I'm using VS Code and it just treats virgil as text files. It would be really nice to have an LSP and VS Code extension, I saw this discussed in another thread, it's possible that I'd be able to work on this once we have JSON support. I think syntax highlighting, show definition on hover, jump to definition / usage, and autocomplete, would dramatically improve the coding experience.

2. I'm wondering what the approach would be to create bindings to something like sqlite? Or is there definitely no C compatibility?

My current goal with virgil is to get to the point where I can write a simple (single node) S3 compatible file server (mostly, I just want the most basic features). To that end I'm hoping to write a server application library (built on top of the networking work currently being done). It feels like something that is not very complex but will still be a milestone to get all the required pieces together.

More broadly I'm also curious about systems / embedded programming, but I wouldn't have any idea where to start learning about that. So, I'm focusing on higher level / web applications which I'm more familiar with.

[k-sareen]

True. But with a virgil LSP server and a Typescript LSP client, it may be easiest to just use a UDP connection. Best to survey what other languages do though. Should look at what smaller, niche languages like Zig do instead of some industrial behemoth like Java. I remember trying to read the Java/Rust/C LSP src code was a real pain. Maybe this has all improved in the last 1-2 years though.

[mosheduminer]

The zig LSP looks quite small, just a few files (ignoring the fact that the lsp.zig file is almost 10k lines 🙃). Anyway, they seem to be using stdin/stdout.

[k-sareen]

That's cool! Thanks! Once JSON support has landed, we can look into LSP support.

[mosheduminer]

Oh, btw that 10k line file is generated by a codegen.

[k-sareen]

Wew. For an MVP though I think only a few methods like definitions and did{Open,Close} are required.

[k-sareen]

This is a separate thing, but I feel like having an import statement + namespaces/modules will really help larger applications (for example use lib.net.Net) and streamline the build process.

Edit: As an aside, how does one get updates about the "Discussions" tab? Do you have to subscribe to something? I've never used it before.

[mosheduminer]

Another, perhaps better, solution (vs private) would be to allow / require types to be part of components.

[titzer]

I whipped up private types here: 5268136

[k-sareen]

Oh wow! Very cool! I'll give it a whirl. Thanks!

[mosheduminer]

As far as I can tell, types can be made private, but can't be namespaced. Would it be possible to namespace a class to a component (given the lack of any other namespace mechanism)? Say something like this:

private class MyPrivateClass {}

component SomeComponent {
    def TheClass = MyPrivateClass;
}

Which currently doesn't work, because "type "MyPrivateClass" cannot be used as an expression".

[titzer]

Yeah, I'd like to introduce a notion of modules or namespaces at some point.

[k-sareen]

Also I think non-nullable types and some form of error propagation would be very useful as well, as we discussed in #111. Specially a native error type + some kind of operator for quickly propagating errors up the call stack, like the ? operator.

[titzer]

I am thinking of a construct to make initialization of large, complex structures with named members easier. Some ideas:

with expression

class Config {
  var name: string;
  var size: int;
  var doMore: bool;
  var extraThings: Array<int>;
}

def x = Config.new() with {
  name = "foo";
  size = 44;
  doMore = false;
  extraThings = [88, 99, 1000];
};

The idea of the with expression is that it takes an expression on the left and applies the updates to the fields given in the curlies { ... }. For class types with mutable members, it takes an object, mutates the original object and returns it. For tuples, data types, or ADTs, it would create a new value with the updated fields.

It allows for having lots of parameters via a config object:

class Args { ... }
def m(x: Args) {
  ...
}
var r1 = m(Args.new() with { name = "string"; size = 66; });
// or, eliding the .new():
var r2 = m(Args with {name = "myname"; size = 777; });

Wdyt?

[titzer]

Yes, if you omit them, they get their default values (which could be from the field's initializer).

[k-sareen]

Oh that sounds pretty cool! Will a class variable end up as null? I guess so, right? since it's the default for an object. Might be ugly if you accidentally use it without initializing. Maybe only allow primitives/arrays/ADTs variables to be set in a with clause.

It might also be worth considering the use of some other keyword if you plan on adding scopes (/structured concurrency) in the future as with is a common keyword to use in such a case.

[mosheduminer]

Hold up, are there already plans on structured concurrency? I actually made a (very) minimal library based on the rust async book.

[titzer]

I haven't made any concrete plans. I keep seeing that each kind of concurrency has its uses. fork/join parallelism is really useful in some contexts, while co-routines or generators are useful in others, and in others effect handlers are the thing. Still in other situations sometimes you just want a thread. How to reconcile all of these possible directions? Unclear. What I usually do is something pragmatic and low-level that lets me experiment with the next level to see how that should be blessed and integrated into the language.

[k-sareen]

Arguably a systems language should provide a 1:1 threading model (i.e. 1 language thread = 1 OS thread), with either async primitives or some other coroutine facility to make an M:N threading model if possible. Or this is at least from my experience with a bunch of different systems-y languages. M:N threading models are great to think in though. And structured concurrency lends itself really well to M:N threading models. Just my thoughts.