Proposal: unifying modules, records and code blocks

[berekuk]

Resharing from discussion in Slack: https://docs.google.com/document/d/1QK-y35KRWpZSiGvfZY0QB-cfqYg0WbaWA2e8xoSf0-8/edit

Summary

• Allow blocks without final value to evaluate to their bindings
• Import modules as values; bindings-as-record is the common case but we can do others as well
• Records are syntax sugar for blocks

(examples and rationale are explained in the linked document)

[umuro]

@berekuk I am agreeing to changes for inner blocks. Inner blocks can export the variables as records

However, I won't change the implementation for files.

• When I evaluate("x = 1; x+x), I will simply get 2 and I don't want the act of discarding the bindings I've got. So bindings and result will stay separate at the level of files.
• Also, constraining bindings to be returned as part of the result prevents that data to be tagged with other data which is not squiggle values at all. It is very constraining for the unexpected future development of the language. I need to keep squiggle value space clear of compilation artifacts.
• I am saying this, in spite of "just constructing a record at the end" is way to achieve private variables.
• **Also no to this record trickery because there will be cases where the result is a record and it is not bindings at all. ** There will be a record result and bindings in many cases. If we go along files only return records that's all then this won't be possible at all.
• If you say "as long as we keep the result in a property named default, there wont be a confusion", then I refer to the problems above. Still it is ambiguous: If there is a property named default then it must be the bindings.

[berekuk]

and I don't want the act of discarding the bindings I've got.

Why?

Also, constraining bindings to be returned as part of the result prevents that data to be tagged with other data which is not squiggle values at all.

I guess this is related to this part of my doc? "Will we ever need to stick extra metadata values on modules,, e.g. .__version__ or __filename__ properties?"

If I understand you right, then:

• we could have a separate API for this, e.g. getMetadata(project, source_id)
• actually, I don't mind extra methods for introspecting files! getBindings or getAST, or whatever; I just don't want it to be necessary for basic evaluation
• there's the layer of "I'm just running my program", and then there's the layer of "I want to debug my program"; for that debugging part we could implement more powerful APIs
• but bindings won't be enough for debugging, anyway; e.g., you can't introspect local variables in lambda bodies with them

**Also no to this record trickery because there will be cases where the result is a record and it is not bindings at all. ** There will be a record result and bindings in many cases. If we go along files only return records that's all then this won't be possible at all.

I think there's a typo or two here, can you rephrase?

If you say "as long as we keep the result in a property named default, there wont be a confusion", then I refer to the problems above. Still it is ambiguous: If there is a property named default then it must be the bindings.

Again, I'm not sure what you're saying and what's going to cause a confusion, sorry... Can you give an example of code that would be confusing and cause problems, maybe?

[umuro]

Before answering your questions. I will illustrate the ambiguity.

Case 1:
The user decides to keep x private. So he has to write

plot = {title: "My Plot, lineWidth: 4, lineStyle: dotted}
x = 1
y = 2
{plot: plot, x: x}

So far so good but we will see that we can't do this eventually...

Case 2:

plot = {title: "My Plot, lineWidth: 4, lineStyle: dotted}
x = 1
y = 2

No result we build the record {plot,x,y} (The old way)

Case 3:

plot = {title: "My Plot, lineWidth: 4, lineStyle: dotted}
x = 1
y = 2
plot.addColor(red)

I have got a record {title: "My Plot, lineWidth: 4, lineStyle: dotted}, shall I interpret as bindings for variables title, lineStyle and lineWidth?
Do I have to remind the user that to make available plot + bindings, he attempts writing this:

plot = {title: "My Plot, lineWidth: 4, lineStyle: dotted}
x = 1
y = 2
{plot: plot->addColor(red), x: x, y: y}

Then we remind him that not returning the results would accomplish the same thing. So he writes again:

plot = {title: "My Plot, lineWidth: 4, lineStyle: dotted}
x = 1
y = 2
plot2 = plot->addColor(red)

First plot should have stayed private but it was too much to write, he says.

Case 3:

The user adds an expression to the end.

plot = {title: "My Plot, lineWidth: 4, lineStyle: dotted}
x = 1
y = 2
plot2 = plot->addColor(red)
x+1

And then Sam and me come and say we got the result where are the bindings?

At this point we decide to merge result and bindings?
a) Shall we return it always as

{ result: ...,
  bindings: ... }
 So that what is binding and what is binding is clear? Nope then when I import a file I will have to write Y.binding.plot. Ugly
 
 Let's revert to default:

{ default: ...
x: ...
y: ...
plot: ... }

So we conclude that there has to be a default property there and it is the result

Then I want to call Squiggle for simple evaluations. evaluate("1+1") which returns:

{default: 2}

Get used to this we are always constructing records.

evaluate("x=1; x+x} which returns

{ default: 2, x = 1}
Get used to discarding bindings yourself even if you only need the result.

Case 4:

We merged the bindings and result and the user now wants to construct a binding hiding some variables:

plot = {title: "My Plot, lineWidth: 4, lineStyle: dotted}
x = 1
y = 2
{plot: plot, x: x}

Just as we are thinking everything was alright, we get

 {default: {plot:..., x:...}}

Hey I've got a result but where are the bindings?

AMBIGUITY
Because results can be records as well, there is no way to get out of this ambiguity. Whatever decision we take it is messing up another case that involves a record in the results.

Your idea would be applicable if records were used for no other purpose. There are ambiguous cases whatever you decide. Result and Bindings are different things. So the idea that if I return a record at the end then it can count as bindings does not apply in all cases.

Moreover, we are returning back to the idea of partials this way. We will have to forbid using last expressions in included files so that they are safe. Thus, they cannot build a bindings record anyway again.

Sam and Quinn has spent quite a time to make me understand this blunder. This is why I completely changed the code to produce distinct result and bindings. I was the one saying "hey just return the variables in a record to get your bindings. Let's automatically generate a record at the end to produce bindings". I scraped that altogether after realizing the problems.

I was thinking it was totally unnecessary. I was saying: Simple reducer is working and just build your records as you wish. Because to get distinct bindings, I would have to insert a State Monad into the Reducer code and I was horrified by that. I finally inserted the monad code there.

[umuro]

Back to your answers

and I don't want the act of discarding the bindings I've got.

Why?

Performance loss. I have started to use Squiggle in the implementation of Squiggle. And it will go more and more in that direction. That magnifies the performance loss. Sometimes, it is more straightforward to just use the language instead of dealing with complicated corresponding APIs.

---

Debugging. You're right just bindings are not enough. Ozzie already asked me: It would be nice if we could debug the bindings of local blocks. Like you say much more is needed for debugging: lambdas, local variables of inner blocks...

I think there's a typo or two here, can you rephrase?

The whole essay above.

Again, I'm not sure what you're saying

The whole essay above.

[umuro]

Now, all your import ideas are OK

And I don't see any conflict about them and they are achievable. {Except hacking records to hide private variables}

What I see wrong is that "We could do that in a minute because there is no need for bindings". Hacking for immediate results part. In fact, I am putting them into my job queue immediately.

We will do that kind of import and we wont unify bindings and results

[berekuk]

plot = {title: "My Plot, lineWidth: 4, lineStyle: dotted}
x = 1
y = 2
plot.addColor(red)

I assume addColor doesn't mutate plot? It's not clear from your description. (If Squiggle is going to become non-pure in the future, this would have much bigger consequences than just this discussion).

In this case, the author of the code is either confused about some fundamentals of how Squiggle works, or they really intended to return that result.

But also...

plot = {title: "My Plot, lineWidth: 4, lineStyle: dotted}
x = 1
y = 2
plot2 = plot->addColor(red)

You could put plot instead of plot2 and you won't get issues with extra variable. So this is not true:

First plot should have stayed private but it was too much to write, he says.

---

And then Sam and me come and say we got the result where are the bindings?

Let's ask Sam? I haven't heard from him on this proposal yet, it'd be interesting to see if he'd be opposed to it, and why.

At this point we decide to merge result and bindings?

I hope not. I won't comment on code examples below this point because I don't like the variation with default and don't want to see it implemented. You're right about issues with it.

Sam and Quinn has spent quite a time to make me understand this blunder. This is why I completely changed the code to produce distinct result and bindings. I was the one saying "hey just return the variables in a record to get your bindings. Let's automatically generate a record at the end to produce bindings". I scraped that altogether after realizing the problems.

Again, let's see what Sam and Quinn are going to say, then? Maybe I'll manage to convince them that end expression (with implicit generated record if it's missing) is enough. At least I'm hopeful.

[umuro]

@berekuk

Again, let's see what Sam and Quinn are going to say, then? Maybe I'll manage to convince them that end expression (with implicit generated record if it's missing) is enough. At least I'm hopeful,

End expression (with implicit generated record if it's missing): If it is missing then we have to put the result into the bindings! That is the problem. Bindings are needed even if the expression is there or is missing. There is no case only the final expression is OK (for UI).

[Hazelfire]

Hey! So I think this is actually something Umur wanted to implement (and kind of has implemented) a while back, but I pushed back fairly strongly against this for the following reason, please let me know if you have a better way of solving this particular problem / you think I've misunderstood.

In your proposal you have:

No need to keep track of both result and bindings when evaluating squiggle code (that will lead us to [result, bindings] tuples all over the place); code always evaluates to a single SquiggleValue

The problem here is that when writing Observable notebooks, it's really common (in fact, almost always the case) that one wants both the result and bindings from a squiggle execution. For instance, one may write:

household_grant_size = 925 to 1125
household_grant_size

And then both want to see the result of household_grant_size and reference household_grant_size in another squiggle execution. I'd like to do this in one execution, but under your proposal, I'd likely need to do it in 2. One is to store the variable name for later use, and the second is to display the grant size to the user. This is similar to what we currently have with the partials/cell distinction:

I'd ideally like to write another cell that goes like:

household_size = gamma(2, 4)
person_grant_size = household_grant_size / household_size
person_grant_size

My main excitement about ReducerProject is that it does have this [result, bindings] tuple xD. It makes the experience much more like Jupyter Notebook.

There may be a few ways around this that don't have [result, bindings] distinctions, one could be that every squiggle execution simple returns its bindings, and the viewer allows you to select a variable to display to the user from the bindings. That would require much more work than simply returning the bindings and the result.

[Hazelfire]

Another note is that we often don't want to trigger the record viewer on bindings. This is because the number of declared variables at the top level scope could be large, and have already been displayed in different cells, and possibly even different notebooks! It can also include math and utility functions that just pollutes the result that you want to show (which is often just nothing for a collection of bindings).

For instance, Squiggle already has this feature. Check this out! This is my GiveDirectly analysis:

[umuro]

Some notes: @berekuk @Hazelfire

ReducerProject has 2 API functions: getResult, getBindings instead of getResultAndBindingsTuple

The technical reason is that if you won't display the bindings yet or if you don't need them then you are not triggering any conversion process or memory consumption. So I got away from the idea of delivering the result and bindings bundled together. And this way bindings are always there whether there is a result or not. There are no partials anymore to construct bindings if there is no result.

If one wishes a tuple then it is simply (getResult, getBindings)...

Also sometime next week, each file will start to have its own local bindings. Standard library or bindings of depended files will not propagate

---

In fact, when Sam @Hazelfire wanted bindings to be always there I simply could not do it. Because solving that problems was requiring the use of a state monad that seemed to mess the code too much as there is no mention of monads in Rescript like in Haskell.

Of course I had to solve the monad problem for other purposes like collecting performance statistics. I was having the nightmare of changing the result of every function to pass the state around. Then I found a workaround that does not mess or change the code.

The new version of Reducer in ReducerProject is equipped with a State monad to collect bindings, performance statistics, function result caching, and other kinds of debug data in the future. Bindings are there in the state monad even before the result is calculated after 2 months of planning and plotting.

[berekuk]

@Hazelfire, thanks so much! This clarifies a lot.

I have some new ideas now about how to maybe get the best of both worlds, but it might take me some time to figure out and communicate properly. I'll focus on fixing lambda viewers in ReducerProject for today and come back to this thread later.

[berekuk]

This proposal still has its problems and it seems like we'll have to release imports without doing this, but just a quick note...

This is because the number of declared variables at the top level scope could be large, and have already been displayed in different cells, and possibly even different notebooks!

The new 0.5 interpreter doesn't have this problem anymore. Imported bindings are stored in a separate layer (scope) from the local bindings, and we could render just the locals.

[OAGr]

(We have since made progress here, closing)