Better UX for "types with multiple roles" in the IDE

[JaroslavTulach]

Using Enso Type Classes in the IDE

Types playing multiple roles are needed to represent for example table with a single column as both the Table and Column. Such a multi value (e.g. intersection) types are already available to the IDE and ready to be used right now - see guidelines.

• example of "mixing in" multiple types into single value

Static Typing

While intersection types can be used right now, their relation of with static typing is still under development:

• Intersection types & type checks #11600

Todo: IDE Support

Pending tasks:

• use information provided by Deliver full intersection type of a value to the IDE #11583
• expand on that information when necessary

Conversions

IDE should support conversions out of the box to work well with [Enso Type Classes]((#5446 (comment)). Because there is nothing special on type class value - except conversions:

• there is nothing special on the actual type class itself - it is just a type with single private constructor taking two arguments
• once the IDE obtains an instance of such a type class - there is nothing stopping it to use it

The question however is: How to improve UX of obtaining such a type class?

• type classes are closely associated with conversions
• e.g. improving IDE UX means to improve the way IDE works with from conversions

Right now there is no special support for conversions in the IDE. Conversions are only used automatically when converting autoscoped constructors (#9285) and other argument types. However there is no explicit UI for that. This discussion is here to identify the necessary UI changes to make the conversions to type classes more usable for the needs of standard libraries.

TL;DR

The solution to the below listed use-cases is going to be built around values with intersection types and conversions. Such multi values are already present in the Enso runtime and it is just needed to expose them to the IDE properly. With proper information about intersection types of a value at hand and with already provided database of from conversions, the IDE shall be capable to handle all the use-cases.

Seeing All Conversion is possible as they are all listed in the suggestion DB. Preferred Conversions are those that are re-exported via library Main - e.g. those that get imported by from Standard.Xyz import all - those are automatically offered to the user by exposing methods on the target types directly in the component browser (of course with some level of Auto-Domination) - more complicated conversions need to be explicitly imported before being used - IDE has to insert appropriate import statement when user selects (either as argument of Any.to method or by other means) such a conversion.

Treat Column or Vector as a Table is possible because there will be preferred conversions offering such simple metamorphosis. Treat single column Table as a Column will be handled at creation time by creating multi value representing both types at once - e.g. table as well as column. IDE uses the list of all types participating in value's intersection type to compose list of suitable methods to be displayed in the component browser.

Remaining tasks

• Ensure EnsoMultiValue returns some Meta.type_of #11480
• Deliver full intersection type of a value to the IDE #11481
• Allow (multi_value : Table) : Column to succeed #11482

Use-Cases

Treat Column or Vector as a Table

There is a simple conversion from a Column to Table and there is a simple conversions of a Vector to (unnamed) Column. Can the IDE's component browser display methods of a Table when one is working with Column or Vector object?

Treat single column Table as a Column

When there is a Table object with a single Column it makes sense to treat it as both: as a Table and as (the single) Column. Can the IDE's component browser display methods of a Table and Column in such a case? Can we prevent methods from a Column to be displayed in the component browser, when a table has zero or more than one column?

Possible solution: described here.

Preferred Conversions

Currently there is one type class adhering to the blueprint: Comparable. A lot of classes is convertible to Comparable. However it may not be wise to show Comparable methods to the user as the Comparable.from conversion may not be that important.

We need a way to specify which conversions should be included by default and to control what (methods from possible conversions) appears in the component browser.

Auto-Domination

The functions from the type classes that a value can be converted to should appear in the component browser in the normal way. When the instance has a method already with the same name that dominates (and only instance one should be shown).

Seeing All Conversion

By default the component browser shows only preferred conversion (see above). However there should be a way to see all the possible conversions.

Possible solution: there is method Any.to available on every object. When selected, the drop down widget shall offer all the conversions. However probably sorted by preference.

Displaying Conversions

The IDE should identify to the user that a conversion (e.g. a cast like op1 : Table) may be needed to invoke certain method. The IDE should identify to the user that a conversion is taking place in the code graph - either explicitly or implicitly. There should be a way to analyze/navigate to the actual conversion that's taken a place.

[somebody1234]

just throwing ideas out there to start the conversation:

• I think usage of such type classes should be transparent to the user - they shouldn't need to know about all the intermediate types to find out what they can do with this type
  • However this still comes with considerations. should we deduplicate typeclass instances? memoize them?
    • e.g. should they be shown as (foo:Type).method? as a separate node? completely invisible until hover? (visually) as a pipeline (foo|> _:Type |> _.method)?
    • If we go with this I guess it should be made very explicit that the methods are on a different class? e.g. [via Type] in the docs panel
  • On the GUI <-> LS side I assume this would entail either API changes, or (potentially recursive) fetching of valid conversions
    • Depending on how liberal we are with conversions, perhaps (in the future) we may want to hide some conversions from being automatically done? Perhaps some conversions, while they would make sense in certain very specific contexts, would just add clutter to the Component Browser the vast majority of the time

[somebody1234]

the need for deduplication

the idea was (foo: Type).method_a and (foo: Type).method_b - i assume in most cases the performance benefit is minimal though. not sure if there's a situation in which doing the type conversion may be expensive, and of course i'm not sure if that's worth having dedicated support in the IDE, depending on how common of a usecase this is

[JaroslavTulach]

• however the use-case Treat single column Table as a Column requires the IDE to actually try the conversion to find out if it is possible - how that's going to be done still needs to be designed

Here is an idea. By default the return type of a from conversion is the target class - e.g. Comparable in case of:

Comparable.from (that:Integer) = ...

the idea is to add an explicit signature indicating an error may be thrown:

Comparable.from (that:Complex) -> Comparable ! Not_Available_Error = ...

and deliver such an explicit signature to the IDE via suggestion DB. The IDE would know that conversions explicitly typed to yield an error need to be executed first on the that object to verify they are valid. This would solve the problem of treating single column Table as a Column:

Column.from (that:Table) -> Column ! Too_Many_Columns_Error = 
    if that.column_count != 1 then Error.throw Too_Many_Columns_Error else
        that.at 0

The signature is only useful for selecting which conversions shall be checked. We don't want to explicitly check all the conversions as that might be too slow. E.g. we need a way to identify the suspicious ones that need the check. However the signature isn't needed for execution - Error.throw is a regular return value that can be emitted from any method regardless of its signature.

[somebody1234]

i think this should be promoted to a top-level comment for visibility?

i think that that... maybe sounds good on paper? but with similar reasoning to above, if the conversion is expensive (as above, i'm not sure how common that will be), then the IDE is potentially doing the full, expensive conversion, just to figure out whether to show an entry in the suggestion DB.

if we are going to try every conversion before displaying then performance becomes a bit more of a concern.

an alternative suggestion would be to add another method:

Column.can_convert_from (that:Table) -> Boolean = that.column_count == 1

although it's not ideal, since then you have to make sure that the logic is (more or less) identical to the conditions under which the conversion throws an error

We don't want to explicitly check all the conversions as that might be too slow

yeah, that's a very good point. i guess there are a number of ways to do this? perhaps we should think about what kind of metadata goes where (as a tag in comments, vs e.g. the presence/absence of a method, or a method returning a boolean)?

[JaroslavTulach]

going to try every conversion before displaying then performance becomes a bit more of a concern.

No, we will not try every conversion.

We don't want to explicitly check all the conversions as that might be too slow

We want to check just the more "advanced" and we need to identify them somehow.

The issue with conversions is: they are "mapping a type to another type" - if the mapping is possible for all instances of that type, then the suggestion DB is good enough.

However in the case of "table with one column" - the mapping only works for subset of Table instances. There are few options:

1. try the conversion and let it fail
2. invent some special logic like can_convert_from
3. create new type Single_Column_Table and define Column.from (that:Single_Column_Table) and Table.from (that:Single_Column_Table) - e.g. define a special type for the "special" instances
4. change return type of methods that return Table to Table | Column and instead of single column tables return Column
5. associate a single value with multiple types.

The idea 2 is annoyingly complex and moreover "premature optimization is source of all evil". Option 3 may not be ideal for the user neither for library writes. Option 4 requires a lot of changes thru out the Standard.Table API. 1 seems easy to start with. However right now my favorit is option 5!

[JaroslavTulach]

Primary and Secondary Types of a Value

An interesting option to deal with Single_Column_Table is to use a multi value. There already is a concept like that in Enso. It is possible to create a value x: Text & Integer which is both a Text and an Integer. It is then possible to cast such a value to both of these types and invoke any of their methods on the value.

This could be a natural solution for the single column table. If a Table value feels it is also a Column, then it could explicitly convert itself into Table & Column. The IDE would then just need to know that the value at hand actually have more than a single type.

The only problem is the current behavior of -> Table return check. Right now, if there is a such check or if the value is casted by : Table, then all the other (secondary) types are discarded. This would have to be changed:

• Method return type checks avoid conversion when the type fits #10882

The above PR already tried such an approach and it seems to work. The only complain was that one expects x : Table to ensure only methods of Table are available on such a result (otherwise static analysis has no chance to detect impossible calls). Maybe that can be solved by hiding all the other methods, but still allowing x : Table : Column to succeeds without a generic conversion Column.from (that:Table) being available.

[jdunkerley]

Giving an approximate library developer / end user spec:

• We will need to be able to specify which conversions should be included by default (to control what appears).
  • Some type conversions may only be possible in some cases (e.g. Table with a single column can be treated as a Column).
• The functions from the type classes that a value can become should appear in the component browser in the normal way.
  • When the instance has a method already with the same name that dominates (and only instance once should be shown).
• When a user chooses a function which needs to be converted it should have the conversion in the component added automatically (e.g. :Column on a Vector).
  • I imagine this would be visible on the component (with :Column appearing next to the input icon) or similar.

[JaroslavTulach]

The description of this discussion now includes Use-Cases section.

We will need to be able to specify which conversions should be included by default (to control what appears).

Covered by Preferred Conversions use-case.

Some type conversions may only be possible in some cases (e.g. Table with a single column can be treated as a Column).

Covered by Treat single column Table as a Column use-case.

The functions from the type classes that a value can become should appear in the component browser in the normal way.

Covered by Auto-Domination use-case.

...chooses a function which needs to be converted...

Added Displaying Conversions use-case, although I am not sure I covered the essential requirement.

[JaroslavTulach]

One of the (not yet properly formulated) use-cases is related to the interoperability between

• Table
• Column
• and Column of various types
  • Column of Integer
  • Column of dates
  • etc.

If there is a Column holding just Integervalues, then it should offer sum, max, + operation. However Column of dates should not offer sum, at all. @jdunkerley can you please update this comment to turn this fuzzy view of mine into some material example?

[jdunkerley]

How does Both.combine relate with Jaroslav's suggested & ascription?

Could we do c: Column&Integer_Column instead of the Both? It seems that they are essentially the same concept.

Are we okay for the x = v : T typecheck on a value v : T&U to 'hide' the U type, meaning that only methods of T will be visible on such x, until we call x : U/U.from x to do a conversion that 'unhides' the additional type? Asking because this is the foundation upon which most of static typechecking lies.

The Both.combine is not meant to be the real syntax - very much just implying the behavior I think we want.

Both.combine table column was my shorthand for saying it was a Table&Column with it's value being table when a Table (the default) and column when cast to Column.

I think methods on x:Table&Column only has the methods of Table. To access the Column it would need to be explicitly cast.

This leads to the complication of ideally col_a:Integer_Column should be freely able to use Column methods for backwards compatibility but then maybe we just have to put the place holders on it and accept the issue for now.

[jdunkerley]

I guess we "deprecate" the non-explicit methods on Column and go with returning Column&Integer_Column. The user can then access the methods still in same way in component browser and old code still works.

[JaroslavTulach]

// ## EXPLICIT
Table.from (that:Text) = ...

* _Should not be auto-invoked in parameter passing._ (could be expensive to do an explicit conversion, should a user need to choose?)

If we want to avoid conversions in parameter passing, then I suggest to not make them available via Main - e.g. with from Standard.Xyz import all. Then:

• such a conversion isn't going to be visible for the code
• IDE will know such a conversion exists from Suggestion DB
• e.g IDE can offer such conversion for .to method
• when user chooses such a conversion, IDE can import it

If we hide the explicit conversions this way and require explicit import to enable them, we may not need the ## EXPLICIT documentation tag.

[radeusgd]

If we want to avoid conversions in parameter passing, then I suggest to not make them available via Main - e.g. with from Standard.Xyz import all. Then:

• such a conversion isn't going to be visible for the code
• IDE will know such a conversion exists from Suggestion DB
• e.g IDE can offer such conversion for .to method
• when user chooses such a conversion, IDE can import it

If we hide the explicit conversions this way and require explicit import to enable them, we may not need the ## EXPLICIT documentation tag.

@JaroslavTulach I'm not sure if this is enough. This does not give us 2 kinds of conversions but rather it gives us 2 kinds of conversion visibility. The problem is that if I'm working on a single workflow and on one of my nodes I decide to use an 'Explicit' conversion working like described above, then the IDE will add the import, and from now on - all nodes in my current workflow will begin to resolve this new conversion. So the conversion becomes enabled 'globally' (well scoped to current workflow, but for the user that is 'global'). And moreover it cannot easily be disabled (as at least currently the IDE does not have a way of automatically cleaning unused imports - but let's say that's not an inherent problem but rather a lacking feature).

I'm not sure I know exactly what @jdunkerley meant by explicit conversions (please correct me if I misunderstood), but if I get the overall idea correctly, I think we want to mark some conversions as 'explicit' to ensure they are never applied automatically in parameter passing (so importing them should not change their resolvability), but at the same time have them available in the dropdown of the .to method.

---

With some further thought...

I'm making an assumption that when in My_Proj I'm calling Mod.foo x with value x of type T, the scope for resolving conversions is of the function definition module Mod, and not of the call-site My_Proj. As far as I remember the conversions code, that assumption should be correct.

In that case, I guess the scoped conversions are not that bad, because most of the time, the x : T type assertions are done in library code where these conversions will explicitly be not imported. So it does indeed (to some extent) achieve that when passing parameters to library methods, such conversions will not be applied automatically and they will only apply in user workflows, once the conversion gets imported by the IDE.

Still, I'm not entirely sure this is the best solution we can have.

1. We still have the problem that once the user imports the conversion in their workflow, it is automatically implicitly applied everywhere the user adds a type ascription in that workflow. I know that users will probably be rarely do that. But the import does influence the whole workflow and this may cause some unexpected changes of behaviour that can confuse users - which is not good.
2. Additionally (but related to (1)), I'm overall hesitant about explicit imports controlling resolvability of conversions. I know we already have this feature and I think it is OK for library code written by power users. But it feels to me that it may make more harm than good for user projects. It may not be immediately clear for the user that removing an import disables some conversions in their code - especially as, distinct from imports of functions/modules, the name of the import is not immediately related to such conversion.

---

Thus it feels like it may be better to still have some way to define an 'explicit' conversion that never takes part in the automatic resolution, but can only be called explicitly.

I guess we could even do it without modifying the engine (not sure if we should). Technically, we could define an ExplicitConversionSPI and use similar logic like we do for FileFormatSPI to display possible selections on the .to dropdown. Then the to method would work in 2 phases: in one phase it would check if there is an 'automatic' conversion and if not found, it could consult the SPI to see if an explicit conversion was registered, and then use that. Not at all sure if this is remotely a sensible idea, just throwing ideas around.

[JaroslavTulach]

I'm not sure I know exactly what @jdunkerley meant by explicit conversions

The example I got was: Table.from (that:Text) - e.g. quite expensive "conversion" which you don't want to enable implicitly.

@JaroslavTulach I'm not sure if this is enough. This does not give us 2 kinds of conversions but rather it gives us 2 kinds of conversion visibility.... all nodes in my current workflow will begin to resolve this new conversion.

Alas, that's correct. Unless IDE invokes the conversion method explicitly:

table = Standard.Table.Extra_Conversions.from text

which isn't the nicest syntax, but it avoids the "import problem".

the import does influence the whole workflow and this may cause some unexpected changes of behaviour that can confuse users - which is not good.

The Displaying Conversions use-case could be expanded to ask the IDE to visualize when a conversion is taking a place and allow the user to see what conversion has been applied. This was a request stated by Wojciech few months ago and also something Kustosz mentioned - he was using a language with complex conversion rules that he never understood properly, but the IDE was showing which conversion is used when and Marcin could live with it.

[JaroslavTulach]

Thank you all for your comments. To demonstrate the desired state, let's consider following Enso module and its behavior in the IDE:

from Standard.Base import all
import Standard.Visualization

type R
    private V a:Text i:Integer

    as_text self = self
    as_both self = self
    as_int self = self

Text.from (that:R) = that.a
Integer.from (that:R) = that.i

main =
    n = R.V "Hi" 3
    t1 = n.as_text
    b2 = n.as_both
    i3 = n.as_int

first of all, there are some obvious problems in the current IDE:

• private isn't recognized as a keyword by the IDE's editor #11475
• IDE's editor corrupts code when return types are inserted #11476

but at the end I managed to negotiate and settle with the IDE on:

from Standard.Base import all
import Standard.Visualization

type R
    private V a:Text i:Integer

    as_text self = self:Text
    as_int_text self = 
        r = (self : Integer & Text)
        r
    as_text_int self = 
        r = (self : Text & Integer)
        r
    as_int self = self:Integer

Text.from (that:R) = that.a
Integer.from (that:R) = that.i

main =
    n = R.V "Hi" 3
    t1 = n.as_text
    b2 = n.as_int_text
    i3 = n.as_int
    node1 = n.as_text_int

with that code we can see that t1 and i3 are properly typed:

however b2 and node1 aren't type correctly.

[JaroslavTulach]

b2 and node1 components don't have the right type in the IDE. Neither they show the right content in component browser:

We need to make sure:

• types of Integer & Text and Text & Integer are properly shown in the IDE
  • fixed as Ensure EnsoMultiValue returns some Meta.type_of #11480
  • delivered by Deliver full intersection type of a value to the IDE #11583
• component browser offers methods of both Integer and Text such such intersection types
• the value is properly narrowed down when a method of secondary type is selected - e.g. b2:Text . length
  • Instructions how to do it are available.

Just like in the above example.