[Python] --fableLib fable-library crashing dependencies.

[Freymaurer]

I am not sure if this is intended or not, but using --fableLib fable-library will crash the dependencies of the published files:

This is the command i use for transpilation

dotnet fable <path> -o <out_path> --noCache --lang py --fableLib fable-library --noReflection

This results in an empty fable_modules folder and the following error message:

Traceback (most recent call last):
  File "c:\Users\Kevin\Desktop\test\py.py", line 1, in <module>
    from siren_dsl import siren, git
  File "C:\Users\Kevin\Desktop\test\.venv\Lib\site-packages\siren_dsl\__init__.py", line 1, in <module>
    from .siren_types import SirenElement
  File "C:\Users\Kevin\Desktop\test\.venv\Lib\site-packages\siren_dsl\siren_types.py", line 5
    from .fable-library.list import (FSharpList, of_array, singleton as singleton_1, empty as empty_1, fold, map)
               ^
SyntaxError: invalid syntax

This is what the python file looks like

from __future__ import annotations
from collections.abc import Callable
from dataclasses import dataclass
from typing import (Any, TypeVar)
from .fable-library.list import (FSharpList, of_array, singleton as singleton_1, empty as empty_1, fold, map)
from .fable-library.option import (value as value_2, map as map_1, default_arg)
from .fable-library.reflection import (TypeInfo, union_type, string_type, list_type, option_type, record_type, bool_type, class_type)
from .fable-library.seq import (to_list, delay, append, singleton, collect, empty)
from .fable-library.string_ import (join, to_text, printf)
from .fable-library.types import (Array, Union, Record, to_string, seq_to_string)
from .fable-library.util import IEnumerable_1
from .yaml import (Yaml_line, Yaml_level, Yaml_AST, Yaml_root, Yaml_write)

This is what the dependencies look like:

[MangelMaxime]

In theory --fableLib is used to consume fable-library from another source than the included version coming with each Fable release and having this flag means that we don't make a local copy of the embedded fable-library folder.

For example, in JavaScript you write --fableLib @fable-org/fable-library-js in order to consume fable-library from the NPM packages.

The idea being, if you publish several Fable compiled package from NPM, it will allows them to use the save file which avoid problems with equality because the constructor class is different etc.

With all this introduction, we need to check how Python handle import from registry like PyPi or the NPM equivalent in Python and check if we are actually generated the correct code.

It is also important to note, that at the time of writing we are only publishing pre-compiled version of fable-library on NPM.

The published version of fable-library on PyPi is from the early implementation of Fable Python support and outdated. We should add support for releasing fable-library on PyPi just like done for NPM.

To do so, we should create a fable organisation if PyPi support organisation. Like that we can have core maintainer like @dbrattli and myself at minimum with access to the package for access security.

To sum up if we want to consume fable-library from PyPi:

• Check what we should generate when using --fableLib fable-library
• Create a Fable organisation on PyPi (members at minimum @dbrattli and me)
• Add ownership on fable-library to the Fable organisation
• Configure this repository so we can compile / push fable-library to PyPi

@dbrattli Do you agree with this actions plan ?

[Freymaurer]

So a simple fix would be --fableLib fable_library? So that fable just replaces import with that string?

Edit: Of course only after an update to the library

[MangelMaxime]

So a simple fix would be --fableLib fable_library? So that fable just replaces import with that string?

If this is how we access the fable-library module from PyPi yes that should be enough once the package is updated. Of course, it would suppose that you install the package locally.

Edit: I should have some time next week to look into working on automating the publishing of the package

[dbrattli]

There used to be a hidden feature that if you did --fableLib site-packages then Fable would try to import fable-library from the installed python packages. I tried it but I cannot get it to work. But I like the idea of the simple fix and we could also merge this effort with the pyo3 (rust) bindings since we need a fable-library build for multiple architectures i.e it needs to be built in CI and pushed to PyPI. So yes I agree on the action plan.

[dbrattli]

I've done a couple of stabs at this but I'm a bit confused about what to do with e.g other NuGet dependencies. Should those also be imported from site-packages / PYTHON_PATH or should we continue to import those from fable_modules? What is the motivation for wanting to import fable-library from another source, and why should that reason not apply to every other dependency you have as well? 🤔

[Freymaurer]

So we do actually have such a situation. Our libraries ARCtrl and FsSpreadsheet, both are fable libraries, which are also transpiled to native js, share a common dependency (which is also f# with fable). I think @MangelMaxime was investigating how to handle this.

[dbrattli]

Is that common dependency also available on PyPI? What is the problem you get? It would be really great if we could describe the problem at a higher level, i.e as a developer I want to ... but the problems is that ... This issue starts with a traceback, but I would like a better description of what you are trying to solve and what the problem is

[Freymaurer]

As a consortium we are interested of publishing libraries to multiple programming languages based on one common code base, but the problem is that complex systems are currently not fully supported by fable.

As a developer i want to publish multiple libraries based on fable libraries to multiple programming languages, but dependency handling is not supported (as far as i know).

The following example:

classDiagram
    class fscore["FsSpreadsheet.Core"] {
        isFableCompatible: true
    }
    class fspy["FsSpreadsheet.Python"] {
        isFableCompatible: false
    }
    class ARCtrl {
        isFableCompatible: true
    }
    ARCtrl ..> fscore
    fspy ..> fscore

Loading

We want to publish two libraries ARCtrl and FsSpreadsheet.Python to pypi. They have a shared dependency on FsSpreadsheet.Core. Currently, both ARCtrl and FsSpreadsheet.Python have this dependency as part of fable_modules. pip is unable to resolve this dependency which makes tracking between what versions of ARCtrl and FsSpreadsheet.Python are based on the same FsSpreadsheet.Core and therefore compatible difficult.

I hope this helps 🙂

[MangelMaxime]

I've done a couple of stabs at this but I'm a bit confused about what to do with e.g other NuGet dependencies. Should those also be imported from site-packages / PYTHON_PATH or should we continue to import those from fable_modules?

Only fable-library should be imported from the source specified from --fableLib.

Regarding others libraries imports, there is this issue #3737 which tracks the ideas of allowing user to redirect a NuGet import to source from another source like NPM, PyPi, etc.

Consuming pre-compiled library comes with limitations for example, I think you will not be able to expose an inline API in a pre-compiled library because this is something done at compile which would have already happened when publishing that lib.

What is the motivation for wanting to import fable-library from another source, and why should that reason not apply to every other dependency you have as well? 🤔

One reason, for importing fable-library from NPM, PyPi is for example if you published several F# compiled libraries to NPM, PyPi.

It allows them to use the same source for fable-library and at least for NPM this is required for comparaison to works. Indeed, the way Fable implements comparaison between types is by testing the constructor. So even if you compare something like Some 1 if it is not coming from the same fable-library then the constructor is different and the comparaison fails.