Possible options to synchronize ASR in LFortran and LPython

[kp992]

Possible options to synchronize ASR in LFortran and LPython

A common solution to keep a library like libasr synchronized between multiple projects (LFortran and LPython) on GitHub is to use dedicated repository of libasr and then include it in the projects as a dependency or submodule.

Options

Option 1: Use Git Submodules

• In both LFortran and LPython, add libasr as a submodule:

git submodule add libasr_github path/to/libasr

• This will link both projects to the same version of libasr, and updates to libasr can be pulled into LPython and LFortran by running git submodule update.

Option 2: Use Git Subtree

• Git Subtree allows to "merge" the library libasr into LFortran and LPython but without the dependency tracking of submodules. It simplifies some workflows but requires manual updates.

git subtree add --prefix=path/to/libasr libasr_github main --squash

Option 3: Use a Package Manager

• If libasr can be packaged (e.g., as a C++ library using CMake), we can publish it to a package repository (such as GitHub Packages), and reference the version in LFortran and LPython.

Option 4: Automated Sync workflows

All the above approaches provide a solution where we pull out libasr.
This approach is a bit different where we keep libasr in both the projects as we have currently.
We can add automated CI workflows in Github Actions to the following tasks:

• Check on every commit to main branch
• If there is a patch in libasr of LFortran, trigger an automated pull request to LPython and vice-versa. If the pull request passes all the tests in LPython, then merge it automatically.
• Apply the same to standalone libasr library on github.

This steps ensure that all the three projects(LPython, LFortran and libasr) are in sync (may be with some delay of an hour).

The main drawback of this approach is that there can be several cases of merge conflicts when simultaneous development is going on both the projects at the same time.

Option 5: MonoRepo

This is the easiest approach both for developers and maintainers. We can merge all the projects in the single repository and provide a flag so that we can build/test individual projects to save developers' time. While merging the pull request we can run the whole pipeline.

What do you think @certik is the best way to move forward with?

[certik]

Thanks @kp992 for writing this up! The problem with Option 5 is that every change we make to ASR we now must also get working with all LPython tests. So even if locally we build LFortran separately, we will all still have to build LPython anyway just to get CI to pass. Right now I don't want to slow us down on our main objective of delivering LFortran to beta.

So that leaves probably Option 4. I would treat LFortran as our main "upstream" for libasr for now, we should port most of our tests for lists, tuples, dicts, unsigned and symbolics into LFortran as well (expose in the frontend and test). That would only leave some corner cases that are only triggered by LPython. We should implement all major features in LFortran first. And we port them to LPython second. Whether automatically (that would be nice), or manually.

[kp992]

Thanks @certik. Should we move forward with Option 4 and for now considering LFortran as the source of truth and opening automated PRs in libasr and LPython?

[Shaikh-Ubaid]

I am in the favour of Option 5, but maybe not now like Ondrej mentioned. For Option 4, I have concerns that most likely the automated PRs towards LPython would/might fail at the CI, since I think ASR is still developing and thus fast changing.

Assuming a lot of PRs with failing CIs build up at LPython, it more or less leaves LPython in the current state (where the LibASR is not synced).

[certik]

To move forward, we will keep using LFortran as our main driver. I personally would not bother with automated PRs, I think they will add to our maintenance of things, and I don't want to spend resources right now on that.

For LPython we simply manually upgrade libasr from LFortran from time to time. And all improvements in LPython must be submitted via LFortran first, and then just sync the libasr.