Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

API for general program improvement without slicing #463

Open
marov opened this issue Dec 16, 2021 · 0 comments
Open

API for general program improvement without slicing #463

marov opened this issue Dec 16, 2021 · 0 comments

Comments

@marov
Copy link
Contributor

marov commented Dec 16, 2021

Is your feature request related to a problem? Please describe.
Currently Linea can extract ("slice") source code required to produce some variable.
i.e. linea.slice("c") in

a = 2
b = 2
c = a + b
d = c / a

would produce

a = 2
b = 2
c = a + b

as a does not depend ond
However, one may want to output the entire code, including d

Describe the solution you'd like
"Slicing" the entire code would produce the code identical to the input, so what is the point?
In the trivial example above - none, however, leveraging additional capabilities of Linea we can produce useful outcomes:

  • Airflow (Perfect. Argo, Ray ...) DAG encompassing the entire code
  • Modularization for effectiveness and efficiency - we may output 2 separate functions producing a and d, which can be run parallel, restarted/cache independently etc.
  • Replacing in-mem pandas object by distributed datasets (Dask, Spar, Ray ...) for scalability.
@yifanwu yifanwu changed the title Full code slicing API for general program improvement without slicing Dec 16, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant