-
-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[FEATURE] Optimise chained transformations #22
Comments
That's a good idea. Do you have any clues about how we would be able to get the information that calls are being chained? |
We could instantiate an object that keeps track in a dict of what methods have been called whenever we call a transformation, re-orders them and executes them. For example if we do |
Mmhh yes sounds good! Just, how do you know when the last call occurs to execute them all? |
I guess here's why Spark has transformations and actions haha We could do the same, but I don't like it very much |
haha most probably ^^ Without doing something of the sort, I don't really see yet how we could do it... I'll think about it but my intuition is that we can't without either manipulate AST or having explicit call to execute transformations |
Describe the solution you'd like
We should optimise chained transformations in a similar way as what Apache Spark does: they should be lazy and re-ordered so that transformations reducing the number of elements are always executed first.
The text was updated successfully, but these errors were encountered: