Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[query] Expose references via ExecuteContext #14686

Open
wants to merge 1 commit into
base: ehigham/py4j-backend-extensions
Choose a base branch
from

Conversation

ehigham
Copy link
Member

@ehigham ehigham commented Sep 17, 2024

This change is split out from a larger refactoring effort on the various Backend
implementations. The goals of this effort are to provide query-level
configuration to the backend that's currently tied to the lifetime of a backend,
reduce code duplication and reduce state duplication.

In this change, I'm restoring references to the execute context [1] and
decoupling them from the backend. In a future change, they'll be lifted out of
the backend implementations altogether. This is to reduce the surface area of
the Backend interface to the details that are actually different.

Both the Local and Spark backend have state that's manipulated from python via
various py methods. These pollute the Backend interface [2] and so have been
extracted into the trait Py4JBackendExtensions. In future changes, this will
become a facade that owns state set in python.

Notes
[1] "Restoring" old behaviour I foolishly removed in fe5ed32
[2] "Pollute" in that they obfuscate what's different about backend query plan
and execution

@ehigham ehigham marked this pull request as ready for review September 17, 2024 00:43
@ehigham ehigham force-pushed the ehigham/ctx-references branch 2 times, most recently from 3518244 to e0b89a5 Compare September 17, 2024 15:47
@ehigham ehigham force-pushed the ehigham/cloud-credentials branch from 9cff2fc to d14679d Compare September 19, 2024 20:52
@ehigham ehigham force-pushed the ehigham/ctx-references branch from 92da6f0 to 4576acf Compare September 19, 2024 20:52
@ehigham ehigham force-pushed the ehigham/cloud-credentials branch from d14679d to ed2e8e8 Compare October 1, 2024 19:44
@ehigham ehigham force-pushed the ehigham/ctx-references branch from 4576acf to 8a42e93 Compare October 1, 2024 19:44
@ehigham ehigham force-pushed the ehigham/cloud-credentials branch from ed2e8e8 to fc06f02 Compare October 8, 2024 19:18
@ehigham ehigham force-pushed the ehigham/ctx-references branch from 341e6a7 to 1682964 Compare October 8, 2024 19:18
@ehigham ehigham force-pushed the ehigham/cloud-credentials branch from fc06f02 to a101b55 Compare October 8, 2024 20:30
@ehigham ehigham force-pushed the ehigham/ctx-references branch from 1682964 to b1c0d04 Compare October 8, 2024 20:30
@ehigham ehigham force-pushed the ehigham/cloud-credentials branch from a101b55 to 1682a0f Compare October 16, 2024 20:02
@ehigham ehigham force-pushed the ehigham/ctx-references branch from b1c0d04 to 84ddcf3 Compare October 16, 2024 20:02
@ehigham ehigham force-pushed the ehigham/cloud-credentials branch from 1682a0f to 6e63e69 Compare October 16, 2024 21:30
@ehigham ehigham force-pushed the ehigham/ctx-references branch from 84ddcf3 to a2ff477 Compare October 16, 2024 21:30
@ehigham ehigham force-pushed the ehigham/cloud-credentials branch from 6e63e69 to d31f732 Compare October 17, 2024 14:59
@ehigham ehigham force-pushed the ehigham/ctx-references branch 2 times, most recently from df5c723 to adc0602 Compare October 21, 2024 15:24
@ehigham ehigham force-pushed the ehigham/cloud-credentials branch from 82e6e7a to a70aa24 Compare October 21, 2024 18:50
@ehigham ehigham force-pushed the ehigham/ctx-references branch from 756ba40 to 539e25f Compare October 29, 2024 21:47
@ehigham ehigham force-pushed the ehigham/cloud-credentials branch from 4cf050e to 01aedac Compare October 31, 2024 15:47
@ehigham ehigham force-pushed the ehigham/ctx-references branch from 539e25f to 2b266d0 Compare October 31, 2024 15:47
@ehigham ehigham force-pushed the ehigham/cloud-credentials branch from 01aedac to 6e36a7a Compare November 4, 2024 22:15
@ehigham ehigham force-pushed the ehigham/ctx-references branch from 2b266d0 to b069621 Compare November 4, 2024 22:15
@ehigham ehigham force-pushed the ehigham/cloud-credentials branch from 6e36a7a to e121222 Compare November 19, 2024 17:08
@ehigham ehigham force-pushed the ehigham/ctx-references branch 2 times, most recently from d229670 to 07a390d Compare November 20, 2024 17:00
@ehigham ehigham force-pushed the ehigham/cloud-credentials branch from d21ad57 to 0a14104 Compare December 12, 2024 21:51
@ehigham ehigham force-pushed the ehigham/ctx-references branch from 07a390d to 234bda8 Compare December 12, 2024 21:51
@ehigham ehigham force-pushed the ehigham/cloud-credentials branch from 0a14104 to 4b932f1 Compare December 12, 2024 22:00
@ehigham ehigham force-pushed the ehigham/ctx-references branch from 234bda8 to 92087ca Compare December 12, 2024 22:01
@ehigham ehigham force-pushed the ehigham/cloud-credentials branch from 4b932f1 to 64042bc Compare December 12, 2024 22:02
@ehigham ehigham force-pushed the ehigham/ctx-references branch from 92087ca to d7df668 Compare December 12, 2024 22:02
Base automatically changed from ehigham/cloud-credentials to main December 16, 2024 17:19
@ehigham ehigham force-pushed the ehigham/ctx-references branch from d7df668 to a33bb7f Compare December 16, 2024 18:49
@ehigham ehigham changed the base branch from main to ehigham/py4j-backend-extensions December 16, 2024 18:49
@ehigham ehigham force-pushed the ehigham/py4j-backend-extensions branch from b8b22b8 to 2d4edc1 Compare December 17, 2024 16:47
@ehigham ehigham force-pushed the ehigham/ctx-references branch from a33bb7f to b19073d Compare December 17, 2024 16:47
@ehigham ehigham force-pushed the ehigham/py4j-backend-extensions branch from 2d4edc1 to 79ecea8 Compare December 17, 2024 20:00
@ehigham ehigham force-pushed the ehigham/ctx-references branch from b19073d to 75f4fd5 Compare December 17, 2024 20:00
@ehigham ehigham force-pushed the ehigham/py4j-backend-extensions branch from 79ecea8 to 1d36659 Compare January 13, 2025 16:36
@ehigham ehigham force-pushed the ehigham/ctx-references branch from 75f4fd5 to 5c0308f Compare January 13, 2025 16:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant