chore: bump dataproc image version #94
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
17 tasks
project-defiant
force-pushed
the
spark-image
branch
from
19b4b0a to
fdcc26a
Compare
ireneisdoomed
approved these changes
Jan 24, 2025
There was a problem hiding this comment.
Thank you for updating and testing!!!
Great that it works. Only question is whether we should separate Gentropy's main entry point and cluster set up script from the Gentropy version. I know these 2 files have been virtually identical for a long time.
In general, I don't think it makes sense having to define Gentropy's locations in every single YAML. It's OK for now, but maybe we want to do this differently in the future.
Thanks anyway!
| @@ -105,6 +105,7 @@ def create_cluster( | |||
| num_preemptible_workers=num_preemptible_workers, | |||
| num_workers=num_workers, | |||
| image_version=GCP_DATAPROC_IMAGE, | |||
| internal_ip_only=False, | |||
There was a problem hiding this comment.
Was this argument the reason why the cluster didn't have internet access?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Context
Bump the version of the dataproc image to use pyspark 3.5.1 -> https://cloud.google.com/dataproc/docs/concepts/versioning/dataproc-release-2.2
Implementations
GENTROPY_REFcluster metadata variable.The solution above allows for quicker iteration (there is no need to wait for the github actions to complete before the dags with gentropy steps can be run).
Below you can see the dataproc image 2.2 working with gentropy from
pyspark-bumpbranch.