Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEA] Remove support for Spark 3.1.x #10955

Open
sameerz opened this issue May 31, 2024 · 3 comments
Open

[FEA] Remove support for Spark 3.1.x #10955

sameerz opened this issue May 31, 2024 · 3 comments
Assignees
@sameerz
Labels
feature request New feature or request

Comments

@sameerz
Copy link
Collaborator

sameerz commented May 31, 2024

Is your feature request related to a problem? Please describe.
Remove support for Spark 3.1, as it is more than two years old per https://spark.apache.org/news/ . It no longer receives community updates.

Describe the solution you'd like

  1. Remove Spark 3.1.x shims
  2. We should have removed documentation references to 3.1 support at this point. We need to scrub and reconfirm there are no mentions of Spark 3.1.x support.

Describe alternatives you've considered

Additional context
@sameerz sameerz added feature request New feature or request ? - Needs Triage Need team to review and classify labels May 31, 2024
@mattahrens mattahrens removed the ? - Needs Triage Need team to review and classify label Jun 4, 2024
@pxLi
Copy link
Collaborator

pxLi commented Jun 5, 2024 •
edited
Loading

From CICD perspective,

  1. Pick a new default - Done (confirmed as 3.2.0)
  2. changes to CICD setup based on the new default (still building 3.1.x shims) - Done (6/14)
  3. developer help switch the default shim to a 3.2.x version (stop building 3.1.x shims)
  4. remove all the 3.1.x code

In these steps it should not block any nightly pipelines.

NvTimLiu added a commit to NvTimLiu/spark-rapids that referenced this issue Jun 11, 2024
@NvTimLiu
Drop spark-3.1.x support for spark-rapids
169e15a 
CI part for the issue: NVIDIA#10955

We'll drop spark-3.1.x support from branch-24.08, change the default spark version string to 3.2.0

This change needs to work together with the source code update for dropping spark-3.1.x support

Will keep monitoring CI jobs status after all the related changes get merged

Signed-off-by: Tim Liu <[email protected]>
@NvTimLiu NvTimLiu mentioned this issue Jun 11, 2024
Closed
@gerashegalov gerashegalov mentioned this issue Jun 13, 2024
Open
@pxLi
Copy link
Collaborator

pxLi commented Jun 14, 2024

@sameerz We have done above step 2.
feel free to ask some developers to help deal with the drop support work and switch 2408 defaults in spark-rapids and spark-rapids-private. @NvTimLiu will help with the test scripts during the process.

Also as @gerashegalov mentioned in #11061, it would be nice to re-consider a better way to do shims version management~

NvTimLiu added a commit to NvTimLiu/spark-rapids that referenced this issue Jul 3, 2024
@NvTimLiu
Drop spark-3.1.x support for spark-rapids
830409c 
CI part for the issue: NVIDIA#10955

We'll drop spark-3.1.x support from branch-24.08, change the default spark version string to 3.2.0

This change needs to work together with the source code update for dropping spark-3.1.x support

Will keep monitoring CI jobs status after all the related changes get merged

Signed-off-by: Tim Liu <[email protected]>
@NvTimLiu
Copy link
Collaborator

NvTimLiu commented Aug 5, 2024

We still have some following up issues for this removing spark31x shims ticket, suppose dev team will help on them, thanks! cc @jlowe

#11160

#11178

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sameerz sameerz

Labels
feature request New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@mattahrens @sameerz @pxLi @NvTimLiu