Add Spark 4.0.0 Build Profile and Other Supporting Changes [databricks] #10994

razajafri · 2024-06-06T23:49:33Z

This change adds the profiles needed to build the plugin for Spark 4.0.0

This PR depends on #10993

fixes #9259

Signed-off-by: Raza Jafri <[email protected]>

razajafri · 2024-06-07T15:51:54Z

This is in draft until #10993 #10992 are merged

razajafri · 2024-06-07T17:49:35Z

pre-commit task failing for Spark400 as the private repo changes haven't been merged.

build/buildall

pom.xml

scala2.13/pom.xml

razajafri · 2024-06-14T17:06:09Z

@gerashegalov do you have any more questions on this PR?

gerashegalov · 2024-06-14T21:42:26Z

The 400 Pr check is failing. We need to wait until spark-rapids-private artifact is published https://github.com/NVIDIA/spark-rapids/actions/runs/9519648358/job/26243644463?pr=10994

mythrocks · 2024-06-24T17:08:45Z

I was wondering if this could be checked in before #11066. It doesn't need to wait, does it?

gerashegalov · 2024-06-24T22:39:06Z

@razajafri @mythrocks there are two pending issues with this PR indicated by the failing check

❌ mvn[compile,RAT,scalastyle,docgen] / package-tests-scala213 (400, true) (pull_request)

buildver=400 cannot be built due to the corresponding spark-rapids-private dependency being missing
buildver=400 check should also be moved under JDK17

It currently fails with

Error:  Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.5.0:enforce (enforce-java-version) on project rapids-4-spark-parent_2.13: 
Error:  Rule 0: org.apache.maven.enforcer.rules.version.RequireJavaVersion failed with message:
Error:  Support for Spark 4.0.0 is only available with Java 17+

Update

spark-rapids/pom.xml

Lines 871 to 873 in 4b44903

    
           <jdk17.buildvers> 
        
               330 
        
           </jdk17.buildvers>

We need to exclude 400 from from PR checks based on JDK8,11

razajafri · 2024-06-25T05:58:24Z

Depends on #11092

There is a bytecode incompatibility which is why we are skipping these until we add support for it. For details please see the following two issues NVIDIA#11174 NVIDIA#10203

razajafri · 2024-07-12T03:14:23Z

build

pxLi · 2024-07-12T06:58:55Z

scala2.13/pom.xml

                <executions>
                  <execution>
-                    <id>enforce-maven</id>
+                    <id>default-cli</id>


changing execution id here caused the buildver fail to get correct default builder var (330) from maven life cycle if not specify buildver= in mvn run

Unexpected buildver value 311 for a Scala 2.13 build, only Apache Spark versions 3.3.0 (330) and higher are supported, no vendor builds such as 330db

Yes I don't think the enforcer was actually working before changing the id. We need this value from command line to run the enforcer

~~OK then you are changing the expected behavior of running mvn command with default buildver.~~

~~This means we cannot simply mvn verify w/o specify buildver in scala213. If this is expected then we must doc this for all occurences.~~

see below

I ran below in scala2.13 folder,

mvn help:evaluate -Dexpression=buildver -q -DforceStdout

only showed 311 in this PR,
for latest upstream, it could print out correct 330.

please make sure it could print out the correct builder for default profile locally

# try check mvn help:active-profiles

upstream:

Active Profiles for Project 'com.nvidia:rapids-4-spark-parent_2.13:pom:24.08.0-SNAPSHOT': The following profiles are active: - release330 (source: com.nvidia:rapids-4-spark-parent_2.13:24.08.0-SNAPSHOT)

vs
this PR:

Active Profiles for Project 'com.nvidia:rapids-4-spark-parent_2.13:pom:24.08.0-SNAPSHOT': The following profiles are active: - enforce-cloudera-jdk-version (source: com.nvidia:rapids-4-spark-parent_2.13:24.08.0-SNAPSHOT)

Maven is prioritizing it over the default release activation (comment out this profile could result correct builder). I think you could just add the enforcer to cdh profiles directly if it's just a JDK version check

@razajafri @gerashegalov Please note I've merge PR dropping 31x shims: #11159

In this PR there're some files' change which are conflicted[modified or deleted in MR11159], can you help sync those files with the TOT source on the upstream branch-24.08? Thanks!

I ran below in scala2.13 folder,

mvn help:evaluate -Dexpression=buildver -q -DforceStdout

only showed 311 in this PR, for latest upstream, it could print out correct 330.

please make sure it could print out the correct builder for default profile locally

# try check mvn help:active-profiles

upstream:

Active Profiles for Project 'com.nvidia:rapids-4-spark-parent_2.13:pom:24.08.0-SNAPSHOT': The following profiles are active: - release330 (source: com.nvidia:rapids-4-spark-parent_2.13:24.08.0-SNAPSHOT)

vs this PR:

Active Profiles for Project 'com.nvidia:rapids-4-spark-parent_2.13:pom:24.08.0-SNAPSHOT': The following profiles are active: - enforce-cloudera-jdk-version (source: com.nvidia:rapids-4-spark-parent_2.13:24.08.0-SNAPSHOT)

Maven is prioritizing it over the default release activation (comment out this profile could result correct builder). I think you could just add the enforcer to cdh profiles directly if it's just a JDK version check

Looks like when building Scala2.13 + JDK17, it enables enforce-cloudera-jdk-version with condition <jdk>[17,)</jdk>

Then the 330 is disabled unless we manually add -Dbuildver=330, otherwise buildver falls back to the initial value of 311

I created the enforce-cloudera-jdk-version to avoid code duplication as maven doesn't allow profiles inheritance. I have updated cloudera profiles now

This is what I see now

mvn help:active-profiles

in the root project dir

Active Profiles for Project 'com.nvidia:rapids-4-spark-parent_2.12:pom:24.08.0-SNAPSHOT': The following profiles are active: - release320 (source: com.nvidia:rapids-4-spark-parent_2.12:24.08.0-SNAPSHOT)

This is what I see in scala2.13 folder

Active Profiles for Project 'com.nvidia:rapids-4-spark-parent_2.13:pom:24.08.0-SNAPSHOT': The following profiles are active: - release330 (source: com.nvidia:rapids-4-spark-parent_2.13:24.08.0-SNAPSHOT)

…cloudera profiles

razajafri · 2024-07-12T22:00:38Z

build

gerashegalov

This PR contains different source code changes. Can you reconcile the title with the PR intent.

Technically, more shimming is needed due to a recent upstream change:

14:47:51,237 [ERROR] [Error] /home/user/gits/NVIDIA/spark-rapids.worktrees/spark400/sql-plugin/src/main/spark332db/scala/org/apache/spark/sql/hive/rapids/shims/GpuRowBasedHiveGenericUDFShim.scala:36: type mismatch;
 found   : Any
 required: () => Any
14:47:51,240 [ERROR] one error found

pom.xml

…hanges

gerashegalov

LGTM, let us make sure that all unresolved issues are captured as GH issues.

gerashegalov · 2024-07-16T06:25:07Z

.github/workflows/mvn-verify-check.yml

-    needs: cache-dependencies
+    needs: set-scala213-versions


We need an issue how to better handle cache for scala2.13 build dependencies

gerashegalov · 2024-07-16T06:27:25Z

build

razajafri added 4 commits June 6, 2024 19:43

POM changes for Spark 4.0.0

0c3a1ba

Signed-off-by: Raza Jafri <[email protected]>

validate buildver and scala versions

1631ab4

more pom changes

3271bfd

fixed the scala-2.12 comment

5d2b867

razajafri force-pushed the SP-9259-POM-changes branch from b7f175f to 5d2b867 Compare June 7, 2024 02:44

more fixes for scala-2.13 pom

58806bd

gerashegalov reviewed Jun 7, 2024

View reviewed changes

build/buildall Outdated Show resolved Hide resolved

build/buildall Outdated Show resolved Hide resolved

addressed comments

540c732

sameerz added the Spark 4.0+ Spark 4.0+ issues label Jun 7, 2024

gerashegalov mentioned this pull request Jun 9, 2024

binary-dedupe changes for Spark 4.0.0 [databricks] #10993

Merged

razajafri commented Jun 12, 2024

View reviewed changes

pom.xml Outdated Show resolved Hide resolved

scala2.13/pom.xml Outdated Show resolved Hide resolved

add in shim check to account for 400

5faecf8

razajafri marked this pull request as ready for review June 14, 2024 17:05

razajafri requested review from jlowe, revans2, tgravescs, GaryShen2008 and NvTimLiu as code owners June 14, 2024 17:05

razajafri added 3 commits June 24, 2024 21:36

add 400 for premerge tests against jdk 17

5de0fff

temporarily remove 400 from snapshotScala213

bb784f0

Merge remote-tracking branch 'origin/branch-24.08' into HEAD

140eb7b

razajafri added 2 commits June 24, 2024 23:12

fixed 2.13 pom

41e1982

Remove 400 from jdk17 as it will compile with Scala 2.12

85a2f6d

Merge remote-tracking branch 'origin/branch-24.08' into HEAD

f599413

pxLi mentioned this pull request Jul 10, 2024

Drop spark31x shims [databricks] #11159

Merged

razajafri added 4 commits July 11, 2024 18:13

Skip Opcode tests

0544c6f

There is a bytecode incompatibility which is why we are skipping these until we add support for it. For details please see the following two issues NVIDIA#11174 NVIDIA#10203

Merge remote-tracking branch 'origin/branch-24.08' into HEAD

a43a68d

upmerged and fixed the new compile error introduced

2cf7351

addressed review comments

cabcda0

pxLi reviewed Jul 12, 2024

View reviewed changes

razajafri added 4 commits July 12, 2024 12:20

Merge remote-tracking branch 'origin/branch-24.08' into HEAD

c19fccf

Removed jdk17 cloudera check and moved it inside the 321,330 and 332 …

feeabfc

…cloudera profiles

fixed upmerge conflicts

54a3ee4

reverted renaming of id

48fe626

razajafri requested a review from gerashegalov July 15, 2024 19:11

gerashegalov reviewed Jul 15, 2024

View reviewed changes

pom.xml Outdated Show resolved Hide resolved

razajafri changed the title ~~POM Changes for Spark 4.0.0 [databricks]~~ Add Spark 4.0.0 Build Profile and Other Supporting Changes [databricks] Jul 15, 2024

razajafri added 3 commits July 15, 2024 15:50

Merge remote-tracking branch 'origin/branch-24.08' into SP-9259-POM-c…

166e4c6

…hanges

Fixed HiveGenericUDFShim

6cbce78

addressed review comments

8d0351b

razajafri force-pushed the SP-9259-POM-changes branch from 8d74758 to e177024 Compare July 15, 2024 23:36

reverted the debugging code

5b0e36d

razajafri force-pushed the SP-9259-POM-changes branch from e177024 to 5b0e36d Compare July 15, 2024 23:39

generated Scala 2.13 poms

66419b8

gerashegalov approved these changes Jul 16, 2024

View reviewed changes

razajafri mentioned this pull request Jul 16, 2024

Handle cache for scala2.13 build dependencies #11199

Closed

razajafri merged commit a41a616 into NVIDIA:branch-24.08 Jul 16, 2024
43 checks passed

razajafri deleted the SP-9259-POM-changes branch July 16, 2024 14:21

YanxuanLiu mentioned this pull request Jul 22, 2024

Cache dependencies for scala 2.13 [skip ci] #11231

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Spark 4.0.0 Build Profile and Other Supporting Changes [databricks] #10994

Add Spark 4.0.0 Build Profile and Other Supporting Changes [databricks] #10994

razajafri commented Jun 6, 2024 •

edited

Loading

razajafri commented Jun 7, 2024

razajafri commented Jun 7, 2024

razajafri commented Jun 14, 2024

gerashegalov commented Jun 14, 2024

mythrocks commented Jun 24, 2024

gerashegalov commented Jun 24, 2024

razajafri commented Jun 25, 2024

razajafri commented Jul 12, 2024

pxLi Jul 12, 2024 •

edited

Loading

razajafri Jul 12, 2024

pxLi Jul 12, 2024 •

edited

Loading

pxLi Jul 12, 2024 •

edited

Loading

NvTimLiu Jul 12, 2024

NvTimLiu Jul 12, 2024 •

edited

Loading

razajafri Jul 12, 2024

razajafri Jul 12, 2024

razajafri commented Jul 12, 2024

gerashegalov left a comment

gerashegalov left a comment

gerashegalov Jul 16, 2024

razajafri Jul 16, 2024

gerashegalov commented Jul 16, 2024

Add Spark 4.0.0 Build Profile and Other Supporting Changes [databricks] #10994

Add Spark 4.0.0 Build Profile and Other Supporting Changes [databricks] #10994

Conversation

razajafri commented Jun 6, 2024 • edited Loading

razajafri commented Jun 7, 2024

razajafri commented Jun 7, 2024

razajafri commented Jun 14, 2024

gerashegalov commented Jun 14, 2024

mythrocks commented Jun 24, 2024

gerashegalov commented Jun 24, 2024

razajafri commented Jun 25, 2024

razajafri commented Jul 12, 2024

pxLi Jul 12, 2024 • edited Loading

Choose a reason for hiding this comment

razajafri Jul 12, 2024

Choose a reason for hiding this comment

pxLi Jul 12, 2024 • edited Loading

Choose a reason for hiding this comment

pxLi Jul 12, 2024 • edited Loading

Choose a reason for hiding this comment

NvTimLiu Jul 12, 2024

Choose a reason for hiding this comment

NvTimLiu Jul 12, 2024 • edited Loading

Choose a reason for hiding this comment

razajafri Jul 12, 2024

Choose a reason for hiding this comment

razajafri Jul 12, 2024

Choose a reason for hiding this comment

razajafri commented Jul 12, 2024

gerashegalov left a comment

Choose a reason for hiding this comment

gerashegalov left a comment

Choose a reason for hiding this comment

gerashegalov Jul 16, 2024

Choose a reason for hiding this comment

razajafri Jul 16, 2024

Choose a reason for hiding this comment

gerashegalov commented Jul 16, 2024

razajafri commented Jun 6, 2024 •

edited

Loading

pxLi Jul 12, 2024 •

edited

Loading

pxLi Jul 12, 2024 •

edited

Loading

pxLi Jul 12, 2024 •

edited

Loading

NvTimLiu Jul 12, 2024 •

edited

Loading