[BUG] Unable to use Spark Rapids with Spark Thrift Server #9966
-
|
Describe the bug Our objective is to activate Spark Rapids (SQLPlugin) with Spark Thrift Server. However, we encountered some exception related to ClassNotFound. For your reference, Spark Thrift Server is also known as Distributed SQL Engine. Steps/Code to reproduce bug You need to launch Spark Thrift Server with $SPARK_HOME/sbin/start-thriftserver.sh with following steps:
Expected behavior Under folder $SPARK_HOME/logs, you will see a log related to Spark Thrift Server with following exception: Environment details (please complete the following information)
Additional context These exceptions only happen with Thrift Server. With the same configurations, I am able to launch spark-shell and execute whatever sql commands that I want. |
Beta Was this translation helpful? Give feedback.
Replies: 17 comments 2 replies
-
|
Have you tried not placing the RAPIDS Accelerator jar in the Spark jars directory and instead of specifying the driver/executor classpath via the command-line and configs, use the --jars flag instead? e.g.: This worked for me, I was able to show tables and select elements from a table using pyhive to connect to the Spark thriftserver and verified via the Hive thiftserver log that the RAPIDS Accelerator was being used during the queries. |
Beta Was this translation helpful? Give feedback.
-
|
I have tried to reproduce as well with different modalities for jar submission. So far I have not been able to.
This looks like an outdated note, $ cat ~/dist/spark-3.5.0-bin-hadoop3/RELEASE
Spark 3.5.0 (git revision ce5ddad9903) built for Hadoop 3.3.4
Build flags: -B -Pmesos -Pyarn -Pkubernetes -Psparkr -Pscala-2.12 -Phadoop-3 -Phive -Phive-thriftserver
This has worked for me too, but this is my least-favorite deployment option. It typically is only required with a standalone mode (your case) but only when the RapidsShuffleManager is used as well (not present in your conf). It does not look like the case here but mixing
this is not necessary when you already placed your jar under In your setup it looks cleanest if you remove the jar from $SPARK_HOME/jars and start thriftserver with --jars. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @gerashegalov @jlowe. Thanks for your helps. Here is a youtube link which shows how I encountered this error. Furthermore, I am using Spark 3.3.0 instead of newest spark version 3.5.0 so that I need recompile my spark package. I can try with spark 3.5.0 later. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the demo @LIN-Yu-Ting. I was using Standard Spark build for 3.3.0 works with beeline for me. And again I am not sure why you need to recompile Spark for hivethriftserver. It should already be there. At any rate, can you provide your exact build command to double-check if it is about custom build? |
Beta Was this translation helpful? Give feedback.
-
|
@gerashegalov However, when I execute SQL query using Superset through PyHive then I got the above exception, which is quite weird. |
Beta Was this translation helpful? Give feedback.
-
|
@jlowe @gerashegalov I got more information from logs of Spark Thrift Server which might give us more insight. Actually, error occurs when superset is executing a command Can you please try from your side to execute this command to see whether you can reproduce errors or not ? Thanks a lot. Note: from my side to either beeline or PyHive. All are executable. Only SQL query from superset will fail. |
Beta Was this translation helpful? Give feedback.
-
|
Might the error be possibly generated by multiple sessions in Spark Thrift JDBC server ? It seems that Superset will generate multiple sessions to Spark Thrift Server as following SparkUI shown. When query is sent by either beeline or PyHive package, then only one session exists in Spark Thrift Server. In this case, no matter what SQL command you are executing, everything is fine. |
Beta Was this translation helpful? Give feedback.
-
|
We have seen issues with multisession notebooks such as Databricks. I will give a try on Superset, not familiar with it yet. |
Beta Was this translation helpful? Give feedback.
-
|
@LIN-Yu-Ting another suggestion to quickly unblock while we are looking at it. Classloading issues are likely to go away if you build our artifact from scratch using the instructions for a single-spark-version build To this end check out or download the source for the version tag. In your case the Apache Spark version you want to build for is 3.3.0 which can be accomplished by running from local repo's root dir: mvn package -pl dist -am -Dbuildver=330 -DallowConventionalDistJar=true -DskipTestsSince the tests are skipped you do not need a GPU on the machine used for the build. The artifact will be under: |
Beta Was this translation helpful? Give feedback.
-
|
@gerashegalov Thanks for providing this workaround. I have tried to build locally and replace the jar. However, unfortunately, I still got the same error as before. Anyway, I appreciate. |
Beta Was this translation helpful? Give feedback.
-
|
@LIN-Yu-Ting Can you double-check that your jar is indeed "conventional"? The output from running this command below should be $ jar tvf dist/target/rapids-4-spark_2.12-23.12.0-SNAPSHOT-cuda11.jar | grep -c spark3xx
0 |
Beta Was this translation helpful? Give feedback.
-
|
@gerashegalov Here is the printscreen after executing this command: 
|
Beta Was this translation helpful? Give feedback.
-
|
with the new jar you built can you place it back into the $SPARK_HOME/jars/ directory and try it there? remove it from the --jars parameter. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks a lot for confirming @LIN-Yu-Ting. Can we try one more thing? Can you start the thrift server with additional params to enable verbose classloading: and grep the the thrift server / Driver log for rapids-4-spark jar to rule out additional jars on the classpath In branch-24.02 we also have a new feature of detecting duplicate jars automatically #9654. You may want to try this #9867 (comment) again but with HEAD of branch-24.02. You can try the default but better add |
Beta Was this translation helpful? Give feedback.
-
|
I was able to reproduce this I confirmed that it goes away with the simple Even with the simple jar we have While the workaround for |
Beta Was this translation helpful? Give feedback.
-
|
I confirmed that with simple -DallowConventionalDistJar=true build and with a static classpath $SPARK_HOME/jar, there is no NoClassDefFoundError anymore. Thanks a lot @gerashegalov. |
Beta Was this translation helpful? Give feedback.
-
|
Going back to our multi-shim production jar, It looks like there is a race condition that affects all the sessions if the user classloader deployment --jars is used. After So another workaround is to run a single session beeline before allowing traffic to the thrift server from superset. We should review the usage of lazy vals |
Beta Was this translation helpful? Give feedback.
-
|
Is there a long term fix for this? I am running into this problem and it is painful and cryptic. This workaround doesn't work when using Nessie/Iceberg jars because it complains about not being able to parse the response - probably something to do with the nessie/iceberg syntax. |
Beta Was this translation helpful? Give feedback.
-
|
I have worked around this issue by putting the jars in |
Beta Was this translation helpful? Give feedback.
Going back to our multi-shim production jar, It looks like there is a race condition that affects all the sessions if the user classloader deployment --jars is used.
After
supersetsessions make GpuOverrides throw NoClassDeFoundErrors, connecting a singlebeelinesession reproduces the same NoClassDeFoundError. The good news is that the reverse is also true. After the HiveThriftServer2 is "pre-warmed" with a single session from beeline, superset's metadata queries start succeeding without NoClassDefFoundError.So another workaround is to run a single session beeline