Error starting webserver at spark shell

[vaibhavkhanduja]

I followed the steps as mentioned in front Readme but when starting "start()" at the shell prompt error is thrown. Any pointers?

start()
java.lang.NoClassDefFoundError: Could not initialize class de.frosner.dds.core.DDS$
at de.frosner.dds.webui.server.SprayServer$.start(SprayServer.scala:183)
at de.frosner.dds.webui.server.SprayServer$.start(SprayServer.scala:172)
... 52 elided

[FRosner]

Hi @vaibhavkhanduja thanks for reporting this.

I haven't used DDS in a while but I'll try to help you. Can you provide me with the command you used to start it?

What version of Spark and DDS are you using?

[vaibhavkhanduja]

I checked out latest version of spark, and compiled the spark shell. Checked out spawncamping-dds from github and compiled all with scala 11.8. Did following:

a) /bin/spark-shell --jars spawncamping-dds-_.jar
Replaced jars above with one in my local dir

b) import de.frosner.dds.core.DDS._
import de.frosner.dds.webui.server.SprayServer._

c) start()

to see above error ...

[FRosner]

What is the DDS jar in your local dir exactly? Can you give your full spark-shell command? Where did you get the jar from?

Do you have the full stacktrace?

[FRosner]

I haven't actually tested against any new version of Spark because the project is kind of on hold. Sorry for that.

[FRosner]

Does it also happen with Scala 2.10?

[vaibhavkhanduja]

yes same with Scala 2.10

What is the DDS jar in your local dir exactly? Can you give your full spark-shell command? Where did you get the jar from?

I build DDS jars both for 2.10 and 2.11

./bin/spark-shell --jars ./spawncamping-dds/target/scala-2.10/spawncamping-dds-4.0.0-gamma-SNAPSHOT_2.10.jar

[FRosner]

Please try combining the Scala versions in a matching way. You mentioned that you compiled Spark against Scala 2.11 so please try to use DDS with 2.11 as well. They of course need to be the same. Let me know if it works :) -- Sent from my phone. Please excuse my brevity. On 12/24/2016, 02:16 Vaibhav Khanduja <[email protected]> wrote: yes same with Scala 2.10 What is the DDS jar in your local dir exactly? Can you give your full spark-shell command? Where did you get the jar from? I build DDS jars both for 2.10 and 2.11 ./bin/spark-shell --jars ./spawncamping-dds/target/scala-2.10/spawncamping-dds-4.0.0-gamma-SNAPSHOT_2.10.jar — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[FRosner]

Also try with a pre-built jar please (https://github.com/FRosner/spawncamping-dds/releases) to avoid building it wrongly or some problem in the master branch.

Please make sure the Scala versions of DDS and Spark match.

[vaibhavkhanduja]

Nope same error 👎 ... I am using scala 2.11 with spark and same with DDS. BTW, fyi this is on OSX MaC. May be I am missing some simple configuration?

[vaibhavkhanduja]

@FRosner this is larger stack trace .. if it helps

java.lang.NoClassDefFoundError: org/apache/spark/sql/DataFrame
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2688)
at java.lang.Class.getDeclaredMethod(Class.java:2115)
at org.reflections.util.Utils.getMemberFromDescriptor(Utils.java:88)
at org.reflections.util.Utils.getMethodsFromDescriptors(Utils.java:101)
at org.reflections.Reflections.getMethodsAnnotatedWith(Reflections.java:443)
at de.frosner.replhelper.Helper.(Helper.scala:30)
at de.frosner.replhelper.Helper$.apply(Helper.scala:134)
at de.frosner.dds.core.DDS$.(DDS.scala:40)
at de.frosner.dds.core.DDS$.(DDS.scala)
at de.frosner.dds.webui.server.SprayServer$.start(SprayServer.scala:183)
at de.frosner.dds.webui.server.SprayServer$.start(SprayServer.scala:172)
... 52 elided
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.DataFrame
at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 64 more

[vaibhavkhanduja]

From stack overflow:

"In Spark 2.0 class DataFrame was removed, currently there is only type DataFrame = Dataset[Row]. Please change dependency version to 2.0 or change cluster version to < 2.0"

http://stackoverflow.com/questions/39635163/spark-exception-in-thread-main-java-lang-noclassdeffounderror-org-apache-sp

[vaibhavkhanduja]

Well! finally got it working 👍

• Moved to Spark version 1.6.x
• Build DDS library with Scala version 2.10

Walah!!

@FRosner Thanks for your help here. We can probably keep this open to support Spark 2.x?

[FRosner]

I opened #296 for that but I am currently not able to spend any time on this project unfortunately.

[vaibhavkhanduja]

Are you guys open for a pull request? Thanks, Sent from my iPhone. Please excuse for typos and brevity of this message.

…

On Dec 25, 2016, at 4:41 AM, Frank Rosner ***@***.***> wrote: I opened #296 for that but I am currently not able to spend any time on this project unfortunately. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[FRosner]

Sure :) Are you seriously considering to use it or just trying out? -- Sent from my phone. Please excuse my brevity.

[vaibhavkhanduja]

For now trying out ... I like it is light and works with spark as it is out main tool Sent from my iPhone. Please excuse for typos and brevity of this message.

…

On Dec 25, 2016, at 9:40 AM, Frank Rosner ***@***.***> wrote: Sure :) Are you seriously considering to use it or just trying out? -- Sent from my phone. Please excuse my brevity. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[FRosner]

Sure so feel free to try it out and if you want to port it to Spark 2 I'm happy to merge it :) Next thing I need to port is https://github.com/FRosner/drunken-data-quality <https://github.com/FRosner/drunken-data-quality>