Dr Elephant on Cloudera

[sri840]

Hi All,

I want to install Dr elephant in our clouder 6.3.2 version, can any one provide me the installation steps for installing in dr elephant ?

Below is the version in our CDH cluster
hadoop version
Hadoop 3.0.0-cdh6.3.2

When we have given this version in Compile.sh and Compile.conf we are getting below error

[error] (*:update) sbt.ResolveException: unresolved dependency: org.apache.hadoop#hadoop-mapreduce-client-core;3.0.0-cdh6.3.2: not found
[error] unresolved dependency: org.apache.hadoop#hadoop-mapreduce-client-common;3.0.0-cdh6.3.2: not found
[error] unresolved dependency: org.apache.hadoop#hadoop-common;3.0.0-cdh6.3.2: not found
[error] unresolved dependency: org.apache.hadoop#hadoop-hdfs;3.0.0-cdh6.3.2: not found
[error] unresolved dependency: org.apache.hadoop#hadoop-auth;3.0.0-cdh6.3.2: not found
[error] Total time: 25 s, completed Apr 12, 2020 2:46:11 PM

[mareksimunek]

Try to compile it without specifying hadoop and spark versions (just with default)
In @ShubhamGupta29 said:

Dr.Elephant currently doesn't support Hadoop3 with ATS v2. But you can use Dr.E with Hadoop3 in prod given that you Yarn REST APIs and history servers are in sync with what Dr.Elephant is excepting.

more in #657

There could be more info in wiki... I had huge headache to make it run with spark 2.

[sri840]

@ShubhamGupta29

Could you please provide the steps for installing Dr elephant on CDH 6.3.1 version ?

Could you please also tell me how to sync Yarn REST APIs and history servers.

[ShubhamGupta29]

Hi @sri840, some basic queries;

• What changes have you made to the source code/dependencies?

• What Spark version you are using?

• Have you cloned Dr.Elephant's master branch?

• What are the current issues apart from mentioned above?

[sri840]

@ShubhamGupta29

I didnt make any changes to source code.
Below are the versions

CDH VERSION :-CDH6.3.1
spark 2.4.0

Could you please me the steps for cloning Dr.Elephant's master branch?

Could you please provide the steps for starting for installing Dr Elephant in our environment ? Currently we are migrating from SQL server to Big data Environment. We are using spark applications and we want to get recommendations for Spark jobs .

Thanks in advance for helping me.

Thanks
Srikanth

[sri840]

@ShubhamGupta29

Can you please provide update on above request ?

Thanks
Srikanth

[ShubhamGupta29]

@sri840
steps for setting up master branch is same as mentioned here.
Compile Dr.E with Spark 1.4 and Hadoop 2.3 only, this way it would work.
By the error log with which you opened the issue, it seems like you changed the Hadoop version, kindly compile with stock versions only for now.

[sri840]

@ShubhamGupta29

Thanks for providing the information. I will let you know if i face any issues.

[ShubhamGupta29]

@sri840 sure.

[sri840]

@ShubhamGupta29

Just want to check with you , after compile Dr.E with spark 1.4 and Hadoop 2.3 , can we use it in Spark 2.4 and Cdh 6.3.1 version ?

Thanks
Srikanth

[ShubhamGupta29]

@sri840 it should work fine, we are using it in the same way and it works fine. One note: that we are using the REST APIs of Spark to fetch metrics and using it successfully depends a lot on your Spark History Server's performance. But you can increase the timeout for your REST requests. Feel free to reach out if you face issues.
Also, let me know if we can modify anything in the documentation of Dr.Elephant in case you faced issues which can be resolved by proper documentation

[sri840]

@ShubhamGupta29
I am getting below error while compling the Dr elephant, Can you please provide video or url how to install Dr elephant from end to end ?

[info] *** 6 TESTS FAILED ***
[error] Failed: Total 474, Failed 45, Errors 0, Passed 428, Skipped 1
[error] Failed tests:
[error] com.linkedin.drelephant.tuning.PSOParamGeneratorTest
[error] rest.RestAPITest
[error] com.linkedin.drelephant.BackfillTest
[error] controllers.MetricsControllerTest
[error] com.linkedin.drelephant.tony.fetchers.TonyFetcherTest
[error] com.linkedin.drelephant.util.SparkUtilsTest
[error] (test:test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 36 s, completed May 16, 2020 9:25:22 PM
Build failed...

[ShubhamGupta29]

@ShubhamGupta29
I am getting below error while compling the Dr elephant, Can you please provide video or url how to install Dr elephant from end to end ?

[info] *** 6 TESTS FAILED ***
[error] Failed: Total 474, Failed 45, Errors 0, Passed 428, Skipped 1
[error] Failed tests:
[error] com.linkedin.drelephant.tuning.PSOParamGeneratorTest
[error] rest.RestAPITest
[error] com.linkedin.drelephant.BackfillTest
[error] controllers.MetricsControllerTest
[error] com.linkedin.drelephant.tony.fetchers.TonyFetcherTest
[error] com.linkedin.drelephant.util.SparkUtilsTest
[error] (test:test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 36 s, completed May 16, 2020 9:25:22 PM
Build failed...

Hi @sri840,
Tests for PSOParamGeneratorTest and TonyFetcherTest are commonly seen, but not sure why BackfillTest, SparkUtilsTest, MetricsControllerTest, RestAPITest are failing. Did you change anything in the code?