Support for Spark 2.3/2.4 in Dr.Elephant #683

ShubhamGupta29 · 2020-04-20T06:22:32Z

Currently, Dr.Elephant at max supports Spark 2.2.3. We need to support the latest versions of Spark(at least 2.3 and 2.4). This needs several changes, will update the issue as proceeds further.

mareksimunek · 2020-04-20T15:33:09Z

@ShubhamGupta29 Just FYI:
I managed to run https://github.com/songgane/dr-elephant/tree/feature/support_spark_2.x
and it worked with spark 2.3+.
There is couple of tests which needs to be fixed (I skipped them).

I have question:

It doesn't show executor memory used
Same for GC statistics

Is it because in SHS 2.3 it's not available or there is some work needed to see these metrics.

Anyway glad to see progress for 2.3+ spark.

ShubhamGupta29 · 2020-04-20T20:02:13Z

@mareksimunek I will surely go through your changes.
Just some questions:

which spark version did you use for compilation (was that change done in compile.conf)
are you fetching data from EventLogs or SHS's REST API

For your Heuristics related issues, I need to check how are you retrieving and transforming data

mareksimunek · 2020-04-21T09:51:16Z

I used hadoop 2.3.0 and spark 2.1.2
https://github.com/songgane/dr-elephant/blob/feature/support_spark_2.x/compile.conf

I think, I tried rebase to current master and with higher versions there were more failing tests so I stick with 2.1 and skipped less tests :).

I am fetching with:

  <fetcher>
    <applicationtype>spark</applicationtype>
    <classname>com.linkedin.drelephant.spark.fetchers.SparkFetcher</classname>
    <params>
      <use_rest_for_eventlogs>true</use_rest_for_eventlogs>
      <should_process_logs_locally>true</should_process_logs_locally>
      <event_log_location_uri>/spark2-history/</event_log_location_uri>
      <spark_log_ext>.snappy</spark_log_ext>
    </params>
  </fetcher>

ShubhamGupta29 · 2020-04-21T22:03:46Z

@mareksimunek the issue you mentioned It doesn't show executor memory used, is it for every job or for some jobs value is available?

mareksimunek · 2020-04-22T09:38:54Z

@ShubhamGupta29 every spark job, I suspect it's beacuse of this.
https://github.com/songgane/dr-elephant/blame/feature/support_spark_2.x/app/org/apache/spark/deploy/history/SparkDataCollection.scala#L178

That info.memUsed is only available when the job is running, but I am not sure if Dr.Elephant is fetching this information. And when the job ended the information is gone. Because when I check SHS of completed job there is everywhere Peak memory: 0.

Spark history verison : 2.3.0.2.6.5.0-292 (its 2.3 with some HDP patches)

xglv1985 · 2020-04-24T08:14:35Z

@ShubhamGupta29 every spark job, I suspect it's beacuse of this.
https://github.com/songgane/dr-elephant/blame/feature/support_spark_2.x/app/org/apache/spark/deploy/history/SparkDataCollection.scala#L178

That info.memUsed is only available when the job is running, but I am not sure if Dr.Elephant is fetching this information. And when the job ended the information is gone. Because when I check SHS of completed job there is everywhere Peak memory: 0.

Spark history verison : 2.3.0.2.6.5.0-292 (its 2.3 with some HDP patches)

@mareksimunek Hello, buddy, I met the same case as yours: mem/executor/storage info can not be fetched from SHS once the job end. Have your problem solved? Thanks.

ShubhamGupta29 · 2020-04-26T14:46:52Z

@mareksimunek @xglv1985 need one help from you guysin debugging the issue, can you confirm if the value of memUsed is Non-Zero in the response for REST API endpoint [/executors].

mareksimunek · 2020-04-26T19:17:34Z

@ShubhamGupta29 yep its zero.
Checked http://someHost:18081/api/v1/applications/application_1587409317223_1104/1/executors

[ {
    "id" : "driver",
    "hostPort" : "someHost:37121",
    "isActive" : true,
    "rddBlocks" : 0,
    "memoryUsed" : 0,
    "diskUsed" : 0,
    "totalCores" : 0,
    "maxTasks" : 0,
    "activeTasks" : 0,
    "failedTasks" : 0,
    "completedTasks" : 0,
    "totalTasks" : 0,
    "totalDuration" : 0,
    "totalGCTime" : 0,
    "totalInputBytes" : 0,
    "totalShuffleRead" : 0,
    "totalShuffleWrite" : 0,
    "isBlacklisted" : false,
    "maxMemory" : 407057203,
    "addTime" : "2020-04-25T21:08:51.911GMT",
    "executorLogs" : {
      "stdout" : "http://someHost:8042/node/containerlogs/container_e54_1587409317223_1104_01_000001/fulltext/stdout?start=-4096",
      "stderr" : "http://someHost:8042/node/containerlogs/container_e54_1587409317223_1104_01_000001/fulltext/stderr?start=-4096"
    },
    "memoryMetrics" : {
      "usedOnHeapStorageMemory" : 0,
      "usedOffHeapStorageMemory" : 0,
      "totalOnHeapStorageMemory" : 407057203,
      "totalOffHeapStorageMemory" : 0
    }
  }, {
    "id" : "9",
    "hostPort" : "someHost2.dev.dszn.cz:33108",
    "isActive" : true,
    "rddBlocks" : 0,
    "memoryUsed" : 0,
    "diskUsed" : 0,
    "totalCores" : 3,
    "maxTasks" : 3,
    "activeTasks" : 0,
    "failedTasks" : 0,
    "completedTasks" : 56,
    "totalTasks" : 56,
    "totalDuration" : 846816,
    "totalGCTime" : 31893,
    "totalInputBytes" : 0,
    "totalShuffleRead" : 661719258,
    "totalShuffleWrite" : 747129542,
    "isBlacklisted" : false,
    "maxMemory" : 3032481792,
    "addTime" : "2020-04-25T21:09:08.100GMT",
    "executorLogs" : {
      "stdout" : "http://someHost2.dev.dszn.cz:8042/node/containerlogs/container_e54_1587409317223_1104_01_000011/fulltext/stdout?start=-4096",
      "stderr" : "http://someHost2.dev.dszn.cz:8042/node/containerlogs/container_e54_1587409317223_1104_01_000011/fulltext/stderr?start=-4096"
    },
    "memoryMetrics" : {
      "usedOnHeapStorageMemory" : 0,
      "usedOffHeapStorageMemory" : 0,
      "totalOnHeapStorageMemory" : 3032481792,
      "totalOffHeapStorageMemory" : 0
    }
  }.....

Correction, its even reporting zero memoryUsed on running job trough SHS rest API. Should I set something to spark.executor.extraJavaOptions? To show me these stats?

From MR its getting memory stats from this setting. Am I right?
mapreduce.task.profile.params= -agentlib:hprof=cpu=samples,heap=sites,force=n,thread=y,verbose=n,file=%s

xglv1985 · 2020-04-27T03:24:23Z

@mareksimunek @xglv1985 need one help from you guysin debugging the issue, can you confirm if the value of memUsed is Non-Zero in the response for REST API endpoint [/executors].

@ShubhamGupta29 yes, I also proved it. The mem field is 0 in response json

ShubhamGupta29 · 2020-04-27T09:40:42Z

@mareksimunek @xglv1985 thanks for the prompt response, I am able to support Spark2.3 and will make the changes public soon. I am debugging this memUsed = 0 issue as this problem is still there with Spark2.3. I am debugging the issue and will be in contact with you guys.
One more query I have for you, can you paste here the values you are getting for these metrics in /executor API response.
Metrics:
"memoryMetrics" : { "usedOnHeapStorageMemory" "usedOffHeapStorageMemory" "totalOnHeapStorageMemory" "totalOffHeapStorageMemory" }

xglv1985 · 2020-04-27T09:45:01Z

@mareksimunek @xglv1985 thanks for the prompt response, I am able to support Spark2.3 and will make the changes public soon. I am debugging this memUsed = 0 issue as this problem is still there with Spark2.3. I am debugging the issue and will be in contact with you guys.
One more query I have for you, can you paste here the values you are getting for these metrics in /executor API response.
Metrics:
"memoryMetrics" : { "usedOnHeapStorageMemory" "usedOffHeapStorageMemory" "totalOnHeapStorageMemory" "totalOffHeapStorageMemory" }

sure:

"memoryMetrics" : {
"usedOnHeapStorageMemory" : 0,
"usedOffHeapStorageMemory" : 0,
"totalOnHeapStorageMemory" : 1099746508,
"totalOffHeapStorageMemory" : 4000000000
}

xglv1985 · 2020-04-27T09:54:58Z

@mareksimunek @xglv1985 thanks for the prompt response, I am able to support Spark2.3 and will make the changes public soon. I am debugging this memUsed = 0 issue as this problem is still there with Spark2.3. I am debugging the issue and will be in contact with you guys.
One more query I have for you, can you paste here the values you are getting for these metrics in /executor API response.
Metrics:
"memoryMetrics" : { "usedOnHeapStorageMemory" "usedOffHeapStorageMemory" "totalOnHeapStorageMemory" "totalOffHeapStorageMemory" }

By the way, @ShubhamGupta29
I use dr.elephant to analyze spark 2.3 event log, and every job analysis result is as the follow. I found except "Spark Configuration", every field is empty. Is this normal? Thanks!

Spark Configuration
Severity: Moderate [Explain]

Spark Executor Metrics
Severity: None

Executor input bytes distribution | min: 0 B, p25: 0 B, median: 0 B, p75: 0 B, max: 0 B
Executor shuffle read bytes distribution | min: 0 B, p25: 0 B, median: 0 B, p75: 0 B, max: 0 B
Executor shuffle write bytes distribution | min: 0 B, p25: 0 B, median: 0 B, p75: 0 B, max: 0 B
Executor storage memory used distribution | min: 0 B, p25: 0 B, median: 0 B, p75: 0 B, max: 0 B
Executor storage memory utilization rate | 0.000
Executor task time distribution | min: 0 sec, p25: 0 sec, median: 0 sec, p75: 0 sec, max: 0 sec
Executor task time sum | 0
Total executor storage memory allocated | 1.96 GB
Total executor storage memory used | 0 B

Spark Job Metrics
Severity: None

Spark completed jobs count | 0
Spark failed jobs count | 0
Spark failed jobs list |
Spark job failure rate | 0.000
Spark jobs with high task failure rates

Spark Stage Metrics
Severity: None

Spark completed stages count | 0
Spark failed stages count | 0
Spark stage failure rate | 0.000
Spark stages with high task failure rates |
Spark stages with long average executor runtimes

Executor GC
Severity: None

GC time to Executor Run time ratio | NaN
Total Executor Runtime | 0
Total GC time | 0

ShubhamGupta29 · 2020-04-27T10:14:05Z

@xglv1985 no this is not normal. Can you tell me which branch or source code are you using?

xglv1985 · 2020-04-27T10:19:19Z

@xglv1985 no this is not normal. Can you tell me which branch or source code are you using?

@ShubhamGupta29
dr-elephant_987

ShubhamGupta29 · 2020-04-27T10:31:56Z

Can you provide the link as linkedin/dr-elephant doesn't have any branch named dr-elephant_987

xglv1985 · 2020-04-27T10:44:37Z

Can you provide the link as linkedin/dr-elephant doesn't have any branch named dr-elephant_987

@ShubhamGupta29 I forked my own dr-elephant from linkedin/dr-elephant master. I only put "SparkFetcher" in my conf xml file, with <use_rest_for_eventlogs>true</use_rest_for_eventlogs>
<should_process_logs_locally>true</should_process_logs_locally>. Is there any other configuration that may cause these empty field? I will debug more deeply. Thanks

mareksimunek · 2020-04-27T12:17:29Z

@xglv1985 if you are using current master, you can't see any metrics from spark 2.3+
More in:
#389
Check your logs there will be some parsing error. That's why i am using fork as said above.

That's why there is ongoing work from @ShubhamGupta29 to support this version.

Metrics:
"memoryMetrics" : { "usedOnHeapStorageMemory" "usedOffHeapStorageMemory" "totalOnHeapStorageMemory" "totalOffHeapStorageMemory" }

Thanks for update @ShubhamGupta29 They are already included in my post
#683 (comment)

xglv1985 · 2020-04-27T12:32:40Z

@xglv1985 if you are using current master, you can't see any metrics from spark 2.3+
More in:
#389
Check your logs there will be some parsing error. That's why i am using fork as said above.

That's why there is ongoing work from @ShubhamGupta29 to support this version.

Metrics:
"memoryMetrics" : { "usedOnHeapStorageMemory" "usedOffHeapStorageMemory" "totalOnHeapStorageMemory" "totalOffHeapStorageMemory" }

Thanks for update @ShubhamGupta29 They are already included in my post
#683 (comment)

@mareksimunek Thanks very much, I saw the same problem with mine, in link you gave.
Then let's looking forward to the updated dr-elephant by @ShubhamGupta29

ShubhamGupta29 · 2020-04-28T12:06:32Z

@mareksimunek @xglv1985, I have made the changes for Spark2.3 (these are the foundation changes, will fix the tests and other cleanups in some time). If possible can you guys try this personal branch, it has changes for Spark2.3.

mareksimunek · 2020-04-29T19:36:04Z

@ShubhamGupta29 nice, the ShubhamGupta29/test23 works like a charm. It now even shows GC stats.
executor memory used still not showing, but I suppose if it's not available in SHS it won't be seen in elephant. (Do you have any news if there is something to do to make it available in SHS)

ShubhamGupta29 · 2020-04-30T02:08:37Z

@mareksimunek working on the same, after going through Spark's code got some idea of why this metric is not getting populated. For now, testing the changes and soon add those to the branch and also trying to support Spark 2.4 too.
@mareksimunek and @xglv1985, can you guys fill the survey in #685, it would be helpful for us to make Dr.Elephant more OS community-friendly.

xglv1985 · 2020-04-30T02:11:45Z

ok I saw it yesterday and I will fill the survey today. | | hikari | | 邮箱：[email protected] | Signature is customized by Netease Mail Master On 04/30/2020 10:08, Shubham Gupta wrote: @mareksimunek working on the same, after going through Spark's code got some idea of why this metric is not getting populated. For now, testing the changes and soon add those to the branch and also trying to support Spark 2.4 too. @mareksimunek and @xglv1985, can you guys fill the survey in #685, it would be helpful for us to make Dr.Elephant more OS community-friendly. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

ShubhamGupta29 · 2020-04-30T06:36:53Z

@xglv1985 did you get a chance to use the changes done for Spark2.3? Feedback for the changes will make it easy to start the effort for merging the changes to the master branch for users' ease.

xglv1985 · 2020-04-30T08:33:06Z

sure, I will try your personal branch, and will feedback to you during May 1st to 5th. At 2020-04-30 14:37:06, "Shubham Gupta" <[email protected]> wrote: @xglv1985 did you get a chance to use the changes done for Spark2.3? Feedback for the changes will make it easy to start the effort for merging the changes to the master branch for users' ease. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

ShubhamGupta29 · 2020-05-18T05:57:56Z

@mareksimunek and @xglv1985 I have made some more changes for Spark 2.3 support, kindly try this branch whenever you guys have time. Also for memory heuristics, there is a change needed in Spark Conf, add spark.eventLog.logBlockUpdates.enabled if not there already and make this value as true.

mareksimunek · 2020-05-22T22:27:44Z

@ShubhamGupta29
Hi, thanks for update and sorry for late response.

I needed delete some tests to compile.

	deleted:    test/com/linkedin/drelephant/tony/fetchers/TonyFetcherTest.java
	deleted:    test/com/linkedin/drelephant/tuning/PSOParamGeneratorTest.java

I missed that it's storage memory which is showing cached RDDs. And it seems it works fine, after setting spark.eventLog.logBlockUpdates.enabled to job :)
Is there also way to add peak memory? In which I most interested in.
I noticed in event log there is:
{"ID":111,"Name":"internal.metrics.peakExecutionMemory","Update":96381057,"Value":96381057,"Internal":true,"Count Failed Values":true}

ShubhamGupta29 · 2020-05-23T15:48:41Z

@mareksimunek thanks for the reply and testing out the provided version.

TonYFetcherTest doesn't work when compiling locally so it makes sense to remove it. For PSOParamGeneratorTest if you want to try you can fix it by doing pip install inspyred as it fixed the test for me.
Glad after setting spark.eventLog.logBlockUpdates.enabled the metric is getting populated for you.
I am also looking for a way to provide this metric (Peak Memory Used), can you provide me the event name from which you got this metrics(internal.metrics.peakExecutionMemory).

Also, let me know any other issue you are facing or any suggestion you have for Dr.Elephant. Hope Dr.Elephant is proving useful for you and your team.

mareksimunek · 2020-05-23T17:01:45Z

@ShubhamGupta29

Event name for internal.metrics.peakExecutionMemory used is called "Event":"SparkListenerTaskEnd"
But I am not sure if thats it, only juding by its name :). I attached event log from the job.
eventLogs-application_1587409317223_6508-1.zip

So far it seems it's working like a charm. I am trying to push it through in our team (now its running on small testing cluster) and with working spark metrics it will be much easier to get approval to work on that, thanks for the progress.

Question: Are you using 1 dr elephant installation per cluster or do you have 1 dr elephant analyzing more clusters.

ShubhamGupta29 · 2020-05-23T18:27:55Z

The current Dr.Elephant allow the analysis of jobs only from single RM(single cluster).

xglv1985 · 2020-05-26T03:26:56Z

@mareksimunek and @xglv1985 I have made some more changes for Spark 2.3 support, kindly try this branch whenever you guys have time. Also for memory heuristics, there is a change needed in Spark Conf, add spark.eventLog.logBlockUpdates.enabled if not there already and make this value as true.

@ShubhamGupta29 First sorry for the late response. Thanking for your branch feature_spark2.3, I now have run it up. This is my screen capture:

The good new is that it has more dimensions than the past versions of dr.elephant. But the detail of each dimension has disappeared, and I will double check the configuration.

xglv1985 · 2020-05-27T09:24:21Z

@mareksimunek and @xglv1985 I have made some more changes for Spark 2.3 support, kindly try this branch whenever you guys have time. Also for memory heuristics, there is a change needed in Spark Conf, add spark.eventLog.logBlockUpdates.enabled if not there already and make this value as true.

@ShubhamGupta29 First sorry for the late response. Thanking for your branch feature_spark2.3, I now have run it up. This is my screen capture:

The good new is that it has more dimensions than the past versions of dr.elephant. But the detail of each dimension has disappeared, and I will double check the configuration.

I have found the reason why the details disappeared. The "feature_2.3" branch use "org.avaje.ebeanorm.avaje-ebeanorm-3.2.4.jar" and "org.avaje.ebeanorm.avaje-ebeanorm-agent-3.2.2.jar", which will lead to details missing. I replaced these two jars with
''avaje-ebeanorm-3.2.2.jar" and "avaje-ebeanorm-agent-3.2.1.jar", which the old version dr.elephant depends on, and then the details came back :)

ShubhamGupta29 · 2020-05-27T13:35:07Z

@mareksimunek and @xglv1985 I have made some more changes for Spark 2.3 support, kindly try this branch whenever you guys have time. Also for memory heuristics, there is a change needed in Spark Conf, add spark.eventLog.logBlockUpdates.enabled if not there already and make this value as true.

@ShubhamGupta29 First sorry for the late response. Thanking for your branch feature_spark2.3, I now have run it up. This is my screen capture:

The good new is that it has more dimensions than the past versions of dr.elephant. But the detail of each dimension has disappeared, and I will double check the configuration.

I have found the reason why the details disappeared. The "feature_2.3" branch use "org.avaje.ebeanorm.avaje-ebeanorm-3.2.4.jar" and "org.avaje.ebeanorm.avaje-ebeanorm-agent-3.2.2.jar", which will lead to details missing. I replaced these two jars with
''avaje-ebeanorm-3.2.2.jar" and "avaje-ebeanorm-agent-3.2.1.jar", which the old version dr.elephant depends on, and then the details came back :)

@xglv1985 did you replace this in the ClassPath?

xglv1985 · 2020-05-27T13:39:08Z

@mareksimunek and @xglv1985 I have made some more changes for Spark 2.3 support, kindly try this branch whenever you guys have time. Also for memory heuristics, there is a change needed in Spark Conf, add spark.eventLog.logBlockUpdates.enabled if not there already and make this value as true.

@ShubhamGupta29 First sorry for the late response. Thanking for your branch feature_spark2.3, I now have run it up. This is my screen capture:

The good new is that it has more dimensions than the past versions of dr.elephant. But the detail of each dimension has disappeared, and I will double check the configuration.

I have found the reason why the details disappeared. The "feature_2.3" branch use "org.avaje.ebeanorm.avaje-ebeanorm-3.2.4.jar" and "org.avaje.ebeanorm.avaje-ebeanorm-agent-3.2.2.jar", which will lead to details missing. I replaced these two jars with
''avaje-ebeanorm-3.2.2.jar" and "avaje-ebeanorm-agent-3.2.1.jar", which the old version dr.elephant depends on, and then the details came back :)

@xglv1985 did you replace this in the ClassPath?

Yes, I did.

mareksimunek · 2020-05-27T15:19:22Z

@ShubhamGupta29 Hi, did you have time to look into PeakExecutionMemory ?

ShubhamGupta29 · 2020-05-28T18:24:07Z

@mareksimunek didn't get a chance to look into it but surely will do it over the weekend.

mareksimunek · 2020-07-16T13:47:54Z

@ShubhamGupta29 I know you are probably busy, but I hope you will find a way to look at it :).

ShubhamGupta29 · 2020-07-16T18:22:08Z

Hi @mareksimunek, sorry was caught up in some other tasks. I am working on an approximate value for PeakExecutionMemory as after a lot of finding I got to know that there is no way of getting this value with making changes to Spark source code. Possibly in the coming week, I will push the changes. Also, let me know if Dr.Elephant's support for Spark2.3 is working fine.

mareksimunek · 2020-07-17T08:59:44Z

@ShubhamGupta29 Support for spark 2.3 works fine :)).

RaphaelDucay · 2020-08-11T12:35:47Z

@ShubhamGupta29
First of all thanks for all your precious work in order to help adapting Dr elephant to newer spark versions.
I have an hortonworks cluster HDP 2.6.5 ( == hadoop 2.7.3) running :

spark 1.6.3 on yarn
spark 2.3.2 on yarn

I have a few questions :

Do you have a fully working branch / tag / project-fork that i could use in order to make dr elephant work on spark 2.3.2 (which uses scala 2.11 i remind) ?
If yes, could you just tell me if there is any changes in term of "config/packaging/deployment" different from the steps described in the "quick setup guide" (from the master branch) in order to make this work for spark 2.3.2 other than adapting the env variables and conf to make them point on correct spark 2.3.2 folders and config ?
Finally, is it possible to have one Dr Elephant instance that deals with spark 1.6.3 AND spark 2.3.2 applications and what would be the procedure to make that happen ?

Thanks a lot in advance for your time !

ShubhamGupta29 · 2020-08-14T18:02:41Z

Hi @RaphaelDucay ,
I will try to answer you queries in the respective order:

We have a branch which compiles and work for Spark2.3 also Scala_2.11 is default for Spark2.3, here is the link. There is nothing different from the Quick Guide to set this branch up. Just update the spark_version to 2.3.2 in compile.conf file.

This branch is in Beta and is currently used by considerable number of customers. Kindly provide your feedback too.

To keep things simple, I would say that single Dr.Elephant instance cannot have support for Spark_1.6 and Spark_2.3. Kindly tell us your use-case so we can assist in you better way.

RaphaelDucay · 2020-08-18T13:14:49Z

@ShubhamGupta29 Thanks a lot for the feeedback !
We are making a POC on both

master for spark 1.6.3
feature_spark2.3 for spark 2.3.2
So far we took the easy path (using 2 different Dr elephant instances)
We are struggling a little bit on implementing TLS support in order to allow DrElephant to communique with HDP services through TLS (same as) Dr Elephant does not work against YARN in HTTPS mode #472
We used Enabling Dr Elephant to work with YARN in HTTPS mode #475 so far and tried to make it work mimicking https://github.com/linkedin/dr-elephant/pull/475/files as we saw that test failures in the pull request (that still hasn't been merged btw) are only due to coverage
Btw, feel free to let us know if you have stable "beta" branches with
TLS for HDP
TLS for HDP + spark 2.3.2 support
as (so far) we do it by hand crossing fingers not to brake unit tests which we cant fully run because we had to comment same tests as Dr. Elephant is not compiled #514 in order to compile the master branch with hdp 2.7.3/ spark 1.6.3

I will keep you updated !

RaphaelDucay · 2020-08-25T10:13:09Z

@ShubhamGupta29 Ok so we made it for spark 1.6.3
For spark 2.X We are facing issues :
Finally our spark 2 version is 2.3.0
We are using (as you suggested me) this branch https://github.com/ShubhamGupta29/dr-elephant/tree/feature_spark2.3
He are the logs of our failing compilation attempt

[warn] /dr-elephant-sources/app/org/apache/spark/deploy/history/SparkDataCollection.scala:124: abstract type pattern T is unchecked since it is eliminated by erasure
[warn] seq.foreach { case (item: T) => list.add(item)}
[warn] ^
[error] /dr-elephant-sources/app/org/apache/spark/status/CustomAppStatusListener.scala:628: value getPartitions is not a member of org.apache.spark.status.LiveRDD
[error] liveRDD.getPartitions().foreach { case (_, part) =>
[error] ^
[error] /dr-elephant-sources/app/org/apache/spark/status/CustomAppStatusListener.scala:629: value executors is not a member of Any
[error] part.executors.foreach { executorId =>
[error] ^
[error] /dr-elephant-sources/app/org/apache/spark/status/CustomAppStatusListener.scala:639: value getDistributions is not a member of org.apache.spark.status.LiveRDD
[error] liveRDD.getDistributions().foreach { case (executorId, rddDist) =>
[error] ^
[error] /dr-elephant-sources/app/org/apache/spark/status/CustomAppStatusListener.scala:640: type mismatch;
[error] found : Any
[error] required: String
[error] liveExecutors.get(executorId).foreach { exec =>
[error] ^
[warn] one warning found
[error] four errors found
[error] (compile:compileIncremental) Compilation failed

Do you have an idea on how to fix this ?

Thanks in advance !

yanxiaole · 2020-09-24T10:18:01Z

@mareksimunek and @xglv1985 I have made some more changes for Spark 2.3 support, kindly try this branch whenever you guys have time. Also for memory heuristics, there is a change needed in Spark Conf, add spark.eventLog.logBlockUpdates.enabled if not there already and make this value as true.

@ShubhamGupta29 First sorry for the late response. Thanking for your branch feature_spark2.3, I now have run it up. This is my screen capture:

The good new is that it has more dimensions than the past versions of dr.elephant. But the detail of each dimension has disappeared, and I will double check the configuration.

I have found the reason why the details disappeared. The "feature_2.3" branch use "org.avaje.ebeanorm.avaje-ebeanorm-3.2.4.jar" and "org.avaje.ebeanorm.avaje-ebeanorm-agent-3.2.2.jar", which will lead to details missing. I replaced these two jars with
''avaje-ebeanorm-3.2.2.jar" and "avaje-ebeanorm-agent-3.2.1.jar", which the old version dr.elephant depends on, and then the details came back :)

@xglv1985 did you replace this in the ClassPath?

Yes, I did.

hi @ShubhamGupta29 , @xglv1985
is there a more suitable way? right now it seems the jars have to be renamed...

shagneet330 · 2021-02-02T11:01:01Z

hi @ShubhamGupta29 , @xglv1985 ,
Facing the same issue. Not able to double click on these metric dimensions. Using feature_2.3 branch. How can this be corrected?

ShubhamGupta29 · 2021-02-02T17:00:43Z

Hi @shagneet330 the fix is provided in the above comment.
I have found the reason why the details disappeared. The "feature_2.3" branch use "org.avaje.ebeanorm.avaje-ebeanorm-3.2.4.jar" and "org.avaje.ebeanorm.avaje-ebeanorm-agent-3.2.2.jar", which will lead to details missing. I replaced these two jars with ''avaje-ebeanorm-3.2.2.jar" and "avaje-ebeanorm-agent-3.2.1.jar", which the old version dr.elephant depends on, and then the details came back :)

But I would suggest using the latest Ember UI, which would be available if your compilation went well. You can access the new UI by adding new# after Dr.E endpoint. e.g. http://hostname:8080/new#, if you can see UI like below then there should be no issue.

shagneet330 · 2021-02-02T17:07:18Z

@ShubhamGupta29 Tried with http://hostname:8080/new# but this doesn't seem to load. Are these changes available in "feature_2.3" branch?

ShubhamGupta29 · 2021-02-03T05:38:03Z

It is available in feature_2.3, but your compilation should be successful with Ember. NPM should be available in your system. You see this kind of log during compilation:

`
"############################################################################"

"npm installation found, we'll compile with the new user interface"
"############################################################################"
`
Need to monitor the compilation as you face issues while compiling new UI.

tcluzhe · 2021-02-05T02:44:11Z

@ShubhamGupta29 would you help add spark application name in the new UI？

Ashnee1990 · 2021-07-08T19:19:37Z

@ShubhamGupta29,
I am facing issue while compiling .the feature_spark2.3, master and finalspark23 branch. Could you please help.

Below is the error.

elephant_spark23/dr-elephant/app/org/apache/spark/status/CustomAppStatusListener.scala:628: value getPartitions is not a member of org.apache.spark.status.LiveRDD
[error] liveRDD.getPartitions().foreach { case (_, part) =>
[error] ^
[error] elephant_spark23/dr-elephant/app/org/apache/spark/status/CustomAppStatusListener.scala:629: value executors is not a member of Any
[error] part.executors.foreach { executorId =>
[error] ^
[error] elephant_spark23/dr-elephant/app/org/apache/spark/status/CustomAppStatusListener.scala:639: value getDistributions is not a member of org.apache.spark.status.LiveRDD
[error] liveRDD.getDistributions().foreach { case (executorId, rddDist) =>
[error] ^
[error] elephant_spark23/dr-elephant/app/org/apache/spark/status/CustomAppStatusListener.scala:640: type mismatch;
[error] found : Any
[error] required: String
[error] liveExecutors.get(executorId).foreach { exec =>
[error] ^
[warn] one warning found
[error] four errors found
[error] (compile:compileIncremental) Compilation failed

Ashnee1990 · 2021-07-13T21:26:43Z

@ShubhamGupta29 @xglv1985 ,

Do we have resolved Peak Memory Used . Because I am still not able to see it.

ShubhamGupta29 added the feature request label Apr 20, 2020

ShubhamGupta29 self-assigned this Apr 20, 2020

ShubhamGupta29 changed the title ~~Support for Spark 2.4 in Dr.Elephant~~ Support for Spark 2.3/2.4 in Dr.Elephant Apr 20, 2020

ShubhamGupta29 mentioned this issue May 23, 2020

Tracking peak memory usage #689

Closed

Support for Spark 2.3/2.4 in Dr.Elephant #683

Support for Spark 2.3/2.4 in Dr.Elephant #683

Comments

ShubhamGupta29 commented Apr 20, 2020

mareksimunek commented Apr 20, 2020

ShubhamGupta29 commented Apr 20, 2020

mareksimunek commented Apr 21, 2020

ShubhamGupta29 commented Apr 21, 2020

mareksimunek commented Apr 22, 2020 • edited Loading

xglv1985 commented Apr 24, 2020

ShubhamGupta29 commented Apr 26, 2020

mareksimunek commented Apr 26, 2020 • edited Loading

xglv1985 commented Apr 27, 2020

ShubhamGupta29 commented Apr 27, 2020

xglv1985 commented Apr 27, 2020

xglv1985 commented Apr 27, 2020

Spark Configuration Severity: Moderate [Explain]

Spark Executor Metrics Severity: None

Spark Job Metrics Severity: None

Spark Stage Metrics Severity: None

Executor GC Severity: None

ShubhamGupta29 commented Apr 27, 2020

xglv1985 commented Apr 27, 2020

ShubhamGupta29 commented Apr 27, 2020

xglv1985 commented Apr 27, 2020

mareksimunek commented Apr 27, 2020

xglv1985 commented Apr 27, 2020

ShubhamGupta29 commented Apr 28, 2020 • edited Loading

mareksimunek commented Apr 29, 2020

ShubhamGupta29 commented Apr 30, 2020

xglv1985 commented Apr 30, 2020 via email

ShubhamGupta29 commented Apr 30, 2020

xglv1985 commented Apr 30, 2020 via email

ShubhamGupta29 commented May 18, 2020

mareksimunek commented May 22, 2020

ShubhamGupta29 commented May 23, 2020

mareksimunek commented May 23, 2020

ShubhamGupta29 commented May 23, 2020

xglv1985 commented May 26, 2020

xglv1985 commented May 27, 2020 • edited Loading

ShubhamGupta29 commented May 27, 2020

xglv1985 commented May 27, 2020

mareksimunek commented May 27, 2020

ShubhamGupta29 commented May 28, 2020 • edited Loading

mareksimunek commented Jul 16, 2020

ShubhamGupta29 commented Jul 16, 2020

mareksimunek commented Jul 17, 2020

RaphaelDucay commented Aug 11, 2020

ShubhamGupta29 commented Aug 14, 2020

RaphaelDucay commented Aug 18, 2020 • edited Loading

RaphaelDucay commented Aug 25, 2020 • edited Loading

yanxiaole commented Sep 24, 2020

shagneet330 commented Feb 2, 2021

ShubhamGupta29 commented Feb 2, 2021

shagneet330 commented Feb 2, 2021

ShubhamGupta29 commented Feb 3, 2021 • edited Loading

tcluzhe commented Feb 5, 2021

Ashnee1990 commented Jul 8, 2021 • edited Loading

Ashnee1990 commented Jul 13, 2021 • edited Loading

mareksimunek commented Apr 22, 2020 •

edited

Loading

mareksimunek commented Apr 26, 2020 •

edited

Loading

Spark Configuration
Severity: Moderate [Explain]

Spark Executor Metrics
Severity: None

Spark Job Metrics
Severity: None

Spark Stage Metrics
Severity: None

Executor GC
Severity: None

ShubhamGupta29 commented Apr 28, 2020 •

edited

Loading

xglv1985 commented May 27, 2020 •

edited

Loading

ShubhamGupta29 commented May 28, 2020 •

edited

Loading

RaphaelDucay commented Aug 18, 2020 •

edited

Loading

RaphaelDucay commented Aug 25, 2020 •

edited

Loading

ShubhamGupta29 commented Feb 3, 2021 •

edited

Loading

Ashnee1990 commented Jul 8, 2021 •

edited

Loading

Ashnee1990 commented Jul 13, 2021 •

edited

Loading