[TOREE-552] Downgrade Guava 14.0.1 to align with Spark #220
Conversation
| val sparkAll = Def.setting{ | ||
| Seq( | ||
| sparkCore.value % "provided" excludeAll( | ||
| // Exclude netty (org.jboss.netty is for 3.2.2.Final only) |
There was a problem hiding this comment.
The recent version of Spark does not depend on Netty 3.x, so this exclusion is redundant.
https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.12/3.3.2
pan3793
changed the title
|
CI reports failures, investigating |
| // This class is derived from Guava v32.1.2 | ||
| // https://github.com/google/guava/blob/v32.1.2/guava/src/com/google/common/reflect/ClassPath.java | ||
| // | ||
| // TOREE-552: Apache Toree should use Guava 14 to align with Apache Spark, unfortunately, ClassPath | ||
| // from Guava 14 does not work well with Java 11+. | ||
| // See detail at: https://github.com/google/guava/issues/3249 |
There was a problem hiding this comment.
This is a simple copy with limited changes:
- switch
Loggerfrom JDK logging to SFL4J - directly call
System.getPropertyto getpath.separatorandjava.class.pathinstead of usingcom.google.common.base.StandardSystemProperty(since Guava 15)
There was a problem hiding this comment.
This looks like an exception because it is the first Java file in the project, though sbt works well with Java source files, if requested, I can translate it to Scala.
pan3793
changed the title
|
@lresende would you mind taking a look? |
|
So, #194 updated Guava, and why now, when we downgrade to the same previous version, do we need some new workarounds? If Spark uses this, can't we import the additional files from Spark? |
Yes, I explained in the PR description. "As a workaround, this PR fork a
In short, Spark 3.x must use Guava 14.0.1. The restriction comes from the transitive deps of Spark, including Hive and Hadoop, see details in discussions in SPARK-44811 SPARK-36676. Since Guava 14's |
What changes were proposed in this pull request?
This PR proposes to downgrade Guava 14.0.1 to align with Apache Spark.
The previous Guava upgrade is to make the
ClassPathcompatible with Java 11, google/guava#3249. As a workaround, this PR fork aClassPathfrom Guava v32.1.2 to make it happy with Java 11.Why are the changes needed?
Upgrading Guava in Spark is a "Mission Impossible" at the current stage. See details at SPARK-44811 SPARK-36676
The current tests do not cover the use cases of Spark with Hive, and based on the above Spark tickets, the Guava upgrading would break Hive usage