Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TEZ-4549. Upgrade Hadoop Version to 3.4.0. #342

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

TEZ-4549. Upgrade Hadoop Version to 3.4.0. #342

wants to merge 1 commit into from
+1 −1

Conversation

slfan1989
Copy link
Contributor

@slfan1989 slfan1989 commented Apr 3, 2024 •
edited
Loading

JIRA: TEZ-4549. Upgrade Hadoop Version to 3.4.0.

hadoop has been upgraded to 3.4.0, try to upgrade hadoop to 3.4.0.

Local compilation successful.

[INFO] tez ................................................ SUCCESS [  2.206 s]
[INFO] hadoop-shim ........................................ SUCCESS [ 27.433 s]
[INFO] tez-api ............................................ SUCCESS [ 36.299 s]
[INFO] tez-build-tools .................................... SUCCESS [  0.233 s]
[INFO] tez-common ......................................... SUCCESS [  2.093 s]
[INFO] tez-runtime-internals .............................. SUCCESS [  8.123 s]
[INFO] tez-runtime-library ................................ SUCCESS [ 16.348 s]
[INFO] tez-mapreduce ...................................... SUCCESS [ 10.233 s]
[INFO] tez-examples ....................................... SUCCESS [  1.769 s]
[INFO] tez-dag ............................................ SUCCESS [ 18.068 s]
[INFO] tez-tests .......................................... SUCCESS [ 18.191 s]
[INFO] tez-ext-service-tests .............................. SUCCESS [  6.580 s]
[INFO] tez-ui ............................................. SUCCESS [ 46.579 s]
[INFO] tez-plugins ........................................ SUCCESS [  0.469 s]
[INFO] tez-protobuf-history-plugin ........................ SUCCESS [ 11.401 s]
[INFO] tez-yarn-timeline-history .......................... SUCCESS [  5.569 s]
[INFO] tez-yarn-timeline-history-with-acls ................ SUCCESS [  4.206 s]
[INFO] tez-yarn-timeline-cache-plugin ..................... SUCCESS [ 37.988 s]
[INFO] tez-yarn-timeline-history-with-fs .................. SUCCESS [  3.089 s]
[INFO] tez-history-parser ................................. SUCCESS [ 16.163 s]
[INFO] tez-aux-services ................................... SUCCESS [  8.207 s]
[INFO] tez-tools .......................................... SUCCESS [  0.103 s]
[INFO] tez-perf-analyzer .................................. SUCCESS [  0.057 s]
[INFO] tez-job-analyzer ................................... SUCCESS [  2.805 s]
[INFO] tez-javadoc-tools .................................. SUCCESS [  0.905 s]
[INFO] hadoop-shim-impls .................................. SUCCESS [  0.058 s]
[INFO] hadoop-shim-2.8 .................................... SUCCESS [  1.576 s]
[INFO] tez-dist ........................................... SUCCESS [ 30.981 s]
[INFO] Tez ................................................ SUCCESS [  0.287 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  05:18 min
[INFO] Finished at: 2024-04-03T22:58:57+08:00
[INFO] ------------------------------------------------------------------------

@tez-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 37s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+1 💚 mvninstall 18m 7s master passed
+1 💚 compile 2m 28s master passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1
+1 💚 compile 2m 13s master passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06
+1 💚 javadoc 1m 50s master passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1
+1 💚 javadoc 1m 11s master passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06
_ Patch Compile Tests _
+1 💚 mvninstall 6m 11s the patch passed
+1 💚 compile 2m 34s the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1
+1 💚 javac 2m 34s the patch passed
+1 💚 compile 2m 18s the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06
+1 💚 javac 2m 18s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 xml 0m 1s The patch has no ill-formed XML file.
+1 💚 javadoc 1m 15s the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1
+1 💚 javadoc 1m 10s the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06
_ Other Tests _
-1 ❌ unit 3m 53s root in the patch failed.
+1 💚 asflicense 0m 37s The patch does not generate ASF License warnings.
45m 48s
Reason Tests
Failed junit tests tez.common.TestTezCommonUtils
Subsystem Report/Notes
Docker ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-342/1/artifact/out/Dockerfile
GITHUB PR #342
Optional Tests dupname asflicense javac javadoc unit xml compile
uname Linux 1ec75fbf2bd4 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/tez.sh
git revision master / f080031
Default Java Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu222.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~22.04-b06
unit https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-342/1/artifact/out/patch-unit-root.txt
Test Results https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-342/1/testReport/
Max. process+thread count 96 (vs. ulimit of 5500)
modules C: . U: .
Console output https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-342/1/console
versions git=2.34.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

ayushtkn
ayushtkn reviewed Apr 7, 2024
View reviewed changes
Copy link
Member

@ayushtkn ayushtkn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We usually couple Hadoop upgrade on Tez & Hive together, they have a history of not working when not on same version, everytime we didn't, it creates classpath issues when running queries in hive, we had to quickly push the last Tez release to unblock hive, since hive with 3.3.6 wasn't compatible with Tez on hadoop-3.3.1, some Htrace stuff...

Hive doesn't compile as of now with Hadoop-3.4.0, so, I think we should hold this till Hive gets that sorted

@slfan1989
Copy link
Contributor Author

We usually couple Hadoop upgrade on Tez & Hive together, they have a history of not working when not on same version, everytime we didn't, it creates classpath issues when running queries in hive, we had to quickly push the last Tez release to unblock hive, since hive with 3.3.6 wasn't compatible with Tez on hadoop-3.3.1, some Htrace stuff...

Hive doesn't compile as of now with Hadoop-3.4.0, so, I think we should hold this till Hive gets that sorted

Thank you for explanation!

@Aggarwal-Raghav
Copy link
Contributor

From changes POV, Hadoop 3.4.0 has zookeeper version 3.8.3 which uses logback. We have to explicitly exclude it from hadoop dependecies, otherwise it will lead to class loader issue. For example, in hive, If logback jar gets picked up first then hive-log4j2.properties won't be honored.
Same is mentioned in description of apache/hadoop#6582

@zhangbutao zhangbutao mentioned this pull request Oct 11, 2024
Open
4 tasks
@zhangbutao
Copy link
Contributor

From changes POV, Hadoop 3.4.0 has zookeeper version 3.8.3 which uses logback. We have to explicitly exclude it from hadoop dependecies, otherwise it will lead to class loader issue. For example, in hive, If logback jar gets picked up first then hive-log4j2.properties won't be honored. Same is mentioned in description of apache/hadoop#6582

@Aggarwal-Raghav We can trying to upgrading Hadoop version to 3.4.0 in Hive apache/hive#5500.
Any thought about refine this change apache/hive#5500?
Thanks.

@Aggarwal-Raghav
Copy link
Contributor

Have added comments on hive PR apache/hive#5500

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ayushtkn ayushtkn ayushtkn left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@slfan1989 @tez-yetus @Aggarwal-Raghav @zhangbutao @ayushtkn