Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TEZ-4561: Improve reported exception when DAGAppMaster is shutting down. #365

Merged
merged 6 commits into from
Sep 3, 2024

Conversation

ayushtkn
Copy link
Member

No description provided.

@tez-yetus

This comment was marked as outdated.

@ayushtkn
Copy link
Member Author

I think we can live with this checkstyle warning, creating an accessor method looks like an overkill to me

@ayushtkn
Copy link
Member Author

Added a test.
Sharing the output of exception
When State is INITED

org.apache.tez.dag.api.TezUncheckedException: Cannot get ApplicationACLs before all services have started, The current service state is INITED

When State is STOPPED

org.apache.tez.dag.api.TezUncheckedException: Cannot get ApplicationACLs before all services have started, The current service state is STOPPED. The shutdown hook started at Fri Aug 16 16:08:09 IST 2024

@tez-yetus

This comment was marked as outdated.

@tez-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 30s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ master Compile Tests _
+1 💚 mvninstall 16m 24s master passed
+1 💚 compile 0m 43s master passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu322.04
+1 💚 compile 0m 38s master passed with JDK Private Build-1.8.0_422-8u422-b05-1~22.04-b05
+1 💚 checkstyle 1m 30s master passed
+1 💚 javadoc 0m 37s master passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu322.04
+1 💚 javadoc 0m 21s master passed with JDK Private Build-1.8.0_422-8u422-b05-1~22.04-b05
+0 🆗 spotbugs 1m 51s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 1m 48s master passed
_ Patch Compile Tests _
+1 💚 mvninstall 0m 28s the patch passed
+1 💚 compile 0m 31s the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu322.04
+1 💚 javac 0m 31s the patch passed
+1 💚 compile 0m 26s the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~22.04-b05
+1 💚 javac 0m 26s the patch passed
-0 ⚠️ checkstyle 0m 21s tez-dag: The patch generated 1 new + 57 unchanged - 3 fixed = 58 total (was 60)
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 javadoc 0m 10s the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu322.04
+1 💚 javadoc 0m 10s the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~22.04-b05
+1 💚 findbugs 1m 14s the patch passed
_ Other Tests _
+1 💚 unit 5m 8s tez-dag in the patch passed.
+1 💚 asflicense 0m 17s The patch does not generate ASF License warnings.
32m 16s
Subsystem Report/Notes
Docker ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-365/3/artifact/out/Dockerfile
GITHUB PR #365
JIRA Issue TEZ-4561
Optional Tests dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
uname Linux 06ebcb967f9f 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/tez.sh
git revision master / cd6ceec
Default Java Private Build-1.8.0_422-8u422-b05-1~22.04-b05
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu322.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~22.04-b05
checkstyle https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-365/3/artifact/out/diff-checkstyle-tez-dag.txt
Test Results https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-365/3/testReport/
Max. process+thread count 222 (vs. ulimit of 5500)
modules C: tez-dag U: tez-dag
Console output https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-365/3/console
versions git=2.34.1 maven=3.6.3 findbugs=3.0.1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@@ -1743,6 +1747,13 @@ public void setQueueName(String queueName) {
}
}

private String getShutdownTimeString() {
if (shutdownHandler != null && shutdownHandler.shutdownTime != null) {
return ". The shutdown hook started at " + shutdownHandler.shutdownTime;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's strange that this separate string generation method takes care of the closing point of the previous sentence, I believe that should go back to getApplicationACLs

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I moved the "." back to the {{getApplicationACLs}}

@tez-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 27m 8s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ master Compile Tests _
+1 💚 mvninstall 16m 59s master passed
+1 💚 compile 0m 44s master passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu322.04
+1 💚 compile 0m 39s master passed with JDK Private Build-1.8.0_422-8u422-b05-1~22.04-b05
+1 💚 checkstyle 1m 31s master passed
+1 💚 javadoc 0m 36s master passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu322.04
+1 💚 javadoc 0m 23s master passed with JDK Private Build-1.8.0_422-8u422-b05-1~22.04-b05
+0 🆗 spotbugs 1m 53s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 1m 51s master passed
_ Patch Compile Tests _
+1 💚 mvninstall 0m 28s the patch passed
+1 💚 compile 0m 31s the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu322.04
+1 💚 javac 0m 31s the patch passed
+1 💚 compile 0m 27s the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~22.04-b05
+1 💚 javac 0m 27s the patch passed
-0 ⚠️ checkstyle 0m 20s tez-dag: The patch generated 1 new + 57 unchanged - 3 fixed = 58 total (was 60)
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 javadoc 0m 10s the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu322.04
+1 💚 javadoc 0m 10s the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~22.04-b05
+1 💚 findbugs 1m 15s the patch passed
_ Other Tests _
+1 💚 unit 5m 6s tez-dag in the patch passed.
+1 💚 asflicense 0m 17s The patch does not generate ASF License warnings.
59m 40s
Subsystem Report/Notes
Docker ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-365/4/artifact/out/Dockerfile
GITHUB PR #365
JIRA Issue TEZ-4561
Optional Tests dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
uname Linux 249eda176fba 5.15.0-119-generic #129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/tez.sh
git revision master / 174d4e3
Default Java Private Build-1.8.0_422-8u422-b05-1~22.04-b05
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu322.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~22.04-b05
checkstyle https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-365/4/artifact/out/diff-checkstyle-tez-dag.txt
Test Results https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-365/4/testReport/
Max. process+thread count 205 (vs. ulimit of 5500)
modules C: tez-dag U: tez-dag
Console output https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-365/4/console
versions git=2.34.1 maven=3.6.3 findbugs=3.0.1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@abstractdog
Copy link
Contributor

nit: what about changing shutdownTime to private?
not necessarily because of the checkstyle warning (private + accessor methods), but instead to keep it inline with other fields of DAGAppMasterShutdownHandler, like:

    private AtomicBoolean shutdownHandled = new AtomicBoolean(false);
    private long sleepTimeBeforeExit = TezConstants.TEZ_DAG_SLEEP_TIME_BEFORE_EXIT;

I guess the usage scope of those are similar

@tez-yetus

This comment was marked as outdated.

@tez-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 34s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ master Compile Tests _
+1 💚 mvninstall 19m 8s master passed
+1 💚 compile 0m 45s master passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu322.04
+1 💚 compile 0m 40s master passed with JDK Private Build-1.8.0_422-8u422-b05-1~22.04-b05
+1 💚 checkstyle 1m 42s master passed
+1 💚 javadoc 0m 40s master passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu322.04
+1 💚 javadoc 0m 23s master passed with JDK Private Build-1.8.0_422-8u422-b05-1~22.04-b05
+0 🆗 spotbugs 2m 0s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 1m 58s master passed
_ Patch Compile Tests _
+1 💚 mvninstall 0m 27s the patch passed
+1 💚 compile 0m 32s the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu322.04
+1 💚 javac 0m 32s the patch passed
+1 💚 compile 0m 26s the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~22.04-b05
+1 💚 javac 0m 26s the patch passed
+1 💚 checkstyle 0m 21s tez-dag: The patch generated 0 new + 57 unchanged - 3 fixed = 57 total (was 60)
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 javadoc 0m 10s the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu322.04
+1 💚 javadoc 0m 9s the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~22.04-b05
+1 💚 findbugs 1m 14s the patch passed
_ Other Tests _
+1 💚 unit 5m 7s tez-dag in the patch passed.
+1 💚 asflicense 0m 19s The patch does not generate ASF License warnings.
35m 42s
Subsystem Report/Notes
Docker ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-365/6/artifact/out/Dockerfile
GITHUB PR #365
JIRA Issue TEZ-4561
Optional Tests dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
uname Linux 125dfebe230f 5.15.0-119-generic #129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/tez.sh
git revision master / 174d4e3
Default Java Private Build-1.8.0_422-8u422-b05-1~22.04-b05
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu322.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~22.04-b05
Test Results https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-365/6/testReport/
Max. process+thread count 233 (vs. ulimit of 5500)
modules C: tez-dag U: tez-dag
Console output https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-365/6/console
versions git=2.34.1 maven=3.6.3 findbugs=3.0.1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@ayushtkn
Copy link
Member Author

@abstractdog I have changed the variable to private & used setter & getter, there was some FindBugs warning, which I have dodged, lemme know if things look good

@abstractdog abstractdog self-requested a review September 3, 2024 09:57
Copy link
Contributor

@abstractdog abstractdog left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@ayushtkn ayushtkn merged commit 76490d4 into apache:master Sep 3, 2024
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants