TEZ-4584: TezUtilsInternal.readUserSpecifiedTezConfiguration throws com.google.protobuf.CodedInputStream exception #376

liujiayi771 · 2024-10-30T08:41:18Z

com.google.protobuf.InvalidProtocolBufferException: Protocol message was too large.  May be malicious.  Use CodedInputStream.setSizeLimit() to increase the size limit.
    at com.google.protobuf.InvalidProtocolBufferException.sizeLimitExceeded(InvalidProtocolBufferException.java:110)
    at com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:755)
    at com.google.protobuf.CodedInputStream.isAtEnd(CodedInputStream.java:701)
    at com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:99)
    at org.apache.tez.dag.api.records.DAGProtos$ConfigurationProto.<init>(DAGProtos.java:19303)
    at org.apache.tez.dag.api.records.DAGProtos$ConfigurationProto.<init>(DAGProtos.java:19267)
    at org.apache.tez.dag.api.records.DAGProtos$ConfigurationProto$1.parsePartialFrom(DAGProtos.java:19369)
    at org.apache.tez.dag.api.records.DAGProtos$ConfigurationProto$1.parsePartialFrom(DAGProtos.java:19364)
    at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200)
    at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:217)
    at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:223)
    at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
    at org.apache.tez.dag.api.records.DAGProtos$ConfigurationProto.parseFrom(DAGProtos.java:19561)
    at org.apache.tez.common.TezUtilsInternal.readUserSpecifiedTezConfiguration(TezUtilsInternal.java:77)
    at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:2351)

It is same as https://issues.apache.org/jira/browse/TEZ-4142. Fix this issue by calling CodedInputStream.setSizeLimit.

tez-yetus · 2024-10-30T09:30:59Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	23m 54s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+1 💚	mvninstall	16m 58s	master passed
+1 💚	compile	0m 29s	master passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu322.04
+1 💚	compile	0m 29s	master passed with JDK Private Build-1.8.0_422-8u422-b05-1~22.04-b05
+1 💚	checkstyle	1m 25s	master passed
+1 💚	javadoc	0m 37s	master passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu322.04
+1 💚	javadoc	0m 23s	master passed with JDK Private Build-1.8.0_422-8u422-b05-1~22.04-b05
+0 🆗	spotbugs	1m 18s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	1m 15s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 15s	the patch passed
+1 💚	compile	0m 15s	the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu322.04
+1 💚	javac	0m 15s	the patch passed
+1 💚	compile	0m 14s	the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~22.04-b05
+1 💚	javac	0m 14s	the patch passed
+1 💚	checkstyle	0m 9s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	javadoc	0m 8s	the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu322.04
+1 💚	javadoc	0m 8s	the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~22.04-b05
+1 💚	findbugs	0m 36s	the patch passed
		_ Other Tests _
+1 💚	unit	0m 35s	tez-common in the patch passed.
+1 💚	asflicense	0m 17s	The patch does not generate ASF License warnings.
		49m 11s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-376/1/artifact/out/Dockerfile
GITHUB PR	#376
JIRA Issue	TEZ-4584
Optional Tests	dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
uname	Linux 8d7cb4329e85 5.15.0-122-generic #132-Ubuntu SMP Thu Aug 29 13:45:52 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / `607b2bc`
Default Java	Private Build-1.8.0_422-8u422-b05-1~22.04-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu322.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~22.04-b05
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-376/1/testReport/
Max. process+thread count	101 (vs. ulimit of 5500)
modules	C: tez-common U: tez-common
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-376/1/console
versions	git=2.34.1 maven=3.6.3 findbugs=3.0.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

…om.google.protobuf.CodedInputStream exception

tez-yetus · 2024-10-31T06:23:01Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 13s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+1 💚	mvninstall	19m 40s	master passed
+1 💚	compile	0m 29s	master passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu322.04
+1 💚	compile	0m 28s	master passed with JDK Private Build-1.8.0_422-8u422-b05-1~22.04-b05
+1 💚	checkstyle	1m 27s	master passed
+1 💚	javadoc	0m 39s	master passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu322.04
+1 💚	javadoc	0m 24s	master passed with JDK Private Build-1.8.0_422-8u422-b05-1~22.04-b05
+0 🆗	spotbugs	1m 24s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	1m 22s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 15s	the patch passed
+1 💚	compile	0m 14s	the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu322.04
+1 💚	javac	0m 14s	the patch passed
+1 💚	compile	0m 14s	the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~22.04-b05
+1 💚	javac	0m 14s	the patch passed
+1 💚	checkstyle	0m 9s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	javadoc	0m 8s	the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu322.04
+1 💚	javadoc	0m 8s	the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~22.04-b05
+1 💚	findbugs	0m 37s	the patch passed
		_ Other Tests _
+1 💚	unit	0m 37s	tez-common in the patch passed.
+1 💚	asflicense	0m 19s	The patch does not generate ASF License warnings.
		28m 28s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-376/2/artifact/out/Dockerfile
GITHUB PR	#376
JIRA Issue	TEZ-4584
Optional Tests	dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
uname	Linux d17062f9d326 5.15.0-122-generic #132-Ubuntu SMP Thu Aug 29 13:45:52 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / `607b2bc`
Default Java	Private Build-1.8.0_422-8u422-b05-1~22.04-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu322.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~22.04-b05
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-376/2/testReport/
Max. process+thread count	101 (vs. ulimit of 5500)
modules	C: tez-common U: tez-common
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-376/2/console
versions	git=2.34.1 maven=3.6.3 findbugs=3.0.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

ayushtkn

TEZ-4142. had a test as well, is it possible to extend a test for this case as well?

liujiayi771 · 2024-11-06T14:00:39Z

@ayushtkn I tried to add a test case. However, I found that this modification is no longer needed in master, including the modifications in TEZ-4142 are no longer needed as well.

I encountered this issue in version 0.10.2, when the protobuf version was 2.5.0. In that version, the CodedInputStream.DEFAULT_SIZE_LIMIT was 64MB.

https://github.com/protocolbuffers/protobuf/blob/774d630bde574f5fcbb6dae6eaa0f91f7bc12967/java/src/main/java/com/google/protobuf/CodedInputStream.java#L561

However, the protobuf in master branch has now been upgraded to 3.25.5, where the DEFAULT_SIZE_LIMIT is now set to Integer.MAX_VALUE by default, so this issue no longer exists.

https://github.com/protocolbuffers/protobuf/blob/9d0ec0f92b5b5fdeeda11f9dcecc1872ff378014/java/core/src/main/java/com/google/protobuf/CodedInputStream.java#L42

ayushtkn · 2024-11-07T06:47:55Z

cool, we can close this then, thanx for checking

liujiayi771 changed the title ~~:TEZ-4584: TezUtilsInternal.readUserSpecifiedTezConfiguration throws com.google.protobuf.CodedInputStream exception~~ TEZ-4584: TezUtilsInternal.readUserSpecifiedTezConfiguration throws com.google.protobuf.CodedInputStream exception Oct 30, 2024

TEZ-4584: TezUtilsInternal.readUserSpecifiedTezConfiguration throws c…

0712849

…om.google.protobuf.CodedInputStream exception

liujiayi771 force-pushed the TEZ-4584 branch from bc1f63f to 0712849 Compare October 31, 2024 05:54

ayushtkn reviewed Nov 6, 2024

View reviewed changes

ayushtkn closed this Nov 7, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TEZ-4584: TezUtilsInternal.readUserSpecifiedTezConfiguration throws com.google.protobuf.CodedInputStream exception #376

TEZ-4584: TezUtilsInternal.readUserSpecifiedTezConfiguration throws com.google.protobuf.CodedInputStream exception #376

liujiayi771 commented Oct 30, 2024

tez-yetus commented Oct 30, 2024

tez-yetus commented Oct 31, 2024

ayushtkn left a comment

liujiayi771 commented Nov 6, 2024

ayushtkn commented Nov 7, 2024

TEZ-4584: TezUtilsInternal.readUserSpecifiedTezConfiguration throws com.google.protobuf.CodedInputStream exception #376

TEZ-4584: TezUtilsInternal.readUserSpecifiedTezConfiguration throws com.google.protobuf.CodedInputStream exception #376

Conversation

liujiayi771 commented Oct 30, 2024

tez-yetus commented Oct 30, 2024

tez-yetus commented Oct 31, 2024

ayushtkn left a comment

Choose a reason for hiding this comment

liujiayi771 commented Nov 6, 2024

ayushtkn commented Nov 7, 2024