Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

The merge operation could fail when there's channel exception for coverage files #802

Open
hadi88 opened this issue Jul 17, 2023 · 5 comments
Open

The merge operation could fail when there's channel exception for coverage files #802

hadi88 opened this issue Jul 17, 2023 · 5 comments

Comments

@hadi88
Copy link
Contributor

hadi88 commented Jul 17, 2023

We've observed that CoverageIdStrategy.obtainFirstId fails during the merge process with FileLockInterruptionException, ClosedByInterruptException, and other channels exceptions. Handling such exceptions could significantly improve robustness of the merge operation.

Unfortunately, I don't have a reproducer for this issue.
@fmeum
Copy link
Contributor

fmeum commented Jul 18, 2023 •
edited
Loading

It would be really helpful to have a reproducer for this, but I can see that being difficult to come up with.

We have observed rare inconsistencies in the shared coverage ID file in the past that resulted in a crash, but never managed to identify the problem. I haven't seen cases of these exceptions being thrown yet, but will take another look at the code and check whether we can handle them (more) gracefully.

@fmeum
Copy link
Contributor

fmeum commented Jul 18, 2023

@hadi88 Do you have stack traces that you could share?

I can attempt a speculative fix that retries obtaining the lock in case of interrupts, but I would first like to confirm that this has a chance to fix the issue you are seeing.

@hadi88
Copy link
Contributor Author

hadi88 commented Jul 18, 2023

Initially, I observed a check failure in commitIdCount:

Caused by: java.lang.IllegalStateException: Check failed.
	at com.code_intelligence.jazzer.agent.FileSyncCoverageIdStrategy.commitIdCount(CoverageIdStrategy.kt:175)
	... 34 more

This exception masks the root cause. When I changed the check failure into an if-condition, I observed the following two stack traces:

ERROR: Coverage IDs are out of sync	
com.code_intelligence.jazzer.agent.CoverageIdException: Failed to synchronize coverage IDs
	at com.code_intelligence.jazzer.agent.FileSyncCoverageIdStrategy.obtainFirstId(CoverageIdStrategy.kt:167)
	at com.code_intelligence.jazzer.agent.FileSyncCoverageIdStrategy.withIdForClass(CoverageIdStrategy.kt:90)
	at com.code_intelligence.jazzer.agent.RuntimeInstrumentor.instrument(RuntimeInstrumentor.kt:215)
	at com.code_intelligence.jazzer.agent.RuntimeInstrumentor.transformInternal(RuntimeInstrumentor.kt:185)
	at com.code_intelligence.jazzer.agent.RuntimeInstrumentor.transform(RuntimeInstrumentor.kt:96)
	at com.code_intelligence.jazzer.agent.RuntimeInstrumentor.transform(RuntimeInstrumentor.kt:161)
	at java.instrument/sun.instrument.TransformerManager.transform(TransformerManager.java:188)
	at java.instrument/sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:563)
	at java.base/java.lang.ClassLoader.defineClass1(Native Method)
	at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1018)
	at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
	at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:800)
	at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:698)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:621)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:182)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:523)
	at java.base/java.lang.ClassLoader.defineClass1(Native Method)
	at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1018)
	at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
	at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:800)
	at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:698)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:621)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:182)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:523)
	at java.base/java.lang.Class.getDeclaredConstructors0(Native Method)
	at java.base/java.lang.Class.privateGetDeclaredConstructors(Class.java:3139)
	at java.base/java.lang.Class.getConstructor0(Class.java:3344)
	at java.base/java.lang.Class.getConstructor(Class.java:2151)
	at io.netty.channel.ReflectiveChannelFactory.<init>(ReflectiveChannelFactory.java:34)
	at io.netty.bootstrap.AbstractBootstrap.channel(AbstractBootstrap.java:110)
	...
Caused by: java.nio.channels.ClosedByInterruptException
	at java.base/java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:199)
	at java.base/sun.nio.ch.FileChannelImpl.endBlocking(FileChannelImpl.java:162)
	at java.base/sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:228)
	at com.code_intelligence.jazzer.agent.CoverageIdStrategyKt.readFully(CoverageIdStrategy.kt:213)
	at com.code_intelligence.jazzer.agent.FileSyncCoverageIdStrategy.obtainFirstId(CoverageIdStrategy.kt:123)
	... 43 more

ERROR: Coverage IDs are out of sync	
com.code_intelligence.jazzer.agent.CoverageIdException: Failed to synchronize coverage IDs
	at com.code_intelligence.jazzer.agent.FileSyncCoverageIdStrategy.obtainFirstId(CoverageIdStrategy.kt:167)
	at com.code_intelligence.jazzer.agent.FileSyncCoverageIdStrategy.withIdForClass(CoverageIdStrategy.kt:90)
	at com.code_intelligence.jazzer.agent.RuntimeInstrumentor.instrument(RuntimeInstrumentor.kt:215)
	at com.code_intelligence.jazzer.agent.RuntimeInstrumentor.transformInternal(RuntimeInstrumentor.kt:185)
	at com.code_intelligence.jazzer.agent.RuntimeInstrumentor.transform(RuntimeInstrumentor.kt:96)
	at com.code_intelligence.jazzer.agent.RuntimeInstrumentor.transform(RuntimeInstrumentor.kt:161)
	at java.instrument/sun.instrument.TransformerManager.transform(TransformerManager.java:188)
	at java.instrument/sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:563)
	at java.base/java.lang.ClassLoader.defineClass1(Native Method)
	at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1018)
	at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
	at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:800)
	at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:698)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:621)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:182)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:523)
	at java.base/java.lang.ClassLoader.defineClass1(Native Method)
	at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1018)
	at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
	at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:800)
	at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:698)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:621)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:182)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:523)
	at java.base/java.lang.ClassLoader.defineClass1(Native Method)
	at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1018)
	at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
	at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:800)
	at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:698)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:621)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:182)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:523)
	...
Caused by: java.nio.channels.FileLockInterruptionException
	at java.base/sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1138)
	at java.base/java.nio.channels.FileChannel.lock(FileChannel.java:1065)
	at com.code_intelligence.jazzer.agent.FileSyncCoverageIdStrategy.obtainFirstId(CoverageIdStrategy.kt:119)
	... 46 more

@fmeum
Copy link
Contributor

fmeum commented Jul 25, 2023

It would be interesting to learn what causes these exceptions. Is your fuzz test multi-threaded? Does your JVM support concurrent class loading?

I am asking as I haven't seen these failures anywhere else yet and handling them properly isn't trivial. With more information, I might be able to come up with a reproducer.

@ghost
Copy link

ghost commented Feb 26, 2024

@hadi88 - I wrote to you about the other issue, same here. Ping me so we can look at it more closely together?
david[dot]merian [at] code-intelligence[dot]com

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@fmeum @hadi88