How to get a heap dump when fuzzing causes OOM

[centic9]

I have cases where running fuzzing stops with OOM, but re-running the single failure file is not reproducing the OOM.

I assume there are cases where memory accumulates over time when many different inputs are processed and thus the memory is slowly accumulating.

In order to analyze this, I would like to get a heap-dump when Jazzer encounters the OOM.

In a normal Java application I would use -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp to get a .hprof file which can be analyzed afterwards.

However when I tried to specify this option via --jvm_args I do not get a .hprof file. Maybe because Jazzer handles the OOM differently internally.

Is there a way to get a .hprof file?

I am running the Jazzer standalone binary currently, would this be possible via JUnit-based fuzzing?

[fmeum]

Are you using the standalone JAR? --jvm_args is actually a no-op for that as it only applies to the native Jazzer launcher binary. The JAR version of Jazzer just runs in the JVM that you started it in.

Could you try setting these settings via JAVA_TOOL_OPTIONS? That should be picked up by the JVM regardless of how you start Jazzer. I just checked that this works on a very simple fuzz test that OOMs directly.

[centic9]

I originally used the "jazzer" binary, but now switched to running java with the "standalone.jar" to make sure the options are passed properly.

I found the following:

Running with "-rss_limit_mb=1024" causes a different type of OOM which does not lead to a hprof, probably because it happens in native code?

#256    pulse  cov: 885 ft: 2342 corp: 82/588Kb exec/s: 128 rss: 1309Mb
==2707355== ERROR: libFuzzer: out-of-memory (used: 1319Mb; limit: 1024Mb)
   To change the out-of-memory limit use -rss_limit_mb=<N>

MS: 0 ; base unit: 0000000000000000000000000000000000000000
artifact_prefix='./'; Test unit written to ./oom-1768237133a1a4f574239dbbd2c9b9abb8ca94bc
SUMMARY: libFuzzer: out-of-memory

If I removed "-rss_limit_mb" and leave the same value as "-Xmx", I do not get an out of memory at all.

[fmeum]

-rss_limit_mb is a libFuzzer option that limits the memory consumed by the entire process, which includes JVM memory. Since this is hard to control (JVM memory usage is more than just the Java heap), Jazzer sets this to a value that is unlikely to trigger libFuzzer OOMs, which lack any kind of helpful Java stack trace. All in all, it's best to not use this option until you are fuzzing native libraries.

Could you share your fuzz test? If you don't see an OOM but think that you should that sounds more like a fuzz test setup issue than something Jazzer could cause. We can't create memory out of thin air :⁠-⁠)

[centic9]

I'm actually trying to hunt down an OOM reported as part of oss-fuzz runs of project Apache POI.

See https://oss-fuzz.com/testcase-detail/4828378899021824 for the full log.

So it seems oss-fuzz itself uses this option currently and runs into this "special" out of memorry :( .

[fmeum]

I don't have access to the logs unfortunately. -Xmx covers only the Java heap, so if you want to reproduce something that OOMs with --rss_limit_mb=X in OSS-Fuzz, you may have to supply something like 75% of X for -Xmx. If we you choose a really low -Xmx, can you still not reproduce the failure locally?

[centic9]

Seems I am stuck here, oss-fuzz documentation states that this is on-purpose and cannot be adjusted: https://google.github.io/oss-fuzz/faq/#how-do-you-handle-timeouts-and-ooms

[centic9]

I found that Xmx and rss_limit are related in a slightly unexpected way:

• If you INCREASE Xmx, you actually take AWAY memory from Jazzer as the rss_limit controls Java-memory plus native memory
• So LOWERING Xmx seems to provide more memory to Jazzer and thus seems to avoid the native OOM that we are seeing

This is done via google/oss-fuzz#11006 now.