exp2json vs poc_extract.sh discrepancy

[AbhishekShah212]

Hi,

Thanks for this excellent dataset and the terrific documentation. I hope it will be widely used.

Problem: I invoked run.sh (captainrc below). After it finished, I expected the files in directory poc to match the triggerability results generated from using python3 ../benchd/exp2json.py workdir/ long_run.json. It does not match.

The results all match for Libfuzzer, but break for Angora. I include a small snippet below, but can include the entire output if it would help.

I investigated a bit further, and it seems something is off with the run_once.sh invoked from the poc_extract.sh script but I am unsure how to proceed. I also see in the FAQ The [fuzzer]/run_once.sh scripts in Magma are intended to emulate the fuzzer's execution environment to detect faults., suggesting this is intended?

My question is for triggerability in Angora, should I trust the *.json results or the files in poc ? Do you know why this might happen?

(base) user@host magma-1.1.0/tools/captain$ ls workdir/poc/ | uniq | rev | cut -c 4- | rev | uniq

angora_libxml2_xmllint_AAH037.
angora_libxml2_xmllint_AAH041.
libfuzzer_libxml2_libxml2_xml_read_memory_fuzzer_AAH037.
libfuzzer_libxml2_libxml2_xml_read_memory_fuzzer_AAH041.


(base) user@host magma-1.1.0/tools/captain$ python -m json.tool long_run.json
{
    "results": {
        "angora": {
            "libxml2": {
                "libxml2_xml_read_memory_fuzzer": {
                    "0": {
                        "reached": {
                            "AAH032": 15,
                            "AAH029": 15,
                            "AAH037": 10,
                            "AAH024": 15,
                            "AAH035": 25,
                            "AAH034": 15,
                            "AAH031": 75,
                            "AAH026": 15,
                            "AAH041": 15
                        },
                        "triggered": {
                            "AAH037": 15, # does not appear in poc/*
                            "AAH026": 25, # does not appear in poc/*
                            "AAH041": 15 # does not appear in poc/*
                        }
                    }
                },
                "xmllint": {
                    "0": {
                        "reached": {
                            "AAH032": 15,
                            "AAH029": 15,
                            "AAH037": 10,
                            "AAH024": 15,
                            "AAH035": 15,
                            "AAH034": 20,
                            "AAH031": 3615,
                            "AAH026": 15,
                            "AAH041": 15
                        },
                        "triggered": {
                            "AAH037": 12970,
                            "AAH041": 25
                        }
                    }
                }
            },
      ...
        "libfuzzer": {
            "libxml2": {
                "libxml2_xml_read_memory_fuzzer": {
                    "0": {
                        "reached": {
                            "AAH032": 15,
                            "AAH029": 15,
                            "AAH037": 10,
                            "AAH024": 15,
                            "AAH035": 15,
                            "AAH034": 15,
                            "AAH031": 15,
                            "AAH026": 15,
                            "AAH041": 15
                        },
                        "triggered": {
                            "AAH037": 15,
                            "AAH041": 15
                        }
                    }
                }
            },

Here is the captainrc

# This file contains the configuration for the run.sh script. It follows the
# Bash syntax and is sourced by the script to access the variables. Variables
# are mandatory unless marked with [brackets].

###
## Configuration parameters
###

# WORKDIR: path to directory where shared volumes will be created
WORKDIR=./workdir

# REPEAT: number of campaigns to run per program (per fuzzer)
REPEAT=1

# [WORKER_MODE]: defines the type of CPU resources to allocate (default: 1)
# - 1: logical cores (possibly SMT-enabled)
# - 2: physical cores
# - 3: physical sockets (1 worker per CPU socket)
# WORKER_MODE=1

# [WORKERS]: number of worker threads (default: all cores)
# WORKERS=3

# [WORKER_POOL]: a space-separated list of logical cores to allocate
# WORKER_POOL="1 3 5 7 9"

# [CAMPAIGN_WORKERS]: number of workers to allocate for a campaign (default: 1)
# CAMPAIGN_WORKERS=1

# [TIMEOUT]: time to run each campaign. This variable supports one-letter
# suffixes to indicate duration (s: seconds, m: minutes, h: hours, d: days)
# (default: 1m)
TIMEOUT=6h

# [POLL]: time (in seconds) between polls (default: 5)
POLL=5

# [CACHE_ON_DISK]: if set, the cache workdir is mounted on disk instead of
# in-memory (default: unset)
# CACHE_ON_DISK=1

# [NO_ARCHIVE]: if set, campaign workdirs will not be tarballed (default: unset)
NO_ARCHIVE=1

# [TMPFS_SIZE]: the size of the tmpfs mounted volume. This only applies when
# CACHE_ON_DISK is not set (default: 50g)
# TMPFS_SIZE=16g

# [MAGMA]: path to magma root (default: ../../)
# MAGMA=/path/to/magma/

# [CANARY_MODE]: defines the mode of canaries at compile time (default: 1)
# - 1: without fixes, with canaries
# - 2: without fixes, without canaries
# - 3: with fixes, without canaries
# CANARY_MODE=3

# [ISAN]: if set, build the benchmark with ISAN/fatal canaries (default: unset)
ISAN=1

# [HARDEN]: if set, build the benchmark with hardened canaries (default: unset)
# HARDEN=1

# [POC_EXTRACT]: if set, run the extract.sh script after the campaign is done
# (default: unset)
POC_EXTRACT=1


###
## Campaigns to run
###

# FUZZERS: an array of fuzzer names (from magma/fuzzers/*) to evaluate
FUZZERS=(angora libfuzzer)

# [fuzzer_TARGETS]: an array of target names (from magma/targets/*) to fuzz with
# `fuzzer`. The `fuzzer` prefix is a fuzzer listed in the FUZZERS array
# (default: all targets)

# [fuzzer_target_PROGRAMS]: an array of program names (from
# magma/targets/target/configrc) to use as execution drivers when fuzzing the
# `target`
# afl_libtiff_PROGRAMS=(tiffcp)

# [fuzzer_target_FUZZARGS]: a string containing fuzzer/target-specific arguments
# when fuzzing `target` with `fuzzer`
# afl_libpng_FUZZARGS="-x /magma_shared/png.dict"

# [fuzzer_CAMPAIGN_WORKERS]: overrides the global CAMPAIGN_WORKERS setting
# afl_CAMPAIGN_WORKERS=3

[hazimeh]

The monitor results (summarized in the json report) are the most accurate, because they are collected at runtime, while the fuzzer is still running.
It could be, as you mentioned, that the PoC extraction script malfunctions with Angora.
Could you provide me with a copy of your workdir?

[AbhishekShah212]

Thanks for the quick reply.

Here is a link to the workdir: https://drive.google.com/file/d/1IiDrRlArDaQ-awim79PenuDK3Ztwlfbu/view?usp=sharing