AFLChurn++ Experiment targeting bug-based benchmarks #2022
Conversation
|
@DonggeLiu you mentioned that there are 15 bug-based benchmarks. As there are more than 15 benchmarks, do you know which ones they are? |
|
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-12-aflchurnplusplus --fuzzers afl aflplusplus aflchurnplusplus |
It's on this tag: https://github.com/google/fuzzbench/tree/SBFT'23. |
Do you want to run this on the bug-based or coverage based? |
On the bug based, thanks! |
Then would you please rebase this PR on the following tag, which has the bug-based benchmarks? The current base branch (master) does not have those benchmarks.
|
I did rebase my fork of fuzzbench on that tag, but I'm unsure how to target a tag in a PR, I'm only able to target branches (not tags)
Is there something else I should be doing? |
|
Running a test exp below. |
|
Hmm, I will need an extra config to allow PR experiments when the base is not master |
|
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-13-aflchurnplusplus --fuzzers aflchurnplusplus --benchmarks arrow_arrow-ipc-stream-fuzz_1a34a0 |
|
Experiment |
|
Once the test experiment is done (and if all looks well) can you please run the following: /gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-14-aflchurnplusplus --fuzzers afl aflplusplus aflchurnplusplus --benchmarks arrow_arrow-ipc-stream-fuzz_1a34a0 aspell_aspell_fuzzer_e8eb74 assimp_assimp_fuzzer_4d451f bloaty_fuzz_target_52948c ffmpeg_ffmpeg_demuxer_fuzzer_7adeef file_magic_fuzzer_2d5f85 grok_grk_decompress_fuzzer_9cd001 harfbuzz_hb-shape-fuzzer_17863b lcms_cms_transform_all_fuzzer_97d37d libaom_av1_dec_fuzzer_6e1848 libpcap_fuzz_filter_98b0a2 libxml2_xml_e85b9b mbedtls_fuzz_dtlsclient_7c6b0e php_php-fuzz-parser_0dbedb systemd_fuzz-network-parser_288baf That includes all the bug benchmarks. I would like to run the test on AFL and AFL++ as well, so I have data to make direct comparisons |
|
The report looks good so far, I will start the new one now to minimize our waiting time. |
|
But since you have modified afl++, could you please set this to fuzzbench/service/experiment-config.yaml Line 18 in d8a7723 |
|
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-14-aflchurnplusplus --fuzzers afl aflplusplus aflchurnplusplus --benchmarks arrow_arrow-ipc-stream-fuzz_1a34a0 aspell_aspell_fuzzer_e8eb74 assimp_assimp_fuzzer_4d451f bloaty_fuzz_target_52948c ffmpeg_ffmpeg_demuxer_fuzzer_7adeef file_magic_fuzzer_2d5f85 grok_grk_decompress_fuzzer_9cd001 harfbuzz_hb-shape-fuzzer_17863b lcms_cms_transform_all_fuzzer_97d37d libaom_av1_dec_fuzzer_6e1848 libpcap_fuzz_filter_98b0a2 libxml2_xml_e85b9b mbedtls_fuzz_dtlsclient_7c6b0e php_php-fuzz-parser_0dbedb systemd_fuzz-network-parser_288baf |
|
Experiment |
|
As AFLChurn++ is a regression fuzzer, I would also like to be able to compare with results of fuzzing on the bug-introducing commit. I have added new benchmarks for benchmarks previously tested but changed the commit they are using. If the CIs pass, can you please run this for me: /gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-17-aflchurnplusplus --fuzzers afl aflplusplus aflchurnplusplus --benchmarks arrow_arrow-ipc-stream-fuzz_eee13b aspell_aspell_fuzzer_aed7cd assimp_assimp_fuzzer_bdee65 bloaty_fuzz_target_f01ea5 ffmpeg_ffmpeg_demuxer_fuzzer_fe85af grok_grk_decompress_fuzzer_d9ff920 lcms_cms_transform_all_fuzzer_a9796f libpcap_fuzz_filter_bc594f Thank you! |
|
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-17-aflchurnplusplus --fuzzers afl aflplusplus aflchurnplusplus --benchmarks arrow_arrow-ipc-stream-fuzz_eee13b aspell_aspell_fuzzer_aed7cd assimp_assimp_fuzzer_bdee65 bloaty_fuzz_target_f01ea5 ffmpeg_ffmpeg_demuxer_fuzzer_fe85af grok_grk_decompress_fuzzer_d9ff920 lcms_cms_transform_all_fuzzer_a9796f libpcap_fuzz_filter_bc594f |
|
Experiment |
|
@DonggeLiu the in progress experiment report doesn't appear to be working, is there a problem with the experiment? |
Yes, there appears to be a quota error, based on the base-image build log. |
|
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-18-aflchurnplusplus --fuzzers afl aflplusplus aflchurnplusplus --benchmarks arrow_arrow-ipc-stream-fuzz_eee13b aspell_aspell_fuzzer_aed7cd assimp_assimp_fuzzer_bdee65 bloaty_fuzz_target_f01ea5 ffmpeg_ffmpeg_demuxer_fuzzer_fe85af grok_grk_decompress_fuzzer_d9ff920 lcms_cms_transform_all_fuzzer_a9796f libpcap_fuzz_filter_bc594f |
|
Experiment |
|
Hey, just a couple changes to benchmarks so that they're valid for my analysis. Can you please run this for me (afl has been intentionally excluded): /gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-31-aflchurnplusplus --fuzzers aflplusplus aflchurnplusplus --benchmarks arrow_arrow-ipc-stream-fuzz_1a34a0 arrow_arrow-ipc-stream-fuzz_eee13b libaom_av1_dec_fuzzer_5a4757 libaom_av1_dec_fuzzer_cbbc5c systemd_fuzz-network-parser_46c3b1 systemd_fuzz-network-parser_82d7a1 |
|
/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-31-aflchurnplusplus --fuzzers aflplusplus aflchurnplusplus --benchmarks arrow_arrow-ipc-stream-fuzz_1a34a0 arrow_arrow-ipc-stream-fuzz_eee13b libaom_av1_dec_fuzzer_5a4757 libaom_av1_dec_fuzzer_cbbc5c systemd_fuzz-network-parser_46c3b1 systemd_fuzz-network-parser_82d7a1 |
A follow on from #2012
Testing AFLChurn++ on this old branch with bug-based benchmarks