Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add incremental benchmarking scripts #44

Open
wants to merge 87 commits into
base: master
Choose a base branch
from
Open

Add incremental benchmarking scripts #44

wants to merge 87 commits into from

Conversation

jerhard
Copy link
Member

@jerhard jerhard commented Nov 23, 2022 •
edited by sim642
Loading

This PR adds python scripts for benchmarking the incremental analysis. These scripts were originally added by @stilscher in goblint/analyzer#778. In the discussion there, we decided to move the scripts to the bench repo.

Compared to #778 the following changes were made:

  • Project specific settings are to be defined in projects.py
  • a configuration for sqlite was added
  • Project name is to be passed as a command line argument for the scripts
  • the command that goblint is started with and the commit-id of the analyzed repo are recorded in an output-file

Closes #47
Closes #48

TODO

jerhard and others added 15 commits November 23, 2022 13:05
@jerhard @stilscher
Move scripts for incremental benchmarking from the analyzer to the be…
ccc2a6a 
…nch repo.

These scripts were written by Sarah Tilscher.

Co-authored-by: Sarah Tilscher <[email protected]>
@jerhard
Change incremental benchmarking scripts for running with sqlite
73cebe1
@jerhard
Update path for conf
c9e5566
@jerhard
Add build_compdb_sqlite.sh
e138867
@jerhard
Update configuration for sqlite
7bd5d92
@jerhard
efficiency.py: Make it configurable which project to benchmark increm…
dffaeac 
…ental goblint on.
@jerhard
Make precision take paramters for which project to run on
764372e
@jerhard
Do not use dataclasses for compatibility with python 3.6
c773156
@jerhard
Improve error message
7009d93
@jerhard
Change annotation of type for compatibility with Python 3.6
11e92bf
@jerhard
Add configuration for incremental postsolver to sqlite benchmarking c…
2b7eafd 
…onfig.
@jerhard
Utils.analyze_commit: Fix handling of case that file list is passed.
1500752
@jerhard
Print analyze command and commit hash in config.out created by increm…
79c940f 
…ental benchmarking script.
@jerhard
Add -v option to analyze commands in efficiency script.
a248b6b
@jerhard
Merge branch 'master' into incremental_benchmarking_scripts
075194b
@jerhard jerhard marked this pull request as ready for review November 23, 2022 12:39
@jerhard jerhard mentioned this pull request Nov 23, 2022
Open
@jerhard
Add figlet as project.
6d1a67d 
Not completely working yet, as conflicting flags are collected in the compile_commands file for figlet.
@jerhard jerhard mentioned this pull request Nov 29, 2022
Open
@jerhard
Adapt regular expression to extract runtime from output.
595de4c
@jerhard
incremental benchmarking: Use Makefile instead of compiledb for figlet.
8e5888f
@jerhard
Add biuld_compdb_figlet.sh
8bcc304
@jerhard
Add preliminary setup for chrony incremental benchmarks.
97ea54e 
Make does not succeed yet, so this requires further adaption.
@jerhard
Reset time range of considered commits for zstd.
587a2dd 
Resets the range of considered commits to be the same as originally in analyzer/#778.
@jerhard
Update build script for chrony.
0345ce7
@jerhard
Fix project configuration for chrony to not provide files (but use co…
c943f9f 
…mpiledb).
@jerhard
Merge branch 'incremental_benchmarking_scripts_chrony' into increment…
5fc065e 
…al_benchmarking_scripts
@jerhard
Use github mirror for chrony..
2639efb 
The repo at tuxfamily does not allow for multiple connections/parallel clones.
@jerhard
Plot.py: Create cummulative graphs again, combining results from 3 co…
1eb1a82 
…nfigs in one graph.

Comments out the creation using "paper_efficiency_graphs" and "paper_precision_graph".
.
@jerhard jerhard force-pushed the incremental_benchmarking_scripts branch from 5ca878e to e3db195 Compare June 15, 2023 08:59
jerhard and others added 27 commits June 15, 2023 18:05
@jerhard
Refactor collect_data to reduce redundancy.
a6d27d9
@jerhard
Refactor collect_data further to reduce redundancy.
670cf3e
@jerhard
Collec changed/added/removed functions from child (incremental) config.
376a176
@jerhard
Adapt match for changed/added/removed functions for changed goblint o…
9b67c07 
…utput.
@jerhard
Extract CPU times for total runtimes, analysis and solving times.
4f24c89
@jerhard
Collect_data: Check whether result directory exists before iterating …
e4dc7fe 
…over it.

This avoids related errors in the command prompt.
@stilscher
additionally plot efficiency bar plots
7e6e8a7
@jerhard
Interactive confs: set incremental.detect-renames to false.
ddaabd7
@jerhard
Extract incremental analysis of child commit in precision script.
e169eff
@jerhard
Exclude figlet from precision run.
249ebf5
@jerhard
Precision script: fix config file look-up.
de4208b
@jerhard
Incremental efficiency script: Do not limit commits to 50 relevant Lo…
bbb3e7d 
…C changes.
@jerhard
Precision: run all configurations.
793496c
@jerhard
Fix parameters for increm rel with incr. postsolver.
2e50ca1
@jerhard
Incremental precision script: perform merge results for all configs.
5c9d7e1
@jerhard
Interactive precision: iterate over configurations in outer instead o…
c7d9e15 
…f inner loop.
@jerhard
Use float instead of np.float as np.float has been removed.
54e6359
@jerhard
Plot separate efficiency graphs for <= 50 loc changes and > 50 loc ch…
92af047 
…anges.
@jerhard
Efficiency plots: add efficiency_bar_plots again.
c020267
@jerhard
Interactive efficiency: plot cpu_time instead of wall time.
ca79b69
@jerhard
Add printouts for debugging.
086955b
@jerhard
Remove spurious --enable from parameters passed in precision script.
02d9efc
@jerhard
Output precision results into different files, extend plot script to …
8657739 
…produce plots for each of them.
@jerhard
Add underscore in file names of results produced by precision script …
26fc662 
…and the precision plots.
@jerhard
Precision script: explicitely pass load-dir/save-dir for incremental …
20571d3 
…data.
@jerhard
Remove detect-renames field for compatibility with sttt-2022 goblint.
41d5558
@jerhard
Adapt script to extract changed functions from goblint versino sttt-2…
6158b17 
…022.
@michael-schwarz
Copy link
Member

@stilscher: I assume you simply forgot about the review request here?

Unless there are objections, I would propose to merge this.

@sim642
Copy link
Member

sim642 commented Jan 22, 2024

At minimum, one should check if the confs are still valid for Goblint's schema.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@stilscher stilscher Awaiting requested review from stilscher

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
4 participants
@jerhard @michael-schwarz @sim642 @stilscher