Add simple benchmarking #74
Conversation
|
So my thought here is that this is generally useful - and clar shouldn't know the difference between a "benchmark" and a "test". It should just run things and time them. If some consumer wants to use clar to run both benchmarks and tests, then I think that there are a couple of options:
|
|
I think we can avoid having two different clars built by passing in a benchmark flag; that shouldn't be too bad. There's a few reasons I chose We can certainly solve this by putting all the benchmarks as But it's not that big a deal at the end of the day. Having benchmarks be normal tests does mean they get tested the same as the rest even when we're not benchmarking. |
|
Sorry for the long delay - this looks like what I had in mind! This seems to be missing the |
|
I did notice it was missing earlier, but completely forgot to add it the second time around as well :/ It's there now. |
clar/time.h
Outdated
| struct timespec tp; | ||
|
|
||
| if (clock_gettime(CLOCK_MONOTONIC, &tp) == 0) { | ||
| return (double tp.tv_sec + (double tp.tv_nsec / 1.0E9; |
There was a problem hiding this comment.
My compiler is unhappy with this. :(
|
I really like this. My only feedback is that I might like to see the benchmark reports printed at the end of the entire execution, instead of the end of an execution for each suite. eg: Seems a little weird for output (weirder still with |
|
Yeah, that makes sense. |
Starting a function name with 'bench_' will now also store it and make clar run it, but only when invoked with the '-b' flag. In order to reduce how much clar overhead we measure, we put the timing code around the function call, which means we end up timing every function, but we only store the elapsed time in benchmark mode. The overhead from calling the timer is minimal, so it shouldn't present an issue to add these calls for every test. This is not meant for micro-benchmarks, but for situations where we expect the operations to take at least tens or hundreds of milliseconds and we expect the operations not to be easily mapped into the "repeat N times" method to suss out the noise, so we don't include this functionality. This is for "perform very expensive operation" to track high-scale changes though time. Thus we also provide cl_reset_timer() to allow the function under test to perform specific setup steps that it does not want to count towards the timing.
Instead of separating the benchmark tests from "normal" tests, store timings for all successful tests and let the user decide which tests they'd like to run via the -s option.
Instead of intermixing benchmark results with the suite outputs, keep them all until the end.
|
@ethomson I finally got around to finishing this up; the timing results are now shown at the end of a multi-suite run. |
Starting a function name with 'bench_' will now also store it and make
clar run it, but only when invoked with the '-b' flag.
In order to reduce how much clar overhead we measure, we put the timing
code around the function call, which means we end up timing every
function, but we only store the elapsed time in benchmark mode. The
overhead from calling the timer is minimal, so it shouldn't present an
issue to add these calls for every test.
This is not meant for micro-benchmarks, but for situations where we
expect the operations to take at least tens or hundreds of milliseconds
and we expect the operations not to be easily mapped into the "repeat N
times" method to suss out the noise, so we don't include this
functionality. This is for "perform very expensive operation" to track
high-scale changes though time.
Thus we also provide cl_reset_timer() to allow the function under test
to perform specific setup steps that it does not want to count towards
the timing.
This is the simplest thing I could come up with which would be useful for us to test libgit2 performance over time, without going into microbenchmarking, but for the "merge thousand-entry trees" situation. The output is machine-readable, so we can keep historical records in some system. /cc @ethomson
The thing with
global_is_benchis ugly but we're now in two different modes, so the number of tests in each case is different, and it seemed less terrible than looking into_clarfrom the outside, or searching for clar flags inmain.c.