What Makes a Good Computational Benchmark?

We think four key ingredients are necessary for a modern computational benchmark suite [1]:

1. Application-specific list of problems that
   • span different difficulty levels
   • emphasize/exercise features of a computational system relevant to the application
   • are general enough to represent different types of problems encountered frequently
   • evolve as computational systems (algorithm+software implementation+hardware combinations) advance
2. Precisely defined quantities of interest and reliable reference solutions that
   • are relevant to the application
   • include measurement or analytical-solution based references
   • permit (much) more accurate results to be obtained/used as reference
3. Performance measures that
   • include error and computational cost measures
   • quantify computational power available to simulations and normalize costs across platforms
4. Online databases of results that
   • publicize the performance of various methods/tools for solving the problems in the benchmark suite

References

[1] J. W. Massey, C. Liu, and A. E. Yilmaz, "Benchmarking to close the credibility gap: A computational BioEM benchmark suite," in Proc. URSI EMTS, Aug. 2016.