Breaking: Switch to BSON for storage #48
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Moved most of the `load_benchmarks` logic out of the Jogger and into the main package. Added dispatching for `:latest` and `:oldest` for loading results based on the mtime of the results. Added additional test cases to checkout loading benchmarks with :latest
Added test to check that Example.jl/benchmark/trial is deleted Added `cleanup_example()` helper for cleaning up post test Fixed `:latest` / `:oldest` tests to cleanup Example.jl prior to testing
Redefined a few lines down and the later def is cleaner
Previously, a key of `["bench_foo.jl", 0.1]` would be saved as ``["bench_foo.jl", "0.1"]` and thus not match when checked with `setdiff` / `intersect` Now checks for a unique serialization of keys, and then match on those keys iff there is an injective (1:1) mapping. Keys that are not injective (1:n), are always retuned, but realistically why would you have them? Fixes #43
This reverts commit 35ed40a.
Remove tune! methods and use `_get_benchmarks` for more inputs Instead of copying use `BenchmarkTools.loadparams!` for an inplace copy Now throw error if given a refereance dict that's missing benchmarking results
BSON correctly seralizes data without stringifying it, which fixes the following seralization errors: - `Jogger.judge(:latest, UUID)` fails #45 - `save_benchmarks` modifies types #43 Maintain backwards compatablity (ie loading old results), but fully depreciate the old format - Won't fix `save_benchmarks` modifies types #43 or `Jogger.judge(:latest, UUID)` fails #45 for `*.json.gz` files - New features build on non-stringified results may not work
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
While implementing
load_benchmarks(:latest)it became apparent that stringifying everything before saving wasn't ideal (#45, #46, #43), and attempting to invert stringifying (35ed40a) started to balloon (Adding support forjudgewas getting hacky). So instead I've decided to switch to a serializer that wouldn't stringify everything.Serialization.serialize- Smallest file size of all methods. But lacks stability (by design) across versions and machines. While benchmarking results tend to lose value quickly, I do want to support judging between versions / across machines.JLD2- Large file size, even when compressedJLD- Would add a dependency on HDF5, large file size, and slowJSON- Stringifies most data, mangling the keys for benchmark groups.Backwards compatibility
PkgJogger v0.4 and later will save results using BSON to
*.bson.gzfiles. Loading results withload_benchmarksfrom*.json.gzfiles is supported by PkgJogger v0.4, but will be removed in a future release.Other Changes
This PR also adds support for
JogExample.load_benchmarks(:lastest)andJogExample.load_benchmarks(:oldest)to load the most recent / oldest results from disk. The:latestand:oldestidentifiers are supported byjudgeandtune!as well.