Combine measurements from multiple trials and experiments for the same code revision #164
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…y based on runId Grouping based on trialId is insufficient, since we want to support getting measurements from multiple trials, for instance, when re-executing experiments. We don’t really need the trialId for anything anymore. It’s really just a database artifact.
- we now do not keep the invocation number anymore - instead, we keep different invocations from different trials in a list
There can be multiple experiments for the same commit id. This is generally a little problematic, but since we designed the user interface around commit ids, I think this is something we need to handle. I would assume that multiple experiments do not introduce extra changes, and should just be treated like multiple trials from the same experiment. This is most relevant for the warmup data visualization and the profiles. Profiles should be easy enough. We will simply show a list of available profiles. For warmup data plotting, well, currently everything is going to be shown on the same plot. Not sure how well that works, but it’s a rare case, so, we can solve it of there are actually problems showing up. We also now need just the runId in the button info for profile and warmup buttons. We simply use the full commit hashes from the page header where we already have them.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In this PR, we combine data in the comparison view for multiple experiment ids and trial ids.
We essentially stop using expId and trialId to distinguish data sets from now on.
Instead, we rely solely on commit ids.
The user interface of ReBenchDB is all around commit ids, so, it makes sense to combine multiple experiments that provide data for a specific revision, and the assumption is that there's no change in the setup, environment, benchmarks, or executors between different experiments on the same source revision.
As a consequence, the plots to visualize warmup data may need to show data from multiple trials/experiments.
The profiling info now can be a list of profiles, which are all shown under the corresponding revision.
We do not track invocation numbers anymore in the data for computing comparisons.
The reason is that we assume invocations to be independent, and it doesn't matter in which order they appeared.
This makes it simple to combine invocations from different experiments and trials, which are for the same experiments.
This fixes #158 and #159.