Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature - Long(er) term test result tracking. #1056

Open
jflo opened this issue Mar 19, 2024 · 0 comments
Open

Feature - Long(er) term test result tracking. #1056

jflo opened this issue Mar 19, 2024 · 0 comments

Comments

@jflo
Copy link
Contributor

jflo commented Mar 19, 2024

The Besu team ❤️ having a resource like hivetests.ethdevops.io to consult. It is a very helpful piece of infrastructure which we do not have to maintain! We would like to track Besu's performance on the wide variety of test suites and integrate it into our periodic reviews; it's important to be aware of how our test results are trending, hopefully always improving. So we're suggesting a feature that may be useful to all client teams- simple access to historical test results, over a medium-term timeframe.

This data wouldn't need to be deeply analyzed over time, we just want to keep a running metric of what test suites are passing at what rates. This could be as simple as appending to a static CSV file every night and making that available. Possible implementation details:

  • Separate the test results json from all logging output from clients and simulators. The logs are a ton of data, which we don't need and those can be pruned much more aggressively. The test results json files are much smaller, and should be retained for longer.
  • clientInfo fields would need to be populated consistently. They seem to only be populated for the first test a client executes. Including this consistently on each test makes aggregating easier.
  • Periodic job to aggregate results. Nightly is probably fine.
  • Means to serve the results up for use by client teams.

Eventually, we'd like to get to a point where we can proudly display a HIVE-PASSING badge on our github home :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant