Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Re-enable benchmark CI step #1656

Merged
merged 27 commits into from
May 22, 2024
Merged

Re-enable benchmark CI step #1656

merged 27 commits into from
May 22, 2024

Conversation

erthalion
Copy link
Contributor

@erthalion erthalion commented May 3, 2024 •
edited
Loading

Description

Currently benchmark CI step is disabled. Turn it on with following changes:

  • Use berserker instread of hackbench
  • Use regular ttest instead of IQR
  • Compare CPU/Memory usage instead of hackbench time

#run-benchmark

Checklist

  • Investigated and inspected CI test results
  • Updated documentation accordingly

Testing Performed

Manual testing.

@erthalion erthalion added build-qa-containers Force PR to build QA related containers run-benchmark Ask to run benchmark on a PR and compare it with the baseline and removed build-qa-containers Force PR to build QA related containers labels May 7, 2024
@erthalion erthalion force-pushed the feature/berserker branch 2 times, most recently from 98b646d to 39a9c89 Compare May 8, 2024 16:03
@github-actions GitHub Actions
Copy link

github-actions bot commented May 8, 2024 •
edited
Loading

VM Method Baseline CPU median (%) Test CPU median (%) CPU P-value
rhel.rhel-8 core-bpf 32.38 34.44 🟢
rhel.rhel-8 ebpf 24.56 24.95 🟢
rhel.rhel-9 core-bpf 42.4 42.63 🟢
rhel.rhel-9 ebpf 22.88 66.97 🔴
ubuntu-os.ubuntu-2004-lts core-bpf 45.59 44.06 🟢
ubuntu-os.ubuntu-2004-lts ebpf 32.02 35.28 🟢
ubuntu-os.ubuntu-2204-lts core-bpf 46.82 43.67 🟢
ubuntu-os.ubuntu-2204-lts ebpf 34.22 33.38 🟢
VM Method Baseline Memory median (MiB) Test Memory median (MiB) Memory P-value
rhel.rhel-8 core-bpf 48.55 47.1 🟢
rhel.rhel-8 ebpf 30.52 34.68 🟢
rhel.rhel-9 core-bpf 51.94 50.92 🟢
rhel.rhel-9 ebpf 36.18 94.13 🔴
ubuntu-os.ubuntu-2004-lts core-bpf 51.07 49.66 🟢
ubuntu-os.ubuntu-2004-lts ebpf 29.23 29.51 🟢
ubuntu-os.ubuntu-2204-lts core-bpf 53.07 52.54 🟢
ubuntu-os.ubuntu-2204-lts ebpf 30.52 29.44 🟢

@Stringy
Copy link
Collaborator

Stringy commented May 9, 2024

Just a heads up on the socat integration test failures - the socat image is using alpine:latest which pulls in a new version of socat that seems to consistently fail (I've not looked into precisely what's wrong with it) but we can pin the image to alpine:3.18.4 to stick with the old and working version.

I had to do this in the syscall latency branch as well: 65cf135

@erthalion erthalion added update-baseline Instructs the pipeline to update the baseline with performance data from this PR and removed update-baseline Instructs the pipeline to update the baseline with performance data from this PR labels May 10, 2024
@erthalion erthalion marked this pull request as ready for review May 15, 2024 08:02
@erthalion erthalion requested a review from a team as a code owner May 15, 2024 08:02
Stringy
Stringy reviewed May 15, 2024
View reviewed changes
.github/workflows/benchmarks.yml Show resolved Hide resolved
.github/workflows/benchmarks.yml Outdated Show resolved Hide resolved
docs/troubleshooting.md Outdated Show resolved Hide resolved
docs/troubleshooting.md Outdated Show resolved Hide resolved
docs/troubleshooting.md Outdated Show resolved Hide resolved
integration-tests/scripts/baseline/main.py Show resolved Hide resolved
integration-tests/scripts/baseline/main.py Outdated Show resolved Hide resolved
integration-tests/scripts/baseline/main.py Show resolved Hide resolved
integration-tests/scripts/baseline/main.py Outdated Show resolved Hide resolved
@erthalion
Re-enable benchmark CI step
a50a2f1 
Currently benchmark CI step is disabled. Turn it on with following
changes:

* Use berserker instread of hackbench
* Use regular ttest instead of IQR
* Compare CPU/Memory usage instead of hackbench time
@erthalion
Formatting improvements
c58e217 
Adjust reporting for new format, plus add reset command for cleaning up
the baseline if needed.
@erthalion
Bump QA tag to build a new image
782d032
@erthalion
Use correct image
8e6ee89
@erthalion
An experiment to enable benchmark
035cb41
@erthalion
Rely on a label
ccda220
@erthalion
Search for perf results under the right path
558a683
@erthalion
Update auth part
f470ae1
@erthalion
Allow to manually update the baseline
66efac3
@erthalion
Increase duration
3598791
@erthalion
Use entrypoint argument for workload
e47a958 
Integration test image doesn't have container project directory. Built
workload configuration into the image and switch between them via
entrypoint argument.
@erthalion
Switch separator
c3eb28e
@erthalion
Split memory and cpu for readability
b605b3d
@erthalion erthalion requested a review from Stringy May 22, 2024 12:13
Stringy
Stringy approved these changes May 22, 2024
View reviewed changes
Copy link
Collaborator

@Stringy Stringy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! It'll be nice to have these back

@erthalion erthalion merged commit 5e846a9 into master May 22, 2024
60 of 61 checks passed
@erthalion erthalion deleted the feature/berserker branch May 22, 2024 13:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Stringy Stringy Stringy approved these changes

Assignees
No one assigned
Labels
run-benchmark Ask to run benchmark on a PR and compare it with the baseline
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@erthalion @Stringy