Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add performance benchmarking scripts and run in CI #1978

Closed
wants to merge 49 commits into from
Closed

Add performance benchmarking scripts and run in CI #1978

wants to merge 49 commits into from

Commits on Jan 18, 2024

  1. benchmarks: Upload initial models from JuliaDynamics 

    https://github.com/JuliaDynamics/ABMFrameworksComparison/tree/5551d7abf1611d377b3b32346c7774f176af4c65
    @EwoutH
    EwoutH committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    dae101a View commit details
    Browse the repository at this point in the history

  2. benchmarks: Add dictionary with configurations

    @EwoutH
    EwoutH committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    0c284be View commit details
    Browse the repository at this point in the history

  3. benchmarks: Update configurations, use relative imports

    @EwoutH
    EwoutH committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    c67ad70 View commit details
    Browse the repository at this point in the history

  4. benchmarks: Add single script to run all benchmarks

    @EwoutH
    EwoutH committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    9de2647 View commit details
    Browse the repository at this point in the history

Commits on Jan 19, 2024

  1. benchmarks: Update configurations 

    Few less replications, some more seeds. Every benchmark now takes between 10 and 20 seconds (on my machine).
    @EwoutH
    EwoutH committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    b72cdb1 View commit details
    Browse the repository at this point in the history

  2. benchmarks: Add generated pickle files to gitignore 

    That allows switching branches without benchmarks results disappearing
    @EwoutH
    EwoutH committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    80d61be View commit details
    Browse the repository at this point in the history

  3. benchmarks: Update global script to calculate and save stuff 

    Prints some stuff when running and saves a pickle file after running.
    @EwoutH
    EwoutH committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    b6611af View commit details
    Browse the repository at this point in the history

  4. benchmarks: Write a script to statistically compare runs 

    - The bootstrap_speedup_confidence_interval function calculates the mean speedup and its confidence interval using bootstrapping, which is more suitable for paired data.
- The mean speedup and confidence interval are calculated for both initialization and run times.
- Positive values indicate an increase in time (longer duration), and negative values indicate a decrease (shorter duration).
- The results are displayed in a DataFrame with the percentage changes and their confidence intervals.
    @EwoutH
    EwoutH committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    d038758 View commit details
    Browse the repository at this point in the history

  5. benchmarks: Remove seperate benchmark scripts

    @EwoutH
    EwoutH committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    d31bb27 View commit details
    Browse the repository at this point in the history

  6. benchmarks: Black and ruff

    @EwoutH
    EwoutH committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    5a17e8e View commit details
    Browse the repository at this point in the history

  7. benchmarks: Add GitHub Actions workflow

    @EwoutH
    EwoutH committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    70149af View commit details
    Browse the repository at this point in the history

  8. update versions

    @EwoutH
    EwoutH committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    d2f1704 View commit details
    Browse the repository at this point in the history

  9. Update checkout

    @EwoutH
    EwoutH committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    2915730 View commit details
    Browse the repository at this point in the history

  10. Update benchmarks.yml

    @EwoutH
    EwoutH committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    0b5bba9 View commit details
    Browse the repository at this point in the history

  11. install dependencies

    @EwoutH
    EwoutH committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    e3d9355 View commit details
    Browse the repository at this point in the history

  12. test checkout first

    @EwoutH
    EwoutH committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    2de025e View commit details
    Browse the repository at this point in the history

  13. checkout main

    @EwoutH
    EwoutH committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    f026679 View commit details
    Browse the repository at this point in the history

  14. Run on all PRs

    @EwoutH
    EwoutH committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    0b5949a View commit details
    Browse the repository at this point in the history

  15. Add PR repo to checkout

    @EwoutH
    EwoutH committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    2f97a85 View commit details
    Browse the repository at this point in the history

  16. Add working directories

    @EwoutH
    EwoutH committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    6da4316 View commit details
    Browse the repository at this point in the history

  17. "../benchmarks"

    @EwoutH
    EwoutH committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    2794921 View commit details
    Browse the repository at this point in the history

  18. no string

    @EwoutH
    EwoutH committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    4065ea6 View commit details
    Browse the repository at this point in the history

  19. ./benchmarks

    @EwoutH
    EwoutH committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    4b8d908 View commit details
    Browse the repository at this point in the history

  20. Update benchmarks.yml

    @EwoutH
    EwoutH committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    1a5b5d5 View commit details
    Browse the repository at this point in the history

  21. Update benchmarks.yml

    @EwoutH
    EwoutH committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    134edea View commit details
    Browse the repository at this point in the history

  22. Update benchmarks.yml

    @EwoutH
    EwoutH committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    64506ca View commit details
    Browse the repository at this point in the history

  23. remove other workflows

    @EwoutH
    EwoutH committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    63f366e View commit details
    Browse the repository at this point in the history

  24. Python path

    @EwoutH
    EwoutH committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    c5bb3a4 View commit details
    Browse the repository at this point in the history

  25. install tqdm

    @EwoutH
    EwoutH committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    78e34d6 View commit details
    Browse the repository at this point in the history

  26. Update benchmarks.yml

    @EwoutH
    EwoutH committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    1379ece View commit details
    Browse the repository at this point in the history

  27. working directory benchmarks

    @EwoutH
    EwoutH committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    723caab View commit details
    Browse the repository at this point in the history

  28. test

    @EwoutH
    EwoutH committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    b12d6ab View commit details
    Browse the repository at this point in the history

  29. artifact circus

    @EwoutH
    EwoutH committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    8be4bda View commit details
    Browse the repository at this point in the history

  30. don't clean on checkout

    @EwoutH
    EwoutH committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    c78a58a View commit details
    Browse the repository at this point in the history

  31. Revert "don't clean on checkout" 

    This reverts commit c78a58a.
    @EwoutH
    EwoutH committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    dcdeca6 View commit details
    Browse the repository at this point in the history

  32. enable benchmark again

    @EwoutH
    EwoutH committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    d54f477 View commit details
    Browse the repository at this point in the history

  33. install tabulate

    @EwoutH
    EwoutH committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    ba1680b View commit details
    Browse the repository at this point in the history

  34. Resolve deprecation of set-output 

    Warning: The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
    @EwoutH
    EwoutH committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    f50cc4e View commit details
    Browse the repository at this point in the history

  35. 1 replication to go fast

    @EwoutH
    EwoutH committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    bc61345 View commit details
    Browse the repository at this point in the history

  36. single quotes to work in Python 3.11 and lower

    @EwoutH
    EwoutH committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    e99a11a View commit details
    Browse the repository at this point in the history

  37. base 64 again

    @EwoutH
    EwoutH committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    71be9ad View commit details
    Browse the repository at this point in the history

  38. Try via text file

    @EwoutH
    EwoutH committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    4ac96ee View commit details
    Browse the repository at this point in the history

  39. Revert "Try via text file" 

    This reverts commit 4ac96ee.
    @EwoutH
    EwoutH committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    b880496 View commit details
    Browse the repository at this point in the history

  40. use actions/github-script@v7

    @EwoutH
    EwoutH committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    9836e19 View commit details
    Browse the repository at this point in the history

  41. Add fancy colors! 

    If the 95% confidence interval is:
- fully below -3%: 🟢
- fully above +3%: 🔴
- else: 🔵
    @EwoutH
    EwoutH committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    f649925 View commit details
    Browse the repository at this point in the history

  42. Add permission to write to issues 

    See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#permissions
    @EwoutH
    EwoutH committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    dfacbd3 View commit details
    Browse the repository at this point in the history

  43. Remove explicit token 

    You never knows what helps
    @EwoutH
    EwoutH committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    da5c9d9 View commit details
    Browse the repository at this point in the history

  44. Try using pull_request_target

    @EwoutH
    EwoutH committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    ad116b1 View commit details
    Browse the repository at this point in the history

  45. run on ready_for_review and opened

    @EwoutH
    EwoutH committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    6369622 View commit details
    Browse the repository at this point in the history