Skip to content

Commit

Permalink
ci: add downgrade testing (#300)
Browse files Browse the repository at this point in the history
  • Loading branch information
@avik-pal
avik-pal authored Nov 22, 2024
1 parent 1ef3992 commit 97bf7ad
Show file tree
Hide file tree
Showing 3 changed files with 62 additions and 5 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/CI.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,7 @@ jobs:
- uses: codecov/codecov-action@v5
if: steps.run_tests.outcome == 'success'
with:
file: lcov.info
files: lcov.info
docs:
name: Documentation
runs-on: ubuntu-latest
Expand Down
57 changes: 57 additions & 0 deletions .github/workflows/downgrade.yml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
name: Downgrade
on:
pull_request:
branches:
- main
push:
branches:
- main

concurrency:
# Skip intermediate builds: always.
# Cancel intermediate builds: only if it is a pull request build.
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: ${{ startsWith(github.ref, 'refs/pull/') }}

jobs:
downgrade:
# if: ${{ !contains(github.event.head_commit.message, '[skip tests]') && github.base_ref == github.event.repository.default_branch }}
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
test_group:
- core
- neural_networks
- integration
steps:
- uses: actions/checkout@v4
- uses: julia-actions/setup-julia@v2
with:
version: "1.10"
- uses: julia-actions/julia-downgrade-compat@v1
with:
skip: "ReactantCore"
- name: "Install Dependencies and Run Tests"
run: |
import Pkg
Pkg.Registry.update()
# Install packages present in subdirectories
dev_pks = Pkg.PackageSpec[]
for path in ("lib/ReactantCore",)
push!(dev_pks, Pkg.PackageSpec(; path))
end
Pkg.develop(dev_pks)
Pkg.instantiate()
Pkg.test(; coverage="user")
shell: julia --color=yes --code-coverage=user --depwarn=yes --project=. {0}
id: run_tests
env:
JULIA_PKG_SERVER_REGISTRY_PREFERENCE: eager
REACTANT_TEST_GROUP: ${{ matrix.test_group }}
- uses: julia-actions/julia-processcoverage@v1
if: steps.run_tests.outcome == 'success'
- uses: codecov/codecov-action@v5
if: steps.run_tests.outcome == 'success'
with:
files: lcov.info
8 changes: 4 additions & 4 deletions Project.toml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -31,16 +31,16 @@ ReactantNNlibExt = "NNlib"
ReactantStatisticsExt = "Statistics"

[compat]
AbstractFFTs = "1"
AbstractFFTs = "1.5"
Adapt = "4"
ArrayInterface = "7.10"
CEnum = "0.4, 0.5"
Downloads = "1.6"
Enzyme = "0.13.15"
EnzymeCore = "0.8"
GPUArraysCore = "0.1, 0.2"
EnzymeCore = "0.8.6"
GPUArraysCore = "0.1.6, 0.2"
LinearAlgebra = "1.10"
NNlib = "0.9"
NNlib = "0.9.24"
OrderedCollections = "1"
Preferences = "1.4"
ReactantCore = "0.1.2"
Expand Down

1 comment on commit 97bf7ad

@github-actions
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reactant.jl Benchmarks

Benchmark suite Current: 97bf7ad Previous: 1ef3992 Ratio
ViT base (256 x 256 x 3 x 32)/forward/CPU/Reactant (optimize = :after_enzyme) 6210362195 ns 5413804517 ns 1.15
ViT base (256 x 256 x 3 x 32)/forward/CPU/Reactant 5735036003 ns 6129080328 ns 0.94
ViT base (256 x 256 x 3 x 32)/forward/CPU/Reactant (optimize = :before_enzyme) 5275879695 ns 6885334947 ns 0.77
ViT base (256 x 256 x 3 x 32)/forward/CPU/Reactant (optimize = :only_enzyme) 7431765067 ns 7488256383 ns 0.99
ViT base (256 x 256 x 3 x 32)/forward/CPU/Lux 33171194144 ns 33764137689 ns 0.98
ViT small (256 x 256 x 3 x 4)/forward/CPU/Reactant (optimize = :after_enzyme) 1553038587 ns 1572509918 ns 0.99
ViT small (256 x 256 x 3 x 4)/forward/CPU/Reactant 1543277580 ns 1540574169 ns 1.00
ViT small (256 x 256 x 3 x 4)/forward/CPU/Reactant (optimize = :before_enzyme) 1534690574 ns 1551622605 ns 0.99
ViT small (256 x 256 x 3 x 4)/forward/CPU/Reactant (optimize = :only_enzyme) 3296506322 ns 3307639474 ns 1.00
ViT small (256 x 256 x 3 x 4)/forward/CPU/Lux 2274870712 ns 2559807136 ns 0.89
ViT tiny (256 x 256 x 3 x 32)/forward/CPU/Reactant (optimize = :after_enzyme) 2176196590 ns 2130919371 ns 1.02
ViT tiny (256 x 256 x 3 x 32)/forward/CPU/Reactant 2149612180 ns 2129693004 ns 1.01
ViT tiny (256 x 256 x 3 x 32)/forward/CPU/Reactant (optimize = :before_enzyme) 2187958767 ns 2133876774 ns 1.03
ViT tiny (256 x 256 x 3 x 32)/forward/CPU/Reactant (optimize = :only_enzyme) 3926981418 ns 3968926719 ns 0.99
ViT tiny (256 x 256 x 3 x 32)/forward/CPU/Lux 5699632049 ns 6168170173.5 ns 0.92
ViT tiny (256 x 256 x 3 x 4)/forward/CPU/Reactant (optimize = :after_enzyme) 1487184830 ns 1426118988 ns 1.04
ViT tiny (256 x 256 x 3 x 4)/forward/CPU/Reactant 1479317708 ns 1426378204 ns 1.04
ViT tiny (256 x 256 x 3 x 4)/forward/CPU/Reactant (optimize = :before_enzyme) 1485573585 ns 1409132409 ns 1.05
ViT tiny (256 x 256 x 3 x 4)/forward/CPU/Reactant (optimize = :only_enzyme) 3328538147 ns 3150696061 ns 1.06
ViT tiny (256 x 256 x 3 x 4)/forward/CPU/Lux 1128541230.5 ns 1640702004.5 ns 0.69
ViT tiny (256 x 256 x 3 x 16)/forward/CPU/Reactant (optimize = :after_enzyme) 1775441707 ns 1694201241 ns 1.05
ViT tiny (256 x 256 x 3 x 16)/forward/CPU/Reactant 1781070041 ns 1694859743 ns 1.05
ViT tiny (256 x 256 x 3 x 16)/forward/CPU/Reactant (optimize = :before_enzyme) 1770243638 ns 1692209732 ns 1.05
ViT tiny (256 x 256 x 3 x 16)/forward/CPU/Reactant (optimize = :only_enzyme) 3622272410 ns 3438162145 ns 1.05
ViT tiny (256 x 256 x 3 x 16)/forward/CPU/Lux 3191550201.5 ns 3437779789.5 ns 0.93
ViT small (256 x 256 x 3 x 16)/forward/CPU/Reactant (optimize = :after_enzyme) 2260327329 ns 2125368450 ns 1.06
ViT small (256 x 256 x 3 x 16)/forward/CPU/Reactant 2242602405 ns 2180362988 ns 1.03
ViT small (256 x 256 x 3 x 16)/forward/CPU/Reactant (optimize = :before_enzyme) 2236526873 ns 2113675356 ns 1.06
ViT small (256 x 256 x 3 x 16)/forward/CPU/Reactant (optimize = :only_enzyme) 4035446288 ns 3901836386 ns 1.03
ViT small (256 x 256 x 3 x 16)/forward/CPU/Lux 6075855044 ns 6764812162 ns 0.90
ViT small (256 x 256 x 3 x 32)/forward/CPU/Reactant (optimize = :after_enzyme) 3029617019 ns 2977712820 ns 1.02
ViT small (256 x 256 x 3 x 32)/forward/CPU/Reactant 3038364901 ns 3003667317 ns 1.01
ViT small (256 x 256 x 3 x 32)/forward/CPU/Reactant (optimize = :before_enzyme) 3047756026 ns 3046943792 ns 1.00
ViT small (256 x 256 x 3 x 32)/forward/CPU/Reactant (optimize = :only_enzyme) 5020610211 ns 4857405620 ns 1.03
ViT small (256 x 256 x 3 x 32)/forward/CPU/Lux 21336851068 ns 9041023986 ns 2.36
ViT base (256 x 256 x 3 x 16)/forward/CPU/Reactant (optimize = :after_enzyme) 3358460165 ns 3109229306 ns 1.08
ViT base (256 x 256 x 3 x 16)/forward/CPU/Reactant 3528781027 ns 3163244884 ns 1.12
ViT base (256 x 256 x 3 x 16)/forward/CPU/Reactant (optimize = :before_enzyme) 3221856514 ns 3152278733 ns 1.02
ViT base (256 x 256 x 3 x 16)/forward/CPU/Reactant (optimize = :only_enzyme) 5157120407 ns 5077029212 ns 1.02
ViT base (256 x 256 x 3 x 16)/forward/CPU/Lux 12293712706 ns 15973751896 ns 0.77
ViT base (256 x 256 x 3 x 4)/forward/CPU/Reactant (optimize = :after_enzyme) 1889217446 ns 1845679933 ns 1.02
ViT base (256 x 256 x 3 x 4)/forward/CPU/Reactant 1906465434 ns 1847427891 ns 1.03
ViT base (256 x 256 x 3 x 4)/forward/CPU/Reactant (optimize = :before_enzyme) 1910603050 ns 1823520802 ns 1.05
ViT base (256 x 256 x 3 x 4)/forward/CPU/Reactant (optimize = :only_enzyme) 3769045152 ns 3578977105 ns 1.05
ViT base (256 x 256 x 3 x 4)/forward/CPU/Lux 3405687272 ns 3805640188.5 ns 0.89

This comment was automatically generated by workflow using github-action-benchmark.

Please sign in to comment.