Cleanup

[avik-pal]

• Use Tracker.@grad_from_chainrules
• Cleanup DynamicExpressions handling.

[github-actions]

Benchmark Results

Benchmark suite	Current: a005297	Previous: faf1b13	Ratio
Dense(2 => 2)/cpu/reverse/ReverseDiff (compiled)/(2, 128)	3584.8125 ns	3653.125 ns	0.98
Dense(2 => 2)/cpu/reverse/Zygote/(2, 128)	7243.5 ns	7507.5 ns	0.96
Dense(2 => 2)/cpu/reverse/Tracker/(2, 128)	20859 ns	20860 ns	1.00
Dense(2 => 2)/cpu/reverse/ReverseDiff/(2, 128)	9680 ns	9886 ns	0.98
Dense(2 => 2)/cpu/reverse/Flux/(2, 128)	9121 ns	8766.75 ns	1.04
Dense(2 => 2)/cpu/reverse/SimpleChains/(2, 128)	4498.375 ns	4524.75 ns	0.99
Dense(2 => 2)/cpu/forward/NamedTuple/(2, 128)	1124.5159235668789 ns	1117.9358974358975 ns	1.01
Dense(2 => 2)/cpu/forward/ComponentArray/(2, 128)	1180.9407407407407 ns	1195.6870229007634 ns	0.99
Dense(2 => 2)/cpu/forward/Flux/(2, 128)	1789.0172413793102 ns	1805.7636363636364 ns	0.99
Dense(2 => 2)/cpu/forward/SimpleChains/(2, 128)	179.10084033613447 ns	179.2450980392157 ns	1.00
Dense(20 => 20)/cpu/reverse/ReverseDiff (compiled)/(20, 128)	17332 ns	17524 ns	0.99
Dense(20 => 20)/cpu/reverse/Zygote/(20, 128)	17482 ns	18605 ns	0.94
Dense(20 => 20)/cpu/reverse/Tracker/(20, 128)	37480 ns	37831 ns	0.99
Dense(20 => 20)/cpu/reverse/ReverseDiff/(20, 128)	28303 ns	28513 ns	0.99
Dense(20 => 20)/cpu/reverse/Flux/(20, 128)	19857 ns	20068 ns	0.99
Dense(20 => 20)/cpu/reverse/SimpleChains/(20, 128)	17072 ns	17167.5 ns	0.99
Dense(20 => 20)/cpu/forward/NamedTuple/(20, 128)	3874.75 ns	3899.75 ns	0.99
Dense(20 => 20)/cpu/forward/ComponentArray/(20, 128)	3961.25 ns	3971.25 ns	1.00
Dense(20 => 20)/cpu/forward/Flux/(20, 128)	4910.642857142857 ns	4930.714285714285 ns	1.00
Dense(20 => 20)/cpu/forward/SimpleChains/(20, 128)	1658.1 ns	1666.2 ns	1.00
Conv((3, 3), 3 => 3)/cpu/reverse/ReverseDiff (compiled)/(64, 64, 3, 128)	40186467 ns	41918346.5 ns	0.96
Conv((3, 3), 3 => 3)/cpu/reverse/Zygote/(64, 64, 3, 128)	82486916 ns	83032906 ns	0.99
Conv((3, 3), 3 => 3)/cpu/reverse/Tracker/(64, 64, 3, 128)	81308896.5 ns	103393678.5 ns	0.79
Conv((3, 3), 3 => 3)/cpu/reverse/ReverseDiff/(64, 64, 3, 128)	91726627.5 ns	91745276 ns	1.00
Conv((3, 3), 3 => 3)/cpu/reverse/Flux/(64, 64, 3, 128)	78139080 ns	78118255 ns	1.00
Conv((3, 3), 3 => 3)/cpu/reverse/SimpleChains/(64, 64, 3, 128)	11866455 ns	11920766 ns	1.00
Conv((3, 3), 3 => 3)/cpu/forward/NamedTuple/(64, 64, 3, 128)	14246534 ns	14238692 ns	1.00
Conv((3, 3), 3 => 3)/cpu/forward/ComponentArray/(64, 64, 3, 128)	14134714 ns	14119873 ns	1.00
Conv((3, 3), 3 => 3)/cpu/forward/Flux/(64, 64, 3, 128)	11331670 ns	12170472 ns	0.93
Conv((3, 3), 3 => 3)/cpu/forward/SimpleChains/(64, 64, 3, 128)	6374715 ns	6379162 ns	1.00
vgg16/cpu/reverse/Zygote/(32, 32, 3, 1)	114197677 ns	117904196 ns	0.97
vgg16/cpu/reverse/Zygote/(32, 32, 3, 16)	876131684 ns	872552516 ns	1.00
vgg16/cpu/reverse/Zygote/(32, 32, 3, 64)	3094435863 ns	3104938287 ns	1.00
vgg16/cpu/reverse/Tracker/(32, 32, 3, 1)	151677856 ns	165023531 ns	0.92
vgg16/cpu/reverse/Tracker/(32, 32, 3, 16)	735392239 ns	843147014 ns	0.87
vgg16/cpu/reverse/Tracker/(32, 32, 3, 64)	2881895034 ns	2934328537 ns	0.98
vgg16/cpu/reverse/Flux/(32, 32, 3, 1)	79609927 ns	88342252 ns	0.90
vgg16/cpu/reverse/Flux/(32, 32, 3, 16)	671999722.5 ns	794101907 ns	0.85
vgg16/cpu/reverse/Flux/(32, 32, 3, 64)	2804429428 ns	3211387067 ns	0.87
vgg16/cpu/forward/NamedTuple/(32, 32, 3, 1)	29179065 ns	29055401 ns	1.00
vgg16/cpu/forward/NamedTuple/(32, 32, 3, 16)	303678823 ns	304789763 ns	1.00
vgg16/cpu/forward/NamedTuple/(32, 32, 3, 64)	945966862 ns	951651894.5 ns	0.99
vgg16/cpu/forward/ComponentArray/(32, 32, 3, 1)	28979963 ns	28675149 ns	1.01
vgg16/cpu/forward/ComponentArray/(32, 32, 3, 16)	306661469 ns	302923380 ns	1.01
vgg16/cpu/forward/ComponentArray/(32, 32, 3, 64)	940096790.5 ns	958662417 ns	0.98
vgg16/cpu/forward/Flux/(32, 32, 3, 1)	23309415 ns	23972068.5 ns	0.97
vgg16/cpu/forward/Flux/(32, 32, 3, 16)	186259229 ns	210433134 ns	0.89
vgg16/cpu/forward/Flux/(32, 32, 3, 64)	717405784 ns	813505664 ns	0.88
Conv((3, 3), 64 => 64)/cpu/reverse/ReverseDiff (compiled)/(64, 64, 64, 128)	1032198098 ns	1065144944 ns	0.97
Conv((3, 3), 64 => 64)/cpu/reverse/Zygote/(64, 64, 64, 128)	1875023817 ns	1877704952.5 ns	1.00
Conv((3, 3), 64 => 64)/cpu/reverse/Tracker/(64, 64, 64, 128)	2143579280 ns	2132865236 ns	1.01
Conv((3, 3), 64 => 64)/cpu/reverse/ReverseDiff/(64, 64, 64, 128)	2478167384 ns	2343891524 ns	1.06
Conv((3, 3), 64 => 64)/cpu/reverse/Flux/(64, 64, 64, 128)	1827666855.5 ns	1883974195 ns	0.97
Conv((3, 3), 64 => 64)/cpu/forward/NamedTuple/(64, 64, 64, 128)	337604124 ns	338343060 ns	1.00
Conv((3, 3), 64 => 64)/cpu/forward/ComponentArray/(64, 64, 64, 128)	346027030 ns	336049584 ns	1.03
Conv((3, 3), 64 => 64)/cpu/forward/Flux/(64, 64, 64, 128)	370771942.5 ns	349315098 ns	1.06
Conv((3, 3), 1 => 1)/cpu/reverse/ReverseDiff (compiled)/(64, 64, 1, 128)	11847432 ns	11707469 ns	1.01
Conv((3, 3), 1 => 1)/cpu/reverse/Zygote/(64, 64, 1, 128)	29502844 ns	29403961 ns	1.00
Conv((3, 3), 1 => 1)/cpu/reverse/Tracker/(64, 64, 1, 128)	19305497 ns	19129419 ns	1.01
Conv((3, 3), 1 => 1)/cpu/reverse/ReverseDiff/(64, 64, 1, 128)	23963085.5 ns	23896016.5 ns	1.00
Conv((3, 3), 1 => 1)/cpu/reverse/Flux/(64, 64, 1, 128)	18091050 ns	17979594 ns	1.01
Conv((3, 3), 1 => 1)/cpu/reverse/SimpleChains/(64, 64, 1, 128)	1168061 ns	1165109 ns	1.00
Conv((3, 3), 1 => 1)/cpu/forward/NamedTuple/(64, 64, 1, 128)	4427561 ns	4405531 ns	1.01
Conv((3, 3), 1 => 1)/cpu/forward/ComponentArray/(64, 64, 1, 128)	4258649 ns	4226646 ns	1.01
Conv((3, 3), 1 => 1)/cpu/forward/Flux/(64, 64, 1, 128)	2093020 ns	2067222 ns	1.01
Conv((3, 3), 1 => 1)/cpu/forward/SimpleChains/(64, 64, 1, 128)	211419 ns	198624 ns	1.06
Dense(200 => 200)/cpu/reverse/ReverseDiff (compiled)/(200, 128)	295380 ns	291579 ns	1.01
Dense(200 => 200)/cpu/reverse/Zygote/(200, 128)	266687 ns	264828 ns	1.01
Dense(200 => 200)/cpu/reverse/Tracker/(200, 128)	367795 ns	366479 ns	1.00
Dense(200 => 200)/cpu/reverse/ReverseDiff/(200, 128)	411787 ns	407346 ns	1.01
Dense(200 => 200)/cpu/reverse/Flux/(200, 128)	274852 ns	273283 ns	1.01
Dense(200 => 200)/cpu/reverse/SimpleChains/(200, 128)	408220 ns	407576 ns	1.00
Dense(200 => 200)/cpu/forward/NamedTuple/(200, 128)	81202 ns	81292 ns	1.00
Dense(200 => 200)/cpu/forward/ComponentArray/(200, 128)	81712 ns	81824 ns	1.00
Dense(200 => 200)/cpu/forward/Flux/(200, 128)	86541 ns	86612 ns	1.00
Dense(200 => 200)/cpu/forward/SimpleChains/(200, 128)	104605 ns	104647 ns	1.00
Conv((3, 3), 16 => 16)/cpu/reverse/ReverseDiff (compiled)/(64, 64, 16, 128)	185926128.5 ns	186673336 ns	1.00
Conv((3, 3), 16 => 16)/cpu/reverse/Zygote/(64, 64, 16, 128)	344547709 ns	345534133 ns	1.00
Conv((3, 3), 16 => 16)/cpu/reverse/Tracker/(64, 64, 16, 128)	396062486 ns	402937773 ns	0.98
Conv((3, 3), 16 => 16)/cpu/reverse/ReverseDiff/(64, 64, 16, 128)	460171397 ns	458052460.5 ns	1.00
Conv((3, 3), 16 => 16)/cpu/reverse/Flux/(64, 64, 16, 128)	371512830 ns	369972104.5 ns	1.00
Conv((3, 3), 16 => 16)/cpu/reverse/SimpleChains/(64, 64, 16, 128)	346392175 ns	339401356 ns	1.02
Conv((3, 3), 16 => 16)/cpu/forward/NamedTuple/(64, 64, 16, 128)	54300019 ns	55577402.5 ns	0.98
Conv((3, 3), 16 => 16)/cpu/forward/ComponentArray/(64, 64, 16, 128)	54130216 ns	55687376 ns	0.97
Conv((3, 3), 16 => 16)/cpu/forward/Flux/(64, 64, 16, 128)	49668320.5 ns	59754155 ns	0.83
Conv((3, 3), 16 => 16)/cpu/forward/SimpleChains/(64, 64, 16, 128)	28796269 ns	28806305.5 ns	1.00
Dense(2000 => 2000)/cpu/reverse/ReverseDiff (compiled)/(2000, 128)	19162712 ns	18921264 ns	1.01
Dense(2000 => 2000)/cpu/reverse/Zygote/(2000, 128)	19689112 ns	19484060.5 ns	1.01
Dense(2000 => 2000)/cpu/reverse/Tracker/(2000, 128)	23587339 ns	23321717 ns	1.01
Dense(2000 => 2000)/cpu/reverse/ReverseDiff/(2000, 128)	24197863 ns	24042051 ns	1.01
Dense(2000 => 2000)/cpu/reverse/Flux/(2000, 128)	19760425 ns	19592196 ns	1.01
Dense(2000 => 2000)/cpu/forward/NamedTuple/(2000, 128)	6540506 ns	6499211 ns	1.01
Dense(2000 => 2000)/cpu/forward/ComponentArray/(2000, 128)	6540563 ns	6476020 ns	1.01
Dense(2000 => 2000)/cpu/forward/Flux/(2000, 128)	6585432 ns	6472317 ns	1.02

This comment was automatically generated by workflow using github-action-benchmark.

[codecov]

Codecov Report

Attention: Patch coverage is 80.00000% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 87.46%. Comparing base (faf1b13) to head (a005297).

Files	Patch %	Lines
src/layers/extension.jl	66.66%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     LuxDL/Lux.jl#629      +/-   ##
==========================================
+ Coverage   87.42%   87.46%   +0.03%     
==========================================
  Files          43       43              
  Lines        2250     2241       -9     
==========================================
- Hits         1967     1960       -7     
+ Misses        283      281       -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.