add CUDA extension

[CarloLucibello]

Building on FluxML/NNlib.jl#492, this removes CUDA as a dependency and implements option (1) in #2265, meaning that the CUDA functionality is now unlocked with the explicit double import using CUDA, cuDNN in users' code.

Other notes:

• Minimum supported julia version becomes 1.9
• This is a breaking change
• Moving to option (2) in how to make CUDA functionalities an extension #2265 requires minimal changes to this PR and the creation of a FluxCUDA package.
• I didn't add/remove/change any test, just moved them around
• I'm closely following the structure of the AMDGPU extension

[codecov-commenter]

Codecov Report

Patch coverage: 18.00% and project coverage change: -1.86 ⚠️

Comparison is base (24b1eb2) 80.87% compared to head (dc9e252) 79.02%.

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2268      +/-   ##
==========================================
- Coverage   80.87%   79.02%   -1.86%     
==========================================
  Files          29       31       +2     
  Lines        1710     1721      +11     
==========================================
- Hits         1383     1360      -23     
- Misses        327      361      +34     

Impacted Files	Coverage Δ
ext/FluxAMDGPUExt/FluxAMDGPUExt.jl	0.00% <0.00%> (ø)
ext/FluxAMDGPUExt/batchnorm.jl	0.00% <ø> (ø)
ext/FluxAMDGPUExt/conv.jl	0.00% <ø> (ø)
ext/FluxAMDGPUExt/functor.jl	0.00% <0.00%> (ø)
ext/FluxCUDAExt/functor.jl	0.00% <0.00%> (ø)
ext/FluxCUDAExt/utils.jl	0.00% <0.00%> (ø)
ext/FluxCUDAcuDNNExt/FluxCUDAcuDNNExt.jl	0.00% <0.00%> (ø)
src/Flux.jl	66.66% <ø> (ø)
src/losses/Losses.jl	100.00% <ø> (ø)
src/utils.jl	97.31% <ø> (-0.02%)	⬇️
... and 2 more

... and 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[ToucheSir]

I'm pretty ambivalent on whether 1) or 2) would be better, but 2) might let us keep compat with older Julia versions if we made it load NNlibCUDA and/or the Flux CUDA code on <1.9.

[CarloLucibello]

I'm pretty ambivalent on whether 1) or 2) would be better, but 2) might let us keep compat with older Julia versions if we made it load NNlibCUDA and/or the Flux CUDA code on <1.9.

let's discuss this in #2265

[CarloLucibello]

@ToucheSir your suggestion works!

[CarloLucibello]

This should be ready to go. After merging this I will wait a couple of days, do some local testing, then tag the v0.14 release. Cc @ToucheSir @mcabbott @darsnack

[ToucheSir]

I'm seeing this warning on the GPU CI despite all the tests passing. Do you know what's going on with that?

Edit: if I print out the error from this, it's still the same not found in dependencies one :(

[ToucheSir]

Ok, tried some ideas locally and this appears to work:

try
    Base.require(Main, :cuDNN)

Not sure how general this is, but I haven't encountered any issues yet.

[CarloLucibello]

This works!

[CarloLucibello]

@ToucheSir everything seems fine now. Should we pull the trigger?

[darsnack]

No complaints from me

[IanButterworth]

Awesome!