Add Enzyme tests

[devmotion]

I had tried to run Enzyme tests, and it seems currently quite a few tests fail/error.

[yebai]

The fact that it doesn't segfault in so many tests is a good sign.

Related: TuringLang/Turing.jl#1887

[wsmoses]

@devmotion can you rerun on latest Enzyme main?

[wsmoses]

fyi the test failures appear to be due to currently unimplemented derivatives of lapack functions. probably should disable those tests.

[wsmoses]

@devmotion @yebai at this point, everything in this PR works except:

• differentiating through Lapack functions is not yet supported by Enzyme
• your test code uses an active return for a function which returns a vector (this must be duplicated).

Moreover all the errors are relatively nice.

at this point there aren't Enzyme bugs here and merging this is now on your end (presumably marking the code using lapack as unsupported atm, and fixing your test handler to not use an active vector for that input)

[devmotion]

Thank you, I'll try to have a look at and fix the remaining issues this week.

[devmotion]

The latest commit added a way to mark some Enzyme tests as broken (and I also introduced a bug in the test setup 😄) but it also reveals that there are many more issues - so far only a tiny initial testset was run but no tests of the log densities of the uni-, multi- and matrixvariate distributions we're actually interested in.

[devmotion]

@wsmoses Is this an enzyme bug or did I call autodiff incorrectly? https://github.com/TuringLang/DistributionsAD.jl/actions/runs/5872312798/job/15923610267?pr=254#step:6:303

[wsmoses]

Definitely seems like a bug (that we need a minimal reproducer for)

…

On Wed, Aug 16, 2023 at 4:26 PM David Widmann ***@***.***> wrote: @wsmoses <https://github.com/wsmoses> Is this an enzyme bug or did I call autodiff incorrectly? https://github.com/TuringLang/DistributionsAD.jl/actions/runs/5872312798/job/15923610267?pr=254#step:6:303 — Reply to this email directly, view it on GitHub <#254 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJTUXBUFJ5TP4AJUKMV3ADXVRY3HANCNFSM6AAAAAA2LPVOW4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[yebai]

One slightly annoying thing I found about the Enzyme error message is printing a large amount of LLVM code when compilation fails (see here for an example). This is useful when developing Enzyme (at least in the early stages) but not very helpful for users of Enzyme. Very few people can read such LLVM code and figure out why Enzyme couldn't compile. They also add clutter to CI logs. I suggest disabling such LLVM messages by default but providing an API for developers to activate them when needed.

[wsmoses]

@devmotion did you have a chance to isolate that 1.6 error that crashes Julia (so we can ensure it does not)?

[wsmoses]

@devmotion can you repush (its not letting me rerun tests for some reason)

[wsmoses]

@devmotion hm I still can't retrigger this, if you can have the tests run again.

[yebai]

@wsmoses you should have the priveledges to trigger CI (re-)runs -- can you try again?

[yebai]

I think Enzyme might still struggle with a few MvNomal tests, which was revealed after 8b9fa77

[wsmoses]

open issues and we'll get those sorted!

[wsmoses]

Oh yeah runtime activity isn't supported yet for forward mode blas, so my recommendation here is to turn on runtime activity more selectively for which tests need it

[mhauru]

Some open issues that are causing test failures here. Some might just need marking as broken, others seem more in need of fixing:

• Missing Rmath derivatives EnzymeAD/Enzyme.jl#1620 (marked as broken)
• Implement besseli EnzymeAD/Enzyme.jl#1603 (marked as broken)
• xlogy grad wrong at 0 EnzymeAD/Enzyme.jl#1601
• Cannot deduce type for QuadGK EnzymeAD/Enzyme.jl#1599
• Cholesky solve segfault EnzymeAD/Enzyme.jl#1649
• unwrapMode != UnwrapMode::LegalFullUnwrap with Julia <1.8 EnzymeAD/Enzyme.jl#1629
• Indeterministic illegal instruction: 4 EnzymeAD/Enzyme.jl#1626
• Turing model crashes without runtimeActivity EnzymeAD/Enzyme.jl#1623
  • Likely a Julia compiler bug
• Illegal type analysis - BigFloat EnzymeAD/Enzyme.jl#1621
• Turing.jl error with AdvancedHMC EnzymeAD/Enzyme.jl#1696

See also:

• Simpler error messages for missing gradient rules EnzymeAD/Enzyme.jl#1625
• Raise Julia errors rather than crashing on Assertion failures EnzymeAD/Enzyme.jl#1634

I was in the process of checking if I could find more issues in this test suite, but got distracted by TuringLang/DynamicPPL.jl#643 and related issues. Will get back to this once I'm done minimising what I've found there.

[yebai]

I marked distributions that involve ccalls without rules as broken so CI won't fail. Enzyme seem to run in most cases but could return incorrect gradients some distribution density functions.

[wsmoses]

Cool, open an issue for whatever is wrong and we’ll get it fixed

[mhauru]

Draft PR for doing the reverse, namely adding Distributions.jl integration tests into Enzyme: EnzymeAD/Enzyme.jl#1819

[wsmoses]

We're going to need individual issues for any failures. The PR above doesn't presently work as it hits a version issue with QuadGK.

[yebai]

Closed in favour of EnzymeAD/Enzyme.jl#1819