You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Now that we have a concrete use case for it, I think this is worth considering. The rough structure would be similar to what Zygote automatically generates now. Namely, keep a stack of pullbacks (from rrule_via_ad) around when running forward, and apply them in reverse in the pullback.
The main challenge lies in constraining the type of the stack and/or the final output. Push comes to shove I believe this is possible with return_type, but ideally keeping track of a running overall pullback/output type would be sufficient. It may be simpler to only take this path if the eltype of layers is concrete, but I'm not sure of how to do that with just dispatch.
Another consideration is double-differentiation, but IIUC we can use the opt-out functionality in CRC for this?
The text was updated successfully, but these errors were encountered:
Now that we have a concrete use case for it, I think this is worth considering. The rough structure would be similar to what Zygote automatically generates now. Namely, keep a stack of pullbacks (from
rrule_via_ad
) around when running forward, and apply them in reverse in the pullback.The main challenge lies in constraining the type of the stack and/or the final output. Push comes to shove I believe this is possible with
return_type
, but ideally keeping track of a running overall pullback/output type would be sufficient. It may be simpler to only take this path if the eltype oflayers
is concrete, but I'm not sure of how to do that with just dispatch.Another consideration is double-differentiation, but IIUC we can use the opt-out functionality in CRC for this?
The text was updated successfully, but these errors were encountered: