Add freeze!/thaw!
#112
Add freeze!/thaw!
#112
Conversation
src/adjust.jl
Outdated
| julia> s # Leaf(..., true) means frozen | ||
| (x = (Leaf(Momentum{Float32}(0.01, 0.9), [0.0], true), ()), y = Leaf(Momentum{Float32}(0.01, 0.9), [3.14159])) |
There was a problem hiding this comment.
Maybe just a clear show would be nice here?
There was a problem hiding this comment.
Like make it a keyword, you mean?
I'd like to make them colourful too... #53
|
Shall we do this? Would be nice to include in FluxML/Flux.jl#2114 while sorting that out. |
docs/src/index.md
Outdated
|
|
||
| Optimisers.update!(opt, net, gradient(m -> sum(m(x)), net)...); | ||
|
|
||
| opt # bias = Leaf(Momentum{Float32}(0.01, 0.9), Float32[0.0, 0.0], frozen = true) |
There was a problem hiding this comment.
This would be nice as a doctest.
There was a problem hiding this comment.
I tried & now everything is broken locally, I don't understand Documenter.
Note that the tests at present don't load Flux at all. And that I think testing the output of this block probably means testing a bunch of Flux printing which I'm not sure this package should ever break if that changes.
There was a problem hiding this comment.
Fair point. In that case, this line could be eliminated entirely and the comment moved to directly before/after the freeze! call. Also duplicating it after the thaw! call to show frozen = false.
opt_state = Optimisers.setup(Optimisers.Momentum(), net);
...
Optimisers.freeze!(opt_state.layers[3])
# opt_state.layers[3].bias = Leaf(Momentum{Float32}(0.01, 0.9), Float32[0.0, 0.0], frozen = true)
...
Optimisers.thaw!(opt)
# opt_state.layers[3].bias = Leaf(Momentum{Float32}(0.01, 0.9), Float32[0.0, 0.0], frozen = false)
...There was a problem hiding this comment.
Last commit does something like this
| using Flux, Optimisers | ||
|
|
||
| x = randn(Float32, 28, 28, 1, 1); | ||
| net = @autosize (size(x)...,) Chain( |
There was a problem hiding this comment.
what's the point of splatting here instead of just using size(x)?
There was a problem hiding this comment.
The macro at present wants to see a tuple expression, not a call. It could be made to accept anything.
Replaces #49 with something much simpler: now that
Leafis mutable, you can simply pass it the appropriate limb of the state tree, rather than having some explicit address mechanism.Addresses some of #107
PR Checklist