How to get latest minimizer (u) instead of the best one for optimization?

[ThummeTo]

Question❓

Hi,

I really like the quality of the documentation, however I struggle to find a functionality I am pretty sure that it exists:
If I get this correct, Optimization.jl always returns the minimizer u for the best (minimum/maximum) f evaluation - this is nice and often what I want.
However in some cases one is more interested in the latest u - so the last try to find a new optimum (think e.g. of having some stochastic effects in your f). Is there a keyword or similar?

I would assume to find this on this page of the docu:
Common-Solver-Options-(Solve-Keyword-Arguments)

Thanks in advance!

[Vaibhavdixit02]

I really like the quality of the documentation,

I am genuinely surprised 🤣

You can use a callback to store the us and pass it to the solve call -

ustore = []
function my_storing_callback(state, l)
    push!(ustore, state.u)
    return false
end

[ThummeTo]

I am genuinely surprised 🤣

Yeah, I actually found the things that I'd been looking for, so 👍 (except the one above)

Thank you! Ok, this is actually the way I am doing it right now... maybe this is something for a feature request (e.g. adding u_latest to the result struct) or does this feel like an exotic requirement?

[Vaibhavdixit02]

Since some solvers already store this in their results I haven't seen a reason to do this and increase the memory usage even more for the results (and you as the user can do it with the callback)

[ChrisRackauckas]

If some solvers already store this, then you just use the same reference and there's no extra memory cost?

[ThummeTo]

I have another example where this is very common: Stochastic batching when optimizing ML models... without the option to pick the latest instead of the best minimizer, the optimization will return the minimizer of the best batch element loss - which might be a completely outdated parameter set ...

Also: The callback gets called one last time for the best element after optimization finished, so with the code above:

ustore = []
function my_storing_callback(state, l)
    push!(ustore, copy(state.u)) # copy required because inplace modification of `state.u`
    return false
end
ustore[end] # the best value
ustore[end-1] # actually the latest value during optimization

I didn't check the implementation, but wouldn't it be possible to let a keyword decide if either the state best or latest gets stored without additional memory required?

Thanks!

[Vaibhavdixit02]

I am not sure if this was specific for Optimisers.jl solvers since you mention stochastic methods, if it is you can pass save_best=false as kwarg in the solve call to disable saving the best and then it'll just give you the latest set of parameters and loss

[lsmohan]

Hi

Thanks for all the great work with the sciml ecosystem! I am working on a paper related to NeuralODE and came across the following problem where the returned optimization parameters were not the "best" I am asking this here rather than in Discourse since it seems related to this thread.

Here is a MWP that seems to be show that the returned objective value is not the "best" even when save_best=true is set in the OptimizationProblem. I am trying this on a mac M2.

==

using Zygote
using Optimization, OptimizationOptimisers

rosenbrock(u, p) = (p[1] - u[1])^2 + p[2] * (u[2] - u[1]^2)^2
x0 = zeros(2)

optf = OptimizationFunction(rosenbrock, AutoZygote())
prob = OptimizationProblem(optf, x0, [1.0, 100.0], save_best=true)
function cb(state, l)
@show l
false
end
sol = solve(prob, OptimizationOptimisers.Adam(0.01), maxiters=1000, callback=cb)

sol.objective

==

sol.objectve returns the latest objective evaluation rather than some of the better values (to the tume of 1e-13 evaluated earlier)

My Project.toml is as follows

Status ~/Julia/ODE/Lux/Project.toml
[6e4b80f9] BenchmarkTools v1.5.0
[336ed68f] CSV v0.10.15
[13f3f980] CairoMakie v0.12.18
[b0b7db55] ComponentArrays v0.15.20
[f68482b8] Cthulhu v2.16.0
[a93c6f00] DataFrames v1.7.0
[82cc6244] DataInterpolations v6.6.0
[0c46a032] DifferentialEquations v7.15.0
[b4f34e82] Distances v0.10.12
[5789e2e9] FileIO v1.16.6
[1fa38f19] Format v1.3.7
[28b8d3ca] GR v0.73.10
[f67ccb44] HDF5 v0.17.2
[033835bb] JLD2 v0.5.10
[b2108857] Lux v1.4.4
[33e6dc65] MKL v0.7.0
[6f286f6a] MultivariateStats v0.10.3
[76087f3c] NLopt v1.1.1
[872c559c] NNlib v0.9.26
[429524aa] Optim v1.10.0
[7f7a1694] Optimization v4.0.5
[36348300] OptimizationOptimJL v0.4.1
[42dfb2eb] OptimizationOptimisers v0.3.6
[1dea7af3] OrdinaryDiffEq v6.90.1
[d96e819e] Parameters v0.12.3
[1ed8b502] SciMLSensitivity v7.72.0
[10745b16] Statistics v1.11.1
[2913bbd2] StatsBase v0.34.4
⌅ [09ab397b] StructArrays v0.6.18
[5d786b92] TerminalLoggers v0.1.7
[b8865327] UnicodePlots v3.7.1
[ddb6d928] YAML v0.4.12
[ade2ca70] Dates v1.11.0
[56ddb016] Logging v1.11.0
[3fa0cd96] REPL v1.11.0
[9a3f8284] Random v1.11.0

Regards
Mohan

[Vaibhavdixit02]

Does it work if you pass it in solve instead of OptimizationProblem?

[lsmohan]

No. It still returns the same objective.

[lsmohan]

I did some investigation. Looks like the check before the last iteration is not activated. (Line 133 of OptimizationOptimisers.jl).

I modified that from
if i == length(data)*epochs #Last iter, revert to best.
to
if iterations == length(data)*epochs #Last iter, revert to best.
and it seems to work now.

[Vaibhavdixit02]

That makes sense, this wasn't updated when we changed to separating out the maxiters and epochs args, do you want to do a PR?

[lsmohan]

Sure. I will create one by tomorrow.