You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
As described at http://blog.dlib.net/2017/12/a-global-optimization-algorithm-worth.html LIPO is supposed to alternate a random search of candidate values for parameters with a local optimiser. The random search depends on the Lipschitz constant (which is taken as the greatest gradient seen thus far), with the highest peak being selected as the next "random" point to choose. Therefore, for relatively flat 1D functions with a relatively sharp feature discovered, this is far from random - it's really more like choosing roughly the midpoint of the largest gap observed thus far. The video makes this obvious if viewed in this light.
lipo version 1.2.2 with dlib version 19.22.0 seems to be sensitive, under certain circumstances, to having an apparently completely different algorithm for the random search component. This apparently also can ignore boundary conditions under certain circumstances.
I found this behaviour whilst optimising a "real" problem of many parameters, but was able to generate a toy 1D function that exhibits the same behaviour when being optimised. The function has a parameter that must be positive, and we think the optimum probably lies in the range 0...6.5. The toy function does indeed have a relatively wide Gaussian peak at x=5.0, with a couple of different slopes and some asymptotic behaviour near 0. The peak at x=5.0 is relatively wide, with the entire region from 4.6 - 5.4 being more optimal than anywhere else on the function, so >10% of x-values within the 0...6.5 bounds are in the correct area. This means a truly random search should end up in the correct region after ~10 random samples. So this shouldn't be a "hard" function to optimise.
However, with certain bounds set, the optimiser goes badly wrong.
Setting the lower bound to 0.0 gives the expected behaviour, and the optimal point is found after ~12 calls. Sample output is below.
That is however unsafe, as the function is undefined at x=0, so we should set the bounds slightly higher than that. If you set the lower bound to a number higher than 0.0 but lower than about 0.006, so use 0.001 for example, things go wrong. The optimiser tries for the 6th observation a point well outside the bounds (47.9) without having tested any point higher than the starting point of 1.3, so the upper bounds expansion should not have happened by that stage. No further points are tested above the initial point, and so the false optima at around x=0.7 is found instead. The "random" search component simply seems not to have happened at all. Sample output is below.
Things get even more weird if you change from a maximizing problem to a minimizing one (make the function return +res rather than -res, and set maximize to false at init). The optimizer doesn't seem to actually minimize - the third observation is made at x=4.959837885048982,y=0.7184097543520752 yet this doesn't get exhaustively searched, and the optimum returned is the starting point of x=1.3,y=0.8061923076923078. Sample output below.
Test code:
import numpy as np
from lipo import GlobalOptimizer
# This test function makes LIPO go badly wrong if lower bound is 0.001 instead of 0.0
def testfunc(x):
res = 0.767+0.03*x+(0.8-min(x,0.8))*0.03+0.00025/x-0.2*np.exp(-0.5*(((x-5.)/0.25)**2))
print(f"x={x},y={res}")
return -res
to_adj = {'x':1.3}
lower_bounds={'x':0.00}
upper_bounds={'x':6.5}
flexible_bounds = {'x':[False,True]}
print(lower_bounds,upper_bounds,flexible_bounds)
evaluations = [(to_adj, testfunc(**to_adj))]
search = GlobalOptimizer(
testfunc,
lower_bounds=lower_bounds,
upper_bounds=upper_bounds,
evaluations=evaluations,
maximize=True,
epsilon=0.001,
flexible_bounds = flexible_bounds,
flexible_bound_threshold=0.3
)
num_function_calls = 60
search.run(num_function_calls)
print("optimal:",search.optimum)
As described at http://blog.dlib.net/2017/12/a-global-optimization-algorithm-worth.html LIPO is supposed to alternate a random search of candidate values for parameters with a local optimiser. The random search depends on the Lipschitz constant (which is taken as the greatest gradient seen thus far), with the highest peak being selected as the next "random" point to choose. Therefore, for relatively flat 1D functions with a relatively sharp feature discovered, this is far from random - it's really more like choosing roughly the midpoint of the largest gap observed thus far. The video makes this obvious if viewed in this light.
lipo version 1.2.2 with dlib version 19.22.0 seems to be sensitive, under certain circumstances, to having an apparently completely different algorithm for the random search component. This apparently also can ignore boundary conditions under certain circumstances.
I found this behaviour whilst optimising a "real" problem of many parameters, but was able to generate a toy 1D function that exhibits the same behaviour when being optimised. The function has a parameter that must be positive, and we think the optimum probably lies in the range 0...6.5. The toy function does indeed have a relatively wide Gaussian peak at x=5.0, with a couple of different slopes and some asymptotic behaviour near 0. The peak at x=5.0 is relatively wide, with the entire region from 4.6 - 5.4 being more optimal than anywhere else on the function, so >10% of x-values within the 0...6.5 bounds are in the correct area. This means a truly random search should end up in the correct region after ~10 random samples. So this shouldn't be a "hard" function to optimise.
However, with certain bounds set, the optimiser goes badly wrong.
Setting the lower bound to 0.0 gives the expected behaviour, and the optimal point is found after ~12 calls. Sample output is below.
That is however unsafe, as the function is undefined at x=0, so we should set the bounds slightly higher than that. If you set the lower bound to a number higher than 0.0 but lower than about 0.006, so use 0.001 for example, things go wrong. The optimiser tries for the 6th observation a point well outside the bounds (47.9) without having tested any point higher than the starting point of 1.3, so the upper bounds expansion should not have happened by that stage. No further points are tested above the initial point, and so the false optima at around x=0.7 is found instead. The "random" search component simply seems not to have happened at all. Sample output is below.
Things get even more weird if you change from a maximizing problem to a minimizing one (make the function return +res rather than -res, and set maximize to false at init). The optimizer doesn't seem to actually minimize - the third observation is made at x=4.959837885048982,y=0.7184097543520752 yet this doesn't get exhaustively searched, and the optimum returned is the starting point of x=1.3,y=0.8061923076923078. Sample output below.
Test code:
Sample output for lower_bounds = 0.0
Sample output for lower_bounds = 0.001
Sample output for lower_bounds = 0.001 and function returning +res and minimize=False
The text was updated successfully, but these errors were encountered: