forum.alglib.net

In the LM optimizer, I sometimes hit max iterations when one direction hessian is huge, although the hessian is fairly constant and in theory it should converge much quicker. After a bit debug I found that it is due to the fact that damping factor lambda v is set to a too large number, and this number is carried on to next generation, therefore essentially it is doing a steepest descent back and forth along the shallow valley. Then I put in code to reset lambda v to small number every time a hessian is calculated, (not including secant), and it works. I can't think of any significant side effect (besides the fact it will take several attempts to increase lambda to a high level if indeed needed) of this change and would like to get a confirmation in this forum, or if this is wrong, please shed some light on the topic:)

Two questions:
1. did you set scale of the variables with MinLMSetScale() call?
2. can you post your code, in particular - function being optimized and initial guess?

Significant downside of your approach is that iteration cost can increase - every time you increase lambda, additional Cholesky decomposition is needed. Algorithm should be able to automatically decrease lambda when needed - that's why I want to study your problem in more details.

1. Yes I did set scales but it is hard for me to know the shape ahead of time so the best I can do is to set them more or less equal, at least not orders or magnitude apart.

2. A very simple example is the following parabola:
errorValues[0] = x;
errorValues[1] = 1000*y;
obviously this is only a dumb example because scaling can help, but let's assume we don't know that since in real world that's what I am facing. Initial value is x=1, y=1. The path the optimizer takes is to reduce y to almost zero and then start to bounce in the valley.

Here is what I added to alglib.minlm.minlmiteration(minlmstate state) function: