forum.alglib.net
http://forum.alglib.net/

Levenberg Marquardt Code
http://forum.alglib.net/viewtopic.php?f=2&t=546		 Page 1 of 1
Author:  ShanbinZhao [ Fri Mar 02, 2012 8:20 pm ]
Post subject:  Levenberg Marquardt Code
In the LM optimizer, I sometimes hit max iterations when one direction hessian is huge, although the hessian is fairly constant and in theory it should converge much quicker. After a bit debug I found that it is due to the fact that damping factor lambda v is set to a too large number, and this number is carried on to next generation, therefore essentially it is doing a steepest descent back and forth along the shallow valley. Then I put in code to reset lambda v to small number every time a hessian is calculated, (not including secant), and it works. I can't think of any significant side effect (besides the fact it will take several attempts to increase lambda to a high level if indeed needed) of this change and would like to get a confirmation in this forum, or if this is wrong, please shed some light on the topic:)
Author:  Sergey.Bochkanov [ Sat Mar 03, 2012 11:12 am ]
Post subject:  Re: Levenberg Marquardt Code
Two questions:
1. did you set scale of the variables with MinLMSetScale() call?
2. can you post your code, in particular - function being optimized and initial guess?

Significant downside of your approach is that iteration cost can increase - every time you increase lambda, additional Cholesky decomposition is needed. Algorithm should be able to automatically decrease lambda when needed - that's why I want to study your problem in more details.
Author:  ShanbinZhao [ Tue Mar 06, 2012 11:55 pm ]
Post subject:  Re: Levenberg Marquardt Code
1. Yes I did set scales but it is hard for me to know the shape ahead of time so the best I can do is to set them more or less equal, at least not orders or magnitude apart.

2. A very simple example is the following parabola:
errorValues[0] = x;
errorValues[1] = 1000*y;
obviously this is only a dumb example because scaling can help, but let's assume we don't know that since in real world that's what I am facing. Initial value is x=1, y=1. The path the optimizer takes is to reduce y to almost zero and then start to bounce in the valley.

Here is what I added to alglib.minlm.minlmiteration(minlmstate state) function:
Code:
        lbl_5:
            state.needfi = false;
            state.repnfunc = state.repnfunc+1;
            state.repnjac = state.repnjac+1;
           
            //
            // New model
            //
            state.modelage = 0;

            // the following are added
            state.lambdav = double.PositiveInfinity;
            double factor = 0.01;
            for (i = 0; i < n - 1; i++)
            {
              double tmpLambdav = factor * Math.Abs(state.quadraticmodel[i, i]) * Math.Sqrt(state.s[i]);
              if (tmpLambdav > double.Epsilon)
                state.lambdav = Math.Min(state.lambdav, tmpLambdav);
            }
            if (double.IsInfinity(state.lambdav))
              state.lambdav = 1;
Page 1 of 1 All times are UTC
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
http://www.phpbb.com/