Hello.
I use the LevenbergMarquardt algorithm with linear inequalities for least squares fitting. The target function has an analytic jacobian, but the optimizer's behavior seems strange to me: optimization is faster if I use a numerical derivative. I will study the problem in detail and give technical details at the end.
The calculation of the target function with the calculation of jacobian lasts 15 times longer than the calculation without jacobian. And this is good, because the optimization takes place in the 40dimensional space. So, the optimization should be faster with jacobian. But it's not. With different parameters, optimization lasts 35% longer with an analytic derivative. For an unknown reason, the optimizer makes more steps with the analytic derivative. I have performed a hundred solutions of the regression problem and got that the version with the numerical derivative makes 181 steps on the average, with the standard deviation 87. Of 100 times, two stopped because they reached the limit of 500 steps. With analytic jacobian enabled, the optimizer makes 185 steps on average, with a standard deviation of 166. Out of 100 times, 16 stopped at 500 steps.
I use rough stopping criterion alglib::minlmsetcond(state_, 0.0001, 500) because it is faster. If you put step norm 0.e12, it gets worse. The numerical derivative makes 206 steps with a standard deviation of 87 and 3/100 end in step 500. The analytical jacobian takes 202 steps with a standard deviation of 178 and 21/100 ending in step 500.
Why is the number of steps different? Why does analytical jacobian need more steps? 500iterations stop at the right minimum, by the way. I also checked the jacobian with alglib::minlmoptguardgradient(), so it works correctly.
Profiling shows that alglib_impl::minlmiteration takes a lot of CPU time. You can expect the optimizer to take more steps, so the target function with the analytic jacobian will take a larger share than the function with the numerical derivative. But it is not. With a numerical derivative, minlmiteration takes 50% of the CPU, and with an analytic jacobian it takes 60%. And that's weird, too.
I have set boundbox and linear inequality constraints A*x < 0. Is it normal that minlmiteration works so long? In the target function, I fill and solve 130 linear systems AX = BY. 100 of them are 1x1 and the rest are sparse. The largest matrix is A 253x253. I know that free alglib doesn't have vectorization, but does the optimizer should work longer than this target function? At the moment, one regression solution takes about 1 second, and minlmiteration takes 0.6 sec of it.
Technical characteristics: Alglib::minlmcreatevj() method Generally speaking, optimization is launched in each thread, but if you leave one thread, nothing changes. All measurements are performed in singlethreaded mode. alglib::minlmsetcond(state, 0.0001, 500); alglib::minlmsetacctype(state, 1); If a numerical derivative is used, the step of differentiation is 0.001. boundbox restrictions and linear inequality A * x < 0. Optimization space has dimension 40. All coordinates are small and of the same order: 0.001 < x < 100 gcc: O2 marchnative
The code is not simple, but if needed, it's here: Running the optimizier: https://github.com/SteshinSS/khnum/blob/master/src/solver/solver.cpp#L54
Setting parameters: https://github.com/SteshinSS/khnum/blob/master/src/solver/solver.cpp#L88
Running the alglib::minlmoptimize() https://github.com/SteshinSS/khnum/blob/master/src/solver/solver.cpp#L141
Thank you in advance for your answer.
