ALGLIB forum
It is currently Tue Jul 16, 2024 2:35 am

All times are UTC

Forum rules

1. This forum can be used for discussion of both ALGLIB-related and general numerical analysis questions
2. This forum is English-only - postings in other languages will be removed.

Post new topic Reply to topic  [ 1 post ] 
Author Message
 Post subject: Different number of steps of Levenberg-Marquardt
PostPosted: Sat Apr 11, 2020 1:20 pm 

Joined: Sat Apr 11, 2020 10:49 am
Posts: 1

I use the Levenberg-Marquardt algorithm with linear inequalities for least squares fitting. The target function has an analytic jacobian, but the optimizer's behavior seems strange to me: optimization is faster if I use a numerical derivative. I will study the problem in detail and give technical details at the end.

The calculation of the target function with the calculation of jacobian lasts 15 times longer than the calculation without jacobian. And this is good, because the optimization takes place in the 40-dimensional space. So, the optimization should be faster with jacobian. But it's not. With different parameters, optimization lasts 35% longer with an analytic derivative. For an unknown reason, the optimizer makes more steps with the analytic derivative. I have performed a hundred solutions of the regression problem and got that the version with the numerical derivative makes 181 steps on the average, with the standard deviation 87. Of 100 times, two stopped because they reached the limit of 500 steps. With analytic jacobian enabled, the optimizer makes 185 steps on average, with a standard deviation of 166. Out of 100 times, 16 stopped at 500 steps.

I use rough stopping criterion alglib::minlmsetcond(state_, 0.0001, 500) because it is faster. If you put step norm 0.e-12, it gets worse. The numerical derivative makes 206 steps with a standard deviation of 87 and 3/100 end in step 500. The analytical jacobian takes 202 steps with a standard deviation of 178 and 21/100 ending in step 500.

Why is the number of steps different? Why does analytical jacobian need more steps? 500-iterations stop at the right minimum, by the way. I also checked the jacobian with alglib::minlmoptguardgradient(), so it works correctly.

Profiling shows that alglib_impl::minlmiteration takes a lot of CPU time. You can expect the optimizer to take more steps, so the target function with the analytic jacobian will take a larger share than the function with the numerical derivative. But it is not. With a numerical derivative, minlmiteration takes 50% of the CPU, and with an analytic jacobian it takes 60%. And that's weird, too.

I have set bound-box and linear inequality constraints A*x < 0. Is it normal that minlmiteration works so long? In the target function, I fill and solve 130 linear systems AX = BY. 100 of them are 1x1 and the rest are sparse. The largest matrix is A 253x253. I know that free alglib doesn't have vectorization, but does the optimizer should work longer than this target function? At the moment, one regression solution takes about 1 second, and minlmiteration takes 0.6 sec of it.

Technical characteristics:
Alglib::minlmcreatevj() method
Generally speaking, optimization is launched in each thread, but if you leave one thread, nothing changes. All measurements are performed in single-threaded mode.
alglib::minlmsetcond(state, 0.0001, 500);
alglib::minlmsetacctype(state, 1);
If a numerical derivative is used, the step of differentiation is 0.001.
bound-box restrictions and linear inequality A * x < 0.
Optimization space has dimension 40. All coordinates are small and of the same order: 0.001 < x < 100
gcc: -O2 -march-native

The code is not simple, but if needed, it's here:
Running the optimizier:

Setting parameters:

Running the alglib::minlmoptimize()

Thank you in advance for your answer.

Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 1 post ] 

All times are UTC

Who is online

Users browsing this forum: No registered users and 9 guests

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group