Hi! Ideally, these algorithms should just work without any knowledge of the underlying math. The idea is that being a physicist is already good enough:)
However, exponential fitting is extremely difficult. No, it is EXTREMELY difficult. In particular, its success sometimes depends on algorithm implementation in a nonobvious manner. Some design decision that greatly improve robustness in many important cases turn out to degrade performance in this case. The be specific, I am a bit concerned by the fact that ALGLIB way of regularizing steps may slow down convergence on convolved problems exactly like yours. I explain it in order for you to understand that there is a possibility that you did everything right, and that is ALGLIB that needs to get better.
Now, several questions first: * did you fail to get past the first iteration when using analytic derivatives  or numerical gradient? * how many variables do you have? * sometimes exponential problems get easier when you solve them with tight box constraints first (to prevent accidental escape from the initial point), and then relax constraints and resolve. Did you try something like that?
And recommendations: * for your case (accurate computations over inaccurate experimental data) a small differentiation step is recommended. Something proportional to sqrt(machine_epsilon), like 1E7 or 1E6. But not less, you do not want rounding errors to amplify. * epsx can be chosen higher than the numerical differentiation step, like 1e3 or 1e4 for the beginning.
