Open main menu

Changes

Relax 3.3.0

1,243 bytes added, 06:55, 8 September 2014
m
R2eff parameter formatting and model links.
* Improved axis label for plotting R<sub>1&rho;</sub>/R2 as function of effective field w_eff. [https://gna.org/support/?3124 Sr #3124]: Grace graphs production for R<sub>1&rho;</sub> analysis with R<sub>2eff</sub> as function of &Omega;<sub>eff</sub>. [https://gna.org/support/?3138 Sr #3138]: Interpolating theta through spin-lock offset [Omega], rather than spin-lock field strength [w1].
* Added sub-title to the plot of R<sub>1&rho;</sub>/R2 as function of effective field. This is to add information about how the effective field has been interpolated. [https://gna.org/support/?3124 Sr #3124]: Grace graphs production for R<sub>1&rho;</sub> analysis with R<sub>2eff</sub> as function of &Omega;<sub>eff</sub>. [https://gna.org/support/?3138 Sr #3138]: Interpolating theta through spin-lock offset [Omega], rather than spin-lock field strength [w1].
* Added functionality to plot R<sub>1&rho;</sub>/R2 as function of effective field w_eff, for the [[R2eff ]] model. Also renamed a function, to better reflect is functionality. The hard-coding of which models to plot, has been removed. If the exp-type is R<sub>1&rho;</sub>, then the plotting will commence. [https://gna.org/support/?3124 Sr #3124]: Grace graphs production for R<sub>1&rho;</sub> analysis with R<sub>2eff</sub> as function of &Omega;<sub>eff</sub>. [https://gna.org/support/?3138 Sr #3138]: Interpolating theta through spin-lock offset [Omega], rather than spin-lock field strength [w1].
* Added the spin specific residue name and spin_id to the title of the dispersion plots. This is handy, since it is often of interest to have this information at hand, when looking through many graphs. [https://gna.org/support/?3124 Sr #3124]: Grace graphs production for R<sub>1&rho;</sub> analysis with R<sub>2eff</sub> as function of &Omega;<sub>eff</sub>. [https://gna.org/support/?3138 Sr #3138]: Interpolating theta through spin-lock offset [Omega], rather than spin-lock field strength [w1].
* Renamed and improved epydoc information for interpolating function for dispersion values. [https://gna.org/support/?3124 Sr #3124]: Grace graphs production for R<sub>1&rho;</sub> analysis with R<sub>2eff</sub> as function of &Omega;<sub>eff</sub>. [https://gna.org/support/?3138 Sr #3138]: Interpolating theta through spin-lock offset [Omega], rather than spin-lock field strength [w1].
* Improved description in GUI text for user function relax_disp.plot_disp_curves(). The improved description now explains the new features. [https://gna.org/support/?3124 Sr #3124]: Grace graphs production for R<sub>1&rho;</sub> analysis with R<sub>2eff</sub> as function of &Omega;<sub>eff</sub>. [https://gna.org/support/?3138 Sr #3138]: Interpolating theta through spin-lock offset [Omega], rather than spin-lock field strength [w1].
* Extended graph labelling, file naming and return of data for multiple CPMG graphs types. [https://gna.org/support/?3124 Sr #3124]: Grace graphs production for R<sub>1&rho;</sub> analysis with R<sub>2eff</sub> as function of &Omega;<sub>eff</sub>. [https://gna.org/support/?3138 Sr #3138]: Interpolating theta through spin-lock offset [Omega], rather than spin-lock field strength [w1].
* Added system test Relax_disp.test_kteilum_fmpoulsen_makke_check_graphs to check all CPMG graph combinations of: y_axis_types = [Y_AXIS_R2_EFF, Y_AXIS_R2_R1RHO]; x_axis_types = [X_AXIS_DISP, X_AXIS_THETA, X_AXIS_W_EFF]; interpolate_types = [INTERPOLATE_DISP]. This is a total of 6 graphs. The graphs will in most cases be totally equal, since the theta angle is calculated to 90 degrees, and R1 is returned as 0.0, then R2=(R<sub>1&rho;</sub> - R1 cos^2(theta)) / sin^2(theta) = R<sub>1&rho;</sub> = R2eff R<sub>2eff</sub> for CPMG models. [https://gna.org/support/?3124 Sr #3124]: Grace graphs production for R<sub>1&rho;</sub> analysis with R<sub>2eff</sub> as function of &Omega;<sub>eff</sub>. [https://gna.org/support/?3138 Sr #3138]: Interpolating theta through spin-lock offset [Omega], rather than spin-lock field strength [w1].
* Added graphs to check against for system test: Relax_disp.test_kteilum_fmpoulsen_makke_check_graphs. [https://gna.org/support/?3124 Sr #3124]: Grace graphs production for R<sub>1&rho;</sub> analysis with R<sub>2eff</sub> as function of &Omega;<sub>eff</sub>. [https://gna.org/support/?3138 Sr #3138]: Interpolating theta through spin-lock offset [Omega], rather than spin-lock field strength [w1].
* Expanded ex. to example in help text for function. [https://gna.org/support/?3124 Sr #3124]: Grace graphs production for R<sub>1&rho;</sub> analysis with R<sub>2eff</sub> as function of &Omega;<sub>eff</sub>. [https://gna.org/support/?3138 Sr #3138]: Interpolating theta through spin-lock offset [Omega], rather than spin-lock field strength [w1].
* Moved the auto_analyses plot of special R<sub>1&rho;</sub> graphs into the check of has_r1rho_exp_type(). [https://gna.org/support/?3135 Sr #3135]: Optimisation of the R1 relaxation rate for the off-resonance R<sub>1&rho;</sub> relaxation dispersion models.
* Moved the auto-analyses plotting and writing of r2, r2a and r2b for CPMG models into test of has_cpmg_exp_type(). [https://gna.org/support/?3135 Sr #3135]: Optimisation of the R1 relaxation rate for the off-resonance R<sub>1&rho;</sub> relaxation dispersion models.
* Replaced in auto analysis all instances of [[No Rex]] and '[[R2eff' ]] with its equivalent defined variables. [https://gna.org/support/?3135 Sr #3135]: Optimisation of the R1 relaxation rate for the off-resonance R<sub>1&rho;</sub> relaxation dispersion models.* Re-arranged plotting and writing in auto-analyses of relax disp, when model is [[R2eff]]. [https://gna.org/support/?3135 Sr #3135]: Optimisation of the R1 relaxation rate for the off-resonance R<sub>1&rho;</sub> relaxation dispersion models.
* Removed unused variables in auto-analyses of relax_disp. [https://gna.org/support/?3135 Sr #3135]: Optimisation of the R1 relaxation rate for the off-resonance R<sub>1&rho;</sub> relaxation dispersion models.
* Implemented the list "MODEL_LIST_NEST", which define which model are used for nesting. [https://gna.org/support/?3135 Sr #3135]: Optimisation of the R1 relaxation rate for the off-resonance R<sub>1&rho;</sub> relaxation dispersion models.
* Split target function of model [[TAP03]], into a calc and two func_TAP03* variants. One target function will use measured R1 values, while one target function will use the fitted R1 values. They will use the same calculation function. [https://gna.org/support/?3135 Sr #3135]: Optimisation of the R1 relaxation rate for the off-resonance R<sub>1&rho;</sub> relaxation dispersion models.
* Split target function of model [[MP05]], into a calc and two func_MP05* variants. One target function will use measured R1 values, while one target function will use the fitted R1 values. They will use the same calculation function. [https://gna.org/support/?3135 Sr #3135]: Optimisation of the R1 relaxation rate for the off-resonance R<sub>1&rho;</sub> relaxation dispersion models.
* Fix for system test Relax_disp.test_r1rho_kjaergaard_auto_check_graphs(), where the special R<sub>1&rho;</sub> graphs is no longer produced for the [[R2eff ]] models. [https://gna.org/support/?3135 Sr #3135]: Optimisation of the R1 relaxation rate for the off-resonance R<sub>1&rho;</sub> relaxation dispersion models.
* Split target function of model ns_r1rho_2site, into a calc and two func_ns_r1rho_2site* variants. One target function will use measured R1 values, while one target function will use the fitted R1 values. They will use the same calculation function. [https://gna.org/support/?3135 Sr #3135]: Optimisation of the R1 relaxation rate for the off-resonance R<sub>1&rho;</sub> relaxation dispersion models.
* Removed num_points to be used in target and lib function of model ns_r1rho_2site. [https://gna.org/support/?3135 Sr #3135]: Optimisation of the R1 relaxation rate for the off-resonance R<sub>1&rho;</sub> relaxation dispersion models.
* Added more models to be tested in system test Relax_disp.test_r1rho_kjaergaard_auto. [https://gna.org/bugs/?22461 Bug #22461]: NS R1rho 2-site_fit_r1 has extremely high &chi;<sup>2</sup> value in system test Relax_disp.test_r1rho_kjaergaard_missing_r1. [https://gna.org/support/?3135 Sr #3135]: Optimisation of the R1 relaxation rate for the off-resonance R<sub>1&rho;</sub> relaxation dispersion models.
* Implemented the sorting of models, for auto-analyses. [https://gna.org/support/?3135 Sr #3135]: Optimisation of the R1 relaxation rate for the off-resonance R<sub>1&rho;</sub> relaxation dispersion models.
* Implemented partial reading of results file. Before reading a results file, it is determined if the file exists. This makes is possible to read a directory with partial results from a previous analysis. This can be handsome, if reading R2eff R<sub>2eff</sub> values in R<sub>1&rho;</sub> experiments, and the error estimation has been prepared with a high number of Monte Carlo simulations. [https://gna.org/support/?3135 Sr #3135]: Optimisation of the R1 relaxation rate for the off-resonance R<sub>1&rho;</sub> relaxation dispersion models.* Added keyword to relax_disp auto analysis, if R2eff R<sub>2eff</sub> values should be optimised. Here optimisation means minimisation and Monte Carlo simulations of the error. [https://gna.org/support/?3135 Sr #3135]: Optimisation of the R1 relaxation rate for the off-resonance R<sub>1&rho;</sub> relaxation dispersion models.* Modified system test Relax_disp.test_r1rho_kjaergaard_missing_r1 to load previous R2eff R<sub>2eff</sub> values, and not optimise them. [https://gna.org/support/?3135 Sr #3135]: Optimisation of the R1 relaxation rate for the off-resonance R<sub>1&rho;</sub> relaxation dispersion models.
* Fix in back end for relax_disp.parameter_copy, where r2a and r2b should be skipped, since it has already been copied. [https://gna.org/support/?3135 Sr #3135]: Optimisation of the R1 relaxation rate for the off-resonance R<sub>1&rho;</sub> relaxation dispersion models.
* Locked values in system test Relax_disp.test_r1rho_kjaergaard_missing_r1. This is possible after locking the R2eff R<sub>2eff</sub> values and errors from a previous run. [https://gna.org/support/?3135 Sr #3135]: Optimisation of the R1 relaxation rate for the off-resonance R<sub>1&rho;</sub> relaxation dispersion models.
* Removed MODEL_NS_CPMG_2SITE_EXPANDED to be analysed in system test test_hansen_cpmg_data_missing_auto_analysis. The new ordering of models, will make MODEL_NS_CPMG_2SITE_EXPANDED be analysed first, and results copied to model [[CR72]]. This will interfere with the old results. [https://gna.org/support/?3135 Sr #3135]: Optimisation of the R1 relaxation rate for the off-resonance R<sub>1&rho;</sub> relaxation dispersion models.
* Lowered the precision of &delta;&omega; for model [[NS CPMG 2-site expanded]], in system test est_hansen_cpmg_data_auto_analysis_numeric. Model [[NS CPMG 2-site expanded]] is now analysed before MODEL_CR72, which alter the values a bit. [https://gna.org/support/?3135 Sr #3135]: Optimisation of the R1 relaxation rate for the off-resonance R<sub>1&rho;</sub> relaxation dispersion models.
* Lowering of precision in system test Relax_disp.test_r1rho_kjaergaard_missing_r1(). This is due to 64/32-bit issues, between analysing on Linux computer, and testing on Mac computer.
* Made the GUI selection of models for relaxation dispersion more simple. After the implementation of a function which will translate the models, the [[No Rex]] model will be converted to the [[No Rex]] model for R<sub>1&rho;</sub> off-resonance. Also the corresponding 'R1 fit' model will be chosen instead, if R1 data has not been loaded. This makes the model selection easier in the GUI interface.
* Bugfix for Relax_disp.test_bug_21715_clustered_indexerror, where only '[[R2eff']], [[No Rex]] is analysed. This special case was not tested in the translating function.
* Shortening the text in the auto_analysis, and raises a warning if R1 data has not been loaded.
* Rewrote the logic of the key-word 'optimise_r2eff' in the auto-analyses of relax disp. If [[R2eff ]] result file exist in the 'pre_run_dir', this is loaded. If the results contain both values, and errors, then no optimisation is performed on the [[R2eff ]] model. Unless the 'optimise_r2eff' flag is raised, which is not standard.
* Fixes for the Relax_disp.test_hansen_cpmg_data_missing_auto_analysis system test on MS Windows. This is for 32-bit MS Windows systems where the parameter checks need to be loosened.
* Fixes for the Relax_disp.test_r1rho_kjaergaard_missing_r1 system test on MS Windows. This is for 32-bit MS Windows systems where the parameter checks need to be loosened.
* Implemented to the back_end of the value.copy, a "force" flag to overwrite destination value.
* Implemented to the front_end of the value.copy, a "force" flag to overwrite destination value.
* Copying the R2eff R<sub>2eff</sub> value from the reading of [[R2eff ]] results in the auto-analyses of relax_disp.
* Fix for the misspelled is_r1_optimised() function.
* The r1_fit flag is stored in the Disp_minimise_command class and passed into the target function. This matches the details at http://thread.gmane.org/gmane.science.nmr.relax.scm/22850/focus=6736. The specific_analyses.relax_disp.optimisation.Disp_minimise_command class calls is_r1_optimised() and stores the result. This is then passed into the relaxation dispersion target function class.
* Added the offset column to the spectrum list GUI element for the dispersion analysis. This is to complete [https://gna.org/task/?7820 task #7820]. The spectrum list GUI element add_offset() method has been added to insert the offset column when the relax_disp_flag is set. This is called by the update_data() method to fill and update the GUI element.
* Implemented the GUI test Relax_disp.test_bug_22501_close_all_analyse to catch [https://gna.org/bugs/?22501 bug #22501], 'Close all analyses' raises error.
* Inserted intermediate system test, to profile R2eff R<sub>2eff</sub> calculation for R<sub>1&rho;</sub>. System test is: Relax_disp.test_bug_9999_slow_r1rho_r2eff_error_with_mc. This system test actually fails, if one tries to do a grid search. This is related to the R2eff R<sub>2eff</sub> values stored as dictionary, and pipe_control.minimise.grid_setup() will fail. The function 'isNaN(values[i])' cannot handle dictionary.* Modified intermediate system test Relax_disp.test_bug_9999_slow_r1rho_r2eff_error_with_mc to see if the initial grid search for 'i0' I<sub>0</sub> and 'R2eff' R<sub>2eff</sub> estimation can be skipped. This is done by converting the exponential curve, to a linear curve, and calculate the best parameters by a line of best fit by least squares. This seems like a promising method as an initial estimator of 'i0' I<sub>0</sub> and 'r2eff'R<sub>2eff</sub>. For 500 to 2000 Monte Carlo simulations, this could have an influence on the timings, as all grid searchs could be skipped.
* Modified system test test_bug_9999_slow_r1rho_r2eff_error_with_mc to save data arrays. This is to prepare a profiling script.
* Added start script with basic data for profiling the relax curve fit.
* Created a number of system tests to check the diffusion tensor PDB representation. This is to prevent [https://gna.org/bugs/?22502 bugs such as #22502] from ever reappearing. The PDB file contents are hardcoded into the tests and checked. The tests include Structure.test_create_diff_tensor_pdb_ellipsoid, Structure.test_create_diff_tensor_pdb_oblate, Structure.test_create_diff_tensor_pdb_prolate, and Structure.test_create_diff_tensor_pdb_sphere.
* Improved data checking for all of the Structure system tests. Before looping over the structural data, the number of lines in the real file and the newly generated file are compared. This avoids the situation whereby an empty file is produced, accidentally allowing the test to pass.
* Modified following functions (time points are now saved at the [ei][mi][oi][di] index level, at this index level all time points are saved for the R2eff R<sub>2eff</sub> point): interpolate_disp(), to interpolate time points, all time points through the original dispersion points di, are collected and then made unique - this time list can potentially be the largest of all time lists; interpolate_offset(), to interpolate time points, all time points through the original offset points, and then dispersion points di, are collected and then made unique - this time list can potentially be the largest of all time lists; plot_disp_curves_to_file(), to acquire the original relax_times points; return_r2eff_arrays(), to save all time points on the level of [ei][mi][oi][di]. At this index level, it will be a numpy array list with all time values used for fitting. [https://gna.org/bugs/?22461 Bug #22461]: NS R1rho 2-site_fit_r1 has extremely high &chi;<sup>2</sup> value in system test Relax_disp.test_r1rho_kjaergaard_missing_r1.
* Modified back_calc_r2eff() to accept interpolated time points. [https://gna.org/bugs/?22461 Bug #22461]: NS R1rho 2-site_fit_r1 has extremely high &chi;<sup>2</sup> value in system test Relax_disp.test_r1rho_kjaergaard_missing_r1.
* Modified target function of relax dispersion, to use the new list of time points, which are of higher dimension. [https://gna.org/bugs/?22461 Bug #22461]: NS R1rho 2-site_fit_r1 has extremely high &chi;<sup>2</sup> value in system test Relax_disp.test_r1rho_kjaergaard_missing_r1.
* Added Nikolai's original Matlab code to the lib.dispersion.ns_r1rho_2site module docstring. This is the code taken directly form the original funNumrho.m file, which was the origin of the code in this module.
* Further extended the profiling script for curve fitting. Now profiling is in place for the implemented C code method in relax. A similar code should now be devised for numpy array for comparing. But this profiling shows that when contraints=True, is slowing down this procedure by a factor 10 X.
* Further improved the profiling of relax curve fit. This profiling shows, that Python code is about twice as slow as the C code implemented. But it also shows that optimising with scipy.optimize.leastsq is 20 X faster. It also gives reasonable error values. Combining a function for a linear fit to guess the initial values, together with scipy optimise, will be an extreme time win for estimating R2eff R<sub>2eff</sub> values fast. A further test would be to use relax Monte Carlo simulations for say 1000-2000 iterations, and compare to the errors extracted from estimated covariance.* Added verification script, that shows that using scipy.optimize.leastsq reaches the exact same parameters as minfx for exponential curve fitting. The profiling shows that scipy.optimize.leastsq is 10X as fast as using minfx (with no linear constraints). scipy.optimize.leastsq is a wrapper around wrapper around MINPACK's lmdif and lmder algorithms. MINPACK is a FORTRAN90 library which solves systems of nonlinear equations, or carries out the least squares minimization of the residual of a set of linear or nonlinear equations. The verification script also shows, that a very heavy and time consuming Monte Carlo simulation of 2000 steps, reaches the same errors as the errors reported by scipy.optimize.leastsq. The return from scipy.optimize.leastsq, gives the estimated covariance. Taking the square root of the covariance corresponds with 2X error reported by minfx. This could be an extremely time saving step, when performing model fitting in R<sub>1&rho;</sub>, where the errors of the R2eff R<sub>2eff</sub> values, are estimated by Monte Carlo simulations. The following setup illustrates the problem. This was analysed on a MacBook Pro, 13-inch, Late 2011 with no multi-core setup. Script running is: test_suite/shared_data/dispersion/Kjaergaard_et_al_2013/2_pre_run_r2eff.py. This script analyses just the R2eff R<sub>2eff</sub> values for 15 residues. It estimates the errors of R2eff R<sub>2eff</sub> based on 2000 Monte Carlo simulations. For each residues, there is 14 exponential graphs. The script was broken after 35 simulations. This was measured to 20 minutes. So 500 simulations would take about 4.8 Hours. The R2eff R<sub>2eff</sub> values and errors can by scipy.optimize.leastsq can instead be calculated in: 15 residues * 0.02 seconds = 0.3 seconds.* Moved the target function for minimisation of exponential fit into the target functions folder. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.* Implemented initial system test Relax_disp.test_estimate_r2eff for setting up the new user function to estimate R2eff R<sub>2eff</sub> and errors by scipy. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.* Added front end user function relax_disp.r2eff_estimate to estimate R2eff R<sub>2eff</sub> and errors by exponential curve fitting in scipy.optimize.leastsq. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.* Modified check for model, to accept model as input, for error printing. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.* Implemented back end for estimating R2eff R<sub>2eff</sub> and errors by exponential curve fitting with scipy.optimize.leastsq. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.* Documentation fix for new exponential target function. Also added new function to estimate R2eff R<sub>2eff</sub> and I<sub>0</sub> parameters, before minimisation. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.* Small changes to verification scripts, to use &chi;<sup>2</sup> function and use the scaling matrix correct. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.* Split up of system test test_r1rho_kjaergaard_missing_r1, into a verification part. This is to test the new r2eff R<sub>2eff</sub> estimation, which should get the parameter values, as a this 2000 Monto Carlo simulation. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.
* Modified system test Relax_disp.test_estimate_r2eff. This is to compare against errors simulated with 2000 MC. The parameters are comparable, but not equal. Mostly, it seems that the errors from scipy.optimize.leastsq, are twice as high than the Monte Carlo simulations. This affect model fitting, and the calculated &chi;<sup>2</sup> value.
* Added system test Relax_disp.test_estimate_r2eff_error(). This is to get insight in the error difference between 2000 Monto Carlo simulations and then scipy.optimize.leastsq.
* Add dependency check for scipy.optimize.leastsq. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.
* Lowering precision in system test Relax_disp.test_r1rho_kjaergaard_missing_r1. This is R1 estimation with MODEL_NS_R1RHO_2SITE. The lowering of precision is due different system precision.
* Reused the dependency check "scipy_module", since leastsq() has been part of Scipy since 2003.
* Isolated all code related to user function relax_disp.r2eff_estimate into independent module file. All has been isolated to: specific_analyses.relax_disp.estimate_r2eff.
* Split function to minimise with scipy.optimize.leastsq out in estimate_r2eff module. This is to prepare for implementing with minfx.
* Implemented first try to minimise with minfx in estimate_r2eff() function. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.
* Implementation of the target_functions.relax_fit.jacobian() function. This follows from the discussions at http://thread.gmane.org/gmane.science.nmr.relax.devel/6807. The function will calculate the Jacobian matrix for the exponential curve-fitting module. The Jacobian can be used to directly calculate the covariance matrix, for example as described at https://www.gnu.org/software/gsl/manual/html_node/Computing-the-covariance-matrix-of-best-fit-parameters.html. The Jacobian is calculated using the help of the new exponential_dI() and exponential_dR() functions in the target_functions/exponential.c file. These calculate the partial derivatives of the exponential curve with respect to each model parameter separately. The implementation still needs testing and debugging.
* Fixes for the new target_functions.relax_fit.jacobian() function. The Python list of lists is now correctly created and returned.
* Turned off the optimisation constraints for the '[[R2eff' ]] model in the dispersion auto-analysis. This follows from http://thread.gmane.org/gmane.science.nmr.relax.scm/22977/focus=6829. This model does not require constraints at all, and the constraints only cause the optimisation to take 10x longer to complete. Therefore the constraint flag has been set to False for the model.* Initial try to form the Jacobian and Hessian matrix for exponential decay. This can be tried with system test: relax -s Relax_disp.test_estimate_r2eff_error. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.* Intermediate step in estimate R2eff R<sub>2eff</sub> module. It seems that minfx is minimising in a quadratic space because of the power of &chi;<sup>2</sup>, while the general input to scipy.optimize does not do this. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.* Cleaned up target function for leastsq, since arguments to function can be extracted from class. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.* Tried to implement with scipy.optimize.fmin_ncg and scipy.optimize.fmin_cg, but cannot get it to work. The matrices are not aligned well. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.
* Implemented the chi-squared gradient as a C module for the target functions. This simply translates the Python code into C to allow any target function C modules to build its own gradient function.
* Implemented the target_functions.relax_fit.dfunc() gradient function. This is using the Python/C interface to provide a Python function for calculating and returned the chi-squared gradient for the exponential curves.
* The parameter index is now passed into exponential_dI0() and exponential_dR(). This is for the relaxation curve-fitting C module so that the indices are not hardcoded.
* The I<sub>0</sub> and R parameter indices are now defined in the target_function/relax_fit.h header file. This is to abstract the exponential curve parameter indices even more.
* Big cleanup of estimate R2eff R<sub>2eff</sub> module. This is to make the documentation more easy to read and understand. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.
* Created 2 unit tests for the target_functions.relax_fit relax C module. These check the func() and dfunc() Python methods exposed by the module.
* The relax_fit C module unit tests now check if the parameter scaling is functional.
* Added several comments to the R2eff R<sub>2eff</sub> estimate module. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.
* Added a script and log file for calculating the numerical gradient for an exponential curve. This uses the data at http://thread.gmane.org/gmane.science.nmr.relax.devel/6807/focus=6840 and calculates the gradient using the scipy.misc.derivative() function both at the minimum and at a point away from the minimum. The values will be used to construct a unit test to check the C module implementation.
* Created a unit test to check the dfunc() function of the relax_fit C module off the minimum.
* Fix for the test_dfunc_off_minimum() unit test. This is the test class test_suite.unit_tests._target_functions.test_relax_fit.Test_relax_fit. The wrong gradient was being scaled.
* Switched the optimisation algorithm in test_suite/system_tests/scripts/relax_fit.py. This script, used by the Relax_fit.test_curve_fitting_height and Relax_fit.test_curve_fitting_volume system tests, now uses the BFGS optimisation. This is to demonstrate that the exponential curve gradient function dfunc() is implemented correctly and that more advanced optimisation algorithms can be used (excluding those that require the full Hessian d2func() function).
* Got the method of 'Steepest descent' to work properly, by specifying the Jacobian correctly. The Jacobian was derived according to the &chi;<sup>2</sup> function. The key point was to evaluate the two derivative arrays for all times points, and then sum each of the two arrays together, before constructing the Jacobian. This clearly shows the difference between minfx and scipy.optimize.leastsq. scipy.optimize.leastsq takes as input a function F(x0), which should return the array of weighted differences between function value and measured values: "1. / self.errors * (self.calc_exp(self.times, *params) - self.values)". This will be an array with number of elements 'i' corresponding to number of elements. scipy.optimize.leastsq then internally evaluates the sum of squares -> sum[ (O - E)**2 ], and minimises this. This is the &chi;<sup>2</sup>. Minfx requires the function to minimise before hand. So, the "func" should be &chi;<sup>2</sup>. Then the dfunc, and d2func, should be derivative of &chi;<sup>2</sup>, but all elements in the array should still be summed together. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.* Got the Quasi-Newton BFGS to work. This uses only the gradient, this gets the same results as 2000 Monte Carlo with simplex and scipy.optimize.leastsq. Error estimation still not provided. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.* Removed all code regarding scipy.optimize fmin_cg and fmin_ncg. This problem should soon be able to be solved with minfx. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.* Added initial documentation for multifit_covar. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.* Modified profiling script to use the new estimate R2eff R<sub>2eff</sub> module. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.* Modified verify error script, to use new estimate R2eff R<sub>2eff</sub> module. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.* Removed all unnecessary code from estimate R2eff R<sub>2eff</sub> module. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.* More removal of code. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.
* Changed the array declarations in the target_functions/exponential C file and header. Instead of using the pointer format of *xyz, the array format of xyz[] is now being used. These are equivalent and the later is more obvious that this is an array.
* Changed the array declarations in the target_functions/c_chi2 C file and header. Instead of using the pointer format of *xyz, the array format of xyz[] is now being used. These are equivalent and the later is more obvious that this is an array.
* Fixes for the Hessian.py script for numerical integrating the Hessian for an exponential curve.
* Implemented two unit tests to check the Hessian of the target_functions.relax_fit.d2func() function. This compares the calculated Hessian to the numerically integrated values from the test_suite/shared_data/curve_fitting/numeric_gradient/Hessian.py script, showing that the d2func() function is implemented correctly.
* Modified profiling script, but it seems that the dfunc from target_functions.relax_fit does not work. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.* Modified estimate R2eff R<sub>2eff</sub> module, to use C code. But system test Relax_disp.test_estimate_r2eff_error shows that the Jacobian is not correctly implemented to be called in minfx.
* Created an initial test suite data directory for a mixed R<sub>1&rho;</sub> + CPMG dispersion analysis. The generate.py script will be extended in the future to generate both synthetic R<sub>1&rho;</sub> and CPMG data for a common exchange process. Such a data combination should show some minor flaws in the current design of the dispersion analysis and will help to solve these.
* Improvements to the pipe_control.minimise.reset_min_stats() function. The minimise statistics resetting is now more elegantly implemented. And the sim_index keyword argument is accepted by the function and individual Monte Carlo simulation elements can now be reset.
* Modified wrapper function for curve_fit, to only change to list type, if the type is a ndarray. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.
* The model-free reset_min_stats() function has been replaced with the pipe_control.minimise version. The specific_analyses.model_free.optimisation.reset_min_stats() function has been deleted and instead the pipe_control.minimise version is being used.
* Implemented the first try to compute the variance of R2eff R<sub>2eff</sub> and I<sub>0</sub>, by the covariance. This uses the Jacobian matrix. The errors calculated, are though way to small compared 2000 Monte Carlo simulations. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.* Tried to implement the Jacobian from C code. This though also report errors which are to small. Maybe some scaling is wrong. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.
* Modified profiling script to calculate timings. The timings for C code are: Simplex, with constraints = 2.192; Simplex, without constraints = 0.216; BFGS, without constraints = 0.079; Newton, without constraints = 0.031; This is pretty pretty fast. To this profiling script, I would also now add some verification on calculations.
* Tried to verify solution to profiling script. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.* Set the constraints=False when doing Monte Carlo simulations for R2effR<sub>2eff</sub>. This is to speed up the Monte Carlo simulations by a factor X10, when estimating the error for R2effR<sub>2eff</sub>. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.* Implemented the use of "Newton" as minimisation algorithm for R2eff R<sub>2eff</sub> curve fitting instead of simplex. Running the test script: test_suite/shared_data/dispersion/Kjaergaard_et_al_2013/2_pre_run_r2eff.py. For 50 Monte Carlo simulations, the time drop from: 3 minutes and 13 s, to 1 min an 5 seconds. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.
* Changed the relax_fit.py sample script to use Newton rather than Simplex optimisation. This can lead to significantly faster optimisation times, as shown in the commit message http://article.gmane.org/gmane.science.nmr.relax.scm/23081.
* Changed the optimisation description in the relaxation curve-fitting chapter of the manual. The script example has been converted to match the sample script, replacing the Nelder-Mead simplex algorithm with Newton optimisation, and removing the argument turning diagonal scaling off. All the text about only the simplex algorithm being supported due to the missing gradients and Hessians in the C module have been deleted. The text that linear constraints are not supported has also been removed - but this was fixed when the logarithmic barrier constraint algorithm was added to minfx.
* By using minfx, and the reported Jacobian, it is now possible to get the exact same error estimation as scipy.optimize.leastsq. The fatal error was to set the weighting matrix with diagonal elements as the error. There weights are 1/errors**2. There is though some unanswered questions left. The Jacobian used, is the direct derivative of the function. It is not the &chi;<sup>2</sup> derivative Jacobian. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.* Fixed naming of functions, to better represent what they do in module of estimating R2effR<sub>2eff</sub>. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.* Implemented the Jacobian of exponential function in Python code. This now also gets the same error as leastsq and C code. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.* Tried to implement a safety test for linearly-dependent columns in the covariance matrix. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.
* Fixes for the relax_disp.r2eff_estimate user function documentation. This is to allow the relax manual to compile again as the original documentation was causing LaTeX failures.
* Clean up of the declarations in the target_functions.relax_fit C module. The Python list objects are now declared at the start of the functions, and then PyList_New() is called later on. This allows the code to compile on certain Windows systems.
* Removed the user function to estimate the R2eff R<sub>2eff</sub> values and errors with scipy.optimize.leastsq. With the newly implemented Jacobian and Hessian of the exponential decay function, the front-end to scipy.optimize.leastsq does not serve a purpose. This is because minfx is now as fast as scipy.optimize.leastsq, and can estimate the errors from the Jacobian to the exact same numbers as scipy.optimize.leastsq. In addition to that, the covariance can be calculated by QR decomposition. This adds additional feature for checking for a singular matrix. The back-end will still be kept in place for the coming tim, but could be removed later. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.* Added front-end to the new user function relax_disp.r2eff_err_estimate(), which will estimate the R2eff R<sub>2eff</sub> errors from a pipe and spins with optimised values of R2eff and IR<sub>02eff</sub>. and I<sub>0</sub>. The covariance matrix can be calculated from the optimised parameters, and the Jacobian. Big care should be taken not to directly trust these results, since the errors are quite different compared to the Monte Carlo simulations. This implementation, will reach the exact same error estimation as scipy.optimize.leastsq. But with much better control over the data, and insight into the calculations. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.* Added method to automatically perform error analysis on peak heights. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.* Modified system test Relax_disp.test_estimate_r2eff() to first do a grid search, then minimise and then estimate the errors for R2eff and IR<sub>2eff</sub> and I<sub>0</sub>. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.* Added back-end to estimate R2eff R<sub>2eff</sub> errors. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff and R<sub>2eff</sub> and associated errors for exponential curve fitting.* Fix to system test test_estimate_r2eff_error(), to first delete the old error estimations. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.* Added several tests to: test_estimate_r2eff_error, to compare different output from algorithms. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.* Cleaned up code in R2eff error module. R<sub>2eff</sub> error module. Also removed a non working Hessian matrix. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.* Moved code around, and made function multifit_covar() independent of class object. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.* Inserted checks for C module is available in module for estimating R2eff error. R<sub>2eff</sub> error. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.* Removed unnecessary call to experimental Exp class. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.* Renamed system test, that test the user function for estimating the R2eff error: R<sub>2eff</sub> error: test_estimate_r2eff_err, test the user function. test_estimate_r2eff_err_methods, test different methods for getting the error. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.* Added system test, Relax_disp.test_estimate_r2eff_err_auto and extended functionality to the auto-analyses protocol. If "exp_mc_sim_num" is set to "-1" and sent to the auto-analyses, the errors of R2eff R<sub>2eff</sub> will be estimated from the covariance matrix. These errors is HIGHLY likely to be wrong, but can be used in an initial test fase, to rapidly produce data for plotting data. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff and associated R<sub>2eff</sub> and associated errors for exponential curve fitting.* Added script, to be used in GUI test. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.* Added GUI test Relax_disp.test_r2eff_err_estimate, to test the setting of MC sim to -1 for exponential R2eff R<sub>2eff</sub> error estimation. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff and associated errors R<sub>2eff</sub> and associated errors for exponential curve fitting.* Added keyword "exp_mc_sim_num", to the auto-analyses in the GUI. This sets the number of Monte Carlo simulations for R2eff R<sub>2eff</sub> error estimation in exponential curve fitting. When setting to -1, the errors are estimated from the covariance matrix. These errors are highly likely to be wrong, but can be used in Rapid testing of data and plotting. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.* Tried to click the "fit_r1" button in the GUI test, but receives an error: relax --gui-tests Relax_disp.test_r2eff_err_estimate, "AttributeError: 'SpinContainer' object has no attribute 'r1'". [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.* Moved the mc_sim_num GUI element in the analysis tab ip, as it is executed first. Also modified the tooltip. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.* Added a warning to the auto-analyses about error estimation from the covariance. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.* Removed yet another comma from GUI tooltip. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.
* Formatting changes for the lib.periodic_table module. This is in preparation for extending the information content of this module.
* Modified system test 'test_estimate_r2eff_err_auto', to use the GUI script. It seems to work perfect. This is to test against GUI script: test_r2eff_err_estimate [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.* Modified test_estimate_r2eff_err_auto, to set r1_fit to False. This still make the system test pass, and fit R1. So this means R1 fit button is not functioning properly. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.* Fix for warning message in the auto-analyses in the GUI. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.* Tried to improve docstring for API documentation. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.
* Added all of the IUPAC 2011 atomic weights to the lib.periodic_table module. These will be useful for correctly calculating the centre of mass of a molecule.
* The lib.periodic_table method for adding elements is now private.
* Created the unit test infrastructure for the lib.periodic_table module. This includes one unit test of the lib.periodic_table.periodic_table.atomic_weight() function which has not been implemented yet.
* Implemented the lib.periodic_table.periodic_table.atomic_weight() method. This returns the standard atomic weight of the atom as a float.
* Yet another try to make the API documentation working. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.* Implemented system test Relax_disp.verify_estimate_r2eff_err_compare_mc for testing R2eff R<sub>2eff</sub> error as function of Monte Carlo simulation. Note, since the name does not start with "test", but with "verify", this test will not be issued in the system test suite. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff and associated errors for exponential curve fitting.
* Converted the periodic table in lib.periodic_table into a dictionary type object. The new Element container has been added for storing the information about each element in the table. The Periodic_table object used the atomic symbol as a key for each Element instance.
* Modified system test test Relax_disp.test_estimate_r2eff_err_methods() to show the difference between using the direct function Jacobian, or the &chi;<sup>2</sup> function Jacobian. Added also the functionality to the estimate R2eff module, to switch between using the different Jacobians. The results show, that R2eff can be estimated better.
* Fix for time not extracted for CPMG experiments in target_function. [https://gna.org/bugs/?22461 Bug #22461]: NS R1rho 2-site_fit_r1 has extremely high chi2 value in system test Relax_disp.test_r1rho_kjaergaard_missing_r1.
* Fix for interpolating time points, when producing xmgrace files for CPMG experiments. [https://gna.org/bugs/?22461 Bug #22461]: NS R1rho 2-site_fit_r1 has extremely high chi2 value in system test Relax_disp.test_r1rho_kjaergaard_missing_r1.
* Correction for catastrophic implementation of Monte Carlo simulations for exponential curve-fitting R2eff R<sub>2eff</sub> values in the dispersion analysis. A wrong implemented "else if" statement, would add the intensity for the simulated intensity together with the original intensity. This means that all intensity values send to minimisation would be twice as high than usually (if spectra were not replicated). This was discovered for Monte Carlo simulations of R2eff R<sub>2eff</sub> errors in exponential fit. This will affect all analyses using full relaxation exponential curves until now. By pure luck, it seems that the effect of this would be that R2eff R<sub>2eff</sub> errors are half the value they should be. A further investigation shows, that for the selected data set, this had a minimum on influence on the fitted parameters, because the chi2 value would be scaled up by a factor 4. [https://gna.org/bugs/?22554 Bug #22554]: The distribution of intensity with errors in Monte Carlo simulations are markedly more narrow than expected. [https://gna.org/task/?7822 Task #7822]: Implement user function to estimate R2eff R<sub>2eff</sub> and associated errors for exponential curve fitting.
* Added a minfx minimum version check to the dep_check module. This is to avoid problems such as that reported at [http://gna.org/bugs/?22408 bug #22408].
Trusted, Bureaucrats
4,228

edits