Difference between revisions of "Tutorial for model free SBiNLab"
(28 intermediate revisions by the same user not shown) | |||
Line 6: | Line 6: | ||
For references, see [http://www.nmr-relax.com/refs.shtml relax references]: | For references, see [http://www.nmr-relax.com/refs.shtml relax references]: | ||
+ | * [[Model-free_analysis_single_field#Protocol|See this description of the protocol by Edward]] and image [http://www.nmr-relax.com/manual/The_diffusion_seeded_paradigm.html The diffusion seeded paradigm] | ||
+ | * [http://www.nmr-relax.com/manual/Model_free_analysis.html Link to the manual] | ||
+ | * [http://www.nmr-relax.com/manual/The_model_free_models.html Summary of model-free models] | ||
+ | * [http://www.nmr-relax.com/manual/molmol_macro_apply.html#SECTION081284600000000000000 Summary of parameter meaning and value to pymol visualization] | ||
* d'Auvergne, E. J. and Gooley, P. R. (2008). [http://dx.doi.org/10.1007/s10858-007-9214-2 Optimisation of NMR dynamic models I. Minimisation algorithms and their performance within the model-free and Brownian rotational diffusion spaces. J. Biomol. NMR, 40(2), 107-119.] | * d'Auvergne, E. J. and Gooley, P. R. (2008). [http://dx.doi.org/10.1007/s10858-007-9214-2 Optimisation of NMR dynamic models I. Minimisation algorithms and their performance within the model-free and Brownian rotational diffusion spaces. J. Biomol. NMR, 40(2), 107-119.] | ||
* d'Auvergne, E. J. and Gooley, P. R. (2008). [http://dx.doi.org/10.1007/s10858-007-9213-3 Optimisation of NMR dynamic models II. A new methodology for the dual optimisation of the model-free parameters and the Brownian rotational diffusion tensor. J. Biomol. NMR, 40(2), 121-133.] | * d'Auvergne, E. J. and Gooley, P. R. (2008). [http://dx.doi.org/10.1007/s10858-007-9213-3 Optimisation of NMR dynamic models II. A new methodology for the dual optimisation of the model-free parameters and the Brownian rotational diffusion tensor. J. Biomol. NMR, 40(2), 121-133.] | ||
Line 31: | Line 35: | ||
* [https://github.com/nmr-relax/relax/blob/master/sample_scripts/model_free/modsel.py modsel.py] Script for model-free model selection. | * [https://github.com/nmr-relax/relax/blob/master/sample_scripts/model_free/modsel.py modsel.py] Script for model-free model selection. | ||
* [https://github.com/nmr-relax/relax/blob/master/sample_scripts/model_free/molmol_plot.py molmol_plot.py] Script for generating Molmol macros for highlighting model-free motions | * [https://github.com/nmr-relax/relax/blob/master/sample_scripts/model_free/molmol_plot.py molmol_plot.py] Script for generating Molmol macros for highlighting model-free motions | ||
− | * [https://github.com/nmr-relax/relax/blob/master/sample_scripts/model_free/palmer.py palmer.py] Script for model-free analysis using Art Palmer's program 'Modelfree4' | + | * [https://github.com/nmr-relax/relax/blob/master/sample_scripts/model_free/palmer.py palmer.py] Script for model-free analysis using Art Palmer's program 'Modelfree4'. Download from http://comdnmr.nysbc.org/comd-nmr-dissem/comd-nmr-software |
* [https://github.com/nmr-relax/relax/blob/master/sample_scripts/model_free/remap.py remap.py] Script for mapping the model-free space. | * [https://github.com/nmr-relax/relax/blob/master/sample_scripts/model_free/remap.py remap.py] Script for mapping the model-free space. | ||
* [https://github.com/nmr-relax/relax/blob/master/sample_scripts/model_free/single_model.py single_model.py] This script performs a model-free analysis for the single model 'm4'. | * [https://github.com/nmr-relax/relax/blob/master/sample_scripts/model_free/single_model.py single_model.py] This script performs a model-free analysis for the single model 'm4'. | ||
Line 152: | Line 156: | ||
cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t158" >> deselect.txt | cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t158" >> deselect.txt | ||
cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t159" >> deselect.txt | cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t159" >> deselect.txt | ||
+ | </source> | ||
+ | |} | ||
+ | |||
+ | And a new one, changing the NOE error, and deselecting spins found from consistency test.<br> | ||
+ | {| class="mw-collapsible mw-collapsed wikitable" | ||
+ | ! See commands | ||
+ | |- | ||
+ | | | ||
+ | <source lang="bash"> | ||
+ | mkdir 20171010_model_free_5_HADDOCK | ||
+ | cp 20171010_model_free/*.dat 20171010_model_free_5_HADDOCK | ||
+ | cp 20171010_model_free/*.pdb 20171010_model_free_5_HADDOCK | ||
+ | |||
+ | # Get scripts | ||
+ | cd 20171010_model_free_5_HADDOCK | ||
+ | git init | ||
+ | git remote add origin git@github.com:tlinnet/relax_modelfree_scripts.git | ||
+ | git fetch | ||
+ | git checkout -t origin/master | ||
+ | |||
+ | # Change NOE error | ||
+ | sed -i 's/0.1*$/0.05/' NOE_600MHz_new.dat | ||
+ | sed -i 's/0.1*$/0.05/' NOE_750MHz.dat | ||
+ | |||
+ | # Make deselection | ||
+ | echo "#" > deselect.txt | ||
+ | cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t158" >> deselect.txt | ||
+ | cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t157" >> deselect.txt | ||
+ | cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t17" >> deselect.txt | ||
+ | cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t159" >> deselect.txt | ||
+ | cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t120" >> deselect.txt | ||
+ | cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t59" >> deselect.txt | ||
+ | cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t98" >> deselect.txt | ||
+ | cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t49" >> deselect.txt | ||
+ | cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t76" >> deselect.txt | ||
+ | cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t155" >> deselect.txt | ||
+ | cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t156" >> deselect.txt | ||
+ | cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t48" >> deselect.txt | ||
+ | cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t154" >> deselect.txt | ||
+ | </source> | ||
+ | |} | ||
+ | |||
+ | And a new one, without changing the NOE error, and deselecting spins found from consistency test.<br> | ||
+ | {| class="mw-collapsible mw-collapsed wikitable" | ||
+ | ! See commands | ||
+ | |- | ||
+ | | | ||
+ | <source lang="bash"> | ||
+ | mkdir 20171010_model_free_6_HADDOCK | ||
+ | cp 20171010_model_free/*.dat 20171010_model_free_6_HADDOCK | ||
+ | cp 20171010_model_free/*.pdb 20171010_model_free_6_HADDOCK | ||
+ | |||
+ | # Get scripts | ||
+ | cd 20171010_model_free_6_HADDOCK | ||
+ | git init | ||
+ | git remote add origin git@github.com:tlinnet/relax_modelfree_scripts.git | ||
+ | git fetch | ||
+ | git checkout -t origin/master | ||
+ | |||
+ | # Make deselection | ||
+ | echo "#" > deselect.txt | ||
+ | cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t158" >> deselect.txt | ||
+ | cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t157" >> deselect.txt | ||
+ | cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t17" >> deselect.txt | ||
+ | cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t159" >> deselect.txt | ||
+ | |||
+ | cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t59" >> deselect.txt | ||
+ | cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t98" >> deselect.txt | ||
+ | cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t76" >> deselect.txt | ||
+ | cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t155" >> deselect.txt | ||
+ | cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t156" >> deselect.txt | ||
+ | cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t120" >> deselect.txt | ||
+ | |||
+ | cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t49" >> deselect.txt | ||
+ | cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t48" >> deselect.txt | ||
+ | cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t154" >> deselect.txt | ||
+ | |||
+ | cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t33" >> deselect.txt | ||
+ | cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t67" >> deselect.txt | ||
</source> | </source> | ||
|} | |} | ||
Line 177: | Line 260: | ||
=== relax 11_test_consistency.py - Consistency test of our data === | === relax 11_test_consistency.py - Consistency test of our data === | ||
Before running the analysis, it is wise to run a script for [[Tutorial_for_model_free_SBiNLab#Other_script_inspiration_for_checking|consistency testing]]. | Before running the analysis, it is wise to run a script for [[Tutorial_for_model_free_SBiNLab#Other_script_inspiration_for_checking|consistency testing]]. | ||
+ | |||
+ | See here: | ||
+ | * Morin & Gagne (2009a) [http://dx.doi.org/10.1007/s10858-009-9381-4 Simple tests for the validation of multiple field spin relaxation data. J. Biomol. NMR, 45: 361-372.] | ||
+ | |||
+ | Highlights: | ||
+ | * Comparing results obtained at different magnetic fields should, in the case of perfect consistency and assuming the absence of conformational exchange, yield equal values independently of the magnetic field. | ||
+ | * avoid the potential extraction of erroneous information as well as the waste of time associated to dissecting inconsistent datasets using numerous long model-free minimisations with different subsets of data. | ||
+ | * The authors prefer the use of the spectral density at zero frequency J(0) alone since it '''does not rely''' on an estimation of the global correlation time '''tc/tm''', neither on a measure of theta, the angle between the 15N–1H vector and the principal axis of the 15N chemical shift tensor. Hence, J(0) is less likely to be affected by incorrect parameterisation of input parameters. | ||
See content of: | See content of: | ||
[https://github.com/tlinnet/relax_modelfree_scripts/blob/master/11_test_consistency.py 11_test_consistency.py] | [https://github.com/tlinnet/relax_modelfree_scripts/blob/master/11_test_consistency.py 11_test_consistency.py] | ||
− | |||
− | |||
<source lang="bash"> | <source lang="bash"> | ||
relax 11_test_consistency.py -t 11_test_consistency.py.log | relax 11_test_consistency.py -t 11_test_consistency.py.log | ||
Line 218: | Line 307: | ||
# or | # or | ||
− | tmux new -s | + | tmux new -s m1 |
− | mpirun -np | + | mpirun -np 22 relax --multi='mpi4py' 12_Model_1_I_local_tm.py -t 12_Model_1_I_local_tm.log |
</source> | </source> | ||
Line 244: | Line 333: | ||
relax 13_Model_2_II_sphere.py -t 13_Model_2_II_sphere.log | relax 13_Model_2_II_sphere.py -t 13_Model_2_II_sphere.log | ||
# Or | # Or | ||
− | mpirun -np | + | mpirun -np 5 relax --multi='mpi4py' 13_Model_2_II_sphere.py -t 13_Model_2_II_sphere.log |
# When relax is running, push: Ctrl+b and then d, to disconnect without exit | # When relax is running, push: Ctrl+b and then d, to disconnect without exit | ||
Line 254: | Line 343: | ||
relax 13_Model_3_III_prolate.py -t 13_Model_3_III_prolate.log | relax 13_Model_3_III_prolate.py -t 13_Model_3_III_prolate.log | ||
# Or | # Or | ||
− | mpirun -np | + | mpirun -np 5 relax --multi='mpi4py' 13_Model_3_III_prolate.py -t 13_Model_3_III_prolate.log |
</source> | </source> | ||
Line 262: | Line 351: | ||
relax 13_Model_4_IV_oblate.py -t 13_Model_4_IV_oblate.log | relax 13_Model_4_IV_oblate.py -t 13_Model_4_IV_oblate.log | ||
# Or | # Or | ||
− | mpirun -np | + | mpirun -np 5 relax --multi='mpi4py' 13_Model_4_IV_oblate.py -t 13_Model_4_IV_oblate.log |
</source> | </source> | ||
Line 270: | Line 359: | ||
relax 13_Model_5_V_ellipsoid.py -t 13_Model_5_V_ellipsoid.log | relax 13_Model_5_V_ellipsoid.py -t 13_Model_5_V_ellipsoid.log | ||
# Or | # Or | ||
− | mpirun -np | + | mpirun -np 5 relax --multi='mpi4py' 13_Model_5_V_ellipsoid.py -t 13_Model_5_V_ellipsoid.log |
</source> | </source> | ||
Line 292: | Line 381: | ||
The script will ask for input of MC numbers. So just run it. | The script will ask for input of MC numbers. So just run it. | ||
+ | [https://github.com/tlinnet/relax_modelfree_scripts/blob/master/14_intermediate_final.py 14_intermediate_final.py] | ||
<source lang="bash"> | <source lang="bash"> | ||
tmux new -s final | tmux new -s final | ||
− | relax 14_intermediate_final.py -t 14_intermediate_final. | + | relax 14_intermediate_final.py -t 14_intermediate_final.log |
</source> | </source> | ||
This does: | This does: | ||
* Option: Collect current best result from Model 2-5, and make MC simulations, and finalize to get current results files | * Option: Collect current best result from Model 2-5, and make MC simulations, and finalize to get current results files | ||
− | ** Make analysis script for palmer Modelfree4 | + | ** [http://comdnmr.nysbc.org/comd-nmr-dissem/comd-nmr-software Make analysis script for palmer Modelfree4] |
** Get more spin information | ** Get more spin information | ||
* Make a pymol file, that collects all of relax pymol command files into 1 pymol session | * Make a pymol file, that collects all of relax pymol command files into 1 pymol session | ||
Line 305: | Line 395: | ||
** Make a python plot file for plotting this results | ** Make a python plot file for plotting this results | ||
+ | === Per iteration get: chi2, k, tm === | ||
Afterwards, plot the data. | Afterwards, plot the data. | ||
<source lang="bash"> | <source lang="bash"> | ||
python results_collected.py | python results_collected.py | ||
+ | </source> | ||
+ | |||
+ | === Pymol macro === | ||
+ | You also get a pymol folder. | ||
+ | |||
+ | See here for info how the macro is applied | ||
+ | * [http://www.nmr-relax.com/manual/molmol_macro_apply.html#SECTION081284600000000000000 Summary of parameter meaning and value to pymol visualization] | ||
+ | |||
+ | Run with | ||
+ | <source lang="bash"> | ||
+ | pymol 0_0_apply_all_pymol_commands.pml | ||
</source> | </source> | ||
Latest revision as of 10:40, 22 October 2017
Contents
- 1 Background
- 2 Script inspiration
- 3 Scripts - Part 2
- 4 To run on Haddock
- 5 Useful commands to log file
- 6 rsync files
- 7 About the protocol
- 8 See also
Background
This is a tutorial for Lau and Kaare in SBiNLab, and hopefully others.
To get inspiration of example scripts files and see how the protocol is performed, have a look here:
For references, see relax references:
- See this description of the protocol by Edward and image The diffusion seeded paradigm
- Link to the manual
- Summary of model-free models
- Summary of parameter meaning and value to pymol visualization
- d'Auvergne, E. J. and Gooley, P. R. (2008). Optimisation of NMR dynamic models I. Minimisation algorithms and their performance within the model-free and Brownian rotational diffusion spaces. J. Biomol. NMR, 40(2), 107-119.
- d'Auvergne, E. J. and Gooley, P. R. (2008). Optimisation of NMR dynamic models II. A new methodology for the dual optimisation of the model-free parameters and the Brownian rotational diffusion tensor. J. Biomol. NMR, 40(2), 121-133.
Script inspiration
model-free : Script inspiration for setup and analysis
The distribution of relax includes a folder sample_scripts/model_free which contain a folder with scripts for analysis.
It can be seen here: https://github.com/nmr-relax/relax/tree/master/sample_scripts/model_free
Here is the current list
- back_calculate.py. Back-calculate and save relaxation data starting from a saved model-free results file.
- bmrb_deposition.py Script for creating a NMR-STAR 3.1 formatted file for BMRB deposition of model-free results.
- cv.py Script for model-free analysis using cross-validation model selection.
- dasha.py Script for model-free analysis using the program Dasha.
- dauvergne_protocol.py Script for black-box model-free analysis.
- diff_min.py Demonstration script for diffusion tensor optimisation in a model-free analysis.]
- final_data_extraction.py Extract Data to Table
- generate_ri.py Script for back-calculating the relaxation data.
- grace_S2_vs_te.py Script for creating a grace plot of the simulated order parameters vs. simulated correlation times.
- grace_ri_data_correlation.py Script for creating correlations plots of experimental verses back calculated relaxation data.
- map.py Script for mapping the model-free space for OpenDX visualisation.
- mf_multimodel.py This script performs a model-free analysis for the models 'm0' to 'm9' (or 'tm0' to 'tm9').
- modsel.py Script for model-free model selection.
- molmol_plot.py Script for generating Molmol macros for highlighting model-free motions
- palmer.py Script for model-free analysis using Art Palmer's program 'Modelfree4'. Download from http://comdnmr.nysbc.org/comd-nmr-dissem/comd-nmr-software
- remap.py Script for mapping the model-free space.
- single_model.py This script performs a model-free analysis for the single model 'm4'.
- table_csv.py Script for converting the model-free results into a CSV table.
- table_latex.py Script for converting the model-free results into a LaTeX table.
Other script inspiration for checking
The distribution of relax includes a folder sample_scripts/ which contain a folder with scripts for analysis.
It can be seen here: https://github.com/nmr-relax/relax/tree/master/sample_scripts
R1 / R2 Calculation
- relax_fit.py Script for relaxation curve fitting.
- relax_curve_diff.py Script for creating a Grace plot of peak intensity differences.
The resultant plot is useful for finding bad points or bad spectra when fitting exponential curves determine the R1 and R2 relaxation rates. If the averages deviate systematically from zero, bias in the spectra or fitting will be clearly revealed. To use this script, R1 or R2 exponential curve fitting must have previously have been carried out the program state saved to the file 'rx.save' (either with or without the .gz or .bz2 ). The file name of the saved state can be changed at the top of this script.
NOE calculation
- noe.py Script for calculating NOEs.
Test data
- jw_mapping.py Script for reduced spectral density mapping.
- consistency_tests.py Script for consistency testing.
Severe artifacts can be introduced if model-free analysis is performed from inconsistent multiple magnetic field datasets. The use of simple tests as validation tools for the consistency assessment can help avoid such problems in order to extract more reliable information from spin relaxation experiments. In particular, these tests are useful for detecting inconsistencies arising from R2 data. Since such inconsistencies can yield artifactual Rex parameters within model-free analysis, these tests should be use routinely prior to any analysis such as model-free calculations. This script will allow one to calculate values for the three consistency tests J(0), F_eta and F_R2. Once this is done, qualitative analysis can be performed by comparing values obtained at different magnetic fields. Correlation plots and histograms are useful tools for such comparison, such as presented in Morin & Gagne (2009a) J. Biomol. NMR, 45: 361-372.
Other representations
- angles.py Script for calculating the protein NH bond vector angles with respect to the diffusion tensor.
- xh_vector_dist.py Script for creating a PDB representation of the distribution of XH bond vectors.
- diff_tensor_pdb.py Script for creating a PDB representation of the Brownian rotational diffusion tensor.
Scripts - Part 2
We now try to setup things a little more efficient.
Relax is able to read previous results file, so let us divide the task up into:
- 1: Load the data and save as state file. Inspect in GUI before running.
- 2: Run the Model 1: local_tm.
- 3: Here make 4 scripts. Each of them only depends on Model 1:
- Model 2: sphere
- Model 3: prolate
- Model 4: oblate
- Model 5: ellipsoid
- 4: Make an intermediate 'final' model script. This will automatically detect files from above.
Prepare data
We make a new folder and try.
See commands |
---|
mkdir 20171010_model_free_2_HADDOCK
cp 20171010_model_free/*.dat 20171010_model_free_2_HADDOCK
cp 20171010_model_free/*.pdb 20171010_model_free_2_HADDOCK
# Get scripts
cd 20171010_model_free_2_HADDOCK
git init
git remote add origin git@github.com:tlinnet/relax_modelfree_scripts.git
git fetch
git checkout -t origin/master
|
And a new one, changing the NOE error
See commands |
---|
mkdir 20171010_model_free_3_HADDOCK
cp 20171010_model_free/*.dat 20171010_model_free_3_HADDOCK
cp 20171010_model_free/*.pdb 20171010_model_free_3_HADDOCK
# Get scripts
cd 20171010_model_free_3_HADDOCK
git init
git remote add origin git@github.com:tlinnet/relax_modelfree_scripts.git
git fetch
git checkout -t origin/master
# Change NOE error
sed -i 's/0.1*$/0.05/' NOE_600MHz_new.dat
sed -i 's/0.1*$/0.05/' NOE_750MHz.dat
|
And a new one, changing the NOE error, and deselecting N-terminal.
Consistency test, found that this stretch contained outliers.
See commands |
---|
mkdir 20171010_model_free_4_HADDOCK
cp 20171010_model_free/*.dat 20171010_model_free_4_HADDOCK
cp 20171010_model_free/*.pdb 20171010_model_free_4_HADDOCK
# Get scripts
cd 20171010_model_free_4_HADDOCK
git init
git remote add origin git@github.com:tlinnet/relax_modelfree_scripts.git
git fetch
git checkout -t origin/master
# Change NOE error
sed -i 's/0.1*$/0.05/' NOE_600MHz_new.dat
sed -i 's/0.1*$/0.05/' NOE_750MHz.dat
# Make deselection
echo "#" > deselect.txt
cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t151" >> deselect.txt
cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t152" >> deselect.txt
cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t153" >> deselect.txt
cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t154" >> deselect.txt
cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t155" >> deselect.txt
cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t156" >> deselect.txt
cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t157" >> deselect.txt
cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t158" >> deselect.txt
cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t159" >> deselect.txt
|
And a new one, changing the NOE error, and deselecting spins found from consistency test.
See commands |
---|
mkdir 20171010_model_free_5_HADDOCK
cp 20171010_model_free/*.dat 20171010_model_free_5_HADDOCK
cp 20171010_model_free/*.pdb 20171010_model_free_5_HADDOCK
# Get scripts
cd 20171010_model_free_5_HADDOCK
git init
git remote add origin git@github.com:tlinnet/relax_modelfree_scripts.git
git fetch
git checkout -t origin/master
# Change NOE error
sed -i 's/0.1*$/0.05/' NOE_600MHz_new.dat
sed -i 's/0.1*$/0.05/' NOE_750MHz.dat
# Make deselection
echo "#" > deselect.txt
cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t158" >> deselect.txt
cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t157" >> deselect.txt
cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t17" >> deselect.txt
cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t159" >> deselect.txt
cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t120" >> deselect.txt
cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t59" >> deselect.txt
cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t98" >> deselect.txt
cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t49" >> deselect.txt
cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t76" >> deselect.txt
cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t155" >> deselect.txt
cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t156" >> deselect.txt
cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t48" >> deselect.txt
cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t154" >> deselect.txt
|
And a new one, without changing the NOE error, and deselecting spins found from consistency test.
See commands |
---|
mkdir 20171010_model_free_6_HADDOCK
cp 20171010_model_free/*.dat 20171010_model_free_6_HADDOCK
cp 20171010_model_free/*.pdb 20171010_model_free_6_HADDOCK
# Get scripts
cd 20171010_model_free_6_HADDOCK
git init
git remote add origin git@github.com:tlinnet/relax_modelfree_scripts.git
git fetch
git checkout -t origin/master
# Make deselection
echo "#" > deselect.txt
cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t158" >> deselect.txt
cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t157" >> deselect.txt
cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t17" >> deselect.txt
cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t159" >> deselect.txt
cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t59" >> deselect.txt
cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t98" >> deselect.txt
cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t76" >> deselect.txt
cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t155" >> deselect.txt
cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t156" >> deselect.txt
cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t120" >> deselect.txt
cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t49" >> deselect.txt
cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t48" >> deselect.txt
cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t154" >> deselect.txt
cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t33" >> deselect.txt
cat R1_600MHz_new_model_free.dat | grep -P "ArcCALD\t67" >> deselect.txt
|
11_read_data_GUI_inspect.py - Read data GUI inspect
This will read the data and save as a state.
The GUI can be a good place to inspect the setup and files.
See content of: 11_read_data_GUI_inspect.py
Run with
relax 11_read_data_GUI_inspect.py -t 11_read_data_GUI_inspect.log
To check in GUI
- relax -g
- File -> Open relax state
- In folder "result_10" open "result_10_ini.bz2"
- View -> Data pipe editor
- Right click on pipe, and select "Associate with a new auto-analysis"
relax 11_test_consistency.py - Consistency test of our data
Before running the analysis, it is wise to run a script for consistency testing.
See here:
- Morin & Gagne (2009a) Simple tests for the validation of multiple field spin relaxation data. J. Biomol. NMR, 45: 361-372.
Highlights:
- Comparing results obtained at different magnetic fields should, in the case of perfect consistency and assuming the absence of conformational exchange, yield equal values independently of the magnetic field.
- avoid the potential extraction of erroneous information as well as the waste of time associated to dissecting inconsistent datasets using numerous long model-free minimisations with different subsets of data.
- The authors prefer the use of the spectral density at zero frequency J(0) alone since it does not rely on an estimation of the global correlation time tc/tm, neither on a measure of theta, the angle between the 15N–1H vector and the principal axis of the 15N chemical shift tensor. Hence, J(0) is less likely to be affected by incorrect parameterisation of input parameters.
See content of: 11_test_consistency.py
relax 11_test_consistency.py -t 11_test_consistency.py.log
#Afterwards, go into the folder at plot data.
python plot_txt_files.py
./grace2images.py
12_Model_1_I_local_tm.py - Only run local_tm
Now we only run Model 1.
- DIFF_MODEL = ['local_tm']
- GRID_INC = 11 # This is the standard
- MC_NUM = 0 # This has no influence in Model 1-5
- MAX_ITER = 20 # Stop if it has not converged in 20 rounds
Normally between 8 to 15 multiple rounds of optimisation of the are required for the proper execution of this script.
This is can also be see here in Figure 2.
- d'Auvergne, E. J. and Gooley, P. R. (2008). Optimisation of NMR dynamic models II. A new methodology for the dual optimisation of the model-free parameters and the Brownian rotational diffusion tensor. J. Biomol. NMR, 40(2), 121-133.
Relax should stop calculation, if a model does not converge.
See content of: 12_Model_1_I_local_tm.py
We use tmux to make a terminal-session, we can get back to, if our own terminal connection get closed.
Run with
# Make terminal-session
tmux new -s m1
relax 12_Model_1_I_local_tm.py -t 12_Model_1_I_local_tm.log
# or
tmux new -s m1
mpirun -np 22 relax --multi='mpi4py' 12_Model_1_I_local_tm.py -t 12_Model_1_I_local_tm.log
You can then in another terminal follow the logfile by
less +F 12_Model_I_local_tm.log
- To scroll up and down, use keyboard: Ctrl+c
- To return to follow mode, use keyboard: Shift+f
- To exit, use keyboard: Ctrl+c and then: q
13_Model_2-5 - Run Model 2 to 5
When Model 1 is completed, then make 4 terminal windows and run them at the same time.
These scripts do:
- Read the state file from before with setup
- Change DIFF_MODEL accordingly
tmux new -s m2
relax 13_Model_2_II_sphere.py -t 13_Model_2_II_sphere.log
# Or
mpirun -np 5 relax --multi='mpi4py' 13_Model_2_II_sphere.py -t 13_Model_2_II_sphere.log
# When relax is running, push: Ctrl+b and then d, to disconnect without exit
tmux new -s m3
relax 13_Model_3_III_prolate.py -t 13_Model_3_III_prolate.log
# Or
mpirun -np 5 relax --multi='mpi4py' 13_Model_3_III_prolate.py -t 13_Model_3_III_prolate.log
tmux new -s m4
relax 13_Model_4_IV_oblate.py -t 13_Model_4_IV_oblate.log
# Or
mpirun -np 5 relax --multi='mpi4py' 13_Model_4_IV_oblate.py -t 13_Model_4_IV_oblate.log
tmux new -s m5
relax 13_Model_5_V_ellipsoid.py -t 13_Model_5_V_ellipsoid.log
# Or
mpirun -np 5 relax --multi='mpi4py' 13_Model_5_V_ellipsoid.py -t 13_Model_5_V_ellipsoid.log
To join session
# List
tmux list-s
# Join either
tmux a -t m1
tmux a -t m2
tmux a -t m3
tmux a -t m4
tmux a -t m5
14_intermediate_final.py - Inspection during model optimization
During running of model 2-5, the current results can be inspected with this nifty scripts.
The script will ask for input of MC numbers. So just run it.
tmux new -s final
relax 14_intermediate_final.py -t 14_intermediate_final.log
This does:
- Option: Collect current best result from Model 2-5, and make MC simulations, and finalize to get current results files
- Make analysis script for palmer Modelfree4
- Get more spin information
- Make a pymol file, that collects all of relax pymol command files into 1 pymol session
- Option: Collect all chi2 and number of params k, for each iteration per model
- Make a python plot file for plotting this results
Per iteration get: chi2, k, tm
Afterwards, plot the data.
python results_collected.py
Pymol macro
You also get a pymol folder.
See here for info how the macro is applied
Run with
pymol 0_0_apply_all_pymol_commands.pml
To run on Haddock
Have a look here, how to get standalone python Anaconda linux. Also have a look here OpenMPI.
# SSH in
ssh haddock
# Test with shell
mpirun -np 6 echo "hello world"
# Test with python
mpirun -np 6 python -m mpi4py helloworld
# Test with relax
mpirun -np 6 relax --multi='mpi4py'
# Look for: Processor fabric: MPI 2.2 running via mpi4py with 5 slave processors & 1 master. Using MPICH2 1.4.1.
Now we run 04_run_default_with_tolerance_lim.py with more power!
We use tmux to make a terminal-session, we can get back to,
if our own terminal connection get closed.
- start a new session: tmux
- re-attach a detached session: tmux attach
# Make terminal-session
tmux
# Start relax
mpirun -np 20 relax --multi='mpi4py' 04_run_default_with_tolerance_lim.py -t 04_run_default_with_tolerance_lim.log
Useful commands to log file
While the analysis is running, these commands could be used to check the logfile for errors
### Check convergence
# For chi2
cat 04_run_default_with_tolerance_lim.log | grep -A 10 "Chi-squared test:"
# For other tests
cat 04_run_default_with_tolerance_lim.log | grep -A 10 "Identical "
cat 04_run_default_with_tolerance_lim.log | grep -A 10 "Identical model-free models test:"
cat 04_run_default_with_tolerance_lim.log | grep -A 10 "Identical diffusion tensor parameter test:"
cat 04_run_default_with_tolerance_lim.log | grep -A 10 "Identical model-free parameter test:"
# To look for not converged errors
# For chi2
cat 04_run_default_with_tolerance_lim.log | grep -B 7 "The chi-squared value has not converged."
# For other tests
cat 04_run_default_with_tolerance_lim.log | grep -B 7 " have not converged."
cat 04_run_default_with_tolerance_lim.log | grep -B 7 "The model-free models have not converged."
cat 04_run_default_with_tolerance_lim.log | grep -B 7 "The diffusion parameters have not converged."
cat 04_run_default_with_tolerance_lim.log | grep -B 7 "The model-free parameters have not converged."
You can then inspect the logfile by less: 10-tips for less
less 04_run_default_with_tolerance_lim.log
To find pattern: We have to escape with \ for special character like: ()[] etc.
# Search forward
/Value \(iter 14\)
/The chi-squared value has not converged
n or N – for next match in forward / previous match in backward
- To return to follow mode, use keyboard: Shift+f
- To exit, use keyboard: Ctrl+c and then: q
rsync files
rsync files after completion to Sauron
When a run is completed, then sync files to Sauron file server.
Make a rsync_to_sbinlab.sh file with content
See file content |
---|
#!/bin/bash
read -p "Username on sauron :" -r
RUSER=$REPLY
SAURON=10.61.4.60
PROJ=`basename "$PWD"`
FROM=${PWD}
TO=${RUSER}@${SAURON}:/data/sbinlab2/${RUSER}/Downloads
# -a: "archive"- archive mode; equals -rlptgoD (no -H,-A,-X). syncs recursively and preserves symbolic links, special and device files, modification times, group, owner, and permissions.
# We want to remove the -o and -g options:
# -o, --owner preserve owner (super-user only)
# -g, --group preserve group
# -rlptD : Instead or
# -a --no-o --no-g
# -z: Compression over network
# -P: It combines the flags --progress and --partial. The first of these gives you a progress bar for the transfers and the second allows you to resume interrupted transfers:
# -h, Output numbers in a more human-readable format.
# Always double-check your arguments before executing an rsync command.
# -n
echo "I will now do a DRY RUN, which does not move files"
read -p "Are you sure? y/n :" -n 1 -r
echo ""
if [[ $REPLY =~ ^[Yy]$ ]]; then
rsync -rlptDPzh -n ${FROM} ${TO}
else
echo "Not doing DRY RUN"
fi
echo ""
echo "I will now do the sync of files"
read -p "Are you sure? y/n :" -n 1 -r
echo ""
if [[ $REPLY =~ ^[Yy]$ ]]; then
rsync -rlptDPzh ${FROM} ${TO}
else
echo "Not doing anything"
fi
Make it executable and run chmod +x rsync_to_sbinlab.sh
#run
./rsync_to_sbinlab2.sh
|
rsync files from BIO to home mac
To inspect from home mac.
Make a rsync_from_bio_to_home.sh file with content
See file content |
---|
#!/bin/bash
read -p "Username on bio:" -r
RUSER=$REPLY
BIO=ssh-bio.science.ku.dk
#PROJ=Desktop/kaare_relax
PROJ=Desktop/kaare_relax/20171010_model_free_HADDOCK
PROJDIR=`basename "$PROJ"`
FROM=${RUSER}@${BIO}:/home/${RUSER}/${PROJ}
TO=${PWD}/${PROJDIR}
# -a: "archive"- archive mode; equals -rlptgoD (no -H,-A,-X). syncs recursively and preserves symbolic links, special and device files, modification times, group, owner, and permissions.
# We want to remove the -o and -g options:
# -o, --owner preserve owner (super-user only)
# -g, --group preserve group
# -rlptD : Instead or
# -a --no-o --no-g
# -z: Compression over network
# -P: It combines the flags --progress and --partial. The first of these gives you a progress bar for the transfers and the second allows you to resume interrupted transfers:
# -h, Output numbers in a more human-readable format.
# Always double-check your arguments before executing an rsync command.
# -n
echo "I will now do a DRY RUN, which does not move files"
read -p "Are you sure? y/n :" -n 1 -r
echo ""
if [[ $REPLY =~ ^[Yy]$ ]]; then
rsync -rlptDPzh -n ${FROM} ${TO}
else
echo "Not doing DRY RUN"
fi
echo ""
echo "I will now do the sync of files"
read -p "Are you sure? y/n :" -n 1 -r
echo ""
if [[ $REPLY =~ ^[Yy]$ ]]; then
rsync -rlptDPzh ${FROM} ${TO}
else
echo "Not doing anything"
fi
Make it executable and run chmod +x rsync_from_bio_to_home.sh
#run
./rsync_from_bio_to_home.sh
|
About the protocol
Model I - 'local_tm'
This will optimise the diffusion model whereby all spin of the molecule have a local tm value, i.e. there is no global diffusion tensor. This model needs to be optimised prior to optimising any of the other diffusion models. Each spin is fitted to the multiple model-free models separately, where the parameter tm is included in each model.
Model II - 'sphere'
This will optimise the isotropic diffusion model. Multiple steps are required, an initial optimisation of the diffusion tensor, followed by a repetitive optimisation until convergence of the diffusion tensor. In the relax script UI each of these steps requires this script to be rerun, unless the conv_loop flag is True. In the GUI (graphical user interface), the procedure is repeated automatically until convergence. For the initial optimisation, which will be placed in the directory './sphere/init/', the following steps are used:
- The model-free models and parameter values for each spin are set to those of diffusion model MI.
- The local tm parameter is removed from the models.
- The model-free parameters are fixed and a global spherical diffusion tensor is minimised
- For the repetitive optimisation, each minimisation is named from 'round_1' onwards. The initial 'round_1' optimisation will extract the diffusion tensor from the results file in './sphere/init/', and the results will be placed in the directory './sphere/round_1/'. Each successive round will take the diffusion tensor from the previous round. The following steps are used:
- The global diffusion tensor is fixed and the multiple model-free models are fitted to each spin.
- AIC model selection is used to select the models for each spin.
- All model-free and diffusion parameters are allowed to vary and a global optimisation of all parameters is carried out.
Model III - 'prolate'
The methods used are identical to those of diffusion model MII, except that an axially symmetric diffusion tensor with Da >= 0 is used. The base directory containing all the results is './prolate/'.
Model IV -'oblate'
The methods used are identical to those of diffusion model MII, except that an axially symmetric diffusion tensor with Da <= 0 is used. The base directory containing all the results is './oblate/'.
Model V - 'ellipsoid'
The methods used are identical to those of diffusion model MII, except that a fully anisotropic diffusion tensor is used (also known as rhombic or asymmetric diffusion). The base directory is './ellipsoid/'
'final'
Once all the diffusion models have converged, the final run can be executed. This is done by setting the variable diff_model to 'final'. This consists of two steps, diffusion tensor model selection, and Monte Carlo simulations. Firstly AIC model selection is used to select between the diffusion tensor models. Monte Carlo simulations are then run solely on this selected diffusion model. Minimisation of the model is bypassed as it is assumed that the model is already fully optimised (if this is not the case the final run is not yet appropriate).
The final black-box model-free results will be placed in the file 'final/results'.