Open main menu

Changes

Run relax at Google Cloud Computing

1,014 bytes removed, 12:47, 27 October 2017
Switch to the {{relax source}} and {{relax url}} templates for the code URLs.
The [https://en.wikipedia.org/wiki/Google_Cloud_Platform Google Cloud Platform] is a cloud computing platform that can be used for cheaply running relax on a cluster.
 
== Background ==
This follows the same idea as from [[Run_relax_at_cloud.sagemath.com|Run relax at cloud.sagemath.com]]
== Create a Virtual Machine instance ==
* Click then at '''Try Compute Engine''' or go Go to https://console.developers.google.com/compute* Click '''Create instance'''* '''Name:''' relax* '''Zone:''' Don't change this. As of Nov 2015, it seems that if you change this, you cannot select more than 2 vCPU'shome* '''Machine type:''' From In the very bottomtop right corner, create '''8 vCPUs'''. 32 cores need an upgrade there is a link to you account, [https://cloud.google.com/free-trial/#/faq and the free trial only allows for 8 vCPU's.]* '''Boot disk:Activate Google Cloud Shell''' 10 GB of Ubuntu 14.04 LTS* '''Zone:''' [https://cloud.google.com/compute/docs/zones Now change Clicking this to:] us-central1-f* Let should start a terminal in half the rest be default and click '''Create'''window.
== Access Virtual Machine instance ==Links* Go to [https://consolecloud.developersgoogle.com/cloud-shell/docs/examples See examples here]* [https://cloud.google.com/sdk/gcloud/reference/compute/instances/create instance creation examples here]* In the line of '''relax''', click '''SSH''' * This should open a '''browser terminal window'''[https://cloud.google.com/compute/docs/zones Google zones]
<source lang== Get relax and unpack =="bash"># List all compute instances in the projectgcloud compute instances listgcloud compute zones listgcloud compute machine-types list | grep us-central1-fgcloud compute machine-types list | grep us-central1-f | sort -k3 -ngcloud compute images list
{{#tag:source|Create instancev={{current version gcloud compute instances create relax}}--zone us-central1-f --machine-type n1-highcpu-4 --image ubuntu-14-04 --boot-disk-size 10GB gcloud compute instances list
# Get relaxOrcurl http://download.gna.org/relax/gcloud compute instances create relax-$v.GNU-Linux.x86_64.tar.bz2 zone us-central1-f --machine-type n1-highcpu-4 --image rhel-6-v20160921 --o relaxboot-$v.GNUdisk-Linux.x86_64.tar.bz2size 10GB
# Make home binmkdir -p $HOME</bintar xvjf relax-{{current version relax}}.GNU-Linux.x86_64.tar.bz2rm relax-{{current version relax}}.GNU-Linux.x86_64.tar.bz2ln -s $HOME/relax-{{current version relax}}/relax $HOME/binsource>
# Add $HOME/bin to PATHThen you can SSH into the instanceecho '' <source lang="bash">> $HOME/.bashrcecho 'export PATHPROJ=relax-1ZONE=$PATH:$HOME/bin' >> $HOME/.bashrcus-central1-fsource $HOME/.bashrcPRID=instance-1|langGCC="bash"gcc}}gcloud compute --project $PROJ ssh --zone $ZONE $PRID
Solve depencies. See also [[Installation_linux|Installation linux]]{{#tag:source|sudo aptgcloud compute --project "relax-1" ssh --get zone "us-y install pythoncentral1-numpy|lang=f" "bashinstance-1"}}</source>
Then test relax== Access Virtual Machine instance =={{#tag* Go to https:source|//console.developers.google.com/compute* In the line of '''relax -i|lang="bash"''', click '''SSH''' }}* This should open a '''browser terminal window'''
Then install other == Install packagesSolve depencies. See also [[Installation_linux|Installation linux]]{{#tag:source|sudo apt-get -y install python-scipy python-matplotlib python-pipsudo apt-get -y install openmpi-bin openmpi-doc libopenmpi-devsudo pip install mpi4pysudo pip install epydocat server ==
# Check installationFor '''rapid deployment''' use this command. This will install current version of relax -i|lang="bash"}}and trunk.
See the list of deploy scripts here* {{relax url|path== Inspect server ==devel_scripts/deploy_scripts/}}
Install these very good server packagesSolve depenciesThen download the script that matches the system. See alsp [[Installation_linux|Installation linux]]{{#tag:source|sudo apt-get -y install htop# Select a version script# Redhat Version 6VERS=deploy_google_computing_redhat_6_86_x64.shVERS=deploy_google_computing_redhat_6_86_x64_upgrade_python.sh# Centos Version 6VERS=deploy_google_computing_centos_6_86_x64_upgrade_python.sh# Redhat Version 7|langVERS="bash"deploy_google_computing_redhat_7_86_x64.sh# Ubuntu}}VERS=deploy_google_computing_ubuntu_14-04_86_x64.sh
# Then check serverset download URLURL={{#tag:sourcerelax url|uptimewhoamilscpuhtop -u `whoami`path=devel_scripts/deploy_scripts/$VERS|langview="bash"raw}}
Then check mpirun{{#tag:source|Download and readlmpirun sudo yum --versiony install wgetmpirun --report-bindings -np 4 echo "hello world"mpirun --np 8 relax --multi='mpi4py'wget $URL $VERS|lang="bash"mv $VERS* $VERS}}cat $VERS
==Install trunk of relax==# Load functionssource $VERS
=== Solve dependencies. ===See alsp [[Installation_linux|Installation linux]]{{#tag:source|Do installationsudo apt-get -y install subversion scons graceinstallandcheck|lang="bash"
}}
=== Checkout '''Check installation of relax and build. ==='''
{{#tag:source|
svn co svn://svn.gna.org/svn/relax/trunk mpirun --np 2 relax_trunk--multi='mpi4py' # Buildcd relax_trunkscons # Link to executablecd $HOMEmkdir -p $HOME/binln -s $HOME/relax_trunk/relax $HOME/bin/relax_trunk # See relax inforelax_trunk time -ix
|lang="bash"
}}
==Test openmpi =install Minfx===See [[Minfx]]{{#tag:source|v={{current version minfx}}
mkdir -p $HOME/DownloadsFor BASH shell<br>cd $HOME{{relax source|path=devel_scripts/Downloadsopenmpi_test_install_bash.sh|view=raw|text=See the script here}}curl http{{#tag:source|URL={{relax url|path=devel_scripts//downloadopenmpi_test_install_bash.gna.org/minfx/minfx-$v.tar.gz -o minfx-sh|view=raw}}wget $v.tar.gzURLtar -xzf minfx-$vmv openmpi_test_install_bash.tarsh* openmpi_test_install_bash.gzcd minfx-$vsudo pip install .cd $HOMEsh
# Test relaxSource functionsrelax_trunk -isource openmpi_test_install_bash.shtestopenmpi|lang="bash"
}}
For TCSH shell<br>{{relax source|path=devel_scripts/openmpi_test_install_tcsh.sh|view=raw|text=install Bmrblib===See [[Bmrblib]]the script here}}{{#tag:source|v=alias get 'set noglob; wget {{current version bmrblibrelax url|path=devel_scripts/openmpi_test_install_tcsh.sh|view=raw}}; mv openmpi_test_install_tcsh.sh* openmpi_test_install_tcsh.sh;unset noglob'get
mkdir -p $HOME/Downloadscd $HOME/Downloadscurl http://download.gna.org/bmrblib/bmrblib-$v.tar.gz -o bmrblib-$v.tar.gz# Source functionstar -xzf bmrblib-$v.tarsource openmpi_test_install_tcsh.gzcd bmrblib-$vsudo pip install .cd $HOME # Test relaxrelax_trunk -ish|lang="bash"
}}
==Check installation of relax==
{{#tag:source|
relax_trunk --time -x
|lang="bash"
}}
==Transfer files back and forth to the server==
SeSee here
* https://cloud.google.com/compute/docs/tutorials/transfer-files
=== Install gcloudon your local computer===
See https://cloud.google.com/sdk/
On your own computer
<source lang="pythonbash">
cd $HOME/Downloads
curl https://sdk.cloud.google.com | bash
This will start a browser to authenticate you. When it ask for a ID, just write "test" and say no to cloning.
Then go to:=== ssh-config with gcloud ===* See https://console.developerscloud.google.com/sdk/gcloud/reference/compute* In the line of relax, click SSH, and drop down to: '''View gcloud command'''./config-ssh
It could look like this. Try this in your own terminal. This should make some SSH keys and bring you to the server<source lang="text">gcloud compute config--project "relax-1" ssh --zone "us-central1-f" "relax"</source> This should bring you makes SSHing to virtual machine instances easier by adding an alias for each instance to the google server.<source lang="bash">lscpucat $HOME/.ssh/authorized_keys exit # At your own computerls -l $HOME/.ssh/google*cat $HOMEuser SSH configuration (~/.ssh/configssh-add -lssh-add $HOME/) file.ssh/google_compute_enginessh-add -l</source>
=== Copy files ===
See: https://cloud.google.com/sdk/gcloud/reference/compute/copy-files
 
The ssh command was
<source lang="bash">
bash
PROJ=relax-1
ZONE=us-central1-f
PRID=relax
gcloud compute --project $PROJ ssh --zone $ZONE $PRID</source> We make a test file and copy it<source langGCC="bash">touch test.txtgcloud compute --project $PROJ copy-files test.txt $PRID:~ --zone $ZONE # Try a directorymkdir testdircp test.txt testdirgcloud compute --project $PROJ copy-files testdir $PRID:~ --zone $ZONE # Back againcd testdirgcloud compute --project $PROJ copy-files $PRID:~/testdir . --zone $ZONE</source> === ssh-config with gcloud ===See https://cloud.google.com/sdk/gcloud/reference/compute/config-ssh gcloud compute config-ssh makes SSHing to virtual machine instances easier by adding an alias for each instance to the user SSH configuration (~/.ssh/config) file. <source lang="bash">gcc
gcloud compute --project $PROJ config-ssh
cat $HOME/.ssh/config
 
# ssh in
ssh $PRID.$ZONE.$PROJ
exit
# Use sed to change value of your local computes ssh config file
grep "$PRID.$ZONE.$PROJ" $HOME/.ssh/configsed "s/$PRID.$ZONE.$PROJ/gcc_relax/g" $HOME/.ssh/configsed -i.bak "s/$PRID.$ZONE.$PROJ/gcc_relax$GCC/g" $HOME/.ssh/config
ls -l $HOME/.ssh/config*
grep 'gcc_relax' "$GCC" $HOME/.ssh/configssh gcc_relax$GCCexit
</source>
{{#tag:source|
#First fast check
cd $HOME/Downloads
touch test2.txt
scp test2.txt gcc_relax$GCC:ssh gcc_relax$GCC
ls
rm test2.txt
exit
rm test2.txt
# Or use rsync
mkdir dirtestdeletethisdirrsync --progress -avzhe ssh ./dirtest gcc_relaxdeletethisdir $GCC:ssh gcc_relax$GCC
ls
rmdir dirtestdeletethisdir
exit
rmdir deletethisdir
|lang="bash"
}}
 
== Use relax on a test dataset from a tutorial ==
This follows some example data from the tutorial here [[Tutorial_for_sorting_data_stored_as_numpy_to_on-resonance_R1rho_analysis|Tutorial for sorting data stored as numpy to on resonance R1rho_analysis]]
 
{{relax source|path=test_suite/shared_data/dispersion/Paul_Schanda_2015_Nov|text=Overview of directory}}
 
Copy data to local dir
{{#tag:source|
cd $HOME
cp -r $HOME/relax_trunk/test_suite/shared_data/dispersion/Paul_Schanda_2015_Nov .
cd Paul_Schanda_2015_Nov
|lang="bash"
}}
 
 
{{relax source|path=test_suite/shared_data/dispersion/Paul_Schanda_2015_Nov/1_tutorial_prepare_data.py|text=Prepare data}}
{{#tag:source|
cat 1_tutorial_prepare_data.py
python 1_tutorial_prepare_data.py
ls -latr
|lang="bash"
}}
 
{{relax source|path=test_suite/shared_data/dispersion/Paul_Schanda_2015_Nov/2_tutorial_load_data.py|text=Run analysis}}
{{#tag:source|
cat 2_tutorial_load_data.py
mpirun --np 2 relax_trunk --multi='mpi4py' 2_tutorial_load_data.py
ls -latr
|lang="bash"
}}
 
{{relax source|path=test_suite/shared_data/dispersion/Paul_Schanda_2015_Nov/3_tutorial_clustered.py|text=Run clustered analysis}}
 
{{#tag:source|
cat 3_tutorial_clustered.py
mpirun --np 2 relax_trunk --multi='mpi4py' 3_tutorial_clustered.py
ls -latr
|lang="bash"
}}
 
See results
{{#tag:source|
cd results_cluster/final
ls -latr
cat phi_ex.out
cat kex.out
|lang="bash"
}}
===By terminal===
<source lang="bash">
PROJ=relax-1
ZONE=us-central1-f
PRID=relax
GCC=gcc
 
# Get at list of projects
gcloud compute --project "relax-1143" $PROJ instances list
# Start it, it takes 30 seconds.
gcloud compute --project "relax-1143" $PROJ instances start --zone "europe-west1-c" "relax"$ZONE $PRIDssh gcc_relax$GCC
exit
# Stop it through terminal
gcloud compute --project "relax-1143" $PROJ instances stop --zone "europe-west1-c" "relax"$ZONE $PRID
# This should now not work
ssh gcc_relax$GCC
</source>
==Deploy google datalab to your server==
{{warning|This does not work}}
 
Read more here: http://techcrunch.com/2015/10/13/google-launches-cloud-datalab-an-interactive-tool-for-exploring-and-visualizing-data
 
Note: '''Cloud Datalab uses AppEngine and Managed VMs beta, which is available in the US region only.'''<br>
'''You cannot change the location after the project has been created.'''<br>
See Location at: https://appengine.google.com/
'''THIS IS NOT WORKING'''Stop your server: <source lang="bash">gcloud compute --project $PROJ instances stop --zone $ZONE $PRID</source>
Read more here: httpCloud Datalab is deployed as an Google App Engine application module in a Google Developers Console project. <br>[https://techcrunchconsole.developers.google.com/2015flows/10enableapi?apiid=compute_component&redirect=https:/13/console.developers.google-launches-cloud-datalab-an-interactive-tool-for-exploring-and-visualizing-data.com/&_ga=1.74861778.1584957856.1448496848 Click here to → Enable the Google Compute Engine API.]
Visit:
* Launch datalab
* Sign in
* Start your server: gcloud compute --project "relax-1143" instances start --zone "europe-west1-c" "relax" ssh gcc_relax* Deploy to your project Note: '''Cloud Datalab uses AppEngine and Managed VMs beta, which is available in the US region only.''' ===Move the project to another zone===* See https://cloud.google.com/compute/docs/instances/moving-instance-across-zones* See zones: https://cloud.google.com/compute/docs/zones<source lang="bash">gcloud compute --project "relax-1143" instances start --zone "europe-west1-c" "relax"gcloud compute --project "relax-1143" instances move "relax" --zone "europe-west1-c" --destination-zone "us-central1-f"</source> === Reconfigure ssh config===<source lang="bash">gcloud compute config-ssh --removecat $HOME/.ssh/config gcloud compute --project "relax-1143" config-sshcat $HOME/.ssh/config # ssh inssh relax.us-central1-f.relax-1143exit # Use sed to change value of your local computes ssh config filegrep 'relax.us-central1-f.relax-1143' $HOME/.ssh/configsed 's/relax.us-central1-f.relax-1143/gcc_relax/g' $HOME/.ssh/configsed -i.bak 's/relax.us-central1-f.relax-1143/gcc_relax/g' $HOME/.ssh/configls -l $HOME/.ssh/config*grep 'gcc_relax' $HOME/.ssh/configssh gcc_relax</source>===Deploy again=== Cloud Datalab is deployed as an Google App Engine application module in a Google Developers Console project. Click here to Enable the Google Compute Engine API.<br>https://console.developers.google.com/flows/enableapi?apiid=compute_component&redirect=https://console.developers.google.com/&_ga=1.83746902.1584957856.1448496848 The appengine should be changed to US<br>https://appengine.google.com/ <br>https://cloud.google.com/appengine/docs/developers-console/#server-location '''You cannot change the location after the project has been created.''''
== See also ==
[[Category:Installation]]
Trusted, Bureaucrats
4,223

edits