Run relax at Google Cloud Computing
The Google Cloud Platform is a cloud computing platform that can be used for cheaply running relax on a cluster.
- 1 Background
- 2 Pricing
- 3 Start free trial
- 4 Create a Virtual Machine instance
- 5 Access Virtual Machine instance
- 6 Install packages at server
- 7 Test openmpi
- 8 Transfer files back and forth to the server
- 9 Use relax on a test dataset from a tutorial
- 10 STOP the google instance after test to stop the billing of computation
- 11 Upgrade account
- 12 Deploy google datalab to your server
- 13 See also
This follows the same idea as from Run relax at cloud.sagemath.com
Make a rapid deployment of relax on a google computer cluster, and calculate for 2 months with 8 CPU's, for free.
For this example of Google Cloud Computing, we wan't to have access to as many computer cores as possible
- The pricing explained here: https://cloud.google.com/pricing/
- A pricing calculator is here: https://cloud.google.com/products/calculator/
At November 2015, google give free trial of 300$. https://cloud.google.com/free-trial/
Note, there are some limitations:
For this time, the most important limitation is 8 CPUs.
For the price calculation of:
- 1 server
- using Free: Debian, CentOS, ...
- with a regular VM Class
- Instance type of (HIGH CPU) n1-highcpu-8 (vCPUs: 8, RAM: 7.20 GB)
- 0 SSD
- Datacenter location United States
- Average days per month each server is running: 1 days, per month.
This 1 day price was of November 2015, $8.06. Constantly running is $171.70 per month.
This essentially gives you 2 months of free computation with 8 CPU's.
Start free trial
Please first see this video:
- https://cloud.google.com/compute/docs/linux-quickstart or https://www.youtube.com/watch?v=BnEhYaUY4sA
Go to: https://cloud.google.com/free-trial/ and click Start your free trial.
You need to sign up with a credit card. To prevent mis-abuse from robots and hackers.
Then go to: https://console.developers.google.com
- You should already have a project called My First Project.
- Create a new project in top right corner of the browser under the dropdown of My First Project.
- Call the new project name relax, Project ID relax-1, App Engine location at us-central. NOTE: App engine HAS TO BE located in united states, if datalab should be enabled later.
Create a Virtual Machine instance
- Go to https://console.developers.google.com/home
- In the top right corner, there is a link to Activate Google Cloud Shell.
- Clicking this should start a terminal in half the window.
# List all compute instances in the project gcloud compute instances list gcloud compute zones list gcloud compute machine-types list | grep us-central1-f gcloud compute machine-types list | grep us-central1-f | sort -k3 -n gcloud compute images list # Create instance gcloud compute instances create relax --zone us-central1-f --machine-type n1-highcpu-4 --image ubuntu-14-04 --boot-disk-size 10GB gcloud compute instances list # Or gcloud compute instances create relax --zone us-central1-f --machine-type n1-highcpu-4 --image rhel-6-v20160921 --boot-disk-size 10GB
Then you can SSH into the instance
PROJ=relax-1 ZONE=us-central1-f PRID=instance-1 GCC=gcc gcloud compute --project $PROJ ssh --zone $ZONE $PRID gcloud compute --project "relax-1" ssh --zone "us-central1-f" "instance-1"
Access Virtual Machine instance
- Go to https://console.developers.google.com/compute
- In the line of relax, click SSH
- This should open a browser terminal window
Install packages at server
For rapid deployment use this command. This will install current version of relax and trunk.
See the list of deploy scripts here
Then download the script that matches the system.
# Select a version script # Redhat Version 6 VERS=deploy_google_computing_redhat_6_86_x64.sh VERS=deploy_google_computing_redhat_6_86_x64_upgrade_python.sh # Centos Version 6 VERS=deploy_google_computing_centos_6_86_x64_upgrade_python.sh # Redhat Version 7 VERS=deploy_google_computing_redhat_7_86_x64.sh # Ubuntu VERS=deploy_google_computing_ubuntu_14-04_86_x64.sh # Then set download URL URL=https://sourceforge.net/p/nmr-relax/code/ci/master/tree/devel_scripts/deploy_scripts/$VERS?format=raw # Download and readl sudo yum -y install wget wget $URL $VERS mv $VERS* $VERS cat $VERS # Load functions source $VERS # Do installation installandcheck
Check installation of relax
mpirun --np 2 relax_trunk --multi='mpi4py' relax_trunk --time -x
For BASH shell
See the script here
URL=https://sourceforge.net/p/nmr-relax/code/ci/master/tree/devel_scripts/openmpi_test_install_bash.sh?format=raw wget $URL mv openmpi_test_install_bash.sh* openmpi_test_install_bash.sh # Source functions source openmpi_test_install_bash.sh testopenmpi
For TCSH shell
See the script here
alias get 'set noglob; wget https://sourceforge.net/p/nmr-relax/code/ci/master/tree/devel_scripts/openmpi_test_install_tcsh.sh?format=raw; mv openmpi_test_install_tcsh.sh* openmpi_test_install_tcsh.sh;unset noglob' get # Source functions source openmpi_test_install_tcsh.sh
Transfer files back and forth to the server
Install gcloud on your local computer
On your own computer
cd $HOME/Downloads curl https://sdk.cloud.google.com | bash source $HOME/.bash_profile gcloud init
This will start a browser to authenticate you. When it ask for a ID, just write "test" and say no to cloning.
ssh-config with gcloud
gcloud compute config-ssh makes SSHing to virtual machine instances easier by adding an alias for each instance to the user SSH configuration (~/.ssh/config) file.
PROJ=relax-1 ZONE=us-central1-f PRID=relax GCC=gcc gcloud compute --project $PROJ config-ssh cat $HOME/.ssh/config # Use sed to change value of your local computes ssh config file sed -i.bak "s/$PRID.$ZONE.$PROJ/$GCC/g" $HOME/.ssh/config ls -l $HOME/.ssh/config* grep "$GCC" $HOME/.ssh/config ssh $GCC exit
Use scp and rsync when gcloud has been ssh-config configured
#First fast check cd $HOME/Downloads touch test2.txt scp test2.txt $GCC: ssh $GCC ls rm test2.txt exit rm test2.txt # Or use rsync mkdir deletethisdir rsync --progress -avzhe ssh ./deletethisdir $GCC: ssh $GCC ls rmdir deletethisdir exit rmdir deletethisdir
Use relax on a test dataset from a tutorial
This follows some example data from the tutorial here Tutorial for sorting data stored as numpy to on resonance R1rho_analysis
Copy data to local dir
cd $HOME cp -r $HOME/relax_trunk/test_suite/shared_data/dispersion/Paul_Schanda_2015_Nov . cd Paul_Schanda_2015_Nov
cat 1_tutorial_prepare_data.py python 1_tutorial_prepare_data.py ls -latr
cat 2_tutorial_load_data.py mpirun --np 2 relax_trunk --multi='mpi4py' 2_tutorial_load_data.py ls -latr
cat 3_tutorial_clustered.py mpirun --np 2 relax_trunk --multi='mpi4py' 3_tutorial_clustered.py ls -latr
cd results_cluster/final ls -latr cat phi_ex.out cat kex.out
STOP the google instance after test to stop the billing of computation
- Tick the relax project and click the "Stop" button.
PROJ=relax-1 ZONE=us-central1-f PRID=relax GCC=gcc # Get at list of projects gcloud compute --project $PROJ instances list # Start it, it takes 30 seconds. gcloud compute --project $PROJ instances start --zone $ZONE $PRID ssh $GCC exit # Stop it through terminal gcloud compute --project $PROJ instances stop --zone $ZONE $PRID # This should now not work ssh $GCC
To use 32 CPU's you need to upgrade your account
Deploy google datalab to your server
|Warning This does not work|
Note: Cloud Datalab uses AppEngine and Managed VMs beta, which is available in the US region only.
You cannot change the location after the project has been created.
See Location at: https://appengine.google.com/
Stop your server:
gcloud compute --project $PROJ instances stop --zone $ZONE $PRID
Cloud Datalab is deployed as an Google App Engine application module in a Google Developers Console project.
Click here to → Enable the Google Compute Engine API.
Do the following
- Launch datalab
- Sign in
- Deploy to your project.