Run relax at Google Cloud Computing

The Google Cloud Platform is a cloud computing platform that can be used for cheaply running relax on a cluster.

Background

This follows the same idea as from Run relax at cloud.sagemath.com

Make a rapid deployment of relax on a google computer cluster, and calculate for 2 months with 8 CPU's, for free.

Pricing

For this example of Google Cloud Computing, we wan't to have access to as many computer cores as possible

• The pricing explained here: https://cloud.google.com/pricing/
• A pricing calculator is here: https://cloud.google.com/products/calculator/

At November 2015, google give free trial of 300$. https://cloud.google.com/free-trial/

Note, there are some limitations:

• https://cloud.google.com/free-trial/docs/free-trial-quotas
• https://cloud.google.com/free-trial/#/faq

For this time, the most important limitation is 8 CPUs.

8 CPU's

For the price calculation of:

• 1 server
• using Free: Debian, CentOS, ...
• with a regular VM Class
• Instance type of (HIGH CPU) n1-highcpu-8 (vCPUs: 8, RAM: 7.20 GB)
• 0 SSD
• Datacenter location United States
• Average days per month each server is running: 1 days, per month.

This 1 day price was of November 2015, $8.06. Constantly running is $171.70 per month.

This essentially gives you 2 months of free computation with 8 CPU's.

Start free trial

Please first see this video:

• https://cloud.google.com/compute/docs/linux-quickstart or https://www.youtube.com/watch?v=BnEhYaUY4sA

Go to: https://cloud.google.com/free-trial/ and click Start your free trial.

You need to sign up with a credit card. To prevent mis-abuse from robots and hackers.

Then go to: https://console.developers.google.com

• You should already have a project called My First Project.
• Create a new project in top right corner of the browser under the dropdown of My First Project.
• Call the new project name relax, Project ID relax-1, App Engine location at us-central. NOTE: App engine HAS TO BE located in united states, if datalab should be enabled later.

Create a Virtual Machine instance

• Go to https://console.developers.google.com/home
• In the top right corner, there is a link to Activate Google Cloud Shell.
• Clicking this should start a terminal in half the window.

Links

• See examples here
• instance creation examples here
• Google zones

# List all compute instances in the project
gcloud compute instances list
gcloud compute zones list
gcloud compute machine-types list | grep us-central1-f
gcloud compute machine-types list | grep us-central1-f | sort -k3 -n
gcloud compute images list

# Create instance
gcloud compute instances create relax --zone us-central1-f --machine-type n1-highcpu-4 --image ubuntu-14-04 --boot-disk-size 10GB 
gcloud compute instances list

# Or
gcloud compute instances create relax --zone us-central1-f --machine-type n1-highcpu-4 --image rhel-6-v20160921 --boot-disk-size 10GB

Then you can SSH into the instance

PROJ=relax-1
ZONE=us-central1-f
PRID=instance-1
GCC=gcc
gcloud compute --project $PROJ ssh --zone $ZONE $PRID

gcloud compute --project "relax-1" ssh --zone "us-central1-f" "instance-1"

Access Virtual Machine instance

• Go to https://console.developers.google.com/compute
• In the line of relax, click SSH
• This should open a browser terminal window

Install packages at server

For rapid deployment use this command. This will install current version of relax and trunk.

See the list of deploy scripts here

• https://sourceforge.net/p/nmr-relax/code/ci/master/tree/devel_scripts/deploy_scripts/

Then download the script that matches the system.

# Select a version script
# Redhat Version 6
VERS=deploy_google_computing_redhat_6_86_x64.sh
VERS=deploy_google_computing_redhat_6_86_x64_upgrade_python.sh
# Centos Version 6
VERS=deploy_google_computing_centos_6_86_x64_upgrade_python.sh
# Redhat Version 7
VERS=deploy_google_computing_redhat_7_86_x64.sh
# Ubuntu
VERS=deploy_google_computing_ubuntu_14-04_86_x64.sh

# Then set download URL
URL=https://sourceforge.net/p/nmr-relax/code/ci/master/tree/devel_scripts/deploy_scripts/$VERS?format=raw

# Download and readl
sudo yum -y install wget
wget $URL $VERS
mv $VERS* $VERS
cat $VERS

# Load functions
source $VERS

# Do installation
installandcheck

Check installation of relax

mpirun --np 2 relax_trunk --multi='mpi4py'
relax_trunk --time -x

Test openmpi

For BASH shell
See the script here

URL=https://sourceforge.net/p/nmr-relax/code/ci/master/tree/devel_scripts/openmpi_test_install_bash.sh?format=raw
wget $URL
mv openmpi_test_install_bash.sh* openmpi_test_install_bash.sh

# Source functions
source openmpi_test_install_bash.sh
testopenmpi

For TCSH shell
See the script here

alias get 'set noglob; wget https://sourceforge.net/p/nmr-relax/code/ci/master/tree/devel_scripts/openmpi_test_install_tcsh.sh?format=raw; mv openmpi_test_install_tcsh.sh* openmpi_test_install_tcsh.sh;unset noglob'
get

# Source functions
source openmpi_test_install_tcsh.sh

Transfer files back and forth to the server

See here

• https://cloud.google.com/compute/docs/tutorials/transfer-files

Install gcloud on your local computer

See https://cloud.google.com/sdk/

On your own computer

cd $HOME/Downloads
curl https://sdk.cloud.google.com | bash
source $HOME/.bash_profile
gcloud init

This will start a browser to authenticate you. When it ask for a ID, just write "test" and say no to cloning.

ssh-config with gcloud

See https://cloud.google.com/sdk/gcloud/reference/compute/config-ssh

gcloud compute config-ssh makes SSHing to virtual machine instances easier by adding an alias for each instance to the user SSH configuration (~/.ssh/config) file.

PROJ=relax-1
ZONE=us-central1-f
PRID=relax
GCC=gcc
gcloud compute --project $PROJ config-ssh
cat $HOME/.ssh/config

# Use sed to change value of your local computes ssh config file
sed -i.bak "s/$PRID.$ZONE.$PROJ/$GCC/g" $HOME/.ssh/config
ls -l $HOME/.ssh/config*
grep "$GCC" $HOME/.ssh/config
ssh $GCC
exit

Use scp and rsync when gcloud has been ssh-config configured

#First fast check
cd $HOME/Downloads
touch test2.txt
scp test2.txt $GCC:
ssh $GCC
ls
rm test2.txt
exit
rm test2.txt

# Or use rsync
mkdir deletethisdir
rsync --progress -avzhe ssh ./deletethisdir $GCC:
ssh $GCC
ls
rmdir deletethisdir
exit
rmdir deletethisdir

Use relax on a test dataset from a tutorial

This follows some example data from the tutorial here Tutorial for sorting data stored as numpy to on resonance R1rho_analysis

Overview of directory

Copy data to local dir

cd $HOME
cp -r $HOME/relax_trunk/test_suite/shared_data/dispersion/Paul_Schanda_2015_Nov .
cd Paul_Schanda_2015_Nov

Prepare data

cat 1_tutorial_prepare_data.py 
python 1_tutorial_prepare_data.py
ls -latr

Run analysis

cat 2_tutorial_load_data.py
mpirun --np 2 relax_trunk --multi='mpi4py' 2_tutorial_load_data.py
ls -latr

Run clustered analysis

cat 3_tutorial_clustered.py 
mpirun --np 2 relax_trunk --multi='mpi4py' 3_tutorial_clustered.py 
ls -latr

See results

cd results_cluster/final
ls -latr
cat phi_ex.out
cat kex.out

STOP the google instance after test to stop the billing of computation

See: https://cloud.google.com/compute/docs/instances/stopping-or-deleting-an-instance

By browser

Go to:

• https://console.developers.google.com/compute
• Tick the relax project and click the "Stop" button.

By terminal

PROJ=relax-1
ZONE=us-central1-f
PRID=relax
GCC=gcc

# Get at list of projects
gcloud compute --project $PROJ instances list

# Start it, it takes 30 seconds.
gcloud compute --project $PROJ instances start --zone $ZONE $PRID
ssh $GCC
exit

# Stop it through terminal
gcloud compute --project $PROJ instances stop --zone $ZONE $PRID
# This should now not work
ssh $GCC

Upgrade account

To use 32 CPU's you need to upgrade your account

See

• https://cloud.google.com/free-trial/docs/upgrade-paid-account
• https://console.developers.google.com/compute/quotas
• https://cloud.google.com/free-trial/docs/free-trial-quotas

Deploy google datalab to your server

Warning  This does not work

Read more here: http://techcrunch.com/2015/10/13/google-launches-cloud-datalab-an-interactive-tool-for-exploring-and-visualizing-data

Note: Cloud Datalab uses AppEngine and Managed VMs beta, which is available in the US region only.
You cannot change the location after the project has been created.
See Location at: https://appengine.google.com/

Stop your server:

gcloud compute --project $PROJ instances stop --zone $ZONE $PRID

Cloud Datalab is deployed as an Google App Engine application module in a Google Developers Console project.
Click here to → Enable the Google Compute Engine API.

Visit:

• https://cloud.google.com/datalab
• https://cloud.google.com/datalab/getting-started

Do the following

• Launch datalab
• Sign in
• Deploy to your project.