View source for Run relax at Google Cloud Computing

The [https://en.wikipedia.org/wiki/Google_Cloud_Platform Google Cloud Platform] is a cloud computing platform that can be used for cheaply running relax on a cluster.

== Background ==
This follows the same idea as from [[Run_relax_at_cloud.sagemath.com|Run relax at cloud.sagemath.com]]

Make a rapid deployment of relax on a google computer cluster, and calculate for 2 months with 8 CPU's, '''for free'''.

== Pricing ==
For this example of '''Google Cloud Computing''', we wan't to have access to as many computer cores as possible

* The pricing explained here: https://cloud.google.com/pricing/
* A pricing calculator is here: https://cloud.google.com/products/calculator/

At November 2015, google give free trial of '''300$'''. https://cloud.google.com/free-trial/

Note, there are some limitations:
* https://cloud.google.com/free-trial/docs/free-trial-quotas
* https://cloud.google.com/free-trial/#/faq

For this time, the most important limitation is 8 CPUs.

===8 CPU's===

'''For the price calculation of:'''
* '''1''' server
* using '''Free: Debian, CentOS, ...'''
* with a '''regular''' VM Class
* Instance type of (HIGH CPU) '''n1-highcpu-8    (vCPUs: 8, RAM: 7.20 GB)'''
* 0 '''SSD'''
* Datacenter location '''United States'''
* Average days per month each server is running: '''1''' days, per month. 
This '''1 day''' price was of November 2015, '''$8.06'''. Constantly running is $171.70 per month. <br>

'''This essentially gives you 2 months of free computation with 8 CPU's. '''

== Start free trial ==
Please first see this video: 
* https://cloud.google.com/compute/docs/linux-quickstart or https://www.youtube.com/watch?v=BnEhYaUY4sA

Go to: https://cloud.google.com/free-trial/ and click '''Start your free trial'''.

You need to sign up with a credit card. To prevent mis-abuse from robots and hackers.

'''Then go to: https://console.developers.google.com'''
* You should already have a project called '''My First Project'''. 
* '''Create a new project''' in top right corner of the browser under the dropdown of '''My First Project'''.
* Call the new project name '''relax''', Project ID '''relax-1''', '''App Engine location''' at '''us-central'''. NOTE: App engine HAS TO BE located in united states, if datalab should be enabled later.

== Create a Virtual Machine instance ==
* Go to https://console.developers.google.com/home
* In the top right corner, there is a link to '''Activate Google Cloud Shell'''.
* Clicking this should start a terminal in half the window.

Links
* [https://cloud.google.com/cloud-shell/docs/examples See examples here]
* [https://cloud.google.com/sdk/gcloud/reference/compute/instances/create instance creation examples here]
* [https://cloud.google.com/compute/docs/zones Google zones]

<source lang="bash">
# List all compute instances in the project
gcloud compute instances list
gcloud compute zones list
gcloud compute machine-types list | grep us-central1-f
gcloud compute machine-types list | grep us-central1-f | sort -k3 -n
gcloud compute images list

# Create instance
gcloud compute instances create relax --zone us-central1-f --machine-type n1-highcpu-4 --image ubuntu-14-04 --boot-disk-size 10GB 
gcloud compute instances list

# Or
gcloud compute instances create relax --zone us-central1-f --machine-type n1-highcpu-4 --image rhel-6-v20160921 --boot-disk-size 10GB 

</source>

Then you can SSH into the instance
<source lang="bash">
PROJ=relax-1
ZONE=us-central1-f
PRID=instance-1
GCC=gcc
gcloud compute --project $PROJ ssh --zone $ZONE $PRID

gcloud compute --project "relax-1" ssh --zone "us-central1-f" "instance-1"
</source>

== Access Virtual Machine instance ==
* Go to https://console.developers.google.com/compute
* In the line of '''relax''', click '''SSH''' 
* This should open a '''browser terminal window'''

== Install packages at server ==

For '''rapid deployment''' use this command. This will install current version of relax and trunk. 

See the list of deploy scripts here
* http://svn.gna.org/viewcvs/relax/trunk/devel_scripts/deploy_scripts/

Then download the script that matches the system.
<source lang="bash">
# Select a version script
VERS=deploy_google_computing_redhat_6_86_x64.sh
VERS=deploy_google_computing_redhat_6_86_x64_upgrade_python.sh
VERS=deploy_google_computing_ubuntu_14-04_86_x64.sh

# Then set download URL
URL=http://svn.gna.org/viewcvs/*checkout*/relax/trunk/devel_scripts/deploy_scripts/$VERS

# Download and readl
wget $URL
cat $VERS

# Load functions
source $VERS

# Do installation
installandcheck
</source>


'''Check installation of relax'''
{{#tag:source|
mpirun --np 2 relax_trunk --multi='mpi4py'
relax_trunk --time -x
|lang="bash"
}}

== Test openmpi ==

[http://svn.gna.org/viewcvs/*checkout*/relax/trunk/devel_scripts/openmpi_test_install.sh?content-type=text%2Fplain See the script here]
<source lang="bash">
URL=http://svn.gna.org/viewcvs/*checkout*/relax/trunk/devel_scripts/openmpi_test_install.sh
wget $URL
$SHELL openmpi_test_install.sh
</source>

==Transfer files back and forth to the server==
See here
* https://cloud.google.com/compute/docs/tutorials/transfer-files

=== Install gcloud on your local computer===
See https://cloud.google.com/sdk/

On your own computer
<source lang="bash">
cd $HOME/Downloads
curl https://sdk.cloud.google.com | bash
source $HOME/.bash_profile
gcloud init
</source>

This will start a browser to authenticate you. When it ask for a ID, just write "test" and say no to cloning.

=== ssh-config with gcloud ===
See https://cloud.google.com/sdk/gcloud/reference/compute/config-ssh

gcloud compute config-ssh makes SSHing to virtual machine instances easier by adding an alias for each instance to the user SSH configuration (~/.ssh/config) file.

<source lang="bash">
PROJ=relax-1
ZONE=us-central1-f
PRID=relax
GCC=gcc
gcloud compute --project $PROJ config-ssh
cat $HOME/.ssh/config

# Use sed to change value of your local computes ssh config file
sed -i.bak "s/$PRID.$ZONE.$PROJ/$GCC/g" $HOME/.ssh/config
ls -l $HOME/.ssh/config*
grep "$GCC" $HOME/.ssh/config
ssh $GCC
exit
</source>

=== Use scp and rsync  when gcloud has been ssh-config configured===

{{#tag:source|
#First fast check
cd $HOME/Downloads
touch test2.txt
scp test2.txt $GCC:
ssh $GCC
ls
rm test2.txt
exit
rm test2.txt

# Or use rsync
mkdir deletethisdir
rsync --progress -avzhe ssh ./deletethisdir $GCC:
ssh $GCC
ls
rmdir deletethisdir
exit
rmdir deletethisdir
|lang="bash"
}}

== Use relax on a test dataset from a tutorial ==
This follows some example data from the tutorial here [[Tutorial_for_sorting_data_stored_as_numpy_to_on-resonance_R1rho_analysis|Tutorial for sorting data stored as numpy to on resonance R1rho_analysis]]

[http://svn.gna.org/viewcvs/relax/trunk/test_suite/shared_data/dispersion/Paul_Schanda_2015_Nov Overview of directory]

Copy data to local dir
{{#tag:source|
cd $HOME
cp -r $HOME/relax_trunk/test_suite/shared_data/dispersion/Paul_Schanda_2015_Nov .
cd Paul_Schanda_2015_Nov
|lang="bash"
}}


[http://svn.gna.org/viewcvs/*checkout*/relax/trunk/test_suite/shared_data/dispersion/Paul_Schanda_2015_Nov/1_tutorial_prepare_data.py?content-type=text%2Fplain Prepare data]
{{#tag:source|
cat 1_tutorial_prepare_data.py 
python 1_tutorial_prepare_data.py
ls -latr
|lang="bash"
}}

[http://svn.gna.org/viewcvs/*checkout*/relax/trunk/test_suite/shared_data/dispersion/Paul_Schanda_2015_Nov/2_tutorial_load_data.py?revision=28131&content-type=text%2Fplain Run analysis]
{{#tag:source|
cat 2_tutorial_load_data.py
mpirun --np 8 relax_trunk --multi='mpi4py' 2_tutorial_load_data.py
ls -latr
|lang="bash"
}}

[http://svn.gna.org/viewcvs/*checkout*/relax/trunk/test_suite/shared_data/dispersion/Paul_Schanda_2015_Nov/3_tutorial_clustered.py?content-type=text%2Fplain Run clustered analysis]

{{#tag:source|
cat 3_tutorial_clustered.py 
mpirun --np 8 relax_trunk --multi='mpi4py' 3_tutorial_clustered.py 
ls -latr
|lang="bash"
}}

See results
{{#tag:source|
cd results_cluster/final
ls -latr
cat phi_ex.out
cat kex.out 
|lang="bash"
}}

==STOP the google instance after test to stop the billing of computation==
See: https://cloud.google.com/compute/docs/instances/stopping-or-deleting-an-instance

===By browser===
Go to:
* https://console.developers.google.com/compute
* Tick the '''relax''' project and click the "Stop" button.

===By terminal===
<source lang="bash">
PROJ=relax-1
ZONE=us-central1-f
PRID=relax
GCC=gcc

# Get at list of projects
gcloud compute --project $PROJ instances list

# Start it, it takes 30 seconds.
gcloud compute --project $PROJ instances start --zone $ZONE $PRID
ssh $GCC
exit

# Stop it through terminal
gcloud compute --project $PROJ instances stop --zone $ZONE $PRID
# This should now not work
ssh $GCC
</source>

==Upgrade account==
To use 32 CPU's you need to upgrade your account

See
* https://cloud.google.com/free-trial/docs/upgrade-paid-account
* https://console.developers.google.com/compute/quotas
* https://cloud.google.com/free-trial/docs/free-trial-quotas

==Deploy google datalab to your server==
{{warning|This does not work}}

Read more here: http://techcrunch.com/2015/10/13/google-launches-cloud-datalab-an-interactive-tool-for-exploring-and-visualizing-data

Note: '''Cloud Datalab uses AppEngine and Managed VMs beta, which is available in the US region only.'''<br>
'''You cannot change the location after the project has been created.'''<br>
See Location at: https://appengine.google.com/

Stop your server: 
<source lang="bash">
gcloud compute --project $PROJ instances stop --zone $ZONE $PRID
</source>

Cloud Datalab is deployed as an Google App Engine application module in a Google Developers Console project. <br>
[https://console.developers.google.com/flows/enableapi?apiid=compute_component&redirect=https://console.developers.google.com/&_ga=1.74861778.1584957856.1448496848 Click here to → Enable the Google Compute Engine API.]

Visit: 
* https://cloud.google.com/datalab
* https://cloud.google.com/datalab/getting-started

Do the following
* Launch datalab
* Sign in
* Deploy to your project.

== See also ==
[[Category:Installation]]