Difference between revisions of "Run relax at Google Cloud Computing"

From relax wiki
Jump to navigation Jump to search
(Switch to the {{relax source}} and {{relax url}} templates for the code URLs.)
 
(3 intermediate revisions by one other user not shown)
Line 95: Line 95:
  
 
See the list of deploy scripts here
 
See the list of deploy scripts here
* http://svn.gna.org/viewcvs/relax/trunk/devel_scripts/deploy_scripts/
+
* {{relax url|path=devel_scripts/deploy_scripts/}}
  
 
Then download the script that matches the system.
 
Then download the script that matches the system.
<source lang="bash">
+
{{#tag: source|
 
# Select a version script
 
# Select a version script
 
# Redhat Version 6
 
# Redhat Version 6
 
VERS=deploy_google_computing_redhat_6_86_x64.sh
 
VERS=deploy_google_computing_redhat_6_86_x64.sh
 
VERS=deploy_google_computing_redhat_6_86_x64_upgrade_python.sh
 
VERS=deploy_google_computing_redhat_6_86_x64_upgrade_python.sh
 +
# Centos Version 6
 +
VERS=deploy_google_computing_centos_6_86_x64_upgrade_python.sh
 
# Redhat Version 7
 
# Redhat Version 7
 
VERS=deploy_google_computing_redhat_7_86_x64.sh
 
VERS=deploy_google_computing_redhat_7_86_x64.sh
Line 109: Line 111:
  
 
# Then set download URL
 
# Then set download URL
URL=http://svn.gna.org/viewcvs/*checkout*/relax/trunk/devel_scripts/deploy_scripts/$VERS
+
URL={{relax url|path=devel_scripts/deploy_scripts/$VERS|view=raw}}
  
 
# Download and readl
 
# Download and readl
 
sudo yum -y install wget
 
sudo yum -y install wget
wget $URL
+
wget $URL $VERS
 +
mv $VERS* $VERS
 
cat $VERS
 
cat $VERS
  
Line 121: Line 124:
 
# Do installation
 
# Do installation
 
installandcheck
 
installandcheck
</source>
+
| lang="bash"
 +
}}
  
  
Line 134: Line 138:
  
 
For BASH shell<br>
 
For BASH shell<br>
[http://svn.gna.org/viewcvs/*checkout*/relax/trunk/devel_scripts/openmpi_test_install_bash.sh?content-type=text%2Fplain See the script here]
+
{{relax source|path=devel_scripts/openmpi_test_install_bash.sh|view=raw|text=See the script here}}
<source lang="bash">
+
{{#tag: source|
URL=http://svn.gna.org/viewcvs/*checkout*/relax/trunk/devel_scripts/openmpi_test_install_bash.sh
+
URL={{relax url|path=devel_scripts/openmpi_test_install_bash.sh|view=raw}}
 
wget $URL
 
wget $URL
 +
mv openmpi_test_install_bash.sh* openmpi_test_install_bash.sh
  
 
# Source functions
 
# Source functions
 
source openmpi_test_install_bash.sh
 
source openmpi_test_install_bash.sh
 
testopenmpi
 
testopenmpi
</source>
+
| lang="bash"
 +
}}
  
 
For TCSH shell<br>
 
For TCSH shell<br>
[http://svn.gna.org/viewcvs/*checkout*/relax/trunk/devel_scripts/openmpi_test_install_tcsh.sh?content-type=text%2Fplain See the script here]
+
{{relax source|path=devel_scripts/openmpi_test_install_tcsh.sh|view=raw|text=See the script here}}
<source lang="bash">
+
{{#tag: source|
set URL=http://svn.gna.org/viewcvs/*checkout*/relax/trunk/devel_scripts/openmpi_test_install_tcsh.sh
+
alias get 'set noglob; wget {{relax url|path=devel_scripts/openmpi_test_install_tcsh.sh|view=raw}}; mv openmpi_test_install_tcsh.sh* openmpi_test_install_tcsh.sh;unset noglob'
wget $URL
+
get
  
 
# Source functions
 
# Source functions
 
source openmpi_test_install_tcsh.sh
 
source openmpi_test_install_tcsh.sh
testopenmpi
+
| lang="bash"
</source>
+
}}
 +
 
  
 
==Transfer files back and forth to the server==
 
==Transfer files back and forth to the server==
Line 220: Line 227:
 
This follows some example data from the tutorial here [[Tutorial_for_sorting_data_stored_as_numpy_to_on-resonance_R1rho_analysis|Tutorial for sorting data stored as numpy to on resonance R1rho_analysis]]
 
This follows some example data from the tutorial here [[Tutorial_for_sorting_data_stored_as_numpy_to_on-resonance_R1rho_analysis|Tutorial for sorting data stored as numpy to on resonance R1rho_analysis]]
  
[http://svn.gna.org/viewcvs/relax/trunk/test_suite/shared_data/dispersion/Paul_Schanda_2015_Nov Overview of directory]
+
{{relax source|path=test_suite/shared_data/dispersion/Paul_Schanda_2015_Nov|text=Overview of directory}}
  
 
Copy data to local dir
 
Copy data to local dir
Line 231: Line 238:
  
  
[http://svn.gna.org/viewcvs/*checkout*/relax/trunk/test_suite/shared_data/dispersion/Paul_Schanda_2015_Nov/1_tutorial_prepare_data.py?content-type=text%2Fplain Prepare data]
+
{{relax source|path=test_suite/shared_data/dispersion/Paul_Schanda_2015_Nov/1_tutorial_prepare_data.py|text=Prepare data}}
 
{{#tag:source|
 
{{#tag:source|
 
cat 1_tutorial_prepare_data.py  
 
cat 1_tutorial_prepare_data.py  
Line 239: Line 246:
 
}}
 
}}
  
[http://svn.gna.org/viewcvs/*checkout*/relax/trunk/test_suite/shared_data/dispersion/Paul_Schanda_2015_Nov/2_tutorial_load_data.py?revision=28131&content-type=text%2Fplain Run analysis]
+
{{relax source|path=test_suite/shared_data/dispersion/Paul_Schanda_2015_Nov/2_tutorial_load_data.py|text=Run analysis}}
 
{{#tag:source|
 
{{#tag:source|
 
cat 2_tutorial_load_data.py
 
cat 2_tutorial_load_data.py
Line 247: Line 254:
 
}}
 
}}
  
[http://svn.gna.org/viewcvs/*checkout*/relax/trunk/test_suite/shared_data/dispersion/Paul_Schanda_2015_Nov/3_tutorial_clustered.py?content-type=text%2Fplain Run clustered analysis]
+
{{relax source|path=test_suite/shared_data/dispersion/Paul_Schanda_2015_Nov/3_tutorial_clustered.py|text=Run clustered analysis}}
  
 
{{#tag:source|
 
{{#tag:source|

Latest revision as of 12:47, 27 October 2017

The Google Cloud Platform is a cloud computing platform that can be used for cheaply running relax on a cluster.

Background

This follows the same idea as from Run relax at cloud.sagemath.com

Make a rapid deployment of relax on a google computer cluster, and calculate for 2 months with 8 CPU's, for free.

Pricing

For this example of Google Cloud Computing, we wan't to have access to as many computer cores as possible

At November 2015, google give free trial of 300$. https://cloud.google.com/free-trial/

Note, there are some limitations:

For this time, the most important limitation is 8 CPUs.

8 CPU's

For the price calculation of:

  • 1 server
  • using Free: Debian, CentOS, ...
  • with a regular VM Class
  • Instance type of (HIGH CPU) n1-highcpu-8 (vCPUs: 8, RAM: 7.20 GB)
  • 0 SSD
  • Datacenter location United States
  • Average days per month each server is running: 1 days, per month.

This 1 day price was of November 2015, $8.06. Constantly running is $171.70 per month.

This essentially gives you 2 months of free computation with 8 CPU's.

Start free trial

Please first see this video:

Go to: https://cloud.google.com/free-trial/ and click Start your free trial.

You need to sign up with a credit card. To prevent mis-abuse from robots and hackers.

Then go to: https://console.developers.google.com

  • You should already have a project called My First Project.
  • Create a new project in top right corner of the browser under the dropdown of My First Project.
  • Call the new project name relax, Project ID relax-1, App Engine location at us-central. NOTE: App engine HAS TO BE located in united states, if datalab should be enabled later.

Create a Virtual Machine instance

Links

# List all compute instances in the project
gcloud compute instances list
gcloud compute zones list
gcloud compute machine-types list | grep us-central1-f
gcloud compute machine-types list | grep us-central1-f | sort -k3 -n
gcloud compute images list

# Create instance
gcloud compute instances create relax --zone us-central1-f --machine-type n1-highcpu-4 --image ubuntu-14-04 --boot-disk-size 10GB 
gcloud compute instances list

# Or
gcloud compute instances create relax --zone us-central1-f --machine-type n1-highcpu-4 --image rhel-6-v20160921 --boot-disk-size 10GB

Then you can SSH into the instance

PROJ=relax-1
ZONE=us-central1-f
PRID=instance-1
GCC=gcc
gcloud compute --project $PROJ ssh --zone $ZONE $PRID

gcloud compute --project "relax-1" ssh --zone "us-central1-f" "instance-1"

Access Virtual Machine instance

Install packages at server

For rapid deployment use this command. This will install current version of relax and trunk.

See the list of deploy scripts here

Then download the script that matches the system.

# Select a version script
# Redhat Version 6
VERS=deploy_google_computing_redhat_6_86_x64.sh
VERS=deploy_google_computing_redhat_6_86_x64_upgrade_python.sh
# Centos Version 6
VERS=deploy_google_computing_centos_6_86_x64_upgrade_python.sh
# Redhat Version 7
VERS=deploy_google_computing_redhat_7_86_x64.sh
# Ubuntu
VERS=deploy_google_computing_ubuntu_14-04_86_x64.sh

# Then set download URL
URL=https://sourceforge.net/p/nmr-relax/code/ci/master/tree/devel_scripts/deploy_scripts/$VERS?format=raw

# Download and readl
sudo yum -y install wget
wget $URL $VERS
mv $VERS* $VERS
cat $VERS

# Load functions
source $VERS

# Do installation
installandcheck


Check installation of relax

mpirun --np 2 relax_trunk --multi='mpi4py'
relax_trunk --time -x

Test openmpi

For BASH shell
See the script here

URL=https://sourceforge.net/p/nmr-relax/code/ci/master/tree/devel_scripts/openmpi_test_install_bash.sh?format=raw
wget $URL
mv openmpi_test_install_bash.sh* openmpi_test_install_bash.sh

# Source functions
source openmpi_test_install_bash.sh
testopenmpi

For TCSH shell
See the script here

alias get 'set noglob; wget https://sourceforge.net/p/nmr-relax/code/ci/master/tree/devel_scripts/openmpi_test_install_tcsh.sh?format=raw; mv openmpi_test_install_tcsh.sh* openmpi_test_install_tcsh.sh;unset noglob'
get

# Source functions
source openmpi_test_install_tcsh.sh


Transfer files back and forth to the server

See here

Install gcloud on your local computer

See https://cloud.google.com/sdk/

On your own computer

cd $HOME/Downloads
curl https://sdk.cloud.google.com | bash
source $HOME/.bash_profile
gcloud init

This will start a browser to authenticate you. When it ask for a ID, just write "test" and say no to cloning.

ssh-config with gcloud

See https://cloud.google.com/sdk/gcloud/reference/compute/config-ssh

gcloud compute config-ssh makes SSHing to virtual machine instances easier by adding an alias for each instance to the user SSH configuration (~/.ssh/config) file.

PROJ=relax-1
ZONE=us-central1-f
PRID=relax
GCC=gcc
gcloud compute --project $PROJ config-ssh
cat $HOME/.ssh/config

# Use sed to change value of your local computes ssh config file
sed -i.bak "s/$PRID.$ZONE.$PROJ/$GCC/g" $HOME/.ssh/config
ls -l $HOME/.ssh/config*
grep "$GCC" $HOME/.ssh/config
ssh $GCC
exit

Use scp and rsync when gcloud has been ssh-config configured

#First fast check
cd $HOME/Downloads
touch test2.txt
scp test2.txt $GCC:
ssh $GCC
ls
rm test2.txt
exit
rm test2.txt

# Or use rsync
mkdir deletethisdir
rsync --progress -avzhe ssh ./deletethisdir $GCC:
ssh $GCC
ls
rmdir deletethisdir
exit
rmdir deletethisdir

Use relax on a test dataset from a tutorial

This follows some example data from the tutorial here Tutorial for sorting data stored as numpy to on resonance R1rho_analysis

Overview of directory

Copy data to local dir

cd $HOME
cp -r $HOME/relax_trunk/test_suite/shared_data/dispersion/Paul_Schanda_2015_Nov .
cd Paul_Schanda_2015_Nov


Prepare data

cat 1_tutorial_prepare_data.py 
python 1_tutorial_prepare_data.py
ls -latr

Run analysis

cat 2_tutorial_load_data.py
mpirun --np 2 relax_trunk --multi='mpi4py' 2_tutorial_load_data.py
ls -latr

Run clustered analysis

cat 3_tutorial_clustered.py 
mpirun --np 2 relax_trunk --multi='mpi4py' 3_tutorial_clustered.py 
ls -latr

See results

cd results_cluster/final
ls -latr
cat phi_ex.out
cat kex.out

STOP the google instance after test to stop the billing of computation

See: https://cloud.google.com/compute/docs/instances/stopping-or-deleting-an-instance

By browser

Go to:

By terminal

PROJ=relax-1
ZONE=us-central1-f
PRID=relax
GCC=gcc

# Get at list of projects
gcloud compute --project $PROJ instances list

# Start it, it takes 30 seconds.
gcloud compute --project $PROJ instances start --zone $ZONE $PRID
ssh $GCC
exit

# Stop it through terminal
gcloud compute --project $PROJ instances stop --zone $ZONE $PRID
# This should now not work
ssh $GCC

Upgrade account

To use 32 CPU's you need to upgrade your account

See

Deploy google datalab to your server

Warning  This does not work

Read more here: http://techcrunch.com/2015/10/13/google-launches-cloud-datalab-an-interactive-tool-for-exploring-and-visualizing-data

Note: Cloud Datalab uses AppEngine and Managed VMs beta, which is available in the US region only.
You cannot change the location after the project has been created.
See Location at: https://appengine.google.com/

Stop your server:

gcloud compute --project $PROJ instances stop --zone $ZONE $PRID

Cloud Datalab is deployed as an Google App Engine application module in a Google Developers Console project.
Click here to → Enable the Google Compute Engine API.

Visit:

Do the following

  • Launch datalab
  • Sign in
  • Deploy to your project.

See also