Difference between revisions of "Run relax at Google Cloud Computing"

Revision as of 18:12, 28 November 2015

Background

This follows the same idea as from Run relax at cloud.sagemath.com

Make a rapid deployment of relax on a google computer cluster, and calculate for 2 months with 8 CPU's, for free.

Pricing

For this example of Google Cloud Computing, we wan't to have access to as many computer cores as possible

The pricing explained here: https://cloud.google.com/pricing/
A pricing calculator is here: https://cloud.google.com/products/calculator/

At November 2015, google give free trial of 300$. https://cloud.google.com/free-trial/

Note, there are some limitations:

For this time, the most important limitation is 8 CPUs.

8 CPU's

For the price calculation of:

1 server
using Free: Debian, CentOS, ...
with a regular VM Class
Instance type of (HIGH CPU) n1-highcpu-8 (vCPUs: 8, RAM: 7.20 GB)
0 SSD
Datacenter location United States
Average days per month each server is running: 1 days, per month.

This 1 day price was of November 2015, $8.06. Constantly running is $171.70 per month.

This essentially gives you 2 months of free computation with 8 CPU's.

Start free trial

Please first see this video:

https://cloud.google.com/compute/docs/linux-quickstart or https://www.youtube.com/watch?v=BnEhYaUY4sA

Go to: https://cloud.google.com/free-trial/ and click Start your free trial.

You need to sign up with a credit card. To prevent mis-abuse from robots and hackers.

Then go to: https://console.developers.google.com

You should already have a project called My First Project.
Create a new project in top right corner of the browser under the dropdown of My First Project.
Call the new project name relax, Project ID relax-1, App Engine location at us-central. NOTE: App engine HAS TO BE located in united states, if datalab should be enabled later.

Create a Virtual Machine instance

Click then at Try Compute Engine or go to https://console.developers.google.com/compute
Click Create instance
Name: relax
Zone: Don't change this. As of Nov 2015, it seems that if you change this, you cannot select more than 2 vCPU's
Machine type: From the very bottom, create 8 vCPUs. 32 cores need an upgrade to you account, and the free trial only allows for 8 vCPU's.
Boot disk: 10 GB of Ubuntu 14.04 LTS
Zone: Now change this to: us-central1-f
Let the rest be default and click Create.

Access Virtual Machine instance

Go to https://console.developers.google.com/compute
In the line of relax, click SSH
This should open a browser terminal window

Get relax and unpack

v=5.1.0

# Get relax
curl http://download.gna.org/relax/relax-$v.GNU-Linux.x86_64.tar.bz2 -o relax-$v.GNU-Linux.x86_64.tar.bz2

# Make home bin
mkdir -p $HOME/bin
tar xvjf relax-5.1.0.GNU-Linux.x86_64.tar.bz2
rm relax-5.1.0.GNU-Linux.x86_64.tar.bz2
ln -s $HOME/relax-5.1.0/relax $HOME/bin

# Add $HOME/bin to PATH
echo '' >> $HOME/.bashrc
echo 'export PATH=$PATH:$HOME/bin' >> $HOME/.bashrc
source $HOME/.bashrc

Solve depencies. See also Installation linux

sudo apt-get -y install python-numpy

Then test relax

relax -i

Then install other packages Solve depencies. See also Installation linux

sudo apt-get -y install python-scipy python-matplotlib python-pip
sudo apt-get -y install openmpi-bin openmpi-doc libopenmpi-dev
sudo pip install mpi4py
sudo pip install epydoc

# Check installation
relax -i

Inspect server

Install these very good server packages Solve depencies. See alsp Installation linux

sudo apt-get -y install htop

Then check server

uptime
whoami
lscpu
htop -u `whoami`

Then check mpirun

mpirun --version
mpirun --report-bindings -np 4 echo "hello world"
mpirun --np 8 relax --multi='mpi4py'

Install trunk of relax

Solve dependencies.

See alsp Installation linux

sudo apt-get -y install subversion scons grace

Checkout relax and build.

svn co svn://svn.gna.org/svn/relax/trunk relax_trunk

# Build
cd relax_trunk
scons
 
# Link to executable
cd $HOME
mkdir -p $HOME/bin
ln -s $HOME/relax_trunk/relax $HOME/bin/relax_trunk
 
# See relax info
relax_trunk -i

install Minfx

See Minfx

v=1.0.12

mkdir -p $HOME/Downloads
cd $HOME/Downloads
curl http://download.gna.org/minfx/minfx-$v.tar.gz -o minfx-$v.tar.gz
tar -xzf minfx-$v.tar.gz
cd minfx-$v
sudo pip install .
cd $HOME

# Test relax
relax_trunk -i

install Bmrblib

See Bmrblib

v=1.0.4

mkdir -p $HOME/Downloads
cd $HOME/Downloads
curl http://download.gna.org/bmrblib/bmrblib-$v.tar.gz -o bmrblib-$v.tar.gz
tar -xzf bmrblib-$v.tar.gz
cd bmrblib-$v
sudo pip install .
cd $HOME

# Test relax
relax_trunk -i

Check installation of relax

relax_trunk --time -x

Transfer files back and forth to the server

Se

https://cloud.google.com/compute/docs/tutorials/transfer-files

Install gcloud

See https://cloud.google.com/sdk/

On your own computer

cd $HOME/Downloads
curl https://sdk.cloud.google.com | bash
source $HOME/.bash_profile
gcloud init

This will start a browser to authenticate you. When it ask for a ID, just write "test" and say no to cloning.

Then go to:

https://console.developers.google.com/compute
In the line of relax, click SSH, and drop down to: View gcloud command.

It could look like this. Try this in your own terminal. This should make some SSH keys and bring you to the server

gcloud compute --project "relax-1" ssh --zone "us-central1-f" "relax"

This should bring you to the google server.

lscpu
cat $HOME/.ssh/authorized_keys 
exit

# At your own computer
ls -l $HOME/.ssh/google*
cat $HOME/.ssh/config
ssh-add -l
ssh-add $HOME/.ssh/google_compute_engine
ssh-add -l

Copy files

See: https://cloud.google.com/sdk/gcloud/reference/compute/copy-files

The ssh command was

bash
PROJ=relax-1
ZONE=us-central1-f
PRID=relax
gcloud compute --project $PROJ ssh --zone $ZONE $PRID

We make a test file and copy it

touch test.txt
gcloud compute --project $PROJ copy-files test.txt $PRID:~ --zone $ZONE

# Try a directory
mkdir testdir
cp test.txt testdir
gcloud compute --project $PROJ copy-files testdir $PRID:~ --zone $ZONE

# Back again
cd testdir
gcloud compute --project $PROJ copy-files $PRID:~/testdir . --zone $ZONE

ssh-config with gcloud

See https://cloud.google.com/sdk/gcloud/reference/compute/config-ssh

gcloud compute config-ssh makes SSHing to virtual machine instances easier by adding an alias for each instance to the user SSH configuration (~/.ssh/config) file.

gcloud compute --project $PROJ config-ssh
cat $HOME/.ssh/config

# ssh in
ssh $PRID.$ZONE.$PROJ
exit

# Use sed to change value of your local computes ssh config file
grep "$PRID.$ZONE.$PROJ" $HOME/.ssh/config
sed "s/$PRID.$ZONE.$PROJ/gcc_relax/g" $HOME/.ssh/config
sed -i.bak "s/$PRID.$ZONE.$PROJ/gcc_relax/g" $HOME/.ssh/config
ls -l $HOME/.ssh/config*
grep 'gcc_relax' $HOME/.ssh/config
ssh gcc_relax
exit

Use scp and rsync when gcloud has been ssh-config configured

#First fast check
touch test2.txt
scp test2.txt  gcc_relax:
ssh gcc_relax
ls
exit

# Or use rsync
mkdir dirtest
rsync --progress -avzhe ssh ./dirtest gcc_relax:
ssh gcc_relax
ls
rmdir dirtest
exit

STOP the google instance after test to stop the billing of computation

See: https://cloud.google.com/compute/docs/instances/stopping-or-deleting-an-instance

By browser

Go to:

https://console.developers.google.com/compute
Tick the relax project and click the "Stop" button.

By terminal

# Get at list of projects
gcloud compute --project $PROJ instances list

# Start it, it takes 30 seconds.
gcloud compute --project $PROJ instances start --zone $ZONE $PRID
ssh gcc_relax
exit

# Stop it through terminal
gcloud compute --project $PROJ instances stop --zone $ZONE $PRID
# This should now not work
ssh gcc_relax

Upgrade account

To use 32 CPU's you need to upgrade your account

See

Deploy google datalab to your server

THIS IS NOT WORKING

Visit:

Do the following

Launch datalab
Sign in
Start your server:

gcloud compute --project "relax-1143" instances start --zone "europe-west1-c" "relax"
ssh gcc_relax

Deploy to your project

Note: Cloud Datalab uses AppEngine and Managed VMs beta, which is available in the US region only.

Move the project to another zone

gcloud compute --project "relax-1143" instances start --zone "europe-west1-c" "relax"
gcloud compute --project "relax-1143" instances move "relax" --zone "europe-west1-c" --destination-zone "us-central1-f"

Reconfigure ssh config

gcloud compute config-ssh --remove
cat $HOME/.ssh/config

gcloud compute --project "relax-1143" config-ssh
cat $HOME/.ssh/config
 
# ssh in
ssh relax.us-central1-f.relax-1143
exit
 
# Use sed to change value of your local computes ssh config file
grep 'relax.us-central1-f.relax-1143' $HOME/.ssh/config
sed 's/relax.us-central1-f.relax-1143/gcc_relax/g' $HOME/.ssh/config
sed -i.bak 's/relax.us-central1-f.relax-1143/gcc_relax/g' $HOME/.ssh/config
ls -l $HOME/.ssh/config*
grep 'gcc_relax' $HOME/.ssh/config
ssh gcc_relax

Deploy again

Cloud Datalab is deployed as an Google App Engine application module in a Google Developers Console project. Click here to Enable the Google Compute Engine API.
https://console.developers.google.com/flows/enableapi?apiid=compute_component&redirect=https://console.developers.google.com/&_ga=1.83746902.1584957856.1448496848

The appengine should be changed to US
https://appengine.google.com/
https://cloud.google.com/appengine/docs/developers-console/#server-location

You cannot change the location after the project has been created.'