Difference between revisions of "Run relax at Google Cloud Computing"

From relax wiki
Jump to navigation Jump to search
Line 62: Line 62:
 
== Install packages at server ==
 
== Install packages at server ==
  
−
For '''rapid deployment''' use this command. This will install current version of relax and trunk.
+
For '''rapid deployment''' use this command. This will install current version of relax and trunk. [http://svn.gna.org/viewcvs/*checkout*/relax/trunk/devel_scripts/deploy_google_computing_ubuntu.sh?content-type=text%2Fplain See it here]
 
<source lang="bash">
 
<source lang="bash">
 
sudo curl http://svn.gna.org/viewcvs/*checkout*/relax/trunk/devel_scripts/deploy_google_computing_ubuntu.sh | bash
 
sudo curl http://svn.gna.org/viewcvs/*checkout*/relax/trunk/devel_scripts/deploy_google_computing_ubuntu.sh | bash

Revision as of 20:52, 28 November 2015

Background

This follows the same idea as from Run relax at cloud.sagemath.com

Make a rapid deployment of relax on a google computer cluster, and calculate for 2 months with 8 CPU's, for free.

Pricing

For this example of Google Cloud Computing, we wan't to have access to as many computer cores as possible

At November 2015, google give free trial of 300$. https://cloud.google.com/free-trial/

Note, there are some limitations:

For this time, the most important limitation is 8 CPUs.

8 CPU's

For the price calculation of:

  • 1 server
  • using Free: Debian, CentOS, ...
  • with a regular VM Class
  • Instance type of (HIGH CPU) n1-highcpu-8 (vCPUs: 8, RAM: 7.20 GB)
  • 0 SSD
  • Datacenter location United States
  • Average days per month each server is running: 1 days, per month.

This 1 day price was of November 2015, $8.06. Constantly running is $171.70 per month.

This essentially gives you 2 months of free computation with 8 CPU's.

Start free trial

Please first see this video:

Go to: https://cloud.google.com/free-trial/ and click Start your free trial.

You need to sign up with a credit card. To prevent mis-abuse from robots and hackers.

Then go to: https://console.developers.google.com

  • You should already have a project called My First Project.
  • Create a new project in top right corner of the browser under the dropdown of My First Project.
  • Call the new project name relax, Project ID relax-1, App Engine location at us-central. NOTE: App engine HAS TO BE located in united states, if datalab should be enabled later.

Create a Virtual Machine instance

Access Virtual Machine instance

Install packages at server

For rapid deployment use this command. This will install current version of relax and trunk. See it here 

sudo curl http://svn.gna.org/viewcvs/*checkout*/relax/trunk/devel_scripts/deploy_google_computing_ubuntu.sh | bash

Inspect server

Install these very good server packages Solve depencies. See alsp Installation linux 

sudo apt-get -y install openmpi-bin openmpi-doc libopenmpi-dev
sudo apt-get -y install htop

Then check server 

uptime
whoami
lscpu
htop -u `whoami`

Then check mpirun 

mpirun --version
mpirun --report-bindings -np 4 echo "hello world"

Get relax and unpack

v=5.1.0

# Get relax
curl http://download.gna.org/relax/relax-$v.GNU-Linux.x86_64.tar.bz2 -o relax-$v.GNU-Linux.x86_64.tar.bz2

# Make home bin
mkdir -p $HOME/bin
tar xvjf relax-5.1.0.GNU-Linux.x86_64.tar.bz2
rm relax-5.1.0.GNU-Linux.x86_64.tar.bz2
ln -s $HOME/relax-5.1.0/relax $HOME/bin

# Add $HOME/bin to PATH
echo '' >> $HOME/.bashrc
echo 'export PATH=$PATH:$HOME/bin' >> $HOME/.bashrc
source $HOME/.bashrc

Solve depencies. See also Installation linux 

sudo apt-get -y install python-numpy
sudo apt-get -y install python-scipy python-matplotlib python-pip
sudo pip install mpi4py
sudo pip install epydoc

# Check installation
relax -i

Install trunk of relax

See alsp Installation linux 

sudo apt-get -y install subversion scons grace

Checkout relax and build 

svn co svn://svn.gna.org/svn/relax/trunk relax_trunk

# Build
cd relax_trunk
scons
 
# Link to executable
cd $HOME
mkdir -p $HOME/bin
ln -s $HOME/relax_trunk/relax $HOME/bin/relax_trunk
 
# See relax info
relax_trunk -i

install Minfx See Minfx 

v=1.0.12

mkdir -p $HOME/Downloads
cd $HOME/Downloads
curl http://download.gna.org/minfx/minfx-$v.tar.gz -o minfx-$v.tar.gz
tar -xzf minfx-$v.tar.gz
cd minfx-$v
sudo pip install .
cd $HOME

# Test relax
relax_trunk -i

install Bmrblib See Bmrblib 

v=1.0.4

mkdir -p $HOME/Downloads
cd $HOME/Downloads
curl http://download.gna.org/bmrblib/bmrblib-$v.tar.gz -o bmrblib-$v.tar.gz
tar -xzf bmrblib-$v.tar.gz
cd bmrblib-$v
sudo pip install .
cd $HOME

# Test relax
relax_trunk -i

Check installation of relax 

mpirun --np 8 relax_trunk --multi='mpi4py'
relax_trunk --time -x

Transfer files back and forth to the server

Se

Install gcloud

See https://cloud.google.com/sdk/

On your own computer 

cd $HOME/Downloads
curl https://sdk.cloud.google.com | bash
source $HOME/.bash_profile
gcloud init

This will start a browser to authenticate you. When it ask for a ID, just write "test" and say no to cloning.

Then go to:

It could look like this. Try this in your own terminal. This should make some SSH keys and bring you to the server 

gcloud compute --project "relax-1" ssh --zone "us-central1-f" "relax"

This should bring you to the google server. 

lscpu
cat $HOME/.ssh/authorized_keys 
exit

# At your own computer
ls -l $HOME/.ssh/google*
cat $HOME/.ssh/config
ssh-add -l
ssh-add $HOME/.ssh/google_compute_engine
ssh-add -l

Copy files

See: https://cloud.google.com/sdk/gcloud/reference/compute/copy-files

The ssh command was 

bash
PROJ=relax-1
ZONE=us-central1-f
PRID=relax
GCC=gcc
gcloud compute --project $PROJ ssh --zone $ZONE $PRID

We make a test file and copy it 

touch test.txt
gcloud compute --project $PROJ copy-files test.txt $PRID:~ --zone $ZONE

# Try a directory
mkdir testdir
cp test.txt testdir
gcloud compute --project $PROJ copy-files testdir $PRID:~ --zone $ZONE

# Back again
cd testdir
gcloud compute --project $PROJ copy-files $PRID:~/testdir . --zone $ZONE

ssh-config with gcloud

See https://cloud.google.com/sdk/gcloud/reference/compute/config-ssh

gcloud compute config-ssh makes SSHing to virtual machine instances easier by adding an alias for each instance to the user SSH configuration (~/.ssh/config) file. 

gcloud compute --project $PROJ config-ssh
cat $HOME/.ssh/config

# ssh in
ssh $PRID.$ZONE.$PROJ
exit

# Use sed to change value of your local computes ssh config file
grep "$PRID.$ZONE.$PROJ" $HOME/.ssh/config
sed "s/$PRID.$ZONE.$PROJ/$GCC/g" $HOME/.ssh/config
sed -i.bak "s/$PRID.$ZONE.$PROJ/$GCC/g" $HOME/.ssh/config
ls -l $HOME/.ssh/config*
grep "$GCC" $HOME/.ssh/config
ssh $GCC
exit

Use scp and rsync when gcloud has been ssh-config configured

#First fast check
touch test2.txt
scp test2.txt  $GCC:
ssh $GCC
ls
exit

# Or use rsync
mkdir dirtest
rsync --progress -avzhe ssh ./dirtest $GCC:
ssh $GCC
ls
rmdir dirtest
exit

STOP the google instance after test to stop the billing of computation

See: https://cloud.google.com/compute/docs/instances/stopping-or-deleting-an-instance

By browser

Go to:

By terminal

# Get at list of projects
gcloud compute --project $PROJ instances list

# Start it, it takes 30 seconds.
gcloud compute --project $PROJ instances start --zone $ZONE $PRID
ssh $GCC
exit

# Stop it through terminal
gcloud compute --project $PROJ instances stop --zone $ZONE $PRID
# This should now not work
ssh $GCC

Upgrade account

To use 32 CPU's you need to upgrade your account

See

Deploy google datalab to your server

Read more here: http://techcrunch.com/2015/10/13/google-launches-cloud-datalab-an-interactive-tool-for-exploring-and-visualizing-data

Note: Cloud Datalab uses AppEngine and Managed VMs beta, which is available in the US region only.
You cannot change the location after the project has been created.
See Location at: https://appengine.google.com/

Start your server: 

gcloud compute --project $PROJ instances start --zone $ZONE $PRID
ssh $GCC

Cloud Datalab is deployed as an Google App Engine application module in a Google Developers Console project. Click here to → Enable the Google Compute Engine API.


Visit:

Do the following

  • Launch datalab
  • Sign in
  • Deploy to your project. This takes 10 min.

See also

Retrieved from "http://wiki.nmr-relax.com/wiki/index.php?title=Run_relax_at_Google_Cloud_Computing&oldid=6570"