Difference between revisions of "Run relax at Google Cloud Computing"
Line 335: | Line 335: | ||
See | See | ||
+ | * https://cloud.google.com/free-trial/docs/upgrade-paid-account | ||
* https://console.developers.google.com/compute/quotas | * https://console.developers.google.com/compute/quotas | ||
* https://cloud.google.com/free-trial/docs/free-trial-quotas | * https://cloud.google.com/free-trial/docs/free-trial-quotas |
Revision as of 16:34, 28 November 2015
Contents
- 1 Background
- 2 Pricing
- 3 Start free trial
- 4 Create a Virtual Machine instance
- 5 Access Virtual Machine instance
- 6 Get relax and unpack
- 7 Inspect server
- 8 Install trunk of relax
- 9 Check installation of relax
- 10 Transfer files back and forth to the server
- 11 STOP the google instance after test to stop the billing of computation
- 12 Upgrade account
- 13 See also
Background
This follows the same idea as from Run relax at cloud.sagemath.com
Make a rapid deployment of relax on a google computer cluster, and calculate for 2 months with 8 CPU's, for free.
Pricing
For this example of Google Cloud Computing, we wan't to have access to as many computer cores as possible
- The pricing explained here: https://cloud.google.com/pricing/
- A pricing calculator is here: https://cloud.google.com/products/calculator/
At November 2015, google give free trial of 300$. https://cloud.google.com/free-trial/
Note, there are some limitations:
- https://cloud.google.com/free-trial/docs/free-trial-quotas
- https://cloud.google.com/free-trial/#/faq
For this time, the most important limitation is 8 CPUs.
8 CPU's
For the price calculation of:
- 1 server
- using Free: Debian, CentOS, ...
- with a regular VM Class
- Instance type of (HIGH CPU) n1-highcpu-8 (vCPUs: 8, RAM: 7.20 GB)
- 0 SSD
- Datacenter location Europe
- Average days per month each server is running: 1 days, per month.
This 1 day price was of November 2015, $8.06. Constantly running is $171.70 per month.
This essentially gives you 2 months of free computation with 8 CPU's.
Start free trial
Please first see this video:
- https://cloud.google.com/compute/docs/linux-quickstart or https://www.youtube.com/watch?v=BnEhYaUY4sA
Go to: https://cloud.google.com/free-trial/ and click Start your free trial.
You need to sign up with a credit card. To prevent mis-abuse from robots and hackers.
Then go to: https://console.developers.google.com
- You should already have a project called My First Project.
- Create a new project in top right corner of the browser under the dropdown of My First Project.
- Call the new project relax with App Engine location at europe-west
Create a Virtual Machine instance
- Click then at Try Compute Engine or go to https://console.developers.google.com/compute
- Click Create instance
- Name: relax
- Zone: Don't change this. As of Nov 2015, it seems that if you change this, you cannot select more than 2 vCPU's
- Machine type: From the very bottom, create 8 vCPUs. 32 cores need an upgrade to you account, and the free trial only allows for 8 vCPU's.
- Boot disk: 10 GB of Ubuntu 14.04 LTS
- Zone: Now change this to: europe-west1-c
- Let the rest be default and click Create.
Access Virtual Machine instance
- Go to https://console.developers.google.com/compute
- In the line of relax, click SSH
- This should open a browser terminal window
Get relax and unpack
v=5.1.0
# Get relax
curl http://download.gna.org/relax/relax-$v.GNU-Linux.x86_64.tar.bz2 -o relax-$v.GNU-Linux.x86_64.tar.bz2
# Make home bin
mkdir -p $HOME/bin
tar xvjf relax-5.1.0.GNU-Linux.x86_64.tar.bz2
rm relax-5.1.0.GNU-Linux.x86_64.tar.bz2
ln -s $HOME/relax-5.1.0/relax $HOME/bin
# Add $HOME/bin to PATH
echo '' >> $HOME/.bashrc
echo 'export PATH=$PATH:$HOME/bin' >> $HOME/.bashrc
source $HOME/.bashrc
Solve depencies. See also Installation linux
sudo apt-get -y install python-numpy
Then test relax
relax -i
Then install other packages Solve depencies. See also Installation linux
sudo apt-get -y install python-scipy python-matplotlib python-pip
sudo apt-get -y install openmpi-bin openmpi-doc libopenmpi-dev
sudo pip install mpi4py
sudo pip install epydoc
# Check installation
relax -i
Inspect server
Install these very good server packages Solve depencies. See alsp Installation linux
sudo apt-get -y install htop
Then check server
uptime
whoami
lscpu
htop -u `whoami`
Then check mpirun
mpirun --version
mpirun --report-bindings -np 4 echo "hello world"
mpirun --np 8 relax --multi='mpi4py'
Install trunk of relax
Solve dependencies.
See alsp Installation linux
sudo apt-get -y install subversion scons grace
Checkout relax and build.
svn co svn://svn.gna.org/svn/relax/trunk relax_trunk
# Build
cd relax_trunk
scons
# Link to executable
cd $HOME
mkdir -p $HOME/bin
ln -s $HOME/relax_trunk/relax $HOME/bin/relax_trunk
# See relax info
relax_trunk -i
install Minfx
See Minfx
v=1.0.12
mkdir -p $HOME/Downloads
cd $HOME/Downloads
curl http://download.gna.org/minfx/minfx-$v.tar.gz -o minfx-$v.tar.gz
tar -xzf minfx-$v.tar.gz
cd minfx-$v
sudo pip install .
cd $HOME
# Test relax
relax_trunk -i
install Bmrblib
See Bmrblib
v=1.0.4
mkdir -p $HOME/Downloads
cd $HOME/Downloads
curl http://download.gna.org/bmrblib/bmrblib-$v.tar.gz -o bmrblib-$v.tar.gz
tar -xzf bmrblib-$v.tar.gz
cd bmrblib-$v
sudo pip install .
cd $HOME
# Test relax
relax_trunk -i
Check installation of relax
relax_trunk --time -x
Transfer files back and forth to the server
Se
Install gcloud
See https://cloud.google.com/sdk/
On your own computer
cd $HOME/Downloads
curl https://sdk.cloud.google.com | bash
source $HOME/.bash_profile
gcloud init
This will start a browser to authenticate you. When it ask for a ID, just write "test" and say no to cloning.
Then go to:
- https://console.developers.google.com/compute
- In the line of relax, click SSH, and drop down to: View gcloud command.
It could look like this. Try this in your own terminal. This should make some SSH keys and bring you to the server
gcloud compute --project "relax-1143" ssh --zone "europe-west1-c" "relax"
This should bring you to the google server.
lscpu
cat $HOME/.ssh/authorized_keys
exit
# At your own computer
ls -l $HOME/.ssh/google*
cat $HOME/.ssh/config
ssh-add -l
ssh-add $HOME/.ssh/google_compute_engine
ssh-add -l
Copy files
See: https://cloud.google.com/sdk/gcloud/reference/compute/copy-files
The ssh command was
gcloud compute --project "relax-1143" ssh --zone "europe-west1-c" "relax"
We make a test file and copy it
touch test.txt
gcloud compute --project "relax-1143" copy-files test.txt relax:~ --zone "europe-west1-c"
# Try a directory
mkdir testdir
cp test.txt testdir
gcloud compute --project "relax-1143" copy-files testdir relax:~ --zone "europe-west1-c"
# Back again
cd testdir
gcloud compute --project "relax-1143" copy-files relax:~/testdir . --zone "europe-west1-c"
ssh-config with gcloud
See https://cloud.google.com/sdk/gcloud/reference/compute/config-ssh
gcloud compute config-ssh makes SSHing to virtual machine instances easier by adding an alias for each instance to the user SSH configuration (~/.ssh/config) file.
gcloud compute --project "relax-1143" config-ssh
cat $HOME/.ssh/config
# ssh in
ssh relax.europe-west1-c.relax-1143
exit
# Use sed to change value of your local computes ssh config file
grep 'relax.europe-west1-c.relax-1143' $HOME/.ssh/config
sed 's/relax.europe-west1-c.relax-1143/gcc_relax/g' $HOME/.ssh/config
sed -i.bak 's/relax.europe-west1-c.relax-1143/gcc_relax/g' $HOME/.ssh/config
ls -l $HOME/.ssh/config*
grep 'gcc_relax' $HOME/.ssh/config
ssh gcc_relax
Use scp and rsync when gcloud has been ssh-config configured
#First fast check
touch test2.txt
scp test2.txt gcc_relax:
ssh gcc_relax
ls
exit
# Or use rsync
mkdir dirtest
rsync --progress -avzhe ssh ./dirtest gcc_relax:
ssh gcc_relax
ls
rmdir dirtest
exit
STOP the google instance after test to stop the billing of computation
See: https://cloud.google.com/compute/docs/instances/stopping-or-deleting-an-instance
By browser
Go to:
- https://console.developers.google.com/compute
- Tick the relax project and click the "Stop" button.
By terminal
# Get at list of projects
gcloud compute --project "relax-1143" instances list
# Start it, it takes 30 seconds.
gcloud compute --project "relax-1143" instances start --zone "europe-west1-c" "relax"
ssh gcc_relax
exit
# Stop it through terminal
gcloud compute --project "relax-1143" instances stop --zone "europe-west1-c" "relax"
# This should now not work
ssh gcc_relax
Upgrade account
To use 32 CPU's you need to upgrade your account
See
- https://cloud.google.com/free-trial/docs/upgrade-paid-account
- https://console.developers.google.com/compute/quotas
- https://cloud.google.com/free-trial/docs/free-trial-quotas