General Usage of Our Beowulf Cluster

The Key F1 raises the window under the mouse. The Key F2 lowers the window under the mouse. The Key F3 iconizes the window under the mouse. The Key F4 kills the window under the mouse.

Linux as all Unix systems is famous for the powerful command line utilities it offers. Refer to the following articles if you have little knowledge about Linux , LinuxPlusShell and Command-line interface.

Please go through Online Introduction to Unix to acquire the minimum skills on Unix.

Then go through Linux Shell Scripting Tutorial.

Refer to the openSUSE 10.3 Reference for the introduction to the important Linux commands and secure network operations.

Reading entire Chapters 19 and 36 is strongly advisable:

19: Working with the Shell	pp. 271-296
36: SSH-Secure Network Operations	pp. 589-596

The most important reference Sections are:

19.3: Important Linux Commands	pp. 280-290
36.2: The ssh Program	p. 590
36.3: spc-Secure Copy	p. 590

Refer to the Wikipedia article below for a list of frequently used Unix commands.

Bash is a command-line interpreter provided with the majority of Linux distributions. It allows you to execute the command-line utilities and programs:

http://en.wikipedia.org/wiki/Bash

http://www.linuxconfig.org/Bash_scripting_Tutorial

http://bash-hackers.org/wiki/doku.php

http://wooledge.org:8000/BashPitfalls

If you face the bash message "command not found" your shell did not recognise the command you executed. It might occur because of the incorrect spelling of command name, e.g. "gfrtran" instead of "gfortran". In the other case the executed command might not be present in your system or you do not have sufficient user rights to run it.

X11 Forwarding

The following IP addresses are available in our cluster. They are all 64 bits machines. The IP address inside the bracket "()" is the old IP address before we moved to the Engineering Building. So when you see the sentence which involves the old IP addresses please obtain the new IP address using the list below.

File copy from Hokusai to our cluster

In this section, we assume fileB is your tar file which you want to send from Riken. If you do not know tar, then please read through Handling TAR-files


1. Login to 10.109.21.47 using ssh -XC
2. In the x-terminal where you logged into 10.109.21.47, you run xterm -fa monospace-18 & to create your second x-window. The x-window where you logged into 10.109.21.47 is called the first x-window in this section.
3. In the first x-window, login to Riken and run
   cd /home/fumie/data; ln -s /home/fumie/data/yourusername/yourappropriatedirectory/fileB . in Hokusai
4. In the second x-window, run sudo /usr/local/bin/scpfromriken fileB and run cd /rikendata
5. In the second x-window, you will find /rikendata/fileB and run
   scp -P 22111 -C -r /rikendata/fileB yourusername@your.local.machine.IP:/your/local/directory/. Here, never use mv . After the data transfer, in this second x-window, run rm /rikendata/fileB
6. In the first x-window, run rm /home/fumie/data/fileB
7. In the first x-window, remove all the data you produced in Riken immediately and then exit .

Run the serial jobs in Hokusai
1. Login to Riken
2. Run pjshowrsc in order to check the resources availability and decide which cluster you want to use based on your needs and the load of each cluster in Hokusai
3. Go to your directory by cd /home/fumie/data/yourowndirectory/appropriatedirectory
4. Decide which cluster you use: If your job is less than 96GB of memory then please use MPC.
5. Compile your program at the frontend with the compiler options appropriate for your needs as follows
   • On ACSL (Application Computing Server with Large Memory )
   1. module load intel
   2. Produce a Makefile which performs the compilation of ifort filename.f90 Your Fortran code should not have write(100,*)a . Instead you should use format the data output.
      Run man ifort to identify the compiler options and add them to your Makefile
      and make clean; make
   • On MPC
   1. module load sparc
   2. Produce a Makefile which performs the compilation of frtpx filename.f90
      Run man frtpx to identify the compiler options and add them to your Makefile
      and make clean; make
6. Produce a job submission script as follows and tweak the parameters and options based on your needs
   • On ASCL
   cat g00-username.sh
#!/bin/sh
#------ pjsub option --------#
#PJM -L rscunit=gwacsl
#PJM -L rscgrp=batch
#PJM -L vnode=1
#PJM -L vnode-core=1
#PJM -L vnode-mem=30Gi
#PJM -L elapse=23:59:00
#PJM -j

export OMP_NUM_THREADS=1
time ./a.out -Wl,-T

   • On MPC
   cat g00-username.sh
#!/bin/sh
#------ pjsub option --------#
#PJM -L rscunit=bwmpc
#PJM -L rscgrp=batch
#PJM -L vnode=([core=1][core-mem=100Mi])
#PJM -L elapse=23:59:00
#PJM -j

export OMP_NUM_THREADS=1
time ./a.out -Wl,-T

7. Run your job at the backend by submitting the job via pjsub g00-username.sh for example. Please make sure that the job script name has your name by replacing username with yours; otherwise the jobs are killed by Fumie.
8. Check your job status via pjstat and if needed, kill the job via pjdel jobID.

Run the OpenMP jobs in Hokusai
1. Login to Riken
2. Run pjshowrsc in order to check the resources availability and decide which cluster you want to use based on your needs and the load of each cluster in Hokusai
3. Go to your directory by cd /home/fumie/data/yourowndirectory/appropriatedirectory
4. Decide which cluster you use: If your job is less than 32GB of memory then please use MPC.
5. Compile your program at the frontend with the compiler options appropriate for your needs as follows
   • On ASCL
   1. module load intel
   2. Produce a Makefile which performs the compilation of ifort -parallel -qopenmp filename.f90 Your Fortran code should not have write(100,*)a . Instead you should use format the data output.
      Run man ifort to identify the compiler options and add them to your Makefile
      and make clean; make
   • On MPC
   1. module load intel
   2. Produce a Makefile which performs the compilation of ifort -parallel -qopenmp filename.f90
      Run man ifort to identify the compiler options and add them to your Makefile
      and make clean; make
6. Produce a job submission script as follows and tweak the parameters and options based on your needs. The parameter core-mem ,presented in the following examples of the job submission script, can be changed to node-mem. When you use node-mem, you have to set the memory size to the total memory size required by your job.
   • On ASCL
   cat g00-username.sh
#!/bin/sh
#------ pjsub option --------#
#PJM -L rscunit=gwacsl
#PJM -L rscgrp=batch
#PJM -L vnode=1
#PJM -L vnode-core=32
#PJM -L core-mem=5Gi
#PJM -L elapse=23:59:00
#PJM -j

export OMP_NUM_THREADS=32
time ./a.out -Wl,-T
   • On MPC
   cat g00-username.sh
#!/bin/sh
#------ pjsub option --------#
#PJM -L rscunit=bwmpc
#PJM -L rscgrp=batch
#PJM -L vnode=1
#PJM -L vnode-core=11
#PJM -L core-mem=900Mi
#PJM -L elapse=23:59:00
#PJM -j

export OMP_NUM_THREADS=8
time ./a.out -Wl,-T

7. Run your job at the backend by submitting the job via pjsub g00-username.sh for example. Please make sure that the job script name has your name by replacing username with yours; otherwise the jobs are killed by Fumie.
8. Check your job status via pjstat and if needed, kill the job via pjdel jobID.

Run the MPI jobs in Hokusai
1. Login to Riken
2. Run pjshowrsc in order to check the resources availability and decide which cluster you want to use based on your needs and the load of each cluster in Hokusai
3. Go to your directory by cd /home/fumie/data/yourowndirectory/appropriatedirectory
4. Compile your program at the frontend with the compiler options appropriate for your needs as follows
   • On MPC
   1. run module load sparc
   2. Produce a Makefile which performs the compilation of mpifrtpx -Kquickdbg filename.f90
      Run man mpifrtpx to identify the compiler options and add them to your Makefile
      and make clean; make
5. Produce a job submission script as follows and tweak the parameters and options based on your needs.
   • On MPC using many nodes and many cores per node
   cat g00-username.sh
#!/bin/sh
#------ pjsub option --------#
#PJM -L rscunit=bwmpc
#PJM -L rscgrp=batch
#PJM -L vnode=(core=10)
#PJM --mpi proc=10
#PJM -L elapse=20:00:00
#PJM -j
#---------------Program execuation---------#
mpirun ./a.out -wl,-T

6. On MPC using many nodes and one core per node
cat g00-username.sh
#!/bin/sh
#------ pjsub option --------#
#PJM -L rscunit=bwmpc
#PJM -L rscgrp=batch
#PJM -L node=10
#PJM --mpi proc=10
#PJM -L elapse=20:00:00
#PJM -j
#---------------Program execuation---------#
mpirun ./a.out -wl,-T

7. Run your job at the backend by submitting the job via pjsub g00-username.sh for example. Please make sure that the job script name has your name by replacing username with yours; otherwise the jobs are killed by Fumie.
8. Check your job status via pjstat and if needed, kill the job via pjdel jobID.

Run the Python with tensorflow or PyTorch jobs in Hokusai
1. Login to Riken
2. Run pjshowrsc in order to check the resources availability and decide which cluster you want to use based on your needs and the load of each cluster in Hokusai
3. Go to your directory by cd /home/fumie/data/yourowndirectory/appropriatedirectory
4. IF NEEDED, install necessary software/libraries such as
   wget https://www.python.org/ftp/python/3.6.4/Python-3.6.4.tgz
tar zxf Python-3.6.4.tgz
cd Python-3.6.4
mkdir build; cd build
../configure --prefix=$HOME/local/python3 --with-tcltk-includes="-I/$HOME/local/python3/include" --with-tcltk-libs="-L/$HOME/local/python3/lib -ltcl8.6 -ltk8.6"
make ; make install
export PATH=$HOME/local/python3/bin:$PATH
export LD_LIBRARY_PATH=$HOME/local/python3/lib:$LD_LIBRARY_PATH
pip3 install numpy
pip3 install http://download.pytorch.org/whl/cu90/torch-0.4.0-cp36-cp36m-linux_x86_64.whl
pip3 install torchvision
pjsub --interact -L rscunit=bwmpc
export PATH=$HOME/local/python3/bin:$PATH
export LD_LIBRARY_PATH=$HOME/local/python3/lib:$LD_LIBRARY_PATH
cd $HOME/data/chao/download/
pip3 install ./nibabel-2.2.1-py2.py3-none-any.whl
export LD_LIBRARY_PATH=$HOME/local/python3/lib:$LD_LIBRARY_PATH
wget https://developer.nvidia.com/compute/cuda/9.0/Prod/local_installers/cuda_9.0.176_384.81_linux-run
sh cuda_9.0.176_384.81_linux-run
export PATH=$HOME/local/cuda-9.0/bin:$PATH
tar zxf cudnn-9.0-linux-x64-v7.1.tgz
cp ./cuda/include/cudnn.h /home/fumie/local/cuda-9.0/include/
cp ./cuda/lib64/libcudnn* /home/fumie/local/cuda-9.0/lib64/
chmod a+r /home/fumie/local/cuda-9.0/include/cudnn.h
chmod a+r /home/fumie/local/cuda-9.0/lib64/libcudnn*
export LD_LIBRARY_PATH=$HOME/local/cuda-9.0/lib64:$LD_LIBRARY_PATH
pip3 install tensorflow-gpu
module unload python
unset PYTHONPATH
module load anaconda
conda create --name py2 python=2
source activate py2
conda install keras==2.1.3

5. Test your code from frontend by running jobs at the backend by running
   pjsub --interact -L rscunit=gwacsg,rscgrp=interact_gpu -x gpu_per_vnode=1
export PATH=$HOME/local/cuda-9.0/bin:$HOME/local/python3/bin:$PATH
export LD_LIBRARY_PATH=$HOME/local/cuda-9.0/lib64:$HOME/local/python3/lib:$LD_LIBRARY_PATH
python3
import torch
torch.cuda.is_available()
from tensorflow.python.client import device_lib as _device_lib
for x in _device_lib.list_local_devices():

6. Produce a job submission script as follows and tweak the parameters and options based on your needs.
   • On MPC for CPU version using many nodes and many cores per node
   cat g00-username.sh
#!/bin/sh
#------ pjsub option --------#
#PJM -L rscunit=bwmpc
#PJM -L rscgrp=batch
#PJM -L vnode-core=1
#PJM -L vnode-mem=100Mi
#PJM -L elapse=20:00:00
#PJM -j
#---------------Program execuation---------#
export OMP_NUM_THREADS=1
export PATH=$HOME/local/python3/bin:$PATH
export LD_LIBRARY_PATH=$HOME/local/python3/lib:$LD_LIBRARY_PATH
python3 ./pytorch_CV.py

   • On ASCL for CPU version using 1 node
   cat g00-username.sh
#!/bin/sh
#------ pjsub option --------#
#PJM -L rscunit=gwacsl
#PJM -L rscgrp=batch
#PJM -L vnode=1
#PJM -L vnode-core=2
#PJM -L vnode-mem=40Gi
#PJM -L elapse=23:59:59
#PJM -j
#---------------Program execuation---------#
export PATH=$HOME/local/python3/bin:$PATH
export LD_LIBRARY_PATH=$HOME/local/python3/lib:$LD_LIBRARY_PATH
python3 ./pytorch_CV.py

   • On ACSG for GPU version using 1 node
   cat g00-username.sh
#!/bin/sh
#----- pjsub option -------#
#PJM -L rscunit=gwacsg
#PJM -L rscgrp=gpu
#PJM -L vnode=1
#PJM -L vnode-core=17
#PJM -L vnode-mem=58800Mi
#PJM -x gpu_per_vnode=1
#PJM -L elapse=23:59:59
#PJM -j
#---------------Program execuation---------#
export PATH=$HOME/local/python3/bin:$HOME/local/cuda-9.0/bin:$PATH
export LD_LIBRARY_PATH=$HOME/local/python3/lib:$HOME/local/cuda-9.0/lib64:$LD_LIBRARY_PATH
python3 ./pytorch_rk.py

7. Run your job at the backend by submitting the job via pjsub g00-username.sh for example. Please make sure that the job script name has your name by replacing username with yours; otherwise the jobs are killed by Fumie.
8. Check your job status via pjstat and if needed, kill the job via pjdel jobID.