[wip] Using a Bramble as a Gramble

The Bramble has been setup and running in a previous post.

1- Making sure MPI works fine

In the NFS folder create a mpi_hello.c file with

#include 
#include 

int main(int argc, char** argv) {
 int myrank, nprocs;

 MPI_Init(&argc, &argv);
 MPI_Comm_size(MPI_COMM_WORLD, &nprocs);
 MPI_Comm_rank(MPI_COMM_WORLD, &myrank);

 printf("Hello from processor %d of %d\n", myrank, nprocs);

 MPI_Finalize();
 return 0;
}

and compile it with mpicc

mpicc mpi_hello.c -o mpi_hello

Run with

mpirun -np  -host <host1,host2,...> mpi_hello

and it should output something as

Hello from processor 2 of 20
Hello from processor 7 of 20

If you monitor the activity of each node, for example with htop -d 1, you will a transient increase in CPU activity.

2- To use Gromacs, it has to be compiled with correct options.

We are installing Gromacs at the NFS /apps folder. To work with MPI, the correct MPI flags are required:

cd /apps
wget ftp://ftp.gromacs.org/pub/gromacs/gromacs-5.1.4.tar.gz
tar xvf gromacs-5.1.4.tar.gz
cd gromacs-5.1.4
mkdir build
cd build
cmake .. -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++ -DGMX_MPI=on -DGMX_BUILD_MDRUN_ONLY=on -DCMAKE_INSTALL_PREFIX=:/apps/gromacs-5.1.4-mpi -DBUILD_SHARED_LIBS=off -DGMX_BUILD_OWN_FFTW=ON -DGMX_DEFAULT_SUFFIX=mpi
make -j 4
make check
make install

To add the programs to the environment list just edit the /home/ubuntu/.bashrc and add the line

source /apps/gromacs/5.1.4/bin/GMXRC

and restart the terminal or source the profile file.

3- Run some Gromacs job.

Jobs can now be easily distributed over the nodes using

mpirun -np  -host <hostname1,hostname2,...> gmx_mpi mdrun

4- Playing with options to speed-up

4.1- Compiling with own FFTW

cd /apps
wget http://www.fftw.org/fftw-3.3.6-pl1.tar.gz
cd fftw-3.3.6-pl1
./configure CC=mpicc --enable.mpi
make -j 4
make install

4.1- Enabling sub-cycle counters at command time by recompiling with the new FFTW libraries and with GMX_CYCLE_SUBCOUNTERS:

cd /apps
wget ftp://ftp.gromacs.org/pub/gromacs/gromacs-5.1.4.tar.gz
tar xvf gromacs-5.1.4.tar.gz
mv gromacs-5.1.4 gromacs-5.1.4-dev
cd gromacs-5.1.4-dev
mkdir build
cd build
cmake .. -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++ -DGMX_MPI=on -DGMX_BUILD_MDRUN_ONLY=on -DCMAKE_INSTALL_PREFIX=:/apps/gromacs-5.1.4-dev -DBUILD_SHARED_LIBS=off -DGMX_BUILD_OWN_FFTW=ON -DGMX_DEFAULT_SUFFIX=mpi_dev -DGMX_CYCLE_SUBCOUNTERS=on 
make -j 4
make check
make install

[wip] Setting up a Bramble

0- Hardware

5 Raspberry Pi 3 cards
5 microSHDC 16 GB cards (class 10, 90 MB/s read speed)
5 basic USB-micro USB B cale
[Compact 10-Port USB Charger] Anker PowerPort 10

1- Get a Linux image. We used Ubuntu Server Standard 16.04 for Raspberry Pi 3 from Ubuntu Pi Flavour Maker. Other optins include Rasbian Jessie distro, the official Raspberry Debian distro and Ubuntu Raspberry.

2- Write the image to an SD card; We used a Windows machine for this with  Win32DiskImagerLinux and Mac instructions are also available. Download, extract the files and run as admin; select the iso file; select the drive where the SD card is (make sure it is the correct drive); click write and wait for the write to finish. This has to be done for all SD cards.

3- For each Pi card, insert the SD card and power it. Update the system and install some required packages:

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install openssh-server build-essential cmake

4- Configure the IP addresses in each node:

4.1- edit the /etc/hosts file in all nodes:

127.0.0.1 localhost
192.168.0.1 rasp1
192.168.0.2 rasp2
192.168.0.3 rasp3
192.168.0.4 rasp4
192.168.0.5 rasp5

4.2- in each node, assign the correct hostname in the /etc/hostname file:

rasp1 # for the head node; rasp2~5 for the remaining nodes

4.3- reboot the nodes
4.4- configure passwordless ssh, as the user that will run the cluster; on each node:

ssh-keygen -t rsa
#copy it to each of the other nodes:
ssh-copy-id @

5- Configure NFS

At the head node:

sudo apt-get install nfs-kernel-server
sudo cp /etc/exports /etc/exports.back
#add to /etc/exports on head
/home/ubuntu *(rw,sync,no_subtree_check)
/apps *(rw,sync,no_subtree_check)
sudo exportfs -a
sudo service nfs-kernel-server start

At the compute nodes:

sudo apt-get install nfs-common

On all machines:

sudo mkdir /apps
sudo chown -R ubuntu /apps

sudo ufw allow from 192.168.126.134
sudo ufw allow from 192.168.126.233
sudo ufw allow from 192.168.126.6
sudo ufw allow from 192.168.126.17
sudo ufw allow from 192.168.126.3

sudo mount rasp1:/home/ubuntu /home/ubuntu
sudo mount rasp1:/apps /apps

#insert the following lines at the of /etc/fstab
sudo nano /etc/fstab
rasp1:/home/goncalo /home/goncalo nfs
rasp1:/apps /apps nfs

sudo mount -a

6- Expand the SD card contents in order to use all the SD card space available by using raspi-config:

sudo apt-get install raspi-config
sudo raspi-config
#select 1. Expand Filesystem

 

NOTES

locale was wrong for us, the default is en_GB, it had to be changed:

sudo localedef -i pt_PT -f UTF-8 pt_PT.UTF-8