[wip] Using a Bramble as a Gramble

The Bramble has been setup and running in a previous post.

1- Making sure MPI works fine

In the NFS folder create a mpi_hello.c file with


int main(int argc, char** argv) {
 int myrank, nprocs;

 MPI_Init(&argc, &argv);
 MPI_Comm_size(MPI_COMM_WORLD, &nprocs);
 MPI_Comm_rank(MPI_COMM_WORLD, &myrank);

 printf("Hello from processor %d of %d\n", myrank, nprocs);

 return 0;

and compile it with mpicc

mpicc mpi_hello.c -o mpi_hello

Run with

mpirun -np  -host <host1,host2,...> mpi_hello

and it should output something as

Hello from processor 2 of 20
Hello from processor 7 of 20

If you monitor the activity of each node, for example with htop -d 1, you will a transient increase in CPU activity.

2- To use Gromacs, it has to be compiled with correct options.

We are installing Gromacs at the NFS /apps folder. To work with MPI, the correct MPI flags are required:

cd /apps
wget ftp://ftp.gromacs.org/pub/gromacs/gromacs-5.1.4.tar.gz
tar xvf gromacs-5.1.4.tar.gz
cd gromacs-5.1.4
mkdir build
cd build
make -j 4
make check
make install

To add the programs to the environment list just edit the /home/ubuntu/.bashrc and add the line

source /apps/gromacs/5.1.4/bin/GMXRC

and restart the terminal or source the profile file.

3- Run some Gromacs job.

Jobs can now be easily distributed over the nodes using

mpirun -np  -host <hostname1,hostname2,...> gmx_mpi mdrun

4- Playing with options to speed-up

4.1- Compiling with own FFTW

cd /apps
wget http://www.fftw.org/fftw-3.3.6-pl1.tar.gz
cd fftw-3.3.6-pl1
./configure CC=mpicc --enable.mpi
make -j 4
make install

4.1- Enabling sub-cycle counters at command time by recompiling with the new FFTW libraries and with GMX_CYCLE_SUBCOUNTERS:

cd /apps
wget ftp://ftp.gromacs.org/pub/gromacs/gromacs-5.1.4.tar.gz
tar xvf gromacs-5.1.4.tar.gz
mv gromacs-5.1.4 gromacs-5.1.4-dev
cd gromacs-5.1.4-dev
mkdir build
cd build
make -j 4
make check
make install

Compiling GROMACS 2016.1 in Ubuntu 16.04 with GPU support

  1. Install Intel Paralell XE cluster from binary file with ALL components

This install the Intel C, C++ and F compiler, the MKL libraries and the Intel MPI version. To run on one single machine, mpi is not required. To run across multiples machines, a mpi implementation is required, either Intel, MPICH or OpenMPI.

  1. Set environment variables (optionally add to .bashrc)

source intel/bin/compilervars.sh -arch intel64 -platform linux
export PATH=$PATH:"/opt/intel"
export MKLROOT="/opt/intel"
export CC=icc
export CXX=icc
export F77=ifort
export CFLAGS="-O3 -ipo- -static -std=c99 -fPIC -DMKL_LP64 -DM_PI=3.1415926535897932384"
export CPPFLAGS="-I$MKLROOT/include -I$MKLROOT/include/fftw"
export LDFLAGS="-L$MKLROOT/lib/intel64 -L$MKLROOT/../compiler/lib/intel64"
export LD_LIBRARY_PATH="$MKLROOT/lib/intel64:$MKLROOT/../compiler/lib/intel64:$LD_LIBRARY_PATH"

  1. Compile GROMACS
    tar xfz gromacs-2016.1.tar.gz
    cd gromacs-2016.1
    mkdir build
    cd build
    #make -j 6
    make check
    sudo make install
    source /opt/gromacs-2016.1-mod/bin/GMXRC


INTEL Parallel Studio XE Install Guide for Linux

NVIDIA CUDA Quick Start Guide

NVIDIA CUDA Installation Guide for Linux