[wip] Setting up a Bramble

0- Hardware

5 Raspberry Pi 3 cards
5 microSHDC 16 GB cards (class 10, 90 MB/s read speed)
5 basic USB-micro USB B cale
[Compact 10-Port USB Charger] Anker PowerPort 10

1- Get a Linux image. We used Ubuntu Server Standard 16.04 for Raspberry Pi 3 from Ubuntu Pi Flavour Maker. Other optins include Rasbian Jessie distro, the official Raspberry Debian distro and Ubuntu Raspberry.

2- Write the image to an SD card; We used a Windows machine for this with  Win32DiskImagerLinux and Mac instructions are also available. Download, extract the files and run as admin; select the iso file; select the drive where the SD card is (make sure it is the correct drive); click write and wait for the write to finish. This has to be done for all SD cards.

3- For each Pi card, insert the SD card and power it. Update the system and install some required packages:

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install openssh-server build-essential cmake

4- Configure the IP addresses in each node:

4.1- edit the /etc/hosts file in all nodes: localhost rasp1 rasp2 rasp3 rasp4 rasp5

4.2- in each node, assign the correct hostname in the /etc/hostname file:

rasp1 # for the head node; rasp2~5 for the remaining nodes

4.3- reboot the nodes
4.4- configure passwordless ssh, as the user that will run the cluster; on each node:

ssh-keygen -t rsa
#copy it to each of the other nodes:
ssh-copy-id @

5- Configure NFS

At the head node:

sudo apt-get install nfs-kernel-server
sudo cp /etc/exports /etc/exports.back
#add to /etc/exports on head
/home/ubuntu *(rw,sync,no_subtree_check)
/apps *(rw,sync,no_subtree_check)
sudo exportfs -a
sudo service nfs-kernel-server start

At the compute nodes:

sudo apt-get install nfs-common

On all machines:

sudo mkdir /apps
sudo chown -R ubuntu /apps

sudo ufw allow from
sudo ufw allow from
sudo ufw allow from
sudo ufw allow from
sudo ufw allow from

sudo mount rasp1:/home/ubuntu /home/ubuntu
sudo mount rasp1:/apps /apps

#insert the following lines at the of /etc/fstab
sudo nano /etc/fstab
rasp1:/home/goncalo /home/goncalo nfs
rasp1:/apps /apps nfs

sudo mount -a

6- Expand the SD card contents in order to use all the SD card space available by using raspi-config:

sudo apt-get install raspi-config
sudo raspi-config
#select 1. Expand Filesystem



locale was wrong for us, the default is en_GB, it had to be changed:

sudo localedef -i pt_PT -f UTF-8 pt_PT.UTF-8

Compiling NWCHEM under Ubuntu with MPICH support

Note: Some code is written in the csh shell (setenv) but should be written in bash (export ="name")

1. Install required packages:

sudo apt-get install python-dev gfortran libopenblas-dev libopenmpi-dev openmpi-bin tcsh make 

2. Set environment variables at a /home/$USER/.nwchem_login file containing

export USE_MPI=y
export PYTHONHOME=/usr
export BLASOPT="-lopenblas -lpthread -lrt"
export BLAS_SIZE=4
export USE_64TO32=y
setenv NWCHEM_TOP /nwchem

The following environment variables need to be set when NWChem is compiled with MPI:

setenv USE_MPI y
setenv USE_MPIF y
setenv USE_MPIF4 y
setenv MPI_LOC /openmpi-1.4.3 (for example, if you are using OpenMPI)
setenv MPI_LIB /openmpi-1.4.3/lib
setenv MPI_INCLUDE /openmpi-1.4.3/include
setenv LIBMPI "-lmpi_f90 -lmpi_f77 -lmpi -lpthread" 

Adding one of the following blocks according to the implemented MPI:

setenv MPI_LOC /usr/local #location of mpich installation
setenv MPI_LIB $MPI_LOC/lib
setenv MPI_INCLUDE $MPI_LOC/include
setenv LIBMPI "-lfmpich -lmpich -lpmpich"


setenv MPI_LOC /usr/local #location of mpich2 installation
setenv MPI_LIB $MPI_LOC/lib
setenv MPI_INCLUDE $MPI_LOC/include
setenv LIBMPI "-lmpich -lopa -lmpl -lrt -lpthread"


setenv MPI_LOC /usr/local #location of openmpi installation
setenv MPI_LIB $MPI_LOC/lib
setenv MPI_INCLUDE $MPI_LOC/include
setenv LIBMPI "-lmpi_f90 -lmpi_f77 -lmpi -ldl -Wl,--export-dynamic -lnsl -lutil"

3. Source the .nwchem_login file at the .bashrc file:

source .nwchem_login

4. Compile the software:

make nwchem_config NWCHEM_MODULES="all python"
make 64_to_32


cd $NWCHEM_TOP/src
make nwchem_config
make FC=gfortran >& make.log

Sometimes a default.nwchemrc file at the user home directory is required to list the directories NWChem should use:

nwchem_basis_library /usr/local/NWChem/data/libraries/
nwchem_nwpw_library /usr/local/NWChem/data/libraryps/
ffield amber
amber_1 /usr/local/NWChem/data/amber_s/
amber_2 /usr/local/NWChem/data/amber_q/
amber_3 /usr/local/NWChem/data/amber_x/
amber_4 /usr/local/NWChem/data/amber_u/
spce    /usr/local/NWChem/data/solvents/spce.rst
charmm_s /usr/local/NWChem/data/charmm_s/
charmm_x /usr/local/NWChem/data/charmm_x/

In this case, edit the .bashrc file to include

source nwchem_default

Setting up Gaussian09 on Ubuntu

This has worked without issues on Ubuntu 15.04 and above to install the Gaussian 09 binaries.

  1. Copy all Gaussian files to a folder under the home folder, for example
  1. Create a scratch directory, for example
  1. Create a file at the same folder to set environment variables, for example

with the following content

alias g09="/home/user/g09/g09"
export g09root = "/home/user"
export GAUSS_EXEDIR="/home/user/g09"
export GAUSS_SCRDIR="/home/user/g09scratch"
export g09root GAUSS_SCRDIR
  1. Source the parameters by including the line
source /home/user/.login

at the end of the /home/user/.bashrcfile.
5. Jobs can be run from the terminal at the job file location simply by issuing the command

g09 filename

A. In different systems g09 will complain of different missing libraries. Typically sudo apt-get install of the missing libraries will solve this. Some frequent missing libraries and packages are:

B. Some scripts may not run if you are compiling the executables, requiring some modifications:
sed -i 's/gau-machine/.\/gau-machine/' *
sed -i 's/set-mflags/.\/set-mflags/' bldg09
sed -i 's/bsd\/set-mflags/.\/bsd\/set-mflags/' bldg09
sed -i 's/gau-hname/.\/gau-hname/' set-mflags
sed -i 's/cachesize/.\/cachesize/' set-mflags
cp gau-unlimit ..
cp cachesize ..
cp set-mflags ..
cp ../gau-machine .
sed -i 's/getline/get_line/' fsplit.c

C. Gaussian may complain that files are world accessible. The easiest way to solve this issue is to grant read/write permission only to user.

sudo chown -R root:gauss09 g09
sudo chmod -R o-rwx g09
sudo chmod -R 777 gv

Compiling GROMACS 2016.1 in Ubuntu 16.04 with GPU support

  1. Install Intel Paralell XE cluster from binary file with ALL components

This install the Intel C, C++ and F compiler, the MKL libraries and the Intel MPI version. To run on one single machine, mpi is not required. To run across multiples machines, a mpi implementation is required, either Intel, MPICH or OpenMPI.

  1. Set environment variables (optionally add to .bashrc)

source intel/bin/compilervars.sh -arch intel64 -platform linux
export PATH=$PATH:"/opt/intel"
export MKLROOT="/opt/intel"
export CC=icc
export CXX=icc
export F77=ifort
export CFLAGS="-O3 -ipo- -static -std=c99 -fPIC -DMKL_LP64 -DM_PI=3.1415926535897932384"
export CPPFLAGS="-I$MKLROOT/include -I$MKLROOT/include/fftw"
export LDFLAGS="-L$MKLROOT/lib/intel64 -L$MKLROOT/../compiler/lib/intel64"
export LD_LIBRARY_PATH="$MKLROOT/lib/intel64:$MKLROOT/../compiler/lib/intel64:$LD_LIBRARY_PATH"

  1. Compile GROMACS
    tar xfz gromacs-2016.1.tar.gz
    cd gromacs-2016.1
    mkdir build
    cd build
    #make -j 6
    make check
    sudo make install
    source /opt/gromacs-2016.1-mod/bin/GMXRC


INTEL Parallel Studio XE Install Guide for Linux

NVIDIA CUDA Quick Start Guide

NVIDIA CUDA Installation Guide for Linux