Compiling with OpenMPI

[ckande]

For anyone interested in compiling with OpenMPI, this was the mk file that I used. The overall combination was:

- Intel 13.1
- Intel MKL (BLAS, LAPACAK, SCALAPACK, BLACS)
- OPENMPI-1.6.5 (./configure --with-tm --with-openib)
- FFTW-2.1.5 (./configure) (MKL FFTW2XF did not work for me)
- XERCES-2.8.0 (./runConfigure -p linux -r none -s -c icc -x icpc)

mk file

Code: Select all

# modules required: intel impi mkl fftw2
# Before using make, use:
# $ module load intel mkl fftw2
# $ module swap mvapich2 impi 
#-------------------------------------------------------------------------------
#
# Copyright (c) 2013 The Regents of the University of California
#
# This file is part of Qbox
#
# Qbox is distributed under the terms of the GNU General Public License 
# as published by the Free Software Foundation, either version 2 of 
# the License, or (at your option) any later version.
# See the file COPYING in the root directory of this distribution
# or <http://www.gnu.org/licenses/>.
#
#-------------------------------------------------------------------------------
#
#  stampede.mk
#
#-------------------------------------------------------------------------------
#
#XERCESCDIR=$(HOME)/opt/xerces/2.8.0_32_mvapich2
XERCESCDIR=$(HOME)/opt/xerces/2.8.0_32
MKLROOT = $(HOME)/opt/intel/mkl
FFTWDIR = $(HOME)/opt/fftw/2.1.5
#_mvapich2

PLTOBJECTS =
CXX = mpicxx
LD = mpicxx

#PLTFLAGS += -DUSE_FFTW
PLTFLAGS += -DUSE_MPI -DSCALAPACK -DADD_
#PLTFLAGS += -D__linux__
PLTFLAGS += -DUSE_XERCES
PLTFLAGS += -DUSE_FFTW
PLTFLAGS += -D_LARGEFILE64_SOURCE  -D_FILE_OFFSET_BITS=64
#PLTFLAGS += -DPARALLEL_FS
#PLTFLAGS += -DCHOLESKY_REMAP=16
#PLTFLAGS += -DMPICH_IGNORE_CXX_SEEK

INCLUDE = -I$(FFTWDIR)/include -I$(XERCESCDIR)/include

CXXFLAGS= -O3 $(INCLUDE) $(PLTFLAGS) $(DFLAGS) #-g -debug parallel #-openmp #-vec-report1

LDFLAGS = $(LIBPATH) $(LIBS) #-g -debug parallel #-openmp
LIBPATH =

LIBS =  -L$(MKLROOT)/lib/intel64 -lmkl_scalapack_lp64 -lmkl_intel_lp64  \
         -lmkl_sequential -lmkl_core -lmkl_blacs_openmpi_lp64 -lpthread \
         -lmkl_lapack95_lp64 -lm \
         -L$(FFTWDIR)/lib -lfftw \
         -L$(XERCESCDIR)/lib -lxerces-c
         #-luuid \
         #-Bstatic \
         #-Bdynamic

#-------------------------------------------------------------------------------

[fgygi]

Thanks for the post.
Also note a particular feature of openmpi: the parameters used with the mpirun command are important to determine which transport layer is used (e.g. infiniband vs gigabit ethernet). On our cluster, using the default parameters causes openmpi to use gigabit ethernet, which gives very poor performance for Qbox. In order to get infiniband to be used, the following parameters must be used with the mpirun command:

Code: Select all

--mca btl sm,self,openib

Note that these parameters may depend on the openmpi installation on your cluster.