MPgrafic

A parallel version of Ed. Bertschinger’s grafic package for generating ICs in MPI

par Simon Prunet, Pichon Christophe (mercredi 22 février 2006)

MPgrafic 0.3

an example of large run

MPgrafic is a parallel MPI version of grafic-1 which can produce large cosmological initial conditions on a cluster without requiring shared memory. The real Fourier transforms are carried in place using fftw, while minimizing the amount of used memory (at the expense of performance), in the spirit of grafic-1. The writting of the output file is also carried in parallel. Other than the technical parallelisation, it provides 3 extensions over grafic-1
it can produce power spectra with baryon wiggles ( DJ Eisenstein and W. Hu, Ap. J. 496 ) ;
it has the optionnal ability to load a lower resolution noise map corresponding to the low frequency component which will fix the larger scale modes of the simulation (extra flag 0/1 at the end of the input process) in the spirit of grafic-2.
it can be used in conjunction with constrfield, which generates initial conditions phases from a list of local constraints on density, tidal field density gradient and velocity.

The source archive can be downloaded via ftp.

it runs interactively using the command (on 15 processors)

mpirun -np 15 mpgrafic

The program takes the same parameters as those described in grafic-1 (see the documentation of grafic-1 for the meaning of those parameters) + the extension 0 at the end (if no low frequency padding is required) or



1

name_of_white_noise_associated_with_constraint

then the initial condition will satisfy the low frequency phases associated with the white noise file. This second mode requires special care in making sure that both cosmologies (used for the constraints and for the higher resolution ICs) are consistant. (only crude consistencies checks are carried by mpgrafic).

In practice, the launch process can be automated as follow : if lauch.graphic contains



#!/bin/bash

mpgrafic <param

and param contains



4

0.3,0.7,70    

0.05

1

 -0.92

0.01,100

 -50


1

0









1

1234

white.dat

0

swhite.dat

then the command

mpirun -np 4 launch.grafic

will launch non interactively mpgrafic (on 4 processors) with the above cosmology and settings. (standard lambda CDM cosmogony with baryon wiggles $\Omega_b=0.05$ a la Hu for a 50 $h^{-1}$ Mpc box using the seed 1234)

If the parameter file ends with



1

swhite.dat

it will load the phases from the smaller white noise file contained in swhite.dat and pad the higher frequencies with the phase drawn from the seed 1234.

Performance : the program takes 10 min walltime (on a quadri opteron) when procucing the 8 fields corresponding to a simulation. It takes about 200 min of CPU for a simulation on 12 processors of HPC1.

Portability : mpgrafic is written in fortran 90 and was tested with ifort and lam-7. It aims at producing files which are identical to those produced by grafic-1 ; hence grafic-2 can be used to further refine locally some sub regions.

Notes : the random generation avoids replicate sequences for the different processors ; the fft is carried in double precision, unless the flag PRECISION=SING is used ;