Quick Start: 3D Blast Wave - hyschive/gamer-fork GitHub Wiki

This demo will illustrate the following features:

  • Hybrid MPI/OpenMP/GPU
  • HDF5 output
  • yt analysis

It is recommended to first check Quick Start: 1D Shock Tube.

1. Install the external packages required for this demo.

2. Generate Makefile by configure.py.

cp ../example/test_problem/Hydro/BlastWave/generate_make.sh ./
sh generate_make.sh --openmp=true --gpu=true --mpi=true --hdf5=true
Execution results 
   ...
   ...
========================================
Makefile is created.
========================================

3. Compile the code.

make clean
make -j
Execution results 
   ...
   ...
Compiling GAMER --> Successful!

4. Create a working directory and copy the GAMER executable and example files of the test problem.

cd ../bin
mkdir blastwave
cd blastwave
cp -r ../../example/test_problem/Hydro/BlastWave/* .
cp ../gamer .
ls
Execution results 
Input__Flag_Lohner  Input__Parameter  Input__TestProb  README  clean.sh  gamer  plot_profile.gpt  plot_slice.py

5. Edit the input file Input__Parameter to set the total number of OpenMP threads per MPI process. The following example will launch 10 threads per process.

OMP_NTHREAD      10      # number of OpenMP threads (<=0=auto) [-1]

Important

See Hybrid MPI/OpenMP/GPU for the recommended configuration of the number of MPI processes and OpenMP threads.

6. Run the code with MPI (e.g., mpirun, mpiexec, aprun).

Important

Please consult the documentation of your system, especially about how to compile and launch "hybrid MPI/OpenMP" jobs.

The following example uses OpenMPI 1.8.4 to launch 2 MPI processes and 10 threads per process, assuming that there are two 10-core CPUs.

mpirun -np 2 -map-by ppr:1:socket:pe=10 ./gamer

Tip

See more options of MPI:

Execution results 
   ...
   ...
Time: 4.8438672e-03 -> 4.8932853e-03,   Step:     114 ->     115,   dt_base: 4.9418079e-05
Time: 4.8932853e-03 -> 4.9427910e-03,   Step:     115 ->     116,   dt_base: 4.9505779e-05
Time: 4.9427910e-03 -> 4.9923843e-03,   Step:     116 ->     117,   dt_base: 4.9593262e-05
Time: 4.9923843e-03 -> 5.0000000e-03,   Step:     117 ->     118,   dt_base: 7.6156993e-06
Output_DumpData_Total_HDF5 (DumpID = 10) ...
Output_DumpData_Total_HDF5 (DumpID = 10) ... done
Output_DumpData_Part (DumpID = 10) ...
Output_DumpData_Part (DumpID = 10) ... done
End_GAMER ...
End_MemFree ... done
End_GAMER ... done


~ GAME OVER ~

7. The code should have generated several log files Record__*, a series of 1D text data files Diag_*, and a series of HDF5 snapshots Data_*.

ls
Execution results 
Data_000000  Data_000005  Data_000010  Diag_000004  Diag_000009         README               Record__Note         clean.sh
Data_000001  Data_000006  Diag_000000  Diag_000005  Diag_000010         Record__Dump         Record__PatchCount   gamer
Data_000002  Data_000007  Diag_000001  Diag_000006  Input__Flag_Lohner  Record__LoadBalance  Record__Performance  plot_profile.gpt
Data_000003  Data_000008  Diag_000002  Diag_000007  Input__Parameter    Record__MemInfo      Record__TimeStep     plot_slice.py
Data_000004  Data_000009  Diag_000003  Diag_000008  Input__TestProb     Record__NCorrUnphy   Record__Timing

h5ls -r Data_000010
Execution results 
/                        Group
/GridData                Group
/GridData/Dens           Dataset {6016, 8, 8, 8}
/GridData/Engy           Dataset {6016, 8, 8, 8}
/GridData/MomX           Dataset {6016, 8, 8, 8}
/GridData/MomY           Dataset {6016, 8, 8, 8}
/GridData/MomZ           Dataset {6016, 8, 8, 8}
/Info                    Group
/Info/InputPara          Dataset {SCALAR}
/Info/KeyInfo            Dataset {SCALAR}
/Info/Makefile           Dataset {SCALAR}
/Info/SymConst           Dataset {SCALAR}
/Tree                    Group
/Tree/Corner             Dataset {6016, 3}
/Tree/Father             Dataset {6016}
/Tree/LBIdx              Dataset {6016}
/Tree/Sibling            Dataset {6016, 26}
/Tree/Son                Dataset {6016}

8. Validate the OpenMP configuration by following step 5 in Quick Start: 1D Shock Tube -- CPU-only with OpenMP. The example setup given above leads to something like

OpenMP Diagnosis
***********************************************************************************
OMP__SCHEDULE                   DYNAMIC
OMP__SCHEDULE_CHUNK_SIZE        1
OMP__NESTED                     OFF

CPU core IDs of all OpenMP threads (tid == thread ID):
------------------------------------------------------------------------
 Rank        Host  NThread  tid-00  tid-01  tid-02  tid-03  tid-04  tid-05  tid-06  tid-07  tid-08  tid-09
    0    golub121       10       2       0       4       8      10      12      14      16      18       6
    1    golub121       10       1       3       5       7       9      11      13      15      17      19
***********************************************************************************

Note

Here the two MPI processes run on the same node golub121 and all OpenMP threads use different CPU cores.

9. Validate the GPU configuration by following step 3 in Quick Start: 1D Shock Tube -- Hybrid OpenMP/GPU. Especially, make sure that the MPI processes running on the same node access different GPU ID (unless that is what you want).

10. Plot a HDF5 snapshot with yt. You can use the sample script plot_script/plot_slice.py.

cd plot_script
python plot_slice_gas.py -h
Execution results 
usage: plot_slice_gas.py [-h] -s IDX_START -e IDX_END [-d DIDX] [-i PREFIX]

Plot gas density slices for the blast wave test

optional arguments:
  -h, --help    show this help message and exit
  -s IDX_START  first data index
  -e IDX_END    last data index
  -d DIDX       delta data index [1]
  -i PREFIX     data path prefix [../]

Let's plot Data_000010.

python plot_slice_gas.py -s 10 -e 10
Execution results 
Command-line arguments:
-------------------------------------------------------------------
plot_slice_gas.py -s 10 -e 10
-------------------------------------------------------------------

yt : [WARNING  ] 2017-12-11 20:34:09,787 Cannot determine code units ==> Use units_override to specify the units
yt : [WARNING  ] 2017-12-11 20:34:09,788 Assuming length unit = 1.0 cm
yt : [WARNING  ] 2017-12-11 20:34:09,789 Assuming time unit = 1.0 s
yt : [WARNING  ] 2017-12-11 20:34:09,789 Assuming mass unit = 1.0 g
yt : [INFO     ] 2017-12-11 20:34:09,842 Parameters: current_time              = 0.005
yt : [INFO     ] 2017-12-11 20:34:09,842 Parameters: domain_dimensions         = [32 32 32]
yt : [INFO     ] 2017-12-11 20:34:09,843 Parameters: domain_left_edge          = [ 0.  0.  0.]
yt : [INFO     ] 2017-12-11 20:34:09,844 Parameters: domain_right_edge         = [ 1.  1.  1.]
yt : [INFO     ] 2017-12-11 20:34:09,845 Parameters: cosmological_simulation   = 0
yt : [INFO     ] 2017-12-11 20:34:14,783 xlim = 0.000000 1.000000
yt : [INFO     ] 2017-12-11 20:34:14,784 ylim = 0.000000 1.000000
yt : [INFO     ] 2017-12-11 20:34:14,786 xlim = 0.000000 1.000000
yt : [INFO     ] 2017-12-11 20:34:14,787 ylim = 0.000000 1.000000
yt : [INFO     ] 2017-12-11 20:34:14,810 Making a fixed resolution buffer of (('gas', 'density')) 800 by 800
yt : [INFO     ] 2017-12-11 20:34:18,923 Saving plot Data_000010_Slice_z_density.png

display Data_000010_Slice_z_density.png
Execution results

blastwave


Links

Add a custom footer
⚠️ **GitHub.com Fallback** ⚠️