HDF5 Format - PrincetonUniversity/athena GitHub Wiki
Athena++ uses its own HDF5 format when writing .athdf files. Below are the full specifications of the file format within the HDF5 framework. Integers are stored as 32-bit or 64-bit big-endian ints. Floating point values are stored in the native byte order of the machine that wrote the file (little-endian on x86 and arm64); HDF5 converts byte order when reading, so files are portable between machines. "Strings" here refer to variable length char arrays with at most 20 characters (excluding terminating null).
Two types below depend on the output settings:
- The data type of the cell-centered datasets is set by
data_format(see Outputs): 32-bit floats by default, or 64-bit floats if-h5doublewas specified at compile time. It can also be a 16-, 32-, 64-, or 128-bit float or an 8-, 16-, 32-, or 64-bit unsigned int. In the code this ish5out_t. - The mesh float type of
Time,RootGridX1/X2/X3, and the coordinate datasets is the same as the data type for 32-bit and wider floats, and is the default floating point type (32-bit, or 64-bit with-h5double) for 16-bit float and unsigned int data. In the code this ismesh_t.
-
NumCycles- scalar 32-bit int
- cycle number at which file is written
-
Time- scalar of the mesh float type
- simulation time at which file is written
-
Coordinates- scalar string
- name of coordinate system (matches configure option and
.cppfile)
-
RootGridX1/RootGridX2/RootGridX3- triples of the mesh float type
- minimum, maximum, and geometric ratio in x1/x2/x3-direction
-
RootGridSize- triple of 32-bit ints
- numbers of cells at root level in x1-, x2-, and x3-directions
-
NumMeshBlocks- scalar 32-bit int
- total number of MeshBlocks in the simulation
-
MeshBlockSize- triple of 32-bit ints
- numbers of cells in each MeshBlock in x1-, x2-, and x3-directions
-
MaxLevel- scalar 32-bit int
- highest level of mesh refinement present, with root being 0
-
NumVariables- array of 32-bit ints
- length is number of cell-centered datasets in file
- each entry is number of scalar variables in corresponding dataset
-
DatasetNames- array of strings
- length matches that of
NumVariables - each entry is name of cell-centered dataset
- order matches that of
NumVariables
-
VariableNames- array of strings
- length equals sum of entries in
NumVariables - each entry is name of cell-centered variable
- order matches that of
DatasetNames, with variables ordered by their index within the dataset
Let "NBlocks" be the value of the NumMeshBlocks attribute. Let "nx1," "nx2," and "nx3" be the values in the MeshBlockSize attribute.
-
Levels- (NBlocks) array of 32-bit ints
- Refinement levels of MeshBlocks, with root being 0
-
LogicalLocations- (NBlocks)×(3) array of 64-bit ints
- For each MeshBlock, the offsets in the x1-, x2-, and x3-directions from the minimum edge of the grid
- Counting is done as though entire grid is at the refinement level of the MeshBlock in question
-
x1f/x2f/x3f- (NBlocks)×(nx1/nx2/nx3+1) array of the mesh float type
- Values of interface locations along x1/x2/x3-direction
-
x1v/x2v/x3v- (NBlocks)×(nx1/nx2/nx3) array of the mesh float type
- Values of cell centers along x1/x2/x3-direction
- Cell-centered datasets
- One for each entry in
DatasetNames - Each one is an (NVars)×(NBlocks)×(nx3)×(nx2)×(nx1) array of the data type
- NVars varies between datasets and is given by corresponding entry in
NumVariables
- One for each entry in
The following names of datasets and variables may be output depending on what is requested via the variable argument in the <output> block. (Variables and datasets will not be included if they are not included in the simulation due to the selected physics.)
| Output Variable | Dataset Names | Variable Names |
|---|---|---|
| prim | prim | rho, press, vel1, vel2, vel3 |
| B | Bcc1, Bcc2, Bcc3 | |
| cons | cons | dens, Etot, mom1, mom2, mom3 |
| B | Bcc1, Bcc2, Bcc3 | |
| d | hydro | rho |
| p | hydro | press |
| v | hydro | vel1, vel2, vel3 |
| D | hydro | dens |
| E | hydro | Etot |
| m | hydro | mom1, mom2, mom3 |
| bcc | B | Bcc1, Bcc2, Bcc3 |
| uov | uov | user_out_var0, user_out_var1, ... |
When Configuring Athena+++ with both the -mpi and -hdf5 flags, it is important that the HDF5 library was built with MPI support. The solver will not compile if only the serial HDF5 routines are available.
If you are building HDF5 from source, the HDF5 library source code contains both serial HDF5 and Parallel HDF5 (PHDF5). Simply specify the MPI compiler when configuring the installer and pass the following flag:
CC=mpicc ./configure --enable-parallel ...
A POSIX compliant file system and an MPI library with MPI-I/O are requirements for Parallel HDF5. A parallel file system is necessary for good performance. To check if an installed HDF5 library was linked with the MPI library, check the output of h5cc -showconfig.
On the Princeton University Research Computing (RC) clusters, there are a few currently recommended compiler and library settings for compiling Athena++ with MPI and Parallel HDF5:
- Intel OneAPI + OpenMPI (best overall for Intel hardware)
module load anaconda3/2024.10
module load intel-tbb/2021.13 intel-rt/2024.2 intel/2024.2
module load intel-oneapi/2024.2
module load openmpi/oneapi-2024.2/4.1.8
module load hdf5/oneapi-2024.2/openmpi-4.1.6/1.14.4
- Intel OneAPI + IntelMPI (less stable/reliable; seems to be better for communication/IO bound sims when it works)
module load anaconda3/2024.10
module load intel-tbb/2021.13 intel-rt/2024.2 intel/2024.2
module load intel-oneapi/2024.2
module load intel-mpi/oneapi/2021.13
module load hdf5/oneapi-2024.2/intel-mpi/1.14.4
- GCC + OpenMPI (less optimized, compiles faster, works on all RC hardware)
module load anaconda3/2024.10
module load openmpi/gcc/4.1.6
module load hdf5/gcc/openmpi-4.1.6/1.14.4
They are loaded using the Environment Modules package via module load intel/2024.2, for example, and the anaconda3/2024.10 is just to have a more featureful Python setup, which is necessary for the vis scripts.
There is a default serial HDF5 on the RC clusters, but it is still recommended to load an HDF5 module that was compiled with the same compiler you are using to compile Athena++.
Matt Coleman is currently the point of contact for issues related to running Athena++ on Princeton Research Computing's clusters.
Last updated on 8/1/2025