ELBDM Hybrid Scheme - hyschive/gamer-fork GitHub Wiki
The hybrid scheme in GAMER is designed to efficiently simulate the behavior of Extremely Light Bosonic Dark Matter (ELBDM; also referred to as Fuzzy Dark Matter or Wave Dark Matter) in cosmological simulations.
- Switch to
psidm
branch
git checkout psidm
- Copy test problem
cp -r example/test_problem/LSS_Hybrid bin/
- Compile GAMER
cd src
cp ../bin/LSS_Hybrid/generate_make.sh ./
sh generate_make.sh
- Download initial condition
cd ../bin/LSS_Hybrid/
sh download_light_halo.sh
-
To use spectral interpolation
- Download table
sh download_spectral_interpolation_tables.sh
- Set
OPT__FLU_INT_SCHEME
andOPT__REF_FLU_INT_SCHEME
to8
inInput__Parameter
-
Run GAMER
mpirun -map-by ppr:2:socket:pe=8 --report-bindings ./gamer 1>>log 2>&1
-
Plot results at
$z=0$ :- Slice plots
python plot_script/plot_slice.py -s 69 -e 69
Execution results
- Projection plots
python plot_script/plot_projection.py -s 69 -e 69
Execution results
- Power spectrum
python plot_script/plot_power_spectrum.py -s 0 -e 69
Execution results
The wave scheme (GRAMFE_MATMUL
) is used for the dark grey and black grids.
The relative mass conservation error at
The wave-based solver in GAMER uses a finite-difference or spectral method to discretise the Schrödinger equation. The drawback of solving the wave formulation is the need to resolve the de Broglie wavelength of the matter wave even in regions where the density is low and smooth. The reason is that the velocity is related to the gradient of the phase of the wave function, i.e. a given velocity translates into a phase that varies on the scale of the de Broglie wavelength
In contrast to wave simulations, simulations of the fluid formulation do not need to resolve the de Broglie wavelength to correctly capture large-scale dynamics. In fact, they even allow the adoption of a Lagrangian picture via smoothed-particle hydrodynamics methods incorporating the quantum pressure. The possible simulation volumes are much closer to those attainable in traditional
The ELBDM hybrid scheme solves the fluid formulation of the Schrödinger-Poisson equations on a coarse grid on large scales and the wave formulation on a refined grid on small scales thereby combining the advantages of both approaches: The coarse grid on large scales need not resolve the de Broglie wavelength while the wave formulation on small scales correctly describes interference effects.
The hybrid scheme is particularly suited for cosmological simulations where a significant part of the simulation domain features a smooth density field and high infall velocities.
- Simulating the formation and evolution of cosmic structures under the influence of ELBDM physics.
- Investigating the core-halo structure of dark matter-dominated galaxies.
- Exploring the role of quantum pressure and interference in dark matter dynamics.
The configure.py script is the starting point for setting up a simulation. For the hybrid scheme, ensure the following flags are set:
-
--elbdm_scheme = ELBDM_HYBRID
: Determines the scheme type for ELBDM simulations. There are two options:-
ELBDM_WAVE
: Wave-only scheme. -
ELBDM_HYBRID
: Fluid-wave hybrid scheme.
-
-
--hybrid_scheme
: This option is specific to the hybrid scheme, defining the fluid dynamics treatment:-
HYBRID_UPWIND
: First-order upwind scheme. It is diffusive but stable. -
HYBRID_FROMM
: Second-order Fromm scheme. It has no limiter but can be unstable. -
HYBRID_MUSCL
: Second-order MUSCL scheme. It includes a limiter, balancing accuracy and stability.
For zoom-in and fluid-only simulations, use
--hybrid_scheme HYBRID_MUSCL
since unrefined regions may become unstable otherwise. For fully refined simulations use--hybrid_scheme HYBRID_FROMM
since it is slightly faster and more accurate thanHYBRID_MUSCL
. In case you encounter instabilities, test the more diffusive, first-order scheme--hybrid_scheme HYBRID_UPWIND
. -
-
ELBDM_FIRST_WAVE_LEVEL
determines on which level the hybrid scheme switches to the wave solver. This level requires careful consideration as it directly impacts the accuracy and performance of your simulation. It must be set based on trial and error. If it is set too low, the hybrid scheme becomes inefficient because a large part of the simulation domain will use the wave solver. If it is set too high, simulation regions with destructive interference tend to be overrefined. See theLSS_Hybrid
test problem for a rough estimate of how to set this parameter. -
OPT__FLAG_INTERFERENCE
must be set in conjunction with theInput__Flag_Interference
file. It refines fluid levels based on the quantum pressure and should generally be used in all hybrid simulations to ensure that regions where the fluid scheme fails are refined. - The
DT__HYBRID_*
parameters control the time-step size based on different aspects of the hybrid solver. Set these to negative for automatic determination, which is recommended unless you have specific requirements. The constants were determined empirically.DT__HYBRID_VELOCITY
with a value of up to3.5
has never shown any instability in tests, but is set to1.0
by default. -
OPT__LB_EXCHANGE_FATHER
is crucial for proper load balancing in MPI runs withELBDM_MATCH_PHASE
and should always be set to 1 for the hybrid scheme. Its purpose is to ensure that all MPI ranks exchange information about the father patches of physical patches. This is required for backward phase matching in the restriction operation. -
ELBDM_MATCH_PHASE
ensures that the phase at the fluid-wave-level transitions is unwrapped during the restriction operation. Alternatively, one could unwrap the phase directly in the fluid solver. The latter approach has the advantage that one could turn offELBDM_MATCH_PHASE
(andOPT__LB_EXCHANGE_FATHER
in MPI runs), but has the disadvantage that one needs to ensure that the resolution on the fluid solver levels is high enough to resolve$2\pi$ phase jumps.
The Input__Flag_Interference
file is used in GAMER to control when to refine regions with interference within the ELBDM hybrid scheme. This file allows users to specify parameters that determine when and where to apply interference conditions throughout different levels of the simulation.
The Input__Flag_Interference
file consists of rows of parameters, each corresponding to a specific level in the simulation. The columns represent different conditions that must be met for interference to be flagged. The parameters are as follows:
Level: The refinement level in the adaptive mesh at which the parameters will apply.
QP: The quantum potential threshold for flagging interference.
Density: The matter density condition. In the given file, this is set to 0, implying that density is not used as a criterion.
PhaseLap: The minimum phase laplacian to consider for interference effects.
OnlyAtExtrema: A flag to indicate whether to consider interference only at extrema of the wave function (0 = No, 1 = Yes).
An example configuration might look as follows
Level QP Density PhaseLap OnlyAtExtrema
0 0.03 0 1.0 0
1 0.03 0 1.0 0
2 0.03 0 1.0 0
...
10 0.03 0 1.0 0
In the above configuration, for all levels from 0 to 10, the simulation will flag interference when the quantum potential exceeds 0.03 or the phase Laplacian is at least 1.0, regardless of density and not exclusively at wave function extrema. The thresholds were determined empirically. Raising the Density parameter prevents refinement in low-density regions, but can make the scheme unstable. Setting OnlyAtExtrema = 1
has also been observed to be unstable in cosmological simulations.
The ELBDM hybrid scheme is shipped with a series of dedicated test problems highlighting different features of the hybrid scheme.
This test case simulates a pair of vortices moving linearly in a two-dimensional space. It evolves the wave function of the vortices over a fixed period under periodic boundary conditions. The initial setup involves a background wave function with an added periodic component along the x and y axes. If needed, a wave in the z-direction can be added as an option. The fluid scheme is used on level 0 and the wave scheme is used on level 1 close to the vortices. This test demonstrates refinement using Input__Flag_Interference
and also demonstrates the
Characteristics:
2D simulation with a possible addition in the z-direction.
Vortices move linearly with a specific wave function described by given parameters (A, kx, ky, Omega, Phase0).
If mass conservation is important, it can be ensured by enabling CONSERVE_MASS due to the periodic boundaries.
Run the test case in the default configuration (purely 2D) and plot the density and phase field at the final time using python plot_slice -s 20 -e 20 -i ./
. The L1 errors are as follows: error of density =
This test problem is aimed at simulating a rotating pair of vortices in two dimensions. The boundary conditions for this simulation are based on the analytical solution, and it involves a wave function that is a function of the radial distance and azimuthal angle. The fluid scheme is used on level 0 and the wave scheme is used on level 1 close to the vortices. Like the linear vortex pair test, this test demonstrates refinement using Input__Flag_Interference
and also demonstrates the
Characteristics:
2D simulation of a rotating vortex pair.
The wave function is described in polar coordinates and evolves over time.
This simulation references the work of Tzihong Chiueh and others, which discusses vortex turbulence in the context of linear Schrödinger wave mechanics.
Run the test case in the default configuration (purely 2D) and plot the density and phase field at the final time using python plot_slice -s 9 -e 9 -i ./
. The L1 errors are as follows: error of density =
This simulation investigates the gravitational collapse that can occur in one, two, or three dimensions. The focus is on the evolution of small-amplitude plane waves superimposed on a homogeneous background density, a setup that can help understand the early stages of structure formation in the universe.
Default setup and characteristics:
Designed to study gravitational collapse in multiple dimensions.
Periodic boundary conditions are used.
The hybrid scheme's refinement capabilities are showcased using Input__Flag_Interference.
Compared to a comoving reference simulation, it supports multi-dimensional collapse without assuming a comoving frame.
Simulation parameters include:
Perturbation_Amplitude: Maximum amplitude of the perturbations.
Perturbation_BgAmplitude: Amplitude of the background density.
Perturbation_N: Number of Fourier modes in the perturbation, with a maximum of 4.
Perturbation_NDim: Dimensionality of the problem, with unused dimensions set to zero amplitude.
Run the test case in the default configuration (purely 2D) and plot the density and phase field at the final time using python plot_slice -s 20 -e 20 -i ./
.
The result should look as follows:
Note that the initial resolution is lower and adaptive mesh refinement is used in this test problem.
A reference solution obtained with a
It is obvious that the two disagree. This is confirmed by plotting the respective 2D power spectra with the script example/yt/extract_power_spectrum.py
. Therefore, it is clear that the default refinement settings still need modifications to obtain reliable results.
This simulation is aimed at modeling cosmological large-scale structures using the hybrid scheme. The test problem is comprehensive, including scripts for setting up initial conditions (IC), converting wave IC to hybrid IC, and providing information on resolution and computational details relevant to the cosmological context.
Default setup and characteristics:
Initial conditions can be obtained through the provided scripts, which also facilitate the conversion to the hybrid IC format.
The simulation uses a base-level resolution and transitions between a fluid solver at lower levels to a wave solver at higher refinement levels.
The resolution is set to strike a balance between performance and accuracy.
Parameters such as ELBDM_MASS, A_INIT, OMEGA_M0, HUBBLE0, and BOX_SIZE are specified to define the cosmological framework of the simulation.
For more detailed information see the README
file of the test problem and for reference plots, see the quick start section.
The crucial issue in developing a hybrid code combining a fluid solver with a wave solver lies in reconstructing the wave function from the Madelung formulation of the Schrödinger-Poisson equations.
The Madelung transform
It is clear that the wave function can be locally reconstructed from the density and phase fields
We evolve the continuity equation using a first-order upwind scheme (HYBRID_UPWIND
), the second-order Fromm scheme without limiter (HYBRID_FROMM
) or a MUSCL scheme with a linear subgrid model together with the van Albada limiter (HYBRID_MUSCL
). As for the HJ scheme, we use a finite difference scheme. The convection term is discretised via the Sethian-Osher flux. The velocities at the cell faces are then computed as regular finite differences. We treat the cell averages DT__HYBRID_CFL
) and DT__HYBRID_VELOCITY
) were determined empirically.
While the fluid scheme has been observed to be stable in fluid-only runs, hybrid runs pose additional challenges. When vortices form on wave levels, the phase field develops discontinuities and the quantum pressure becomes ill-defined. To handle this case, the fluid scheme implements the following additional checks.
- switch to a first-order scheme in time and space in fully refined patches
- switch to a second-order centered-in-space finite difference scheme wherever the fluid equation produces unphysical data. This modification to the fluid scheme is work-in-progress and must be modified if it proves to be unstable in production runs.
- Issue: Bitwise reproducibility currently fails in the hybrid scheme due to the conversion from RE/IM to DENS/PHAS when storing fields in HDF5.
- Potential Solution: Implement conversion from RE/IM <-> DENS/PHAS using high-precision routines to ensure bitwise identity for significant digits.
- Issue: Non-reproducible overrefinement, similar to what occurs in the heavy halo case, still occurs without clear cause. See this Slack thread for more information.
- Insufficient resolution in the wave region.
- Too large a value for
QP
in the QP refinement criterion. - Use of monotonic interpolation which may not capture the complexity of wave functions.
- Interpolation in low-resolution regions.
- Phase interpolation errors, particularly in regions with complex interference patterns.
- Issue: The fluid scheme may fail in low-density regions, especially around vortices where the wave scheme is used. These failures can propagate through levels via restriction operations.
- Handling: If the fluid solver fails (indicated by negative densities or NaN values), and a refined wave counterpart exists, overwrite the values using second-order finite-difference discretization. But some users still report instabilities, especially together with spectral interpolation
- Global memory accesses in the ELBDM solver are uncoalesced, suggesting potential for optimization by implementing an effective 3D transpose on the GPU.
- Currently
ELBDM_FIRST_WAVE_LEVEL
is fixed. Ideally, the code should be able to adaptively increase the first wave level when the resolution turns out to be insufficient for the wave solver. This would require implementing a detection for insufficient resolution and converting patches back from the wave to the fluid representation.
- Ensure that transitions at the zoom-in boundary are handled correctly. Currently, the mismatch between fluid and wave schemes requires using the wave scheme outside of the zoom-in region.