Multigrid Self Gravity - HongxuanJiang/athenak-transients GitHub Wiki
This page mirrors
docs/wiki/Multigrid-Self-Gravity.mdin the repository, which is the authoritative version.
Adding a <gravity> block turns on Newtonian self-gravity. A geometric multigrid solver
computes the gravitational potential from the gas density on the whole MeshBlock hierarchy.
The force and its work are then added to the hydro or MHD update as a source term. The module
works on uniform meshes, SMR and AMR, with periodic, isolated (multipole) or fixed-potential
boundaries. For hydro runs it also works with sink particles and with local adaptive time
stepping (LAT).
Use it for self-gravitating gas: a star that holds itself together, a collapsing cloud, or a stream that fragments. Do not use it for relativistic runs (it is Newtonian and has no GR version), for curvilinear coordinates (anisotropic Cartesian cells are supported), or on 1D and 2D meshes (the solver is 3D only). The cost is dominated by the Poisson solve, not by the coupling. Under LAT the potential is frozen over a window of fine steps and is solved only between windows.
An isolated problem (a star in vacuum) with multipole boundaries:
<gravity>
four_pi_G = 1.0 # 4 pi G in code units, so G = 1/(4 pi)
mg_bc = multipole # isolated system
threshold = 1.0e-6 # stop when the L2 defect is below this
niteration = 20 # but do at most 20 defect checksEverything is set in <gravity>. There is no <multigrid> block, and a deck that has one
aborts. Under LAT also set solve_dt (see LAT). The keys you will use most:
| key | what it does |
|---|---|
four_pi_G |
The value of |
mg_bc |
Boundary condition of the potential on all non-periodic faces: multipole (isolated), zerograd or zerofixed. Periodic faces follow the mesh. |
threshold, niteration
|
When to stop the iteration. Set at least one. A negative threshold runs exactly niteration V-cycles, a positive one iterates until the defect is that small, and 0 selects an automatic controller. |
solve_dt |
Solve the potential once per solve_dt of code time, at the first cycle at or after each multiple of solve_dt. Required with LAT. Without LAT it makes the solve schedule time-based. |
solve_every |
Solve only every N cycles (ignored when solve_dt > 0). The potential is reused in between. |
reuse_phi_after_amr |
After a regrid, start from the transferred potential instead of a cold restart of the solve. |
rho_grav_min |
Density at which gas is fully coupled to gravity. Gas below it is coupled less or not at all, which is useful for a vacuum background (details in the table below). |
The solver finds the potential four_pi_G. There
is no separate input for four_pi_G = 12.566370614359172. The value
four_pi_G = 1.0 used in the TDE decks means
Two other places need four_pi_G/(4 pi). One is the
analytic black-hole potential of the tde_external problem generator. The other is the sink
particles, but only when the deck has no <units> block. With a <units> block the sinks take
four_pi_G. If you
use <units>, set four_pi_G to
All keys are in <gravity> unless the table says otherwise. Defaults are the values in the
source.
| key | type | default | meaning |
|---|---|---|---|
self_gravity |
bool |
true when <gravity> exists |
Master switch. If false, the solver and the source term are not built. |
four_pi_G |
Real | -1 | The value of |
rho_grav_min |
Real | 0 (negative values act as 0) | Density at which gas is fully coupled to gravity, both as a source of the potential and as a receiver of the force. Between hydro/dfloor and rho_grav_min the coupling weight rises smoothly from 0 to 1 (a cubic smoothstep in log density), and below hydro/dfloor it is 0. If rho_grav_min is at or below hydro/dfloor, it is a hard cut at rho_grav_min. The default 0 couples every cell. |
reuse_phi_after_amr |
bool | false |
Keep the AMR-transferred potential as the first guess of the next solve instead of discarding it. |
| key | type | default | meaning |
|---|---|---|---|
threshold |
Real | -1 | Target L2 defect. Below 0 means a fixed number of V-cycles, above 0 means iterate to that defect, and 0 means the automatic controller (see How it works). |
niteration |
int | -1 | With threshold < 0: the number of V-cycles, which must be at least 1. With threshold > 0: the cap on defect checks (160 if niteration is not positive). With threshold = 0: the length of the first burst of V-cycles (fmg_ncycle + 1 if not set). One of threshold and niteration must be set. |
solve_every |
int | 1 (at least 1) | Solve on cycles with ncycle % N == 0. Ignored if solve_dt > 0. |
solve_dt |
Real | absent | Solve interval in code time, 0 or more. Above 0 it enables a time-based schedule and overrides solve_every (a message is printed if both are set). |
full_multigrid |
bool | true |
Start from a full-multigrid (FMG) ramp instead of plain V-cycles. Used only for a cold solve (a fresh start or a restart). A solve that starts from a valid potential always uses plain V-cycles. |
fmg_ncycle |
int | 1 | V-cycles per level of the FMG ramp. |
defect_check_interval |
int | 2 (at least 1) | V-cycles between defect evaluations. Forced to 1 when show_defect is 2 or more. |
auto_max_extra_cycles |
int | -1, meaning max(6, 3*fmg_ncycle), raised to at least 12 with refinement and to 32 after a regrid |
Cap on the extra cycles of the automatic controller. Read only when threshold = 0. |
warm_final_niter |
int | 4 (at least 1) | V-cycles on the last stage of a warm cycle when the mesh has not changed. Used only when threshold = 0 and niteration is not positive. |
| key | type | default | meaning |
|---|---|---|---|
omega |
Real | 1.15 | Over-relaxation weight of the smoother. |
npresmooth |
int | 1 (at least 1) | Pre-smoothing sweeps per level. |
npostsmooth |
int | 2 with SMR or AMR, else 1 (at least 1) | Post-smoothing sweeps per level. |
coarsest_min_sweeps |
int | 64 (at least 1) | Minimum number of relaxations on the coarsest root grid. |
mg_prolongation |
string |
tricubic with SMR or AMR, else trilinear
|
Interpolation used to prolong the correction. Any value other than tricubic means trilinear. |
mg_nghost |
int | 1 | Ghost cells on the multigrid levels. A value above 1 is fatal on a refined mesh. Its only use is a multi-rank optimisation on uniform meshes. |
mg_fc_symmetric |
bool | false |
Make the coarse/fine operator symmetric by dropping the tangential terms of the fine ghost value. Without it the composite operator is slightly non-symmetric, and the LAT ledger records the effect as E_asym. On the octet levels this changes only the convergence rate, not the solution. |
subtract_average |
bool |
true on a fully periodic mesh, else false
|
Remove the mean of the source and the solution. Forced to true (with a warning) when no face is zerofixed or multipole, and forced to false when a face is multipole. |
root_on_host |
bool |
true if the root grid has at most 4096 blocks |
Keep the root-grid arrays in host memory. |
The defaults are right for most runs. These keys trade extra local work for fewer halo exchanges on several ranks.
| key | type | default | meaning |
|---|---|---|---|
same_exchange_stride |
int | 1 (at least 1) | Switch for the reduced same-level halo exchange. Any value above 1 turns it on (the size of the value does not matter). It applies only on several ranks, only to coarser MeshBlock levels, and not to the first solve after a regrid. |
local_sweeps_per_exchange |
int | 1 (at least 1) | With the reduced exchange on: red and black sweeps done between two exchanges. |
reduced_exchange_max_edge |
int | 4 (at least 0) | With the reduced exchange on: it is used only on levels whose block edge is at most this many cells. |
ca_pack_smoother |
string | rbgs |
Smoother used while the reduced exchange is active on several ranks: rbgs (ordinary red-black Gauss-Seidel), jacobi, chebyshev or chebyshev2. The last two are the same two-stage Chebyshev smoother. Any other value is silently treated as rbgs. It has no effect when the reduced exchange is off. |
ca_pack_cheb_lambda_min, ca_pack_cheb_lambda_max
|
Real | 0.7, 1.95 | The interval of eigenvalues that the two Chebyshev relaxation weights are tuned to damp. They are dimensionless: the eigenvalues of the operator scaled by its diagonal lie between 0 and 2. Used only by the Chebyshev smoother (omega is not used by it). If the upper value is not above the lower one, it is reset to the lower value plus 1. |
distributed_coarse_solve |
bool | false |
Only the owner rank solves the root and octet levels and broadcasts the result. Effective only on several ranks with refinement, and only worth it at rank counts where repeating the coarse levels on every rank is no longer cheap. |
| key | type | default | meaning |
|---|---|---|---|
mg_bc |
string | none |
Default for all non-periodic faces: zerofixed, zerograd (or outflow), multipole or none. none keeps the mesh-derived choice, which is zerograd on every non-periodic face. periodic here is fatal. |
ix1_bc, ox1_bc, ix2_bc, ox2_bc, ix3_bc, ox3_bc
|
string | from mg_bc and the mesh |
Per-face override with the same words as mg_bc. A value that is not recognised is ignored. |
mporder |
int | 4 | Multipole order, 2 or 4. Read only when some face is multipole. |
auto_mporigin |
bool | true |
Recompute the expansion origin as the source's centre of mass at each solve. |
nodipole |
bool | false |
Skip the dipole terms. Fatal together with auto_mporigin. |
mporigin_x1, mporigin_x2, mporigin_x3
|
Real | required if auto_mporigin = false
|
Fixed expansion origin. |
mask_radius |
Real | -1 (off) | Zero the Poisson source outside this radius from the mask origin. It removes only the source: the gas inside still feels the potential. |
mask_origin_x1, mask_origin_x2, mask_origin_x3
|
Real | 0 | Centre of the mask. |
| key | type | default | meaning |
|---|---|---|---|
show_defect |
string or int | "0" |
0 or false is off, 1 or true prints the final defect of each solve, 2 or more prints every iteration. Any other value is fatal. |
show_timing |
bool | false |
Print one timing line per solve with the time of each phase. |
mg_verbose |
int | 0 (at least 0) | Gates exactly one warning: a multipole source whose total mass cancels. It is not a general verbosity switch. |
lat_time_centered_work |
bool | false |
LAT only. Solves again at each window end and adds <gravity> metadata. The tde_external problem generator writes them as history columns (W_cent, E_remap, E_floor, E_asym, and with lat_boundary_flux_diagnostics also B_mass, B_E, B_Wself, B_Wbh). Its requirements are in Requirements and refusals. |
lat_boundary_flux_diagnostics, lat_ledger_debug
|
bool | false |
Extra ledger columns, and a per-window printout on rank 0. The first requires lat_time_centered_work. |
reciprocity_test |
bool | false |
Run a diagnostic of the coarse/fine operator's symmetry at startup, then stop before any evolution. Its tuning keys (reciprocity_*) are for developers. |
| key | default | meaning |
|---|---|---|
hydro_srcterms/self_gravity |
absent | Legacy key. It is ignored when a <gravity> block exists. Without a <gravity> block, true aborts the run. |
problem/external_bh_gravity_source |
true for tde_external, else false
|
Analytic softened black-hole potential, applied by the same source-term kernel but separate from the solved potential. |
problem/bh_grav_rho_min |
0 | The rho_grav_min of the external black-hole branch, with the same ramp above hydro/dfloor. |
sink_particles/newton_g |
four_pi_G/(4 pi) |
The value of <units> block. See What is solved, and in which units. |
The solver finds the potential rho_grav_min.
fine grid smooth --restrict--> smooth --restrict--> ... coarsest: solve
fine grid smooth <-prolong---- smooth <-prolong---- ... (correct upwards)
- Multigrid. A smoothing sweep (red-black Gauss-Seidel with over-relaxation) removes short-wavelength error quickly but long-wavelength error slowly. So the solver restricts the residual to a coarser grid, where those long wavelengths are short, smooths there, and interpolates the correction back up. One trip down and up is a V-cycle. The coarsest grid is relaxed many times. A full-multigrid (FMG) start builds a good first guess by solving on the coarsest level and working up. Later solves start from the previous potential.
- Refinement. The levels between the root grid and the MeshBlocks that AMR or SMR creates are handled on the host as small 2x2x2 groups of cells (octets).
-
Stopping. With a negative
thresholdthe solver does exactlyniterationV-cycles. With a positive one it checks the L2 defect after everydefect_check_intervalV-cycles and stops when it is belowthreshold, or afterniterationchecks (160 ifniterationis not positive). Withthreshold = 0an automatic controller runs a first burst of V-cycles (niterationof them if set, elsefmg_ncycle+ 1) and then extra cycles, until the defect is below$10^{-6}$ times the L2 norm of the source (never less than$10^{-12}$ ), untilauto_max_extra_cyclesextra cycles have run, or until the defect stops improving. The$10^{-6}$ is fixed in the code and is not an input key. A threshold relative to the source carries over between problems, while an absolutethresholddepends on your units. -
Boundaries.
zerogradis a zero-gradient condition andzerofixedsets the potential to zero on the face.multipolesets the potential on the face from a multipole expansion of the source (up to the quadrupole formporder = 2, up to the hexadecapole for 4) about its centre of mass. - Coupling. The force enters as a source term in the form of Mullen, Hanawa and Gammie (2020). The momentum changes by the potential gradient. The energy changes by the same potential differences multiplied by the Godunov mass flux through each cell face. Energy conservation is therefore limited by the multigrid residual and by the age of the potential. The dual-energy auxiliary variable is not changed by this term.
-
When it solves. By default the potential is solved at every Runge-Kutta stage, because
the stage density changes. With
solve_everyabove 1 or withsolve_dt, it is solved only on the last stage of a due cycle and reused in between. The first solve of a run, and the solves after a regrid, always run. Withsolve_dtthe due times sit on the grid 0,solve_dt, 2solve_dt, ... of code time, so a restarted run solves on the same cycles as an uninterrupted one. -
Under LAT. The potential is frozen inside a window, and no Poisson solve runs inside it,
so the source density is never loaded from blocks at different local times. The driver
solves before a window, and again at a window end when the run stops there (with
lat_time_centered_workit solves at every window end). Before a window it first shortens the window to at mostsolve_dt. If the potential is invalid or due, or the window would run past the next due time, it solves first instead of cutting the window.
- Start with
threshold = 0(all the shipped TDE decks use it) for an automatic stopping rule. Use a fixedniteration(withthresholdnegative) for a reproducible cost, or a positivethresholdfor a controlled accuracy. - Leave
full_multigrid = true. It matters only for the first solve and for restarts. - With SMR or AMR the defaults already switch to
npostsmooth = 2and tricubic prolongation. - Leave the parallel tuning keys at their defaults unless you are tuning performance.
- To keep a vacuum background out of the gravity, set
rho_grav_minabove the background density. The TDE decks usehydro/dfloor = 1e-10andrho_grav_min = 1e-9, so the coupling rises from 0 to 1 over that decade.
- Use
mg_bc = multipolefor an isolated system and a periodic mesh for a box. - With all-Neumann boundaries (the default
zerograd) the mean of the source is subtracted, so the equation solved is the Jeans swindle, not the isolated problem. The driver only warns. -
mpordermust be 2 or 4. Withauto_mporigin = falsegive the threemporigin_x*keys.
- Set
gravity/solve_dtto the interval at which the potential should be refreshed. The potential is frozen inside a LAT window, so this key also caps the window length. A smaller value gives a fresher potential and shorter windows. - LAT is available for hydro only in this release, so LAT with self-gravity is for hydro runs.
- Local Adaptive Time Stepping explains how a window works. Its list of supported and refused modules applies here too. The conditions that self-gravity adds are in Requirements and refusals.
Sink particles are deliberately absent from the Poisson source. Their potential enters through
a separate source term that uses the same softened potential as the sink-to-gas reaction, so
the force pair is equal and opposite cell by cell. <sink_particles>/create = true requires a
<gravity> block, and sinks require <hydro>. mask_radius is the deck-level way to drop the
Poisson source outside a radius.
The potential is not written to restart files, so the first solve after a restart is a cold
FMG solve. A restarted run therefore agrees with the uninterrupted run only to the solver
tolerance, not bit for bit. With a solve_every or solve_dt cadence and no LAT, the first
stage after a restart also uses a freshly solved potential, where the uninterrupted run would
still use an older one. Under LAT the potential is solved at the first window start. LAT for MHD is not in this release, and the LAT gravity logic reads the hydro source terms only.
Each of these makes the run exit with an error.
Mesh and setup:
- The mesh is not 3D, or the MeshBlocks are not logically cubic or not a power of two in size.
- The deck has a
<multigrid>block, which is not a valid block name. - There is neither a hydro nor an MHD module.
-
<sink_particles>/create = truewithout a<gravity>block. -
hydro_srcterms/self_gravity = truewithout a<gravity>block.
Gravity keys:
-
four_pi_Gis not set, is 0, or is negative. - Neither
thresholdnorniterationis set, orniteration = 0with a negativethreshold. -
auto_target_defectis set. It is no longer a key. -
mg_bcisperiodicor an unknown word, or a face is periodic for gravity but not for the mesh. -
mporderis not 2 or 4, or bothauto_mporiginandnodipolearetrue. -
mg_nghostis above 1 on a refined mesh. -
show_defectis nottrue,falseor an integer, orsolve_dtis negative.
Time stepping and LAT:
-
time/latwith self-gravity and no positivegravity/solve_dt. The potential is frozen inside a window, and this key bounds the window length. -
time/lat_same_level = truewithlat_neighbor_limiter = face, with self-gravity or with the analytic black hole. The limiter must beallorhybrid. -
time/latwith an<mhd>block, because LAT for MHD is not in this release. LAT also refuses other modules, such as an ion-neutral pack (see the LAT page linked above). -
lat_boundary_flux_diagnostics = truewithoutlat_time_centered_work = true. -
lat_time_centered_work = trueunless all of these hold:time/lat = true,rho_grav_minis at mosthydro/dfloor, there is no positivemask_radius, there is no<sink_particles>block, and eithersolve_dt > 0orsolve_every = 1. At run time it also needs hydro with an energy equation (not isothermal), because the correction is added to the gas energy.
Two root-grid conditions only warn. A root grid that cannot be coarsened to one cell prints a
warning, and its coarsest level is then solved iteratively, which is less efficient. A root
grid with more than 100 coarsest cells prints a warning about the cost of the coarsest solve.
Separately, auto_max_extra_cycles and warm_final_niter are silently ignored unless
threshold = 0.
- Set
show_defect = 1for the final defect of each solve, or 2 for every iteration. -
show_timing = trueprints the time per phase. - The
grav_phioutput variable writes the gravitational potential (the solved potential plus the analytic black-hole potential if enabled). Before the first solve the solved part is zero. It is also added automatically to thehydro_u,hydro_w,mhd_uandmhd_woutput bundles. - The regression tests are in
tst/test_suite/multigrid/. Fromtst/, runpython run_test_suite.py --cpu --test test_suite/multigrid, or use--mpicpuor--gpu. They cover a two-sphere Poisson problem, Jeans waves and 64^3 hydro and MHD solves, with checks on convergence rate and on independence from MeshBlock size and rank count. Further gravity tests are intst/test_suite/nr/.
-
tde_externaluses the solved potential to hold the star together, and applies the black hole as an analytic potential in the same source-term kernel (problem/external_bh_gravity_source). - The decks that switch self-gravity on are the five
inputs/TDE_examples/*decks andinputs/sink_particles/creation.athinput.
- Mullen, Hanawa and Gammie (2020): the flux-consistent gravitational-work source form.
- Local Adaptive Time Stepping: how a window works and how LAT and gravity stay synchronized.