This page mirrors docs/wiki/Local-Adaptive-Time-Stepping.md in the repository, which is the authoritative version.

LAT for MHD and GRMHD. In this release, LAT is available for hydrodynamics, including self-gravity. The development version of the code also supports LAT for MHD and GRMHD, including GRMHD on dynamical spacetimes. These paths are not included in this public release, and a run with time/lat = true and an <mhd> block stops with an error. They are available on request from Hong-Xuan Jiang ([email protected]).

Normally every MeshBlock (a block, for short) advances with the smallest time step found anywhere in the domain. With LAT, each block advances with its own step. A block's step is the largest step it can take on its own, rounded down to a power-of-two multiple of the finest step. That multiple is the block's factor (1, 2, 4, ...), and all blocks with the same factor form a bin. A slow region, such as a large low-density envelope, then needs far fewer updates than the small dense region that sets the global step.

Two mechanisms keep the solution consistent where neighbouring blocks advance at different rates. A block that updates takes its ghost-zone values from a neighbour that is partway through a longer step by interpolating that neighbour's state in time. The flux across such an interface is recorded on both sides, and the mismatch is added back later (refluxing), which keeps the scheme conservative.

Use it when a few blocks need a time step much smaller than most of the domain, for example a star or disk inside a large, cheap envelope, and the run is hydrodynamics (with or without self-gravity, sink particles and an external black hole). With the default lat_same_level = false, all blocks of one refinement level share one factor, so a gain needs SMR or AMR.

Do not use it when almost all blocks need the same step (there is nothing to save, and the boundary refresh in every tick costs extra), when the run needs a module that LAT refuses (see Practical guidance), or when you need the same trajectory on different numbers of MPI ranks and do not set hydro_lat_min_bin_count explicitly.

Add two keys to <time>:

<time>
lat        = true
lat_levels = 3        # blocks may take steps of 1, 2, 4 or 8 times the finest step

With self-gravity, <gravity>/solve_dt is also required. The keys you will touch most often:

All time/ keys are read only when LAT is on. Legacy spellings are listed below the table.

Legacy spellings. Both spellings of a key are accepted. hydro_lat maps to lat, hydro_lat_levels to lat_levels, and hydro_lat_same_level, hydro_lat_same_level_max_ratio, hydro_lat_neighbor_limiter, hydro_lat_diagnostics and hydro_lat_union_stage1 to the same names without hydro_. In the other direction, lat_min_bin_count, lat_gid_reorder and lat_post_amr_rebalance map to the hydro_lat_ names.

Each block works out the largest step it can take on its own. The ratio of that step to the global step, rounded down to a power of two and capped at 2^lat_levels, is the block's factor. Rules then lower factors where a large one would be unsafe, in this order:

1. A cap from the problem generator, if it installs one. The TDE generator forces factor 1 around the excised black hole.
2. Sink pinning: blocks around a sink are forced to factor 1.
3. The refinement-level collapse, unless lat_same_level = true.
4. The minimum bin population, hydro_lat_min_bin_count. A bin above factor 1 with too few blocks is merged into the next bin down.
5. A neighbour limiter that keeps the factors of neighbouring blocks close. Across a refinement interface a finer block never has a larger factor than the coarser one. Between same-level neighbours the ratio is at most lat_same_level_max_ratio.

The factors are recomputed at startup, at every window end and after every regrid.

The largest factor that survives sets the length of a window, which is that many finest steps. Inside a window the driver runs one tick per finest step. On tick k, the blocks whose factor divides k take a step of factor finest steps.

tick             0     1     2     3     4 (window end)
factor 4 block   |-----------------------|
factor 2 block   |-----------|-----------|
factor 1 block   |-----|-----|-----|-----|

The window is shortened when needed: it must fit before tlim and nlim; with self-gravity it never exceeds gravity/solve_dt; and with sinks it is short enough that a sink moves at most about cfl_number * lat_window_motion_cells finest cells.

AMR regridding, outputs (including restart files), the sink operator and the self-gravity solve happen only when every block is at the same time, which means at a window boundary.

When a block updates, its neighbour may be partway through a longer step. The neighbour then supplies ghost values interpolated in time between its own start and end states. The interpolation is linear, or a second-order polynomial built from the Runge-Kutta stages when that is available. A fast block therefore always sees ghost data at the right time.

Each block accumulates the time integral of the flux through its own boundary and subtracts what finer or faster neighbours contributed. When a slow block finishes its step, the difference is added back as a surface correction. This is Berger-Colella refluxing, generalised to time-step bins, including same-level factor mismatches.

LAT changes the load-balance goal from equal cost per rank to work on every rank in every tick. Blocks are ordered so that each rank owns a slice of every bin. On SMR and AMR meshes this ordering is applied at every regrid. With more than one rank, a rebalance at a window end applies it as well, a few windows after startup or after a regrid, and only if the new layout is predicted to improve the window time or the worst tick by at least 5%. A uniform mesh keeps the ordinary Z-order layout. The default minimum bin population scales with the rank count, so the rank count also affects bin membership (see Reproducibility across rank counts).

A refused module makes the run stop with an error message.

Only the rk1 integrator and the rk2-equivalent tableaux (rk2, imex2) are allowed. All other integrators, including rk3, rk4, imex2+ and imex3, are refused.

With the default hydro_lat_min_bin_count = -1 the minimum bin population is 4*nranks, so bin membership depends on the number of ranks. A restart of the same state on a different rank count is still conservative and valid, but it follows a different trajectory. Set the key explicitly to reproduce a run on a different number of ranks. Bin assignment is not stored in checkpoints. It is recomputed from the local time-step limits at the first step after a restart.

Regrids happen only at window ends, and only when at least mesh_refinement/ncycle_check cycles have passed since the previous check. A window is up to 2^lat_levels cycles long, so a smaller ncycle_check simply means a check at every window end.

Set gravity/solve_dt. The window never exceeds this value. When a solve is due, or the window would run past the next solve time, the potential is solved at the start of the window. Shortening the window to land on the due time would instead produce a long descent of ever shorter windows. See Multigrid Self-Gravity for the solver itself.

Blocks near a sink are forced to factor 1, and the sink operator runs once per window. A fast sink limits the window length, because the window must be short enough that the sink moves only a little. If that keeps the windows very short, raise lat_window_motion_cells within the range given in the parameter table.

• LAT needs a single global restart file. Restarting from per-rank restart files (single_file_per_rank = true in the rst output) with lat = true is fatal.
• Outputs and restart files are written only at window ends. An output that falls due in the middle of a window is written at the end of that window.
• A restart does not reuse the old bins, as explained in Reproducibility across rank counts.

• Set lat_diagnostics = true to print the bin table at each stage of the factor selection.
• Each time the LAT partition is built on more than one rank with a largest factor above 1, rank 0 prints one line, Mesh: HD LAT per-bin rank distribution .... It lists how many blocks of each bin land on each rank, and the predicted busy fraction of each rank (util%). Compare it with the measured GPU utilisation.
• Driver: LAT union stage-1 predictor enabled/disabled (...) names the reason when the fused predictor is off.

The remaining cost is mainly communication and task-list launch overhead for the boundary refresh in every tick, which grows with the window length. The design favours correctness and synchronized AMR and gravity over asynchronous per-rank progress. On a static mesh the rank boundaries are fixed to nearly equal block counts, so the order of the blocks is the only thing the LAT layout can change.

• The default bin floor makes the trajectory depend on the rank count (see Reproducibility across rank counts).
• Same-level mixed bins (lat_same_level = true) are a physics-validation risk at large factor ratios, because the interpolation is a dense Runge-Kutta polynomial between the start and end conserved states.
• A window cannot be cut short. The driver shortens windows beforehand so that no limit (tlim, nlim, wall clock) falls inside one, and turns any attempt to leave a window early into a fatal error rather than a silent loss of conservation.

• Example deck: inputs/TDE_examples/tde_05_fallback_lat.athinput. It runs the remap, the settle steps and the fallback in one run: the settle steps run without LAT and LAT starts after the last remap pass.
• Berger and Colella (1989), J. Comput. Phys. 82, 64: the refluxing generalised here to bin boundaries.
• Gottlieb (2009): the SSPRK(2,2) tableau whose dense output is used for mixed-cadence ghosts.