ChaNGa Options - N-BodyShop/changa GitHub Wiki

Here you can find a description of all the parameters supported by ChaNGa. This list is probably incomplete, so please update this page if you notice something missing.

Normally parameters are set by a keyword in the parameter file. Note that these keywords follow a Hungarian Notation style where the first few characters indicate the type of variable. Parameters can be set (or overwritten) by command line switches as well.

Table of Contents Parallel Performance Options Boundary condition options Cosmology options Timestepping options Force accuracy options Smooth Particle Hydrodynamics options SPH Time stepping Input/Output options Checkpoint/Restart options Movie Making options Model Specific Options Grow Mass Options Gasoline Compatibility Options

One of the big differences between PKDGRAV and ChaNGa is its use of the CHARM load balancing system for parallel performance. The CHARM strategy is overdecomposition, i.e. divide the work into many more pieces than you have processors and let the runtime system load balance by appropriately assigning pieces to real processors. Options that control this process are the following:

• nTreePieces: the total number of pieces into which the computation should be split. A rule of thumb is that CHARM needs 8 to 16 times more pieces than processors to do reasonable load balancing, e.g. 512 pieces for a 64 processor run. However, fewer than 1000 particles per piece will result in too much bookkeeping overhead for good performance. The command line version of this is -p <Npieces>.
• nDomainDecompose: strategy for breaking computation into pieces. The command line version of this is -D <nDomain>. Options are
  • 0: Space Filling Curve (SFC). Morton order the particles and split evenly among pieces.
  • 1: Oct tree. Divide along cubical boundaries.
  • 2: Orthogonal Recursive Bisection (ORB). (DEPRECATED. It gave poor performance.)
  • 3: Peano-Hilbert Space Filling Curve. Otherwise similar to 0.

ORB is generally a bad choice. Ironically this is the method PKDGRAV uses. SFC generates more communication but is easier to load balance. Oct might need more pieces for good load balancing.

• dFracNoDomainDecomp : fraction of active particles needed for redetermining domain decomposition boundaries. The default is 0, i.e., domain boundaries are determined every step.
• +balancer <balancer>: specify load balancing scheme. See the CHARM manual for all the possibilities. Reasonable choices are below, but among them MultistepLB_notopo is a good choice for a typical simulation.
  • GreedyLB: use the Greedy algorithm to assign pieces to processors. It works OK with Oct domain decomposition, but generates a lot of communication with SFC.
  • OrbLB: Use an ORB tree to assign pieces to processors. It works OK with SFC (-D 0 or 3) domain decomposition. In the case of SFC domain decomposition, the ORB is one dimensional: the curve is recursively bisected at points to equally split the work among the processors, and pieces that are adjacent on the SFC will be on the same processor. It generally needs more pieces per processor than RefineLB to get good performance, but it should have the least communication overhead. Testing with 3M particle simulations indicates that around 500-1000 particles/treepiece gives optimum performance.
  • RefineLB: Move objects away from the most overloaded processors to reach average. It limits the number of objects migrated. Testing with 3M particle simulations indicates that 1000-2000 particles per treepiece gives optimum performance.
  • MultistepLB: Load balance the work on different rungs separately. This is designed to give good performance in the case where there are a few particles on short timesteps. For the large timesteps it falls back to Orb3dLB, and so gives good performance for single stepping as well.
  • MultistepLB_notopo: Like above, but without using any knowledge of the machine topology. Unless you are running on a Bluegene, the machine topology is not very useful, so this option is preferred over the above.
  • Orb3dLB: Maps domains onto processors based on their centroids in 3D space.
• Note that more work needs to be done in this area. New balancers are currently being developed. There are also a number of flags that can fine tune the load balancing. See the load balancer options in the CHARM manual. Useful options include:
  • +LBNoBackground: ignore background loads when making migration decisions.
  • +LBObjOnly: only load balance migratable objects, and ignore others when making migration decisions.
  • +LBPeriod time: Set minimum time between load balance steps. The default is .5 seconds. If gravity calculations are taking less than a second, then this needs to be made shorter. "0.0" is a good value.

The load balancer can significantly impact performance. See Performance-Analysis for further information on understanding what it does.

• bPrefetch: turn on cache prefetching. The command line version of this is +f. ChaNGa will do a preliminary tree walk to preload the cache before doing the gravity calculation. We suggest that this should always be set.
• nCacheDepth: depth of tree to fetch upon a cache miss. I.e., if a node is fetched, fetch its children (with a value of 1) or grandchildren (2) etc. 4 seems to be a good compromise. The command line version of this is -d <depth>.
• nBucket: Maximum number of particles in the leaves of the tree. This is -b in the command line. This is a time space trade-off. Raise this number and you will save space because the tree is smaller, but gravity calculations will be more expensive. Lower this number and gravity calculations will be faster. If the number is too low, gravity can slow down again due to less processor pipelining.
• nYield: allow charm to interrupt computation to prioritize communication. A lower number results in a higher interrupt frequency. This is -y on the command line.
• bConcurrentSph: enable SPH running concurrently with gravity. This takes advantage of the Charm++ run time system to overlap computation with communication latency. The command line option is +consph.

See Performance Analysis to evaluate how these options effect the parallel performance.

• bPeriodic Use periodic boundary conditions. The command line version is +per.
• nReplicas Number of periodic replicas in each direction. The default is 1 for periodic boundary conditions.
• dPeriod Box size for cubical periodic boundary conditions. The command line option is -L. The default is 1.0. Periodic volumes are centered on 0.0.
• dxPeriod, dyPeriod, dzPeriod : Box size in the x, y and z directions, respectively. The command line options are -Lx, -Ly, -Lz. The default is 1.0.
• bEwald Use Ewald expansion for periodic forces. The command line option is +ewald.
• dEwCut Real space cutoff in the Ewald expansion. The command line option is -ewc. The default is 2.6.
• dEwhCut Fourier space cutoff in the Ewald expansion. The command line option is -ewh. The default is 2.8.

• bComove: Use cosmological comoving coordinates for integration. The default is off.
• dHubble0: Hubble constant in system units. The default is 0.0.
• dOmega0: Comoving density of matter in units of the current critical density.
• dLambda: Cosmological constant in units of the current critical density.
• dOmegaRad: Comoving density of radiation in units of the current critical density.
• dOmegab: Comoving density of baryons in units of the current critical density.
• dQuintess: Comoving density of a "Quintessence" field (with equation of state parameter w = -1/2) in units of the current critical density.
• dRedTo: Final redshift to which to run the simulation.

• dDelta : The timestep to use when integrating the equations of motion in the simulation. The command line version of this is -dt. If multistepping is used, this is the largest allowed timestep that a particle can have.
• nSteps : The total number of steps (of size dDelta) to complete. The command line option for this is -n.
• killAt : Stop the simulation after this timestep. The command line option for this is -killat. This is used for benchmarking or testing the initial part of a simulation.
• bStaticTest : don't move the particle between gravity calculations. This is for a static test of performance. The command line option is +st.
• dEta : The eta parameter is a dimensionless prefix that controls the scale of the timestep picking criterion. That is, to pick smaller, more conservative timesteps, use a smaller eta. The command line version of this is -eta.
• bEpsAccStep: This is the default timestep picking criteria. The timestep is proportional to sqrt(eps/a), where eps is the gravitational softening and a is the magnitude of the acceleration.

The timestep this algorithm picks is <math>\eta \sqrt{\epsilon/a}</math> where ε is the particle's softening length and a is the particle's acceleration.

• bGravStep: This is an alternate timestep picking criteria. The timestep this algorithm picks is <math>\eta/\sqrt{G \rho}</math> where <math>\rho</math>; is an estimate of the local density: <math>\rho = max((m_i + m_j) / r_{ij}^3)</math> where m_i is the mass of the ith particle and the maximum is taken over all interactions with that particle.
• iMaxRung: Maximum rung for timestepping. The base timestep (dDelta) is divided into power-of-2 rungs, and this parameter sets the largest power-of-2 into which the base step can be divided. This parameter is ignored and is here for backwards compatibility with PKDGRAV. The compiled in value is 30.

• dTheta : The opening angle that determines when to open a branch in the tree. Smaller dTheta means more nodes will be opened, resulting in a more accurate, slower, calculation. The command line version of this is -theta.
• dTheta2 : Opening angle to use at later redshifts in a cosmological simulation. The command line option is -theta2.
• daSwitchTheta : Expansion factor (1/1+z) at which to switch from dTheta to dTheta2. The command line option is -aswitchTheta. The default is 1/3 (z = 2).
• iOrder : Order of multiple expansion to use. This is ignored and is here for backwards compatibility with PKDGRAV. In ChaNGa, the multiple order is set by the compile time define -DHEXADECAPOLE. Multipole expansions are to quadrapole order if HEXADECAPOLE is not set.
• bDoGravity : Enable gravitational forces. The command line option is -g. The default is enabled.
• dSoft : Set the gravitational softening of all the particles. The command line option is -e. The default is to read in the softenings from the input file.
• bPhysicalSoft : For cosmology simulations, use physical rather than comoving softenings. The command line option is +PhysSoft. This only has an effect if ChaNGa is compiled with the -DCHANGESOFT define.
• dSoftMax : Maximum comoving softening when using the bPhysicalSoft parameter. The command line option is eMax.
• bSoftMaxMul : Specify that dSoftMax is a multiplier rather than an absolute maximum. The command option is +SMM. If set, the maximum softening is softening times dSoftMax. That is, the softening becomes comoving for 1 + z > dSoftMax.

• nSmooth : The number of particles in the SPH smoothing kernel. The command line option is -nsm. The default is 32.
• bDoGas : Enable gas dynamics. The command line option is +gas.
• bFastGas : Enable fast inverse neighbor finding for SPH. The command line option is +Fgas.
• dFracFastGas : Fraction of active particles below which the FastGas algorithm is used. The command line option is -ffg.
• dhMinOverSoft : Minimum smoothing as a fraction of the gravitational softening. Currently this is ignored.
• bViscosityLimiter : Use Balsara limiter on viscosity. The command line option is +vlim. Default is on.
• iViscosityLimiter : Alternative way to set the Balsara limiter. The command line option is -ivlim.
• bGasAdiabatic : Use an adiabatic equation of state. The command line option is +GasAdiabatic.
• bGasIsothermal: Use an isothermal equation of state. The command line option is +GasIsothermal.
• dMsolUnit: Solar masses per system mass unit. The command line option is -msu.
• dKpcUnit: Kiloparsecs per system length unit. The command line option is -kpcu.

The time unit is determined from the above units under the assumption that G = 1. That is the time unit will be <math>\sqrt{1/G \rho}</math>.

• ddHonHLimit : limit on relative change in smoothing length. For the bFastGas algorithm to be efficient, there needs to be a limit on how much the smoothing length can change per timestep. This sets such a limit on | δh | / h. The default is .1. The command line option is -dhonh.
• dConstAlpha : the alpha parameter in the artificial viscosity formula. The default is 1.0. The command line option is -alpha.
• dConstBeta : the beta parameter in the artificial viscosity formula. The default is 2.0. The command line option is -beta.
• dConstGamma : the adiabatic index (ratio of specific heats) for adiabatic gas. The default is 5/3. The command line option is -gamma.
• dMeanMolWeight : the mean molecular weight in amu. The default is 1.0. The command line option is -mmw.
• dGasConst : Boltzmann gas constant in system units. If dMassUnit and dKpcUnit this is calculated from them. Otherwise the default value is 1.0. The command line option is -gcnst.
• dResolveJeans : Apply a pressure floor to prevent the Jeans length from becoming smaller than the resolved length scales. By default the resolution scale is set by the SPH smoothing length. If ChaNGa is compiled with -DJEANSSOFTONLY, the resolution scale will be set by the gravitational softening length. The default value is "0.0" (i.e., not used). A value of "3.06" matches what is done in other codes (see the AGORA paper Kim et al 2016). The command line option is -resjeans.

• bSphStep : enable SPH based timestep criteria. The command line option is +ss.
• bViscosityLimitdt : limit viscosity contribution to Courant condition according to Balsara switch. Default is off. The command line option is +vlim.
• dEtaCourant : Courant number for SPH timestep criteria. The default is 0.4. The command line option is -etaC.
• dEtauDot : limit relative changes in internal energy. This is only applied if the internal energy is decreasing (either adiabatically or by cooling). The default is 0.25. The command line option is -etau.

• iVerbosity : produce verbose diagnostic output. A higher argument increases the verbosity. The command line option is -v.
• bVDetails : equivalent to iVerbosity = 1. The command line option is +vdetails.
• bBenchmark : Benchmark only mode: no output or checkpoints are written. The command line version of this is +bench. Default is off.
• achInFile : The filename of the initial conditions (in tipsy format). The command line version of this is -I.
• bStandard : The output file is writen in "standard" (i.e., "big endian") format. This is in contrast to "little endian" which is default on Intel architectures. The command line version of this is +std. In ChaNGa, this option is ignored and the output is always in "standard" format.
• achOutFile : The prefix of the output filenames. The command live version of this is -o. The actual outputs will be tipsy standard format files, with filenames of this prefix and a suffix of six digits (left zero-padded) of the timestep for that output. The log file is also prepended with this name.
• iOutInterval : This determines how often to write an output of the simulation. The command line version of this is -oi. An output file will be written every iOutInterval timesteps, and will have the name achOutFile.current_timestep_padded_to_six_digits. Output files are in tipsy standard format.
• iLogInterval : This determines how often an entry will be made into the log file, which has the filename achOutFile.log. You can learn about the contents of the log files on Log-File-Contents page. The command line version of this is -ol.
• bDoDensity : Output a file of mass densities after writing a simulation output. The command line version is +den. Note that this differs from densities in gas particle output in that this is total mass, as opposed to gas mass. The default is on.
• bParaRead : Read input in parallel. The default is on.
• bParaWrite : Write output in parallel. The default is on.
• nIOProcessor : number of processors to use for I/O when reading/writing in parallel. By default ChaNGa uses all processors for parallel I/O, but this can overwhelm file systems. A recommended value to use in this case is the number of physical nodes being run on. The command line version of this is -npio.

• iCheckInterval : This determines how often to write checkpoint files. The command line version of this is -oc. A checkpoint is written every iCheckInterval major timesteps. The checkpoint gets written to a directory basename.chk? where "?" alternates between "0" and "1". After the checkpoint completes, a file "lastcheckpoint" is written to indicate the latest clean checkpoint.
• iWallRunTime : Number of wall clock minutes available for running. The command line version of this is -wall. ChaNGa will attempt to limit its run time to less than this number of minutes. When time is running out, it will write a checkpoint and exit.
• +restart directory : this is only a command line option. Note that if this option is given, all other command arguments and the parameter file will be ignored. The calculation will continue from the checkpoint written in directory using all parameters from that checkpoint.

If a file named STOP is created in the directory in which ChaNGa is running, it will write a checkpoint and exit at the end of the currently running major timestep.

• iStartStep : Timestep at which to begin the simulation. The command line version is -nstart. This is useful if restarting from an output rather than a checkpoint. That is, if iStartStep is specified, then the initial conditions that are provided are actually iStartStep/nSteps through the simulation, and only nSteps-iStartStep more steps will be taken.

ChaNGa has the ability to generate movie frames on the fly in PPM format. The interval between frames is controlled by either:

• dDumpFrameTime : simulation time interval between frames.
• dDumpFrameStep : number of timesteps between frames.

If neither of these options are set, the the movie outputs are disabled.

• iDirectory : number of director files, each producing a separate movie. The command line option is -idr
• bLiveViz : enable real-time simulation rendering. See Research:ChaNGa#Visualization for details. The command line option is +liveviz.

The rendering of the frames is controlled by the .director file. See the movie/director.README file in the ChaNGa distribution for an explanation of this file.

The following are options peculiar to specific types of simulations.

Creating an equilibrium galaxy sometimes involves adiabatically growing one component. ChaNGa has the ability to slowly grow (or shrink) the mass of a subset of particles.

• bDynGrowMass : particles which are growing are also moving, as opposed to being locked in place. The command line option is +gdm.
• nGrowMass : the number of particles which are growing. These are the first particles in the input file. The command line option is -gnm.
• dGrowDeltaM : the total growth in mass per particle. The command line option is -gmdm.
• dGrowStartT : simulation time at which to start growing the particle mass. The command line option is -gmst.
• dGrowEndT : simulation time at which to stop growing the particle mass. The command line option is -gmet.

Some options are there so that Gasoline parameter files can be used without (or with very little) modification.

• dExtraStore : This was a performance option in Gasoline. Here it just specifies how much extra space each TreePiece has to store newly created particles (e.g. stars). There is usually no need to change it from its default value of 0.01.
• bKDK : Use the Kick-Drift-Kick leapfrog time integration scheme, see Quinn et al 1997. This is always set in ChaNGa, so the parameter is recognized but ignored.
• bCannonical : For comoving coordinates, use the cannonical momenta conjugate to the comoving coordinates. For ChaNGa, this option is ignored, and the integration is always done in cannonical coordinates.
• bGeometric : use Katz-Hernquist style geometric mean for SPH force averaging. This is recognized but ignored in ChaNGa.
• bBulkViscosity : use bulk viscosity in SPH. This is default off. In ChaNGa it is recognized but ignored.