Flatiron Quickstart - N-BodyShop/changa GitHub Wiki

Talk slides (PDF) ChaNGAGasolineFlatiron2022.pdf

This is a Quick Start guide to support the Gasoline/ChaNGa tutoral on June 28, 2022 at the NBSE workshop at the Flatiron Institute. It assumes that you are running on the VM supplied and documented by Phil Chang.

Obtaining and Building Charm++

ChaNGa is built on the Charm++ parallel library and runtime system. Version 7.0.0 is assumed for this workshop. It can be downloaded directly from the Charm group's source archive using the Git version control system.

Check out the latest development version of Charm++ from the repository:

    $ git clone https://github.com/UIUC-PPL/charm

This will create a directory named charm. Move to this directory:

$ cd charm

To obtain the current stable release, 7.0.0, checkout the v7.0.0 tag:

$ git checkout v7.0.0

And now build Charm with the netlrts-linux machine layer:

$ ./build ChaNGa netlrts-linux-x86_64 --with-production

Note: the ''ChaNGa'' argument in the above build script is an instruction to build all the charm++ libraries on which ChaNGa depends. ChaNGa itself is built in the next step. C++ is compute intensive to compile; appending a ''-jN'' where N is the number of cores on your system can significantly speed up the build process by compiling in parallel.

Obtaining and Building ChaNGa

ChaNGa itself can also be downloaded from the N-Body Shop source archive using ''git''. ChaNGa depends on a utility module which also needs to be downloaded.

git clone https://github.com/N-BodyShop/changa.git
git clone https://github.com/N-BodyShop/utility.git

This will create a directories named changa and utility. Move to the changa directory:

$ cd changa

Now configure ChaNGa for the science you are simulating. ChaNGa is always able to do simpler cases (e.g. dark matter only cosmology) based on the .param (parameter file) used to start the job. We will additionally enable the galaxy formation code that would be suitable for cosmological galaxies and/or the AGORA isolated galaxy disk and use single executable for all tests. We use GASOLINE2 SPH (hydrodynamics): Geometric Density Force, Diffusion, dt Adjusting and the Wendland kernel, the metal cooling module and superbubble feedback (note that Geometric Densisty Forces, Diffusion, and dt Adjusting are enabled by default, you do not need to use --enable-diffusion --enable-dtadjust --enable-gdforce )

./configure --enable-cooling=metal --enable-superbubble --enable-wendland 

A summary of the compile flags will be displayed when configure is finished. Look out for ''WARNING''s which usually indicate a mis-typed option. Now compile:

make

Again, a ''-j'' option will rung the compile in parallel. The result will be two executables: ''charmrun'' and ''ChaNGa''. ''charmrun'' is a helper program to start all parallel instances of ''ChaNGa'' and start them communicating with each other.

Basic Tests

The changa source directory contains a number of basic tests. These include a Sod shock tube, ''testshock'', a King model, ''teststep'', an Evrard collapse, ''testcollapse'' and a low resolution cosmological volume, ''testcosmo''.

As an example, run the ''teststep'' King model by changing directories.

cd teststep

and running changa in parallel, in this example on 4 processors.

../charmrun +p 4 ++local ../ChaNGa test_pg.param

The ''++local'' flag tells charmrun to run all 4 processes on the local machine. Otherwise it will attempt to ssh to machines, including the localhost, to start the parallel processes.

cd ../testcosmo
../charmrun +p 4 ++local ../ChaNGa cube300.param

Basic Visualization: Building Tipsy

Tipsy is a program that can be used to quickly visualize ChaNGa/Gasoline output. To build it, additional development libraries are needed. Install them with the following:

sudo apt-get install libx11-dev
sudo apt-get install libxt-dev
sudo apt-get install libxaw7-dev
sudo apt-get install libncurses5-dev

Obtain Tipsy using ''git'':

git clone https://github.com/N-BodyShop/tipsy.git

Move to the tipsy code directory:

cd tipsy/code

Configure and make:

./configure; make

This will build an executable ''tipsy''. Move this executable to where you can easily find it. To quickly visualize the ''teststep'' I.C.s:

$ tipsy
openb king_soft.bin 
loads 0
zall

Use the left mouse button to zoom in and the center mouse button to zoom out.

Hydro tests

git clone https://github.com/N-BodyShop/flatiron-tests.git

Suggested first test is evrardcollapse. cd into the evrardcollapse directory and run using the ChaNGA executable you built previously:

cd evrardcollapse
path/charmrun +p 4 ++local path/ChaNGa evrardcollapse >& screen &

Note that path is the path to wherever you put ChaNGa.

Using the screen output option saves the output for later viewing. This is typically done automatically on supercomputer systems.

You can see a radial profile of the collapse final output (step 30) using

python plot_collapse.py

If you have pynbody successfully installed. Typically this can be added to your python3 install with

pip install pynbody

You can also look at this output with tipsy and spin it around.

Analysis of Tipsy format outputs from ChaNGa/Gasoline

A quick tool to examine outputs at the command line is printtipsy which is provided in the flatiron-tests repository. It can be compiled with

cd printtipsy
gcc printtipsy.c -lm -o printtipsy

printtipsy autodetects the format and provides a brief summary of the file contents. It is useful for checking ICs (e.g. ones you make yourself).

pytipsy is a module that reads and writes tipsy format files (and autodetects the format). It is useful for making ICs yourself. If you have pytipsy.py (also provided in the above reposity) in your PYTHONPATH (e.g. the current directory or a place you keep modules) it can be used automatically. You can try it out with the cosmo16 example. You will need to have python working to do this.

cd cosmo16
python make16cubed.py

This creates a 16x16x16 particle dark matter plus gas IC in tipsy format that obeys the units rules for Gasoline/ChaNGA. It is just a simple grid created in python and saved using pytipsy.

Have a look at it with printtipsy

printtipsy -prop cosmo16cubed.std

printtipsy -g -prop cosmo16cubed.std

printtipsy -d -prop cosmo16cubed.std

The total mass in the box is Omega matter following the units rules. The -g gives the gas mass (Omega_gas) and -d the dark mass (Omega_DM). Since we did not perturb the positions no structure develops so this box evolves like a piece of the IGM when simulated.

path/charmrun +p 4 ++local path/ChaNGa cosmo16.param >& screen &

This parameter file is set-up to run cosmo16cubed.std for the age of the universe with 1024 steps. The gas heats and ionizes in response to the cosmic UV background (Haardt + Madau 2012). You can view the behaviour with a python script:

python plot_cosmo16.py

Note that the param file specifies outputs every 16 steps (~ 200 Myr) which is a long time at high redshift. Specifying a .red file with the same name as the output files (AchOutName = cosmo16) like cosmo16.red allows you to indicate redshifts you also want outputs for. ChaNGa/Gasoline will make a best effort to use outputs close to these redshifts. They are also listed in the .log file at runtime.

Other Hydro tests

sod_shocktube ---- relaxed glass sod shock tube (cf. Gasoline2 paper and others)

glass16 ---- example script to make glass ICs

blob ---- test3 of Wengen (Agertz et al. 2007) cold blob in wind tunnel (low res)

AGORA ---- Low resolution AGORA galaxy IC with step 100 (100 Myr) output (Kim et al. 2016)

The blob and the AGORA test are more expensive to run (on a laptop at least, not really on any decent multi-core system). Outputs are provided that you can look at with tipsy or otherwise. The AGORA output can be compared to the results from Kim et al. 2016 which use an ineffective feedback (Thermal energy dumps or no feedback).

You can look at step 100 from AGORA with

python plot_galaxy.py

Note that for superbubble, we plot 'Tinc' not 'temp' to see the average temperature of the hot and cold phases. 'temp' is just the cold phase temperature for two phase particles (or temperature for single phase). See the colormap in the first figure. The galaxy is only partially relaxed at this time after an initial burst of star formation (see second figure). Note that the last point in the histogram is low due to binning.

⚠️ **GitHub.com Fallback** ⚠️