IGRINS‐2: How to Run Pipeline - igrins/plp GitHub Wiki

Setup

If you are just getting started, see How to Install for how to install the IGRINS-2 Pipeline and How to download and process the raw data from the Gemini Archive for how to download and processs up your raw data.

The IGRINS-2 Pipeline runs in Python and there is no separate compilation process. Once it is cloned or downloaded and you have installed the required Python dependencies, you should be able to run it. You are supposed run everything within the source directory. In the source directory, you should see a file called "igr_pipe.py", which is the main executable to run the pipeline. Try to run it like below, and you should see a similar error message.

> python ./igr_pipe.py 
usage: igr_pipe.py [-h]
                   {flat,thar,sky-wvlsol,a0v-ab,stellar-ab,extended-ab,extended-onoff,plot-spec,publish-html}
                   ...
igr_pipe.py: error: too few arguments

Recipe Logs

The IGRINS-2 Pipeline organizes data by nights which follow the date format YYYYMMDD. To run the pipeline, you need to have a recipe log file for a given night listing observation groups and recipe names to be applied. The recipe log file serves to tell the pipeline what to do for a given night. This is necessary as no recipe information is recorded in the fits header or other logs during the observations. These recipe log files are stored in the directory called "recipe_logs".

> ls recipe_logs
20140524.recipes  20140707.recipes  20140710.recipes  20140713.recipes
20140525.recipes  20140708.recipes  20140711.recipes
20140526.recipes  20140709.recipes  20140712.recipes

You might have been provided a recipe log with your data. You might want to modify it, or if one was not provided need to create your own. See How to prepare recipe logs for instruction to prepare the recipe log file. Here we assume that you already have the recipe log.

The run.sh script

To simplify execution of the IGRINS-2 Pipeline, we provide an example script called "run.sh" in the source directory. For most users, reducing a night of IGRINS-2 data simply consists of running it after you have prepared the night's recipe log. This script is set up so that the PLP runs "python igr_pipe.py" for nearly all possible recipes you might encounter in the correct order with the optimal settings. In some circumstances, you might want to create your own modified version of run.sh, but it should work fine for most use cases.

The script looks like the following:

#!/bin/bash

#Simple script to fully run IGRINS PLP on any UTDATE
USER=Default #Name of user running script, will be passed to FTIS headers
VERSION=igrins-2_v3.1 #Version of PLP being used to run script, will be passed to FTIS headers
SLIT_PROFILE_METHOD=column #Method for determining slit profile for optimal extraction, column=use running median per column across detector (new method, default),  full=use single median profile across entire detector (old method, might be needed for very low SNR targets)


echo "Enter UTDATE:" #Prompt user to enter date
read UTDATE #Get date from user
#Clean up previous files, if any
rm -r outdata/$UTDATE
rm -r calib/secondary/$UTDATE
rm -r calib/primary/$UTDATE
#Run PLP
python ./igr_pipe.py flat $UTDATE #Process flat
python ./igr_pipe.py register-sky $UTDATE #First pass at wavelength solution using sky frame (incase THAR lamps don't exist)
python ./igr_pipe.py flexure-setup $UTDATE #Setup files needed for flexure correction
#python ./igr_pipe.py thar $UTDATE #First pass at wavelength solution using ThAr lamps, looks like current PLP version doesn't even use THAR lamp frames
python ./igr_pipe.py wvlsol-sky $UTDATE #Second pass at wavelength solution using OH sky emission
python ./igr_pipe.py a0v-ab $UTDATE --correct-flexure --height-2dspec=100 --mask-cosmics --user=$USER --version=$VERSION --slit-profile-method=$SLIT_PROFILE_METHOD #Reduce A0V standard star data
python ./igr_pipe.py a0v-onoff $UTDATE --correct-flexure --height-2dspec=100 --mask-cosmics --user=$USER --version=$VERSION --slit-profile-method=$SLIT_PROFILE_METHOD
python ./igr_pipe.py tell-wvsol $UTDATE #Use telluric absorption in A0V std star to fine tune wavelength solution
python ./igr_pipe.py stellar-ab $UTDATE --correct-flexure --height-2dspec=100 --mask-cosmics --user=$USER --version=$VERSION --slit-profile-method=$SLIT_PROFILE_METHOD #Reduce stellar sources nod on slit
python ./igr_pipe.py stellar-onoff $UTDATE --correct-flexure --height-2dspec=100 --mask-cosmics --user=$USER --version=$VERSION --slit-profile-method=$SLIT_PROFILE_METHOD #Reduce stellar sources nod off slit
python ./igr_pipe.py extended-ab $UTDATE --correct-flexure --height-2dspec=100 --mask-cosmics --user=$USER --version=$VERSION #Reduce extended sources nod on slit
python ./igr_pipe.py extended-onoff $UTDATE --correct-flexure --height-2dspec=100 --mask-cosmics --user=$USER --version=$VERSION #Reduce extended sources not off slit
python ./igr_pipe.py divide-a0v $UTDATE --user=$USER --version=$VERSION #Reduce stellar sources nod off slit
#python igr_pipe.py plot-spec $UTDATE --html-output #Make and publish HTML preview
#python igr_pipe.py publish-html $UTDATE #Ditto

echo "Done running"
echo $UTDATE

We will explain each step later.

If this is the first time you have run "run.sh," there are a few things you should do once. You might want to make a copy of run.sh and run the copy.

Change the user name from "Default" to a name you want to use for your own data. This will propagate to the FITS headers in the reduced data products and help track who reduced the data.

USER=Default #Name of user running script, will be passed to FTIS headers

Leave VERSION alone unless you are making custom modifications to the Pipeline. SLIT_PROFILE_METHOD sets whether you want to use a running median of the detector columns or the full detector for determining the slit profile used in optimal extraction for STELLAR recipes. You should probably leave it as the default value.

You will need to give the script executable permissions. You only need to do this once.

> chmod +x run.sh

To run the script to reduce a night of IGRINS-2 data, you simply execute it like so:

> ./run.sh

It will ask you for the date of the night you want to reduce. This date will be in the format YYYYMMDD and correspond to both the date for the raw data in the "indata" directory and the recipe file.

> ./run.sh
Enter UTDATE:
20250123

The script will then reduce the entire night automatically and you just need to wait for it to finish. This can take some time, depending on the number of observations taken in that night and how fast your computer is. You will see a lot of output from the Pipeline telling you what it is doing. When it is done, you will see "DONE RUNNING" with the date.

DONE RUNNING
20250123
>

Steps in pipeline processing

Here we will explain each step in running the pipeline, using the "run.sh" script as our guide.

Options

There are a few options the user can set at the top of the script

USER=Default #Name of user running script, will be passed to FTIS headers
VERSION=igrins-2_v3.1 #Version of PLP being used to run script, will be passed to FTIS headers
SLIT_PROFILE_METHOD=column #Method for determining slit profile for optimal extraction, column=use running median per column across detector (new method, default),  full=use single median profile across entire detector (old method, might be needed for very low SNR targets)

User is the name of who is running the Pipeline that gets propagated to the FITS headers for the reduced data products. Should be changed to whoever is running the Pipeline.
VERSION is the software release version of the Pipeline being run, and is propagated to the FITS headers of the reduced data products. The user should not change this unless they are running a custom modified version of the Pipeline.
SLIT_PROFILE_METHOD tells the Pipeline which slit profile type to use when running optimal extraction for the STELLAR recipes to generate the 1D spectrum (the spec.fits files). The default is "column" where the profile is determined column by column across the detector with a running median. This accounts for variations in the profile across the detector especially near order edges, but requires more signal-to-noise from the target. The other option is "full" where median profile will be constructed across the entire detector. This is the old method, and might be needed if the targets are low signal-to-noise, but does not account for variation in the profile across the detector. If you the continuum flux from order to order not lining up at the order edges, we recommend "column," but if you don't care too much about your flux calibration and are working with low signal-to-noise point source targets, you can try "full" to see if this results in better 1D extractions.

Clean up previous files

The "run.sh" script will start by trying to remove any previous data reduction products for the night you are running with the following:

#Clean up previous files, if any
rm -r outdata/$UTDATE
rm -r calib/secondary/$UTDATE
rm -r calib/primary/$UTDATE

This step is not strictly necessary, as the the PLP will simply overwrite existing files, but exists to clean things up in the "outdata" and "calib" directories before rerunning a night.

Process flat

The flats are required calibrations for a night. Processing flats is the very first step you need to run in the pipeline to start reducing a night. They are processed in run.sh using:

> python igr_pipe.py flat $UTDATE #Process flat

Register sky

Registering the sky is the first iteration in the wavelength calibration process and is a required step after the processing the flats. It uses the OH lines in the SKY frame to make a first guess at the wavelength solution:

> python ./igr_pipe.py register-sky $UTDATE #First pass at wavelength solution using sky frame (incase THAR lamps don't exist)

Setup for flexure correction

If you are going to use the flexure correction, which is called for the various recipes with the command line argument "--correct-flexure", you need to run the setup script to make sure the necessary files are generated. You only need to run this step if you are going to use the flexure correction. This is done as follows:

> python ./igr_pipe.py flexure-setup $UTDATE #Setup files needed for flexure correction

Get wavelength solution

This runs the second iteration on the wavelength solution using the OH sky emission lines in the SKY frame, using the solution generated by Register as a first guess. This is a required step.

> python ./igr_pipe.py wvlsol-sky $UTDATE #Second pass at wavelength solution using OH sky emission

Process standard stars

This step reduces the A0V standard star observations used for telluric correction and flux calibration. Generally each science target will have an associated standard star at a similar airmass. These are normally observed nodded on the slit but can be observed nodded off the slit. The PLP will treat all these as stellar sources and use optimal extraction for the 1D spectra.

python ./igr_pipe.py a0v-ab $UTDATE --correct-flexure --height-2dspec=100 --mask-cosmics --user=$USER --version=$VERSION --slit-profile-method=$SLIT_PROFILE_METHOD #Reduce A0V standard star data
python ./igr_pipe.py a0v-onoff $UTDATE --correct-flexure --height-2dspec=100 --mask-cosmics --user=$USER --version=$VERSION --slit-profile-method=$SLIT_PROFILE_METHOD

Process science targets

These recipes will reduce the various science targets. These include stellar (point) and extended (nebular) sources, both nodded on slit (ab) and off slit (onoff).

python ./igr_pipe.py stellar-ab $UTDATE --correct-flexure --height-2dspec=100 --mask-cosmics --user=$USER --version=$VERSION --slit-profile-method=$SLIT_PROFILE_METHOD #Reduce stellar sources nod on slit
python ./igr_pipe.py stellar-onoff $UTDATE --correct-flexure --height-2dspec=100 --mask-cosmics --user=$USER --version=$VERSION --slit-profile-method=$SLIT_PROFILE_METHOD #Reduce stellar sources nod off slit
python ./igr_pipe.py extended-ab $UTDATE --correct-flexure --height-2dspec=100 --mask-cosmics --user=$USER --version=$VERSION #Reduce extended sources nod on slit
python ./igr_pipe.py extended-onoff $UTDATE --correct-flexure --height-2dspec=100 --mask-cosmics --user=$USER --version=$VERSION #Reduce extended sources not off slit

Standard star and science target recipe options

The recipes for processing standards and science targets, specifically a0v-ab, a0v-onoff, stellar-ab, stellar-onoff, extended-ab, and extended-onoff, have several options that can be set.

--correct-flexure

Must be set if you want to turn on the instrumental flexure correction. This will attempt to shift the individual exposures in pixel space in the x-direction to try to keep the OH sky emission lines aligned using the first exposure in the SKY as a reference frame. We highly recommended you use the flexure correction as it normally reduces "p-cygni" type residuals from sky emission and telluric absorption lines. You need to run the "flexure-setup"

*--mask-cosmics

Turns on cosmic ray correction for each individual exposure using astroscrappy. The parameters used have been specifically tuned for the IGRINS H and K bands to avoid masking bright sky emission lines or the target spectrum itself. It might not catch all cosmic rays, but it will catch many of the bright ones. The PLP will attempt to "fill" masked cosmics with data from different exposures of the same nod on a target. We generally recommend you leave the cosmic ray masking on. If you require more thorough cosmic ray correction, we recommend you run your own cosmic ray correction on the raw data before reducing the data with the IGRINS PLP.

*--height-2dspec=100

Sets the height the PLP will interpolate the reduced rectified 2D spectra to in pixels for the spec2d.fits and var2d.fits files found in the "outdata" directory. This is useful when dealing with extended sources and we recommend you use this option. 100 pixels works well, but other values can be used.

*-g 101

Reduce just a single target (line from the recipe file). The number after -g specifies reduce only a single group1 number specified in the group1 column of the recipe log. This almost always is the first frame number for a target. In this example, this would reduce only the target starting with frame number 101.

*-b H

Reduce just the H or K band (or both). Options are H, K, and HK. By default the PLP will reduce both the H and K bands, so this option is typically not needed.

*--basename-postfix="_A" --frac-slit="0,0.5" and --basename-postfix="_B" --frac-slit="0.5,1"

Reduce just the A or B nod of a target. The option --basename-postfix sets the postfix in the filename of the data reduction products and --frac-slit specifies which fraction of the slit is being reduced. These options could be useful for targets with a bad nod or where you might be worried about variation along the slit, such as for precision RVs.

Old instructions

The instructions below were written for v2 of the IGRINS PLP so might be out of date but are kept here for reference.

The order of pipeline processing is

flat
register-sky (use sky frame to do bootstrapping and an initial wavelength solution)
wvlsol-sky (Sky lines for full wavelength solution)
a0v-ab (extract spectra for A0V telluric standard stars)
stellar-ab, extended-ab, extended-onoff

The processed data will be saved under "calib/primary/20140525" and "outdata/20140525". "calib/primary" will contain files required by other recipes. Under the directory "outdata/20140525/qa", several figures should have been generated. It is supposed to help you with the quality assurance, although it is not as complete and descriptive as it meant to be.

For example, to manually process flat for the night of 20140525,

> python ./igr_pipe.py flat 20140525 -s 64 -b "H"

This will process "H" band data of a group whose first ObsID is 64 using recipe "FLAT".

There may be multiple observation groups with recipe name of "FLAT", and the default behavior is to process all the groups and both "H" and "k" band. For this, you can simply do

> python ./igr_pipe.py flat 20140525

Next is to do "register"

> python ./igr_pipe.py register-sky 20140525 -b "H"

As before, this will only process "H" band data. Remove '-b "H"' to process both "H" and "K" bands data.

Next is Sky.

> python ./igr_pipe.py wvlsol-sky 20140525 -b "H"

Now, we have all the calibration files ready to extract spectra.

> python ./igr_pipe.py a0v-ab 20140525 -b "H" -s 16

This will extract H band spectra of A0V star from a group whose first ObsID is 16 (24 Oph). To extract H- and K- spectra of all A0V stars,

> python ./igr_pipe.py a0v-ab 20140525

It is time to extract spectra of target objects.

> python ./igr_pipe.py stellar-ab 20140525 -b "H" -s 11

This will extract H-band spectra of a group with starting obsID of 11 (V2247Oph). Again, to extract all target spectra (with recipe name STELLAR_AB).

> python ./igr_pipe.py stellar-ab 20140525

Here is some more details on the extraction with available options.

The extracted spectra will be saved under "outdata/20140525" with file name ending with "spec.fits". The file should be compatible with tools like IRAF splot. To quick-look view of the spectrum,

> python ./igr_pipe.py plot-spec 20140525 -b "H" -s 11 -i -m

This will give two matplotlib figures. The first figure shows extracted spectra and its S/N. The second figure (lower panel) is target spectra divided by A0V spectra for quick telluric correction. The upper panel is the spectra of (flattened) A0V spectra for comparison. Note that this is for quick-look only. Without "-i" option, a static figure is saved under the "qa" directory. Supplying "--html-output" option will produce files related to html output under "html/20140525". With "-m" option, a model spectra of Vega is multiplied to the divided spectra.

Similar to "stellar-ab", you can process "extended-ab" and "extended-onoff". The extraction of 1d spectra from extended source is not optimal and needs to be improved. "extended-ab" is for compact extended source which was observed in ABBA mode. "extended-onoff" is for extended source with large extent where a separate off position was observed.

> python ./igr_pipe.py extended-ab 20140525

> python ./igr_pipe.py extended-onoff 20140525

Here ia a small script that go through all the necessary recipes.

UTDATE=20140525
for RECIPE in flat register-sky wvlosl-sky a0v-ab stellar-ab extended-ab extended-onoff
do
 python igr_pipe.py $RECIPE $UTDATE
 rc=$?
 if [ $rc != "0" ]; then
   exit $rc
 fi
done

To produce interactive html page,

UTDATE=20140525
python igr_pipe.py plot-spec $UTDATE --html-output
python igr_pipe.py publish-html $UTDATE

In this case, the last recipe "publish-html" produces the extracted spectra under "html/20140525" as html pages.

To check the spectra, you may need to run a web server under the "html" directory. The easiest way is to use python's SimpleHTTPServer module.

cd html
python -m SimpleHTTPServer

This will run a very simple web server listening to port 8000. Now in your browser, visit "http://localhost:8000/20140525/".

You will see a list of targets observed in 20140525. The links on the right side will give you extracted spectra.

If you have a firefox, you don't need to run a webserver.

> firefox html/20140525/index.html

You may be able to do this with other browser. It depends on the security policy of your browser (with chrome, running this way would not give you correctly rendered page because of the default security policy of chrome.)

When some calibration observation is missing.

The current pipeline requires FLAT, THAR, SKY calibrations for each night. Often, some of these calibration files are missing. The easiest workaround is to make a symlink to existing calibration directory. For example, no calibration files are available for 20140712. You may use calibrations from 20140710. What you have to do link "calib/primary/20140710" to "calib/primary/20140712".

> cd calib/primary
> ln -s 20140710 20140712

You can now be able to extract spectra of 20140712 using recipes like "A0V_AB", "STELLAR_AB", etc.