HEASoft Notes - RobertRosca/MPhys-Project-Work GitHub Wiki

HEASoft-Related Notes

When downloading the HEASoft packages make sure to tick Caltools and Futils as well as NuSTAR (or just tick the whole general use section, file around 2GB in size) under the Mission-Specific Tools heading

Setup

Sometimes the ./configure command fails to run on all required folders, may have to manually go to a folder and run ./configure there, then go back to the main heasoft/BUILD_DIR folder and run ./configure again

Added line to headas-init.sh to set HEAEDAS directory on run:

export HEADAS='/home/robert/Documents/heasoft-6.22.1/x86_64-unknown-linux-gnu-libc2.23'

Added following to ./bash_aliases:

alias heainit='echo "Running from ~/.bash_aliases" ; source /home/robert/Documents/heasoft-6.22.1/x86_64-unknown-linux-gnu-libc2.23/headas-init.sh

alias caldbinit='echo "Running from ~/.bash_aliases" ; source /home/robert/Documents/nustar/caldb/software/tools/caldbinit.sh'

Using source instead of export to get variables into correct sub-shell

Data Sources

NuSTAR data FTP:

ftp://heasarc.gsfc.nasa.gov/nustar/

Caldb FTP:

ftp://heasarc.gsfc.nasa.gov/caldb/data/nustar

Data Processing

Introduction

Three main processing stages:

Data Calibration: FITS formatted data (raw downloaded data) processed into calibrated data
Data Screening: Calibrated data filtered using various methods depending on instrument
Products Extraction: Use calibrated, filtered data for research

Effectively leads to three (ish) 'levels' of data:

Level 0: Raw Telemetry packets;

Level 1: Telemetry formatted in FITS format;

Level 1a: Calibrated Events Files;

Level 2: Cleaned and Calibrated Event Files;

Level 3: High-level Scientific Products (e.g. light-curves, energy spectra, sky images)

The folder codes (e.g. nu10012001002) aren't random or just date-based, format is nuObservationID [M][xx]_[ll].ss

Every NuSTAR observation is assigned a unique 11 digit number of the form CPPttxxxvvv that is a combination of an 8 digit target identification number and a 3 digit visit number, where:

C is the "source" category defined as:

  0 Non-pointing data (e.g. IOC) or safe hold
  1 Calibration observations, e.g. Crab nebula
  2 Solar system objects (e.g. the Sun)
  3 Galactic compact sources (stars, CVs, X-ray binaries, isolated 
    neutron stars)
  4 Non-ToO Supernovae, Supernova remnants, and Galactic diffuse emission
  5 Normal galaxies
  6 Active galaxies and quasars
  7 Galaxy clusters and extragalactic diffuse objects
  8 Proposed ToO's and Directors Discretionary Time
  9 Non-proposal ToO's

PP is two digits used to identify the program type as:

  00 is assigned to the first 2 year primary mission (2012 to 2014)
  01, 02 etc increments for each additional year of operation 
     (synchronized with possible GO time)

tt is the program type within that PP. The value is reset at each PP. The tt values are the following:

   01 single observation of an object.
   02 multiple observation (monitoring) of the same object.
   10-59 reserved to tiling/mosaic/raster scan programs. These programs 
         include several different targets (pointings) not at the same 
     sky position but are very close.       
  60-99 reserved to survey programs. These programs comprise observations 
        of several objects identified to carry out a specific science 
    investigation.

xxx is the target number unique for a given C and PP.

vvv is the observation visit number for a given target. Multiple 
    observations may be planned by the observer or due to operational 
scheduling requirements. The starting value is 001.

The bit after the observation ID is for:

where:

'M' is a one character string that identifies the focal Plane Module (A or B);
'xx' is a code to identify the observing mode as defined below:
- 01 (SCIENCE): normal observing scientific mode;
- 02 (OCCULTATION): Earth in the field of view;
- 03 (SLEW): data taken during a spacecraft slew;
- 04 (SAA): South Atlantic Anomaly passages;
- 05 (CALIBRATION): on-board calibration radioactive source in the field of view;
- 06 (SCIENCE_SC): attitude reconstruction from the spacecraft bus star trackers;
'll' indicates for event files the processing level ('uf' for Level 1 and Level 1a files, 'cl' for Level 2 files).
For other data files it describes their content (e.g. 'met' for raw metrology data, 'mast' for mast aspect
solution file, 'ex' for exposure maps);
'ss' is the file extension and indicates the data type (e.g. 'evt' for event files, 'img' for sky images, 'hk'
for housekeeping files, 'lc' for light-curves, 'pha' for energy spectra)

Past this the full data and file structure isn't too important

Data Calibration

Setting up the calibration software is described here: https://heasarc.gsfc.nasa.gov/docs/heasarc/caldb/install.html

Nothing else to add past the instructions

NuSTAR instrument is FPM

Using nupipeline runs all calibration on the raw data sequentially, command is as follows:

nupipeline indir=/data_folder/nuObservationID/ steminputs=nuObservationID outdir=./data_calib_folder

If currently in the nuObservarionID folder, using ${PWD##*/} gives the name of the folder. So if in /Documents/nustar/nustar_archive/80001020002, entering ${PWD##*/} gives 80001020002: command not found

Instead of typing in steminputs=nuObservationID manually, can type in steminputs=nu${PWD##*/}:

nupipeline indir=./ steminputs=nu${PWD##*/} outdir=./out

Full documentation goes into details for how each step works if needed

Data Analysis

Once done the output folder will contain one (or more) nuObservationID.evt file(s), running ds9 nuObservationID.evt opens ds9 to view the image

To view the detection for individual pixels use fdump command to dump the raw data from .fits files as follows:

fdump nuObservatioID /output/file/path/name columns=- rows=-

Where columns=- and rows=- defaults to saving all rows and columns, can put in a number range for rows (rows=1-20 for the first 20 rows) and names of columns (columns="TIME X_DET1" for the time and x-axis detection values) to select specific data

Small tool to convert just the coordinates and time: MPhys-Project-Work/fits_conv_coord.sh

Using the astropy module in Python it's quite easy to view/import .fits files for analysis:

from astropy.io import fits
from astropy.table import Table

hdulist = fits.open(`/path/to/fits')

events = Table(hdulist['events'].data)

hdulist.close()

What happens is:

hdulist loaded from the fits file as astropy.io.fits.hdu.hdulist.HDUList
events selected from the hdulist list: astropy.io.fits.hdu.table.BinTableHDU
Data selected from events: astropy.io.fits.fitsrec.FITS_rec
Data converted from fitsrecord to usable astropy.table.table.Table type
hdulist file closed

There is a Julai FITSIO module, but it throws an error when trying to read these files. Either stick with Python, use Julia importing astropy.io and tables from Python or write up a fits tool for Julia which works with this data

Pandas DataFrames can't handle multi-dimensional columns, so the whole events table can't be converted at once, but the non-multi-dimensional parts can be. Select those parts and then convert to a DataFrame like:

events['TIME', 'X', 'Y', 'DET1X', 'DET1Y', 'DET2X', 'DET2Y'].to_pandas()

Alternatively convert to DataFrames and then to Feather files by:

events['TIME', 'X', 'Y', 'DET1X', 'DET1Y', 'DET2X', 'DET2Y'].to_pandas().to_feather('/path/to/feather/file')

XSELECT

xselect to start, set satellite as NUSTAR, set instrument as FPM

read event nuObservationIDA0N_cl.evt.gz to read the event data in

extract to pull out data, extract ALL to pull out all data

FITS Notes

RA_OBJ - Right Ascension of object, degrees

DEC_OBJ - Declination of object, degrees

RA_NOM - R. A. of nominal aspect point

DEC_NOM - Dec. of nominal aspect point

RA_PNT - R. A. pointing, degrees

DEC_PNT - Dec. pointing, degrees

PA_PNT - Position angle (roll), degrees

EQUINOX - Equinox of celestial coordinate system

RADECSYS - Celestial coordinate system

MJDREFI - MJD reference day

MJDREFF - MJD reference day fraction

TCPRX13 - X image reference pixel

TCRVL13 - X image reference pixel coordinate (degrees)

TCDLT13 - X image scale (degrees/pixel)

TCTYP13 - X image coordinate type

TNULL13 - X null value

TCPRX14 - Y image reference pixel

TCRVL14 - Y image reference pixel coordinate (degrees)

TCDLT14 - Y image scale (degrees/pixel)

TCTYP14 - Y image coordinate type

TNULL14 - Y null value