pipeline‐I overview - EranOfek/AstroPack GitHub Wiki

LAST pipeline-I overview

Main function is: pipeline.last.pipes.pipelineI

Steps

  • Create master dark image + mask

  • Create master flat image + mask

  • Read the entire visit's raw images from disk to memory.

    • Cast image type to single.
    • Flag and remove empty images.
    • Flag and remove images with incorrect size.
    • Estimate the global bcakground and remove images with large background
    • Check that there are not too many pixels with high level
    • Check for image histogram anomalies.
    • Check for a large number of pixels with a fixed value.
    • Estimate the PSF using the ACF.
    • Check for bad PSF.
    • Add additional header keywords.
    • Add file name literals to image header.
    • Add git version to image header.
    • Add raw image ID to header.

  • Basic calibration

    • set saturated pixels bit mask
    • Multiply by the gain
    • Dark subtraction
    • Overscan/darkened pixels subtraction
    • Linearity correction (not applied)
    • Divide by flat image
    • Propagate mask from dark-mask and flat-mask

  • Add Mid JD header keyword

  • Divide the full image to 24 crops (sub-images) with overlap

  • Add Proc image ID to all crops

  • Multi-iteration PSF fitting

    • Measure the image background and variance in 256x256 pixel sub-crops and interpolate to generate a background and variance image.
    • Select bright stars, select good stars (no neighbors, second moment, high S/N), and use them to create a PSF for each crop.
    • The PSF is constructed by using the first moment to measure the stars' positions and the Lanczos3 interpolation to shift all the stars to a common frame.
    • Perform matched filtering with the PSF, a delta function, and the PSF slightly extended by an exponential profile.
    • Select stars with S/N>500, fit their PSF (position and flux), and measure their properties (aper phot, moments, S/N with all filters, annulus background and std, etc.)
    • Subtract the stars from the image
    • Add the subtracted stars to the background and variance images.
    • Only for very bright stars ($>10^5$ photons) add an empirical power-law function to the background and variance images to account for scattered light.
    • Repeat the search, measurments, and subtraction for stars with S/N>50
    • Repeat the search, measurments, and subtraction for stars with S/N>5
    • Write the catalog of sources to the image catalog.
    • Additional steps:
      • Collect all the bit mask values within 3 pix from each source, combine with OR operator, and insert the results into a FLAGS column in the source catalog of the image.
      • Remove cosmic ray / bad pixels based on hypothesis testing against the delta function PSF.
      • Remove sources for which PSF fitting S/N<4.
      • Search for streaks in the images.

  • Astrometry

    • Start with the middle crop of the first epoch, and attempt to solve its astrometry blindly (with rough ~0.5 deg intial position and free rotation)
    • If it fails, try a few additional central crops.
    • Project the solution to the nearest crop and refine the astrometry (i.e., match to known stars and re-solve).
    • Repeat for all the crops in the first epoch.
    • For each crop, and using its astrometric solution, refine the astrometry of all the other epochs.
    • Write WCS (with PV distortions) to header.
    • Add RA, Dec of the sources to the image catalog

  • Update the mask for streaks.

  • Measure the PSF properties and write them to header

    • The fwhm of the PSF
    • Number of local maxima, ratio between two highest local maxima, distance between two highest local maxima
    • Fit the PSF with a 2D elliptical Gaussian and write its parameters.

  • Add Airmass of image to header.

  • Add Healpix ID for each source.

  • Measure background gradients and remove from further processing images with large gradients

  • Write statistics to header:

    • Mean, median, std of Background
    • Number of stars
    • 95 percentil of stellar mag.
    • Median 2nd moments
    • Pixel scale
    • Image rotation
    • PSF $\sqrt{\sum_{i}P_{i}^{2}}$

  • Forced photometry of selected sources blindly

    • For all the stars in ForcedPhotList (a few million) performs blind forced photometry (on the epoh images).
    • Allow the PSF fit to adjust positions in only two iterations and a total shift not exceeding 0.2 pix.
    • Add to the source catalog with FORCED=1 column.

  • Sort the catalog by declination

  • Add the Airmass of each source to the catalog

  • Create a MatchedSource object for each crop (over all epochs)

    • In each crop, over all epochs in the visit, match all the sources within 1.5'' from the position in which they appear in the first image in which they detected.
    • The FORCED and unFORCED sources are matched separately.
    • For selected properties (e.g., X, Y, RA, Dec,...) save the data for all the stars (columns) over all epochs (rows).
    • Use NaN for epochs in which the sources does not appear.
    • Perform relative photometry of all the sources in the visit.
    • Note that the photometry is not calibrated, just relative photometry adjusted!
    • Calculate the rms vs. APER_MAG_3, and store the asymptotic rms in the header of the coadd image (generated next).
    • Calculate the X and Y linear drifts of the matched stars and store them in the header of the coadd image (generated next).

  • For each crop, coadd all the good images in the visit.

    • Register the images by their WCS, using Lanczos3 interpolation.
    • Robust coaddition of the images (min/max=1 + 1 iteration $2.5\sigma$ cipping; with robust std).
    • Repeat the steps doen for the epoch images (e.g., background estimation, source extraction,...).
    • Generate image IDs
    • Add airmass to images
    • Add airmass to individual stars
    • Match sources against known solar system objects (10`` radius).
    • Match sorces against the MergedCat external catalogs

  • Run old photometric calibration to epoch and visit images

    • Fit the stars' magnitudes to GAIA magnitudes with color terms
    • Write results to the headers in the PH_* keywords.
    • Do not adjust the stars' magnitudes.

  • Run photometric calibration by transmission fitting to the visit images

    • Write results to the headers in the PT_* keywords.
    • Calculate aperture corrections and write them to the headers.
    • Adjust the catalog magnitudes in the catalogs based on the calibration and aperture corrections.
    • Assume that the epoch images (of the same crop) have the same parameters while the ZP is adjusted for the relative photometry ZP found in the source matching step. Under this assumption, populate the epoch image headers with the PT_* keywords, aperture corrections, and update the magnitude in the catalogs.

  • Save all the data produced to disk

Data products

  • The following data products are saved to disk
    • Epoch images (per epoch, per crop) / FITS / FITS binary table
      • Catalog of sources
      • Calibrated images (not saved by default).
      • Mask image (not saved by default).
      • PSF image (not saved by default)
    • Visit images (per crop) / FITS / FITS binary table
      • Catalog of sources
      • Calibrated images
      • Mask image
      • PSF image
    • Matched sources object (per crop) / HDF5 file
    • Separate list of known asteroids in visit coadd images (per visit) / mat file
    • Streaks detected in the epoch images (per visit) / mat file