Pipeline Unit test Framework - EranOfek/AstroPack GitHub Wiki

The AstroPack pipeline unit test suite monitors for differences in key scientific outputs that may indicate either potential degradations or improvements in the pipeline’s performance.

This regression test operates on a fixed set of 20 raw science images and flats, simulating a real LAST observatory night's data. It compares the new pipeline results against a precomputed reference output.

Test Setup

  • Input Directory:
    Contains raw .fits science and flat images under New/, with calibration products (darks only) under calib/.

  • Reference Output Directory:
    Contains the fully processed output from a known-good pipeline version (26 Mar 2025).

  • Running the Pipeline:
    The test runs the full pipeline (pipeline.DemonLAST.main) on the input dataset and generates a new output set.

Tests Performed

1. File Count Test

  • Goal: Ensure the total number of output files matches between the new run and the reference output.
  • Check: Recursive file count comparison.

2. Merged Catalog Tests (MatchedSources HDF5)

  • Source Count Check: Number of detected sources must match.
  • Julian Date (JD) Check: Maximum JD difference must be < 1e-5 days.
  • Flags Check:
    • Total number of FLAGS must match.
    • Sum of all FLAGS must match.
  • Photometry Consistency:
    • Median drift in MAG_PSF must be < 0.05 mag.
  • Missing Values Check:
    • Fraction of NaNs in MAG_PSF must match.
  • Background Estimation Check:
    • Mean background level difference must be < 20 counts.

3. Astrometric Consistency Test

  • Goal: Ensure RA/Dec values are stable.
  • Check:
    • Median RA shift must be < 0.1 arcsec.
    • Median Dec shift must be < 0.1 arcsec.

4. Science Catalog Tests (Individual CropID FITS)

  • Centroid Consistency:
    • Median XPEAK - X1 offset must be < 0.1 pixels.
    • Median YPEAK - Y1 offset must be < 0.1 pixels.
    • Less than 10% of sources should have PSF-fit positions deviating > 0.5 pixels from peak positions.
  • Background Noise Consistency:
    • The relative difference between VAR_IM (variance) and BACK_IM (mean background) must have a median < 20%.

Data Requirements

  • Input Stack Size:

    • Raw images: ~19 GB
    • Reference processed output: ~25 GB

  • Directory Structure:

Notes:

  • Only one CropID (default 16) is fully compared in merged catalog tests.
  • SNR distribution comparison and multi-CropID expansion are planned future improvements.
  • The raw and reference stacks should eventually be downloadable through the AstroPack Installer.
  • The unit test must be updated to use path-agnostic directory resolution (no hard-coded user/computer-specific paths).
  • This test ensures stable calibration, photometry, astrometry, and noise properties across AstroPack versions.

Running the Unit Test

From MATLAB:

runtests('test_pipeline_regression', 'Verbose', true)

___