Tasks - EranOfek/AstroPack GitHub Wiki

AstroPack pipeline-related tasks

List of tasks related to AstroPack and the LAST/ULTRASAT pipeline. An old version of this document is available here.

Pipeline

In the following, we list the major components of the LAST/ULTRASAT pipeline and the tasks needed for the completion of each component.

DB

Priority : Medium Lead : Sasha Involved : Chen, Arie QA : ? Status : DEsign is required.

Ready:

  • imProc.header.headers2table - convert headers to table, including operators and column name modifications.
  • imUtil.ccdsec.ccdsecStr2num - convert CCDSEC strings to numbers.
  • tools.bit.bitEncode - Encode values as bits in integer (mex).

Tools:

  • db.util.createUniqueKey

Classes:

  • class: db.Db
    • properties
      • DbType - 'postgress'|'clickhouse'
      • DbName
      • User - can be a function handle that brings user/pass
      • Pass
    • methods:

      • open(self, Args.DbName, Args.User, Args.Pass)

      • close(self)

      • insert(self, Data=table|matrix, TableName, Args.Method='single'|'bulk', Args.ColName) The bulk insert method should include mandatory filtering by the names of the columns actually existing in the particular table, otherwise, it will fail. This can be done by db.AstroDb.getListOfTableColumns followed by imProc.header.writeCSVforBulkInjection .

      • createTable(self, TableName, Args.ColName, Args.ColType, Args.ColPrKey)

      • query(self, Query=string, OutType=table|matrix|AstroCatalog)

      • delete(self, PrKeys)

      • replace(self, PrKeys, ColName, ColValue, ColName, ColValue,...)

  1. Note there is no need to Record, Query, etc. classes at this point.
  2. Note that the inset works only with table/matrix input (no Astro classes) - in the future, we may add another layer.
  3. Note that when table object is used chars should be specified as string array and not as cell arrays.
classdef db.Db < Component

properties
   DbType   % 'postgress'|'clickhouse'
   DbName
   User
   Password = [];  % if empty, then use the PasswordsManager class to find and populate the password
   Host
   Port = 9000;
end

methods
   function insertClickHouseBulk(TableName, Data, Args)
       % Insert table to DB using bulk insertion
       %    Data is either CSV file name or table object.
       %    Steps:
       %    Write the table to CSV file
       %    Insert CSV to DB

   end

   function insertPostGressBulk(TableName, Data, Args)
   end

   function insertClickHouseKeyVal(TableName, Data)
       % Insert data to DB line by line
       %    Data is a table object
       
   end

   function insertPostGressKeyVal(TableName, Data)
   end

   function queryClickHouse(SelectString, Args)
       % Execute select
       % Output is a table object

   end

   function queryPostGress(SelectString, Args)
   end

   function deleteClickHouse(SelectString, Args)
   
   end

   function deletePostGress(SelectString, Args)
   end

   function updateClickHouse(SelectString, Key, Val, Args)

   end

   function updatePostGress(SelectString, Key, Val, Args)
   end

end

Image subtraction

Priority : High Lead : ? Ruslan Involved : Ruslan QA : ? Status : Mostly done

  • Debug existing code: imProc.sub.properSubtraction -> always ongoing
  • Translient - test -> always ongoing
  • Safely remove astrometric noise from Scorr statistic, replace through Z2 statistic
  • Revisit source noises calculation for Scorr statistic
  • Undersampled subtraction I: using multiple pixel-phase PSFs
  • Undersampled subtraction II: Matan’s algorithm.

Transients detection in image subtraction

Priority : Medium Lead : Ruslan Involved : ? QA : ? Status : First version is running

  • Iterate based on needs
  • Consider upgrading Results struct to a dedicated object if structure complexity increases

PSF photometry

Priority : High Lead : Sasha Involved : Eran QA : ? Status : Current version in imProc.sources.psfFitPhot - Need a multi-iteration version and improvement in the fitting.

  • Compare two psf fitting algorithms: imUtil.sources.psfPhotCube (old) with imUtil.psf.psfPhot (new) which uses: imUtil.psf.psfChi2_RangeIter
  • Better/robust convergence - New code in: imUtil.psf.psfChi2_RangeIter, imUtil.psf.psfChi2 - Need to test.
  • Multi-iteration PSF fitting
  • Faster
  • Need to write imUtil.psf.psfFit (a new version of imUtil.source.psfFitPhot) using psfChi2_RangeIter...
  • Optional moment estimation after psf phot.
  • End-to-End code for source finding and measuring - Template in: imProc.sources.mextractor

New background scheme

  • suggestion: Global mode followed by surface fit in which the global mode constraining the surface locus.

Gridded reference images

Priority : Medium Lead : Nora & Eran Involved : QA : ? Status : Mostly done.

  • Functionality in the FileNames class to locate reference images by ID or field-coordinate string. - Finished.
  • In FileNames - I added a property named: BasePathRef - this should be used as the base locator.
  • DemonLAST/getRefImage - Works for single images so far.
  • Approach for Reference image versions. E.g., using the file name version. - Not yet implemented.
  • works only if coordinate string or correct field id in file name

PSF estimation

Priority : Medium Lead : Eran Involved : Sasha QA : Sasha Status : The new version in imProc.psf.populatePSF replaced the old one in most of the code

Background and variance estimation

Priority : Medium Lead : Eran? Involved : Enrico QA : Status : Mostly done with the new function imUtil.background.modeVar_LogHist

  • back/var for subtraction and detection should be different.
  • Better treatment of high-frequency variations
  • Faster algorithms
  • Multiple background estimation while doing multi-iteration PSF fitting

Construct PSF from undersampled images

Priority : Medium Lead : ? Involved : QA : ? Status : Not started

  • Need to derive the equations for the solution.

Astrometry

Priority : Medium Lead : ? Involved : ? QA : ? Status : Current code in imProc.astrometry

  • Explore why sometimes the algorithms do not converge
  • Explore new algorithms

Selecting reference images using the DB

Priority : Medium Lead : Nora Involved : ? QA : ? Status : not implemented, but workaround existing

  • use DB to select references based on image quality and other criteria
  • so far used a python script that generates a bash script to copy refs, many currents references have bad quality (wind shake!)
  • need to deal with different versions

Approach and functionality for ungridded reference images

Priority : Medium Lead : ? Involved : ? QA : ? Status :

  • A scheme to manage reference images: Define grid of reference images (size, overlap). What to interpolate? How to combine subimages? etc.
  • Management
  • Interpolation/stitching tools
  • Faster interpolation
  • Lanczos interpolation

Image registration

Priority : Medium Lead : ? Involved : Enrico QA : ? Status : Existing code in: imProc.transIm.imwarp (some issues), imProc.transIm.interp2wcs

  • Missing: treatment for undersampled images.
  • Requires interpolating to the nearest whole pixel. Sending the images + deltas to undersampled coaddition.

Image coaddition

Priority : Medium Lead : ? Involved : ? QA : ? Status : Current version in imProc.stack.coadd

  • Verify/debug weight options in existing coadd code
  • Robustness to CR
  • Proper coaddition code (basics is in: imUtil.properCoadd.combine_proper). Need a partitioning scheme (operating on small blacks and stitch) + CR treatment
  • Undersampled image coaddition

Forced photometry

Priority : Medium Lead : ? Involved : ? QA : ? Status :

  • Forced photometry on subtraction images (Generalization of Nimrod's code)
  • Forced photometry on a predefined list as part of the pipeline.
  • Adaptive forced photometry for moving sources - refine source trajectory before forced phot.

Relative photometry

Priority : Medium Lead : Involved : Yarin QA : ? Status : Existing code can be improved. Many tools in MatchedSources

  • test MatchedSources/lsqRelPhot
  • Remove sources with bad flags (use searchFlags, countFlags)
  • Investigate residuals
  • Investigate cross-correlation function
  • photometric calibration of the forced photometry reults

Asteroid detection

Priority : Medium Lead : Eran? Involved : David QA : ? Status : Search in individual epochs of a single visit is operational but not tested carefully.

  • imProc.asteroids.searchAsteroids_pmCat called by pipeline.generic.procMergeCoadd - Test carefully and debug.
  • Asteroid orphan search in visit and among visits - main tool is ready: imUtil.asteroids.pairsMotionMatchKDTree
  • Test report generation and submit report: imUtil.asteroids.generateReportMPC
  • Streak detection

Speed optimization of the lowest-level operations

Priority : Medium Lead : Eran Involved: Dan, Chen QA : Dan Status : in progress

PSF class

Priority : Low Lead : Sasha Involved : QA : Status : Mostly Done

  • Add functionality.

External catalogs

Priority : Low Lead : Involved : Sasha QA : ? Status : Mostly in catsHTM

  • An astrometric/photometric catalog for ULTRASAT (catsHTM version)
  • DECLeS-DR9

Photometric calibration

Priority : Low Lead : Simone Involved : QA : ? Status : Current version in imProc.calib.photometricZP

  • A new approach for measuring the transmission.

Source Detection

Priority : Low Lead : ? Involved : Noam? QA : ? Status : Mostly exist

  • There may be a bug when running detection simultaneously with multiple PSFs. Need to test.

Image Masking

Priority : Low/On-demand Lead : Eran Involved : Enrico QA : Status : Many implemented (see BitDictionary, MaskImages, imProc.mask)

  • Need to add some additional functionality (e.g., the low/high gain bit, ghosts, spikes):
  • imUtil.image.doubleGain2singleGain
  • imUtil.image.singleGain2doubleGain
  • imUtil.imProc.lowHighGainUnify
  • imUtil.filter.findDiffractionSpikes
  • imUtil.mask.maskDiffractionSpikes

Master corrections (bias, flat, gain, etc.)

Priority : Low Lead : ? Involved : Enrico QA : Status : Mostly done (in CalibImages, imProc.dark, imProc.flat)

  • Test the options to identify problematic bias/dark images.
  • Test the evolution of bad pixels masks.
  • Build a dark/bias management system - e.g., how to select the right dark/flat image based on Mode/ExpTime/etc.
  • Calib images DB
  • Extrapolating Dark images to different exp times.
  • Test the options to identify problematic flats.
  • Removal of stars from flat images
  • A function for star (low frequency) flat field.
  • Tools to measure gain (in imProc.instChar) - map of per pixel values (at least, for USat)
  • Non-linearity - implemented in CalibImages class, but not activated.
  • Fringing correction

Pipeline tools

Priority : Low Lead : Eran Involved : QA : ? Status : Current functionality in pipeline.generic

  • More functionality
  • Memory efficient version
  • Robustness to individual bad images in sequence

Pipeline Managers

Priority : Low Lead : Eran Involved : QA : ? Status : Current version in pipeline.DemonLAST

  • Cleaning code
  • More options

High level functionality

Priority : On-demand Lead : Involved : QA : ? Status : Some exist

  • DS9analysis - A new class for analysis in the ds9 environment. Add features.

General functionality