The pipeline.DemonLAST class is a demon that excute the LAST image processing pipeline when ever new images are available.

• CI - A CalibImages object that contains the dark, flat and other calibration data.

The following properties are the input parameters for getPath() and getFileName()

• ProjectName - Default is 'LAST';

• Node - LAST Node. Default is 1;

• DataDir - Data directory on the computer (e.g., 'data1' | 'data2'). The default is 1.

• CamNumber - Camera number (1|2|3|4). If empty, then use DataDir to establish the path. Default is [].

• BasePath - Base path - e.g., 'last02w/data1/archive/LAST.01.02.02'. If BasePath is updated, the NewPath, CalibPath, FailedPath, and LogPath will be updated. If you want to change these paths you need to modify them after BasePath is modified.

• NewPath - Path for new images. If start with '/' then this is an absolute path, otherwise, this is a relative path that will be added to the BasePath. The default is 'new'.

• CalibPath - Same as Newpath, but for the directory containing the master calibration images. The default is 'calib'.

• FailedPath - Same as Newpath, but for the directory containing the failed images. The default is 'failed'.

• LogPath - Same as Newpath, but for the directory containing the log files. The default is 'log'.

• ObsCoo - Observatory geodetic position [Long(deg), Lat(deg), Height(m)]. Used for calculating day/night time. Default is [35 30 415].

Hidden properties:

Fields formatting

• FormatCounter - Default is '%03d'.
• FormatCCDID - Default is '%03d'.
• FormatCropID - Default is '%03d'.
• FormatVersion - Default is '%03d'.

Some default parameters:

• DefNewPath - Default is 'new'.
• DefCalibPath - Default is 'calib'.
• DefFailedPath - Default is 'failed'.
• DefLogPath - Default is 'log'.

• constructProjectName - Construct the project name of the form LAST.01.02.03
• getMountNumber - Get the mount number from the computer name.
• camNumber2dataDir - Camera number to data dir.
• camNumber2computerSide - Camera number to the computer side (east|west).
• dataDir2cameraNumber - data dir number to camera number and computer side.
• getBasePath - get base path for LAST computers.
• getArchivePath - get archive (proc/raw) images directory from FileNames object.

• getPath - get the base path, computer, data, camera,...
• deleteDayTimeImages - Delete science images taken when the Sun is above the horizon.
• getBasePathWithOutProjName - Get the BasePath without the last directory.
• moveToDestination - Move list of files in the NewPath dir to the destination.
• setPath - set the BasePath and other paths of the pipeline.DemonLAST object.
• writeStatus - Write ready-to-transfer in the '.status' file.
• writeLog - write a log message to the screen and log file.

To run specific tasks - e.g., generate master dark and flat:

D = pipeline.DemonLAST;
D.prepMasterDark
% or
D.prepMasterFlat

Note that the calibration images will be stored in the CI property which contains a CalibImages object.

Alternatively you can load exiting master calibration images using:

D.loadCalib;

You can also load calibration files nearest to some date or JD. If not specified, then the largest JD will be selected:

D.loadCalib('BiasNearJD',[31 1 2023], 'FlatNearJD',2451545);

You can also tell the pipeline to stop working after it is done (i.e., not waiting for more images). Note that default is false.

D.main('StopWhenDone',true);

Its main method waits for new images in the 'new' directory, process them, store the processed images in the /YYYY/MM/DD/proc/#/ dir, and the raw images are copied to: /YYYY/MM/DD/raw/. Calibration images are stored /calib, and failed images in: /failed,

To execute the LAST pipline:

D = pipeline.DemonLAST;
D.main

To modify the working directory/camera, you can use the following options:

D.BasePath = <Base dir>  % e.g., '/last08e/data1/archive/LAST.01.08.01'
D.DataDir  = <1|2>  %    % e.g., 1 for disk1, or 2 for disk2
D.CamNumber = <1|2|3|4|[]>  % If empty, will use DataDir

Using the 'StartJD' and 'EndJD' optional arguments you can specify a range of dates of images to analyze. For example:

D.main('StartJD',[20 4 2023], 'EndJD',[28 4 2023]);

The main function can calculates a new master bias/flat from a newly available calibration file. By default, it will load existing master bias/flat files and use them. If you want to change this behavior and generate and use new calibration files then use:

D.main('RegenCalib',true);

The main function allows the user to control which data products will be saved, which is done using the following arguments:

• SaveEpochProduct - A cell array of products to write for the single image (epoch) data. To save all the data use: {'Image','Mask','Cat','PSF'}. Default is {[],[],'Cat'} (i.e., saving only the Catalog data.
• SaveVisitProduct - A cell array of products to write for the coadd image (visit) data. Default is {'Image','Mask','Cat','PSF'}.
• SaveMergedCat - A logical indicating if to write the visit merged catalog. Default is true.
• SaveMergedMat - A logical indicating if to write the visit merged matrices (hdf5 file). Default is true.
• SaveAsteroids - A logical indicating if to write the visit asteroid search results (mat file). Default is true.

The main function can be told to work on a non-standard directory containing "new" RAW images once it is called with the argument:

• 'NonStandardNew', 'non-standard new dir'

This is useful for specific data reprocessing and debugging. It is recommended that such an off-site run be carried out with ('UpdateStatusFile', true) in order the processing results are not automatically inserted into the database and uploaded to the storage of standard products. If the data to be reprocessed is already in the packed/archived form, the main can be run with ('UnpackRaw', true) and, optionally, ('RepackRaw', true) flags.

The pipeline should be execute as a LINUX service outside of matlab. The following commands can be used for status, start, stop. Note that each LAST computer should run two pipelines named last-pipeline1 and last-pipeline2. Do not use these commands, unless you know what you are doing.

systemctl status last-pipeline1
systemctl start last-pipeline1
systemctl stop last-pipeline1

The pipeline.DemonLAST class provides flexible directory handling for processing images. Below is a guide on modifying key directories:

D = pipeline.DemonLAST;

This initializes the object, which includes properties like BasePath, NewPath, CalibPath, and others.

Updating the BasePath changes where data is stored and processed. Related paths (e.g., NewPath, CalibPath) are updated automatically:

D.BasePath = '/last05e/data1/archive/LAST.01.05.01_re';

To ensure the pipeline uses the standard calibration directory:

D.DefCalibPath = '/last05e/data1/archive/LAST.01.05.01/calib';

Redirect the pipeline to a different directory for raw images:

D.DefNewPath = '/last05e/data1/archive/LAST.01.05.01/newY';

Execute the pipeline, once the directories are configured:

D.main();

Processed data will be saved under the configured BasePath with the structure:

/<NewBasePath>/YYYY/MM/DD/proc/