Class Hierarchy: Base -> Component -> AstroFileName

AstroFileName is a container class for file names of images and astronomical data product. This class supports the LAST/ULTRASAT file name convention described in Ofek et al. (2023). The idea is to use the same file name convention for all file types.

This class replaces the old FileNames class.

The file names are unique and they are concatenated from several strings separated by underline "_". The rationale for this convention is that all file names have the same structure and can be analyzed using a single routine that uses simple regular expression commands.

The file name format is therefore:

<ProjName>_YYYYMMDD.HHMMSS.FFF_<filter>_<FieldID>_<counter>_<CCDID>_<CropID>_<type>_<level>.<sublevel>_<product>_<version>.<FileType>

For example: USAT_20210909.123456.789_clear_M31_001_2_12_sci_raw_Image_1.fits

If some sub-string is not relevant to the file, then it appears as an empty string. In this case, two (or more) successive underlines (e.g., "__") will appear in the file name. The strings, by their order of appearance in the file names, are:

• Project Name - Project/telescope name. For LAST we use LAST.<Node>.<Mount>.<Camera>, where Node is the node index (1 for the first node in Neot-Smadar, Israel). Mount for the telescope-mount index (e.g., 1 to 12). Camera, for the camera on mount index (e.g., 1 to 4).
• Time - UTC date and time in format YYYYMMDD.HHMMSS.FFF.
• Filter - Filter name (e.g., `clear').
• Field - The field ID. For LAST the field ID is a string of the format ddd+dd, indicating the RA and Dec in decimal degrees.
• Counter - Image counter. For LAST the image counter is usually 1 to 20, indicating the index of the image in the sequence of 20 exposures in a visit.
• CCDID - Detector ID. For LAST this is always 1.
• CropID - Index of the sub-image (1 to 24). 0 or an empty string is reserved for the full image.
• Type -- One of the following (self-explanatory) image types: bias, dark, flat, domeflat, twflat, skyflat, fringe, focus, sci, wave, test.
• Level -- One of the following strings describing the level of processing:
  • raw - A raw image.
  • proc - A single processed image.
  • caodd - The coadd image of the visit or any other sequence of images.
  • ref - A reference image.
  • calib - A calibration image.
• Product -- One of the following keywords describing the file content:
  • Image - Image data.
  • Back - Background image.
  • Var - Variance image (typically background variance only).
  • Mask - A bit mask image.
  • PSF - A PSF image, or cube.
  • Cat - A Catalog of sources detected in the image.
  • MergedMat - A merged matrices of sources detected in multiple epochs of the same field.
  • Asteroid - A file containing information on asteroids detected in the image.
  • Evt - A photon event file.
  • Spec - A spectrum.
• Version - Processing version.

File type extensions are typically, fits, hdf5, or mat.

Each one of the literals is represented by a property in the AstroFileName object.

The FileNames class is also responsible for generating the directory names in which the files are stored. The file path is constructed from the following sub-directories, described in the following options:

<BasePathRef><FieldID>

<BasePath><ProjName>/new <BasePath><ProjName>/calib <BasePath><ProjName>/failed

<BasePath><ProjName>/yyyy/mm/dd/raw <BasePath><ProjName>/yyyy/mm/dd/proc/<SubDir>

Here <SubDir> and <\ProjName> are optional subdirectories. <SubDir> is either a numerical index or has the format HHMMSSv#. yyyy, mm, dd, HH, MM, SS are the date and time strings.

The construction of the date directory is done by changing the directory at local noon. In order to determine the local noon, the time zone is a property of the FileNames class.

BasePath can include the project name - for example: /last2/data/archive/LAST.01.01.02

The AstroFileName can be an array of objects, and each element may contain multiple file names.

The FileNames class contains the following properties

• ProjName
• Time
• JD
• Filter
• FieldID
• Counter
• CCDID
• CropID
• Type
• Level
• Product
• Version
• SubDir
• BasePath
• BasePathRef
• Path - If not empty, then it will be used instead of the BasePath BasePathIncludeProjName - A logical indicating if the BasePath includes the project name. AddSubDir - A logical indicating if to add the <SubDir> to the path name.
• TimeZone - Default is 2 [hr].

All the properties representing literals are strings. However, their setters can accept numeric values and can convert them to strings.

Note that if 'Path' is provided then it will be used instead of the 'BasePath'. So prior to generating path, you may want to set the value of Path to "".

Additional hidden properties:

• FormatCounter - Default format of the Counter literal - "%03d"
• FormatFieldID - Default format of the FieldID literal - "%d"
• FormatCCDID - Default format of the CCDID literal - "%03d"
• FormatCropID - Default format of the CropID literal - "%03d"
• FormatVersion - Default format of the Version literal - "%d"
• FormatProjName - Default format of the ProjName Sub literals - "%02d"
• FormatTime - Time format - "yyyymmdd.HHMMSS.FFF"
• ListType - List of valid Type.
• ListLevel - List of valid Level.
• ListProduct - List of valid product.
• SEPERATOR - Literal seperator - "_"
• FIELDS - List of file name literals ["ProjName", "Time", "Filter", "FieldID", "Counter", "CCDID", "CropID", "Type", "Level", "Product", "Version", "FileType"];

The constructor can be use to create an empty object, read file names in a directory, or parse file names. A few examples:

A = AstroFileName([2 4]);  % create an empty object with 2x4 elements

% read all files in a directory according to file pattern:
A = AstroFileName('LAST.01.*fits','Path','.')
% cal be also achieved using:
A = AstroFileName.dir('LAST.01.*fits');

% read specific file names
A = AstroFileName("LAST.01.08.02_20240109.143054.460_clear_001+30_001_001_001_sci_raw_Image_1_fits")

% read from a table (or similarly from an AstroCatalog object):
% If the input is a table, then attempt to populate the
% file names based on the table columns (e.g., if the
% table has a 'CropID' column, then its content will
% be stored in the AstroFileName CropID property).
T = table; T.CropID=[1 2].';
T.Product=["Image","PSF"].' ; T.JD=2451545+(0:1).';
A = AstroFileName(T)
% Read JD from table and convert it to Time string
A = AstroFileName(T, 'ReadJD',true, 'JD2Time',true)

The ProjName can be composed of several sub-literals separated by dot. The first sub-literal is a string while the rest may be numeric values that will be converted to strings. This fact can be used to populate the ProjName is a more convenient way:

A=AstroFileName(1);
% For example, instead of using:
A.ProjName="LAST.01.01.01";
% you can achieve the same goal using:
A.ProjName={"LAST",1,1,1}

Although all the literals are saved as strings, some of them can be populated by numerical values that will be converted to strings. For example:

A.Counter = [1 2 3];
% is equivalent to:
A.Counter = ["001";"002";003"];

% In a similar way:
A.Version = 1;
A.CCDID = 1;  % will be converted to string using the FormatCCDID property
A.CropID = 1; % will be converted to string using the FormatCropID property
A.FieldID = 1; % will be converted to string using the FormatFieldID property

The literal content may be either a single-element string array or a multi-element string array. However, when file names are generated, all the single-element string array will be converted (using repmat) to the length of the Time property. The literals are always kept as a column vectors.

To count the number of file names in each element of the AstroFileName object (i.e., the number of elements in the Time property), you can use:

AstroFileName.nFiles

To read the content of a literal property:

A=AstroFileName('LAST.01.*fits','Path','.');
A.getProp('Type')
% Alternatively, you can read specific file name indices:
A.getProp('Product',[1 2])

The Time string contains the file name Time. However, it can be converted from/to JD. A few examples:

A=AstroFileName('LAST.01.*fits','Path','.');

% to generate a vector of JD from the Time property (note that this will not populate the JD property):
A.julday

% One can populate the JD property:
A.JD = 2451545;
% and then to use it to populate the Time string:
A.julday2time

% get the SunAlt for the time:
A.sunAlt

A=AstroFileName('LAST.01.*fits','Path','.');
% sort by JD
A.sortBy('JD')
% sort by 'Product'
A.sortBy('Product')

% To select (or reorder) entries of a single element object using indices or logical flags:
A.reorderEntries([2 1])  % I.e., select entry 2 and 1 only

% To search all entries that have a specific literal value:
% The following will return a new AstroFileName with file names for which Type=raw
A.selectByPropVal('Type','raw');

% Select Times/JD in range:
A.selectByDate(2451545, 2460000)

The most important functionality is the construction of file names and path names, from the literals in the object:

% Generate a string array of file names:
A.genFile

% Generate a string array of file names from selected entries (by index or logical flags):
F.genFile([1 2])

% Generate file names, but replace the Product literal with "Image" and Filter by "".
A.genFile([], 'Product','Image', 'Filter',"")

% Generate a single element string array of path (based on the first file name):
A.genPath

% Generate path for all file names:
% NOTE that if the Path property is not empty, then it will be used as is:
A.Path = "";  % you may want to suppress the Path property so that the BasePath will be used.
A.genPath([])

% You can also generate specific directory types:
A.genPath(1,'PathType','new')
A.genPath(1,'PathType','proc')
A.genPath(1,'PathType','raw')
A.genPath(1,'PathType','failed')
A.genPath(1,'PathType','calib')
A.genPath(1,'PathType','ref')

% Generate a full file name including path:
A.genFull

% Generate a full file name for selected entries and replace literals:
A.genFull([1 2], 'genFileArgs',{'Product','PSF'})

% Generate a matrix of strings array
% raws per each file name entry and column for each product entry (default products are: "Image", "Mask", "PSF", "Cat")
A.genProducts
% or selected products:
A.genProducts([],'OutProduct',{'PSF','Image'}, 'AddPath',false)


## Group file names

A few grouping functions are available. These functions get a single object AstroFileName and may generate a vector of AstroFileName objects.
You can either group images taken at sucessive times, by their Counter value, or group by large time gaps between the file names JD:

```matlab
[~, A]=A.groupByCounter;
[~, A]=A.groupByTimeGaps;

You can read properties from header, or write properties to headers using:

A.write2header(AH)
A.readFromHeader(AH)

To delete, move, copy files from one dir to another you can use the moveImages method. A few eaxmples:

ProcPath = A.genPath(1, 'PathType','proc');
A.moveImages('Operator','move', 'DestPath',ProcPath)
% See help for more options