Automated processes - RubinLab/epad-ws GitHub Wiki

Flowchart

The following diagram shows the background processed that are run in ePad. Background flowchart

QueueAndWatcherManager is the manager that controls all the watchers in ePad

and manages three queues

  • DicomSeriesWatcherQueue : a queue of SeriesProcessingDescription
  • XnatSeriesWatcherQueue : a queue of SeriesProcessingDescription
  • PngGeneratorTaskQueue : a queue of GeneratorTask

Has a addDICOMFileToPNGGeneratorPipeline method to manage the png generation pipeline.

Dcm4CheeDatabaseWatcher

Watches for changes in ePAD's DCM4CHEE MySQL database

  • In 5 second intervals, it checks if there are studies with the 'study_status' field set to zero, which indicates that they are a new series. Adds them to ePAD's series watcher queues to be subsequently processed by watchers (currently DICOMSeriesWatcher and XNATSeriesWatcher). Sets the series' processing status in database to IN_PIPELINE
  • In 5 second intervals, it also checks if there are existing remote pac transfers
  • In 50 second intervals, it checks if any series was deleted from dcm4chee directly and deletes them from ePad database.

DicomSeriesWatcher

Runs infinitely.

At 1 am every night:

  • Runs [ImageCheckTask's verify image generation](Nightly tasks#verifyimagegeneration) if it is not disabled by DISABLE_IMAGECHECK configuration
  • Runs RemotePACQueryTask if it is not disabled by DISABLE_REMOTEPAC_QUERY configuration

XnatSeriesWatcher

Monitors the XNAT series queue. The queue contains descriptions of new DICOM series that have been uploaded to ePAD's DCM4CHEE instance (and are modeled by SeriesProcessingDescription objects). Here, we create the XNAT subject and study record for each series. The series data will then be subsequently process by the DICOMSeriesWatcher. This queue is populated by a Dcm4CheeDatabaseWatcher, which monitors a DCM4CHEE MySQL database for new DICOM series.

PngGeneratorProcess

Polls the PngGeneratorTaskQueue infinitely, executes the retrieved task to generate the pngs. Then creates a DicomHeadersTask with the information retrieved from the polled task and executes it to create the tag file using dcm2txt script. The polled task can be one of the following:

QueueAndWatcherManager maintains the PngGeneratorTaskQueue, and decides which task is required when DicomSeriesWatcher adds dicom files to pipeline by calling addDICOMFileToPNGGeneratorPipeline.

EpadSessionWatcher

Checks if there are any sessions that has exceeded its life span (1 hour for now) in one minute intervals and expires it.

Every 15 minutes, checks if the disk has enough space on the dcm4che root directory, epad DicomProxy directory, temp directory and mysql directory and posts an eventlog with a warning if any of the folders has less than MIN_DISKSPACE (500Mb for now) available.

Once every day, runs EpadStatisticsTask to gather the statistics and send to epad-public instance if DISABLE_STATISTICS is not set to true in the configuration.

DicomSeriesProcessingStatusTracker

Singleton class to keep track of all series in the pipeline. Each series has a SeriesPipelineState object describing its processing status.

SeriesPipelineState

Pipeline processing state of a DICOM series. A DicomSeriesProcessingStatusTracker holds these for all series currently in ePAD's pipeline. When a new series is detected, it either runs to completion or until it is idle for a set amount of time (currently 30 seconds). Holds SeriesProcessingDescription, DicomSeriesProcessingState and a lastActivityTimeStamp to check if it is still processing or idle.

SeriesProcessingDescription

Keeps all the information about an uploaded series that is being processed by ePAD. A SeriesPipelineState tracks the processing state of this series when it is in the pipeline. It has numberOfInstances, instances, studyUID, seriesUID, subjectName, subjectID. Calculates the completion percentage by comparing number of instances to the number of completed instances. The instances list holds the DicomImageDescriptions of the completed instances.

DicomSeriesProcessingState

Dicom series processing state can be one of the following NEW, IN_PIPELINE, IN_PNG_GRID_PIPELINE, ERROR, COMPLETE

DicomImageDescription

Holds instanceNum, SOPInstanceUID, error message, PNG file path, header file path and DicomSeriesProcessingState

SeriesProcessingStatus

NO_DICOM(1), IN_PIPELINE(2), DONE(3), ERROR(4)

This is the actual one put in the database for series. Dcm4CheeDatabaseWatcher sets as IN_PIPELINE, DicomSeriesWatcher sets as DONE, png generators sets as ERROR if an error occurs.

RTDICOMProcessingTask

Uses a matlab code to read the RTSTRUCT dicom file, creates a segmentation and an annotation using the extracted information. Puts the mat file produced by the matlab code in the epafd files with a status DONE when it is successful. As it saves the segmentation object, the pngs for that segmentation are actually created by DSOMaskPNGGeneratorTask.

DSOMaskPNGGeneratorTask

Generate the pngs using ImageIO. Uses SourceImage to get the frames as the segmentation objects are multiframe. It also generated the pngs for multi-segment DSOs, the pngs are named in the manner of frameNumber_segmentNumber.png If it fails to generate the pngs, it removes the series from the pipeline and updates the series status to ERROR. Creates an aim file for the dso. Checks if there are aims referencing to this dso and deletes the old aim file. It also updates the start index in the Aim file to point to the instance number segmentation objects start at.

MultiFramePNGGeneratorTask

First tries creating the png using ImageJ library. If that is successful writes the png using ImageIO; if that fails, uses PixelMed library to create the png. To handle floating point parametric maps, it also gets the pixel data using PixelMed and stored them in the database. If both ImageJ and PixelMed fails to create the png it updates the database setting both PNGFileProcessingStatus and SeriesProcessingStatus to ERROR and creates an event log

SingleFrameDICOMPngGeneratorTask

First tries creating the png using [Dcm4Chee and image IO library](PNG generation#dcm4chee-and-imageio) and if that fails, uses PixelMed library to create the png. If that also fails updates the database setting both PNGFileProcessingStatus and SeriesProcessingStatus to ERROR and creates an event log

PNGFileProcessingStatus

NO_DICOM(1), IN_PIPELINE(2), DONE(3), ERROR(4)

This is the actual one put in the database for png files. png generators sets to IN_PIPELINE when the file entry is created, DONE if png creation is successful and ERROR if an error occurs.