RMS Data Files and Utilities - markmac99/ukmon-pitools GitHub Wiki

What does RMS Actually Do?

Starting at dusk and going till dawn, RMS captures short 256-frame videos of the sky. Every block of 256 frames is stored in the CapturedFiles folder in a format known as Four-frame Temporal Pixel - see here for more info. RMS then analyses each video for interesting data. To do this it masks off any bad areas using 'mask.bmp' and looks for movement that looks 'meteoric'. Data capture and analysis happens in parallel to avoid gaps in the data. The captured files are stored in '~/RMS_data/CapturedFiles' in a folder for each date.

Whenever a meteoric movement is found, that block of 256 frames is marked for further analysis. In the morning once data capture has finished, RMS goes back through each tentative detection and re-analyses it in more detail. First it loads the platepar file, 'platepar_cmn2010.cal', which maps the camera's field of view onto the stars. For each confirmed detection, it recalibrates the platepar to allow for any slight movement of the camera caused by wind or temperature changes, then calculates the Magnitude, RA and Dec of each point on the trajectory. The revised platepars and calculated trajectories are stored in platepars_all_recalibrated.json and the FTPdetectinfo files. Files with probable meteors are copied to ~/RMS_data/ArchivedFiles, and uploaded to the server.

thats RMS's main job, and the platepars_all_recalibrated.json and FTPdetectinfo files are the two key scientific outputs of RMS.

What are all the other files?

Overnight

RMS creates the following files during data capture. These are put into a dated folder in ~/RMS_data/CapturedFiles

  • FF*.fits - these are the FourFrame FITs files containing the blocks of 256 frames. The FF format is lossy - see here for more information.
  • FR*.bin - when a fireball is detected, a lossless H.264 video block of a region-of-interest is saved. See here for more info on this format.
  • FS*.bin - these are fieldsum files - basically measuring the brightness of the image.
  • *.pickle - these are python temporary files containing data queued waiting analysis. Deleted after a data run.

in the morning

In the morning, RMS creates the following files in addition to the platepars and FTPdetectinfo files. ~These are put into ~/RMS_data/ArchivedFiles in a dated folder.

  • UFO-orbit compatible CSV file of the detections
  • A map and text file showing the apparent radiant and shower associated with each detection.
  • A Stack of night's detections and captures showing all detections or captures in one image.
  • Google maps KML files showing the camera field of view at 25, 70 and 100km altitude.
  • an all-night timelapse.

RMS also creates thumbnails of the nights captures and detections, astrometry and photometry charts showing the calibration quality and how much the camera moved about during the night, and fieldsums graphs showing how much the sky brightness varied. You will easily be able to identify moonrise and changes in cloud cover on these.

Extra Files created by UKMON

The UKMON toolset also creates a few extra files, using capabilities built into RMS:

  • jpgs of each detection
  • mp4s of each detection (optional)

See here for more information about the toolset.

Other files you can create

There are a few other interesting files you can create using the RMS utility modules in the Utils folder. To see what a utility does, and what parameters it requires, invoke it thus ** open a Terminal window on the Pi. ** run the command:

python -m Utils.utilityname -h

Some of the Utilities

  • '''TrackStack''': this routine creates a stack of meteors, but tracking the stars. For outreach purposes this can be pretty cool.
  • '''BatchFFtoImage''': converts all FF files in a folder to JPG or PNG.
  • '''GenerateTimeLapse''': creates an all-night timelapse in MP4 format.
  • '''GenerateMP4s''': creates an MP4 movie of each individual detection in the target folder.
  • '''FRbinViewer''': view or create MP4 video from the data the FR.bin files in the target folder. For more info see here. Some additional utilities RMS automatically creates these files but you may want to rerun the utilities for example if you've cleaned up the data.
  • '''RMS2UFO''': creates a UFO-compatible CSV file.
  • '''ShowerAssociation''': creates an image and file showing single-shower associations.
  • '''StackFFs''': stacks all FF files in the target folder into a single image. This image is stacked on the background so the stars will trail.

FourFrame Temporal Format

FTP is a way of compressing video which would otherwise take up too much space and be too slow to process.

Each block of 256 frames is converted as follows

  • An image is constructed containing only the brightest value of each pixel across all 256 frames. This is the MaxPixel plane.
  • An image is constructed containing the average value of each pixel across all 256 frames. This is the AvgPixel frame.
  • An image is constructed where each pixel's value is the standard deviation of its value across all 256 frames. This is the StdPixel frame.
  • An image is constructed where each pixels' value is the frame ''number'' in which that pixel was brightest. For example if the pixel at (10,100) was brightest in frame 129, then in this image, pixel (10,100) would have the value 129. This is the MaxFrame frame.

Remarkably, from this it is possible to reconstruct each individual frame well enough to create a crude video!

You can open these images in FITS viewers, however you'll normally just see the MaxPixel frame since it is first. Apps like Pixinsight will load all four frames separately as shown below.

FR Bin Format

If a really bright event is detected, RMS will create a short uncompressed video clip of it. Because RMS is designed to run on low-powered hardware such as a Pi3, the system doesn't capture full-frame video which would take up too much diskspace (200GB per ten second clip) and too much processing power. Instead it calculates the region of interest in each frame and discards the rest of the image.

So the FR bin files are uncompressed H.264 video of the region-of-interest in each frame. A typical FR file is thus only a few hundreds of kilobytes up to a couple of megabytes in size.

However, you can combine the FR file with the corresponding FF file (or even a different FF file) to create a video which overlays the region of interest onto the static full-frame image from the FF file - see here for more information.

An Example FF file, viewed in Pixinsight

ftpfiles

⚠️ **GitHub.com Fallback** ⚠️