🧵 How to synchronise video (eg. Lip camera) - articulateinstruments/AAA-DeepLabCut-Resources GitHub Wiki

Please choose appropriate video settings for your needs before you record video data.
The default settings in AAA should be good for most people, but you can change the settings to suit your needs.

^{Video data you record is not automatically synchronised when you record it. You must tell AAA to synchronise your video data after you have recorded it. This differs from Ultrasound data which is automatically synchronised the first time you load the recording to view it.}

First, please ensure that you are viewing a Task Window in AAA which has a video display. For example, the Record Video (Ultrasound) Task Window that is provided with AAA has a video display in the bottom-right.

To open the Video Setup dialog, please right-click the video display and choose Video Setup...

The Video Setup dialog contains 5 pages which you can navigate using the tabs at the top:

• Sync Hardware
  This page lets you choose how your video data will be time-synchronised with your audio data.
  You should set this up before you start recording.
  

• Sync New Recording
  This is the page where you perform synchronisation for data you have recorded.
  You can control how far apart video frames are and when they should start relative to the audio.
  

• Adjust Old Recording
  This page lets you adjust the synchronisation for recordings which you have previously synchronised.
  

• Video
  This page lets you select how your video hardware sends data to AAA.
  You can also adjust the colour format, brightness, contrast and hue.
  

• Memory
  This page lets you choose how much of your computer's memory AAA will use to store video data as you record it.
  If you plan to create long video recordings (over 45 seconds), please use this page to allocate more memory.
  

• Live Tracker
  This page provides a way to access the DeepLabCut controls, which you can use to view an estimate of the tongue surface shown overlaid on your tongue in real-time.
  

1. On the Sync Hardware page please select whether you have a SyncBrightUp device, a BrightUp device (older; no longer sold), or no video sync device.
2. Select which audio Channel number your sync device is connected to. If you have followed the hardware setup guide then please use Channel 2.
3. On the Sync New Recording page, please set the Frame Rate to:
   • 59.94 if your video signal should be de-interlaced (if you purchased a video camera from us, please select this).
   • 59.97 if your video signal should not be de-interlaced.
4. In the Batch Sync area of the page, please tick the De-interlace box if your video signal should be de-interlaced (if you purchased a video camera from us, please tick this box).
5. Use the date selections From and To to select a range of dates within which to synchronise all video recordings.
6. Click Sync/offset/set rate to perform synchronisation.

7. Close the Video Setup dialog you have been using.
8. Click on one of your recordings to view it.
9. Look through the recording by clicking and dragging the mouse in the video timeline (aka. video chart) and verify that a white square appears in the top-left corner of the video at the same time as the 4 tones of the video sync signal occur in the audio channel that you selected in part 2 above.

The SyncBrightUp provides accurate synchronisation between video and audio data by recording special synchronisation information when you make a recording. To use the device, you must have it correctly connected to your video and audio devices at the time you make the recording.

A video which was recorded with a SyncBrightUp should briefly show a white square in the top-left corner of the video for the duration of a large pulse on the audio sync channel (one of the two sound channels), as shown below:

These should not need to be changed. On very rare occasions, it may be necessary to adjust the Tone Threshold and Pulse Threshold values.

• Tone Frequency should always be 1000
• Tone threshold sets the level at which the tone in the sync signal is detected and will usually be positive and in the range 20-60. Observe the sync signal and note the amplitude and polarity of the 4 tones at the start as a percentage of the y-axis then adjust the threshold to detect this level of signal.
• Pulse threshold sets the level at which 10 vertical sync pulses are detected and is often negative and less than the magnitude of the tone threshold. Observe the sync signal and note the amplitude and polarity of the 14 pulses as a percentage of the y-axis then adjust the threshold to detect this level of signal. Default of -10 should work for most signals.
• Beep Length should be set to the duration of the sync tone pulses = 50ms. In exceptional circumstances if the sync signal has been distorted then this duration can be shortened to cope.
• Beep/flash delay AAA waits until this number of video frames have been received before issuing the command to produce a beep to trigger the synchronisation. This ensures the flash appears in the recorded data.
• Approx frame rate should be kept set anywhere in the range 25-30
• Flash must be within X ms AAA only looks up to this maximum amount of time from the start of the file for the flash. By default it is set to 1.5 seconds. If for any reason the flash is delayed beyond 1.5 seconds, this value can be extended.

Colour format
Select GRAY for the most space efficient 8-bit greyscale image storage. Choose RGB24 if colour recording is required but be aware that the storage requirements are 3x Grayscale.

Hue, saturation and contrast
These values may be changed from the default if you feel the recorded image needs to be adjusted.

Get, Stop and Re-initialise
If the live image freezes for any reason, it is possible to restart it by clicking the button.

Measure available memory
Click this button to find out how much free memory is currently available on your PC

Reserve memory
No matter what recording duration is selected, AAA will only record a maximum amount of video determined by the reserved memory. Move the slider to select the amount of memory you wish to reserve for recording the video data. Things to consider when setting this value:

1. You get approximately 1 second of video data for every 9.2 Mbytes of memory (the default of 410Mbytes provides about 45 seconds)[Note: if video is set to colour then this takes up 3x the memory so 27.6MBytes/sec]
2. Don’t use all of the available memory. Three quarters of the available is probably a good maximum level but is depends on what other applications may be running while recording is in progress.
3. Remember that if you are likely to want to analyse the ultrasound data on another PC it must also have the amount of memory you choose available to it otherwise it will run slowly as it swaps bits of video in and out of its memory.

Sync New recordings
Alignment is done after the data is recorded by selecting the video setup option. Once satisfactory synchronisation settings are chosen, they can be applied to all recordings using a batch processing option.
Although the video frames are produced by the ultrasound (or camera) at a steady rate, there are no time stamps associated with each frame and the video card does not log the arrival time. This means that the best estimate of the time of arrival is the time that the software reads the frame in. This is usually not at regular intervals.

The “frame rate” box provides the option to realign the video frames to a set, evenly-spaced rate. Note that although the nominal broadcast NTSC rate is 29.97 frames per second, ultrasound machines may export video frames at a slightly different rate.

• Calculate mean rate from file: Calculates a rate based on the number of samples recorded over the period of the recording. The precision of this calculation depends on the duration of the recording.
• Calculate frame rate from sync: provides a frame rate estimate based on the 14 pulses produced by the “SyncBrightUp” unit. See the SyncBrightUp manual for details. If the video has not been synchronised it will show the interlaced rate. This needs to be doubled if the video is going to be de-interlaced.

One of these estimates can be used or another rate can be specified based on an independent calculation. Calculate frame rate from sync is a good option.

^{To manually calculate your ultrasound video frame rate: Connect the video signal output from your ultrasound to the audio input and record several seconds. View the recording you have just made. You will see a repeating pattern in the signal. Count approximately 100 of these repetitions, note the duration between the first and last and divide it by the number of repetitions counted. This gives the frame rate.}

With a value in the “Frame Rate” box, click the “Set Rate” button to reassign each video frame to evenly spaced time points at the specified rate. If a mistake is made simply correct the rate and click the “Set Rate” button again.

The button is used to align the flash generated by the SyncBrightUp unit with the EPG sync signal. NOTE: The Sync Hardware setting should be set to “SyncBrightUp”. Exceptionally, if you have the older BrightUp unit one of these Hardware settings can be selected.
If you know that the video output of your ultrasound machine lags behind the actual ultrasound scan by a fixed amount you can set that offset value in the ‘offset by’ box (see Figure 5). There will typically b e a small lag between the time the image is scanned and the time it is sent out of the video port of the ultrasound machine. If the video signal passes through a video mixer then this lag can be significant.

^{To manually assess the video image latency, gently tap an object on a surface in view of the camera. The sound of the impact will show as a pulse on the audio signal. You can also see the strike occur in the video. Make several recordings each with several taps. Follow the rate and synchronisation instructions above, then look through the recordings to establish the average lag.}

The batch sync button will set the rate and synchronise all the recordings for the date range specified in the adjacent boxes. It uses the rate specified in the frame rate box. If the de-interlace box is checked then it will also de-interlace video first. This can take some time as each recording needs to be loaded in turn. If the "Current Client Only" checkbox is checked then only the recordings on the specified date for the current client will be synchronised. This helps limit the number of recordings processed in one batch.

The Adjust Old Recordings tab allows manual offsetting of the video data relative to the audio. Use this to manually offset the video data relative to the waveform.