This page contains instructions for performing Acceptance Testing for the RTC-S1 beta delivery from the OPERA-ADT team. These instructions assume the user has access to the JPL FN-Artifactory, and has Docker installed on their local machine.

Acquiring the RTC-S1 Beta Docker Image

The image is currently hosted on JPL FN-Artifactory, which requires JPL VPN access and JPL credentials. You may also need to be added to the gov.nasa.jpl.opera.adt organization.

Once you have access, the container tarball delivery is available under general/gov/nasa/jpl/opera/adt/rtc_s1/r2/beta/dockerimg_rtc_beta_0.2.tar. Sample inputs and outputs are also available under general/gov/nasa/jpl/opera/adt/rtc_s1/r2/beta/delivery_2_beta_0.2.zip.

Download both images to a location on your local machine. This location will be referred to throughout this instructions as <RTC_DIR> Note that the sample data is quite large, so the download from AF can take some time.

Loading the image into Docker

The first step in running the RTC-S1 image is to load it into Docker via the following command:

docker load -i <RTC_DIR>/dockerimg_rtc_beta_0.2.tar

This should add the Docker image to your local repository with the name opera/rtc and the tag beta_0.2.

Preparing the test data

Once the delivery_2_beta_0.2.zip file is downloaded to your local machine, unpack it to <RTC_DIR>:

cd <RTC_DIR>; unzip delivery_2_beta_0.2.zip

This will create a delivery_2_beta_0.2 directory within <RTC_DIR> containing the following files/directories:

- input_dir/
  |_ opera_burst_database_deploy_2022_1212.sqlite3
  |_ PERU_EPSG4326.tif
  |_ dem.tif
  |_ S1B_IW_SLC__1SDV_20180504T104507_20180504T104535_010770_013AEE_919F.zip
  |_ S1B_OPER_AUX_POEORB_OPOD_20180524T110543_V20180503T225942_20180505T005942.EOF
- expected_output_dir/
  |_ rtc.log
  |_ t069_147169_iw3
     |_ rtc_product_v0.2.h5
     |_ rtc_product_v0.2_VH.tif
     |_ rtc_product_v0.2_VV.tif
  |_ <additional subdirs for each burst from the input SLC>
- output_dir/
- rtc_s1_beta.yaml
- scratch_dir/

In order to execute the SAS, the input file directory, runconfig, scratch and output locations will be mounted into the container instance as Docker Volumes. To help streamline this process, we recommend making the following changes within the delivery_2_beta_0.2 directory:

Create a directory named runconfig, and copy the existing runconfig YAML file into it:

mkdir -p <RTC_DIR>/delivery_2_beta_0.2/runconfig

cp <RTC_DIR>/delivery_2_beta_0.2/rtc_s1_beta.yaml <RTC_DIR>/delivery_2_beta_0.2/runconfig/rtc_s1_beta.yaml
Modify the copied runconfig to point the product_path value under product_group to "output_dir". This is required since we will invoking the SAS image using a different working directory than its default. The product_path section of the Runconfig should be modified to match the following:

    product_group:
        processing_type: 'NOMINAL'

        product_version: 0.2

        # Directory where PGE will place results
        product_path: output_dir  # <- update to point at output_dir

        # Directory where SAS writes temporary data
        scratch_path: scratch_dir
        ...

Make sure the output and scratch directories have write permissions set

chmod ga+w output_dir/ scratch_dir/

Executing the RTC-S1 container on the sample datasets

We're now ready to execute the RTC-S1 Beta. Run the following the command to kick off execution with the test assets:

docker run --rm -u $UID:$(id -g) \
  -w /home/rtc_user \
  -v <RTC_DIR>/delivery_2_beta_0.2/runconfig:/home/rtc_user/runconfig:ro \
  -v <RTC_DIR>/delivery_2_beta_0.2/input_dir:/home/rtc_user/input_dir:ro \
  -v <RTC_DIR>/delivery_2_beta_0.2/output_dir:/home/rtc_user/output_dir \
  -v <RTC_DIR>/delivery_2_beta_0.2/scratch_dir:/home/rtc_user/scratch_dir \
  -i --tty opera/rtc:beta_0.2 --log output_dir/rtc_at.log /home/rtc_user/runconfig/rtc_s1_beta.yaml

Running on the OPERA PGE Dev/CI machine, execution on all bursts took roughly 150 minutes. Once execution is complete, you should see a number of directories within the <RTC_DIR>/delivery_2_beta_0.2/output_dir/ directory, one for each burst ID processed. There should be 28 sub-directories, each containing three output products:

rtc_product_v0.2.h5 : HDF5 file containing the metadata for the product(s)
rtc_product_v0.2_VH.tif : GeoTIFF containing the RTC raster layer for the VH polarization channel
rtc_product_v0.2_VV.tif : GeoTIFF containing the RTC raster layer for the VV polarization channel

Running the Quality Assurance test

Now that we've successfully executed the SAS container and generated outputs, the last step is to perform a QA check against the expected outputs.

For the current beta delivery, the comparison script (rtc_compare.py) is available within container, and can be invoked via a docker run command to compare a single set of burst-based products at a time. The script expects the paths to the expected and output HD5 metadata product, from which it can find the GeoTIFF layers to compare automatically.

The following shell script can be used to automated the comparisons across all burst-based products:

#!/bin/bash

declare -a burst_ids=(  "t069_147169_iw3"
                        "t069_147170_iw1"
                        "t069_147170_iw2"
                        "t069_147170_iw3"
                        "t069_147171_iw1"
                        "t069_147171_iw2"
                        "t069_147171_iw3"
                        "t069_147172_iw1"
                        "t069_147172_iw2"
                        "t069_147172_iw3"
                        "t069_147173_iw1"
                        "t069_147173_iw2"
                        "t069_147173_iw3"
                        "t069_147174_iw1"
                        "t069_147174_iw2"
                        "t069_147174_iw3"
                        "t069_147175_iw1"
                        "t069_147175_iw2"
                        "t069_147175_iw3"
                        "t069_147176_iw1"
                        "t069_147176_iw2"
                        "t069_147176_iw3"
                        "t069_147177_iw1"
                        "t069_147177_iw2"
                        "t069_147177_iw3"
                        "t069_147178_iw1"
                        "t069_147178_iw2"
                        "t069_147178_iw3")

expected_dir=$1
output_dir=$2

for burst_id in "${burst_ids[@]}"; do
  echo "-------------------------------------"
  echo "Comparing results for $burst_id"
  docker run --rm -w /home/rtc_user \
      -v $expected_dir:/home/rtc_user/expected_output_dir \
      -v $output_dir:/home/rtc_user/output_dir \
      -i --tty --entrypoint /bin/bash opera/rtc:beta_0.2 \
      "-i" "-c" "python3 /home/rtc_user/OPERA/RTC/app/rtc_compare.py output_dir/$burst_id/rtc_product_v0.2.h5 expected_output_dir/$burst_id/rtc_product_v0.2.h5"
done

After giving the script a name, such as rtc_compare_products.sh and making it executable, you can compare all expected/actual products with the following command:

rtc_compare_products.sh <RTC_DIR>/delivery_2_beta_0.2/expected_output_dir <RTC_DIR>/delivery_2_beta_0.2/output_dir

The rtc_compare.py script performs a number of checks on both the image layers and metadata for each burst-based set of products. All tests should be marked as PASS, or WARNING for some instances of metadata where we expect a mismatch between input file paths, time stamps etc...

An example report for a single burst product is provided below.

-------------------------------
Comparing results for t069_147169_iw3
*******************************************************
************      TESTING (HDF5 file)      ************
*******************************************************
*** file 1: output_dir/t069_147169_iw3/rtc_product_v0.2.h5
*** file 2: expected_output_dir/t069_147169_iw3/rtc_product_v0.2.h5
-------------------------------------------------------
Checking the dataset.
[PASS]   Dataset 1 of 47: //science/SENTINEL1/identification/productVersion
[PASS]   Dataset 2 of 47: //science/SENTINEL1/RTC/metadata/processingInformation/inputs/demSource
[PASS]   Dataset 3 of 47: //science/SENTINEL1/RTC/metadata/processingInformation/algorithms/demInterpolation
[PASS]   Dataset 5 of 47: //science/SENTINEL1/RTC/grids/frequencyA/zeroDopplerTimeSpacing
[PASS]   Dataset 6 of 47: //science/SENTINEL1/identification/isUrgentObservation
[PASS]   Dataset 7 of 47: //science/SENTINEL1/RTC/metadata/processingInformation/algorithms/radiometricTerrainCorrection
[PASS]   Dataset 8 of 47: //science/SENTINEL1/identification/burstID
[PASS]   Dataset 9 of 47: //science/SENTINEL1/RTC/grids/frequencyA/listOfPolarizations
[PASS]   Dataset 10 of 47: //science/SENTINEL1/RTC/metadata/orbit/referenceEpoch
[PASS]   Dataset 11 of 47: //science/SENTINEL1/identification/processingType
[PASS]   Dataset 12 of 47: //science/SENTINEL1/RTC/metadata/processingInformation/algorithms/RTCVersion
[PASS]   Dataset 13 of 47: //science/SENTINEL1/RTC/grids/frequencyA/xCoordinateSpacing
[PASS]   Dataset 14 of 47: //science/SENTINEL1/identification/diagnosticModeFlag
[PASS]   Dataset 15 of 47: //science/SENTINEL1/RTC/grids/frequencyA/xCoordinates
[PASS]   Dataset 17 of 47: //science/SENTINEL1/RTC/grids/frequencyA/faradayRotationFlag
[PASS]   Dataset 18 of 47: //science/SENTINEL1/RTC/grids/frequencyA/yCoordinates
[PASS]   Dataset 19 of 47: //science/SENTINEL1/RTC/metadata/processingInformation/inputs/l1SlcGranules
[PASS]   Dataset 20 of 47: //science/SENTINEL1/RTC/metadata/orbit/interpMethod
[PASS]   Dataset 21 of 47: //science/SENTINEL1/RTC/metadata/processingInformation/algorithms/S1ReaderVersion
[PASS]   Dataset 22 of 47: //science/SENTINEL1/RTC/grids/frequencyA/polarizationOrientationFlag
[PASS]   Dataset 23 of 47: //science/SENTINEL1/identification/zeroDopplerEndTime
[PASS]   Dataset 24 of 47: //science/SENTINEL1/identification/productType
[PASS]   Dataset 26 of 47: //science/SENTINEL1/RTC/grids/frequencyA/slantRangeSpacing
[PASS]   Dataset 27 of 47: //science/SENTINEL1/RTC/grids/frequencyA/rangeBandwidth
[PASS]   Dataset 28 of 47: //science/SENTINEL1/RTC/grids/frequencyA/projection
[PASS]   Dataset 29 of 47: //science/SENTINEL1/RTC/metadata/orbit/velocity
[PASS]   Dataset 30 of 47: //science/SENTINEL1/identification/listOfFrequencies
[PASS]   Dataset 31 of 47: //science/SENTINEL1/identification/trackNumber
[PASS]   Dataset 32 of 47: //science/SENTINEL1/RTC/metadata/orbit/position
[PASS]   Dataset 33 of 47: //science/SENTINEL1/RTC/metadata/processingInformation/algorithms/geocoding
[PASS]   Dataset 35 of 47: //science/SENTINEL1/RTC/metadata/orbit/orbitType
[PASS]   Dataset 36 of 47: //science/SENTINEL1/identification/lookDirection
[PASS]   Dataset 37 of 47: //science/SENTINEL1/identification/missionId
[PASS]   Dataset 38 of 47: //science/SENTINEL1/identification/boundingPolygon
[PASS]   Dataset 39 of 47: //science/SENTINEL1/RTC/metadata/orbit/time
[PASS]   Dataset 40 of 47: //science/SENTINEL1/identification/absoluteOrbitNumber
[PASS]   Dataset 41 of 47: //science/SENTINEL1/identification/zeroDopplerStartTime
[PASS]   Dataset 42 of 47: //science/SENTINEL1/RTC/grids/frequencyA/layoverShadowMask
[PASS]   Dataset 43 of 47: //science/SENTINEL1/RTC/grids/frequencyA/radiometricTerrainCorrectionFlag
[PASS]   Dataset 44 of 47: //science/SENTINEL1/RTC/grids/frequencyA/centerFrequency
[PASS]   Dataset 45 of 47: //science/SENTINEL1/identification/orbitPassDirection
[PASS]   Dataset 46 of 47: //science/SENTINEL1/RTC/grids/frequencyA/yCoordinateSpacing
[PASS]   Dataset 47 of 47: //science/SENTINEL1/identification/isGeocoded

Checking the attributes.

HDF5 test summary:
    [PASS]  Same dataset structure confirmed.
    [PASS]  Same attributes structure confirmed.
    [PASS]  The datasets of the two HDF files are the same within the tolerance.
            Relative tolerance = 0.0001, Absolute tolerance = 1e-05
    [PASS]  The attributes of the two HDF files are the same within the tolerance
            Relative tolerance = 0.0001, Absolute tolerance = 1e-05
*******************************************************
************   TESTING (VH polarization)   ************
*******************************************************
*** file 1: output_dir/t069_147169_iw3/rtc_product_v0.2_VH.tif
*** file 2: expected_output_dir/t069_147169_iw3/rtc_product_v0.2_VH.tif
-------------------------------------------------------
[PASS]  Comparing number of bands
Comparing RTC-S1 bands...
[PASS]       Band 1 - "
[PASS]  Comparing geotransform
[PASS]  Comparing metadata
*******************************************************
************   TESTING (VV polarization)   ************
*******************************************************
*** file 1: output_dir/t069_147169_iw3/rtc_product_v0.2_VV.tif
*** file 2: expected_output_dir/t069_147169_iw3/rtc_product_v0.2_VV.tif
-------------------------------------------------------
[PASS]  Comparing number of bands
Comparing RTC-S1 bands...
[PASS]       Band 1 - "
[PASS]  Comparing geotransform
[PASS]  Comparing metadata
*******************************************************
************         Overall results       ************
*******************************************************
[PASS]   HDF5 test
[PASS]   VH polarization test
[PASS]   VV polarization test
*******************************************************

For the Acceptance Test to pass, the "Overall results" section for each burst product should display PASS for all three test categories.

RTC‐S1 Beta Acceptance Testing Instructions - nasa/opera-sds-pge GitHub Wiki

Acquiring the RTC-S1 Beta Docker Image

Loading the image into Docker

Preparing the test data

Executing the RTC-S1 container on the sample datasets

Running the Quality Assurance test

⚠️ GitHub.com Fallback ⚠️

RTC‐S1 Beta Acceptance Testing Instructions - nasa/opera-sds-pge GitHub Wiki

Acquiring the RTC-S1 Beta Docker Image

Loading the image into Docker

Preparing the test data

Executing the RTC-S1 container on the sample datasets

Running the Quality Assurance test

⚠️ **GitHub.com Fallback** ⚠️

⚠️ GitHub.com Fallback ⚠️