3. Implementation details - radhikahorti/Embedding-Synthetic-Pattern GitHub Wiki

This chapter discusses the solution chosen and defines the algorithm used in the project. The project flowchart is also included in this chapter.

3.1 Specifications and final system architecture

3.1.1 NVIDIA Jetson AGX Orin kit specifications:

The NVIDIA Jetson AGX Orin kit is created with the highly effective and power efficient AGX Orin module that is capable of generating up to 275 Terra Operations Per Second (TOPS) and is eight times more powerful than the NVIDIA Jetson AGX Xavier in the same small shape. Power can be varied from 15W to 50W. The kit consists of the module, a heat sink, and a reference carrier board, alongside an 802.11ac/abgn wireless Network interface controller, USB-C power controller, and cord, in addition to the quick start.

The NVIDIA Jetson AGX Orin architecture has its foundation on the NVIDIA Ampere architecture GPU and the Arm Cortex - A78AE CPU. It also has next-generation deep learning and vision accelerators built in. High-speed IO, memory bandwidth of 204GB/s, and 32GB of DRAM is provided by the kit. It can also feed numerous AI application pipelines at a given time.The key advantage of Jetson Orin is that developers can now deploy massive and intricate models to address real world problems in the field of AI. To summarize its features:

• It can perform up to 275 TOPS and performs up to eight times better than than its counterpart Xavier in a comparable small shape factor, with power varying from 15W to 50W. It signifies a major improvement in robotics and edge AI. • 12-core Arm Cortex-A78AE v8.2 64-bit CPU, 2048 NVIDIA CUDA cores, 64 tensor cores, and Ampere architecture included • Multiple AI programs may run simultaneously thanks to the on-board 64GB eMMC, 204 GB/s of memory bandwidth, and the kit also provides 32 GB of RAM. • High-speed I/O includes 22 PCIe Gen4 lanes, Gigabit Ethernet, four XFI interfaces for 10 Gigabit Ethernet, a Display Port, sixteen MIPI CSI-2 lanes, USB3.2 interfaces, and a 40-pin header. • PVA v2, a next-generation vision accelerator engine, in addition to a multi-standard video encoder and decoder is also included in the kit. • Jetpack 5.0 supports the entirety of JetPack and use-case specific software platforms, including specific applications for robotics and smart cities , and installs Ubuntu 20.04.

(a) Jetson Orin Kit	(b) Camera Sensors
Figure 3.1: Orin kit and Sensors

3.1.2 e-CAM22 CUOAGX IMX 462 camera specifications:

This is an 8 MP ultra low light MIPI CSI 2 camera for Jetson AGX Orin kit. This camera is based on the SONY STARVIS IMX 462 CMOS image sensor which helps it to produce superior quality images in both visible and low light conditions. This is made possible by its large sensor size of 1/1.2” and pixel size of the frame captured by the sensor is around of 2.8 micrometer.This big sensor size, in addition to the camera sensor’s outstanding resolution, allows it to capture additional features at the scene of interest. • Sensor: IMX462 sensor from SONY®Focus • Type: Fixed focus • Sensor Resolution: 8MP • Chroma: Color • Shutter Type: Electronic Rolling • ShutterOptical Format: 1/1.2” • Output Format: RAW Bayer 10/12-bit • Pixel size: 2.8 um x2.8 um • Sensor Active Area: 3864H x 2180V • Array Size:3864(H) × 2200(V) • Holder: CS-Mount • DFOV:35.75 (with the lens provided by e-con)

3.2 Hardware Setup

(a) Jetson Orin Kit and Camera sensor	(b) Connections
Figure 3.2: Hardware Setup

3.3 Algorithm

The algorithm of the solution for dynamic display of histogram and tone-curve has the following steps: • Step1:From the live capture obtain the RAW Image frames. • Step2:The user can pick an option from the available methods to embed • Step3:Prompt the user to select the Synthetic Pattern to be embedded amongst the available options (QR Code, Dot Pattern, Uniform Color Patch and Noise schedule). • Step4:Input the above three values into the GUI for processing. • Step5:Select the framework for embedding based on the option selected by the user. • Step6:After obtaining the modified raw image array, feed both the input RAWimage and the modified RAW image into the Local Statistics measurement block. • Step7: It determines dissimilar blocks based on contrast, mean brightness, mean intensity, and standard deviation differences using the Chi-square dissimilarity measure for histogram comparison. • Step8: The algorithm iterates across blocks with predetermined sizes, comparing their properties, and storing the locations of blocks that differ. Then, by creating heatmaps that show the locations of the dissimilar blocks in each comparison, it visualizes these differences. • Step9:Encode both the RAW images into JPEG format. • Step10:Generate the SSIM score after comparing the two images for comparing the global statistics.

3.4 Flowchart for Embedding Synthetic Patterns into

Raw Images while Preserving Natural Image Statistics

The flowchart of the solution for Embedding Synthetic Patterns into Raw Images while Preserving Natural Image Statistics can be represented as follows in Figure 3.4. As shown in the flowchart, the input to the GUI are taken in the form of the input RAW image, the option of embedding format from the user and the the synthetic pattern to be embedded selected by the user. After which the appropriate form of embedding is carried out based on the option selected by the user.The corresponding modified image array is obtained as result. The modified array contains the synthetic pattern embedded onto the input RAW image. The next step is comparison of the modified image and the input image which is carried out by comparison of local as well as global statistics. In terms of local statistics, we compare local histogram, contrast, mean luminance, mean intensity and standard deviation. This comparison is then presented in terms of a heatmap wherein the dissimilar blocks are represented as cold regions. For global statistics comparison we make use of SSIM.

Figure 3.3: Flowchart for Embedding Synthetic Patterns into Raw Images while Preserving Natural Image Statistics