3. Implementation details - radhikahorti/Dynamic-Display-of-Histogram-and-Tone-curve GitHub Wiki

This chapter discusses the solution chosen and defines the algorithm used in the project. The project flowchart is also included in this chapter.

3.1 Specifications and final system architecture

3.1.1 NVIDIA Jetson AGX Orin kit specifications:

The NVIDIA Jetson AGX Orin kit is built with a high performance and power efficient AGX Orin module delivering up to 275 Terra Operations Per Second (TOPS) and 8 times the performance when compared to NVIDIA Jetson AGX Xavier in the same compact form. It is power configurable between 15W to 50W. It contains the module with a heat sink and a reference carrier board,802.11ac/abgn wireless Network interface controller, USB-C power controller and cord together with a quick start.

The architecture of NVIDIA Jetson AGX Orin is modelled NVIDIA Ampere architecture GPU and Arm Cortex – A78AE CPU. It also has inbuilt next-generation deep learning and vision accelerators. High-speed IO, 204GB/s of memory bandwidth, and 32GB of DRAM. IT is also capable of feeding multiple concurrent AI application pipelines. The main advantage of Jetson Orin is that now developers can deploy large and complex models to solve problems like natural language understanding, 3D perception, and multisensor fusion.

To summarize its features: • It provides up to 275 TOPS and 8X the performance of NVIDIA Jetson AGX Xavier in the same compact form-factor, power configurable between 15W and 50W. It is a giant leap forward in Robotics and Edge AI. • Built with NVIDIA Ampere architecture with 2048 NVIDIA CUDA cores and 64 tensor cores, 12-core Arm Cortex-A78AE v8.2 64-bit CPU 3MB L2+6MB L3 • Enables multiple concurrent AI applications: on-board 64GB eMMC, 204 GB/s of memory bandwidth, and 32 GB of DRAM • High speed I/O: 22 lanes of PCIe Gen4, Gigabit Ethernet, 4 XFI interfaces for 10 Gigabit Ethernet, a Display Port, 16 lanes of MIPI CSI-2, USB3.2 interfaces and 40-pin header. • Includes next-generation vision accelerator engine: PVA v2, and multi-standard video encoder & decoder • Jetpack 5.0 installs Ubuntu 20.04 and supports entire NVIDIA JetPack and usecase specific software platforms including Isaac for robotics, and Metropolis for smart cities.

Figure 3.1: NVIDIA Jetson AGX Orin Kit

3.1.2 e-CAM22 CUOAGX IMX 462 camera specifications:

This is an 8 MP ultra low light MIPI CSI 2 camera for Jetson AGX Orin kit. This camera is based on the SONY STARVIS IMX 462 CMOS image sensor which helps it to produce superior quality images in both visible and low light conditions. This is made possible by its large sensor size of 1/1.2” and pixel size of 2.9 um.This large sensor size is in conjunction with its high resolution and helps it to capture better details at the scene of interest. • Sensor: IMX485 sensor from SONY®Focus • Type: Fixed focus • Sensor Resolution: 8MP • Chroma: Color • Shutter Type: Electronic Rolling • ShutterOptical Format: 1/1.2 • Output Format: RAW Bayer 10/12-bit • Pixel size: 2.9 um x2.9 um • Sensor Active Area: 3864H x 2180V • Array Size:3864(H) × 2200(V) • Holder: CS-Mount • DFOV:35.75 (with the lens provided by e-con)

Figure 3.2: Camera Sensor

3.2 Hardware Setup

Figure 3.3: (a) Jetson Orin Kit and Camera sensor

Figure 3.3: (b) Connections

3.3 Algorithm

The algorithm of the solution for dynamic display of histogram and tone-curve has the following steps: • Step1:From the live capture obtain the frames • Step2:For each EGLStream frame, obtain the metadata using CaptureMetadata() • Step3:For histogram, read the Bayer histogram data using iBayerHistogram(). • Step4:For tone curve, read the default tone curve values by passing an empty vector to the the getToneMapCurve() interface. • Step5:For setting up a custom tone curve, pass the custom values as a vector through setToneMapCurve() and obtain the same using getToneMapCurve(). • Step6:After obtaining these values,push them into separate arrays for separate channels. • Step7:Push these arrays into the histogram and tone curve texture. • Step8:For scaling the graph obtain the maximum value of each array. • Step9:For calculating the CPU utilization of the daemon use the ps command and pass it through the text texture. • Step10:Finally display the histogram and tone-curve for the given frame along with the CPU utilization in separate windows for the given frame. • Step13:We initialise the Gstreamer video encoder pipeline in order to record the video of the frames to which the tone curve has been applied. The EGLStream from which frames will be consumed, frame rate, and resolution are the input parameters for the same. • Step14:The Gstreamer video source is muxed to a container format for the file output in .mp4 format

3.4 Flowchart for Dynamic Display of Histogram &

Tone Curve.

The flowchart of the solution for Dynamic Display of Histogram and Tone Curve can be represented as follows in Figure 3.3.

As shown in the flowchart, the user can set the proper tone curve in the program’s producer thread. The choice 0 corresponds to linear tone curve, choice 2 for Reinhard tone curve,choice 3 for non-linear tone curve, choice 4 for exponential increase tone curve, choice 5 for S-shaped tone curve and choice 6 for delayed S-shaped tone curve. When compiling the program, the user can access these options directly from the command line by using the -c option.

Coming to the consumer thread of the program, we first obtain the Bayer histogram data and tone curve data using iBayerHistogram() and getToneMapCurve() respectively.Then these values are separated according to the channels R, G and B. To scale the graph, we obtain the maximum value of each array.Also the CPU value is obtained and stored in a file.Next the arrays and the values are passed through the texture and shader program to finally obtain the histogram and tone curve on preview for the given frame.

Figure 3.4: Flowchart for Dynamic Display of Histogram and Tone Curve