Dual Camera Picture‐in‐Picture 26.1 - ArrowElectronics/Agilex-5 GitHub Wiki

Table of Contents

Quick Start Guide
Build Instructions

Quick Start Guide

This Quick Start Guide describes the architecture of this reference design. It also describes the hardware and software setup needed to run it.

  1. Overview
  2. Theory of Operation
  3. Install the Quartus Prime Pro Programmer
  4. Configure the Board for the Demo
  5. Flash the FPGA configuration file.
  6. Run the Demo

Overview

This reference design demonstrates the Agilex-5 MIPI capabilities. Video from dual ams OSRAM Mira220 cameras enters the Agilex-5 via the MIPI DPHY/CSI-2 and goes through a few Video ISP modules and gets triple-buffered in DDR4 memory. From there, it is read, passes through more Video ISP and an HDMI transmitter, on its way to an HDMI display.

One of the video streams is displayed on the screen's background. The second video stream is displayed as a Picture-in-Picture (PiP) window moving around the screen and bouncing off its walls.

The Video ISP used is a minimal set necessary to carry the video across. It is not implemented to do any color correction or auto exposure. These tasks are left to the user to add, if they need them.

Theory of Operation

Both Mira220 cameras are initialized to generate a 960x540 (arbitrary 1/4 HD)image at 30fps, in RAW12 format over 2 data lanes running at 900Mbps per lane. MIPI camera data enters the Agilex-5 via the hard DPHY IP, then goes through the CSI-2 RX IP to be processed and sent out as an axi4 video streaming interface. It passes through a minimal set of Video ISP blocks, namely the White Balance Correction IP and the Demosaic IP, which converts the 12-bit Bayer to 24-bit RGB per pixel and flow as 4 pixels in parallel. The axi4 streaming data enters a Scaler IP. For the main background video, it gets scaled to a full HD format of 1920x1080. For the PiP video, it gets scaled down to an arbitrary 512x300 resolution. From there, the corresponding Video Frame Writers store the frame in 3 rolling buffers located in the DDR4 memory.

On the output side, there are 2 Video Frame Reader IPs, where each one reads from their designated frame buffers as 1 pixel in parallel. The 2 video streams enter a video mixer which blends the PiP video over the background video. The Clocked Video Output IP generates the proper timing for a 1920x1080 @60fps image. The Full Raster to Clocked Video Converter IP converts the AXI streaming video into 24-bit RGB with Sync signals to be forwarded to the external HDMI transmitter.

The Camera configuration file used by the Sensor Configuration Tool v1.4 is provided in the scripts subfolder. The user can modify it to obtain a different resolution. From that tool, a .csv file is exported and edited to look like the .txt file in the same folder. Initialization settings from the .txt file are used to populate the Mira220 .h file in the software/mipi_app subfolder.

In addition to setting up IP blocks in the Platform Designer tool, the user still does more initialization using a NIOS-V CPU. Moving the PiP window around the screen is done via the NIOS-V code.

Block diagram.

The reference design includes:

  • Nios® V/m Microcontroller
  • Onchip RAM (160KB)
  • DDR4 memory
  • System ID peripheral
  • Mailbox Client
  • MIPI DPHY/CSI2
  • Video & Vision Processing IP
  • Push-Button
  • 2x RGB LED
  • Light Weight UART

Configure the Board for the Demo

The following components are required for the demo:

  • AXE5-Falcon (TEI0183) development board,
  • USB C Cable
  • 2x ams OSRAM Mira220 cameras
  • 2x 15-pin to 22-pin FPC cables
  • HDMI Cable
  • HDMI Monitor/TV

Assemble the Hardware

  • Plug the USB Cable into J25, the USB C connector.
  • Plug the HDMI cable in J7, the HDMI connector.
  • Connect an FPC cable between a camera and J8, 22-pin CSI2 connector.
  • Connect the other FPC cable between a camera and J9, 22-pin CSI2 connector.

Flash the FPGA Configuration File

Open the Quartus Programmer

    $ Tools --> Programmer

Add JTAG Hardware. If the Hardware Setup has 'No hardware'

    $ Press the Hardware Setup button
    $   Double click USB Blaster III, then Press Close
    $ Select the FPGA image and click on the **Change File...** button, 
    $   Navigate to the **output_files/falcon_mipi_pip.jic** file and select it
    $ Check the **Program/Configure** box on the .jic row
    $ Click the Start Button

This will take up to a minute to complete.

Run the Demo

The demo runs without using a terminal for user interaction, but it is optional to perform other tasks.

(optional) Open a terminal

     $ Open Tera-Term or equivalent terminal program
     $ set its baud rate to 115200

Observe boot progress on the terminal and see available commands,.

Build Instructions

Release Contents
Prerequisites
Build the Reference Design

Release Contents

Latest Source Code Release Contents - Branches and Commit IDs

Component Location Branch Tag/Commit ID
GHRD https://github.com/ArrowElectronics/refdes-agilex5 master QPDS26.1_REL_GSRD_PR/4c386a5259ca487f4fdc80e9d7c9a404b5ec39fc

Prerequisites

  • Host machine running Windows or Linux.
  • Internet connection to download the tools and clone the repositories from github. If you are behind a firewall you will need your system administrator to enable you to get to the git trees.
  • Quartus Prime Pro version 26.1 (with NIOS-V license)
  • Licenses :
    • NIOS-V free license
    • CSI-2 and DSI-2 free licenses
    • Video and Vision Processing IP evaluation License
    • Synaptic Labs xSPI IP License
  • Ashling RiscFree IDE for Intel FPGAs 26.1, if modifying software

Build the Reference Design

Open a niosv shell.

For Windows: Start --> Altera 26.1.xx Pro Edition --> Nios V Command Shell.

For Linux: Open a shell and then enter niosv-shell at the prompt.

Set up the Environment

    $ sudo rm -rf agilex_5
    $ mkdir agilex_5
    $ cd agilex_5
    $ export TOP_FOLDER=`pwd`

Clone the repository

    $ cd $TOP_FOLDER
    $ git clone -b QPDS26.1_REL_AGILEX5_REFDES https://github.com/ArrowElectronics/refdes-agilex5 refdes-agilex5
    $ cd refdes-agilex5/axe5_falcon/falcon_mipi_pip_refdes

Build the Software

Create the User Application

Create the User BSP

    niosv-bsp \
      --create \
      --qsys=./top_system.qsys \
      --quartus-project=./falcon_mipi_pip.qpf \
      --type=hal \
      ./software/mipi_bsp/settings.bsp

Create the project

    niosv-app\
      --bsp_dir=software/mipi_bsp\
      --app_dir=software/mipi_app\
      --srcs=software/mipi_app/niosv_mipi_mira220.c

Create the project Makefile

    cmake \
      -G "Unix Makefiles" \
      -DCMAKE_BUILD_TYPE=Debug\
      -B software/mipi_app/build/Debug\
      -S software/mipi_app

Build the project

    cmake --build software/mipi_app/build/Debug

Build the Hardware

Compile the Quartus project

    quartus_sh --flow compile falcon_mipi_pip

The following file is created:

  • agilex_5/refdes-agilex5/axe5_falcon/falcon_mipi_pip_refdes/output_files/falcon_mipi_pip.sof

Create the QSPI Flash Image

    quartus_pfg -c ./output_files/falcon_mipi_pip.sof ./output_files/falcon_mipi_pip.jic -o device=MT25QU256 -o flash_loader=A5EE013BB23BE6SCS -o mode=ASX4

The following file is created:

  • agilex_5/refdes-agilex5/axe5_falcon/falcon_mipi_pip_refdes/output_files/falcon_mipi_pip.jic

Return - Table of Contents