Image Recognition with AI 26.1 - ArrowElectronics/Agilex-3 GitHub Wiki

Table of Contents

Quick Start Guide
Build Instructions

Quick Start Guide

This Quick Start Guide describes the architecture of this reference design. It also describes the hardware and software setup needed to run it.

  1. Overview
  2. Theory of Operation
  3. Install the Quartus Prime Pro Programmer
  4. Configure the Hardware for the Demo
  5. Flash the FPGA configuration file.
  6. Run the Demo

Overview

This reference design uses the MIPI DSI-2 video output path to a 1280x800 8" DSI LCD panel to demonstrate user interaction using the touch screen and an object recognition AI engine generated by [El Camino] using the One Ware ONE AI tools. Video data is created internally from drawing on the touch screen, deposited in the hyperRAM video frame buffer, and travels through a minimal set of Video and Vision Processing IP Suite components to the display.

Theory of Operation

The video data in the frame buffer is pre-initialized by the NIOS-V to paint the left half of the LCD screen in Black and the right half in White.

A few on-screen touch buttons and text boxes are used to interact with the user using the touch screen feature.

The user draws 1 of 10 objects (Banana, El Camino logo, Car, Carrot, Castle, Cruise Ship, Flower, Lightning, Traffic Light, Umbrella), with their finger, on the black left half of the screen. As the user is drawing, they are seeing it on the screen and the pattern is being written in the frame buffer as a 128x128 pixel image. When done drawing, the user presses on the “el camino” touch button to initiate the AI engine (one_ai_module) created using the One-Ware AI tools, to identify the image drawn. A Text Box is used to print what was identified along with a percentage probability of what it identified onto the white right half of the LCD screen.

The object drawn on the screen is in a 640x800 pixel region (black half) of the screen, which gets mapped into the left half of the video frame buffer as a scaled 320x400 region. At the same time, the object image is being mapped into a dedicated frame buffer of 128x128 for the AI engine to operate on. So, the touch screen interrupt service routine on the NIOS-V fills the video frame buffer and at the same time, scales the image into a dedicated 128x128 on-chip memory buffer to be acted on by the AI engine.

Block diagram.

The reference design includes:

The project has 3 main subsystems. The clock_system which handles clock generation and reset functions, the niosv_system_0 which has the NIOS-V processor and few peripherals to drive the demo, and the mipi_system_0 which has the video components used in driving the DSI LCD panel.

2.1.1 Clock sub-system

This sub-system adds the PLL and Reset logic.

Reset Release is a mandatory module in any Agilex-3. It ensures that the FPGA enters User mode after all sectors have been configured. The Reset Controller merges the external active-low input and the Reset Release resets and forwards them to the Reset Extender which has a configurable period for how long the final system reset is asserted.

2.2 MIPI DSI sub-system

The AXC3000 evaluation kit has a 256Mb hyperRAM device for data storage. It is used to implement the video frame buffer. The video output path starts with the Video Frame Readers, which pull the image data from the hyperRAM frame buffer and pass it to a Scaler, which resizes the image to fill the LCD display, then a Mixer, which overlays other items onto the background image. After the Mixer, it goes through a Protocol Converter to prepare the streaming axi4 to be “full” variant due to the DSI2 IP requirement, then gets to the DSI2 module, which contains a video timing generator that produces timing for a 1280x800 @60fps video format, which goes to the DPHY to be output to the DSI LCD display.

The NIOS-V CPU is used to initialize most of the aforementioned modules. The initialization will be explained in detail later.

In addition to all of the video elements in this design, the touch screen feature of the LCD panel is being used for user interaction.

The “evaluation” version of the hyperRAM IP maxes out when using an image format of 640x480, so a format that corresponds to half of the horizontal and half of the vertical (640x400) LCD panel resolution was chosen to be in the video frame buffer.

2.2.1 Video Frame Reader (VFR)

The Video Frame Reader (VFR) pulls RGB888 video data from the frame buffer. In this design, the frame buffer has been pre-initialized to be black on the left and white on the right. The video data in the hyperRAM frame buffer is accessed via the Synaptic Labs xSPI Multiple Bus Memory Controller (xSPI-MBMC).

The NIOS-V initializes it to be aware of the video width, height, number of buffers it can access, inter-line offset, and the start address of the memory-mapped frame buffer.

2.2.2 Scaler

Due to the condensed video frame buffer size, RGB data needs to be scaled up by a factor of 2 in both dimensions. i.e. 640x400 -> 1280x800. 1280x800 is the full LCD panel resolution. The Scaler input and output resolution are programmed by the NIOS-V.

2.2.3 Mixer

The Mixer has 8 input sources which can be mixed together to draw on the LCD screen. Input 0 is the default video stream, which we call the background. After each handwriting event, the buffer gets altered, so the user can press the “CLEAR SCREEN” on-screen touch button to re-initialize the frame buffer to its black and white halves.

Input 1 is a Test Pattern Generator that gets initialized by the NIOS-V to generate 8 color bars filling the entire screen, on demand, from a user input into a terminal emulator.

Input 2 is a custom Verilog module (bl_button_160x100_0) that draws a 160x100 pixel “BACKLIGHT” button, allowing the user to increase/decrease LCD backlighting. in2 is always on. Its start coordinates are programmed by the NIOS-V.

Input 3 is a custom Verilog module (clear_screen_160x100) that draws a 160x100 pixel “CLEAR SCREEN” button, allowing the user to restore the original black/white content of the frame buffer. in3 is always on. Its start coordinates are programmed by the NIOS-V.

Input 4 is a custom Verilog module (elcamino_160x100) that draws a 160x100 pixel “el camino + logo” button, allowing the user to start the AI engine to identify the handwritten object; then the NIOS-V uses that output to print the object's name on the right-hand side of the LCD panel via text_box_0 (in5).

Input 5 is a VVP Text Box IP block that is created as 32 characters by 4 lines, using Font size 7. It is used to write the identified object's name along with the percent confidence on the right-hand side of the screen. Its start coordinates and contents are programmed by the NIOS-V.

Input 6 is a VVP Text Box IP block that is created as 32 characters by 1 line, using Font size 7. It is used to label the black screen half at the top of the display region. Its start coordinates and contents are programmed by the NIOS-V.

Input 7 is a custom Verilog module (oneware_logo_0) that draws a 192x40 pixel "ONE-ware logo" at the top of the white half of the LCD screen.

The Mixer is operated in “Lite” variant because the custom IP blocks feeding its inputs do not pass any control data over the streaming interface.

2.2.4 Protocol Converter

Since the Mixer is being operated in “Lite” variant and the DSI2 IP expects “full” variant streaming data, the Protocol Converter IP does this.

The NIOS-V initializes it with the frame width and height values.

2.2.5 MIPI DSI-2

The mipi_dsi2 module is configured for 24-bit RGB data, with 1280 pixels per line, and CRC/ECC insertion. It has a video timing engine that is manually configured, as seen in the figure below, to generate 1280x800 @60fps.

The mipi_dsi2 does not get any further initialization by the NIOS-V.

Other calculations are done as follows:

    60fps = 16.667 ms/frame
    Allowing 928 lines/frame for an 800-line image
    Clock freq = MIPI_data_rate/(PPI_width/2) = 1000x106/(16/2)=125MHz
    HTOTAL=(clock_freq*PIP)/(frame_rate_hz*VTOTAL) = (125x106*1)/(60*928) = 2245
    HB_END = HTOTAL – HACTIVE = 2245 – 1280 = 965
    V1S_VSTART = 40
    V1S_VEND = 88
    V1B_END = 128
    HS_START = 156
    HS_END = 176
    V1S_HSTART = Align with HS_START = 156
    V1S_HEND = Align with HS_START = 156
    VTOTAL = VACTIVE + V1S_VEND = 800 + 128 = 928

2.3 NIOS-V sub-system

The niosv_system holds the NIOS-V /m variant soft processor, along with a few peripherals.

These peripherals are:

onchip_memory : This is 96KB onchip SRAM, made up of M20K memory blocks, to hold NIOS-V execution code.,

jtag_master : This module allows the user to access peripherals on the NIOS-V bus via the System Console tool, which uses TCL commands to perform debugging.

sysid : This is a 32-bit user-programmable register. The NIOS-V code parses it and displays its content on a terminal (via the lw_uart). It is programmed with 0xAC302610. The “AC3” is for Agilex-3 AXC3000, “02610” is for Quartus version 26.1.0.

camera_i2c : This module drives the I2C connected to the CRUVI-HS via the TEC0278-01 MIPI adapter. It is used to initialize the DSI LCD panel.

lw_uart : This module is used for the stdin and stdout of the C-code running on the NIOS-V processor.

rti_100ms : This interval timer is used to generate periodic interrupts for the processor to query the status of the touch screen controller. It is initialized by default to 100ms but the processor code speeds it up to provide more finger touch points as the user draws on the screen.

pio_led : This 3-bit output I/O drives the RGB LED to indicate the status.

Oneware_pulse_mipi :This 1-bit output I/O drives the One Ware AI module to initiate inference.

irq_bridge : This bridge allows the passing of an interrupt from the video frame writer in the mipi_system.

niosv_mm_bridge : This module exports the NIOS-V bus to outside sub-systems, the mipi_system.

Configure the Hardware for the Demo

There are a few hardware items required to use this reference design successfully:

  • AXC3000 Evaluation Board.
  • USB-C to USB-C cable
  • Trenz TEC0278-01 CRUVI-HS to MIPI CSI2/DSI2 adapter.
  • 6"-10" 15-pin FFC cable used with Raspberry Pi cameras/displays.
  • Waveshare 8" DSI LCD panel.
  • 2x jumper wires for 5V and GND between LCD and AXC3000
  • 4-pin 0.1" header to be soldered on MKR pins
  • Quartus Prime Pro 26.1 with free Agilex-3, NIOS-V, and MIPI DSI licenses.
  • Synaptic Labs xSPI Controller.
  • Evaluation license (or run in time-limited, JTAG-tethered mode) for the Altera Video and Vision Processing Suite
  • Quartus Prime Pro 26.1.

Assemble the Hardware

  • Plug one end of the USB-C cable in the DSI LCD panel receptacle
  • Insert the TEC0278-01 CRUVI adapter into the AXC3000 receptacle and tighten the provided screw.
  • connect the 15-pin FFC cable (pins facing away from PCB) into the DSI LCD panel connector and tighten the black tab
  • connect the other end of the 15-pin FFC cable into the TEC0278-01 connector labeled DISPLAY (pins facing the CAMERA connector) and tighten the black tab
  • Connect the jumper wires between the DSI LCD 5V/GND pins and the 5V/GND MKR pins on the AXC3000

Flash the FPGA Configuration File

  • Download the FPGA jic image axc3000_dsi_touch.jic

  • A USB-C cable is required to be plugged into the AXC3000 for JTAG configuration.

Open the Quartus Programmer

    $ Tools --> Programmer

Add JTAG Hardware. If the Hardware Setup has 'No hardware'

    $ Press the Hardware Setup button
    $   Double click USB Blaster III, then Press Close
    $ Select the FPGA icon and click on the **Change File...** button, 
    $   Navigate to the **output_files/axc3000_dsi_touch.jic** file and select it
    $ Check the **Program/Configure** box on the .jic row
    $ Click the Start Button

This will take a few seconds to complete.

Build Instructions

Release Contents
Prerequisites
Build the Reference Design

Release Contents

Latest Source Code Release Contents - Branches and Commit IDs

Component Location Branch Tag/Commit ID
GHRD https://github.com/ArrowElectronics/refdes-agilex3 master QPDS26.1_REL_AGILEX3_REFDES_AI_TOUCH/71879c887b50a4f636405b9adff06fbc4b612c1a

Prerequisites

  • Host machine running Windows or Linux.
  • Internet connection to download the tools and clone the repositories from github. If you are behind a firewall, you will need your system administrator to enable you to get to the git trees.
  • Quartus Prime Pro version 26.1
  • Licenses :
    • NIOS-V free license
    • CSI-2 and DSI-2 free licenses
    • Video and Vision Processing IP evaluation License
    • Synaptic Labs xSPI IP License

Build the Reference Design

Open a niosv shell.

For Windows: Start --> Altera 26.1.xx Pro Edition --> Nios V Command Shell.

For Linux: Open a shell and then enter niosv-shell at the prompt.

Set up the Environment

    $ sudo rm -rf agilex_3
    $ mkdir agilex_3
    $ cd agilex_3
    $ export TOP_FOLDER=`pwd`

Clone the repository

    $ cd $TOP_FOLDER
    $ git clone -b QPDS26.1_REV2_REL_AGILEX3_REFDES https://github.com/ArrowElectronics/refdes-agilex3 refdes-agilex3
    $ cd refdes-agilex3/axc3000/dsi2_touch_ai_refdes

Build the Software

Create the User Application

Create the User BSP

    niosv-bsp -c -p=axc3000_dsi2_touch.qpf -s=top_system.qsys -t=hal \
     --cmd="set_setting hal.make.cflags_optimization {-O3}" \
     --cmd="add_section_mapping .bss niosv_system_0_onchip_memory" \
     --cmd="add_section_mapping .heap niosv_system_0_onchip_memory" \
     --cmd="add_section_mapping .rodata niosv_system_0_onchip_memory" \
     --cmd="add_section_mapping .rwdata niosv_system_0_onchip_memory" \
     --cmd="add_section_mapping .stack niosv_system_0_onchip_memory" \
     --cmd="add_section_mapping .text niosv_system_0_onchip_memory" \
     --cmd="add_section_mapping .exceptions niosv_system_0_onchip_memory" \
     software/mipi_bsp/settings.bsp

     niosv-bsp -g software/mipi_bsp/settings.bsp

Create the project

    niosv-app\
      --bsp_dir=software/mipi_bsp\
      --app_dir=software/mipi_app\
      --srcs=software/mipi_app/niosv_dsi_touch.c

Create the project Makefile

    cmake \
      -G "Unix Makefiles" \
      -DCMAKE_BUILD_TYPE=Debug\
      -DCMAKE_C_FLAGS="-O3" \
      -B software/mipi_app/build/Debug\
      -S software/mipi_app

Build the project

    cmake --build software/mipi_app/build/Debug

Build the Hardware

Compile the Quartus project

    quartus_sh --flow compile axc3000_dsi2_touch

The following file is created:

  • agilex_3/refdes-agilex3/axc3000/dsi2_touch_ai_refdes/output_files/axc3000_dsi2_touch.sof

Create the QSPI Flash Image

    quartus_pfg -c output_files/axc3000_dsi2_touch.sof output_files/axc3000_dsi2_touch.jic -o device=MT25QU256 -o flash_loader=A3CY100BM16AE7S -o mode=ASX4

The following file is created:

  • agilex_3/refdes-agilex3/axc3000/dsi2_touch_ai_refdes/output_files/axc3000_dsi_touch.jic

Validate the Design

To validate the design, the other side of the USB-C cable is plugged into a USB-C 5V power source.

The USB-C port on the AXC3000 is only used for configuring the FPGA and a user interface via a COM port.

Return - Table of Contents

[El Camino]: https://www.elcamino.de/en/