Luckfox Pico Ultra - johnosbb/Automation GitHub Wiki

Luckfox Pico Ultra Linux RV1106 Rockchip AI Development Board

Processor Cortex [email protected]
NPU 0.5TOPS, supports int4, int8 and int16
ISP Input 5M @30fps (Max)
Memory Pro: 128MB DDR2Max: 256MB DDR2
USB USB 2.0 Host/Device
Camera MIPI CSI 2-lane
GPIO 26 × GPIO pins
Ethernet port 10/100M Ethernet controller and embedded PHY
Default Storage SPI NAND FLASH (256MB)

image

Serial Connections

Serial connection is via an FTDI connector with the connections shown below

image

image

Serial Access via Putty

image

image

Login as root and use the luckfox password you configured in buildroot.

SSH Access via Putty

image

Login as root and use the luckfox password you configured in buildroot.

Buildroot

Install dependencies

sudo apt-get install -y git ssh make gcc gcc-multilib g++-multilib module-assistant expect g++ gawk texinfo libssl-dev bison flex fakeroot cmake unzip gperf autoconf device-tree-compiler libncurses5-dev pkg-config bc python-is-python3 passwd openssl openssh-server openssh-client vim file cpio rsync

Install the SDK

git clone https://github.com/LuckfoxTECH/luckfox-pico.git

Set Environment variables

cd {SDK_PATH}/tools/linux/toolchain/arm-rockchip830-linux-uclibcgnueabihf/
source env_install_toolchain.sh

Build Image

./build.sh lunch   # Select the reference board configuration
./build.sh         # One-click automatic compilation

Configuring Buildroot

./build.sh buildrootconfig

fserial To regenerate the RootfS

./build.sh clean rootfs
./build.sh rootfs
./build.sh firmware

For busybox, in the directory sysdrv/source/buildroot/buildroot-2023.02.6

make busybox-menuconfig

buildroot_defconfig in luckfox-pico/config

BR2_arm=y
BR2_cortex_a7=y
BR2_TOOLCHAIN_EXTERNAL=y
BR2_TOOLCHAIN_EXTERNAL_CUSTOM=y
BR2_TOOLCHAIN_EXTERNAL_PATH="../../../../tools/linux/toolchain/arm-rockchip830-linux-uclibcgnueabihf"
BR2_TOOLCHAIN_EXTERNAL_CUSTOM_PREFIX="arm-rockchip830-linux-uclibcgnueabihf"
BR2_TOOLCHAIN_EXTERNAL_GCC_8=y
BR2_TOOLCHAIN_EXTERNAL_HEADERS_5_10=y
BR2_TOOLCHAIN_EXTERNAL_WCHAR=y
BR2_TOOLCHAIN_EXTERNAL_HAS_SSP=y
BR2_TOOLCHAIN_EXTERNAL_CXX=y
BR2_PRIMARY_SITE="http://sources.buildroot.net"
BR2_OPTIMIZE_3=y
BR2_TARGET_GENERIC_HOSTNAME="luckfox pico"
BR2_TARGET_GENERIC_ISSUE="Welcome to luckfox pico"
BR2_TARGET_GENERIC_ROOT_PASSWD="luckfox"
BR2_ROOTFS_OVERLAY="board/pico_max/overlay"
BR2_PACKAGE_BUSYBOX_SHOW_OTHERS=y
BR2_PACKAGE_E2FSPROGS_RESIZE2FS=y
BR2_PACKAGE_PYTHON3=y
BR2_PACKAGE_PYTHON3_SSL=y
BR2_PACKAGE_PYTHON_AIOHTTP=y
BR2_PACKAGE_PYTHON_CLICK=y
BR2_PACKAGE_PYTHON_ITSDANGEROUS=y
BR2_PACKAGE_PYTHON_JINJA2=y
BR2_PACKAGE_PYTHON_PERIPHERY=y
BR2_PACKAGE_PYTHON_PILLOW=y
BR2_PACKAGE_PYTHON_SERIAL=y
BR2_PACKAGE_PYTHON_SETUPTOOLS=y
BR2_PACKAGE_PYTHON_SMBUS_CFFI=y
BR2_PACKAGE_PYTHON_SPIDEV=y
BR2_PACKAGE_PYTHON_WERKZEUG=y
BR2_PACKAGE_STB=y
BR2_PACKAGE_DTC=y
BR2_PACKAGE_DTC_PROGRAMS=y
BR2_PACKAGE_LIBV4L=y
BR2_PACKAGE_LIBV4L_UTILS=y
BR2_PACKAGE_LRZSZ=y
BR2_PACKAGE_NTP=y
BR2_PACKAGE_OPENSSH=y
BR2_PACKAGE_SAMBA4=y
BR2_PACKAGE_BASH=y
BR2_PACKAGE_BASH_LOADABLE_EXAMPLES=y
BR2_PACKAGE_DIALOG=y
BR2_PACKAGE_TIME=y
BR2_PACKAGE_HTOP=y
BR2_PACKAGE_NANO=y
BR2_PACKAGE_DARKNET_EMBEDDED=y
BR2_PACKAGE_HOST_DTC=y

Output Image Components

The Image output from the build system can be found in:

luckfox-pico/out/image

boot.img  download.bin  env.img  idblock.img  oem.img  rootfs.img  sd_update.txt  tftp_update.txt  uboot.img  update.img  userdata.img

Creating the SD Card

On Windows we use the application: SocToolKit.exe in SocTooKit to download the image.

image

  • We omit the update.img file
  • If the Card is not detected one may have to unplug and re-plug the card reader

Programming Nand Flash

image

Flash Organisation

The system uses UBI to manage the NAND flash. UBI is a layer that sits between the raw NAND and the filesystem, providing wear leveling and bad block management.

Several UBI devices and volumes are mounted:

  • ubi0:0: Mounted as the root filesystem (rootfs) (read only).
  • ubi4:0: Mounted with the name "oem".
  • ubi5:0: Mounted with the name "userdata".

Flash Tools

Enable ubi utilities in busybox

image

Configure the network in /etc/network/interfaces

We continue to let /etc/init.d/S40network handle lo and any minimal baseline setup.

We create a new script ``/etc/init.d/S45netselect``` to allow us select between Wifi and Eth0. This script will:

  • Wait for hardware readiness (e.g., sleep for Wi-Fi).
  • Check if eth0 has link.
  • Set up eth0 and add default gateway if it does.
  • Otherwise, start wpa_supplicant, bring up wlan0, and set the route there.

S40network Script

#!/bin/sh
# /etc/init.d/S40network
LOG_FILE="/tmp/S40network.log"

log() {
    echo "[INFO] $1" | tee -a "$LOG_FILE"
}

log "=== Starting S40network ==="
log "=== Disabling USB networking ==="
rm  -rf /etc/init.d/S99usb0config

log "=== Copying Customized files ==="
cp /customised/resolv.conf /etc/
cp /customised/wpa_supplicant.conf /etc/

log "=== Starting S40network ==="
log "Checking for Ethernet link..."
sleep 3

CARRIER=$(cat /sys/class/net/eth0/carrier 2>/dev/null)
log "eth0 carrier = $CARRIER"

if [ "$CARRIER" = "1" ]; then
    log "Ethernet is connected. Using eth0."
    ifconfig eth0 192.168.1.202 netmask 255.255.255.0 up
    ip route del default 2>/dev/null
    ip route add default via 192.168.1.254 dev eth0
else
    log "Ethernet not connected. Falling back to Wifi."
    
    ifconfig eth0 down
    
    log "Killing any existing wpa_supplicant..."
    killall wpa_supplicant 2>/dev/null
    
    log "Starting wpa_supplicant with retry..."
    MAX_RETRIES=3
    TRY=1
    while [ $TRY -le $MAX_RETRIES ]; do
        log "Attempt $TRY: Starting wpa_supplicant..."
        wpa_supplicant -B -i wlan0 -c /etc/wpa_supplicant.conf
        sleep 5
        
        STATUS=$(iw wlan0 link)
        echo "$STATUS" | tee -a "$LOG_FILE"
        echo "$STATUS" | grep -q "Connected"
        if [ $? -eq 0 ]; then
            log "wpa_supplicant connected successfully."
            break
        else
            log "wpa_supplicant not connected, killing and retrying..."
            killall wpa_supplicant 2>/dev/null
        fi
        TRY=$((TRY + 1))
    done
    
    if [ $TRY -gt $MAX_RETRIES ]; then
        log "[ERROR] Failed to connect to Wi-Fi after $MAX_RETRIES attempts."
        exit 1
    fi
    
    log "Configuring wlan0 IP"
    ifconfig wlan0 192.168.1.206 netmask 255.255.255.0 up
    
    log "Deleting existing default route if any"
    ip route del default 2>/dev/null
    
    log "Setting default route for wlan0"
    ip route add default via 192.168.1.254 dev wlan0
    if [ $? -ne 0 ]; then
        log "[ERROR] Failed to add default route for wlan0"
    else
        log "Default route set successfully"
    fi
    
    log "Current route:"
    ip route show | tee -a "$LOG_FILE"
    
    log "Wi-Fi setup complete"
fi

log "Writing resolv.conf"
echo -e "nameserver 8.8.8.8\nnameserver 1.1.1.1" > /etc/resolv.conf

log "S40network script complete."

I then use this modified interfaces file

auto lo
iface lo inet loopback

# eth0 is configured manually by script if link is detected
iface eth0 inet static
    address 192.168.1.202
    netmask 255.255.255.0

# wlan0 is configured manually via init script as well
iface wlan0 inet manual

We need to configure our wifi details in /etc/wpa_supplicant.conf

ctrl_interface=/var/run/wpa_supplicant
ap_scan=1
update_config=1

network={
        ssid="your ssid"
        psk="your key"
        key_mgmt=WPA-PSK
}

I also created a script S99network_fixup, to restorethe default route which appears to get reset by other scripts.

#!/bin/sh
#/etc/init.d/S99network_fixup
echo "FIXUP: Fixing Ethernet configuration..."
echo "[S99network_fixup] Restoring default route for wlan0" >> /var/log/fixroute.log
echo -e "nameserver 8.8.8.8\nnameserver 1.1.1.1" > /etc/resolv.conf
ip route del default 2>/dev/null
ip route add default via 192.168.1.254 dev wlan0
ip route >> /var/log/fixroute.log

The script S21appinit script in luckfox-pico/output/out/rootfs_uclibc_rv1106/etc/init.d launches the RKLunch.sh script starting udhcpc and overriding the static assignment. However, this script loads the required kernel modules so we cannot disable it without loading those modules by some other means. The modifications above are a better option.

We can disable udhcp in the busybox menuconfig in buildroot. This will allow our static interfaces file determine the network settings, but the scripts above are still required.

image

File System Overlay

image

A file system overlay for cusomization can be created as shown above. I created mine in board/pico_max/overlay/etc/network/interfaces

Samba File Transfer

Use the login 'root' with the password you set in buildroot

image

RTSP Streaming

Default URL

rtsp://192.168.1.202/live/0

Loaded Modules Required

rockit                154528 16
rknpu                  22683  1
mpp_vcodec            375759  4 rockit
rga3                   87738  2 rockit
sc3336                  9940  2
os04a10                15198  0
phy_rockchip_csi2_dphy     7674  0
phy_rockchip_csi2_dphy_hw     8161  0
video_rkisp           148435  3 rockit
video_rkcif           130130  3
rk_dvbm                 5813  2 mpp_vcodec,video_rkisp

OEM and Test Scripts

These can be found in

/oem/usr/bin

image

Checking Loaded Modules

[root@luckfox bin]# lsmod
Module                  Size  Used by    Tainted: G
rockit                154528 16
rknpu                  22683  1
mpp_vcodec            375759  4 rockit
rga3                   87738  2 rockit
sc3336                  9940  2
os04a10                15198  0
phy_rockchip_csi2_dphy     7674  0
phy_rockchip_csi2_dphy_hw     8161  0
video_rkisp           148435  3 rockit
video_rkcif           130130  3
rk_dvbm                 5813  2 mpp_vcodec,video_rkisp

Kernel Configuration

vmlinux can be found in: /mnt/buildroot_volume/luckfox-pico/sysdrv/out/bin/board_uclibc_rv1106/vmlinux

Other kernel files can be found in ./out/bin/board_uclibc_rv1106/

TARGET_ARCH=arm
TARGET_UBOOT_CONFIG=luckfox_rv1106_uboot_defconfig rk-emmc.config
TARGET_SPL_CONFIG=
TARGET_KERNEL_CONFIG=luckfox_rv1106_linux_defconfig rv1106-bt.config
TARGET_KERNEL_DTS=rv1106g-luckfox-pico-ultra-w.dts

make[1]: Leaving directory '/mnt/buildroot_volume/luckfox-pico/sysdrv/source/kernel'
make -C /mnt/buildroot_volume/luckfox-pico/sysdrv/source/kernel ARCH=arm savedefconfig
make[1]: Entering directory '/mnt/buildroot_volume/luckfox-pico/sysdrv/source/kernel'
  HOSTCC  scripts/kconfig/conf.o
  HOSTLD  scripts/kconfig/conf
make[1]: Leaving directory '/mnt/buildroot_volume/luckfox-pico/sysdrv/source/kernel'
make: Leaving directory '/mnt/buildroot_volume/luckfox-pico/sysdrv'
[build.sh:info] Kernel Save Defconfig
make: Entering directory '/mnt/buildroot_volume/luckfox-pico/sysdrv'
MDEBUG: KERNEL_DIR = /mnt/buildroot_volume/luckfox-pico/sysdrv/source/kernel
MDEBUG: KERNEL_CFG = luckfox_rv1106_linux_defconfig
cp  /mnt/buildroot_volume/luckfox-pico/sysdrv/source/kernel/defconfig /mnt/buildroot_volume/luckfox-pico/sysdrv/source/kernel/arch/arm/configs/luckfox_rv1106_linux_defconfig
make: Leaving directory '/mnt/buildroot_volume/luckfox-pico/sysdrv'

Kernel Modules

Module Name Description
video_rkcif.ko Rockchip Camera Interface Framework (video capture front-end)
gspca_main.ko Generic Linux USB webcam driver framework (GSPCA)
aic8800_bsp.ko AIC8800 wireless chipset driver (likely Wi-Fi or Bluetooth)
rknpu.ko Rockchip Neural Processing Unit (NPU) driver
aes_generic.ko Generic AES cryptographic algorithm module
rockit.ko Possibly a proprietary Rockchip module (unclear without further context)
sc3336.ko Camera sensor driver (likely for SC3336 CMOS sensor)
rk_dvbm.ko Rockchip Digital Video Broadcasting (DVB) module
mpp_vcodec.ko Rockchip Multi-Media Processing Platform (video codec support)
mac80211.ko Linux 802.11 MAC layer (wireless networking framework)
aic8800_btlpm.ko AIC8800 low-power mode driver
mis5001.ko Camera sensor module (likely for MIS5001 sensor)
ctr.ko Counter mode cryptographic module
rve.ko Rockchip Video Engine (post-processing, composition)
rga3.ko Rockchip Graphics Acceleration 3 (2D raster acceleration)
libarc4.ko ARC4 stream cipher implementation
ghash-generic.ko GHASH algorithm for Galois/Counter Mode (GCM)
phy-rockchip-csi2-dphy-hw.ko Rockchip MIPI CSI-2 D-PHY hardware interface support
ccm.ko Counter with CBC-MAC (CCM) cryptographic module
gf128mul.ko GF(2^128) multiplication support for crypto
ipv6.ko IPv6 network protocol support
cfg80211.ko Linux wireless configuration API
video_rkisp.ko Rockchip Image Signal Processor (ISP) driver
gcm.ko Galois/Counter Mode (GCM) cryptographic support
phy-rockchip-csi2-dphy.ko Rockchip MIPI CSI-2 D-PHY physical layer driver
aic8800_fdrv.ko AIC8800 firmware driver (possibly combined Wi-Fi/BT)

ToolChain File for CMake

toolchain.cmake

# toolchain.cmake

# Set the toolchain prefix
SET(TOOLCHAIN_PREFIX "/mnt/500GB/luckfox-pico/")

# Set the system name and architecture
SET(CMAKE_SYSTEM_NAME Linux)
SET(CMAKE_SYSTEM_PROCESSOR arm)

# Specify the cross-compiler toolchain binaries using the TOOLCHAIN_PREFIX
SET(CMAKE_C_COMPILER "${TOOLCHAIN_PREFIX}/sysdrv/source/buildroot/buildroot-2023.02.6/output/host/bin/arm-rockchip830-linux-uclibcgnueabihf-gcc")
SET(CMAKE_CXX_COMPILER "${TOOLCHAIN_PREFIX}/sysdrv/source/buildroot/buildroot-2023.02.6/output/host/bin/arm-rockchip830-linux-uclibcgnueabihf-g++")
SET(CMAKE_AR "${TOOLCHAIN_PREFIX}/sysdrv/source/buildroot/buildroot-2023.02.6/output/host/bin/arm-rockchip830-linux-uclibcgnueabihf-ar")
SET(CMAKE_ASM_COMPILER "${TOOLCHAIN_PREFIX}/sysdrv/source/buildroot/buildroot-2023.02.6/output/host/bin/arm-rockchip830-linux-uclibcgnueabihf-as")

# Specify the system root directory (this is the root filesystem of the target system)
SET(CMAKE_FIND_ROOT_PATH "${TOOLCHAIN_PREFIX}/sysdrv/source/buildroot/buildroot-2023.02.6/output/host/arm-rockchip830-linux-uclibcgnueabihf/sysroot")

# Direct CMake to search for headers and libraries in the sysroot first
SET(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
SET(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
SET(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)

# Specify the build type (optional)
SET(CMAKE_BUILD_TYPE Release)

# Specify linker flags if necessary (optional)
SET(CMAKE_EXE_LINKER_FLAGS "-static-libgcc -static-libstdc++")

Toolchain

Default Toolchain

../../../tools/linux/toolchain/arm-rockchip830-linux-uclibcgnueabihf

C Library

image

Example Code

There are examples in C for the board runtime in ./rknn-toolkit2/rknpu2/examples/. The equivalent host side example is at ./rknn-toolkit2/rknn_toolkit_lite2/examples/

More extensive examples are in ./rknn-toolkit2/examples/

Streaming Examples

export LUCKFOX_SDK_PATH=/mnt/500GB/luckfox-pico/
./build.sh

The source for the example is in:

./luckfox_pico_rkmpi_example/example/luckfox_pico_rtsp_yolov5/src

Documentation is here

Summary of what does and does not work

The SDK I am using is here: https://github.com/LuckfoxTECH/luckfox-pico.git

What does not work

  1. Enabling OpenCV in the buildroot build. - It appears this version of openCV is not compatible with the board. I cannot find any examples that use this version and produce usable results. Luckfox recommends using the lightweight version, OpenCV-Mobile.

  2. The rknn_toolkit_lite2 toolkit which comes as part of the RKNN-toolkit available at: https://github.com/airockchip/rknn-toolkit2 is not suitable for the Pico Ultra. I think this means we cannot run python code on the Pico Ultra board itself that can talk with the NPU, but this is not clear in the documentation. In fact the documentation states: RKNN-Toolkit-Lite2 provides Python programming interfaces for Rockchip NPU platform to help users deploy RKNN models and accelerate the implementation of AI applications.And under that it says: Support Platform, and then lists RV1103/RV1106.

What does work

  1. There are a series of examples that are available here: https://github.com/LuckfoxTECH/luckfox_ ... pi_example. These appear to work and use a statically compiled libraries, including a library for openCV which works. It would be nice to be able to recompile these libraries and add some more OpenCV modules if possible.

  2. There is the RKNN-toolkit available at: https://github.com/airockchip/rknn-toolkit2/. This is a software development kit for users to perform model conversion, inference and performance evaluation on PC and Rockchip NPU platforms. There is a directory RKNN/rknn-toolkit2/rknpu2/examples which has a number of interesting examples, but it seems most of these are not suitable for the Pico Ultra. There is a subdirectory called RV1106_RV1103 which has two examples: rknn_mobilenet_demo and rknn_yolov5_demo. These work on the Ultra, but do not use OpenCV or show how to interface with the camera. They take static images and use the NPU to infer object detection.

What is not clear

  1. There are a number of code samples in the SDK at: luckflox-pico/media/samples/example which appear to be Rockchip camera test utilities of various sorts. These are all low level and do not use OpenCV, but do give examples of capturing video frames from /dev/video(n). Unfortunately, these is no documentation with these samples. I don't know if they are relevant for the Pico Ultra.

Demo Code

The board runs a demo program that streams video over an RTSP stream

RkLunch-stop.sh #stops the demo
RkLunch.sh # starts the demo

One can also use

killall rkipc

RkLunch.sh

  • Checks various init scripts (none of which exist)
  • Checks various symbolic links (none of which exist)
  • Launches rkipc depending on the camera type it finds.

Examples

MobileNet Demo

MobileNet-Demo refers to a demonstration or implementation of MobileNet, a lightweight deep learning model architecture designed for mobile and edge devices. MobileNet models are known for their efficiency and low computational cost while maintaining high accuracy in various computer vision tasks like image classification, object detection, and segmentation.

MobileNet is optimized for use on devices with limited processing power and memory. It can be used to perform inference tasks in real-time, making it suitable for applications like augmented reality (AR), robotics, or live video analysis. and is efficient enough to run on smartphones and embedded devices without needing high-end GPUs or cloud-based infrastructure.

MobileNet Architecture

MobileNet is a family of models designed by Google for mobile and embedded devices, optimized to run efficiently on hardware with limited resources (such as mobile phones, IoT devices, and other low-power hardware).

The architecture uses depthwise separable convolutions, which reduce the number of parameters and computational requirements compared to traditional convolutions.

MobileNet provides different versions such as MobileNetV1, MobileNetV2, and MobileNetV3, with each successive version offering better performance, accuracy, and efficiency.

Use Cases

Image Classification: Identifying objects or scenes in images with MobileNet on mobile devices. Object Detection: Detecting and classifying multiple objects in an image or video stream. Segmentation: Dividing an image into regions for understanding its structure. On-device AI: MobileNet is ideal for edge AI where on-device processing is necessary, reducing the need for constant cloud communication.

C code for the board

rknn-toolkit2/rknpu2/examples/RV1106_RV1103/rknn_mobilenet_demo

Python Host Side Code

./rknn-toolkit2/rknn-toolkit2/examples/tflite/mobilenet_v1

Running the demo

./rknn_mobilenet_demo ./model/RV1106/mobilenet_v1.rknn model/dog_224x224.jpg
./rknn_mobilenet_demo ./model/RV1106/mobilenet_v1.rknn model/cat_224x224.jpg

Output


[DEBUG] Model initialization status: SUCCESS
rknn_api/rknnrt version: 1.6.0 (2de554906@2024-01-17T14:53:41), driver version: 0.9.2
model input num: 1, output num: 1
[DEBUG] Model input/output numbers seem valid.
input tensors:
  index=0, name=input:0, n_dims=4, dims=[1, 224, 224, 3], n_elems=150528, size=150528, fmt=NHWC, type=UINT8, qnt_type=AFFINE, zp=0, scale=0.007812
[DEBUG] Input tensor 0 attributes dumped.
output tensors:
  index=0, name=MobilenetV1/Logits/SpatialSqueeze:0, n_dims=4, dims=[1, 1, 1, 1008], n_elems=1008, size=1008, fmt=NHWC, type=INT8, qnt_type=AFFINE, zp=-53, scale=0.108075
[DEBUG] Output tensor 0 attributes dumped.
custom string:
[DEBUG] Loaded image properties:
[DEBUG] Original image dimensions: 224x224x3
[DEBUG] Required image dimensions (from model input attr): 224x224x3
[DEBUG] Image loaded and processed successfully.
[DEBUG] Input data copied directly (width == stride).
[DEBUG] Input tensor memory set successfully.
[DEBUG] Output tensor 0 memory set successfully.
Begin perf ...
   0: Elapse Time = 8.93ms, FPS = 112.04
[DEBUG] RKNN run for loop 0 completed.
[DEBUG] First few values of output tensor 0: -3.999451, -2.053772, -4.648010...
19.456787 - 156 - Blenheim spaniel
16.646362 - 155 - Shih-Tzu
13.727844 - 205 - flat-coated retriever
12.430725 - 284 - Siamese cat, Siamese
10.376953 - 260 - chow, chow chow
[DEBUG] RKNN input and output memories destroyed.
[DEBUG] RKNN context destroyed.
[DEBUG] Input image data freed.
[DEBUG] Model initialization status: SUCCESS
rknn_api/rknnrt version: 1.6.0 (2de554906@2024-01-17T14:53:41), driver version: 0.9.2
model input num: 1, output num: 1
[DEBUG] Model input/output numbers seem valid.
input tensors:
  index=0, name=input:0, n_dims=4, dims=[1, 224, 224, 3], n_elems=150528, size=150528, fmt=NHWC, type=UINT8, qnt_type=AFFINE, zp=0, scale=0.007812
[DEBUG] Input tensor 0 attributes dumped.
output tensors:
  index=0, name=MobilenetV1/Logits/SpatialSqueeze:0, n_dims=4, dims=[1, 1, 1, 1008], n_elems=1008, size=1008, fmt=NHWC, type=INT8, qnt_type=AFFINE, zp=-53, scale=0.108075
[DEBUG] Output tensor 0 attributes dumped.
custom string:
[DEBUG] Loaded image properties:
[DEBUG] Original image dimensions: 224x224x3
[DEBUG] Required image dimensions (from model input attr): 224x224x3
[DEBUG] Image loaded and processed successfully.
[DEBUG] Input data copied directly (width == stride).
[DEBUG] Input tensor memory set successfully.
[DEBUG] Output tensor 0 memory set successfully.
Begin perf ...
   0: Elapse Time = 8.90ms, FPS = 112.31
[DEBUG] RKNN run for loop 0 completed.
[DEBUG] First few values of output tensor 0: -3.458984, -2.486145, 0.108093...
12.322632 - 283 - Persian cat
11.565979 - 282 - tiger cat
11.565979 - 286 - cougar, puma, catamount, mountain lion, painter, panther, Felis concolor
10.052673 - 278 - kit fox, Vulpes macrotis
9.728394 - 279 - Arctic fox, white fox, Alopex lagopus
[DEBUG] RKNN input and output memories destroyed.
[DEBUG] RKNN context destroyed.
[DEBUG] Input image data freed.

Yolo5 Demo

C code for the board is in

./rknn-toolkit2/rknpu2/examples/RV1106_RV1103/rknn_yolov5_demo`

Kernel Dependencies

insmod rknpu.ko

Dependencies

  • The demo currently requires a directory model which contains the list of object classes: coco_80_labels_list.txt model
  • Command Line: ./rknn_yolov5_demo ./model/RV1106/yolov5s-640-640.rknn model/image_640_640.jpg ./model/coco_80_labels_list.txt
  • Expected output:
Initialised Model Successfully
rknn_api/rknnrt version: 1.4.1b9 (09eb4be80@2022-10-19T09:51:39), driver version: 0.8.2
model input num: 1, output num: 3
input tensors:
  index=0, name=images, n_dims=4, dims=[1, 640, 640, 3], n_elems=1228800, size=1228800, fmt=NHWC, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003922
output tensors:
  index=0, name=output, n_dims=4, dims=[1, 80, 80, 255], n_elems=1632000, size=1632000, fmt=NHWC, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003860
  index=1, name=283, n_dims=4, dims=[1, 40, 40, 255], n_elems=408000, size=408000, fmt=NHWC, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003922
  index=2, name=285, n_dims=4, dims=[1, 20, 20, 255], n_elems=102000, size=102000, fmt=NHWC, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003915
custom string:
Our model requires NHWC format with height=640,width=640,channels=3
Our image loaded with height=640,width=640,channels=3
Loaded input data from path : bus.jpg
Creating tensor input memory
Copying input data to input tensor memory: width =640, stride=640
Creating tensor output memory
Setting tensor input memory
Setting tensor output memory
Begin perf ...
   0: Elapse Time = 80.04ms, FPS = 12.49
model is NHWC input format
Post processing data
Confidence Threshold: 0.250000
Non-Maximum Suppression (NMS) Threshold: 0.250000
Scale Width: : 1.000000 Scale Height 1.000000
Loading labels
loadLabelName ./model/coco_80_labels_list.txt
Valid Count for Stride 8 : 0
Valid Count for Stride 16 : 13
Valid Count for Stride 32 : 35
result  0: ( 208,  244,  286,  506), person
result  3: ( 479,  238,  560,  526), person
result  8: ( 110,  236,  230,  535), person
result 21: (  94,  130,  553,  464), bus
result 42: (  79,  354,  122,  516), person
person @ (208 244 286 506) 0.884136
person @ (479 238 560 526) 0.863766
person @ (110 236 230 535) 0.832498
bus @ (94 130 553 464) 0.697389
person @ (79 354 122 516) 0.349307
Releasing rknn memory

Camera

OpenCV Compatibility

Cmake for the example:

project(opencv-mobile-test)
cmake_minimum_required(VERSION 3.5)
set(CMAKE_CXX_STANDARD 11)

SET(CMAKE_C_COMPILER "<SDK Directory>/tools/linux/toolchain/arm-rockchip830-linux-uclibcgnueabihf/bin/arm-rockchip830-linux-uclibcgnueabihf-gcc")
SET(CMAKE_CXX_COMPILER "<SDK Directory>/tools/linux/toolchain/arm-rockchip830-linux-uclibcgnueabihf/bin/arm-rockchip830-linux-uclibcgnueabihf-g++")

set(OpenCV_DIR "${CMAKE_CURRENT_SOURCE_DIR}/opencv-mobile-4.8.1-luckfox-pico/lib/cmake/opencv4")
find_package(OpenCV REQUIRED)
include_directories(${OpenCV_INCLUDE_DIRS})

add_executable(opencv-mobile-test main.cpp)

target_link_libraries(opencv-mobile-test ${OpenCV_LIBS})

Basic C code for the example:

#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/imgproc/imgproc.hpp>

#include <unistd.h>   // sleep()

int main()
{
    cv::VideoCapture cap;
    cap.set(cv::CAP_PROP_FRAME_WIDTH, 320);
    cap.set(cv::CAP_PROP_FRAME_HEIGHT, 240);
    cap.open(0);

    const int w = cap.get(cv::CAP_PROP_FRAME_WIDTH);
    const int h = cap.get(cv::CAP_PROP_FRAME_HEIGHT);
    fprintf(stderr, "%d x %d\n", w, h);

    cv::Mat bgr[9];
    for (int i = 0; i < 9; i++)
    {
        cap >> bgr[i];

        sleep(1);
    }

    cap.release();

    // combine into big image
    {
        cv::Mat out(h * 3, w * 3, CV_8UC3);
        bgr[0].copyTo(out(cv::Rect(0, 0, w, h)));
        bgr[1].copyTo(out(cv::Rect(w, 0, w, h)));
        bgr[2].copyTo(out(cv::Rect(w * 2, 0, w, h)));
        bgr[3].copyTo(out(cv::Rect(0, h, w, h)));
        bgr[4].copyTo(out(cv::Rect(w, h, w, h)));
        bgr[5].copyTo(out(cv::Rect(w * 2, h, w, h)));
        bgr[6].copyTo(out(cv::Rect(0, h * 2, w, h)));
        bgr[7].copyTo(out(cv::Rect(w, h * 2, w, h)));
        bgr[8].copyTo(out(cv::Rect(w * 2, h * 2, w, h)));

        cv::imwrite("out.jpg", out);
    }

    return 0;
}

Interboard Comms

Serial

Run luckfox-config and select UART. Enable Uart4

Then confirm on the board that its available

ls /dev/ttyS*
/dev/ttyS1
/dev/ttyS3

Use the stty tool to query its communication parameters:

# stty -F /dev/ttyS3
speed 9600 baud; line = 0; -brkint -imaxbel

Modify the baud rate, where ispeed is the input speed, and ospeed is the output speed:

# stty -F /dev/ttyS3 ispeed 115200 ospeed 115200

Disable echo:

stty -F /dev/ttyS3 -echo

image

⚠️ **GitHub.com Fallback** ⚠️