Luckfox Pico Pro Max Linux RV1106 Rockchip AI Development Board - johnosbb/Automation GitHub Wiki
Luckfox Pico Pro Max Linux RV1106 Rockchip AI Development Board
- Manufacturers Website
- Quick Start Guide
- Firmware Link
- GitHub Repo for LuckFox AI examples
- Youtube Introduction to Pico Max
- RKNN Model Zoo
- yolov5 on LuckFox
- Computer Vision on Pico Max
- RKMPI-example
- Rockchip RKNPU User Guide
- How to Change the RK YOLOv5 Demo to Capture-Real Images
- Rockchip RV1106 User Manual GPIO
| Processor | Cortex [email protected] |
|---|---|
| NPU | 0.5TOPS, supports int4, int8 and int16 |
| ISP | Input 5M @30fps (Max) |
| Memory | Pro: 128MB DDR2Max: 256MB DDR2 |
| USB | USB 2.0 Host/Device |
| Camera | MIPI CSI 2-lane |
| GPIO | 26 × GPIO pins |
| Ethernet port | 10/100M Ethernet controller and embedded PHY |
| Default Storage | SPI NAND FLASH (256MB) |
Serial Connections
Serial connection is via an FTDI connector with the connections shown below
Serial Access via Putty
Login as root and use the luckfox password you configured in buildroot.
SSH Access via Putty
Login as root and use the luckfox password you configured in buildroot.
Buildroot
Install dependencies
sudo apt-get install -y git ssh make gcc gcc-multilib g++-multilib module-assistant expect g++ gawk texinfo libssl-dev bison flex fakeroot cmake unzip gperf autoconf device-tree-compiler libncurses5-dev pkg-config bc python-is-python3 passwd openssl openssh-server openssh-client vim file cpio rsync
Install the SDK
git clone https://github.com/LuckfoxTECH/luckfox-pico.git
Set Environment variables
cd {SDK_PATH}/tools/linux/toolchain/arm-rockchip830-linux-uclibcgnueabihf/
source env_install_toolchain.sh
Build Image
./build.sh lunch # Select the reference board configuration
./build.sh # One-click automatic compilation
Configuring Buildroot
./build.sh buildrootconfig
To regenerate the RootfS
./build.sh clean rootfs
./build.sh rootfs
./build.sh firmware
For busybox, in the directory sysdrv/source/buildroot/buildroot-2023.02.6
make busybox-menuconfig
buildroot_defconfig in luckfox-pico/config
BR2_arm=y
BR2_cortex_a7=y
BR2_TOOLCHAIN_EXTERNAL=y
BR2_TOOLCHAIN_EXTERNAL_CUSTOM=y
BR2_TOOLCHAIN_EXTERNAL_PATH="../../../../tools/linux/toolchain/arm-rockchip830-linux-uclibcgnueabihf"
BR2_TOOLCHAIN_EXTERNAL_CUSTOM_PREFIX="arm-rockchip830-linux-uclibcgnueabihf"
BR2_TOOLCHAIN_EXTERNAL_GCC_8=y
BR2_TOOLCHAIN_EXTERNAL_HEADERS_5_10=y
BR2_TOOLCHAIN_EXTERNAL_WCHAR=y
BR2_TOOLCHAIN_EXTERNAL_HAS_SSP=y
BR2_TOOLCHAIN_EXTERNAL_CXX=y
BR2_PRIMARY_SITE="http://sources.buildroot.net"
BR2_OPTIMIZE_3=y
BR2_TARGET_GENERIC_HOSTNAME="luckfox pico"
BR2_TARGET_GENERIC_ISSUE="Welcome to luckfox pico"
BR2_TARGET_GENERIC_ROOT_PASSWD="luckfox"
BR2_ROOTFS_OVERLAY="board/pico_max/overlay"
BR2_PACKAGE_BUSYBOX_SHOW_OTHERS=y
BR2_PACKAGE_E2FSPROGS_RESIZE2FS=y
BR2_PACKAGE_PYTHON3=y
BR2_PACKAGE_PYTHON3_SSL=y
BR2_PACKAGE_PYTHON_AIOHTTP=y
BR2_PACKAGE_PYTHON_CLICK=y
BR2_PACKAGE_PYTHON_ITSDANGEROUS=y
BR2_PACKAGE_PYTHON_JINJA2=y
BR2_PACKAGE_PYTHON_PERIPHERY=y
BR2_PACKAGE_PYTHON_PILLOW=y
BR2_PACKAGE_PYTHON_SERIAL=y
BR2_PACKAGE_PYTHON_SETUPTOOLS=y
BR2_PACKAGE_PYTHON_SMBUS_CFFI=y
BR2_PACKAGE_PYTHON_SPIDEV=y
BR2_PACKAGE_PYTHON_WERKZEUG=y
BR2_PACKAGE_STB=y
BR2_PACKAGE_DTC=y
BR2_PACKAGE_DTC_PROGRAMS=y
BR2_PACKAGE_LIBV4L=y
BR2_PACKAGE_LIBV4L_UTILS=y
BR2_PACKAGE_LRZSZ=y
BR2_PACKAGE_NTP=y
BR2_PACKAGE_OPENSSH=y
BR2_PACKAGE_SAMBA4=y
BR2_PACKAGE_BASH=y
BR2_PACKAGE_BASH_LOADABLE_EXAMPLES=y
BR2_PACKAGE_DIALOG=y
BR2_PACKAGE_TIME=y
BR2_PACKAGE_HTOP=y
BR2_PACKAGE_NANO=y
BR2_PACKAGE_DARKNET_EMBEDDED=y
BR2_PACKAGE_HOST_DTC=y
Output Image Components
The Image output from the build system can be found in:
luckfox-pico/out/image
boot.img download.bin env.img idblock.img oem.img rootfs.img sd_update.txt tftp_update.txt uboot.img update.img userdata.img
Creating the SD Card
On Windows we use the application: SocToolKit.exe in SocTooKit to download the image.
- We omit the update.img file
- If the Card is not detected one may have to unplug and re-plug the card reader
Programming Nand Flash
Flash Organisation
The system uses UBI to manage the NAND flash. UBI is a layer that sits between the raw NAND and the filesystem, providing wear leveling and bad block management.
Several UBI devices and volumes are mounted:
- ubi0:0: Mounted as the root filesystem (rootfs) (read only).
- ubi4:0: Mounted with the name "oem".
- ubi5:0: Mounted with the name "userdata".
Flash Tools
Enable ubi utilities in busybox
Configure the network in /etc/network/interfaces
# interface file auto-generated by buildroot
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet static
address 192.168.1.202
netmask 255.255.255.0
gateway 192.168.1.254
One has to disable the S21appinit script in luckfox-pico/output/out/rootfs_uclibc_rv1106/etc/init.d in order to stop the RKLunch.sh script starting udhcpc and overriding the static assignment.
We can also disable udhcp in the busybox menuconfig in buildroot. This will allow our static interfaces file determine the network settings.
File System Overlay
A file system overlay for cusomization can be created as shown above.
I created mine in board/pico_max/overlay/etc/network/interfaces
Samba File Transfer
Use the login 'root' with the password you set in buildroot
RTSP Streaming
Default URL
rtsp://192.168.1.202/live/0
Loaded Modules Required
rockit 154528 16
rknpu 22683 1
mpp_vcodec 375759 4 rockit
rga3 87738 2 rockit
sc3336 9940 2
os04a10 15198 0
phy_rockchip_csi2_dphy 7674 0
phy_rockchip_csi2_dphy_hw 8161 0
video_rkisp 148435 3 rockit
video_rkcif 130130 3
rk_dvbm 5813 2 mpp_vcodec,video_rkisp
OEM and Test Scripts
These can be found in
/oem/usr/bin
Checking Loaded Modules
[root@luckfox bin]# lsmod
Module Size Used by Tainted: G
rockit 154528 16
rknpu 22683 1
mpp_vcodec 375759 4 rockit
rga3 87738 2 rockit
sc3336 9940 2
os04a10 15198 0
phy_rockchip_csi2_dphy 7674 0
phy_rockchip_csi2_dphy_hw 8161 0
video_rkisp 148435 3 rockit
video_rkcif 130130 3
rk_dvbm 5813 2 mpp_vcodec,video_rkisp
ToolChain File for CMake
toolchain.cmake
# toolchain.cmake
# Set the toolchain prefix
SET(TOOLCHAIN_PREFIX "/mnt/500GB/luckfox-pico/")
# Set the system name and architecture
SET(CMAKE_SYSTEM_NAME Linux)
SET(CMAKE_SYSTEM_PROCESSOR arm)
# Specify the cross-compiler toolchain binaries using the TOOLCHAIN_PREFIX
SET(CMAKE_C_COMPILER "${TOOLCHAIN_PREFIX}/sysdrv/source/buildroot/buildroot-2023.02.6/output/host/bin/arm-rockchip830-linux-uclibcgnueabihf-gcc")
SET(CMAKE_CXX_COMPILER "${TOOLCHAIN_PREFIX}/sysdrv/source/buildroot/buildroot-2023.02.6/output/host/bin/arm-rockchip830-linux-uclibcgnueabihf-g++")
SET(CMAKE_AR "${TOOLCHAIN_PREFIX}/sysdrv/source/buildroot/buildroot-2023.02.6/output/host/bin/arm-rockchip830-linux-uclibcgnueabihf-ar")
SET(CMAKE_ASM_COMPILER "${TOOLCHAIN_PREFIX}/sysdrv/source/buildroot/buildroot-2023.02.6/output/host/bin/arm-rockchip830-linux-uclibcgnueabihf-as")
# Specify the system root directory (this is the root filesystem of the target system)
SET(CMAKE_FIND_ROOT_PATH "${TOOLCHAIN_PREFIX}/sysdrv/source/buildroot/buildroot-2023.02.6/output/host/arm-rockchip830-linux-uclibcgnueabihf/sysroot")
# Direct CMake to search for headers and libraries in the sysroot first
SET(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
SET(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
SET(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
# Specify the build type (optional)
SET(CMAKE_BUILD_TYPE Release)
# Specify linker flags if necessary (optional)
SET(CMAKE_EXE_LINKER_FLAGS "-static-libgcc -static-libstdc++")
Toolchain
Defualt Toolchain
../../../tools/linux/toolchain/arm-rockchip830-linux-uclibcgnueabihf
C Library
Example Code
There are examples in C for the board runtime in ./rknn-toolkit2/rknpu2/examples/. The equivalent host side example is at ./rknn-toolkit2/rknn_toolkit_lite2/examples/
More extensive examples are in ./rknn-toolkit2/examples/
Streaming Examples
- Clone the repo luckfox_pico_rkmpi_example
export LUCKFOX_SDK_PATH=/mnt/500GB/luckfox-pico/
./build.sh
The source for the example is in:
example/luckfox_pico_rtsp_yolov5/src
Documentation is here
Demo Code
The board runs a demo program that streams video over an RTSP stream
RkLunch-stop.sh #stops the demo
RkLunch.sh # starts the demo
One can also use
killall rkipc
RkLunch.sh
- Checks various init scripts (none of which exist)
- Checks various symbolic links (none of which exist)
- Launches rkipc depending on the camera type it finds.
Examples
MobileNet Demo
MobileNet-Demo refers to a demonstration or implementation of MobileNet, a lightweight deep learning model architecture designed for mobile and edge devices. MobileNet models are known for their efficiency and low computational cost while maintaining high accuracy in various computer vision tasks like image classification, object detection, and segmentation.
MobileNet is optimized for use on devices with limited processing power and memory. It can be used to perform inference tasks in real-time, making it suitable for applications like augmented reality (AR), robotics, or live video analysis. and is efficient enough to run on smartphones and embedded devices without needing high-end GPUs or cloud-based infrastructure.
MobileNet Architecture
MobileNet is a family of models designed by Google for mobile and embedded devices, optimized to run efficiently on hardware with limited resources (such as mobile phones, IoT devices, and other low-power hardware).
The architecture uses depthwise separable convolutions, which reduce the number of parameters and computational requirements compared to traditional convolutions.
MobileNet provides different versions such as MobileNetV1, MobileNetV2, and MobileNetV3, with each successive version offering better performance, accuracy, and efficiency.
Use Cases
Image Classification: Identifying objects or scenes in images with MobileNet on mobile devices. Object Detection: Detecting and classifying multiple objects in an image or video stream. Segmentation: Dividing an image into regions for understanding its structure. On-device AI: MobileNet is ideal for edge AI where on-device processing is necessary, reducing the need for constant cloud communication.
C code for the board
rknn-toolkit2/rknpu2/examples/RV1106_RV1103/rknn_mobilenet_demo
Python Host Side Code
./rknn-toolkit2/rknn-toolkit2/examples/tflite/mobilenet_v1
Yolo5 Demo
C code for the board is in
./rknn-toolkit2/rknpu2/examples/RV1106_RV1103/rknn_yolov5_demo`
Kernel Dependencies
insmod rknpu.ko
Dependencies
- The demo currently requires a directory model which contains the list of object classes: coco_80_labels_list.txt model
- Command Line:
./rknn_yolov5_demo yolov5s-640-640.rknn bus.jpg - Expected output:
Initialised Model Successfully
rknn_api/rknnrt version: 1.4.1b9 (09eb4be80@2022-10-19T09:51:39), driver version: 0.8.2
model input num: 1, output num: 3
input tensors:
index=0, name=images, n_dims=4, dims=[1, 640, 640, 3], n_elems=1228800, size=1228800, fmt=NHWC, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003922
output tensors:
index=0, name=output, n_dims=4, dims=[1, 80, 80, 255], n_elems=1632000, size=1632000, fmt=NHWC, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003860
index=1, name=283, n_dims=4, dims=[1, 40, 40, 255], n_elems=408000, size=408000, fmt=NHWC, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003922
index=2, name=285, n_dims=4, dims=[1, 20, 20, 255], n_elems=102000, size=102000, fmt=NHWC, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003915
custom string:
Our model requires NHWC format with height=640,width=640,channels=3
Our image loaded with height=640,width=640,channels=3
Loaded input data from path : bus.jpg
Creating tensor input memory
Copying input data to input tensor memory: width =640, stride=640
Creating tensor output memory
Setting tensor input memory
Setting tensor output memory
Begin perf ...
0: Elapse Time = 80.04ms, FPS = 12.49
model is NHWC input format
Post processing data
Confidence Threshold: 0.250000
Non-Maximum Suppression (NMS) Threshold: 0.250000
Scale Width: : 1.000000 Scale Height 1.000000
Loading labels
loadLabelName ./model/coco_80_labels_list.txt
Valid Count for Stride 8 : 0
Valid Count for Stride 16 : 13
Valid Count for Stride 32 : 35
result 0: ( 208, 244, 286, 506), person
result 3: ( 479, 238, 560, 526), person
result 8: ( 110, 236, 230, 535), person
result 21: ( 94, 130, 553, 464), bus
result 42: ( 79, 354, 122, 516), person
person @ (208 244 286 506) 0.884136
person @ (479 238 560 526) 0.863766
person @ (110 236 230 535) 0.832498
bus @ (94 130 553 464) 0.697389
person @ (79 354 122 516) 0.349307
Releasing rknn memory