Tutorial： How to port your network to AI FPGA module - DigitalMediaProfessionals/dv-sdk GitHub Wiki

This tutorial use this repository to demonstrate how to port a CNN model. One can get it using the following command:

$ git clone https://github.com/DigitalMediaProfessionals/dv-tutorial

This tutorial introduces how to port a Caffe network to AI FPGA module (referred to as MODULE) with AI processor, using a network definition file mobilenet_deploy.prototxt and a network weight file mobilenet.caffemodel in the Caffe repository.

Please setup the FPGA environment according to Quick Start in advance. Hereafter, it is assumed that SDK is installed under ~/dv-sdk/ and you are in ~/dv-tutorial/ in your PC and MODULE.

Export Network

Export your network. For how to export, see this page. In this tutorial, we use mobilenet_deploy.prototxt and mobilenet.caffemodel instread of your network.

Convert Network

We use the DMP tool for converting the network. Please do the conversion on a PC because resources on MODULE is limited. For setup for the tool, please refer to this page.

At first, please put mobilenet_deploy.prototxt and mobilenet.caffemodel under ~/dv-tutorial.

The tool requires a configuration file. Please create a configuration file tool.ini with a below content. (Configuration in tool.ini is minimum. For more functionalities, see Input Files to the DMP tool.)

; tool.ini
; This file is configuration for DMP tool

[INPUT]
name = CaffeModileNet                     ; 
origin = CAFFE                            ; original deep learning framework
definition = mobilenet_deploy.prototxt    ; network definition file
data = mobilenet.caffemodel               ; network weight file

[OUTPUT]
output_folder = ./     ; Output folder is under a directory where this .ini file exists
generate_source = 1

After this you convert the network by the next command.

$ python3 ~/dv-sdk/tool/convertor.py tool.ini

Then source code of the network (CaffeModileNet_gen.cpp and CaffeModileNet_gen.h) and weight file (CaffeModileNet_weights.bin) is created under CaffeModileNet. Current directory composition must be as below. (The generated files have a prefix as the same as a string specified in name field of tool.ini.)

$ tree
.
|-- CaffeModileNet                    # generated directory
|   |-- CaffeModileNet_gen.cpp        # source code of the network
|   |-- CaffeModileNet_gen.h          # source code of the network
|   |-- CaffeModileNet_weights.bin    # weight file of the network
|-- mobilenet.caffemodel
|-- mobilenet_deploy.prototxt
|-- tool.ini

Write Application Code for AI processor

Let's create an application using the network. This section explains how to use the network by referring to a sample program for simplicity. The sample program simply shows object classes for images specified in command line arguments. Please see sample code after this and put files in tutorial repository under ~/dv/tutorial/.

overview

A flow of the sample program is as below.

  1. Initialize network.
  2. For each image files
    1. read and preprocess image
    2. pass the preprocessed data to the network and execute it
    3. get and output a result from the network

Initialzie network

init_net() initializes the network. This step initializes the network, loads network weight, commit the network and get memory address for network input.

int init_net(CCaffeMobileNet &net, void **input_addr)
{
	if(!net.Initialize()){
		cerr << "fail to initialize network" << endl;
		return -1;
	}
	if(!net.LoadWeights("CaffeMobileNet/CaffeMobileNet_weights.bin")){
		cerr << "fail to load weight" << endl;
		return -1;
	}
	if(!net.Commit()){
		cerr << "fail to commit network" << endl;
		return -1;
	}
	*input_addr = net.get_network_input_addr_cpu();
	return 0;
}

read and preprocess image

read_and_preprocess_image() reads and preprocesses an image. NOTE the following points when preprocessing an image for the network.

  • transpose an image. This page shows the reason.
  • cast an image to FP16 because an input type of the network is FP16.
  • normalize an image according to your network.
// preprocessing
void preproc_image(const uint8_t *src, __fp16 *dst, size_t width, size_t height)
{
	for(size_t c = 0; c < width; c++){
		for(size_t r = 0; r < height; r++){
			dst[(c * height + r) * 3 + 0] = (__fp16)(src[(r * width + c) * 3 + 0] - 128);
			dst[(c * height + r) * 3 + 1] = (__fp16)(src[(r * width + c) * 3 + 1] - 128);
			dst[(c * height + r) * 3 + 2] = (__fp16)(src[(r * width + c) * 3 + 2] - 128);
		}
	}
}

Execute a Network

To execute a network, we call RunNetwork(). Writing an input data to a memory region for a network input, which we get in advance, passes the data to a network.

                // input image and run network
		memcpy(net_input_addr, input_buf, width * height * 6);
		if(!net.RunNetwork()){
			cerr << "fail to run network" << endl;
			ret = -1;
			goto fin_loop;
                }

Get A Result from The Network

We get a result of a network by get_final_output(). Finally we post-process the output and utilize that.

		// get and output result
		net.get_final_output(output);
　　　　　　　　　cout << img_path << "," << categories[argmax(output)] << endl

Build and Execute Application

Eventually, we build and execute an application.

At first we prepare files required for an application. Please put files required for the source code generated by the tool and main.cpp as below.

$ # copy files the network source depends on 
$ cp ~/dv-sdk/application/common/src/dmp_network.cpp \
  ~/dv-sdk/application/common/include/dmp_network.h \
  ~/dv-sdk/application/common/include/stats.h \
  ~/dv-tutorial/src/
$ # copy file main.cpp depends on
$ cp ~/dv-sdk/application/CaffeMobileNet/imagenet_1000_categories.h ~/dv-tutorial/src/

Copy ~/dv/tutorial/ to MODULE and create mainexecution file bymake -j4`.

$ # on MODULE
$ make -j4

After this, execute the program on MODULE by sudo bash run.sh. run.sh predicts classes of images that is used in ~/dv-sdk/application. If the output is like the below, the application successed.

$ # on MODULE
$ sudo bash run.sh
/home/ubuntu/dv-sdk/application/bin/images/img01.jpg , notebook, notebook computer
/home/ubuntu/dv-sdk/application/bin/images/img02.jpg , cup
/home/ubuntu/dv-sdk/application/bin/images/img03.jpg , airliner
/home/ubuntu/dv-sdk/application/bin/images/img04.jpg , Granny Smith
/home/ubuntu/dv-sdk/application/bin/images/img05.jpg , bald eagle, American eagle, Haliaeetus leucocephalus
/home/ubuntu/dv-sdk/application/bin/images/img06.jpg , coil, spiral, volute, whorl, helix
/home/ubuntu/dv-sdk/application/bin/images/img07.jpg , baseball
/home/ubuntu/dv-sdk/application/bin/images/img08.jpg , baseball
/home/ubuntu/dv-sdk/application/bin/images/img09.jpg , basketball
/home/ubuntu/dv-sdk/application/bin/images/img10.jpg , folding chair
...