1Overview

:::info 文档信息

页数：50　·　语言：英文

原始文档：Rockchip_User_Guide_RKNN_Toolkit2_EN-1.2.0.pdf

:::

章节大纲

• 1Overview（P5）
  • 1.1Main function description（P5）
  • 1.2Applicable chip model（P6）
  • 1.3Applicable Operating System（P7）
• 2Requirements/Dependencies（P8）
• 3User Guide（P9）
  • 3.1Installation（P9）
    • 3.1.1Install by pip command（P9）
    • 3.1.2Install by the Docker Image（P9）
  • 3.2Usage of RKNN-Toolkit2（P10）
    • 3.2.1Scenario 1: Inference for Simulation on PC（P11）
      • 3.2.1.1run the non-RKNN model（P11）
    • 3.2.2Scenario 2: Run on Rockchip NPU connected to the P（P12）
      • 3.2.2.1run the non-RKNN model（P13）
      • 3.2.2.2run the RKNN model（P13）
  • 3.3Hybrid Quantization（P14）
    • 3.3.1Instructions of hybrid quantization（P15）
    • 3.3.2Hybrid quantization profile（P15）
    • 3.3.3Usage flow of hybrid quantization（P16）
  • 3.4Example（P18）
  • 3.5RKNN-Toolkit2 API description（P20）
    • 3.5.1RKNN object initialization and release（P20）
    • 3.5.2RKNN model configuration（P21）
    • 3.5.3Loading non-RKNN model（P23）
      • 3.5.3.1Loading Caffe model（P24）
      • 3.5.3.2Loading TensorFlow model（P24）
      • 3.5.3.3Loading TensorFlow Lite model（P25）
      • 3.5.3.4Loading ONNX model（P25）
      • 3.5.3.5Loading DarkNet model（P26）
      • 3.5.3.6Loading PyTorch model（P26）
    • 3.5.4Building RKNN model（P27）
    • 3.5.5Export RKNN model（P29）
    • 3.5.6Loading RKNN model（P29）
    • 3.5.7Initialize the runtime environment（P30）
    • 3.5.8Inference with RKNN model（P31）
    • 3.5.9Evaluate model performance（P34）
    • 3.5.10Evaluating memory usage（P34）
    • 3.5.11Get SDK version（P35）
    • 3.5.12Hybrid Quantization（P36）
      • 3.5.12.1hybrid_quantization_step1（P36）
      • 3.5.12.2hybrid_quantization_step2（P37）
    • 3.5.13Quantitative accuracy analysis（P38）
    • 3.5.14Register Custom OP（P41）
    • 3.5.15List Devices（P41）
  • 3.6Accuracy troubleshooting（P41）
    • 3.6.1Simulator accuracy troubleshooting（P42）
      • 3.6.1.1Troubleshooting the accuracy of the “fp16 model”（P42）
      • 3.6.1.2Troubleshooting the accuracy of the “quantized mod（P45）
        • 3.6.1.2.1Configuration issues（P45）
          • After the above investigation, the accuracy of so（P47）
        • 3.6.1.2.2Quantitative methods issue（P47）
    • 3.6.2Runtime accuracy troubleshooting（P49）

在线预览

若预览无法显示，请直接下载 PDF 后用本地阅读器打开。

---

1.1 Main function description

RKNN-Toolkit2 is a development kit that provides users with model conversion, inference and

performance evaluation on PC and Rockchip NPU platforms. Users can easily complete the following

functions through the Python interface provided by the tool:

1） Model conversion: support to convert Caffe / TensorFlow / TensorFlow Lite / ONNX / Darknet

/ PyTorch model to RKNN model, support RKNN model import/export, which can be used on

Rockchip NPU platform later.

2） Quantization: support to convert float model to quantization model, currently support quantized

methods including asymmetric quantization (asymmetric_quantized-8,

asymmetric_quantized-16). and support hybrid quantization. Asymmetric_quantized-16 not

supported yet.

3） Model inference: Able to simulate Rockchip NPU to run RKNN model on PC and get the

inference result. This tool can also distribute the RKNN model to the specified NPU device to

run, and get the inference results.

4） Performance & Memory evaluation: distribute the RKNN model to the specified NPU device to

run, and evaluate the model performance and memory consumption in the actual device.

5） Quantitative error analysis: This function will give the Euclidean or cosine distance of each

layer of inference results before and after the model is quantized. This can be used to analyze

how quantitative error occurs, and provide ideas for improving the accuracy of quantitative

models.

Note: Some features are limited by the operating system or chip platform and cannot be used on

some operating systems or platforms. The feature support list of each operating system (platform) is as

follows:

Ubuntu 18.04 Windows 7/10 Debian 9.8 / 10 MacOS Mojave / (ARM 64) Catalina Model conversion yes Quantization yes Model inference yes Performance yes evaluation Memory yes evaluation Multiple inputs yes Batch inference yes(part) List devices yes Query SDK yes version Quantitative error yes analysis Visualization Model yes optimization level 2 Applicable chip model		Ubuntu 18.04	Windows 7/10	Debian 9.8 / 10 (ARM 64)	MacOS Mojave / Catalina
	Model conversion	yes
	Quantization	yes
	Model inference	yes
	Performance evaluation	yes
	Memory evaluation	yes
	Multiple inputs	yes
	Batch inference	yes(part)
	List devices	yes
	Query SDK version	yes
	Quantitative error analysis	yes
	Visualization
	Model optimization level	yes

RK3566 

RK3568 

RK3588 / RK3588S 

Note: RK3588 is referred to as RK3588 / RK3588S in the following text.

1.3 Applicable Operating System

RKNN Toolkit2 is a cross-platform development kit. The supported operating systems are as

follows:

Ubuntu: 18.04 (x64) or later 

Requirements/Dependencies

It is recommended to meet the following requirements in the operating system environment: Table 1 Operating system environment

Operating system version Ubuntu18.04(x64)or later Python version 3.6 Python library See doc/requirements*.txt dependencies Note: 1. This document mainly uses Ubuntu 18.04 / Python3.6 as an example. For other o systems, please refer to the corresponding quick start guide: <Rockchip Quick Start RKNN Toolkit2 EN.pdf>. _ _ _ _ _	Operating system version	Ubuntu18.04(x64)or later
	Python version	3.6
	Python library dependencies	See doc/requirements*.txt

User Guide

3.1 Installation

There are two ways to install RKNN-Toolkit2: one is through the Python package installation and

management tool pip, the other is running docker image with full RKNN-Toolkit2 environment. The

specific steps of the two installation ways are described below.

3.1.1

Install by pip command

1. Create virtualenv environment. If there are multiple versions of the Python environment in the

system, it is recommended to use virtualenv to manage the Python environment.

sudo apt install virtualenv
sudo apt-get install python3 python3-dev p	ython3-pip
sudo apt-get install libxslt1-dev zlib1g zlib1	g-dev libglib2.0-0 \
libsm6 libgl1-mesa-glx libprotobuf-dev gcc
virtualenv -p /usr/bin/python3 venv
source venv/bin/activate

2. Install dependent libraries:

pip3 install -r doc/requirements*.txt

3. Install RKNN-Toolkit2

pip3 install package/rknn_toolkit2*.whl

Please select corresponding installation package (located at the packages/ directory) according to

different python versions and processor architectures:

Python3.6 for x86_64: rknn_toolkit2-x.x.x-cp36-cp36m-linux_x86_64.whl



3.1.2

Install by the Docker Image

In docker folder, there is a Docker image that has been packaged for all development requirements, Users only need to load the image and can directly use RKNN-toolkit2, detailed steps are as follows:

1. Install Docker

Please install Docker according to the official manual:

https://docs.docker.com/install/linux/docker-ce/ubuntu/

2. Load Docker image

Execute the following command to load Docker image:

docker load --input rknn-toolkit2-x.x.x-cpxx-docker.tar.gz After loading successfully, execute "docker images" command and the image of rknn-toolkit2 appears as follows: REPOSITORY TAG IMAGE ID CREATED SIZE rknn-toolkit2 x.x.x 4f6bae6686d8 1 hours ago 5.34GB 3. Run image Execute the following command to run the docker image. After running, it will enter the bash environment. docker run -t -i --privileged -v /dev/bus/usb:/dev/bus/usb rknn-toolkit2:x.x.x /bin/bash If you want to map your own code, you can add the "-v <host src folder>:<image dst folder>" parameter, for example: docker run -t -i --privileged -v /dev/bus/usb:/dev/bus/usb -v /your/rknn-toolkit2-x.x.x/examples:/examples rknn-toolkit2:x.x.x /bin/bash
	docker load --input rknn-toolkit2-x.x.x-cpxx-docker.tar.gz
	docker run -t -i --privileged -v /dev/bus/usb:/dev/bus/usb -v
	/your/rknn-toolkit2-x.x.x/examples:/examples rknn-toolkit2:x.x.x /bin/bash

REPOSITORY TAG IMAGE ID	CREATED SIZE
rknn-toolkit2 x.x.x 4f6bae6686d8 1 hours ago 5.34GB

4. Run demo

cd /examples/tflite/mobilenet_v1
python3 test.py

3.2 Usage of RKNN-Toolkit2

Next, the use process of RKNN Toolkit2 under each use scenario will be given in detail.

3.2.1

Scenario 1: Inference for Simulation on PC

In this scenario, RKNN Toolkit2 runs on the PC, and runs the model through the simulator.

Depending on the type of model, this scenario can be divided into two sub-scenarios: one scenario is

that the model is a non-RKNN model, i.e. Caffe, TensorFlow, TensorFlow Lite, ONNX, DarkNet,

PyTorch model, and the other scenario is that the model is an RKNN model which is a proprietary model

of Rockchip with the file suffix "rknn".

Note: Simulator only supported on x86_64 Linux.

3.2.1.1

run the non-RKNN model

When running a non-RKNN model, the RKNN-Toolkit2 usage flow is shown below:

Start

Create RKNN object to initialize RKNN SDK environment

Call config interface to set pre-processing parameters of model

Call load_caffe, load_tensorflow, load_tflite, load_onnx, load_darknet, load_pytorch, load_mxnet interface to load original Caffe, TensorFlow, TensorFlow Lite, ONNX, Darknet, Pytorch or MXNet model

Call build interface to build RKNN model

Call init_runtime interface to initialize the runtime environment

Call eval_perf interface to Call inference interface to Call export_rknn Call eval_memory interface evaluate performance of model to perform inference with input interface to export to get memory useage when get the running time of each layer to get results RKNN model model running. and total running time of model

Call release interface to release RKNN object

End Figure 1 Usage flow of RKNN-Toolkit2 when running a non-RKNN model on PC

Note:

1. The above steps should be performed in order.

2.The model exporting step marked in the blue box is not necessary. If you exported, you can use

load_rknn to load it later on.

3. The order of model inference, performance evaluation and memory evaluation steps marked in

red box is not fixed, it depends on the actual demand.

4. Only when the target hardware platform is Rockchip NPU, we can call eval_perf / eval_memory

interface.

3.2.2

Scenario 2: Run on Rockchip NPU connected to the PC.

Rockchip NPU platforms currently supported by RKNN Toolkit2 include RK3566 / RK3568 /

RK3588.

In this Scenario, In this scenario, RKNN Toolkit2 runs on the PC and connects to the NPU device

through the PC's USB. RKNN Toolkit2 transfers the RKNN model to the NPU device to run, and then

obtains the inference results, performance information, etc. from the NPU device

First, we need to complete the following two steps:

1. Make sure the USB OTG of development board is connected to PC, and call list_devices interface

will show the device. More information about "list_devices" interface can see Scction 3.5.15.

2. "Target" parameter and "device_id" parameter need to be specified when calling "init_runtime"

interface to initialize the runtime environment, where "target" indicates the type of hardware, optional

values are "rk3566", "rk3568" and "rk3588". When multiple devices are connected to PC, "device_id"

parameter needs to be specified. It is a string which can be obtained by calling "list_devices" interface,

for example:

all device(s) with adb mode: VD46C3KM6N Runtime initialization code is as follows:

# RK3566 ret = init runtime(target='rk3366', device id='VGEJY9PW7T') _ _ # RK3568 ret = init runtime(target='rk3568', device id='515e9b401c060c0b') _ _ # RK3588 ret = init runtime(target='rk3588', device id='515e9b401c060c0b') _ _ 2.2.1 run the non-RKNN model If the model is a non-RKNN model (Caffe, TensorFlow, TensorFlow Lite, ONNX, DarkNet Torch), the usage flow and precautions of RKNN-Toolkit2 are the same as the scenario 1(see Section 2.1.1). 2.2.2 run the RKNN model When running an RKNN model, users do not need to set model pre-processing parameters, nor do ey need to build an RKNN model, the usage flow is shown in the following figure.
	# RK3566
	ret = init runtime(target='rk3366', device id='VGEJY9PW7T')
	_ _
	# RK3568
	ret = init runtime(target='rk3568', device id='515e9b401c060c0b') _ _
	# RK3588
	ret = init runtime(target='rk3588', device id='515e9b401c060c0b') _ _

Start

Create RKNN object to initialize RKNN SDK environment

Call load_rknn interface to load RKNN model

Call init_runtime interface to initialize the runtime environment

Call eval_perf interface to evaluate Call inference interface to Call eval_memory interface performance of model to get the perform inference with to get the memory usage running time of each layer and total input to get results when model running. running time of model

Call release interface to release RKNN object

End

Figure 2 Usage flow of RKNN-Toolkit2 when running an RKNN model on PC

Note:

1. The above steps should be performed in order.

2. The order of model inference, performance evaluation and memory evaluation steps marked in

red box is not fixed, it depends on the actual demand.

3. We can call inference / eval_perf / eval_memory only when the target is hardware platform.

4. The import method through load_rknn is only used for the use of hardware platform-related

functions, and functions such as accuracy_analysis cannot be used.

3.3 Hybrid Quantization

The quantization feature can ensure the accuracy of model based on improved model inference

speed. But for some models, the accuracy has dropped a bit. In order to better balance performance and accuracy, we add new feature hybrid quantization. Users can decide which layers to quantize or not

manually, the quantization parameters also can been modified.

Note:

1. The examples/functions directory provides a hybrid quantization example named hybrid_quant.

Users can refer to this example for hybrid quantification practice.

3.3.1

Instructions of hybrid quantization

Currently, RKNN Toolkit2 has three kind of ways to use hybrid quantization: 1. Convert quantized layer to non-quantized (e.g. float16) layer. Due to the low non-quantized computing power on the NPU, the inference speed will be reduced. 3.2 Hybrid quantization profile When using the hybrid quantization feature, the first step is to generate a hybrid quantization profile hich is briefly described in this section. When the hybrid quantization interface hybrid quantization step1 is called, a configuration file of _ _ odel name}.quantization.cfg is generated in the current directory. The configuration file format is as _ llows: custom quantize layers: {} _ _ quantize parameters: _ Preprocessor/sub:0: qtype: asymmetric quantized _ qmethod: layer dtype: int8 min: - -1.0 max: - 1.0 scale: - 0.00784313725490196 zero point: _ - 0 ori min: _ - -1.0 ori max:
	custom qu _	antize laye _	rs: {}
	quantize parameters:
	_ Preprocessor/sub:0:
	qtype: asymmetric quantized _
	qmethod: layer
	dtype: int8
	min:
	- -1.0
	max:
	- 1.0
	scale:
	- 0.00784313725490196
	zero point: _
	- 0
	ori min: _
	- -1.0
	ori max:

• 1.0 ……

The first line of the body of the configuration file is a dictionary of customized quantize layers, add

the layer names and their corresponding quantized type (choose from float16 / int16) to be changed to

customized quantize layers. int16 not supported yet.

Next is the quantization parameter of each operand in the model, and each operand is a dictionary.

The key of each dictionary is tensor_name, the value of dictionary is quantization parameter, if it is not

quantized, the "dtype" value is float16.

3.3.3

Usage flow of hybrid quantization

When using the hybrid quantization function, it can be done in four steps.

Step1, load the original model and generate a quantize configuration file, a model structure file and a

model weight bias file. The specific interface call process is as follows: Start

Create RKNN object to initialize RKNN environment

Call config interface to set pre-processing parameters of model

Call load_caffe / load_tensorflow / load_tflite / load_onnx / load_darknet / load_pytorch / load_mxnet interface to load original Caffe / TensorFlow / TensorFlow Lite / ONNX / DarkNet / PyTorch / MXNet model

Call hybrid_quantization_step1 interface to generate quantization profile({model_name}.quantization.cfg), temporary model file({model_name}.model), data file({model_name}.data)

Call release interface to release RKNN object

End

Figure 3 call process of hybrid quantization step 1

Step 2, Modify the quantization configuration file generated in the first step.

If some quantization layers is changed to a non-quantization layer, find the output operand of 

layer that is not to be quantized, and add these operands name and float16 to

custom_quantize_layers, such as "<operands name>: float16".

Step 3, generate hybrid quantized RKNN model. The specific interface call flow is as follows: Start

Create RKNN object to initialize RKNN environment

Call hybrid_quantization_step2 interface to build hybrid quantized RKNN model

Call export_rknn interface to export RKNN model

Call release interface to release RKNN object

End

Figure 4 call process of hybrid quantization step 3

Step 4, use the RKNN model generated in the previous step to inference.

3.4 Example

The following is the sample code for loading TensorFlow Lite model (see the

example/tflite/mobilenet_v1 directory for details), if it is executed on PC, the RKNN model will run on

the simulator.

import numpy as np
import cv2
from rknn.api import RKNN

def show_outputs(outputs): output = outputs[0][0] output_sorted = sorted(output, reverse=True) top5_str = 'mobilenet_v1\n-----TOP 5-----\n' for i in range(5): value = output_sorted[i] index = np.where(output == value) for j in range(len(index)): if (i + j) >= 5:

break
if value > 0:
topi = '{}: {}\n'.format(index[j], value)
else:
topi = '-1: 0.0\n'
top5 str += topi _
print(top5 str)
_
if name == ' main ':
__ __ __ __
# Create RKNN object
rknn = RKNN(verbose=True)
# Pre-process config
print('--> Config model')
rknn.config(mean values=[128, 128, 128], std values=[128, 128, 128]) _ _
print('done')
# Load model
print('--> Loading model')
ret = rknn.load tflite(model='mobilenet v1 1.0 224.tflite')
_ if ret != 0:			_ _ _
print('Load model failed!')
exit(ret)
print('done')
# Build model
print('--> Buildin		g model')
ret = rknn.build(d		o quantization=True, dataset='./dataset.txt') _
if ret != 0:
p	rint('Build model failed!')
e	xit(ret)
print('	done')
# Export rknn model
print('--> Export rknn model')
ret = rknn.export rknn('./mobilenet v1.rknn') _ _
if ret != 0:
print('Export rknn model failed!')
exit(ret)
print('done')
# Set inputs
img = cv2.imread('./dog 224x224.jpg') _
img = cv2.cvtColor(img, cv2.COLOR BGR2RGB)
_ img = np.expand dims(img, 0) _
# Init runtime environment
print('--> Init runtime environment')

ret = rknn.init runtime() _ if ret != 0: print('Init runtime environment failed!') exit(ret) print('done') # Inference print('--> Running model') outputs = rknn.inference(inputs=[img]) show outputs(outputs) _ print('done') rknn.release() Where dataset.txt is a text file containing the path of the test image. For example, if a picture of g 224x224.jpg in the example/tflite/mobilenet v1 directory, then the corresponding content in _ _ taset.txt is as follows: dog 224x224.jpg _ When performing model inference, the result of this demo is as follows: -----TOP 5----- [156]: 0.93310546875 [155]: 0.0555419921875 [205 284]: 0.003704071044921875 [205 284]: 0.003704071044921875 -1: 0.0 5 RKNN-Toolkit2 API description 5.1 RKNN object initialization and release The initialization/release function group consists of API interfaces to initialize and release the	ret = rknn.init runtime()
	_ if ret != 0:
	print('Init runtime environment failed!')
	exit(ret)
	print('done')
	# Inference
	print('--> Running model')
	outputs = rknn.inference(inputs=[img])
	show outputs(outputs)
	_ print('done')
	rknn.release()

RKNN object as needed. The RKNN() must be called before using all the API interfaces of RKNN-Toolkit2, and call the release() method to release the object when task finished. When the RKNN object is initing, the users can set verbose and verbose_file parameters, used to

show detailed log information of model loading, building and so on. The data type of verbose parameter

is bool. If the value of this parameter is set to True, the RKNN Toolkit2 will show detailed log

information on screen. The data type of verbose_file is string. If the value of this parameter is set to a file

path, the detailed log information will be written to this file (the verbose also need be set to True). The sample code is as follows:

# Show the detailed log information on screen, and saved to # mobilenet_build.log rknn = RKNN(verbose=True, verbose_file='./mobilenet_build.log') # Only show the detailed log information on screen. rknn = RKNN(verbose=True) … rknn.release()

3.5.2 RKNN model configuration Before the RKNN model is built, the model needs to be configured first through the config interface.
API	config
Description	Set model parameters
	mean values: The mean values of the input. The parameter format is a list. The list _ contains one or more mean sublists. The multi-input model corresponds to multiple sublists. The length of each sublist is consistent with the number of channels of the input. For example, if the parameter is [[128,128,128]], it means an input subtract 128 from the values of the three channels. If quant img RGB2BGR is set to True, the RGB2BGR _ _ conversion will be done first, and then the average value will be subtracted.
	std values: The normalized value of the input. The parameter format is a list. The list _ contains one or more normalized value sublists. The multi-input model corresponds to multiple sublists. The length of each sublist is consistent with the number of channels of the input. For example, if the parameter is [[128,128,128]], it means the value of the three channels of an input minus the average value and then divide by 128. If quant img RGB2BGR is set to True, the RGB2BGR conversion will be performed first, _ _ followed by subtracting the mean value and dividing by the normalized value.
	quant img RGB2BGR: Indicates whether the RGB2BGR operation needs to be done first _ _ when loading the quantized image. The default value is False. If there are multiple inputs,

	of quantization parameters or each channel of weight has its own set of quantization parameters. Usually the channel will be more accurate than the layer, default is channel.
	float dtype: Float data type, the data types currently supported are float16, The default _ value is float16。
	optimization level: Model optimization level. By modifying the model optimization level, _ you can turn off some or all of the optimization rules used in the model conversion process. The default value of this parameter is 3, and all optimization options are turned on. When the value is 2 or 1, turn off some optimization options that may affect the accuracy of some models. Turn off all optimization options when the value is 0.
	target platform: Specify which target chip platform the RKNN model is based on. _ RK3566 / RK3568 / RK3588 are currently supported.
	custom string: Add custom string information to rknn model, then can query the _ information at runtime.
Return Value	None
The sample code is as follows: # model config rknn.config(mean values=[[103.94, 116.78, 123.68]], _ std values=[[58.82, 58.82, 58.82]], _ quant img RGB2BGR=True, target platform='rk3566') _ _ _ 3.5.3 Loading non-RKNN model

# model co	nfig
rknn.config	(mean val _	ues=[[103.94, 116.78, 123.68]],
std values=[[58.82, 58.82, 58.82]], _
quant img RGB2BGR=True, target platform='rk3566')
_ _ _

RKNN-Toolkit2 currently supports load non-RKNN models of Caffe, TensorFlow, TensorFlow Lite,

ONNX, DarkNet, PyTorch. There are different calling interfaces when loading models, the loading

interfaces are described in detail below.

3.5.3.1

Loading Caffe model

API	load caffe _
Description	Load Caffe model
Parameter	model: The path of Caffe model structure file (suffixed with ".prototxt" ).
	blobs: The path of Caffe model binary data file (suffixed with ".caffemodel"). The value can be None, RKNN Toolkit2 will randomly generate parameters such as weights.
	input name: When the caffe model has multiple inputs, you can specify the order of the _ input layer names through this parameter, such as ['input1','input2','input3'],note that the name needs to be consistent with the model input name；It can also be set default. The sequence is automatically given by the caffe model file (file suffix with .prototxt).
Return Value	0: Import successfully
	-1: Import failed
The sample code is as follows: # Load the mobilenet v2 Caffe model in the current path _ ret = rknn.load caffe(model='./mobilenet v2.prototxt', _ _ blobs='./mobilenet v2.caffemodel') _ 3.5.3.2 Loading TensorFlow model
API	load tensorflow _
Description	Load TensorFlow model
Parameter	tf pb: The path of TensorFlow model file (suffixed with ".pb"). _
	inputs: The input node (operand name) of model, input with multiple nodes is supported now. All the input node string are placed in a list.
	input size list: The size and number of channels of the image corresponding to the input _ _ node. As in the example of mobilenet v1 model, the input size list parameter should be _ _ _ set to [[224,224,3]].

# Load the mobilenet v2 Caffe model in the current path _
ret = rknn.load caffe( _	model='./mobilenet v2.prototxt', _
b	lobs='./mo	bilenet v2.caffemodel')
	_

	outputs: The output node (operand name) of model, output with multiple nodes is supported now. All the output nodes are placed in a list.
Return value	0: Import successfully
	-1: Import failed

The sample code is as follows:

# Load ssd mobilenet v1 coco 2017 11 17 TF model in the current path _ _ _ _ _ _ ret = rknn.load tensorflow( _ tf pb='./ssd mobilenet v1 coco 2017 11 17.pb', _ _ _ _ _ _ _ inputs=['Preprocessor/sub'], outputs=['concat', 'concat 1'], _ input size list=[[300, 300, 3]]) _ _ 3.5.3.3 Loading TensorFlow Lite model
API	load tflite _
Description	Load TensorFlow Lite model.
Parameter	model: The path of TensorFlow Lite model file (suffixed with ".tflite").
Return Value	0: Import successfully
	-1: Import failed
The sample code is as follows: # Load the mobilenet v1 TF-Lite model in the current path _ ret = rknn.load tflite(model = './mobilenet v1.tflite') _ _

	is as follo
# Load the mobilenet v1 TF-Lite model in the current path _
ret = rknn.load tflite(model = './mobilenet v1.tflite')
_ _

3.5.3.4

Loading ONNX model

API	load onnx _
Description	Load ONNX model
Parameter	model: The path of ONNX model file (suffixed with ".onnx")
	inputs: The input node (operand name) of model, input with multiple nodes is supported

	now. All the input node string are placed in a list.
	input size list: The size and number of channels of the image corresponding to the input _ _ node. As in the example of mobilenet v1 model, the input size list parameter should be _ _ _ set to [[224,224,3]].
	outputs: The output node (operand name) of model, output with multiple nodes is supported now. All the output nodes are placed in a list.
Return Value	0: Import successfully
	-1: Import failed
The sample code is as follows: # Load the arcface onnx model in the current path ret = rknn.load onnx(model = './arcface.onnx') _ 3.5.3.5 Loading DarkNet model
API	load darknet _
Description	Load DarkNet model
Parameter	model: The path	of DarkNet model structure file (suffixed with ".cfg").
	weight: The path of weight file (suffixed with ".weight").
Return Value	0: Import successfully
	-1: Import failed
The sample code is as follows: # Load the yolov3-tiny DarkNet model in the current path ret = rknn.load darknet(model = './yolov3-tiny.cfg', _ weight= './yolov3.weights')

3.5.3.6

Loading PyTorch model

load_pytorch API

Description	Load PyTorch model. Support the Quantization Aware Training (QAT) model, but need update torch version to '1.9.0' or higher.
Parameter	model:The path of PyTorch model structure file (suffixed with ".pt"), and need a model in the torchscript format. Required.
	input size list:The size and number of channels of each input node. For example, _ _ [[1,1,224,224],[1,3,224,224]] means there are two inputs. One of the input shapes is [1,1, 224, 224], and the other input shape is [1,3, 224, 224]. Required.
Return Value	0: Import successfully
	-1: Import failed
The sample code is as follows: # Load the PyTorch model resnet18 in the current path ret = rknn. load pytorch(model = './resnet18.pt', _ input size list=[[1,3,224,224]]) _ _ 3.5.4 Building RKNN model
API	build
Description	Build	correspond	ing RKNN model according to imported model.
Parameter	do quantization: Whether to quantize the model, optional values are True and False. _
	dataset: A input data set for rectifying quantization parameters. Currently supports text file format, the user can place the path of picture( jpg or png ) or npy file which is used for rectification. A file path for each line. Such as: a.jpg b.jpg or a.npy

# Load the PyTorch model resnet18 in the c	urrent path
ret = rknn. load pytorch(model = './resnet1 _	8.pt',
input size list=[[1,3,224,224]]) _ _

Return value	0: Build successfully
	-1: Build failed

The sample code is as follows:

# Build and quantize RKNN model ret = rknn.build(do_quantization=True, dataset='./dataset.txt')

3.5.5 Export RKNN model The RKNN model built by ‘build’ interface can be saved as a file, it can used to model deployment.
API	export rknn _
Description	Save RKNN model in the specified file (suffixed with ".rknn").
Parameter	export path: The path of generated RKNN model file. _
Return Value	0: Export successfully
	-1: Export failed
The sample code is as follows: # save the built RKNN model as a mobilenet v1.rknn file in the current # path _ ret = rknn.export rknn(export path = './mobilenet v1.rknn') _ _ _ 3.5.6 Loading RKNN model
API	load rknn _
Description	Load RKNN model. The loading model is limited to connecting to the NPU hardware for inference or performance data acquisition. It can not be used for simulator or accuracy analysis.
Parameter	path: The path of RKNN model file.
Return Value	0: Load successfully
	-1: Load failed

# save the built RKNN	model as a mobilenet v1.rknn file in the current # path _
ret = rknn.export rknn _	(export p _	ath = './mobilenet v1.rknn') _

The sample code is as follows: # Load the mobilenet_v1 RKNN model in the current path ret = rknn.load_rknn(path='./mobilenet_v1.rknn')

3.5.7

Initialize the runtime environment

Before inference or performance evaluation, the runtime environment must be initialized. This

interface determines the type of runtime (hardware platform or software simulator).

API	init runtime _
Description	Initialize the runtime environment. Set the device information (hardware platform, device ID). Determine whether to enable debug mode to obtain more detailed performance information for performance evaluation.
Parameter	target: Target hardware platform, now supports "RK3566", "RK3568", "RK3588". The default value is "None", which indicates model runs on simulator.
	device id: Device identity number, if multiple devices are connected to PC, this parameter _ needs to be specified which can be obtained by calling "list devices" interface. The default _ value is "None ".
	perf debug: Debug mode option for performance evaluation. In debug mode, the running _ time of each layer can be obtained, otherwise, only the total running time of model can be given. The default value is False.
	eval mem: Whether enter memory evaluation mode. If set True, the eval memory _ _ interface can be called later to fetch memory usage of model running. The default value is False.
	async mode: Whether to use asynchronous mode. When calling the inference interface, it _ involves setting the input picture, model running, and fetching the inference result. If the asynchronous mode is enabled, setting the input of the current frame will be performed simultaneously with the inference of the previous frame, so in addition to the first frame, each subsequent frame can hide the setting input time, thereby improving performance. In

	asynchronous mode, the inference result returned each time is the previous frame. The default value for this parameter is False. Not Supported yet.
	core mask: Sets the NPU cores at runtime. The supported platform is rk3588, and the _ supported configurations are as follows: NPU CORE AUTO: Indicates the automatic scheduling model, which automatically runs _ _ on the currently idle NPU core. NPU CORE 0: Indicates running on the NPU0 core. _ _ NPU CORE 1: Indicates running on the NPU1 core. _ _ NPU CORE 2: Indicates running on the NPU2 core. _ _ NPU CORE 0 1: Indicates running on NPU0 and NPU1 cores at the same time. _ _ _ NPU CORE 0 1 2: Indicates running on NPU0, NPU1, NPU2 cores at the same time. _ _ _ _ The default value is "NPU CORE AUTO". _ _ Not Supported yet.
Return Value	0: Initialize the runtime environment successfully
	-1: Initialize the r	untime environment failed
The sample code is as follows: # Initialize the runtime environment ret = rknn.init runtime(target='rk3566') _ if ret != 0: print('Init runtime environment failed!') exit(ret)

# Initialize	the runtime environment
ret = rknn.init runtime(target='rk3566')
_ if ret != 0:
print('Init runtime environment failed!')
exit(ret)

3.5.8

Inference with RKNN model

This interface kicks off the RKNN model inference and get the result of inference.

API	inference
Description	Use the model to perform inference with specified input and get the inference result.

	Detailed scenarios are as follows: 1. If RKNN Toolkit2 is running on PC and the target is set to Rockchip NPU when initializing the runtime environment, the inference of model is performed on the specified hardware platform. 2. If RKNN Toolkit2 is running on PC and the target is not set when initializing the runtime environment, the inference of model is performed on the simulator.
Parameter	inputs: Inputs to be inferred, such as images processed by cv2. The object type is ndarray list.
	data format: The shape format of input data. Optional values are "nchw", "nhwc". The _ default value is 'nhwc'. Currently only supported 'nhwc'.
	inputs pass through: Pass the input transparently to the NPU driver. In non-transparent _ _ mode, the tool will reduce the mean, divide the variance, etc. before the input is passed to the NPU driver; in transparent mode, these operations will not be performed. The value of this parameter is an array. For example, to pass input0 and not input1, the value of this parameter is [1, 0]. The default value is None, which means that all input is not transparent.
Return Value	results: The result of inference, the object type is ndarray list.
The sample code is as follows: For classification model, such as mobilenet v1, the code is as follows (refer to _

example/tfilte/mobilenet_v1 for the complete code):

# Preform inference for a picture with a model and get a top-5 result …… outputs = rknn.inference(inputs=[img]) show_outputs(outputs) ……

The result of top-5 is as follows: -----TOP 5----- [156]: 0.93310546875 [155]: 0.0555419921875 [205 284]: 0.003704071044921875 [205 284]: 0.003704071044921875

For object detection model, such as ssd_mobilenet_v1, the code is as follows (refer to

example/tensorflow/ssd_mobilenet_v1 for the complete code):

# Perform inference for a picture with a model and get the result of object # detection …… outputs = rknn.inference(inputs=[image]) …… After the inference result is post-processed, the final output is shown in the following picture (the lor of the object border is randomly generated, so the border color obtained will be different each e):
	# Perform inference for a picture with a model and get the result of object
	# detection
	……
	outputs = rknn.inference(inputs=[image])
	……

Figure 3 ssd_mobilenet_v1 inference result

3.5.9

Evaluate model performance

API	eval perf _
Description	Evaluate model performance. Model must run on RK3566 / RK3568 / RK3588 which connected to PC.If setting perf debug to False when initializing runtime environment, the performance information _ is obtained from hardware, which only contains the total running time of model. And if the perf debug is set to True, the running time of each layer will also be captured in detail. _
	is print: Whether to print performance information. The default value is True. _
Return Value	perf result: Performance information (strings). _
The sample code is as follows: # Evaluate model performance …… rknn.eval perf(inputs=[image], is print=True) _ _ …… 3.5.10 Evaluating memory usage
API	eval _	memory
Description	Fetch memory usage when model is running on hardware platform. Model must run on RK3566 / RK3568 / RK3588 which connected to PC.
Parameter	is print: Whether to print memory evaluation results in the canonical format. The default _ value is True.
Return Value	memory detail：Detail information of memory usage. Data format is dictionary. _ Data shows as below： { 'total weight allocation': 4312608 _ _ 'total internal allocation': 1756160, _ _ 'total model allocation': 6068768 _ _ }

# Evaluate model performance
……
rknn.eval perf(inputs=[image], is p _ _	r	int=True)
……

The ‘total_weight_allocation’ represents the memory footprint of the weights in the 

model.

The ‘total_internal_allocation’ represents the memory usage of the internal tensor in 

the model.

The ‘total_model_allocation’ represents the memory footprint of the model, that is, 

the sum of the weight and the memory footprint of the internal tensor.

The sample code is as follows:

For tflite/mobilenet_v1 in examples directory, the memory usage when model running on RK3566 is

printed as follows:

=============================================
Memory Profile Info				Dump
=========================			====================
NPU model memory detail(bytes):
Total Weight Me		mory: 4.11 MiB
Total Internal Te		nsor Memory: 1.67 MiB
Total Memory: 5		.79 MiB
INFO: Whe	n evaluati	ng memory usage, we need consider
the size of	model, curr	ent model size is: 4.33 MiB
=============================================

3.5.11 Get SDK version

API	get sdk version _ _
Description	Get API version and driver version of referenced SDK. Note: Before we use this interface, we must load model and initialize runtime first. And this API can only used on RK3566 / RK3568 / RK3588.
Parameter	None

Return sdk_version：API and driver version. Data type is string.

Value

The sample code is as follows：

# Get SDK version …… sdk version = rknn.get sdk version() _ _ _ print(sdk version) _ …… The SDK version looks like below： ============================================== RKNN VERSION: API: 1.1.0 (e3000ff build: 2021-08-19 18:03:18) DRV: rknn server: 1.1.0 (6d07b2c build: 2021-08-13 16:36:43) _ rknnrt: 1.1.2b14 (99b248b7@2021-12-02T11:10:04) ============================================== 5.12 Hybrid Quantization Not supported yet. 5.12.1 hybrid quantization step1 _ _ When using the hybrid quantization function, the main interface called in the first phase is brid quantization step1, which is used to generate the temporary model file ({model name}.model), _ _ _ e data file ({model name}.data), and the quantization configuration file ({model name}.quantization _ _
	# Get SDK version
	……
	sdk version = rknn.get sdk version() _ _ _
	print(sdk version) _
	……
	==============================================
	RKNN VERSION:
	API: 1.1.0 (e3000ff build: 2021-08-19 18:03:18)
	DRV: rknn server: 1.1.0 (6d07b2c build: 2021-08- _	13 16:36:43)
	rknnrt: 1.1.2b14 (99b248b7@2021-12-02T11:10:04)
	==============================================

cfg). Interface details are as follows:

API	hybrid quantization step1 _ _
Description	Corresponding temporary model files, data files, and quantization profiles are generated according to the loaded original model.

Parameter	dataset: A input data set for rectifying quantization parameters. Currently supports text file format, the user can place the path of picture( jpg or png ) or npy file which is used for rectification. A file path for each line. Such as: a.jpg b.jpg or a.npy b.npy
	rknn batch size: See Building RKNN model _ _
	proposal: Generate hybrid quantization config suggestions.
	proposal dataset size: The size of dataset used for proposal. Because the proposal _ _ function is time-consuming, so the default size is 1.
Return Value	0: success
	-1: failure
The sample code is as follows: # Call hybrid quantization step1 to generate quantization config _ _ …… ret = rknn.hybrid quantization step1(dataset='./dataset.txt') _ _ …… 3.5.12.2 hybrid quantization step2 _ _

# Call hybrid quantiz		ation step1 to generate quantization config
_ _ ……
ret = rknn.h	ybrid qua _	ntization step1(dataset='./dataset.txt') _
……

When using the hybrid quantization function, the primary interface for generating a hybrid quantized

RKNN model phase call is hybrid_quantization_step2. The interface details are as follows:

API	hybrid quantization step2 _ _
Description	The temporary model file, the data file, the quantization profile, and the correction data set are received as inputs, and the hybrid quantized RKNN model is generated.
Parameter	model input: The temporary model file generated in the first step, which is shaped like _

	"{model name}.model". The data type is a string. Required parameter. _
	data input: The model data file generated in the first step, which is shaped like _ "{model name}.data". The data type is a string. Required parameter. _
	model quantization cfg: The modified model quantization profile, whick is shaped like _ _ "{model name}.quantization.cfg". The data type is a string. Required parameter. _
Return Value	0: success
	-1: failure
The sample code is as follows: # Call hybrid quantization step2 to generate hybrid quantized RKNN model _ _ …… ret = rknn.hybrid quantization step2( _ _ model input='./ssd mobilenet v2.model', _ _ _ data input='./ssd mobilenet v2.data', _ _ _ model quantization cfg='./ssd mobilenet v2.quantization.cfg') _ _ _ _ …… 3.5.13 Quantitative accuracy analysis The function of this interface is inference with quantized model and generate outputs of each layers for quantitative accuracy analysis.
API	accura	cy analys _	is
Description	Inference with quantized model and generate snapshot, that is dump tensor data of each layers. It will dump a snapshot of both data types include fp32 & quant for calculate quantitative error. Note: 1. this interface can only be called after build or hybrid quantization step2, and _ _ the original model should be a non-quantized model, otherwise the call will fail. 2. The quantization method used by this interface is consistent with the setting in config.

# Call hybrid quantization step2 to generate hybrid quantized RKNN model _ _
……
ret = rknn.hybrid quantization step2(
_ _ model input='./ssd mobilenet v2.model', _ _ _
data input='./ssd mobilenet v2.data',
_ _ _ model quantization cfg='./ssd mobilenet v2.quantization.cfg')
_ _ _ ……	_

Parameter	inputs: the path list of image (jpg/png/bmp/npy).
	output dir: output directory, all snapshot data will stored here. (default is directory name _ ‘snapshot’) If the target is not set, the following content will be output under ‘output dir’: _  Directory simulator: Save the results of each layer on simulator when the entire quantitative model is fully run (The output has been converted to float32).  Directory golden: Save the results of each layer on simulator when the entire floating-point model is completely run down.  error analysis.txt: Record the the cosine distance (entire error and per layer error) _ _ _ _ between each layer result on simulator and the floating-point model on simulator during the complete calculation of the quantized model. The different of entire error/per layer error is the input of each layer is come from the quantization _ _ _ model or floating-point model. If the target is set, more content will output under ‘output dir’: _  Directory runtime: Save the results of each layer when the entire quantitative model is fully run in NPU (The output has been converted to float32).  error analysis.txt: Record the the cosine distance (entire error) between each layer _ _ result on simulator and each layer on NPU during the complete calculation of the quantized model additionally.
	target: Target hardware platform, now supports "RK3566", "RK3568", "RK3588". The default value is "None". If target is set, the output of each layer of NPU will be obtained, and analyze it’s accuracy.
	device id: Device identity number, if multiple devices are connected to PC, this parameter _ needs to be specified which can be obtained by calling "list devices" interface. The default _ value is "None ".
Return	0: success

Value -1: failure

The sample code is as follows:

……
# Create RKNN object
rknn = RKNN(verbose=True)
# Pre-process config
print('--> Config model')
rknn.config(mean values=[128, 128, 128], std values=[128, 128, 128]) _ _
print('done')
# Load model
print('--> Loading model')
ret = rknn.load tensorflow(tf pb='mobilenet v1.pb', _ _ _
inputs=['input'],
outputs=['Mobilen			etV1/Logits/SpatialSqueeze'],
input size list=[[1, 224, 224, 3]]) _ _
if ret != 0:
print('Load model failed!')
exit(ret)
print('done')
# Build model
print('--> Building model')
ret = rknn.build(do		quantization=True, dataset='dataset.txt') _
if ret != 0:
print('build m		odel failed!')
exit(ret)
print('d	one')
# Accuracy analysis
print('--> Accuracy analysis')
Ret = rknn.accuracy analysis(inputs=['./dog 224x224.jpg']) _ _
if ret != 0:
print('Accuracy analysis failed!')
exit(ret)
print('done')
……

3.5.14 Register Custom OP

Not supported yet.

3.5.15 List Devices

API	list devices _
Description	List connected RK3566 / RK3568 / RK3588. Note: There are currently two device connection modes: ADB and NTB. RK3566 / RK3568 / RK3588 support both ADB and NTB. Make sure their modes are the same when connecting multiple devices
Parameter	None
Return Value	Return adb devices list and ntb devices list. If there are no devices connected to PC, it will _ _ return two empty list.
The sample code is as follows： from rknn.api import RKNN if name == ' main ': __ __ __ __ rknn = RKNN() rknn.list devices() _ rknn.release() The devices list looks like below： ************************* all device(s) with adb mode: VD46C3KM6N *************************

from rknn.api import R		KNN
if name == ' main ':
__ __ rknn =	__ RKNN()	__
rknn.lis	t devices() _
rknn.release()

3.6

Accuracy troubleshooting

The troubleshooting of model accuracy is generally conducted from two aspects, one is the Simulator accuracy investigation, and the other is the runtime accuracy check on board-side. The correct

Simulator result is a prerequisite for correct board-side running. Therefore, it is recommended that users

prioritize the correct Simulator results when dealing with the accuracy of the rknn model, and then

troubleshoot the board-side running accuracy.Therefore, we will give recommendations and solutions for

accuracy problem troubleshooting in terms of Simulator accuracy investigation and board-side accuracy

investigation during runtime.

In addition, the judgment of accuracy can simply use the cosine distance as the basic judgment, but

this is not equal to the final model accuracy, it is only used as a reference. And the judgment of accuracy

must be verified by running the data set finally. Of course, in the process of troubleshooting the following

accuracy problems, you can simply use the cosine distant as a basis for accuracy improving.

3.6.1

Simulator accuracy troubleshooting

The correctness of the Simulator results is a prerequisite for the correct inference on board-side, so it

is necessary to ensure that the simulator results is correct on the PC-side. Rknn-toolkit2 provides the

choice of whether the mode is quantified, so this chapter analyzes the accuracy of the “fp16 model” and

the “quantized model” respectively. Because the correct result of the “fp16 model” is the prerequisite for

the accuracy of the “quantized model”, when there is a problem with the accuracy of the “quantized

model”, it is generally recommended that users first verify the correctness of the “fp16 model”. The

troubleshooting strategies for the “fp16 model” and the “quantized model” will be described in detail

below.

3.6.1.1

Troubleshooting the accuracy of the “fp16 model”

The correct result of the “fp16 model” is a prerequisite to ensure the accuracy of the subsequent

“quantized model”. The user only needs to set the do_quantization parameter to False when using the

build interface of rknn to convert the original model to the “fp16 model”. if the output result of “fp16

model” is wrong, you need to perform the following troubleshooting: 1) Configuration issues

The configuration of the model is mainly concentrated in the config interface of rknn. And

there are a few configuration in other rknn APIs. But not every configuration will cause the accuracy

problems, mainly the parameters that cause the accuracy problems of the “fp16 model” as follows:

mean_values / std_values: The normalized parameters of the model, must ensure that they are

the same as the parameters used in the original model.

input_size_list: The input node shape information of load_tensorflow / load_pytorch, if the

configurations is wrong, it will also lead the wrong inference results.

inputs / outputs: the name of the input and output nodes of load_tensorflow. If the

configuration is wrong, it will also lead the wrong inference results.

parameters of inference interface: The input parameters of rknn's inference interface, mainly

including inputs and data_format. Generally, in the python environment, the image data is read through

cv2.imread. At this time, it should be noted that the image format read by cv2.imread is BGR. If the input

of the original model is BGR (such as most caffe models), Then you can directly transfer the image data

to the inference interface of rknn for inference; and if the input of the original model is RGB, you also

need to call cv2.cvtColor(img, cv2.COLOR_BGR2RGB) to convert the image data to RGB, and then

pass it to the inference interface of rknn inference. In addition, the layout of the image data read by

cv2.imread is NHWC, because the default value of data_format is NHWC, so there is no need to set the

data_format parameter. If the input data of the model is not read through cv2.imread, the user must

clearly know the layout of the input data and set the correct data_format parameter, if it is image data,

ensure that its RGB sequence is consistent with the input RGB sequence of the model.

The inspection of parameter configuration is a very important, and it is the main reason why

many users have wrong output results of the "fp16 model". Specific steps are as follows:

a. Use the original model to perform inference under the original model's inference

framework. For example, the caffe model uses caffe_bvlc or opencv_caffe for inference,

the pytorch model uses the pytorch inference framework for inference, pb and tflite use tensorflow for inference, and onnx uses onnxruntime for inference. etc., and then save the

inference result.

b. Use the original model to perform inference under the inference framework of

rknn-toolkit2. You need to use the same input data as in the previous step, and set the

inference mode of fp16 (do_quantization of rknn's build is set to False), and target

parameter of init_runtime should not be configured or set to None. At this time, the

simulator inside rknn-toolkit2 is used for inference, and the result of the inference is also

saved.

c. Comparing the results of the two inferences, if the results are more consistent (cosine

distance can be used to judge the consistency), it means that there is no problem with the

above configuration.

d. If the results are inconsistent, check whether the above parameters are correct.

If it is confirmed that the above parameter configuration is correct, and the results are still

inconsistent, it may be an internal bugs on the emulation-side.

2. Internal bugs on the emulation-side

This chapter may be related to the internal bug of the Simulator, but the probability of

occurrence is very low. Generally, it is recommended that users use the following two methods to

troubleshoot.

One is to set the verbose parameter to 'Debug' when constructing the rknn object, it will turn on

the debug mode of rknn-toolkit2, and output the accuracy check log during the construction of the rknn

model, according to the result of "check results" in the output log (Cosine similarity and Euclidean

distance), you can determine which step has the problem. If the problem is determined, it is best to

provide the model and log that reproduce the result to the Rockchip NPU team for analysis and solution.

This method uses the interface provided by rknn-toolkit2 for error checking, and is generally suitable for

quickly locating the problem. If there is no related "check results" log output or the problem still cannot be located, the next method can also be used.

The prerequisite of the other method is that the user can obtain the ground truth of the output of

each layer of the original model under the original framework. At this time, the accuracy analysis

interface of rknn can be used to dump the ‘golden’ result of each layer of the floating-point model, and

the cosine similarity is compared with the output results of each layer of the model under the original

framework. If the ‘golden’ result is not aligned with the output result of the first layer of the model under

the original framework (generally it is considered that the cosine similarity is less than 0.99 there is a

little inconsistency, and it is almost considered that the result of this layer is wrong if the cosine similarity

is lower than 0.98), it may still be the previous parameter configuration If there is an error, you need to go

back to the previous step for re-checking. If the output results of the first layer are consistent, but the

middle layer or the final results are inconsistent, it may be caused by a Simulator implementation bug. At

this point, the user can locate the layer where the inconsistency started, and can intercept the model

around this layer, and provide the reproduction model to the Rockchip NPU team for analysis and

solution.

3.6.1.2

Troubleshooting the accuracy of the “quantized model”

After the accuracy of the "fp16 model" is verified, the error of the "fp16 model" is eliminated, the

model can be quantified, and the accuracy of the "quantized model" can be further analyzed. If you

encounter accuracy problems in the "quantized model", the investigation will be conducted mainly from

the following aspects:

3.6.1.2.1 Configuration issues

Similar to the configuration of the "fp16 model", configuration errors can also cause the accuracy of

the "quantized model". On the basis of ensuring the correct configuration of the "fp16 model", the

following parameter configurations should still be checked.

quantized_dtype: The choice of quantization type. The accuracy of different quantization types is very different, and there is also a big difference in runtime performance. Generally, a compromise

quantization type, such as asymmetric_quantized-8, is selected. If asymmetric_quantized-4 is selected,

the best runtime performance can be achieved, but the accuracy is also the worst, so it is only suitable for

a few models that are not sensitive to 4-bit quantization. If asymmetric_quantized-16 is selected, the

accuracy close to the original model can be achieved, but the runtime performance will be relatively poor,

so it is only suitable for use in scenarios that are not sensitive to runtime performance but require very

high accuracy. However, because generally in the NPU, 16-bit quantization and non-quantization (float16)

have little difference in computing performance, it is recommended to choose fp16 (do_quantization of

rknn's build interface is set to False) instead of 16-bit quantization. (The current version of

asymmetric_quantized-16 is temporarily not supported, and asymmetric_quantized-4 is not supported

under rk356x either)

quant_img_RGB2BGR: Indicates whether RGB2BGR needs to be performed first when loading

the quantized image. It is generally used for the caffe model. For more detailed information, please refer

to the quant_img_RGB2BGR parameter description. This parameter configuration error will also cause a

significant decrease in quantization accuracy.

dataset: The configuration of quantitative correction set of rknn's build interface. If you select a

calibration set that is not consistent with the actual deployment scenario, the accuracy may be reduced, or

too many or too few calibration sets will affect the accuracy (generally choose 50 to 200 sheets).

To check the parameter configuration of the "quantified model" specifically, you can generally

follow the steps below:

1. Directly perform "quantized model" inference, and then check the result of the inference and

compare it with the result of the original model in the original inference framework. If the

result is not very different, it can be considered that the quantized_dtype,

quant_img_RGB2BGR and dataset parameters are basically correct.

2. If the result is still very different:

a. If quantized_dtype is configured as a 4-bit algorithm, you can modify quantized_dtype to a higher-bit quantization algorithm.

b. If the input image format of the original model is BGR (more common in caffe models),

you can modify quant_img_RGB2BGR to True at this time. In fact, the RGB sequence of

the input data can be known from the processing code of the input data in the accuracy

verification step of the previous "fp16 model".

c. You can use an image for quantization (leave only one line in dataset.txt), and use this

image for inference. If the accuracy of a single image is improved more at this time, it

means that the previously used quantization correction set is not well selected, and you

can reselect pictures that are more consistent with the deployment scene.

d. If only one image is used for quantization (only one line is left in dataset.txt), you can try

to use more images for quantization, which can be increased to about 50~200.

After the above investigation, the accuracy of some models may already meet the requirements, and

some models may not be accurate enough, you can try the methods in the following chapters (change the

quantization method). But there should not be a situation where the quantized result is completely wrong,

if there is a completely wrong situation, please recheck the above configuration.

3.6.1.2.2

Quantitative methods issue

Some models themselves are not friendly to quantization. At this time, you can try to switch between

different quantization methods and quantization algorithms. At present, there are two main quantization

methods, namely Per-Layer / Per-Channel, corresponding to the quantized_method parameter in rknn's

config interface, and the quantization algorithm is mainly divided into two types, namely Normal /

MMSE, corresponding to rknn's config interface quantized_algorithm parameter in rknn’s config

interface. Steps as follows:

1. If the Per-Layer quantization method was originally used, it can be changed to the Per-Channel

quantization method. In general, the accuracy of the Per-Channel quantization method is much

higher than that of the Per-Layer quantization method, but it may be brings a slight decrease in execution efficiency (negligible)

2. If the quantization method has been changed to Per-Channel, but the accuracy still cannot meet

the requirements, the quantization algorithm can be changed from Normal to MMSE at this

time. This method will greatly increase the quantization time, but will bring better accuracy

than Normal, and it will not affect the performance of the runtime.

If the accuracy is still low after using the above method, it may be that some Ops of the model are

not friendly to the existing quantization algorithm, and the accuracy will drop more after quantization.

For example, when the weight distribution of Conv is very uneven, you can consider using hybrid

quantization to further improve the accuracy of the model. Hybrid quantization can allow different OPs in

the model to use different quantization types. Specific steps are as follows:

1. First use the accuracy analysis interface to analyze the accuracy and find the layer that causes

the accuracy to decrease.

2. Use the hybrid quantization method, and write the name of the output tensor of the suspected

layer into the hybrid quantization configuration file.

3. Complete the steps of hybrid quantification and test the accuracy. (You can use the accuracy

analysis interface to observe the accuracy changes)

Generally, after hybrid quantization, the accuracy of the model can be improved. If the improvement

is not obvious or not enough, you can try to hybrid more layers, but at the same time it will also reduce

the runtime speed, so the hybrid quantization requires users weigh the accuracy and speed by themselves.

There is also a special way that when the Op with reduced accuracy is in the last layer, you can also

choose to run the Op in post-processing, which will also effectively avoid the accuracy problem of this

layer.

So far, after investigating the above reasons, we can basically obtain a quantized model with better

accuracy, and the accuracy problem investigation on the Simulator-side is basically completed.

3.6.2

Runtime accuracy troubleshooting

After the accuracy verification of the "fp16 model" and the "quantized model" on the Simulator side,

the quantization accuracy on the Simulator side should generally be able to meet the needs of the

application, but users may often encounter the problem that the quantization accuracy on the Simulator

side is not bad, but when the inference test is performed on the board through rknn's C API programming,

they find that the accuracy is insufficient or not at all correct. There are generally two reasons for this

kind of problem. One is caused by the code itself that calls rknn's C API programming, such as incorrect

input data, incorrect runtime parameter configuration, or post-processing code error, etc.; the other is

caused by runtime bugs on board-side. When encountering this kind of problem, we can first troubleshoot

the runtime problem of the board-side through the following steps:

1. In the case of configuring the connecting board debugging environment (the environment

configuration method is detailed in the RKNN C API release package), use rknn-toolkit2 to

convert the rknn model and set the target parameter of init_runtime, such as target='rk3566' ,

And connect the board to the PC via USB (refer to chapter 3.2.2), and then perform the

inference with board connection and check whether the inference result is correct. (Because the

Simulator does not strictly simulate the NPU hardware, so the result may not be completely

consistent with the Simulator)

2. If the inference result in step 1 differs greatly from the Simulator result, it can be preliminarily

determined that there is a bug in the runtime when running the model on the board-side. At this

time, the accuracy analysis interface can be used to check the accuracy of the runtime side, just

set the target parameter of accuracy_analysis, such as target=’rk3566’, after the

accuracy_analysis is called, the accuracy analysis results of each layer will be output. If there is

a significant difference between the board-side runtime result and the quantized true value of

the Simulator, the analysis result and the reproduced model can be fed back to the Rockchip

NPU team for repair.

If there is no problem with the above verification, the problem lies in the C/C++ code that the user calls rknn's C API for programming. At this time, the user needs to carefully check whether the

configuration of rknn's C API is configured correctly, and whether the pre-processing and post-processing

processes of the model are correct (need to be exactly the same as the process on the Simulator side). For

the use and configuration of rknn's C API, please refer to the relevant rknn_api documentation.