Note

This is the documentation for the latest development branch and may refer to features that are not available in released versions. If you are looking for the documentation for a specific release, use the drop-down menu on the left and select the desired version.

NNCASE Application Guide

Overview

What is nncase

nncase is a neural network compiler designed for AI accelerators. Currently supported targets include CPU/K210/K510/K230, etc.

Features provided by nncase:

• Supports multi-input and multi-output networks, supports multi-branch structures;

• Static memory allocation, no heap memory required;

• Operator fusion and optimization;

• Supports float and uint8/int8 quantized inference;

• Supports post-training quantization, using floating-point models and quantization calibration sets;

• Flat model, supports zero-copy loading;

Neural network model formats supported by nncase:

• TFLite

• ONNX

nncase Architecture

The nncase software stack includes two parts: compiler and runtime.

Compiler: Used to compile neural network models on a PC, and ultimately generate kmodel files. It mainly includes modules such as importer, IR, Evaluator, Quantize, Transform optimization, Tiling, Partition, Schedule, Codegen, etc.

• Importer: Imports models from other neural network frameworks into nncase;

• IR: Intermediate Representation, divided into Neutral IR (device-independent) imported by the importer and Target IR (device-dependent) generated by lowering conversion from Neutral IR;

• Evaluator: Evaluator provides the interpretation and execution capability of IR, and is often used in scenarios such as Constant Folding/PTQ Calibration;

• Transform: Used for IR conversion and graph traversal optimization, etc.;

• Quantize: Post-training quantization. Adds quantization markers to tensors to be quantized. Based on the input calibration set, calls the Evaluator for interpretation and execution, collects the data range of tensors, inserts quantization/dequantization nodes, and finally optimizes to eliminate unnecessary quantization/dequantization nodes, etc.;

• Tiling: Limited by the relatively small memory capacity of the NPU, large blocks of computation need to be split. In addition, when there is a large amount of data reuse in computation, choosing Tiling parameters will affect latency and bandwidth;

• Partition: Splits the graph according to ModuleType. Each split subgraph will correspond to a RuntimeModule. Different types of RuntimeModules correspond to different Devices (CPU/K230);

• Schedule: Generates computation order and allocates Buffers according to the data dependency relationships in the optimized graph;

• Codegen: Calls the codegen corresponding to each ModuleType for each subgraph to generate RuntimeModule;

Runtime: Integrated into the user App, provides functions such as loading kmodel/setting input data/KPU execution/getting output data.

Development Environment

Operating System

Supported operating systems include Ubuntu 18.04/Ubuntu 20.04/Windows 10/Windows 11.

Software Environment

No.	Software	Version
1	python	3.6/3.7/3.8/3.9/3.10
2	pip	>=20.3
3	numpy	1.19.5
4	onnx	1.9.0
5	onnx-simplifier	0.3.6
6	Onnxoptimizer	0.2.6
7	Onnxruntime	1.8.0
8	dotnet-runtime	7.0

Operator Support

TFLite Operators

Operator	Is Supported
ABS	Yes
ADD	Yes
ARG_MAX	Yes
ARG_MIN	Yes
AVERAGE_POOL_2D	Yes
BATCH_MATMUL	Yes
CAST	Yes
CEIL	Yes
CONCATENATION	Yes
CONV_2D	Yes
COS	Yes
CUSTOM	Yes
DEPTHWISE_CONV_2D	Yes
DIV	Yes
EQUAL	Yes
EXP	Yes
EXPAND_DIMS	Yes
FLOOR	Yes
FLOOR_DIV	Yes
FLOOR_MOD	Yes
FULLY_CONNECTED	Yes
GREATER	Yes
GREATER_EQUAL	Yes
L2_NORMALIZATION	Yes
LEAKY_RELU	Yes
LESS	Yes
LESS_EQUAL	Yes
LOG	Yes
LOGISTIC	Yes
MAX_POOL_2D	Yes
MAXIMUM	Yes
MEAN	Yes
MINIMUM	Yes
MUL	Yes
NEG	Yes
NOT_EQUAL	Yes
PAD	Yes
PADV2	Yes
MIRROR_PAD	Yes
PACK	Yes
POW	Yes
REDUCE_MAX	Yes
REDUCE_MIN	Yes
REDUCE_PROD	Yes
RELU	Yes
PRELU	Yes
RELU6	Yes
RESHAPE	Yes
RESIZE_BILINEAR	Yes
RESIZE_NEAREST_NEIGHBOR	Yes
ROUND	Yes
RSQRT	Yes
SHAPE	Yes
SIN	Yes
SLICE	Yes
SOFTMAX	Yes
SPACE_TO_BATCH_ND	Yes
SQUEEZE	Yes
BATCH_TO_SPACE_ND	Yes
STRIDED_SLICE	Yes
SQRT	Yes
SQUARE	Yes
SUB	Yes
SUM	Yes
TANH	Yes
TILE	Yes
TRANSPOSE	Yes
TRANSPOSE_CONV	Yes
QUANTIZE	Yes
FAKE_QUANT	Yes
DEQUANTIZE	Yes
GATHER	Yes
GATHER_ND	Yes
ONE_HOT	Yes
SQUARED_DIFFERENCE	Yes
LOG_SOFTMAX	Yes
SPLIT	Yes
HARD_SWISH	Yes

ONNX Operators

Operator	Is Supported
Abs	Yes
Acos	Yes
Acosh	Yes
And	Yes
ArgMax	Yes
ArgMin	Yes
Asin	Yes
Asinh	Yes
Add	Yes
AveragePool	Yes
BatchNormalization	Yes
Cast	Yes
Ceil	Yes
Celu	Yes
Clip	Yes
Compress	Yes
Concat	Yes
Constant	Yes
ConstantOfShape	Yes
Conv	Yes
ConvTranspose	Yes
Cos	Yes
Cosh	Yes
CumSum	Yes
DepthToSpace	Yes
DequantizeLinear	Yes
Div	Yes
Dropout	Yes
Elu	Yes
Exp	Yes
Expand	Yes
Equal	Yes
Erf	Yes
Flatten	Yes
Floor	Yes
Gather	Yes
GatherElements	Yes
GatherND	Yes
Gemm	Yes
GlobalAveragePool	Yes
GlobalMaxPool	Yes
Greater	Yes
GreaterOrEqual	Yes
GRU	Yes
Hardmax	Yes
HardSigmoid	Yes
HardSwish	Yes
Identity	Yes
InstanceNormalization	Yes
LayerNormalization	Yes
LpNormalization	Yes
LeakyRelu	Yes
Less	Yes
LessOrEqual	Yes
Log	Yes
LogSoftmax	Yes
LRN	Yes
LSTM	Yes
MatMul	Yes
MaxPool	Yes
Max	Yes
Min	Yes
Mul	Yes
Neg	Yes
Not	Yes
OneHot	Yes
Pad	Yes
Pow	Yes
PRelu	Yes
QuantizeLinear	Yes
RandomNormal	Yes
RandomNormalLike	Yes
RandomUniform	Yes
RandomUniformLike	Yes
ReduceL1	Yes
ReduceL2	Yes
ReduceLogSum	Yes
ReduceLogSumExp	Yes
ReduceMax	Yes
ReduceMean	Yes
ReduceMin	Yes
ReduceProd	Yes
ReduceSum	Yes
ReduceSumSquare	Yes
Relu	Yes
Reshape	Yes
Resize	Yes
ReverseSequence	Yes
RoiAlign	Yes
Round	Yes
Rsqrt	Yes
Selu	Yes
Shape	Yes
Sign	Yes
Sin	Yes
Sinh	Yes
Sigmoid	Yes
Size	Yes
Slice	Yes
Softmax	Yes
Softplus	Yes
Softsign	Yes
SpaceToDepth	Yes
Split	Yes
Sqrt	Yes
Squeeze	Yes
Sub	Yes
Sum	Yes
Tanh	Yes
Tile	Yes
TopK	Yes
Transpose	Yes
Trilu	Yes
ThresholdedRelu	Yes
Upsample	Yes
Unsqueeze	Yes
Where	Yes

API Documentation

The nncase software stack includes compiler and runtime parts, used for model conversion and KPU model inference respectively. Python and C++ APIs are provided for these two parts. See the API manual at: nncase API Manual .

Usage Steps

Environment Setup

• Linux

First install dotnet-sdk-7.0, and configure the dotnet environment variable. Do not install dotnet in an anaconda virtual environment:

sudo apt-get update
sudo apt-get install dotnet-sdk-7.0
export DOTNET_ROOT=/usr/share/dotnet

Then install nncase and nncase-kpu :

pip install nncase nncase-kpu

• Windows

First install dotnet-sdk-7.0. For installation steps, refer to the Microsoft official documentation: Install .NET on Windows .

Then install nncase online. In Release, select the corresponding version of nncase_kpu-2.x.x-py2.py3-none-win_amd64.whl to download, then install locally using pip install.

pip install nncase
pip install nncase_kpu-2.x.x-py2.py3-none-win_amd64.whl

• Docker

If users do not have an Ubuntu environment, they can use the nncase docker (Ubuntu 20.04 + Python 3.8 + dotnet-7.0)

cd /path/to/nncase_sdk
docker pull ghcr.io/kendryte/k230_sdk
docker run -it --rm -v `pwd`:/mnt -w /mnt ghcr.io/kendryte/k230_sdk /bin/bash -c "/bin/bash"

• View Version Information

root@469e6a4a9e71:/mnt# python3
Python 3.8.10 (default, May 26 2023, 14:05:08)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import _nncase
>>> print(_nncase.__version__)
2.9.0

Model Conversion

The nncase user guide document is available at: github: user_guide.

Use nncase to convert tflite/onnx models to kmodel. The key to the model conversion code lies in configuring options according to your own needs, mainly CompileOptions , PTQTensorOptions and ImportOptions.

CompileOptions

The CompileOptions class is used to configure nncase compilation options. The description of each property is as follows:

Property Name	Type	Required	Description
target	string	Yes	Specify the compilation target, e.g. ‘cpu’, ‘k230’
dump_ir	bool	No	Specify whether to dump IR, default is False
dump_asm	bool	No	Specify whether to dump asm assembly file, default is False
dump_dir	string	No	After specifying switches such as dump_ir above, specify the dump directory here, default is “”
input_file	string	No	Used to specify the parameter file path when the ONNX model exceeds 2GB, default is “”
preprocess	bool	No	Whether to enable preprocessing, default is False. The following parameters take effect only when preprocess=True
input_type	string	No	Specify the input data type when preprocessing is enabled, default is “float”. When preprocess is True, it must be specified as “uint8” or “float32”
input_shape	list[int]	No	Specify the shape of the input data when preprocessing is enabled, default is []. Must be specified when preprocess is True
input_range	list[float]	No	Specify the float range after dequantization of the input data when preprocessing is enabled, default is []. Must be specified when preprocess is True and input_type is uint8
input_layout	string	No	Specify the layout of the input data, default is “”
swapRB	bool	No	Whether to reverse data in the channel dimension, default is False
mean	list[float]	No	Normalization mean for preprocessing, default is [0,0,0]
std	list[float]	No	Normalization variance for preprocessing, default is [1,1,1]
letterbox_value	float	No	Specify the padding value for preprocessing letterbox, default is 0
output_layout	string	No	Specify the layout of the output data, default is “”
shape_bucket_enable	bool	Yes	Whether to enable the ShapeBucket function, default is False. Takes effect when dump_ir=True
shape_bucket_range_info	Dict[str, [int, int]]	Yes	The range of variables in each input shape dimension information. The minimum value must be greater than or equal to 1
shape_bucket_segments_count	int	Yes	The number of segments the range of input variables is divided into
shape_bucket_fix_var_map	Dict[str, int]	No	Fix variables in shape dimension information to specific values

For the configuration description of preprocessing, please refer to the API documentation: nncase Model Compilation API Manual Preprocessing Flow. Encapsulating some preprocessing operations within the model can improve the preprocessing efficiency during board inference. Supported preprocessing includes: swapRB(RGB->BGR or BGR->RGB), Transpose(NHWC->NCHW or NCHW->NHWC), Normalization (subtract mean, divide variance), Dequantize, etc. For example: the onnx model requires RGB input, but the image we read using opencv is BGR. Normally, for the preprocessing of onnx model inference, we need to convert BGR to RGB for the onnx model. When converting to kmodel, we can set swapRB to True, so that the kmodel itself contains the preprocessing step of swapping the RB channels. When performing preprocessing for kmodel inference, we can ignore the step of swapping the RB channels and put this step inside the kmodel.

PTQTensorOptions

The PTQTensorOptions class is used to configure nncase PTQ options:

Name	Type	Required	Description
samples_count	int	No	Specify the number of calibration sets used for quantization
calibrate_method	string	No	Specify the quantization method, options are ‘NoClip’, ‘Kld’, default is ‘Kld’
finetune_weights_method	string	No	Specify whether to fine-tune weights, options are ‘NoFineTuneWeights’, ‘UseSquant’, default is ‘NoFineTuneWeights’
quant_type	string	No	Specify the data quantization type, options are ‘uint8’, ‘int8’, ‘int16’. quant_type and w_quant_type cannot both be ‘int16’
w_quant_type	string	No	Specify the weight quantization type, options are ‘uint8’, ‘int8’, ‘int16’. quant_type and w_quant_type cannot both be ‘int16’
quant_scheme	string	No	Path to the import quantization parameter configuration file
quant_scheme_strict_mode	bool	No	Whether to strictly follow quant_scheme for quantization
export_quant_scheme	bool	No	Whether to export the quantization parameter configuration file
export_weight_range_by_channel	bool	No	Whether to export weights quantization parameters in bychannel form. This parameter is recommended to be set to True

For the specific usage flow of mixed quantization, see MixQuant Description.

For the configuration description of quantization, please refer to the API documentation: nncase Model Compilation API Manual PTQ Options Configuration. If the converted kmodel does not achieve the desired effect, you can modify the quant_type and w_quant_type parameters to change the quantization types of model data and weights, but these two parameters cannot both be set to int16.

Calibration Set Settings

Name	Type	Description
data	List[List[np.ndarray]]	The read calibration data

The calibration data used in the quantization process is set through the set_tensor_data method. The interface parameter type is List[List[np.ndarray]]. For example: if the model has one input, the calibration data amount is set to 10, the dimension of the passed-in calibration data is [10,1,3,224,224]; if the model has two inputs, the calibration data amount is set to 10, the dimension of the passed-in calibration data is [[10,1,3,224,224],[10,1,3,320,320]].

ImportOptions

The ImportOptions class is used to configure nncase import options and configure the model to be converted by the compiler. It can be configured for tflite/onnx. The usage example is as follows:

# 读取并导入tflite模型
model_content = read_model_file(model)
compiler.import_tflite(model_content, import_options)

# 读取并导入onnx模型
model_content = read_model_file(model)
compiler.import_onnx(model_content, import_options)

YOLOv8 ONNX to kmodel Example

import os
import argparse
import numpy as np
from PIL import Image
import onnxsim
import onnx
import nncase
import shutil
import math

def parse_model_input_output(model_file,input_shape):
    onnx_model = onnx.load(model_file)
    input_all = [node.name for node in onnx_model.graph.input]
    input_initializer = [node.name for node in onnx_model.graph.initializer]
    input_names = list(set(input_all) - set(input_initializer))
    input_tensors = [
        node for node in onnx_model.graph.input if node.name in input_names]

    # input
    inputs = []
    for _, e in enumerate(input_tensors):
        onnx_type = e.type.tensor_type
        input_dict = {}
        input_dict['name'] = e.name
        input_dict['dtype'] = onnx.mapping.TENSOR_TYPE_TO_NP_TYPE[onnx_type.elem_type]
        input_dict['shape'] = [(i.dim_value if i.dim_value != 0 else d) for i, d in zip(
            onnx_type.shape.dim, input_shape)]
        inputs.append(input_dict)

    return onnx_model, inputs


def onnx_simplify(model_file, dump_dir,input_shape):
    onnx_model, inputs = parse_model_input_output(model_file,input_shape)
    onnx_model = onnx.shape_inference.infer_shapes(onnx_model)
    input_shapes = {}
    for input in inputs:
        input_shapes[input['name']] = input['shape']

    onnx_model, check = onnxsim.simplify(onnx_model, input_shapes=input_shapes)
    assert check, "Simplified ONNX model could not be validated"

    model_file = os.path.join(dump_dir, 'simplified.onnx')
    onnx.save_model(onnx_model, model_file)
    return model_file


def read_model_file(model_file):
    with open(model_file, 'rb') as f:
        model_content = f.read()
    return model_content


def generate_data(shape, batch, calib_dir):
    img_paths = [os.path.join(calib_dir, p) for p in os.listdir(calib_dir)]
    data = []
    for i in range(batch):
        assert i < len(img_paths), "calibration images not enough."
        img_data = Image.open(img_paths[i]).convert('RGB')
        img_data = img_data.resize((shape[3], shape[2]), Image.BILINEAR)
        img_data = np.asarray(img_data, dtype=np.uint8)
        img_data = np.transpose(img_data, (2, 0, 1))
        data.append([img_data[np.newaxis, ...]])
    return np.array(data)


def main():
    parser = argparse.ArgumentParser(prog="nncase")
    parser.add_argument("--target", default="k230",type=str, help='target to run,k230/cpu')
    parser.add_argument("--model",type=str, help='model file')
    parser.add_argument("--dataset_path", type=str, help='calibration_dataset')
    parser.add_argument("--input_width", type=int, default=320, help='model input_width')
    parser.add_argument("--input_height", type=int, default=320, help='model input_height')
    parser.add_argument("--ptq_option", type=int, default=0, help='ptq_option:0,1,2,3,4,5')

    args = parser.parse_args()

    # 更新参数为32倍数
    input_width = int(math.ceil(args.input_width / 32.0)) * 32
    input_height = int(math.ceil(args.input_height / 32.0)) * 32

    # 模型的输入shape，维度要跟input_layout一致
    input_shape=[1,3,input_height,input_width]

    dump_dir = 'tmp'
    if not os.path.exists(dump_dir):
        os.makedirs(dump_dir)

    # onnx simplify
    model_file = onnx_simplify(args.model, dump_dir,input_shape)

    # 设置CompileOptions
    compile_options = nncase.CompileOptions()
    compile_options.target = args.target

    # 是否采用kmodel模型做预处理
    compile_options.preprocess = True
    # onnx模型需要RGB的，k230上的摄像头给出的数据也是RGB格式的，因此不需要开启交换RB
    compile_options.swapRB = False
    # 输入图像的shape
    compile_options.input_shape = input_shape
    # 模型输入格式‘uint8’或者‘float32’
    compile_options.input_type = 'uint8'

    # 如果输入是‘uint8’格式，输入反量化之后的范围
    compile_options.input_range = [0, 1]
    # 预处理的mean/std值，每个channel一个，该数据由YOLOv8源码获取
    compile_options.mean = [0, 0, 0]
    compile_options.std = [1, 1, 1]

    # 设置输入的layout，onnx默认‘NCHW’即可
    compile_options.input_layout = "NCHW"

    # 创建Compiler实例
    compiler = nncase.Compiler(compile_options)

    # 导入onnx模型
    model_content = read_model_file(model_file)
    import_options = nncase.ImportOptions()
    compiler.import_onnx(model_content, import_options)

    # 配置量化方式
    ptq_options = nncase.PTQTensorOptions()
    ptq_options.samples_count = 10

    if args.ptq_option == 0:
        ptq_options.calibrate_method = 'NoClip'
        ptq_options.quant_type = 'uint8'
        ptq_options.w_quant_type = 'uint8'
    elif args.ptq_option == 1:
        ptq_options.calibrate_method = 'NoClip'
        ptq_options.quant_type = 'uint8'
        ptq_options.w_quant_type = 'int16'
    elif args.ptq_option == 2:
        ptq_options.calibrate_method = 'NoClip'
        ptq_options.quant_type = 'int16'
        ptq_options.w_quant_type = 'uint8'
    elif args.ptq_option == 3:
        ptq_options.calibrate_method = 'Kld'
        ptq_options.quant_type = 'uint8'
        ptq_options.w_quant_type = 'uint8'
    elif args.ptq_option == 4:
        ptq_options.calibrate_method = 'Kld'
        ptq_options.quant_type = 'uint8'
        ptq_options.w_quant_type = 'int16'
    elif args.ptq_option == 5:
        ptq_options.calibrate_method = 'Kld'
        ptq_options.quant_type = 'int16'
        ptq_options.w_quant_type = 'uint8'
    else:
        pass

    # 设置校正数据
    ptq_options.set_tensor_data(generate_data(input_shape, ptq_options.samples_count, args.dataset_path))
    compiler.use_ptq(ptq_options)

    # 启动编译
    compiler.compile()

    # 写入kmodel文件
    kmodel = compiler.gencode_tobytes()
    base,ext=os.path.splitext(args.model)
    kmodel_name=base+".kmodel"
    with open(kmodel_name, 'wb') as f:
        f.write(kmodel)


if __name__ == '__main__':
    main()

After the model conversion is successful, to deploy the code on the development board, you need to write C++ code to call the nncase_runtime API for implementation.