Note

This is the documentation for the latest development branch and may refer to features that are not available in released versions. If you are looking for the documentation for a specific release, use the drop-down menu on the left and select the desired version.

NNCASE Application Guide

Overview

What is nncase

nncase is a neural network compiler designed for AI accelerators. The currently supported targets include CPU/K210/K510/K230, etc.

Features provided by nncase:

• Supports multi-input and multi-output networks, supports multi-branch structures;

• Static memory allocation, no heap memory required;

• Operator fusion and optimization;

• Supports float and uint8/int8 quantized inference;

• Supports post-training quantization, using floating-point models and quantization calibration sets;

• Flat model, supports zero-copy loading;

Neural network model formats supported by nncase:

• TFLite

• ONNX

nncase Architecture

The nncase software stack includes two parts: compiler and runtime.

Compiler: Used to compile neural network models on PC, ultimately generating kmodel files. It mainly includes modules such as importer, IR, Evaluator, Quantize, Transform optimization, Tiling, Partition, Schedule, Codegen, etc.

• Importer: Imports models from other neural network frameworks into nncase;

• IR: Intermediate Representation, divided into Neutral IR imported by the importer (device-independent) and Target IR generated by lowering the Neutral IR (device-dependent);

• Evaluator: The Evaluator provides interpretive execution capability for IR, commonly used in scenarios such as Constant Folding/PTQ Calibration;

• Transform: Used for IR conversion and graph traversal optimization, etc.;

• Quantize: Post-training quantization. Add quantization markers to tensors to be quantized. Based on the input calibration set, call the Evaluator for interpretive execution, collect the data range of tensors, insert quantization/dequantization nodes, and finally optimize to eliminate unnecessary quantization/dequantization nodes, etc.;

• Tiling: Limited by the relatively small memory capacity of the NPU, large blocks of computation need to be split. In addition, when there is a large amount of data reuse in computation, selecting Tiling parameters will affect latency and bandwidth;

• Partition: The graph is partitioned by ModuleType. Each partitioned subgraph corresponds to a RuntimeModule. Different types of RuntimeModules correspond to different Devices (CPU/K230);

• Schedule: Based on the data dependency relationships in the optimized graph, generate computation order and allocate Buffers;

• Codegen: Call the codegen corresponding to the ModuleType for each subgraph to generate RuntimeModule;

Runtime: Integrated into the user App, providing functions such as loading kmodel/setting input data/KPU execution/getting output data.

Development Environment

Operating System

Supported operating systems include Ubuntu 18.04/Ubuntu 20.04/Windows 10/Windows 11.

Software Environment

No.	Software	Version
1	python	3.6/3.7/3.8/3.9/3.10
2	pip	>=20.3
3	numpy	1.19.5
4	onnx	1.9.0
5	onnx-simplifier	0.3.6
6	Onnxoptimizer	0.2.6
7	Onnxruntime	1.8.0
8	dotnet-runtime	7.0

Operator Support

TFLite Operators

Operator	Is Supported
ABS	Yes
ADD	Yes
ARG_MAX	Yes
ARG_MIN	Yes
AVERAGE_POOL_2D	Yes
BATCH_MATMUL	Yes
CAST	Yes
CEIL	Yes
CONCATENATION	Yes
CONV_2D	Yes
COS	Yes
CUSTOM	Yes
DEPTHWISE_CONV_2D	Yes
DIV	Yes
EQUAL	Yes
EXP	Yes
EXPAND_DIMS	Yes
FLOOR	Yes
FLOOR_DIV	Yes
FLOOR_MOD	Yes
FULLY_CONNECTED	Yes
GREATER	Yes
GREATER_EQUAL	Yes
L2_NORMALIZATION	Yes
LEAKY_RELU	Yes
LESS	Yes
LESS_EQUAL	Yes
LOG	Yes
LOGISTIC	Yes
MAX_POOL_2D	Yes
MAXIMUM	Yes
MEAN	Yes
MINIMUM	Yes
MUL	Yes
NEG	Yes
NOT_EQUAL	Yes
PAD	Yes
PADV2	Yes
MIRROR_PAD	Yes
PACK	Yes
POW	Yes
REDUCE_MAX	Yes
REDUCE_MIN	Yes
REDUCE_PROD	Yes
RELU	Yes
PRELU	Yes
RELU6	Yes
RESHAPE	Yes
RESIZE_BILINEAR	Yes
RESIZE_NEAREST_NEIGHBOR	Yes
ROUND	Yes
RSQRT	Yes
SHAPE	Yes
SIN	Yes
SLICE	Yes
SOFTMAX	Yes
SPACE_TO_BATCH_ND	Yes
SQUEEZE	Yes
BATCH_TO_SPACE_ND	Yes
STRIDED_SLICE	Yes
SQRT	Yes
SQUARE	Yes
SUB	Yes
SUM	Yes
TANH	Yes
TILE	Yes
TRANSPOSE	Yes
TRANSPOSE_CONV	Yes
QUANTIZE	Yes
FAKE_QUANT	Yes
DEQUANTIZE	Yes
GATHER	Yes
GATHER_ND	Yes
ONE_HOT	Yes
SQUARED_DIFFERENCE	Yes
LOG_SOFTMAX	Yes
SPLIT	Yes
HARD_SWISH	Yes

ONNX Operators

Operator	Is Supported
Abs	Yes
Acos	Yes
Acosh	Yes
And	Yes
ArgMax	Yes
ArgMin	Yes
Asin	Yes
Asinh	Yes
Add	Yes
AveragePool	Yes
BatchNormalization	Yes
Cast	Yes
Ceil	Yes
Celu	Yes
Clip	Yes
Compress	Yes
Concat	Yes
Constant	Yes
ConstantOfShape	Yes
Conv	Yes
ConvTranspose	Yes
Cos	Yes
Cosh	Yes
CumSum	Yes
DepthToSpace	Yes
DequantizeLinear	Yes
Div	Yes
Dropout	Yes
Elu	Yes
Exp	Yes
Expand	Yes
Equal	Yes
Erf	Yes
Flatten	Yes
Floor	Yes
Gather	Yes
GatherElements	Yes
GatherND	Yes
Gemm	Yes
GlobalAveragePool	Yes
GlobalMaxPool	Yes
Greater	Yes
GreaterOrEqual	Yes
GRU	Yes
Hardmax	Yes
HardSigmoid	Yes
HardSwish	Yes
Identity	Yes
InstanceNormalization	Yes
LayerNormalization	Yes
LpNormalization	Yes
LeakyRelu	Yes
Less	Yes
LessOrEqual	Yes
Log	Yes
LogSoftmax	Yes
LRN	Yes
LSTM	Yes
MatMul	Yes
MaxPool	Yes
Max	Yes
Min	Yes
Mul	Yes
Neg	Yes
Not	Yes
OneHot	Yes
Pad	Yes
Pow	Yes
PRelu	Yes
QuantizeLinear	Yes
RandomNormal	Yes
RandomNormalLike	Yes
RandomUniform	Yes
RandomUniformLike	Yes
ReduceL1	Yes
ReduceL2	Yes
ReduceLogSum	Yes
ReduceLogSumExp	Yes
ReduceMax	Yes
ReduceMean	Yes
ReduceMin	Yes
ReduceProd	Yes
ReduceSum	Yes
ReduceSumSquare	Yes
Relu	Yes
Reshape	Yes
Resize	Yes
ReverseSequence	Yes
RoiAlign	Yes
Round	Yes
Rsqrt	Yes
Selu	Yes
Shape	Yes
Sign	Yes
Sin	Yes
Sinh	Yes
Sigmoid	Yes
Size	Yes
Slice	Yes
Softmax	Yes
Softplus	Yes
Softsign	Yes
SpaceToDepth	Yes
Split	Yes
Sqrt	Yes
Squeeze	Yes
Sub	Yes
Sum	Yes
Tanh	Yes
Tile	Yes
TopK	Yes
Transpose	Yes
Trilu	Yes
ThresholdedRelu	Yes
Upsample	Yes
Unsqueeze	Yes
Where	Yes

API Documentation

The nncase software stack includes compiler and runtime parts, which are used for model conversion and KPU model inference respectively. Python and C++ APIs are provided for these two parts.

Usage Steps

Environment Setup

• Linux

First, install dotnet-sdk-7.0 and configure the dotnet environment variable. Do not install dotnet in an anaconda virtual environment:

sudo apt-get update
sudo apt-get install dotnet-sdk-7.0
export DOTNET_ROOT=/usr/share/dotnet

Then install nncase and nncase-kpu :

pip install nncase nncase-kpu

• Windows

First, install dotnet-sdk-7.0. For installation steps, see the Microsoft official documentation: Install .NET on Windows .

Then install nncase online. In Release , select the corresponding version of nncase_kpu-2.x.x-py2.py3-none-win_amd64.whl to download, then install locally using pip install.

pip install nncase
pip install nncase_kpu-2.x.x-py2.py3-none-win_amd64.whl

• Docker

If users do not have an Ubuntu environment, they can use the nncase docker (Ubuntu 20.04 + Python 3.8 + dotnet-7.0)

cd /path/to/nncase_sdk
docker pull ghcr.io/kendryte/k230_sdk
docker run -it --rm -v `pwd`:/mnt -w /mnt ghcr.io/kendryte/k230_sdk /bin/bash -c "/bin/bash"

• View Version Information

root@469e6a4a9e71:/mnt# python3
Python 3.8.10 (default, May 26 2023, 14:05:08)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import _nncase
>>> print(_nncase.__version__)
2.9.0

Model Conversion

The nncase user guide documentation is available at: github: user_guide or gitee: user_guide .

Use nncase to convert tflite/onnx models to kmodel. The key to the model conversion code is configuring options according to your needs, mainly CompileOptions, PTQTensorOptions, and ImportOptions.

CompileOptions

The CompileOptions class is used to configure nncase compilation options. The descriptions of each property are as follows:

Attribute Name	Type	Required	Description
target	string	Yes	Specifies the compilation target, e.g., ‘cpu’, ‘k230’
dump_ir	bool	No	Specifies whether to dump IR, default is False
dump_asm	bool	No	Specifies whether to dump asm assembly files, default is False
dump_dir	string	No	After enabling switches such as dump_ir above, specifies the directory for dumping, default is “”
input_file	string	No	Used to specify the parameter file path when the ONNX model exceeds 2GB, default is “”
preprocess	bool	No	Whether to enable preprocessing, default is False. The following parameters only take effect when preprocess=True
input_type	string	No	Specifies the input data type when preprocessing is enabled, default is “float”. When preprocess is True, it must be specified as “uint8” or “float32”
input_shape	list[int]	No	Specifies the shape of the input data when preprocessing is enabled, default is []. Must be specified when preprocess is True
input_range	list[float]	No	Specifies the floating-point range after dequantization of input data when preprocessing is enabled, default is []. Must be specified when preprocess is True and input_type is uint8
input_layout	string	No	Specifies the layout of the input data, default is “”
swapRB	bool	No	Whether to reverse data in the channel dimension, default is False
mean	list[float]	No	Mean value for preprocessing normalization, default is [0,0,0]
std	list[float]	No	Variance for preprocessing normalization, default is [1,1,1]
letterbox_value	float	No	Specifies the padding value for letterbox preprocessing, default is 0
output_layout	string	No	Specifies the layout of the output data, default is “”
shape_bucket_enable	bool	Yes	Whether to enable the ShapeBucket function, default is False. Takes effect when dump_ir=True
shape_bucket_range_info	Dict[str, [int, int]]	Yes	The range of variables in each input shape dimension information, the minimum value must be greater than or equal to 1
shape_bucket_segments_count	int	Yes	How many segments the range of input variables is divided into
shape_bucket_fix_var_map	Dict[str, int]	No	Fixes variables in shape dimension information to specific values

For the description of preprocessing configuration, please refer to the API documentation: nncase Model Compilation API Manual Preprocessing Flow. Encapsulating some preprocessing operations within the model can improve the preprocessing efficiency during inference on the development board. The supported preprocessing includes: swapRB (RGB->BGR or BGR->RGB), Transpose (NHWC->NCHW or NCHW->NHWC), Normalization (subtract mean and divide by variance), Dequantize, etc. For example, the onnx model requires RGB input, but the image we read using opencv is in BGR format. Normally, for onnx model inference preprocessing, we need to convert BGR to RGB for the onnx model. When converting to kmodel, we can set swapRB to True, so that the kmodel contains the preprocessing step for swapping RB channels. When performing preprocessing for kmodel inference, we can skip the step of swapping RB channels and place this step inside the kmodel.

PTQTensorOptions

The PTQTensorOptions class is used to configure nncase PTQ options:

Name	Type	Required	Description
samples_count	int	No	Specifies the number of calibration sets used for quantization
calibrate_method	string	No	Specifies the quantization method, optional ‘NoClip’, ‘Kld’, default is ‘Kld’
finetune_weights_method	string	No	Specifies whether to fine-tune weights, optional ‘NoFineTuneWeights’, ‘UseSquant’, default is ‘NoFineTuneWeights’
quant_type	string	No	Specifies the data quantization type, optional ‘uint8’, ‘int8’, ‘int16’. quant_type and w_quant_type cannot both be ‘int16’
w_quant_type	string	No	Specifies the weight quantization type, optional ‘uint8’, ‘int8’, ‘int16’. quant_type and w_quant_type cannot both be ‘int16’
quant_scheme	string	No	Path to the import quantization parameter configuration file
quant_scheme_strict_mode	bool	No	Whether to strictly execute quantization according to quant_scheme
export_quant_scheme	bool	No	Whether to export the quantization parameter configuration file
export_weight_range_by_channel	bool	No	Whether to export weights quantization parameters in bychannel format. It is recommended to set this parameter to True

For the specific usage flow of mixed quantization, see MixQuant Instructions.

For the description of quantization configuration, please refer to the API documentation: nncase Model Compilation API Manual PTQ Options Configuration. If the converted kmodel does not achieve the desired effect, you can modify the quant_type and w_quant_type parameters to change the quantization types of model data and weights, but these two parameters cannot be set to int16 at the same time.

Calibration Set Settings

Name	Type	Description
data	List[List[np.ndarray]]	Read calibration data

The calibration data used during quantization is set through the set_tensor_data method. The interface parameter type is List[List[np.ndarray]]. For example: if the model has one input, the calibration data count is set to 10, the dimension of the input calibration data is [10,1,3,224,224]; if the model has two inputs, the calibration data count is set to 10, the dimension of the input calibration data is [[10,1,3,224,224],[10,1,3,320,320]].

ImportOptions

The ImportOptions class is used to configure nncase import options, configuring the model to be converted by the compiler. You can configure tflite/onnx. The usage example is as follows:

# Read and import the tflite model
model_content = read_model_file(model)
compiler.import_tflite(model_content, import_options)

# Read and import the onnx model
model_content = read_model_file(model)
compiler.import_onnx(model_content, import_options)

Example: YOLOv8 onnx to kmodel

import os
import argparse
import numpy as np
from PIL import Image
import onnxsim
import onnx
import nncase
import shutil
import math

def parse_model_input_output(model_file,input_shape):
    onnx_model = onnx.load(model_file)
    input_all = [node.name for node in onnx_model.graph.input]
    input_initializer = [node.name for node in onnx_model.graph.initializer]
    input_names = list(set(input_all) - set(input_initializer))
    input_tensors = [
        node for node in onnx_model.graph.input if node.name in input_names]

    # input
    inputs = []
    for _, e in enumerate(input_tensors):
        onnx_type = e.type.tensor_type
        input_dict = {}
        input_dict['name'] = e.name
        input_dict['dtype'] = onnx.mapping.TENSOR_TYPE_TO_NP_TYPE[onnx_type.elem_type]
        input_dict['shape'] = [(i.dim_value if i.dim_value != 0 else d) for i, d in zip(
            onnx_type.shape.dim, input_shape)]
        inputs.append(input_dict)

    return onnx_model, inputs


def onnx_simplify(model_file, dump_dir,input_shape):
    onnx_model, inputs = parse_model_input_output(model_file,input_shape)
    onnx_model = onnx.shape_inference.infer_shapes(onnx_model)
    input_shapes = {}
    for input in inputs:
        input_shapes[input['name']] = input['shape']

    onnx_model, check = onnxsim.simplify(onnx_model, input_shapes=input_shapes)
    assert check, "Simplified ONNX model could not be validated"

    model_file = os.path.join(dump_dir, 'simplified.onnx')
    onnx.save_model(onnx_model, model_file)
    return model_file


def read_model_file(model_file):
    with open(model_file, 'rb') as f:
        model_content = f.read()
    return model_content


def generate_data(shape, batch, calib_dir):
    img_paths = [os.path.join(calib_dir, p) for p in os.listdir(calib_dir)]
    data = []
    for i in range(batch):
        assert i < len(img_paths), "calibration images not enough."
        img_data = Image.open(img_paths[i]).convert('RGB')
        img_data = img_data.resize((shape[3], shape[2]), Image.BILINEAR)
        img_data = np.asarray(img_data, dtype=np.uint8)
        img_data = np.transpose(img_data, (2, 0, 1))
        data.append([img_data[np.newaxis, ...]])
    return np.array(data)


def main():
    parser = argparse.ArgumentParser(prog="nncase")
    parser.add_argument("--target", default="k230",type=str, help='target to run,k230/cpu')
    parser.add_argument("--model",type=str, help='model file')
    parser.add_argument("--dataset_path", type=str, help='calibration_dataset')
    parser.add_argument("--input_width", type=int, default=320, help='model input_width')
    parser.add_argument("--input_height", type=int, default=320, help='model input_height')
    parser.add_argument("--ptq_option", type=int, default=0, help='ptq_option:0,1,2,3,4,5')

    args = parser.parse_args()

    # Update parameters to multiples of 32
    input_width = int(math.ceil(args.input_width / 32.0)) * 32
    input_height = int(math.ceil(args.input_height / 32.0)) * 32

    # The input shape of the model, dimensions should match input_layout
    input_shape=[1,3,input_height,input_width]

    dump_dir = 'tmp'
    if not os.path.exists(dump_dir):
        os.makedirs(dump_dir)

    # onnx simplify
    model_file = onnx_simplify(args.model, dump_dir,input_shape)

    # Set CompileOptions
    compile_options = nncase.CompileOptions()
    compile_options.target = args.target

    # Whether to use kmodel for preprocessing
    compile_options.preprocess = True
    # The onnx model requires RGB, and the camera data on k230 is also in RGB format, so there is no need to enable RB swap
    compile_options.swapRB = False
    # The shape of the input image
    compile_options.input_shape = input_shape
    # The model input format 'uint8' or 'float32'
    compile_options.input_type = 'uint8'

    # If the input is in 'uint8' format, the range after dequantization
    compile_options.input_range = [0, 1]
    # The mean/std values for preprocessing, one per channel, this data is obtained from the YOLOv8 source code
    compile_options.mean = [0, 0, 0]
    compile_options.std = [1, 1, 1]

    # Set the input layout, onnx default 'NCHW' is fine
    compile_options.input_layout = "NCHW"

    # Create Compiler instance
    compiler = nncase.Compiler(compile_options)

    # Import onnx model
    model_content = read_model_file(model_file)
    import_options = nncase.ImportOptions()
    compiler.import_onnx(model_content, import_options)

    # Configure quantization method
    ptq_options = nncase.PTQTensorOptions()
    ptq_options.samples_count = 10

    if args.ptq_option == 0:
        ptq_options.calibrate_method = 'NoClip'
        ptq_options.quant_type = 'uint8'
        ptq_options.w_quant_type = 'uint8'
    elif args.ptq_option == 1:
        ptq_options.calibrate_method = 'NoClip'
        ptq_options.quant_type = 'uint8'
        ptq_options.w_quant_type = 'int16'
    elif args.ptq_option == 2:
        ptq_options.calibrate_method = 'NoClip'
        ptq_options.quant_type = 'int16'
        ptq_options.w_quant_type = 'uint8'
    elif args.ptq_option == 3:
        ptq_options.calibrate_method = 'Kld'
        ptq_options.quant_type = 'uint8'
        ptq_options.w_quant_type = 'uint8'
    elif args.ptq_option == 4:
        ptq_options.calibrate_method = 'Kld'
        ptq_options.quant_type = 'uint8'
        ptq_options.w_quant_type = 'int16'
    elif args.ptq_option == 5:
        ptq_options.calibrate_method = 'Kld'
        ptq_options.quant_type = 'int16'
        ptq_options.w_quant_type = 'uint8'
    else:
        pass

    # Set calibration data
    ptq_options.set_tensor_data(generate_data(input_shape, ptq_options.samples_count, args.dataset_path))
    compiler.use_ptq(ptq_options)

    # Start compilation
    compiler.compile()

    # Write kmodel file
    kmodel = compiler.gencode_tobytes()
    base,ext=os.path.splitext(args.model)
    kmodel_name=base+".kmodel"
    with open(kmodel_name, 'wb') as f:
        f.write(kmodel)


if __name__ == '__main__':
    main()

After the model conversion is successful, the code needs to be deployed on the development board, which requires using nncase_runtime to write C++ code.

Deployment Code Writing

The deployment code writing takes YOLOv8 detection as an example. The source code is in the src/rtsmart/examples/kpu_run_yolov8 directory. Execute build_app.sh in this directory to obtain the compiled executable files for image inference and camera inference in the k230_bin directory under that directory. The process of using the KPU runtime API to perform inference on the model is as follows.

graph TD; LoadModel("Initialize interpreter instance
Load model") -->SetInput("Get input shape
Initialize input tensors")-->SetOutput("Get output shape
Initialize output tensors")-->GetFrame("Read data to be inferred
Read image/from camera")-->SetPreprocessParam("Set preprocessing parameters, including AI2D preprocessing method configuration, input tensor, output tensor")-->PreProcess("Execute preprocessing so that the read image or video frame conforms to the **model input**")-->KPURun("Execute model inference")-->GetOutput("Get the output pointer of model inference")-->PostProcess("Post-process the output according to the specific scenario")-->DrawResult("Draw the post-processing results on the image/screen");

The preprocessing methods provided by AI2D are implemented in hardware, which can improve running efficiency. Interpreter is used to complete model inference on KPU. Their inputs and outputs are all host_runtime_tensor type data. The model input may be one or multiple; the processing result of AI2D is generally given to the model for use. When initially initializing ai2d_builder and Interpreter, the input and output tensors of the two components are generally initialized together. The schematic diagrams of the two components are as follows:

Because the output tensor of AI2D will be given to the input tensor of the model for inference, when there is a single input, the output tensor of AI2D and the input tensor of Interpreter can be bound as one, which can save the memory of one tensor. If AI2D is not used for preprocessing, OpenCV can be used to preprocess the input data, and then create a host_runtime_tensor. The schematic diagram is as follows: