# NNCASE Application Guide ## Overview ### What is nncase nncase is a neural network compiler designed for AI accelerators. The currently supported targets include CPU/K210/K510/K230, etc. Features provided by nncase: - Supports multi-input and multi-output networks, supports multi-branch structures; - Static memory allocation, no heap memory required; - Operator fusion and optimization; - Supports float and uint8/int8 quantized inference; - Supports post-training quantization, using floating-point models and quantization calibration sets; - Flat model, supports zero-copy loading; Neural network model formats supported by nncase: - TFLite - ONNX ### nncase Architecture ![nncase Architecture](https://www.kendryte.com/api/post/attachment?id=509) The nncase software stack includes two parts: compiler and runtime. Compiler: Used to compile neural network models on PC, ultimately generating kmodel files. It mainly includes modules such as importer, IR, Evaluator, Quantize, Transform optimization, Tiling, Partition, Schedule, Codegen, etc. - Importer: Imports models from other neural network frameworks into nncase; - IR: Intermediate Representation, divided into Neutral IR imported by the importer (device-independent) and Target IR generated by lowering the Neutral IR (device-dependent); - Evaluator: The Evaluator provides interpretive execution capability for IR, commonly used in scenarios such as Constant Folding/PTQ Calibration; - Transform: Used for IR conversion and graph traversal optimization, etc.; - Quantize: Post-training quantization. Add quantization markers to tensors to be quantized. Based on the input calibration set, call the Evaluator for interpretive execution, collect the data range of tensors, insert quantization/dequantization nodes, and finally optimize to eliminate unnecessary quantization/dequantization nodes, etc.; - Tiling: Limited by the relatively small memory capacity of the NPU, large blocks of computation need to be split. In addition, when there is a large amount of data reuse in computation, selecting Tiling parameters will affect latency and bandwidth; - Partition: The graph is partitioned by ModuleType. Each partitioned subgraph corresponds to a RuntimeModule. Different types of RuntimeModules correspond to different Devices (CPU/K230); - Schedule: Based on the data dependency relationships in the optimized graph, generate computation order and allocate Buffers; - Codegen: Call the codegen corresponding to the ModuleType for each subgraph to generate RuntimeModule; Runtime: Integrated into the user App, providing functions such as loading kmodel/setting input data/KPU execution/getting output data. ### Development Environment #### Operating System Supported operating systems include Ubuntu 18.04/Ubuntu 20.04/Windows 10/Windows 11. #### Software Environment | No. | Software | Version | | ---- | ---------------- | -------------------- | | 1 | python | 3.6/3.7/3.8/3.9/3.10 | | 2 | pip | \>=20.3 | | 3 | numpy | 1.19.5 | | 4 | onnx | 1.9.0 | | 5 | onnx-simplifier | 0.3.6 | | 6 | Onnxoptimizer | 0.2.6 | | 7 | Onnxruntime | 1.8.0 | | 8 | dotnet-runtime | 7.0 | ## Operator Support ### TFLite Operators | Operator | Is Supported | | ----------------------- | ------------ | | ABS | Yes | | ADD | Yes | | ARG_MAX | Yes | | ARG_MIN | Yes | | AVERAGE_POOL_2D | Yes | | BATCH_MATMUL | Yes | | CAST | Yes | | CEIL | Yes | | CONCATENATION | Yes | | CONV_2D | Yes | | COS | Yes | | CUSTOM | Yes | | DEPTHWISE_CONV_2D | Yes | | DIV | Yes | | EQUAL | Yes | | EXP | Yes | | EXPAND_DIMS | Yes | | FLOOR | Yes | | FLOOR_DIV | Yes | | FLOOR_MOD | Yes | | FULLY_CONNECTED | Yes | | GREATER | Yes | | GREATER_EQUAL | Yes | | L2_NORMALIZATION | Yes | | LEAKY_RELU | Yes | | LESS | Yes | | LESS_EQUAL | Yes | | LOG | Yes | | LOGISTIC | Yes | | MAX_POOL_2D | Yes | | MAXIMUM | Yes | | MEAN | Yes | | MINIMUM | Yes | | MUL | Yes | | NEG | Yes | | NOT_EQUAL | Yes | | PAD | Yes | | PADV2 | Yes | | MIRROR_PAD | Yes | | PACK | Yes | | POW | Yes | | REDUCE_MAX | Yes | | REDUCE_MIN | Yes | | REDUCE_PROD | Yes | | RELU | Yes | | PRELU | Yes | | RELU6 | Yes | | RESHAPE | Yes | | RESIZE_BILINEAR | Yes | | RESIZE_NEAREST_NEIGHBOR | Yes | | ROUND | Yes | | RSQRT | Yes | | SHAPE | Yes | | SIN | Yes | | SLICE | Yes | | SOFTMAX | Yes | | SPACE_TO_BATCH_ND | Yes | | SQUEEZE | Yes | | BATCH_TO_SPACE_ND | Yes | | STRIDED_SLICE | Yes | | SQRT | Yes | | SQUARE | Yes | | SUB | Yes | | SUM | Yes | | TANH | Yes | | TILE | Yes | | TRANSPOSE | Yes | | TRANSPOSE_CONV | Yes | | QUANTIZE | Yes | | FAKE_QUANT | Yes | | DEQUANTIZE | Yes | | GATHER | Yes | | GATHER_ND | Yes | | ONE_HOT | Yes | | SQUARED_DIFFERENCE | Yes | | LOG_SOFTMAX | Yes | | SPLIT | Yes | | HARD_SWISH | Yes | ### ONNX Operators | Operator | Is Supported | | --------------------- | ------------ | | Abs | Yes | | Acos | Yes | | Acosh | Yes | | And | Yes | | ArgMax | Yes | | ArgMin | Yes | | Asin | Yes | | Asinh | Yes | | Add | Yes | | AveragePool | Yes | | BatchNormalization | Yes | | Cast | Yes | | Ceil | Yes | | Celu | Yes | | Clip | Yes | | Compress | Yes | | Concat | Yes | | Constant | Yes | | ConstantOfShape | Yes | | Conv | Yes | | ConvTranspose | Yes | | Cos | Yes | | Cosh | Yes | | CumSum | Yes | | DepthToSpace | Yes | | DequantizeLinear | Yes | | Div | Yes | | Dropout | Yes | | Elu | Yes | | Exp | Yes | | Expand | Yes | | Equal | Yes | | Erf | Yes | | Flatten | Yes | | Floor | Yes | | Gather | Yes | | GatherElements | Yes | | GatherND | Yes | | Gemm | Yes | | GlobalAveragePool | Yes | | GlobalMaxPool | Yes | | Greater | Yes | | GreaterOrEqual | Yes | | GRU | Yes | | Hardmax | Yes | | HardSigmoid | Yes | | HardSwish | Yes | | Identity | Yes | | InstanceNormalization | Yes | | LayerNormalization | Yes | | LpNormalization | Yes | | LeakyRelu | Yes | | Less | Yes | | LessOrEqual | Yes | | Log | Yes | | LogSoftmax | Yes | | LRN | Yes | | LSTM | Yes | | MatMul | Yes | | MaxPool | Yes | | Max | Yes | | Min | Yes | | Mul | Yes | | Neg | Yes | | Not | Yes | | OneHot | Yes | | Pad | Yes | | Pow | Yes | | PRelu | Yes | | QuantizeLinear | Yes | | RandomNormal | Yes | | RandomNormalLike | Yes | | RandomUniform | Yes | | RandomUniformLike | Yes | | ReduceL1 | Yes | | ReduceL2 | Yes | | ReduceLogSum | Yes | | ReduceLogSumExp | Yes | | ReduceMax | Yes | | ReduceMean | Yes | | ReduceMin | Yes | | ReduceProd | Yes | | ReduceSum | Yes | | ReduceSumSquare | Yes | | Relu | Yes | | Reshape | Yes | | Resize | Yes | | ReverseSequence | Yes | | RoiAlign | Yes | | Round | Yes | | Rsqrt | Yes | | Selu | Yes | | Shape | Yes | | Sign | Yes | | Sin | Yes | | Sinh | Yes | | Sigmoid | Yes | | Size | Yes | | Slice | Yes | | Softmax | Yes | | Softplus | Yes | | Softsign | Yes | | SpaceToDepth | Yes | | Split | Yes | | Sqrt | Yes | | Squeeze | Yes | | Sub | Yes | | Sum | Yes | | Tanh | Yes | | Tile | Yes | | TopK | Yes | | Transpose | Yes | | Trilu | Yes | | ThresholdedRelu | Yes | | Upsample | Yes | | Unsqueeze | Yes | | Where | Yes | ## API Documentation The `nncase` software stack includes `compiler` and `runtime` parts, which are used for model conversion and KPU model inference respectively. Python and C++ APIs are provided for these two parts. ## Usage Steps ### Environment Setup - **Linux** First, install `dotnet-sdk-7.0` and configure the `dotnet` environment variable. Do not install dotnet in an `anaconda` virtual environment: ```shell sudo apt-get update sudo apt-get install dotnet-sdk-7.0 export DOTNET_ROOT=/usr/share/dotnet ``` Then install `nncase` and `nncase-kpu` : ```shell pip install nncase nncase-kpu ``` - **Windows** First, install `dotnet-sdk-7.0`. For installation steps, see the `Microsoft` official documentation: [Install .NET on Windows](https://learn.microsoft.com/zh-cn/dotnet/core/install/windows) . Then install `nncase` online. In [Release](https://github.com/kendryte/nncase/releases) , select the corresponding version of `nncase_kpu-2.x.x-py2.py3-none-win_amd64.whl` to download, then install locally using `pip install`. ```shell pip install nncase pip install nncase_kpu-2.x.x-py2.py3-none-win_amd64.whl ``` - **Docker** If users do not have an Ubuntu environment, they can use the nncase docker (Ubuntu 20.04 + Python 3.8 + dotnet-7.0) ```shell cd /path/to/nncase_sdk docker pull ghcr.io/kendryte/k230_sdk docker run -it --rm -v `pwd`:/mnt -w /mnt ghcr.io/kendryte/k230_sdk /bin/bash -c "/bin/bash" ``` - **View Version Information** ```shell root@469e6a4a9e71:/mnt# python3 Python 3.8.10 (default, May 26 2023, 14:05:08) [GCC 9.4.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import _nncase >>> print(_nncase.__version__) 2.9.0 ``` ### Model Conversion The `nncase` user guide documentation is available at: [github: user_guide](https://github.com/kendryte/nncase/tree/master/examples/user_guide) or [gitee: user_guide](https://gitee.com/kendryte/nncase/tree/master/examples/user_guide) . Use `nncase` to convert `tflite/onnx` models to `kmodel`. The key to the model conversion code is configuring options according to your needs, mainly `CompileOptions`, `PTQTensorOptions`, and `ImportOptions`. #### CompileOptions The `CompileOptions` class is used to configure `nncase` compilation options. The descriptions of each property are as follows: | Attribute Name | Type | Required | Description | | :--------------------------- | :-------------------: | :------: | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | target | string | Yes | Specifies the compilation target, e.g., 'cpu', 'k230' | | dump_ir | bool | No | Specifies whether to dump IR, default is False | | dump_asm | bool | No | Specifies whether to dump asm assembly files, default is False | | dump_dir | string | No | After enabling switches such as dump_ir above, specifies the directory for dumping, default is "" | | input_file | string | No | Used to specify the parameter file path when the ONNX model exceeds 2GB, default is "" | | preprocess | bool | No | Whether to enable preprocessing, default is False. The following parameters only take effect when `preprocess=True` | | input_type | string | No | Specifies the input data type when preprocessing is enabled, default is "float". When `preprocess` is `True`, it must be specified as "uint8" or "float32" | | input_shape | list[int] | No | Specifies the shape of the input data when preprocessing is enabled, default is []. Must be specified when `preprocess` is `True` | | input_range | list[float] | No | Specifies the floating-point range after dequantization of input data when preprocessing is enabled, default is []. Must be specified when `preprocess` is `True` and `input_type` is `uint8` | | input_layout | string | No | Specifies the layout of the input data, default is "" | | swapRB | bool | No | Whether to reverse data in the `channel` dimension, default is False | | mean | list[float] | No | Mean value for preprocessing normalization, default is [0,0,0] | | std | list[float] | No | Variance for preprocessing normalization, default is [1,1,1] | | letterbox_value | float | No | Specifies the padding value for letterbox preprocessing, default is 0 | | output_layout | string | No | Specifies the layout of the output data, default is "" | | shape_bucket_enable | bool | Yes | Whether to enable the ShapeBucket function, default is False. Takes effect when `dump_ir=True` | | shape_bucket_range_info | Dict[str, [int, int]] | Yes | The range of variables in each input shape dimension information, the minimum value must be greater than or equal to 1 | | shape_bucket_segments_count | int | Yes | How many segments the range of input variables is divided into | | shape_bucket_fix_var_map | Dict[str, int] | No | Fixes variables in shape dimension information to specific values | For the description of preprocessing configuration, please refer to the API documentation: [nncase Model Compilation API Manual Preprocessing Flow](./nncase_compile.md#preprocess). Encapsulating some preprocessing operations within the model can improve the preprocessing efficiency during inference on the development board. The supported preprocessing includes: `swapRB` (RGB->BGR or BGR->RGB), `Transpose` (NHWC->NCHW or NCHW->NHWC), `Normalization` (subtract mean and divide by variance), `Dequantize`, etc. For example, the onnx model requires `RGB` input, but the image we read using `opencv` is in `BGR` format. Normally, for onnx model inference preprocessing, we need to convert `BGR` to `RGB` for the onnx model. When converting to kmodel, we can set `swapRB` to `True`, so that the kmodel contains the preprocessing step for swapping `RB` channels. When performing preprocessing for kmodel inference, we can skip the step of swapping `RB` channels and place this step inside the kmodel. #### PTQTensorOptions The `PTQTensorOptions` class is used to configure `nncase PTQ` options: | Name | Type | Required | Description | | --------------------------------- | ------ | -------- | ------------------------------------------------------------------------------------------------------------------------ | | samples_count | int | No | Specifies the number of calibration sets used for quantization | | calibrate_method | string | No | Specifies the quantization method, optional 'NoClip', 'Kld', default is 'Kld' | | finetune_weights_method | string | No | Specifies whether to fine-tune weights, optional 'NoFineTuneWeights', 'UseSquant', default is 'NoFineTuneWeights' | | quant_type | string | No | Specifies the data quantization type, optional 'uint8', 'int8', 'int16'. `quant_type` and `w_quant_type` cannot both be 'int16' | | w_quant_type | string | No | Specifies the weight quantization type, optional 'uint8', 'int8', 'int16'. `quant_type` and `w_quant_type` cannot both be 'int16' | | quant_scheme | string | No | Path to the import quantization parameter configuration file | | quant_scheme_strict_mode | bool | No | Whether to strictly execute quantization according to quant_scheme | | export_quant_scheme | bool | No | Whether to export the quantization parameter configuration file | | export_weight_range_by_channel | bool | No | Whether to export weights quantization parameters in `bychannel` format. It is recommended to set this parameter to `True` | For the specific usage flow of mixed quantization, see [MixQuant Instructions](https://github.com/kendryte/nncase/blob/release/2.0/docs/MixQuant.md). For the description of quantization configuration, please refer to the API documentation: [nncase Model Compilation API Manual PTQ Options Configuration](./nncase_compile.md#ptqtensoroptions). If the converted kmodel does not achieve the desired effect, you can modify the `quant_type` and `w_quant_type` parameters to change the quantization types of model data and weights, but these two parameters cannot be set to `int16` at the same time. #### Calibration Set Settings | Name | Type | Description | | ---- | ---------------------- | ---------------------- | | data | List[List[np.ndarray]] | Read calibration data | The calibration data used during quantization is set through the `set_tensor_data` method. The interface parameter type is `List[List[np.ndarray]]`. For example: if the model has one input, the calibration data count is set to 10, the dimension of the input calibration data is `[10,1,3,224,224]`; if the model has two inputs, the calibration data count is set to 10, the dimension of the input calibration data is `[[10,1,3,224,224],[10,1,3,320,320]]`. #### ImportOptions The ImportOptions class is used to configure nncase import options, configuring the model to be converted by the compiler. You can configure `tflite/onnx`. The usage example is as follows: ```python # Read and import the tflite model model_content = read_model_file(model) compiler.import_tflite(model_content, import_options) # Read and import the onnx model model_content = read_model_file(model) compiler.import_onnx(model_content, import_options) ``` #### Example: YOLOv8 onnx to kmodel ```python import os import argparse import numpy as np from PIL import Image import onnxsim import onnx import nncase import shutil import math def parse_model_input_output(model_file,input_shape): onnx_model = onnx.load(model_file) input_all = [node.name for node in onnx_model.graph.input] input_initializer = [node.name for node in onnx_model.graph.initializer] input_names = list(set(input_all) - set(input_initializer)) input_tensors = [ node for node in onnx_model.graph.input if node.name in input_names] # input inputs = [] for _, e in enumerate(input_tensors): onnx_type = e.type.tensor_type input_dict = {} input_dict['name'] = e.name input_dict['dtype'] = onnx.mapping.TENSOR_TYPE_TO_NP_TYPE[onnx_type.elem_type] input_dict['shape'] = [(i.dim_value if i.dim_value != 0 else d) for i, d in zip( onnx_type.shape.dim, input_shape)] inputs.append(input_dict) return onnx_model, inputs def onnx_simplify(model_file, dump_dir,input_shape): onnx_model, inputs = parse_model_input_output(model_file,input_shape) onnx_model = onnx.shape_inference.infer_shapes(onnx_model) input_shapes = {} for input in inputs: input_shapes[input['name']] = input['shape'] onnx_model, check = onnxsim.simplify(onnx_model, input_shapes=input_shapes) assert check, "Simplified ONNX model could not be validated" model_file = os.path.join(dump_dir, 'simplified.onnx') onnx.save_model(onnx_model, model_file) return model_file def read_model_file(model_file): with open(model_file, 'rb') as f: model_content = f.read() return model_content def generate_data(shape, batch, calib_dir): img_paths = [os.path.join(calib_dir, p) for p in os.listdir(calib_dir)] data = [] for i in range(batch): assert i < len(img_paths), "calibration images not enough." img_data = Image.open(img_paths[i]).convert('RGB') img_data = img_data.resize((shape[3], shape[2]), Image.BILINEAR) img_data = np.asarray(img_data, dtype=np.uint8) img_data = np.transpose(img_data, (2, 0, 1)) data.append([img_data[np.newaxis, ...]]) return np.array(data) def main(): parser = argparse.ArgumentParser(prog="nncase") parser.add_argument("--target", default="k230",type=str, help='target to run,k230/cpu') parser.add_argument("--model",type=str, help='model file') parser.add_argument("--dataset_path", type=str, help='calibration_dataset') parser.add_argument("--input_width", type=int, default=320, help='model input_width') parser.add_argument("--input_height", type=int, default=320, help='model input_height') parser.add_argument("--ptq_option", type=int, default=0, help='ptq_option:0,1,2,3,4,5') args = parser.parse_args() # Update parameters to multiples of 32 input_width = int(math.ceil(args.input_width / 32.0)) * 32 input_height = int(math.ceil(args.input_height / 32.0)) * 32 # The input shape of the model, dimensions should match input_layout input_shape=[1,3,input_height,input_width] dump_dir = 'tmp' if not os.path.exists(dump_dir): os.makedirs(dump_dir) # onnx simplify model_file = onnx_simplify(args.model, dump_dir,input_shape) # Set CompileOptions compile_options = nncase.CompileOptions() compile_options.target = args.target # Whether to use kmodel for preprocessing compile_options.preprocess = True # The onnx model requires RGB, and the camera data on k230 is also in RGB format, so there is no need to enable RB swap compile_options.swapRB = False # The shape of the input image compile_options.input_shape = input_shape # The model input format 'uint8' or 'float32' compile_options.input_type = 'uint8' # If the input is in 'uint8' format, the range after dequantization compile_options.input_range = [0, 1] # The mean/std values for preprocessing, one per channel, this data is obtained from the YOLOv8 source code compile_options.mean = [0, 0, 0] compile_options.std = [1, 1, 1] # Set the input layout, onnx default 'NCHW' is fine compile_options.input_layout = "NCHW" # Create Compiler instance compiler = nncase.Compiler(compile_options) # Import onnx model model_content = read_model_file(model_file) import_options = nncase.ImportOptions() compiler.import_onnx(model_content, import_options) # Configure quantization method ptq_options = nncase.PTQTensorOptions() ptq_options.samples_count = 10 if args.ptq_option == 0: ptq_options.calibrate_method = 'NoClip' ptq_options.quant_type = 'uint8' ptq_options.w_quant_type = 'uint8' elif args.ptq_option == 1: ptq_options.calibrate_method = 'NoClip' ptq_options.quant_type = 'uint8' ptq_options.w_quant_type = 'int16' elif args.ptq_option == 2: ptq_options.calibrate_method = 'NoClip' ptq_options.quant_type = 'int16' ptq_options.w_quant_type = 'uint8' elif args.ptq_option == 3: ptq_options.calibrate_method = 'Kld' ptq_options.quant_type = 'uint8' ptq_options.w_quant_type = 'uint8' elif args.ptq_option == 4: ptq_options.calibrate_method = 'Kld' ptq_options.quant_type = 'uint8' ptq_options.w_quant_type = 'int16' elif args.ptq_option == 5: ptq_options.calibrate_method = 'Kld' ptq_options.quant_type = 'int16' ptq_options.w_quant_type = 'uint8' else: pass # Set calibration data ptq_options.set_tensor_data(generate_data(input_shape, ptq_options.samples_count, args.dataset_path)) compiler.use_ptq(ptq_options) # Start compilation compiler.compile() # Write kmodel file kmodel = compiler.gencode_tobytes() base,ext=os.path.splitext(args.model) kmodel_name=base+".kmodel" with open(kmodel_name, 'wb') as f: f.write(kmodel) if __name__ == '__main__': main() ``` After the model conversion is successful, the code needs to be deployed on the development board, which requires using `nncase_runtime` to write C++ code. ### Deployment Code Writing The deployment code writing takes YOLOv8 detection as an example. The source code is in the `src/rtsmart/examples/kpu_run_yolov8` directory. Execute `build_app.sh` in this directory to obtain the compiled executable files for image inference and camera inference in the `k230_bin` directory under that directory. The process of using the KPU runtime API to perform inference on the model is as follows.
graph TD; LoadModel("Initialize interpreter instance
Load model") -->SetInput("Get input shape
Initialize input tensors")-->SetOutput("Get output shape
Initialize output tensors")-->GetFrame("Read data to be inferred
Read image/from camera")-->SetPreprocessParam("Set preprocessing parameters, including AI2D preprocessing method configuration, input tensor, output tensor")-->PreProcess("Execute preprocessing so that the read image or video frame conforms to the **model input**")-->KPURun("Execute model inference")-->GetOutput("Get the output pointer of model inference")-->PostProcess("Post-process the output according to the specific scenario")-->DrawResult("Draw the post-processing results on the image/screen");
The preprocessing methods provided by `AI2D` are implemented in hardware, which can improve running efficiency. `Interpreter` is used to complete model inference on KPU. Their inputs and outputs are all `host_runtime_tensor` type data. The model input may be one or multiple; the processing result of AI2D is generally given to the model for use. When initially initializing `ai2d_builder` and `Interpreter`, the input and output tensors of the two components are generally initialized together. The schematic diagrams of the two components are as follows: ![AI2D](https://www.kendryte.com/api/post/attachment?id=511) Because the output tensor of `AI2D` will be given to the input tensor of the model for inference, when there is a single input, the output tensor of `AI2D` and the input tensor of `Interpreter` can be bound as one, which can save the memory of one tensor. If `AI2D` is not used for preprocessing, `OpenCV` can be used to preprocess the input data, and then create a `host_runtime_tensor`. The schematic diagram is as follows: ![pipe_inference](https://www.kendryte.com/api/post/attachment?id=510)