AI Development Documentation#
Introduction to Basic Principles of KPU Hardware#
In edge computing scenarios (such as IoT devices, smart cameras, industrial inspection terminals, wearable devices, etc.), devices are typically deployed at sites far from cloud data centers, facing challenges of high real-time requirements, limited network bandwidth, sensitive data privacy, and strict power constraints. Running complex AI models in these scenarios (such as image recognition, object detection, voice wake-up), if relying solely on traditional general-purpose CPUs for computation, often encounters problems of excessive computational load, slow processing speed, and high power consumption, making it difficult to meet the requirements of real-time response and energy efficiency ratio.
KPU (Knowledge Processing Unit) is a hardware acceleration engine specifically designed by Canaan Technology to address edge AI computing challenges. It is essentially a highly optimized deep learning coprocessor/accelerator, whose core function is to efficiently execute dense computational tasks in neural network models (particularly operations such as convolution, matrix multiplication, and activation functions).
Core Advantages of KPU: Specialization and Efficiency. Compared with general-purpose CPUs, KPU’s advantages lie in its specialized architecture:
Parallel Computing Capability: KPU internally contains a large number of processing elements (PE) specifically designed for neural network computation, capable of simultaneously processing massive amounts of data (such as feature maps, weights), significantly accelerating the model inference process.
Optimized Data Flow and Memory Access: Deep optimization for neural network computation patterns (such as data reuse) reduces unnecessary data movement, maximizes memory bandwidth utilization, and lowers latency.
High Energy Efficiency Ratio: The dedicated circuit design avoids the overhead of CPUs executing general-purpose instructions. When performing the same AI computational tasks, KPU typically provides tens to hundreds of times the computational efficiency of CPUs (TOPS/W), achieving high-performance AI processing within the limited power budget of edge devices.
Reduced CPU Load: Offloading heavy AI computational tasks to the KPU for execution frees up valuable CPU resources to handle other critical tasks such as device control, communication, and user interaction, improving overall system responsiveness and stability.
KPU supports various mainstream neural network model structures, suitable for a wide range of edge vision AI application scenarios, including but not limited to:
Image Classification: Identifying object categories in images (such as identifying fruit types, industrial parts).
Object Detection: Locating and identifying multiple targets and their positions in images (such as detecting pedestrians, vehicles, defects).
Semantic Segmentation: Classifying every pixel in an image (such as distinguishing roads, sky, buildings; medical image analysis).
Face Detection and Recognition: On-device face verification, access control and attendance.
Pose Estimation: Analyzing human joint positions (such as fitness movement guidance).
KPU’s Position in the System:
It typically exists as an independent IP core in the SoC (System on Chip), working collaboratively with the CPU, memory, peripherals, etc. The CPU is responsible for system management, task scheduling, and application logic, while computationally intensive AI model inference tasks are efficiently handed over to the KPU for execution. The figure below shows the position of KPU in a typical edge AI SoC.
K230 AI Application Examples Showcase#
To help developers quickly get started and intuitively experience K230’s powerful edge AI capabilities, the K230 Linux source code image includes rich and diverse AI example programs (AI Demos).
These out-of-the-box demos cover two major categories: single-model applications (such as face detection) and multi-model applications (such as hand keypoints). Users only need a single command to compile the application and experience mainstream AI features. They also support make menuconfig compilation configuration, including but not limited to:
Visual Applications: Object recognition, face detection, gesture recognition, human body recognition, license plate recognition, OCR text recognition.
Audio Applications: Keyword recognition (KWS), Chinese text-to-speech (TTS), etc.
Through these demos, developers can quickly validate model performance, become familiar with K230’s AI inference capabilities, and lay a solid foundation for subsequent customized development.
How to Run:
All demo source code is open, with clear structure, uniformly stored in the k230_linux_sdk/buildroot-overlay/package/ai_demo directory. Users can build their own compilation environment and complete the compilation, running, debugging, and in-depth study of aidemo code, understand API calls, data processing flows, and model integration methods, greatly accelerating the development process of their own applications. For specific content, please refer to the document: K230_AI_Demo User Guide.
Notes:
Some demos may not run properly on the K230D chip due to high memory usage.
For the differences between K230 and K230D, please refer to: Product Center
Demo Subdirectory |
Scenario |
Description |
Link |
|---|---|---|---|
anomaly_det |
Anomaly Detection |
The model provided in the anomaly detection example is trained using the patchcore anomaly detection method, which can identify whether there is an anomaly at the glass bottle mouth from the input image. Anomaly detection is commonly applied in industrial image inspection, medical image analysis, security monitoring, and other fields. |
|
bytetrack |
Multi-Object Tracking |
The ByteTrack multi-object tracking example uses YOLOv5 as the object detection algorithm, applies the Kalman filter algorithm for bounding box prediction, and applies the Hungarian algorithm for matching between targets and trajectories. |
|
crosswalk_detect |
Crosswalk Detection |
Crosswalk detection uses the YOLOV5 network. This application detects crosswalks in images or videos and can be used in scenarios such as assisted driving. |
|
demo_mix |
Example Collection |
The example collection integrates examples from k230 such as hand keypoint detection, gesture recognition, dynamic gesture recognition, face pose estimation, and target tracking to implement gesture-controlled application switching. When showing the “1” gesture, it enters dynamic gesture recognition; when showing the “2” gesture, it enters face pose estimation; when showing the “3” gesture, it enters automatic face tracking. When showing the “love” gesture, it exits the current task and enters the switching state. |
|
dynamic_gesture |
Visual Dynamic Gesture Recognition |
Visual dynamic gesture recognition can recognize five actions: waving up, down, left, right, and pinching all fingers together, used for contactless operation control scenarios. The palm detection uses the yolov5 network structure with 1.0-mobilenetV2 as the backbone, palm keypoint detection uses the resnet50 network structure, and dynamic gesture recognition uses the tsm structure with mobilenetV2 as the backbone. |
|
eye_gaze |
Gaze Estimation |
The gaze estimation example predicts where a person is looking based on their face. For video frames or images, it first performs face detection, then performs gaze estimation on each face, predicts the gaze vector, and displays it on the screen with an arrow. This application uses the retina-face network for face detection and L2CS-Net for gaze estimation. Gaze estimation can be applied in the automotive safety field. |
|
face_alignment |
Face Alignment |
The face alignment example can obtain the depth map or normalized projection coordinate encoding map of each face in the image or video. Face detection uses the retina-face network structure with 0.25-mobilenet as the backbone, and the face alignment network is implemented based on 3DDFA (3D Dense Face Alignment). |
|
face_detection |
Face Detection |
The face detection example can obtain the detection box for each face in the image or video as well as the positions of five keypoints on each face: left eye, right eye, nose tip, left mouth corner, and right mouth corner. Face detection uses the retina-face network structure with 0.25-mobilenet as the backbone. |
|
face_emotion |
Facial Expression Recognition |
Facial expression recognition uses two models to implement expression recognition for each person in images/videos. Recognizable expression categories include Neutral, Happiness, Sadness, Anger, Disgust, Fear, and Surprise. Face detection uses the retina-face network structure; expression classification uses mobilenet as the backbone for classification to obtain the person’s expression. |
|
face_gender |
Gender Classification |
The face gender classification example uses two models to implement the function of determining the gender of each person in images/videos. Each person’s gender is represented by M or F, where M stands for Male and F stands for Female. Face detection uses the retina-face network structure; gender classification uses EfficientNetB3 as the backbone for classification to obtain the person’s gender. |
|
face_glasses |
Glasses Wearing Classification |
The glasses wearing classification example uses two models to implement the determination of whether each person in an image/video is wearing glasses. The face detection model uses the retina-face network structure; the face glasses classification model uses SqueezeNet-1.1 as the backbone to determine glasses wearing for each face box. |
|
face_landmark |
Dense Face Keypoints |
The dense face keypoint detection application uses two models to detect 106 keypoints for each face in images/videos, and draws the contours of the face and facial features based on the 106 keypoints, with different contours represented in different colors. Face detection uses the retina-face network structure; dense keypoint detection uses 0.5-mobilenet as the backbone to detect 106 keypoints for each face. The 106 keypoints include the cheek, mouth, eyes, nose, and eyebrow areas of the face. |
|
face_mask |
Mask Wearing Classification |
The mask wearing classification application uses two models to implement the determination of whether each person in images/videos is wearing a mask. In application scenarios where masks are required, if someone is found not wearing a mask, relevant reminders can be issued. The face detection model uses the retina-face network structure; the face mask classification model uses mobilenet-v2 as the backbone to determine mask wearing for each face box. |
|
face_mesh |
3D Face Mesh |
The 3D face mesh can obtain the 3D mesh structure of each face in images or videos. Face detection uses the retina-face network structure with 0.25-mobilenet as the backbone, and the face alignment network is implemented based on 3DDFA (3D Dense Face Alignment). |
|
face_parse |
Face Segmentation |
The face segmentation example uses two models to implement segmentation of each face in images/videos. Face segmentation involves distinguishing facial features such as eyes, nose, and mouth at the pixel level, with different regions represented in different colors. Face detection uses the retina-face network structure, and face region segmentation uses the DeepNetV3 network structure with mobilenet-1.0 as the backbone. |
|
face_pose |
Face Pose Estimation |
Face pose estimation uses two models to implement the function of estimating the angle of each person’s face orientation in images/videos. Face orientation is generally represented by Euler angles (roll/yaw/pitch), where roll represents the degree of left-right head shaking; yaw represents the degree of left-right face rotation; pitch represents the degree of face looking up or down. Face detection uses the retina-face model, and face orientation estimation fits 98 2D keypoints. |
|
face_verification |
Face Identity Verification |
Face identity verification is an identity verification technology based on facial biometric features, aimed at confirming whether an individual is the identity they claim to be. This technology verifies identity by analyzing and comparing the user’s facial features, typically by comparing two images in a face verification system to determine whether the faces in the two images belong to the same person. Face detection uses the retina-face model, and face feature extraction uses ResNet50, outputting 512-dimensional features. |
|
falldown_detect |
Fall Detection |
Fall detection can detect the fall state of people in images or videos. This example uses the yolov5n model. |
|
finger_guessing |
Rock-Paper-Scissors Game |
The rock-paper-scissors game example distinguishes rock, paper, and scissors through hand gesture recognition, including two models: palm detection and 21 hand keypoint recognition. The gesture category is determined by the position constraints of the 21 keypoints. The palm detection part uses the yolov5 network structure with 1.0-mobilenetV2 as the backbone, and the palm keypoint detection part uses the resnet50 network structure. |
|
fitness |
Squat Counter |
The squat counter example implements the squat counting function for people in videos, suitable for fitness state detection and other scenarios. Implemented using the yolov8n-pose model. |
|
head_detection |
Head Detection and Counting |
The head detection and counting example implements the function of obtaining the coordinates and quantity of heads appearing in images or videos. Implemented using the yolov8 model. |
|
helmet_detect |
Helmet Detection |
The helmet detection example implements the detection of whether people appearing in images or videos are wearing helmets, suitable for safety prevention scenarios in the construction and manufacturing industries. Implemented using the yolov5 model. |
|
kws |
Keyword Wake-Up |
Keyword wake-up detects whether the audio stream contains the keywords set during training through the audio recognition model. If the corresponding keyword is detected, a voice response is given. The model provided in this example is trained with WeNet. Positive and negative samples respectively use the “xiaonan” audio collected on the k230 development board and the open-source dataset speech_commands. |
|
licence_det |
License Plate Detection |
License plate detection can detect license plates appearing in images or videos. The license plate detection uses the retinanet network structure. |
|
licence_det_rec |
License Plate Recognition |
License plate recognition can identify the position of license plates in images or videos as well as the license plate information. License plate detection uses the retinanet network structure, and license plate recognition uses the RLNet network structure with MobileNetV3 as the backbone. |
|
object_detect_yolov8n |
YOLOV8 Multi-Object Detection |
The YOLOv8 multi-object detection example implements 80-class detection on the COCO dataset. Uses the yolov8n model. |
|
ocr |
OCR Detection + Recognition |
The OCR recognition example can detect the text position in images or videos and the corresponding text content. The OCR recognition task uses the CRNN network structure, and the OCR detection task uses the DBnet network structure. |
|
person_attr |
Human Attributes |
Human attribute detection can identify the position coordinates, gender, age, whether wearing glasses, and whether holding objects of people in images or videos. Human detection is implemented using the YOLOv5 model, and human attributes are implemented using the PULC human model. |
|
person_detect |
Person Detection |
Person detection can detect the position coordinates of human bodies in images or videos and mark them with detection boxes. This example uses the yolov5 model. |
|
person_distance |
Pedestrian Distance Estimation |
Pedestrian distance estimation first detects pedestrians through pedestrian detection, then estimates the target distance through the size of the detection box in the image. The pedestrian detection uses the yolov5n network structure. Using this application, you can obtain the detection box and estimated distance for each pedestrian in the image or video. This technology can be applied in vehicle assisted driving systems, intelligent transportation, and other fields. This application requires adjusting calculation data based on the camera, and the current example may not recognize accurately. |
|
pose_detect |
Human Keypoint Detection |
The output of the human keypoint detection model is a set of keypoints (17) representing the human body objects in the image or video, along with the confidence score for each point, using lines of different colors to connect the keypoints into the shape of a human body. This example uses the yolov8n-pose model. |
|
pphumanseg |
Portrait Segmentation |
Portrait segmentation refers to identifying the outline range of the human body in images or videos, separating it from the background, and returning the segmented binary image, grayscale image, foreground portrait image, etc., enabling replacement and synthesis of background images. It can be applied to scenarios such as portrait matting, photo synthesis, portrait effects, and background effects, greatly improving the efficiency of image and video tools. This example uses the pphumanseg model. |
|
puzzle_game |
Jigsaw Puzzle Game |
The jigsaw puzzle game can obtain the 21 skeletal keypoint positions of each palm in images or videos. It can also implement the jigsaw puzzle function: spread the thumb and middle finger, place the midpoint next to the empty space’s non-empty space, fit the two fingers, and the current non-empty space will move into the empty space. In the example, palm detection uses the yolov5 network structure with 1.0-mobilenetV2 as the backbone, and palm keypoint detection uses the resnet50 network structure. |
|
segment_yolov8n |
YOLOV8 Multi-Object Segmentation |
The YOLOv8 multi-object segmentation example implements 80-class segmentation masks on the COCO dataset. Uses the yolov8n-seg model. |
|
smoke_detect |
Smoking Detection |
Smoking detection performs real-time monitoring and identification of smoking behavior in images or videos. This example uses the yolov5 model. |
|
space_resize |
Gesture Contactless Zoom |
Gesture contactless zoom can obtain the 21 skeletal keypoint positions of each palm in images or videos, and we implement contactless image zooming through the thumb and middle finger. Palm detection uses the yolov5 network structure with 1.0-mobilenetV2 as the backbone, and palm keypoint detection uses the resnet50 network structure. |
|
sq_hand_det |
Palm Detection |
Palm detection can obtain the detection box for each palm in images or videos. Palm detection uses the yolov5 network structure with 1.0-mobilenetV2 as the backbone. |
|
sq_handkp_class |
Palm Keypoint Gesture Classification |
Palm keypoint gesture classification can obtain the 21 skeletal keypoint positions of each palm in images or videos, and obtains static gestures based on the 2D constraints of the keypoint positions. It supports a total of 9 gestures: fist, five fingers spread, one gesture, yeah gesture, three gesture, eight gesture, six gesture, thumbs up, and thumb-index-pink spread. In this example, palm detection uses the yolov5 network structure with 1.0-mobilenetV2 as the backbone, and palm keypoint detection uses the resnet50 network structure. |
|
sq_handkp_det |
Palm Keypoint Detection |
The palm keypoint detection example can obtain the 21 skeletal keypoint positions of each palm in images or videos. Palm detection uses the yolov5 network structure with 1.0-mobilenetV2 as the backbone, and palm keypoint detection uses the resnet50 network structure. |
|
sq_handkp_flower |
Fingertip Region Flower Classification |
Fingertip region flower recognition can obtain the flower category within the area surrounded by the index fingertips of two palms in images or videos. It can recognize up to 102 flower species. In this example, palm detection uses the yolov5 network structure with 1.0-mobilenetV2 as the backbone, and palm keypoint detection uses the resnet50 network structure. The flower classification uses 1.0-mobilenetV2 as the backbone. |
|
sq_handkp_ocr |
Finger Region OCR Recognition |
Finger region OCR recognition can identify the text within the upper-left region of each palm’s index finger in images or videos. Palm detection uses the yolov5 network structure with 1.0-mobilenetV2 as the backbone, and palm keypoint detection uses the resnet50 network structure. Text detection uses the retinanet network structure, and text recognition uses the RLnet network structure with MobileNetV3 as the backbone. |
|
sq_handreco |
Gesture Recognition |
Gesture recognition can obtain the category of each gesture in images or videos. Only supports three gestures: five fingers spread, eight gesture, and yeah gesture. In this example, palm detection uses the yolov5 network structure with 1.0-mobilenetV2 as the backbone, and gesture recognition uses 1.0-mobilenetV2 as the backbone. |
|
traffic_light_detect |
Traffic Light Detection |
Can detect red, green, and yellow traffic lights in images or videos. This example uses the yolov5 model. |
|
tts_zh |
Chinese Text-to-Speech |
Chinese text-to-speech (text to chinese speech, tts_zh) uses three models. By default, the user inputs text three times to generate wav files corresponding to the text. This example splits the FastSpeech2 model into two models: Encoder+Variance Adaptor is fastspeech1, Decoder is fastspeech2, and the vocoder uses hifigan. Duration features are added after fastspeech1. |
|
vehicle_attr |
Vehicle Attribute Recognition |
Vehicle attribute recognition can identify each vehicle in images or videos and return the vehicle’s position coordinates, vehicle type, and body color. This example uses the yolov5 network structure for vehicle detection and the PULC model for vehicle attribute detection. |
|
virtual_keyboard |
Contactless Virtual Keyboard |
The contactless virtual keyboard can use the virtual keyboard on the screen to output characters. Pinching the thumb and index finger is the input action. In this example, palm detection uses the yolov5 network structure with 1.0-mobilenetV2 as the backbone, and palm keypoint detection uses the resnet50 network structure. |
|
yolop_lane_seg |
Road Lane Line Segmentation |
Road lane line segmentation can perform road surface segmentation in images or videos, detecting lane lines and drivable areas and color-distinguishing them. This example uses the yolop model. |
Basic Process of AI Model Inference#
The basic process of deploying the trained AI model on K230 is shown in the flowchart below:
🏷️ Data Collection:
Data collection refers to the process of gathering raw input data through sensing devices such as cameras and microphones. The quality and quantity of collected data directly determine the effect of model training and inference. Therefore, selecting appropriate collection devices and strategies is crucial.
For better deployment results, it is recommended to use the K230 itself to collect image data to ensure the data distribution is closer to the actual deployment environment.
🏷️ Data Annotation:
Data annotation is the process of adding semantic labels to the collected data, used for training supervised learning models. This process can be done manually or semi-automatically with the help of annotation tools.
For example, an image classification task needs to assign the correct category label to each image; an object detection task needs to add bounding boxes and their category labels to each target in the image. Accurate annotation is crucial for training high-performance models with strong generalization capabilities.
🏷️ Model Training:
The model training stage is one of the important steps in the entire AI application development process. Its main goal is to use the labeled dataset to train a neural network model with generalization capabilities through deep learning methods. During this process, the model continuously adjusts internal parameters to gradually fit the distribution characteristics of the data, so that it can still make accurate and stable predictions when faced with unseen input data.
Model training usually relies on a large amount of high-quality sample data covering diverse scenarios and categories related to the task. The sufficiency of data and the accuracy of annotation directly affect the model’s learning effectiveness and application performance. During training, the neural network model extracts features from the input data, computes prediction outputs, and generates a loss by comparing with the true labels, then adjusts the weight parameters in the network through backpropagation to continuously optimize model performance.
To achieve efficient training, developers need to choose a model structure suitable for the current task, such as MobileNet and ResNet for image classification, YOLO series for object detection, etc. The choice of model depends not only on accuracy requirements but also on factors such as inference speed, model size, and resource constraints of the deployment platform. Especially for edge AI chips like K230, lightweight models have more practical value.
In addition, the training process often needs to be carried out on a computing platform with certain computing power support (such as a GPU server or local high-performance workstation) to ensure that the model completes optimization within a reasonable time. Modern deep learning training typically uses mature training frameworks such as PyTorch and TensorFlow, which provide rich neural network building blocks, optimizers, loss functions, and data processing tools, greatly simplifying the model development process. You can choose the appropriate framework for training based on your technical background and model requirements.
🏷️ Model Conversion and Verification:
Due to the limited computational resources of edge devices, models trained on high-computing platforms cannot be directly deployed. The model must be optimized and quantized through model conversion tools to generate an inference format suitable for the target hardware.
For the K230 chip:
Use KPU (Knowledge Processing Unit) as the neural network acceleration unit;
The supported model format is KModel;
Use the nncase compiler to convert trained ONNX or TFLite models into KModel;
Structural optimization and quantization are performed during the conversion process to reduce model size and computational complexity.
After conversion is complete, functional verification is required to ensure that the model meets application requirements in terms of accuracy, latency, and resource usage.
🏷️ Model Deployment:
The verified KModel can be loaded onto the device through the API provided by the K230 Linux SDK for execution.
The deployment process usually includes the following steps:
Load kmodel;
Read input data such as images/audio;
Perform data preprocessing (such as scaling, normalization, channel rearrangement, etc.);
Run model inference;
Perform result post-processing (such as classification decoding, bounding box filtering, etc.);
Draw/output inference results.
Different models may have different preprocessing and post-processing flows, and the corresponding code logic needs to be manually adapted according to the specific model.
🏷️ Model Tuning:
After deployment is complete, the model still needs to be tuned in terms of performance and effectiveness to fit the actual needs of edge scenarios. Optimization measures include but are not limited to:
Set more reasonable inference thresholds or output strategies;
Adjust model conversion parameters (such as quantization strategy, input resolution);
Improve model structure or training hyperparameters;
Enrich and optimize the dataset;
Optimize the inference process (such as thread scheduling, memory reuse).
Model tuning is a continuous iterative process that helps improve system stability, real-time performance, and energy efficiency.
The above six steps constitute the complete process of AI model deployment and inference on the K230 chip. Each step requires careful design and execution to ensure that the final application has good performance, stability, and user experience.
Training the Model#
Note
🤖 【Scenario Definition】: Implement “printed digit recognition and localization” on the K230 development board.
📌 Task Background: In many AI applications, we often encounter the need to “identify certain things in images”, such as recognizing faces, objects in images, or as in this example, recognizing digits. To better understand the basic process of object detection, we designed a simple small task——recognize the four digits “0”, “1”, “2”, “3” printed on paper and mark their positions in the image.
This task is not complicated, but it can completely practice the entire process from model deployment to image processing and result display. It serves as an introductory tutorial to help everyone quickly master how to deploy AI models on the K230 platform, perform object detection, and display the detection results on the screen.
🎯 Project Goal: Based on the Kendryte K230 AI SoC platform, develop a lightweight, high-performance visual recognition example that implements the following functions:
✅ Recognition Categories: Only recognize the four digit characters “0”, “1”, “2”, “3”;
✅ Recognition Object: Standard font digits printed on paper;
✅ Localization Function: Not only recognize the digit category, but also accurately obtain the position coordinates of each digit in the image (draw detection boxes), providing a basis for subsequent processing or operations;
✅ Running Platform: The application is deployed on the K230 development board, utilizing its AI hardware acceleration, camera input, and screen display capabilities to achieve on-device inference and real-time display.
🖼 Expected Effect:
Data Collection#
Note
👉 Collecting training data is actually very simple! You only need to flash the MicroPython firmware onto the development board first, then find the script——/sdcard/examples/16-AI-Cube/DataCollectionCamera.py, rename it to main.py, and put it in the /sdcard directory. Then power on again (i.e., restart the board), and after running, press the key button on the board to start collecting! Each press takes a photo, and the images are automatically saved to the /sdcard/examples/data/ folder, completely without your management, super worry-free!
Before training the model, data collection is the first step in the entire process and is also a crucial step. High-quality data can not only improve model performance but also enhance the model’s generalization capability in real application scenarios. According to different application needs, data collection can be divided into general scenarios and specialized scenarios, which will be described in detail below.
📌 Data Collection in General Scenarios
In general artificial intelligence tasks, such as image classification, object detection, semantic segmentation, etc., existing public datasets can usually be used to construct training samples. These datasets are organized and released by academic institutions, research organizations, or large enterprises, with good annotation quality and a wide range of application foundations.
For example, common public image datasets include: ImageNet, COCO, MNIST, Fashion-MNIST, CIFAR series, etc., or you can find open-source datasets for corresponding scenarios on the network.
Although public datasets are of high quality, appropriate filtering and processing are still required before actual use to ensure they meet project requirements:
Quality Assurance Processing: Remove blurry, mislabeled, or low-quality samples.
Class Balance: Ensure balanced sample quantity for each class to avoid model bias.
Format Unification: Convert data to a unified format (such as JPEG, PNG, etc.).
Data Augmentation: Expand data volume through rotation, cropping, flipping, adding noise, etc., to improve model robustness.
Build Customized Datasets: Sometimes a single dataset may not meet specific needs. You can build a customized dataset more suitable for business scenarios by combining multiple datasets and re-annotating and cleaning them.
📌 Data Collection in Specialized Scenarios
For some special industries or specific application scenarios (such as industrial quality inspection, agricultural monitoring, security monitoring, medical diagnosis, etc.), it is often necessary to collect data dedicated to that scenario. In this case, public datasets may not accurately reflect the data distribution of the real environment, so customized data collection is required.
In some specific AI deployment scenarios, you can directly use the K230 device for data collection when conditions permit. The data collected this way is closer to the actual deployment environment, helping to improve model performance on the device.
⚠️ Here are some data collection process suggestions:
Clarify Collection Goals: Define collection objects (such as object types, scenes), lighting conditions, angles, resolution, etc.
Clarify Data Task: Different tasks have different requirements for datasets. On one hand, the actual deployment scenario must be considered; on the other hand, task requirements must be considered. For example, classification tasks may require the object to occupy a large area, and large backgrounds may affect the classification effect; while object detection can have multiple objects of different sizes.
Use Appropriate Tools: Use the K230 development board with camera modules, and scripts can be written for automatic collection.
Synchronize Annotation Information: Try to synchronously record label information during the collection process to facilitate later annotation.
Preliminary Quality Check: Eliminate invalid samples such as blurriness, overexposure, and severe occlusion.
Data Annotation#
Note
👉 After getting the collected images, you can start labeling them! According to the requirements of this task, you can use some common annotation tools, such as LabelImg, Labelme, or X-AnyLabeling, to add corresponding categories to the digits in the images and draw target boxes. You can collect images yourself and annotate them manually, which is also quite interesting. Of course, if you don’t want to do it from scratch, we have thoughtfully prepared a ready-made “0/1/2/3 four-class printed digit recognition” dataset, just click here to download: 0/1/2/3 Four-Class Printed Digit Recognition Dataset. Saves time and effort, and you can directly start training without any problem!
Data annotation is one of the key steps in training the model, which involves annotating raw data so that the model can learn the features and patterns of the data. When performing data annotation, the following aspects need to be considered:
Annotation Format: Choose an annotation format suitable for the model, such as XML, JSON, TXT, etc.
Annotation Tool: Choose a suitable annotation tool, such as LabelImg, Labelme, X-AnyLabeling, VIA, etc.
Annotation Quality: Ensure the accuracy and consistency of annotations to avoid annotation errors.
Annotation Strategy: According to task requirements and data characteristics, choose a suitable annotation strategy, such as bounding box annotation, keypoint annotation, etc.
For common vision tasks, it is recommended to use X-AnyLabeling for annotation. Download link: X-AnyLabeling-release.
Model Training#
Note
👉 There are many methods for model training, among which the YOLO series is a particularly commonly used choice now, such as YOLOv5, YOLOv8, or YOLO11. We recommend you use YOLO for training because it has good results, fast speed, and an active community. Even better, the dataset we provide has been organized and can be directly used to train the YOLO model! You just need to jump to this example: YOLO Detection Example, follow the process inside, and replace the dataset part in the example with our prepared “0/1/2/3 four-class printed digit recognition dataset”! The goal of this section is to first train the model well and export it to ONNX format. There are more interesting content waiting for you to unlock later!
Model training is the most important step in the entire AI process, which involves model construction, training, and optimization. When training the model, the following aspects need to be considered:
Model Selection: Choose a suitable model based on task requirements and data characteristics.
Model Construction: Construct the network structure of the model, including input layers, hidden layers, and output layers.
Model Training: Use labeled data for model training, including choosing appropriate loss functions and optimizers.
Model Evaluation: Use the test set to evaluate the model, evaluating model performance and generalization capability.
Model Optimization: Optimize the model based on evaluation results to improve model performance and generalization capability.
The trained model needs to be converted to an onnx model or tflite model, preparing for subsequent use of nncase for model conversion to obtain a kmodel that can be inferred on K230.
Model Conversion#
After training is complete, we obtain an ONNX model file. However, this model cannot be directly run on the K230 using the KPU, because the KPU only supports the Kmodel format.
Therefore, next, we need to use a compiler called nncase to “translate” the ONNX model into a Kmodel, so that the KPU can understand and run it.
Let’s take a brief look at this key tool — nncase!
What is nncase#
Introduction to nncase#
nncase is a neural network compiler specifically designed for AI accelerators. Currently supported backends (targets) include: CPU, K210, K510, K230, etc.
Core features provided by nncase:
Supports multiple-input multiple-output network structures, compatible with common multi-branch model topologies;
Adopts a static memory allocation strategy, with no dependency on runtime heap memory, and controllable resource usage;
Implements operator fusion and graph optimization, effectively reducing redundant computation and improving inference efficiency;
Supports floating-point (float) inference and fixed-point quantized inference (uint8/int8);
Supports Post-Training Quantization (PTQ), which can generate efficient quantized models based on floating-point models and calibration datasets;
The compiled model is a Flat Model with Zero-Copy Loading capability, suitable for resource-constrained embedded scenarios.
Supported model formats:
nncase supports the following model formats exported from mainstream deep learning frameworks:
TFLite (TensorFlow Lite)
ONNX (Open Neural Network Exchange)
You can use training frameworks such as PyTorch and TensorFlow to export models to the above formats, and then use nncase to convert them to KModel for deployment on devices such as the K230.
Architecture overview:
The nncase software stack mainly consists of the following two major components:
Compiler: Converts TFLite or ONNX models exported from high-level frameworks into the KModel format suitable for the target hardware platform, and performs structural optimization, operator scheduling, and quantization processing;
Runtime: Loads and runs KModel on the target device (such as K230), combined with hardware acceleration units (such as the KPU) to achieve high-performance model inference.
🏷️ Compiler: Used to compile neural network models on a PC, ultimately generating kmodel files. It mainly includes modules such as importer, IR, Evaluator, Quantize, Transform optimization, Tiling, Partition, Schedule, and Codegen.
Importer: Imports models from other neural network frameworks into nncase;
IR: Intermediate Representation, divided into Neutral IR imported by the importer (device-independent) and Target IR generated by lowering the Neutral IR (device-dependent);
Evaluator: The Evaluator provides the interpretation and execution capability of the IR, commonly used in scenarios such as Constant Folding/PTQ Calibration;
Transform: Used for IR transformation and graph traversal optimization;
Quantize: Post-training quantization. Adds quantization flags to the tensors to be quantized. Based on the input calibration set, it calls the Evaluator for interpretation and execution, collects the data ranges of tensors, inserts quantization/dequantization nodes, and finally optimizes to eliminate unnecessary quantization/dequantization nodes;
Tiling: Limited by the relatively small memory capacity of the NPU, large blocks of computation need to be split. In addition, when computation involves a large amount of data reuse, the choice of Tiling parameters will affect latency and bandwidth;
Partition: Divides the graph by ModuleType. Each subgraph after partitioning corresponds to a RuntimeModule, and different types of RuntimeModules correspond to different Devices (CPU/K230);
Schedule: Generates the computation order and allocates Buffers according to the data dependency relationships in the optimized graph;
Codegen: Calls the codegen corresponding to the ModuleType for each subgraph to generate the RuntimeModule;
🏷️ Runtime: Integrated into the user application (App), providing functions such as model loading, input setting, inference execution, and output reading. The Runtime interface shields the underlying hardware differences, allowing developers to focus more on the integration of model inference logic and application development.
The Model Conversion section mainly introduces the usage of the nncase compiler and simulator.
Installing the nncase environment#
Setting up nncase in a Linux environment
First, please install .NET SDK 7.0 and configure the DOTNET_ROOT environment variable. Please note that it is not recommended to install dotnet in an Anaconda virtual environment, as this may cause compatibility issues.
sudo apt-get update
sudo apt-get install dotnet-sdk-7.0
export DOTNET_ROOT=/usr/share/dotnet
Next, install nncase and nncase-kpu via pip:
pip install nncase nncase-kpu
Setting up nncase in a Windows environment
First, install .NET SDK 7.0. Please follow the Microsoft official documentation to complete the installation process.
Install the nncase library. You can install the main program nncase online via pip, and download the corresponding version of nncase_kpu from the GitHub Releases page, then install it offline via pip.
pip install nncase
# Please replace `2.x.x` with the actual downloaded version number.
pip install nncase_kpu-2.x.x-py2.py3-none-win_amd64.whl
Using Docker to set up the environment
If you have not configured a local Ubuntu environment, you can directly use the official nncase Docker image. This image is based on Ubuntu 20.04 and comes pre-installed with Python 3.8 and dotnet-sdk-7.0, making it easy to get started quickly.
cd /path/to/nncase_sdk
docker pull ghcr.io/kendryte/k230_sdk
docker run -it --rm -v `pwd`:/mnt -w /mnt ghcr.io/kendryte/k230_sdk /bin/bash
Checking nncase version information
After entering the Python interactive environment, you can use the following command to confirm the currently installed nncase version:
>>> import _nncase
>>> print(_nncase.__version__)
2.11.0
The example output is
2.11.0. Please refer to the actual installed version.
Converting to kmodel using the nncase compiler#
The process of compiling a kmodel mainly includes the following key steps, each with its specific purpose and technical considerations:
Setting compilation options: The core purpose of this step is to adapt the model deployment to the target hardware platform. Since edge computing devices need to explicitly specify the runtime platform to ensure whether the generated kmodel needs to leverage hardware (kpu) acceleration. Meanwhile, configuring preprocessing parameters (such as input normalization parameters) into the kmodel internals can reduce computational overhead during inference and improve overall efficiency.
Initializing the compiler: The initialization of the nncase compiler builds a standardized environment for subsequent conversion work. The compiler completes the initialization process based on the previously configured compilation options.
Importing the original model: The ONNX/TFLite models generated by current mainstream training frameworks (such as TensorFlow/PyTorch) contain generic operators, but the KPU, as a dedicated accelerator, requires specific operator formats. This step transforms the original model into an optimizable intermediate representation through model parsing and operator conversion, laying the foundation for subsequent hardware-related optimizations.
Quantization processing: This is a key step to enhance edge-side inference performance. The FP32 models we trained have high accuracy but suffer from high computational latency and large memory footprint. Through quantization to INT8/INT16: significantly reduces model size, improves computation speed (utilizing hardware fixed-point acceleration instructions), and reduces power consumption (decreasing memory bandwidth requirements). It should be noted that quantization introduces precision loss, so a calibration dataset is needed to help the model determine the range to which the weights and activation values of each layer should be mapped during the quantization process, in order to retain more information and reduce quantization error. The quantization process requires configuring quantization parameters and calibration data. See Compilation Parameter Description for the quantization parameters.
Compiling to generate kmodel: Based on the aforementioned optimizations, the finally generated kmodel is deeply optimized and can be directly deployed to K230 devices for efficient inference.
Conversion Example#
Let’s take the four-class printed digit recognition scenario as an example to convert the ONNX model obtained above into Kmodel. Here is the compilation example script:
# 导入所需库
import os
import argparse
import numpy as np
from PIL import Image # 用于图像读取和处理
import onnxsim # ONNX 模型简化工具
import onnx # ONNX 模型处理工具
import nncase # nncase 编译器 SDK
import shutil
import math
def parse_model_input_output(model_file, input_shape):
# 加载ONNX模型
onnx_model = onnx.load(model_file)
# 获取模型中所有输入节点名称
input_all = [node.name for node in onnx_model.graph.input]
# 获取模型中已经被初始化的参数（如权重等），这些不属于输入数据
input_initializer = [node.name for node in onnx_model.graph.initializer]
# 真实输入 = 所有输入 - 初始化器
input_names = list(set(input_all) - set(input_initializer))
# 从图中提取真实输入张量
input_tensors = [node for node in onnx_model.graph.input if node.name in input_names]
# 提取输入张量的名称、数据类型、形状等信息
inputs = []
for _, e in enumerate(input_tensors):
onnx_type = e.type.tensor_type
input_dict = {}
input_dict['name'] = e.name
# 转换为NumPy数据类型
input_dict['dtype'] = onnx.mapping.TENSOR_TYPE_TO_NP_TYPE[onnx_type.elem_type]
# 如果某维为0，说明ONNX模型未固定shape，使用传入的input_shape代替
input_dict['shape'] = [(i.dim_value if i.dim_value != 0 else d) for i, d in zip(onnx_type.shape.dim, input_shape)]
inputs.append(input_dict)
return onnx_model, inputs
def onnx_simplify(model_file, dump_dir, input_shape):
# 获取模型和输入形状信息
onnx_model, inputs = parse_model_input_output(model_file, input_shape)
# 自动推断缺失的shape信息
onnx_model = onnx.shape_inference.infer_shapes(onnx_model)
# 构造用于onnxsim的输入shape映射
input_shapes = {input['name']: input['shape'] for input in inputs}
# 简化模型
onnx_model, check = onnxsim.simplify(onnx_model, input_shapes=input_shapes)
assert check, "模型简化校验失败"
# 保存简化后的模型
model_file = os.path.join(dump_dir, 'simplified.onnx')
onnx.save_model(onnx_model, model_file)
return model_file
def read_model_file(model_file):
with open(model_file, 'rb') as f:
model_content = f.read()
return model_content
def generate_data(shape, batch, calib_dir):
# 获取数据集中的所有图片路径
img_paths = [os.path.join(calib_dir, p) for p in os.listdir(calib_dir)]
data = []
for i in range(batch):
assert i < len(img_paths), "校准图片数量不足"
# 加载图片，转换为RGB格式
img_data = Image.open(img_paths[i]).convert('RGB')
# 按模型输入尺寸进行缩放
img_data = img_data.resize((shape[3], shape[2]), Image.BILINEAR)
# 转换为NumPy数组
img_data = np.asarray(img_data, dtype=np.uint8)
# 转换为 NCHW 格式
img_data = np.transpose(img_data, (2, 0, 1))
# 增加batch维度
data.append([img_data[np.newaxis, ...]])
return np.array(data)
def main():
# 命令行参数定义
parser = argparse.ArgumentParser(prog="nncase")
parser.add_argument("--target", default="k230", type=str, help='编译目标，例如k230或cpu')
parser.add_argument("--model", type=str, help='输入ONNX模型路径')
parser.add_argument("--dataset_path", type=str, help='PTQ校准数据集路径')
parser.add_argument("--input_width", type=int, default=320, help='模型输入宽度')
parser.add_argument("--input_height", type=int, default=320, help='模型输入高度')
parser.add_argument("--ptq_option", type=int, default=0, help='PTQ选项：0-5')
args = parser.parse_args()
# 输入尺寸向上对齐到32的整数倍，符合硬件要求
input_width = int(math.ceil(args.input_width / 32.0)) * 32
input_height = int(math.ceil(args.input_height / 32.0)) * 32
input_shape = [1, 3, input_height, input_width] # NCHW格式
# 创建临时目录保存中间模型
dump_dir = 'tmp'
if not os.path.exists(dump_dir):
os.makedirs(dump_dir)
# 简化模型
model_file = onnx_simplify(args.model, dump_dir, input_shape)
# 编译选项设置
compile_options = nncase.CompileOptions()
compile_options.target = args.target # 指定目标平台
compile_options.preprocess = True # 启用预处理
compile_options.swapRB = False # 不交换RB通道
compile_options.input_shape = input_shape # 设置输入形状
compile_options.input_type = 'uint8' # 输入图像数据类型
compile_options.input_range = [0, 1] # 输入图像反量化范围
compile_options.mean = [0, 0, 0] # 预处理均值
compile_options.std = [1, 1, 1] # 标准差设为1，不进行归一化
compile_options.input_layout = "NCHW" # 输入数据格式
# 初始化编译器
compiler = nncase.Compiler(compile_options)
# 导入ONNX模型为IR
model_content = read_model_file(model_file)
import_options = nncase.ImportOptions()
compiler.import_onnx(model_content, import_options)
# PTQ选项设置（后训练量化）
ptq_options = nncase.PTQTensorOptions()
ptq_options.samples_count = 10 # 校准样本数量
# 支持6种量化方案（根据精度与性能权衡选择）
if args.ptq_option == 0:
ptq_options.calibrate_method = 'NoClip'
ptq_options.quant_type = 'uint8'
ptq_options.w_quant_type = 'uint8'
elif args.ptq_option == 1:
ptq_options.calibrate_method = 'NoClip'
ptq_options.quant_type = 'uint8'
ptq_options.w_quant_type = 'int16'
elif args.ptq_option == 2:
ptq_options.calibrate_method = 'NoClip'
ptq_options.quant_type = 'int16'
ptq_options.w_quant_type = 'uint8'
elif args.ptq_option == 3:
ptq_options.calibrate_method = 'Kld'
ptq_options.quant_type = 'uint8'
ptq_options.w_quant_type = 'uint8'
elif args.ptq_option == 4:
ptq_options.calibrate_method = 'Kld'
ptq_options.quant_type = 'uint8'
ptq_options.w_quant_type = 'int16'
elif args.ptq_option == 5:
ptq_options.calibrate_method = 'Kld'
ptq_options.quant_type = 'int16'
ptq_options.w_quant_type = 'uint8'
# 设置PTQ校准数据
ptq_options.set_tensor_data(generate_data(input_shape, ptq_options.samples_count, args.dataset_path))
# 应用PTQ
compiler.use_ptq(ptq_options)
# 编译模型
compiler.compile()
# 导出KModel文件
base, ext = os.path.splitext(args.model)
kmodel_name = base + ".kmodel"
with open(kmodel_name, 'wb') as f:
f.write(compiler.gencode_tobytes())
# Python程序主入口
if __name__ == '__main__':
main()
Save the above code as the to_kmodel.py script and use the following conversion command to complete the compilation:
# 你需要将onnx模型换成你训练好的模型
python to_kmodel.py --target k230 --model best.onnx --dataset_path test --input_width 320 --input_height 320 --ptq_option 0
Through the above code, we have successfully obtained the Kmodel model used for recognizing four classes of digits. You may be curious: during the process of converting the model into Kmodel, what exactly do those parameters mean? If I want to convert a different model later, do I need to change the parameters as well? Don’t worry, the following sections will walk you through the specific meanings of these conversion parameters, and teach you how to correctly configure them when converting other models, guiding you step by step so you won’t get lost!
Compilation Parameter Description#
To convert tflite/onnx models into kmodel using nncase compiler, the key to the model conversion code lies in configuring the options according to your own needs, mainly including CompileOptions, PTQTensorOptions, and ImportOptions.
For the nncase user guide documentation, see: github: user_guide or gitee: user_guide.
Compilation Options CompileOptions
The CompileOptions class is used to configure nncase compilation options. Each attribute is described as follows:
Attribute Name |
Type |
Required |
Description |
|---|---|---|---|
target |
string |
Yes |
Specifies the compilation target, such as ‘cpu’, ‘k230’ |
dump_ir |
bool |
No |
Specifies whether to dump IR, default is False |
dump_asm |
bool |
No |
Specifies whether to dump asm assembly files, default is False |
dump_dir |
string |
No |
After enabling dump_ir and other switches above, specifies the dump directory here, default is “” |
input_file |
string |
No |
Used to specify the parameter file path when the ONNX model exceeds 2GB, default is “” |
preprocess |
bool |
No |
Whether to enable preprocessing, default is False. The following parameters only take effect when |
input_type |
string |
No |
Specifies the input data type when preprocessing is enabled, default is “float”. When |
input_shape |
list[int] |
No |
Specifies the shape of input data when preprocessing is enabled, default is []. Must be specified when |
input_range |
list[float] |
No |
Specifies the float range after dequantization of input data when preprocessing is enabled, default is [ ]. Must be specified when |
input_layout |
string |
No |
Specifies the layout of input data, default is “” |
swapRB |
bool |
No |
Whether to reverse the data in the |
mean |
list[float] |
No |
Preprocessing normalization parameter mean, default is [0,0,0] |
std |
list[float] |
No |
Preprocessing normalization parameter variance, default is [1,1,1] |
letterbox_value |
float |
No |
Specifies the padding value for preprocessing letterbox, default is 0 |
output_layout |
string |
No |
Specifies the layout of output data, default is “” |
shape_bucket_enable |
bool |
Yes |
Whether to enable the ShapeBucket function, default is False. Takes effect when |
shape_bucket_range_info |
Dict[str, [int, int]] |
Yes |
The range of variables in each input shape dimension information, the minimum value must be greater than or equal to 1 |
shape_bucket_segments_count |
int |
Yes |
How many segments the range of input variables is divided into |
shape_bucket_fix_var_map |
Dict[str, int] |
No |
Fixes the variables in shape dimension information to specific values |
For the configuration description of preprocessing, please refer to the API documentation: nncase Model Compilation API Manual Preprocessing Flow. Encapsulating part of the preprocessing operations within the model can improve the preprocessing efficiency during inference on the development board. The supported preprocessing includes: swapRB(RGB->BGR or BGR->RGB), Transpose(NHWC->NCHW or NCHW->NHWC), Normalization(subtract mean and divide by variance), Dequantize, etc. For example: the onnx model requires RGB input, but the images we read using opencv are in BGR. Normally, for onnx model inference preprocessing, we need to convert BGR to RGB before feeding it to the onnx model. When converting to kmodel, we can set swapRB to True, so that the kmodel itself contains the preprocessing step for swapping the RB channels. When performing preprocessing for kmodel inference, we can skip the step of swapping the RB channels and place this step inside the kmodel.
Import Options ImportOptions
The ImportOptions class is used to configure nncase import options and configure the model to be converted by the compiler. You can configure tflite/onnx. The usage example is as follows:
# 读取并导入tflite模型
model_content = read_model_file(model)
compiler.import_tflite(model_content, import_options)
# 读取并导入onnx模型
model_content = read_model_file(model)
compiler.import_onnx(model_content, import_options)
Post-Training Quantization Options PTQTensorOptions
The PTQTensorOptions class is used to configure nncase PTQ options:
Name |
Type |
Required |
Description |
|---|---|---|---|
samples_count |
int |
No |
Specifies the number of calibration sets used for quantization |
calibrate_method |
string |
No |
Specifies the quantization method, optional ‘NoClip’, ‘Kld’, default is ‘Kld’ |
finetune_weights_method |
string |
No |
Specifies whether to fine-tune the weights, optional ‘NoFineTuneWeights’, ‘UseSquant’, default is ‘NoFineTuneWeights’ |
quant_type |
string |
No |
Specifies the data quantization type, optional ‘uint8’, ‘int8’, ‘int16’. |
w_quant_type |
string |
No |
Specifies the weight quantization type, optional ‘uint8’, ‘int8’, ‘int16’. |
quant_scheme |
string |
No |
The path to the imported quantization parameter configuration file |
quant_scheme_strict_mode |
bool |
No |
Whether to strictly execute quantization according to quant_scheme |
export_quant_scheme |
bool |
No |
Whether to export the quantization parameter configuration file |
export_weight_range_by_channel |
bool |
No |
Whether to export the weights quantization parameters in |
For the specific workflow of mixed quantization, see MixQuant Description.
For the configuration description of quantization, please refer to the table above. If the converted kmodel does not achieve the desired effect, you can modify the quant_type and w_quant_type parameters to change the quantization types for model data and weights, but these two parameters cannot both be set to int16.
Quantization Calibration Set Settings
Name |
Type |
Description |
|---|---|---|
data |
List[List[np.ndarray]] |
The read calibration data |
The calibration data used during the quantization process is set via the set_tensor_data method, and the interface parameter type is List[List[np.ndarray]]. For example: if the model has one input and the calibration data amount is set to 10, the dimension of the input calibration data is [10,1,3,320,320]; if the model has two inputs and the calibration data amount is set to 10, the dimension of the input calibration data is [[10,1,3,224,224],[10,1,3,320,320]].
Using the nncase Simulator to Verify Conversion Results#
Earlier we discussed how to convert a model into a Kmodel. Now let’s give this model a “checkup” to see how well the conversion went!
Because ONNX and Kmodel may differ in preprocessing, we need to prepare input data according to each model’s own requirements. Then, we run inference using the ONNX model and the Kmodel respectively, save both outputs, and compute the Cosine similarity between them—this is like comparing how “similar” their outputs are.
In a nutshell: we want to see whether the converted Kmodel produces output close to the original ONNX model. If the difference is too large, there may be a problem in the conversion process, and you should go back and review the parameter settings～
After the model conversion succeeds, you can use nncase.Simulator to load the Kmodel and run inference on a local PC, and determine whether the Kmodel output is correct by calculating the cosine similarity between the onnx model and the kmodel model. Note that this process runs on your local computer, not on the k230 development board.
First, install the onnx-related packages in the Python environment:
pip install onnx==1.15.0
pip install onnxruntime==1.19.0
pip install onnxsim==0.4.36
To execute the simulator inference script, you need to add the nncase plugin environment variable:
linux:
# The paths in the commands below refer to the Python environment where nncase is installed. Please adapt them to your environment.
export NNCASE_PLUGIN_PATH=$NNCASE_PLUGIN_PATH:/usr/local/lib/python3.9/site-packages/
export PATH=$PATH:/usr/local/lib/python3.9/site-packages/
source /etc/profile
windows:
Add the Lib/site-packages path under the Python environment where nncase is installed to the system environment variable Path.
For the 4-class digit recognition scenario, here’s the example code to verify output similarity:
import os
import cv2
import numpy as np
import onnxruntime as ort
import nncase
import math
def get_onnx_input(img_path,mean,std,model_input_size):
# 读取图片，图片数据一般是RGB三通道，颜色范围为[0, 255.0]
image_fp32=cv2.imread(img_path)
# 如果模型输入要求是RGB的，则转换为RGB格式，如果要求是BGR的，则不需要转换
image_fp32=cv2.cvtColor(image_fp32, cv2.COLOR_BGR2RGB)
# 缩放成模型输入大小
image_fp32 = cv2.resize(image_fp32, (model_input_size[0], model_input_size[1]))
# 数据类型为float32,
image_fp32 = np.asarray(image_fp32, dtype=np.float32)
# 数据标准化,先归一化到[0,1]范围内，然后减均值除方差
image_fp32/=255.0
for i in range(3):
image_fp32[:, :, i] -= mean[i]
image_fp32[:, :, i] /= std[i]
# 按照模型输入要求处理成NCHW排布或者NHWC排布
image_fp32 = np.transpose(image_fp32, (2, 0, 1))
return image_fp32.copy()
def get_kmodel_input(img_path,mean,std,model_input_size):
# 读取图片，图片数据一般是RGB三通道，颜色范围为[0, 255.0]
image_uint8=cv2.imread(img_path)
# 如果模型输入要求是RGB的，则转换为RGB格式，如果要求是BGR的，则不需要转换
image_uint8=cv2.cvtColor(image_uint8, cv2.COLOR_BGR2RGB)
# 缩放成模型输入大小
image_uint8 = cv2.resize(image_uint8, (model_input_size[0], model_input_size[1]))
# 数据类型为uint8,因为转换kmodel的时候开启了预处理，并且设定了标准化参数，因此这里的输入就不需要实现标准化了
image_uint8 = np.asarray(image_uint8, dtype=np.uint8)
# 按照模型输入要求处理成NCHW排布或者NHWC排布
image_uint8 = np.transpose(image_uint8, (2, 0, 1))
return image_uint8.copy()
def onnx_inference(onnx_path,onnx_input_data):
# 创建 ONNX 推理会话（加载模型）
ort_session = ort.InferenceSession(onnx_path)
# 获取模型输出名称列表，用于后续调用推理
output_names = []
model_outputs = ort_session.get_outputs()
for i in range(len(model_outputs)):
output_names.append(model_outputs[i].name)
# 获取模型的输入信息
model_input = ort_session.get_inputs()[0] # 第一个输入（通常只有一个）
model_input_name = model_input.name # 输入的名称（键）
model_input_type = np.float32 # 输入数据类型，这里假设是 float32
model_input_shape = model_input.shape # 输入张量的形状（维度）
# 处理输入数据,需确保和模型输入形状一致
model_input_data = onnx_input_data.astype(model_input_type).reshape(model_input_shape)
# 执行推理，传入输入名称和数据，返回所有输出结果
onnx_results = ort_session.run(output_names, { model_input_name : model_input_data })
return onnx_results
def kmodel_inference(kmodel_path,kmodel_input_data,model_input_size):
# 初始化nncase 模拟器
sim = nncase.Simulator()
# 读取kmodel
with open(kmodel_path, 'rb') as f:
kmodel = f.read()
# 加载kmodel
sim.load_model(kmodel)
# 读取输入数据
input_shape = [1, 3, model_input_size[1], model_input_size[0]]
dtype = sim.get_input_desc(0).dtype
# 处理输入数据,需确保和模型输入形状一致
kmodel_input = kmodel_input_data.astype(dtype).reshape(input_shape)
# 设置模拟器输入tensor,此处为单输入
sim.set_input_tensor(0, nncase.RuntimeTensor.from_numpy(kmodel_input))
# 模拟器推理kmodel模型
sim.run()
# 获取推理输出
kmodel_results = []
for i in range(sim.outputs_size):
kmodel_result = sim.get_output_tensor(i).to_numpy() # 转换为numpy数组
kmodel_results.append(kmodel_result) # 保存到列表中
return kmodel_results
def cosine_similarity(onnx_results,kmodel_results):
output_size=len(kmodel_results)
# 将每个输出展成一维，然后计算余弦相似度
for i in range(output_size):
onnx_i=np.reshape(onnx_results[i], (-1))
kmodel_i=np.reshape(kmodel_results[i], (-1))
cos = (onnx_i @ kmodel_i) / (np.linalg.norm(onnx_i, 2) * np.linalg.norm(kmodel_i, 2))
print('output {0} cosine similarity : {1}'.format(i, cos))
return
if __name__ == '__main__':
img_path="test.jpg"
mean=[0,0,0]
std=[1,1,1]
model_input_size=[320,320]
# ONNX 模型文件
onnx_model = "best.onnx"
# kmodel 模型文件
kmodel_path="best.kmodel"
# 生成onnx模型输入数据
onnx_input_data = get_onnx_input(img_path,mean,std,model_input_size)
# 生成kmodel模型输入数据
kmodel_input_data = get_kmodel_input(img_path,mean,std,model_input_size)
# onnx模型推理
onnx_results = onnx_inference(onnx_model,onnx_input_data)
# kmodel模型推理
nncase_results = kmodel_inference(kmodel_path,kmodel_input_data,model_input_size)
# 计算输出相似度
cosine_similarity(onnx_results,nncase_results)
Save the above code as a file, replace the models inside the code with your own converted models, and run the script to get the following output:
output 0 cosine similarity : 0.9995334148406982
Generally, when the similarity is greater than 0.99, the model conversion is considered successful and is usable in actual deployment scenarios.
Generating Input Data#
⚠️ Note: When performing inference with the ONNX model and the KModel, the preprocessing steps for the input data must be handled with care. If specific preprocessing operations have been encapsulated inside the KModel, there is no need to manually apply these preprocessing steps to its input data before inference; however, when using the ONNX model for inference, all necessary preprocessing must be explicitly performed outside the model.
The preprocessing operations that KModel supports and can encapsulate include:
Channel order transformation (e.g., RGB ↔ BGR), corresponding to the
SwapRBparameter;Layout conversion (NCHW ↔ NHWC), corresponding to the
input_shapeandinput_layoutparameters;Data standardization, which depends on the
meanandstdparameters;Input dequantization, which depends on the
input_typeandinput_rangeparameters;
For the differences between the ONNX and KModel inference workflows, refer to the following diagram:
When using the ONNX model for inference, since it contains no preprocessing logic itself, the user must complete all required preprocessing steps before feeding the input.
For the KModel, if the preprocess option is enabled during model compilation, the related preprocessing operations will be automatically encapsulated inside the model, and users no longer need to handle them manually during inference.
If preprocess is not enabled, its usage is the same as the ONNX model—preprocessing must still be completed outside the model.
According to the above workflow, developers can construct inference data that conforms to the input specifications as required by the model, so that it can be used during inference. Please note: the data generation process must strictly conform to the model requirements. Different models may have significantly different input processing workflows and cannot be mixed.
The following is example code for data preprocessing:
def get_onnx_input(img_path,mean,std,model_input_size):
# 读取图片，图片数据一般是RGB三通道，颜色范围为[0, 255.0]
image_fp32=cv2.imread(img_path)
# 如果模型输入要求是RGB的，则转换为RGB格式，如果要求是BGR的，则不需要转换
image_fp32=cv2.cvtColor(image_fp32, cv2.COLOR_BGR2RGB)
# 缩放成模型输入大小
image_fp32 = cv2.resize(image_fp32, (model_input_size[0], model_input_size[1]))
# 数据类型为float32,
image_fp32 = np.asarray(image_fp32, dtype=np.float32)
# 数据标准化,先归一化到[0,1]范围内，然后减均值除方差
image_fp32/=255.0
for i in range(3):
image_fp32[:, :, i] -= mean[i]
image_fp32[:, :, i] /= std[i]
# 按照模型输入要求处理成NCHW排布或者NHWC排布
image_fp32 = np.transpose(image_fp32, (2, 0, 1))
return image_fp32.copy()
def get_kmodel_input(img_path,mean,std,model_input_size):
# 读取图片，图片数据一般是RGB三通道，颜色范围为[0, 255.0]
image_uint8=cv2.imread(img_path)
# 如果模型输入要求是RGB的，则转换为RGB格式，如果要求是BGR的，则不需要转换
image_uint8=cv2.cvtColor(image_uint8, cv2.COLOR_BGR2RGB)
# 缩放成模型输入大小
image_uint8 = cv2.resize(image_uint8, (model_input_size[0], model_input_size[1]))
# 数据类型为uint8,因为转换kmodel的时候开启了预处理，并且设定了标准化参数，因此这里的输入就不需要实现标准化了
image_uint8 = np.asarray(image_uint8, dtype=np.uint8)
# 按照模型输入要求处理成NCHW排布或者NHWC排布
image_uint8 = np.transpose(image_uint8, (2, 0, 1))
return image_uint8.copy()
When performing inference with the ONNX model and the KModel, there are several key differences in the preprocessing of input data, mainly reflected in the following aspects:
Standardization: The ONNX model contains no preprocessing logic itself, so its input data must be standardized externally (e.g., subtract the mean and divide by the standard deviation). For the KModel, if normalization parameters (such as
meanandstd) were configured during model conversion, this standardization operation will be encapsulated inside the model, and there is no need to repeat it before inference.Data type difference: The input of the ONNX model is usually of type
float32, whereas the input type of the KModel depends on theinput_type(e.g.,uint8) andinput_rangespecified during model conversion. The KModel will perform internal dequantization during inference, converting the integer type back to an approximate floating-point representation.Channel order handling: If
SwapRBwas not enabled (i.e., the parameter isFalse) during model conversion, the channel order of the input image needs to be converted from BGR to RGB in the external preprocessing stage. IfSwapRB=True, this channel transformation operation will be automatically handled inside the KModel, and does not need to be performed externally.
Overall, the external preprocessing required by the ONNX model equals the KModel’s external preprocessing plus the KModel’s internal preprocessing. The relationship between them can be expressed as follows:
ONNX 模型外部预处理 = KModel 外部预处理 + KModel 内部预处理
Loading the onnx Model and Running Inference#
First, use the onnx model to complete inference and obtain the inference results of the onnx model. The example code is as follows:
def onnx_inference(onnx_path,onnx_input_data):
# 创建 ONNX 推理会话（加载模型）
ort_session = ort.InferenceSession(onnx_path)
# 获取模型输出名称列表，用于后续调用推理
output_names = []
model_outputs = ort_session.get_outputs()
for i in range(len(model_outputs)):
output_names.append(model_outputs[i].name)
# 获取模型的输入信息
model_input = ort_session.get_inputs()[0] # 第一个输入（通常只有一个）
model_input_name = model_input.name # 输入的名称（键）
model_input_type = np.float32 # 输入数据类型，这里假设是 float32
model_input_shape = model_input.shape # 输入张量的形状（维度）
# 处理输入数据,需确保和模型输入形状一致
model_input_data = onnx_input_data.astype(model_input_type).reshape(model_input_shape)
# 执行推理，传入输入名称和数据，返回所有输出结果
onnx_results = ort_session.run(output_names, { model_input_name : model_input_data })
return onnx_results
Loading the kmodel Model and Running Inference#
Then use the successfully converted kmodel to run inference and obtain the inference results of the kmodel. The example code is as follows:
def kmodel_inference(kmodel_path,kmodel_input_data,model_input_size):
# 初始化nncase 模拟器
sim = nncase.Simulator()
# 读取kmodel
with open(kmodel_path, 'rb') as f:
kmodel = f.read()
# 加载kmodel
sim.load_model(kmodel)
# 读取输入数据
input_shape = [1, 3, model_input_size[1], model_input_size[0]]
dtype = sim.get_input_desc(0).dtype
# 处理输入数据,需确保和模型输入形状一致
kmodel_input = kmodel_input_data.astype(dtype).reshape(input_shape)
# 设置模拟器输入tensor,此处为单输入
sim.set_input_tensor(0, nncase.RuntimeTensor.from_numpy(kmodel_input))
# 模拟器推理kmodel模型
sim.run()
# 获取推理输出
kmodel_results = []
for i in range(sim.outputs_size):
kmodel_result = sim.get_output_tensor(i).to_numpy() # 转换为numpy数组
kmodel_results.append(kmodel_result) # 保存到列表中
return kmodel_results
Calculating the Cosine Similarity of the Outputs#
After obtaining the inference results of the onnx model and the kmodel model, calculate the cosine similarity of each output one by one. Generally, when the similarity is above 0.99, the model conversion can be considered successful and ready for deployment. The example code is as follows:
def cosine_similarity(onnx_results,kmodel_results):
output_size=len(kmodel_results)
# 将每个输出展成一维，然后计算余弦相似度
for i in range(output_size):
onnx_i=np.reshape(onnx_results[i], (-1))
kmodel_i=np.reshape(kmodel_results[i], (-1))
cos = (onnx_i @ kmodel_i) / (np.linalg.norm(onnx_i, 2) * np.linalg.norm(kmodel_i, 2))
print('output {0} cosine similarity : {1}'.format(i, cos))
return
Model Deployment#
Note
👉 Previously we converted and verified the kmodel. Next, of course, it’s time to———run it on the board! In this chapter, we’ll talk about how to use the provided nncase runtime API to load the model and perform inference in the K230 Linux environment.
So the question is: we have the model, but how should we prepare the input data? We need to process the input image according to the model’s “taste,” such as size, format, normalization, etc., to make sure it can be “fed correctly.” Then we feed the processed data in and let the model start inference. After inference, the model will give us a bunch of “output results”—what do these results mean? We still need to parse them, such as extracting categories, coordinates, and other useful information.
Finally, of course, we can’t keep it hidden! We’ll display the recognized content on the screen, such as drawing boxes and labeling numbers, so that the entire process—from image acquisition, model inference, to result display—is seamless and runs through the full pipeline!
This chapter will walk you through this complete process and make the model truly “come alive”～
For a practical AI program, it includes not only model inference but also image input, pre/post-processing programs, and result display, among other different modules. The figure below shows a complete block diagram of a typical AI application:
In k230_linux_sdk, the camera and display parts use v4l2 and drm for integration. V4L2 (Video for Linux 2) is the Linux standard interface for video capture, used to manage cameras and video input devices, mainly used to provide input data for AI inference models. DRM (Direct Rendering Manager) is a display/video memory control subsystem in the Linux kernel that manages display output. Its main responsibilities include managing video memory (framebuffer), controlling the display pipeline (CRTC, plane, encoder, connector), providing the KMS (Kernel Mode Setting) interface for image display control, and supporting 2D/3D hardware-accelerated rendering (via GPU drivers).
🚀 Deployment Process Explanation: Deployment can be understood as the process of “making the model actually work.” Below, we introduce it step by step according to the workflow.
1️⃣ Get Image Data (Input Data Source)
First, we need to get an image, usually captured in real-time from a camera, or a test image loaded locally. After getting the image, we can use the image’s data pointer to create a runtime_tensor. On the K230 development board, you typically use the v4l2_drm_dump interface to get a frame of data and obtain data from the virtual address to create the runtime_tensor.
2️⃣ Construct Input Tensor (Prepare Data to Feed the Model)
Once we have the image, we need to “package” it into a format that the model can process: runtime_tensor. This step is to feed the model a standard-structured data.
3️⃣ Pre-processing (ai2d Module) The model has specific requirements for input images, such as size, format, channel order, etc. In this step, we use the ai2d module to process the image tensor into the “shape” required by the model.
4️⃣ Model Inference (Using the KPU Inference Module) After the image is processed, it’s fed into the KPU (K230’s neural network acceleration module) for inference. The KPU will return a result tensor, which contains the model’s output, such as detection boxes, classification probabilities, etc.
5️⃣ Post-processing (Extract Useful Information) The output of the KPU is a bunch of numbers, and we need to parse out these “key points.” For example, what number is recognized? Where is the box located on the image? These all need to be handled by post-processing algorithms. For YOLO models, post-processing includes confidence filtering, NMS (Non-Maximum Suppression), etc.
6️⃣ Display Recognition Results (Visualization) The last step is to “draw” the recognized content! We can draw detection boxes, numeric labels, etc. on the screen, making the results clear at a glance. Generally, two layers are used for display: one is the original image, and the other is the recognition result (such as boxes and numbers). Overlaying them ensures a clearer and more flexible effect.
Summary: The core deployment process is: get image → process into input → feed to model → get results → interpret results → display them! Once this workflow runs through, your model is truly “live and working”! 🎉
💡 Firmware Introduction: Please refer to the steps in the documentation below to set up the build environment and compile the firmware, to ensure that the latest features are supported!
Get Input and Create Tensor#
As we mentioned earlier, once the model is running, it needs input data to start inference, right? So where does this image data come from? In this section, let’s talk about—where the image comes from, and how it’s transformed step by step into a format the model can “eat”!
There are actually two ways to get images: you can use a local image pre-stored on the board (such as a test image you’ve copied in advance), or you can use the on-board MIPI camera to capture real-time footage. Regardless of which method you choose, ultimately we need to get a data pointer in RGB888P format—this is like the “raw material.” If the format of the obtained data does not match, some modules need to be used for conversion.
Note
👉 The difference between RGB888 and RGB888P mainly lies in the arrangement of pixel data in memory.
RGB888 format: The R, G, B components of each pixel are stored consecutively, with data arranged in HWC layout.
Memory layout:
[R0][G0][B0] [R1][G1][B1] [R2][G2][B2] … [Rw][Gw][Bw] …
Each pixel occupies 3 bytes (8 bits R + 8 bits G + 8 bits B = 24 bits).
RGB888P format: All R components are placed together, all G components together, and all B components together, with data arranged in CHW layout.
Memory layout:
[R0][R1][R2]…[Rn] ← All red components
[G0][G1][G2]…[Gn] ← All green components
[B0][B1][B2]…[Bn] ← All blue components
The size of each component’s memory block is width × height bytes.
Therefore, how to create the input tensor needs to be determined based on the input_layout of nncase.CompileOptions when converting the model, or by using netron to view the input layout order of the onnx model.
After getting the image, we can’t feed it directly to the model. We use the API provided by the nncase::runtime module to convert this array into a runtime_tensor (tensor). At this point, the data is “packaged” and can be safely fed into the model for inference!
So what is a tensor? You can think of it as the “language” the model understands—it’s like a box that holds data. The model takes in tensors and outputs tensors after inference. In the nncase::runtime module, this is encapsulated as runtime_tensor. As long as you construct it according to the requirements, you can use it directly.
The figure above illustrates the process of getting an input image and creating a tensor. The model inference input is of type runtime_tensor, which can be created from a data pointer.
Common input data sources include:
Image files
MIPI camera
This section provides a detailed introduction to these two input data sources.
Image File Input#
Read an image from the development board, create a cv::Mat instance, and create a runtime tensor instance from the data pointer of the cv::Mat instance. The example code is as follows. opencv reads in an image; the default cv::Mat instance is in BGR HWC format. First convert it to RGB CHW format, then create an empty tensor, map out the writable memory area on the Host side, copy the data into the tensor, and then synchronize to the device:
#include <opencv2/core.hpp>
#include <opencv2/highgui.hpp>
#include <opencv2/imgcodecs.hpp>
#include <opencv2/imgproc.hpp>
#include <nncase/runtime/interpreter.h>
#include <nncase/runtime/runtime_op_utility.h>
#include "scoped_timing.h"
using namespace std;
using namespace nncase::runtime;
/**
* @brief Single image/frame size
*/
typedef struct FrameCHWSize
{
int channel;
int height; // Height
int width; // Width
} FrameCHWSize;
std::string image_path = "/mnt/test.jpg";
// Read image
cv::Mat ori_img = cv::imread(image_path);
FrameCHWSize image_size={ori_img.channels(),ori_img.rows,ori_img.cols};
// Create an empty vector to store chw image data, convert the read hwc data to chw data
std::vector<uint8_t> chw_vec;
std::vector<cv::Mat> bgrChannels(3);
cv::split(ori_img, bgrChannels);
for (auto i = 2; i > -1; i--)
{
std::vector<uint8_t> data = std::vector<uint8_t>(bgrChannels[i].reshape(1, 1));
chw_vec.insert(chw_vec.end(), data.begin(), data.end());
}
// Create tensor
dims_t in_shape { 1, 3, ori_img.rows, ori_img.cols };
runtime_tensor input_tensor = host_runtime_tensor::create(typecode_t::dt_uint8, in_shape, hrt::pool_shared).expect("cannot create input tensor");
auto input_buf = input_tensor.impl()->to_host().unwrap()->buffer().as_host().unwrap().map(map_access_::map_write).unwrap().buffer();
memcpy(reinterpret_cast<char *>(input_buf.data()), chw_vec.data(), chw_vec.size());
hrt::sync(input_tensor, sync_op_t::sync_write_back, true).expect("write back input failed");
MIPI Video Stream Input#
The k230 vvcam module is responsible for image acquisition and data processing, and supports MIPI interface cameras. The MIPI camera can collect image data through the v4l2 framework interface, and convert the collected image data into runtime_tensor type for kmodel model inference.
Each MIPI camera can output up to 3 image channels (each channel can have different resolutions or different formats). Here we use one channel output as an example. The data processing flow chart is shown below:
The input data during model inference can also come from the MIPI camera’s video stream. To ensure that the output data is in CHW-arranged RGB format, we generally specify the camera output data format as v4l2_fourcc('B', 'G', '3', 'P');. The code for configuring the vvcam device output is as follows. The code below is pseudocode and cannot be actually run:
// Define ISP output resolution and single-channel output resolution
#define SENSOR_WIDTH 640
#define SENSOR_HEIGHT 360
#define SENSOR_CHANNEL 3
#define BUFFER_NUM 3
struct v4l2_drm_context context;
struct v4l2_drm_video_buffer buffer;
v4l2_drm_default_context(&context);
context.device = video_device;
context.display = false;
context.width = SENSOR_WIDTH;
context.height = SENSOR_HEIGHT;
context.video_format = v4l2_fourcc('B', 'G', '3', 'P');
context.buffer_num = BUFFER_NUM;
if (v4l2_drm_setup(&context, 1, NULL)) {
cerr << "v4l2_drm_setup error" << endl;
return;
}
if (v4l2_drm_start(&context)) {
cerr << "v4l2_drm_start error" << endl;
return;
}
After completing the above configuration, call the interface v4l2_drm_dump(&context, 1000) to get a frame of image data from context.buffers[0].mmap. This data is in CHW format RGB data of SENSOR_CHANNEL x SENSOR_HEIGHT x SENSOR_WIDTH.
🏷️ Dual-Channel Image Capture
When running AI model inference on edge devices, since the model computation is large, the inference process is usually time-consuming, ranging from a few milliseconds to several hundred milliseconds. If a single-channel processing flow is used:
Image acquisition → Format conversion → Data pre-processing → Model inference → Result post-processing → Original image drawing → Image display
This serial execution method will cause high image display latency, especially when the model is large or system resources are limited, the screen update becomes obviously slow, affecting the user experience.
To solve this problem, a dual-channel processing architecture is recommended, that is, an asynchronous processing strategy of “one channel for real-time display, the other for model inference”. This architecture effectively reduces display latency and improves picture smoothness by processing image acquisition and model inference in parallel. The dual-channel processing mechanism is as follows:
Display channel: Directly captures images and pushes them to the screen for low-latency real-time picture display. This channel can be implemented using a binding method.
Inference channel: Independently captures images and executes the complete AI inference process (including format conversion, pre-processing, model inference, and post-processing).
OSD layer composition: The model inference results (such as detection boxes, key points, etc.) are drawn as an OSD layer, and are combined with the original image through hardware overlay before being output for display.
Although there will be a certain visual delay in the inference results (i.e., the detection box of the previous frame is displayed on the current frame image), the overall picture continuity is better and the user experience is smoother.
The pseudocode for the dual-channel color image mode is as follows. The code below cannot be run directly and is for illustration only:
#define SENSOR_WIDTH 640
#define SENSOR_HEIGHT 360
#define SENSOR_CHANNEL 3
static std::mutex result_mutex;
std::atomic<bool> ai_stop(false);
// Display thread exit flag
std::atomic<bool> display_stop(false);
static volatile unsigned kpu_frame_count = 0;
static struct timeval tv, tv2;
// Display instance and OSD buffer
static struct display* display;
struct display_buffer* draw_buffer;
cv::Mat draw_frame;
/**
* @brief AI inference processing thread function
**/
static void ai_proc(char *argv[], int video_device) {
struct v4l2_drm_context context;
struct v4l2_drm_video_buffer buffer;
#define BUFFER_NUM 3
/**
* Wait for display_proc to run, ensuring the display drm device is initialized
**/
result_mutex.lock();
result_mutex.unlock();
v4l2_drm_default_context(&context);
context.device = video_device;
context.display = false;
context.width = SENSOR_WIDTH;
context.height = SENSOR_HEIGHT;
context.video_format = v4l2_fourcc('B', 'G', '3', 'P');
context.buffer_num = BUFFER_NUM;
if (v4l2_drm_setup(&context, 1, NULL)) {
cerr << "v4l2_drm_setup error" << endl;
return;
}
if (v4l2_drm_start(&context)) {
cerr << "v4l2_drm_start error" << endl;
return;
}
/**
* Add model initialization, load model, instantiate application class code here
**/
std::vector<std::tuple<int, void*>> tensors;
for (unsigned i = 0; i < BUFFER_NUM; i++) {
tensors.push_back({context.buffers[i].fd, context.buffers[i].mmap});
}
SensorBufManager sensor_buf = SensorBufManager({SENSOR_CHANNEL, SENSOR_HEIGHT, SENSOR_WIDTH},tensors);
while (!ai_stop) {
int ret = v4l2_drm_dump(&context, 1000);
if (ret) {
perror("v4l2_drm_dump error");
continue;
}
// Frame data can be obtained from here to create a tensor
void* frame_data=context.buffers[0].mmap;
/**
* Input pre-processing, model inference, output post-processing, result drawing can be processed in frame_handler, or can be drawn here and then the entire cv::Mat copied to the allocated plane buffer;
**/
v4l2_drm_dump_release(&context);
}
v4l2_drm_stop(&context);
}
/**
* @brief V4L2-DRM display frame processing function (triggered once per frame)
*
* This function is called back by the v4l2_drm_run driver loop and is called when each frame of display data is processed. Its main function is to display the AI inference results.
* @param context V4L2-DRM context structure pointer
* @param displayed Indicates whether the frame has actually been displayed
* @return Returns 0 for normal, returns 'q' to request exit from the main loop (controlled by display_stop flag)
*/
int frame_handler(struct v4l2_drm_context *context, bool displayed)
{
static bool first_frame = true;
if (first_frame) {
result_mutex.unlock();
first_frame = false;
}
static unsigned response = 0, display_frame_count = 0;
response += 1;
if (displayed)
{
if (context[0].buffer_hold[context[0].wp] >= 0)
{
static struct display_buffer* last_drawed_buffer = nullptr;
auto buffer = context[0].display_buffers[context[0].buffer_hold[context[0].wp]];
if (buffer != last_drawed_buffer) {
//---------------------- Draw display results ----------------------
if (draw_buffer->width > draw_buffer->height)
{
// Create temporary BGRA display buffer Mat (for drawing)
cv::Mat temp_img(draw_buffer->height, draw_buffer->width, CV_8UC4);
// Landscape
temp_img.setTo(cv::Scalar(0, 0, 0, 0));
result_mutex.lock();
draw_frame.copyTo(temp_img);
result_mutex.unlock();
//---------------------- Display buffer synchronization ----------------------
// Copy drawing image to actual display buffer
memcpy(draw_buffer->map, temp_img.data, draw_buffer->size);
}
else
{
// Portrait
// Create temporary BGRA display buffer Mat (for drawing)
cv::Mat temp_img(draw_buffer->width, draw_buffer->height, CV_8UC4);
// Portrait st7701: draw landscape image, then rotate back to portrait for display
temp_img.setTo(cv::Scalar(0, 0, 0, 0));
result_mutex.lock();
draw_frame.copyTo(temp_img);
result_mutex.unlock();
// Rotate back to screen orientation
cv::rotate(temp_img, temp_img, cv::ROTATE_90_CLOCKWISE);
//---------------------- Display buffer synchronization ----------------------
// Copy drawing image to actual display buffer
memcpy(draw_buffer->map, temp_img.data, draw_buffer->size);
}
last_drawed_buffer = buffer;
// flush cache
thead_csi_dcache_clean_invalid_range(draw_buffer->map, draw_buffer->size);
display_update_buffer(draw_buffer, 0, 0);
}
}
display_frame_count += 1;
}
// FPS counter
gettimeofday(&tv2, NULL);
uint64_t duration = 1000000 * (tv2.tv_sec - tv.tv_sec) + tv2.tv_usec - tv.tv_usec;
if (duration >= 1000000) {
fprintf(stderr, " poll: %.2f, ", response * 1000000. / duration);
response = 0;
if (display) {
fprintf(stderr, "display: %.2f, ", display_frame_count * 1000000. / duration);
display_frame_count = 0;
}
fprintf(stderr, "camera: %.2f, ", context[0].frame_count * 1000000. / duration);
context[0].frame_count = 0;
fprintf(stderr, "KPU: %.2f", kpu_frame_count * 1000000. / duration);
kpu_frame_count = 0;
fprintf(stderr, " \r");
fflush(stderr);
gettimeofday(&tv, NULL);
}
// If an exit signal is received, return 'q' to indicate the main loop exits
if (display_stop) {
return 'q';
}
return 0;
}
/**
* @brief Display thread function, initializes V4L2-DRM and binds the drawing callback
*
* Configure the corresponding width, height, format, and rotation angle according to the screen orientation (landscape / portrait),
* then call `v4l2_drm_run()` to start the frame processing main loop, which triggers drawing via `frame_handler()` for each frame.
*
* @param video_device Video device number (such as 1 in /dev/video0)
*/
void display_proc(int video_device)
{
struct v4l2_drm_context context;
v4l2_drm_default_context(&context);
context.device = video_device;
// Set width/height/rotation according to screen orientation
if (display->width > display->height)
{
// Landscape
context.width = display->width;
context.height = (display->width * SENSOR_HEIGHT / SENSOR_WIDTH) & 0xfff8;
context.video_format = V4L2_PIX_FMT_NV12;
context.display_format = 0;
context.drm_rotation = rotation_0;
}
else
{
// Portrait
context.width = display->height;
context.height = display->width;
context.video_format = V4L2_PIX_FMT_NV12;
context.display_format = 0;
context.drm_rotation = rotation_90;
}
if (v4l2_drm_setup(&context, 1, &display)) {
std::cerr << "v4l2_drm_setup error" << std::endl;
return;
}
// Allocate OSD display plane and buffer
struct display_plane* plane = display_get_plane(display, DRM_FORMAT_ARGB8888);
draw_buffer = display_allocate_buffer(plane, display->width, display->height);
display_commit_buffer(draw_buffer, 0, 0);
/**
* Initialize display Mat for drawing inference results
**/
if (draw_buffer->width > draw_buffer->height)
{
draw_frame = cv::Mat(draw_buffer->height, draw_buffer->width, CV_8UC4, cv::Scalar(0,0, 0, 0));
}
else{
draw_frame = cv::Mat(draw_buffer->width, draw_buffer->height, CV_8UC4, cv::Scalar(0,0, 0, 0));
}
/**
* Start V4L2-DRM display thread, bind frame processing callback function
**/
std::cout << "press 'q' to exit" << std::endl;
gettimeofday(&tv, NULL);
v4l2_drm_run(&context, 1, frame_handler);
// Clean up resources
if (display) {
display_free_plane(plane);
display_exit(display);
}
return;
}
Image Tensor Preprocessing#
We have successfully obtained the image data and can create a tensor, but here’s the problem—this tensor may not match the model’s “appetite.” For example, the size may be wrong, or the color channels may not match. At this point, we need to step in and process the tensor to make it into a format the model can accept. This entire processing process is called “preprocessing,” and the one that accomplishes this task is today’s protagonist—the ai2d module!
🛠️ Why do we need preprocessing? Models are “picky eaters”—they only accept data of specific sizes and formats, such as: the input must be 320x320 in size; it must be in RGB order, not BGR; and the channel order (CHW vs. HWC) must also match. If these don’t match, recognition will fail, or the model may simply refuse to work and throw an error.
⚡ ai2d Module: Hardware-Accelerated, Lightning-Fast Processing! ai2d is a module on the K230 platform specifically designed for image tensor preprocessing. It runs on hardware, is very fast, and is suitable for embedded real-time tasks. It can help you complete: resizing, cropping, padding, affine transformations, and other operations, transforming image data into tensor data that meets the model’s input requirements.
The figure below shows the input/output flow and format for preprocessing via the ai2d module on the K230 platform:
Introduction to the Preprocessing Process#
When deploying a model, the input image’s runtime_tensor does not necessarily conform to the model’s input specifications. For example, the image captured by the camera may be 1280×720, while the model’s input requires 320×320. In this case, the image needs to be preprocessed.
Preprocessing operations include but are not limited to the following common methods:
Resize: Adjust the original image to the size required by the model’s input;
Crop: Retain the key region of the image and remove redundant parts;
Normalization: Map pixel values to a specified range (such as
[0, 1]or[-1, 1]);Padding: Perform edge padding to maintain the image’s aspect ratio and avoid stretching distortion.
The specific preprocessing methods to be adopted should be configured according to the ONNX model’s training preprocessing pipeline. Meanwhile, during the process of converting the ONNX model to KModel, some preprocessing steps (such as normalization, color channel conversion, etc.) can be encapsulated into the model through compiler parameters. These operations do not need to be reimplemented during deployment and are automatically completed by KModel.
⚠️ Note: A clear understanding of the preprocessing pipeline is required, especially when performing image Aspect Ratio Padding, where users can choose different strategies:
Bilateral Padding: Pad on both the top/bottom and left/right sides of the image to center it;
Unilateral Padding: Pad only on one side of the image (such as top/left or bottom/right) to keep one edge aligned.
Different padding methods will affect the coordinate restoration logic of the model inference output. Therefore, in the post-processing stage, it is necessary to match the corresponding coordinate transformation rules to ensure that the results are correctly mapped back to the original image.
Introduction to the ai2d Module#
In Linux solutions, common image preprocessing operations are typically hardware-accelerated through the nncase ai2d_builder module. This module supports five main preprocessing methods, including:
Resize
Crop
Pad
Affine
Shift
Using the ai2d module can effectively reduce the CPU computation burden and improve preprocessing efficiency, making it suitable for image adaptation operations before model inference. Below are some precautions when using AI2D:
Attention
(1) Affine and Resize are mutually exclusive: They cannot be enabled simultaneously; only one can be selected for geometric transformation.
(2) Shift only supports Raw16 input format, used for high-bit shift operations of specific formats.
(3) Pad Value is configured per channel: A list matching the number of channels of the input image should be provided. For example, an RGB image requires padding values for three channels.
(4) The execution order of functions is Crop → Shift → Resize/Affine → Pad: This order must be followed when configuring multiple preprocessing steps. If the preprocessing pipeline does not conform to this order, it is recommended to initialize multiple ai2d instances and complete the required processing step by step.
By properly configuring the ai2d module, efficient and flexible image preprocessing can be achieved to meet the input data requirements of different models.
Here, taking the aspect-ratio resize-and-pad preprocessing used in the digit recognition task as an example, the usage of the ai2d module is introduced. The core code (this code is for illustration only and cannot be run directly) is as follows:
int input_w=1280;
int input_h=720;
int output_w=320;
int output_h=320;
// -------These two tensors are created by yourself------
runtime_tensor input_tensor;
runtime_tensor output_tensor;
// -------------------------------
// Calculate padding parameters. Here, unilateral padding is chosen, padding only on the right and bottom sides. First, calculate the scaling ratio based on the short side, then calculate the padding pixel width on both sides
float ratiow = (float)output_w / input_w;
float ratioh = (float)output_h / input_h;
float ratio = ratiow < ratioh ? ratiow : ratioh;
int new_w = (int)(ratio * input_w);
int new_h = (int)(ratio * input_h);
int top = 0;
int bottom = output_h - new_h;
int left = 0;
int right = output_w - new_w;
// Set ai2d parameters, including input format, output format, input data type, output data type, crop parameters, shift parameters, pad parameters, resize parameters, affine parameters. Set the used method to true and configure operation parameters. Here, the resize+pad method is used
ai2d_datatype_t ai2d_dtype{ai2d_format::NCHW_FMT, ai2d_format::NCHW_FMT, typecode_t::dt_uint8, typecode_t::dt_uint8};
ai2d_crop_param_t crop_param{false, 0, 0, 0, 0};
ai2d_shift_param_t shift_param{false, 0};
ai2d_pad_param_t pad_param{true, {{0, 0}, {0, 0}, {top, bottom}, {left, right}}, ai2d_pad_mode::constant, {padding[0], padding[1], padding[2]}};
ai2d_resize_param_t resize_param{true, ai2d_interp_method::tf_bilinear, ai2d_interp_mode::half_pixel};
ai2d_affine_param_t affine_param{false, ai2d_interp_method::cv2_bilinear, 0, 0, 127, 1, {0.5, 0.1, 0.0, 0.1, 0.5, 0.0}};
// Set input and output dimensions
dims_t in_shape = {1,3,input_h,input_w};
dims_t out_shape = {1,3,output_h,output_w};
// Initialize the ai2d_builder instance
std::unique_ptr<ai2d_builder> ai2d_builder_; // ai2d builder
// Set parameters
ai2d_builder_.reset(new ai2d_builder(in_shape, out_shape, ai2d_dtype, crop_param, shift_param, pad_param, resize_param, affine_param));
ai2d_builder_.build_schedule();
// Execute preprocessing
ai2d_builder_.invoke(input_tensor,output_tensor_).expect("error occurred in ai2d running");
//Finally, the processed data can be obtained from output_tensor
Regarding the five preprocessing methods of ai2d, examples are provided in the source code at k230_linux_sdk/buildroot-overlay/package/usage_ai2d. Running build_app.sh in the execution directory compiles the executable file, and the compiled product is in the k230_bin directory. It can be copied to the development board for execution and result viewing.
KPU Inference#
We’ve already preprocessed the image and prepared the input tensor—now it’s finally time for the star of the show to make an appearance, and that’s our “Neural Network Acceleration Unit”—KPU!
KPU is the dedicated hardware accelerator on the K230 used to run neural network models. Its role is: hand the model to me, and I’ll handle the inference!
However, before we begin, we need to tell it: hey, which model am I going to use! So you need to place the .kmodel file into the K230 board in advance, and then load this model into KPU in your code.
Next, we need to set up the input—the tensor we processed using the ai2d module earlier comes into play now, serving as the model input to be passed to KPU. Then, we can let KPU start running the model at blazing speed!
Once the model finishes running, KPU will return the results to us. This result is an output tensor, which contains the raw data inferred by the model. However, this format is neither human-readable nor directly usable, so we need to do one more step of “translation”: first, map the output tensor to a data pointer, extract the output data from it, and then perform post-processing operations, such as determining which number was recognized and where its location is.
The figure below shows the process of using KPU for model inference. The model inference process includes loading the model, setting the model input, performing model inference, and obtaining the model output:
KPU is a dedicated acceleration engine for deep learning, designed to accelerate the computational process of neural network models. The API documentation for this module can be found at the following link: nncase KPU Runtime API Documentation. For application examples of KPU, see the source code src/rtsmart/examples/kpu_run_yolov8.
Here is the core code for using the kpu module to perform KPU inference (this code is for illustration only and cannot be run directly):
//Assume this model has one input and one output
runtime_tensor input_tensor;
runtime_tensor output_tensor;
const char* kmodel_path="./test.kmodel";
// Load model
interpreter interp;
std::ifstream ifs(kmodel_path, std::ios::binary);
interp.load_model(ifs).expect("Invalid kmodel");
//Set the input tensor, only one is set here, if there are multiple, you can set by index
interp.input_tensor(0, input_tensor).expect("cannot set input tensor");
//interp.input_tensor(1, input_tensor_1).expect("cannot set input tensor");
//Set the output tensor, only one is set here, if there are multiple, you can set by index
interp.output_tensor(0, output_tensor).expect("cannot set input tensor");
//interp.output_tensor(1, output_tensor_1).expect("cannot set input tensor");
// Execute kpu inference process
interp.run().expect("error occurred in running model");
// Get the pointer to the model output data, from which you can extract data for post-processing
auto buf = output_tensor.impl()->to_host().unwrap()->buffer().as_host().unwrap().map(map_access_::map_read).unwrap().buffer();
float *p_out = reinterpret_cast<float *>(buf.data());
For the four-class printed digit recognition task, the KPU model inference has only one output, with an output shape of [1,8,2100]. The data shape of the output is shown in the figure below:
Post-processing#
The model inference is done! KPU has given us a big “array of numbers” as the result—but don’t celebrate too soon—these numbers look completely meaningless at first glance. So, the next step is to “translate” this data into content that humans can understand, such as: which number appears in the image? Where is it? How reliable is this recognition result?
For example, the output shape of our “four-class printed digit recognition” model is [1, 8, 2100], which means there are a total of 2100 candidate boxes, each described by 8 numbers. What are they specifically? The first 4 are the box’s position (center point X, Y coordinates, plus width and height), and the last 4 are the “scores” for the four digits (0, 1, 2, 3), which represent the model’s confidence in its judgment for each class.
The first thing we need to do is to pick the highest score from these 4 scores, get its class index and corresponding score, which represents the most likely digit for this box and how certain the model is about that digit.
Then, we need to process the position. The model outputs the box as “center point + width and height,” but we are usually more accustomed to the “top-left coordinate + bottom-right coordinate” format, which makes it easier to perform NMS operations later.
So what is NMS (Non-Maximum Suppression)? You can think of it as “deduplication.” Sometimes the model is too “enthusiastic” and produces multiple boxes for the same digit. We don’t need that many—we only keep the one with the highest score, and delete all the others with too much overlap. Clean and tidy! This step is called NMS, and almost all object detection models include this step in their post-processing. It’s very critical!
Finally, there’s one more detail: the model performs inference on the input size. For example, we input a 320×320 image, but the original image might be of a different size. So we also need to “restore” these coordinates proportionally to the original image in order to draw boxes correctly.
After all these operations, we’ve gone from a bunch of “mysterious numbers” output by the model to clear recognition results: which number appears in the image, where it is, how reliable the recognition is, and the boxes are drawn! This step is the legendary “post-processing” stage, and the entire pipeline is considered truly complete!
The figure below illustrates the main work of the post-processing process:
After model inference is complete, the model’s output tensor is mapped to a float type pointer. Users can implement post-processing according to the requirements of their application scenarios. For example, post-processing is implemented on the output of the YOLOv8 model to obtain the coordinates and class information of detection boxes. First, you need to understand the meaning of the output. For the output of [1,8,2100], the 8 represents 4 pieces of coordinate information and scores for 4 classes. The post-processing process needs to find the class index and class score with the highest score, and use the ratio calculated during preprocessing to restore the coordinate information back to the original image size, converting from the center point + width and height format to the top-left and bottom-right coordinate format. Then, a confidence threshold is used to filter out some boxes, and NMS (Non-Maximum Suppression) is used to filter out redundant overlapping boxes. Finally, what is obtained is the detection box information based on the original image. For the four-class printed digit recognition, we provide the core code for the post-processing of this task (this code is for illustration only and cannot be run directly) as follows:
//Define detection box type
typedef struct Bbox{
cv::Rect box;
float confidence;
int index;
}Bbox;
// Post-processing IOU calculation
float get_iou_value(cv::Rect rect1, cv::Rect rect2)
{
int xx1, yy1, xx2, yy2;
xx1 = std::max(rect1.x, rect2.x);
yy1 = std::max(rect1.y, rect2.y);
xx2 = std::min(rect1.x + rect1.width - 1, rect2.x + rect2.width - 1);
yy2 = std::min(rect1.y + rect1.height - 1, rect2.y + rect2.height - 1);
int insection_width, insection_height;
insection_width = std::max(0, xx2 - xx1 + 1);
insection_height = std::max(0, yy2 - yy1 + 1);
float insection_area, union_area, iou;
insection_area = float(insection_width) * insection_height;
union_area = float(rect1.width*rect1.height + rect2.width*rect2.height - insection_area);
iou = insection_area / union_area;
return iou;
}
//NMS non-maximum suppression, bboxes is a list of Bbox instances to be processed, indices is the remaining bboxes box indices after NMS
void nms(std::vector<Bbox> &bboxes, float confThreshold, float nmsThreshold, std::vector<int> &indices)
{
sort(bboxes.begin(), bboxes.end(), [](Bbox a, Bbox b) { return a.confidence > b.confidence; });
int updated_size = bboxes.size();
for (int i = 0; i < updated_size; i++)
{
if (bboxes[i].confidence < confThreshold)
continue;
indices.push_back(i);
for (int j = i + 1; j < updated_size;)
{
float iou = get_iou_value(bboxes[i].box, bboxes[j].box);
if (iou > nmsThreshold)
{
bboxes.erase(bboxes.begin() + j);
updated_size = bboxes.size();
}
else
{
j++;
}
}
}
}
// Post-processing after model inference is complete
//Output pointer obtained from the previous step
float *p_out;
// Label names
std::vector<std::string> classes{"0","1","2","3"};
// Confidence threshold
float conf_thresh=0.25;
// nms threshold
float nms_thresh=0.45;
//Number of classes
int class_num=classes.size();
// output0 [num_class+4,(w/8)*(h/8)+(w/16)*(h/16)+(w/32)*(h/32)]
float *output0 = p_out;
// Feature length for each box, class_num scores + 4 coordinates
int f_len=class_num+4;
// Calculate the total number of output boxes based on the model's input resolution
int num_box=((input_shapes[0][2]/8)*(input_shapes[0][3]/8)+(input_shapes[0][2]/16)*(input_shapes[0][3]/16)+(input_shapes[0][2]/32)*(input_shapes[0][3]/32));
// Allocate box data memory
float *output_det = new float[num_box * f_len];
// Adjust the output data layout from [num_class+4,(w/8)*(h/8)+(w/16)*(h/16)+(w/32)*(h/32)] to [(w/8)*(h/8)+(w/16)*(h/16)+(w/32)*(h/32),num_class+4] for easier subsequent processing
for(int r = 0; r < num_box; r++)
{
for(int c = 0; c < f_len; c++)
{
output_det[r*f_len + c] = output0[c*num_box + r];
}
}
// Parse each box's information, class_num+4 is one box, the first four data are coordinate values, the following class_num are scores. Select the highest score as the recognized class. Because padding+resize was performed at the beginning, the model inference coordinates are based on the preprocessed image, so the box coordinates need to be mapped back to the original image using ratio first
std::vector<Bbox> bboxes;
for(int i=0;i<num_box;i++){
float* vec=output_det+i*f_len;
float box[4]={vec[0],vec[1],vec[2],vec[3]};
float* class_scores=vec+4;
float* max_class_score_ptr=std::max_element(class_scores,class_scores+class_num);
float score=*max_class_score_ptr;
int max_class_index = max_class_score_ptr - class_scores; // Calculate index
if(score>conf_thresh){
Bbox bbox;
// Restore to original image ratio
float x_=box[0]/ratio*1.0;
float y_=box[1]/ratio*1.0;
float w_=box[2]/ratio*1.0;
float h_=box[3]/ratio*1.0;
int x=int(MAX(x_-0.5*w_,0));
int y=int(MAX(y_-0.5*h_,0));
int w=int(w_);
int h=int(h_);
if (w <= 0 || h <= 0) { continue; }
bbox.box=cv::Rect(x,y,w,h);
bbox.confidence=score;
bbox.index=max_class_index;
bboxes.push_back(bbox);
}
}
//Execute non-maximum suppression to eliminate redundant overlapping boxes with lower confidence (NMS)
std::vector<int> nms_result;
nms(bboxes, conf_thresh, nms_thresh, nms_result);
The above code shows the post-processing steps for the YOLOv8 four-class printed digit recognition model.
Result Drawing#
Now we have the recognition results! We know each digit’s “identity” and “location.” The next step is to make these results “visible”—that is, draw the detection boxes on the image and label the digits, telling everyone: “Look! Here’s a 1!” “There’s a 3 over there!”
However, it’s not that simple—your model performed recognition on a 320×320 image, but the screen might be 800×480, 1920×1080, or even another size. If you directly draw the model’s boxes on the screen, the positions might all be off! So we need to do something very important: “map” the image coordinates to screen coordinates, that is, convert the box positions proportionally so that they fit just right on the screen.
When drawing these recognition information, we generally don’t modify the original image directly. Instead, we create a transparent layer called OSD (On-Screen Display) that is the same size as the screen, like sticking a piece of glass film on a photo. We draw boxes and label categories on this layer, without affecting the underlying image.
The final step is to overlay this OSD layer with the original image and display them together on the screen! This way, you can clearly see: each digit has been recognized, and the boxes are drawn properly!
The figure below shows the workflow for drawing the results:
Taking the detection boxes of the four-class printed digit recognition as an example, the detection box coordinates we calculated are based on the input original image resolution. If we want to display them on the screen, we need to convert the coordinates proportionally to the screen coordinate resolution, then draw the effects on an initialized transparent cv::Mat, and then call the V4L2-DRM callback method to achieve overlay display. Here is the core code (this code is for illustration only and cannot be run directly):
/**
* @brief V4L2-DRM display frame handler function (triggered once per frame)
*
* This function is called back by the v4l2_drm_run driver loop and is invoked each time frame display data is processed. Its main function is to display the AI inference results.
* @param context Pointer to the V4L2-DRM context structure
* @param displayed Indicates whether the frame has been actually displayed
* @return Returns 0 for normal operation, returns 'q' to request exit from the main loop (controlled by the display_stop flag)
*/
int frame_handler(struct v4l2_drm_context *context, bool displayed)
{
static bool first_frame = true;
if (first_frame) {
result_mutex.unlock();
first_frame = false;
}
static unsigned response = 0, display_frame_count = 0;
response += 1;
if (displayed)
{
if (context[0].buffer_hold[context[0].wp] >= 0)
{
static struct display_buffer* last_drawed_buffer = nullptr;
auto buffer = context[0].display_buffers[context[0].buffer_hold[context[0].wp]];
if (buffer != last_drawed_buffer) {
//---------------------- Draw display results ----------------------
if (draw_buffer->width > draw_buffer->height)
{
// Create temporary BGRA display buffer Mat (for drawing)
cv::Mat temp_img(draw_buffer->height, draw_buffer->width, CV_8UC4);
// Landscape
temp_img.setTo(cv::Scalar(0, 0, 0, 0));
result_mutex.lock();
draw_frame.copyTo(temp_img);
result_mutex.unlock();
//---------------------- Display buffer synchronization ----------------------
// Copy the drawn image to the actual display buffer
memcpy(draw_buffer->map, temp_img.data, draw_buffer->size);
}
else
{
// Portrait
// Create temporary BGRA display buffer Mat (for drawing)
cv::Mat temp_img(draw_buffer->width, draw_buffer->height, CV_8UC4);
// Portrait st7701: draw in landscape, then rotate back to portrait for display
temp_img.setTo(cv::Scalar(0, 0, 0, 0));
result_mutex.lock();
draw_frame.copyTo(temp_img);
result_mutex.unlock();
// Rotate back to screen orientation
cv::rotate(temp_img, temp_img, cv::ROTATE_90_CLOCKWISE);
//---------------------- Display buffer synchronization ----------------------
// Copy the drawn image to the actual display buffer
memcpy(draw_buffer->map, temp_img.data, draw_buffer->size);
}
last_drawed_buffer = buffer;
// flush cache
thead_csi_dcache_clean_invalid_range(draw_buffer->map, draw_buffer->size);
display_update_buffer(draw_buffer, 0, 0);
}
}
display_frame_count += 1;
}
// FPS counter
gettimeofday(&tv2, NULL);
uint64_t duration = 1000000 * (tv2.tv_sec - tv.tv_sec) + tv2.tv_usec - tv.tv_usec;
if (duration >= 1000000) {
fprintf(stderr, " poll: %.2f, ", response * 1000000. / duration);
response = 0;
if (display) {
fprintf(stderr, "display: %.2f, ", display_frame_count * 1000000. / duration);
display_frame_count = 0;
}
fprintf(stderr, "camera: %.2f, ", context[0].frame_count * 1000000. / duration);
context[0].frame_count = 0;
fprintf(stderr, "KPU: %.2f", kpu_frame_count * 1000000. / duration);
kpu_frame_count = 0;
fprintf(stderr, " \r");
fflush(stderr);
gettimeofday(&tv, NULL);
}
// If an exit signal is received, return 'q' to exit the main loop
if (display_stop) {
return 'q';
}
return 0;
}
/**
* @brief Display thread main function, initializes V4L2-DRM and binds the drawing callback
*
* According to the screen orientation (landscape / portrait), configure the corresponding width, height, format, and rotation angle,
* Then call `v4l2_drm_run()` to start the frame processing main loop, which triggers drawing in `frame_handler()` every frame.
*
* @param video_device Video device number (such as the 1 in /dev/video0)
*/
void display_proc(int video_device)
{
struct v4l2_drm_context context;
v4l2_drm_default_context(&context);
context.device = video_device;
// Set width/height/rotation based on screen orientation
if (display->width > display->height)
{
// Landscape
context.width = display->width;
context.height = (display->width * SENSOR_HEIGHT / SENSOR_WIDTH) & 0xfff8;
context.video_format = V4L2_PIX_FMT_NV12;
context.display_format = 0;
context.drm_rotation = rotation_0;
}
else
{
// Portrait
context.width = display->height;
context.height = display->width;
context.video_format = V4L2_PIX_FMT_NV12;
context.display_format = 0;
context.drm_rotation = rotation_90;
}
if (v4l2_drm_setup(&context, 1, &display)) {
std::cerr << "v4l2_drm_setup error" << std::endl;
return;
}
// Allocate OSD display plane and buffer
struct display_plane* plane = display_get_plane(display, DRM_FORMAT_ARGB8888);
draw_buffer = display_allocate_buffer(plane, display->width, display->height);
display_commit_buffer(draw_buffer, 0, 0);
if (draw_buffer->width > draw_buffer->height)
{
draw_frame = cv::Mat(draw_buffer->height, draw_buffer->width, CV_8UC4, cv::Scalar(0,0, 0, 0));
}
else{
draw_frame = cv::Mat(draw_buffer->width, draw_buffer->height, CV_8UC4, cv::Scalar(0,0, 0, 0));
}
std::cout << "press 'q' to exit" << std::endl;
gettimeofday(&tv, NULL);
v4l2_drm_run(&context, 1, frame_handler);
// Clean up resources
if (display) {
display_free_plane(plane);
display_exit(display);
}
return;
}
Through the above steps, we have basically completed the full steps of developing an application using the k230 Linux SDK. Users need to have a relatively good understanding of the entire model inference process, starting from model conversion.
Display Device Introduction#
For display output, k230 provides two types of display devices. You can choose to use either HDMI/LCD screen.
🏷️ HDMI: The device type is LT9611. You can refer to the API documentation to check the supported resolution, frame rate, and number of OSDs during initialization. Under dual-channel AI inference, another layer is generally created, overlaying an OSD transparent image of the same size as the screen’s display resolution on top to display inference results.
🏷️ LCD: The device type is ST7701. You can refer to the API documentation to check the supported resolution, frame rate, and number of OSDs during initialization. Under dual-channel AI inference, another layer is generally created, overlaying an OSD transparent image of the same size as the screen’s display resolution on top to display inference results.
How to switch the screen? Flash the image on the development board and power it on, then enter /boot. If you use hdmi, no operation is needed. If you use lcd, you need to modify k.dtb and create a symbolic link from ***-lcd.dtb to k.dtb. The command is as follows:
# The dtb file with lcd is for the screen, without it is for hdmi, create symbolic link, remember to replace the name
ln -s k230-***-**.dtb k.dtb
reboot
After restarting, the default display device switches to hdmi or lcd, depending on the k.dtb file you modified.
Class Printed Digit Recognition Deployment Code#
If you use YOLO to implement four-class printed digit detection and recognition, the SDK has already prepared complete code for you, not only supporting inference on a single image, but also supporting continuous recognition of real-time video streams! Whether you want to test the model’s performance on static images or perform real-time detection after connecting a camera, you can quickly get started. You only need to use the kmodel exported in the previous steps, along with our provided example scripts, and you can easily deploy and run them on the K230 development board!
If you want to verify the recognition accuracy and localization performance of the model on images, you can directly run our image recognition code; if you want to experience the “video effect” of the recognition process in real-time, try the dual-channel video recognition code to see if the boxes can accurately track the digits when they appear on the screen!
Next, you can boldly try the deployment process and experience the running effect of edge AI on the K230. AI can read the digital world you capture!
💡 Firmware Introduction: Please refer to the steps in the document below to set up the compilation environment and compile the firmware to ensure that the latest features are supported! The document tutorial is available at: Firmware Compilation.
yolo Compilation#
Navigate to the buildroot-overlay/package/yolo directory and execute build_app.sh. The compiled firmware is generated in the k230_bin directory.
Model File Copy#
After the development board is flashed with the firmware and powered on, you can find a virtual USB drive CanMV at the disk location. The disk is divided into two partitions: /sdcard and /data. Copy the converted kmodel, the compiled executable elf file, test images, and label files (txt format, one class per line) to a directory on the development board.
Image Recognition Command#
Here is the complete four-class printed digit recognition image inference command. You can use the kmodel obtained from the above steps for testing:
./yolo.elf -model_type yolov5 -task_type detect -task_mode image -image_path test.jpg -kmodel_path best.kmodel -labels_txt_filepath number_labels.txt -conf_thres 0.35 -nms_thres 0.65 -mask_thres 0.5 -debug_mode 0
Dual-channel Video Recognition Command#
Here is the complete four-class printed digit recognition video inference command. You can use the kmodel obtained from the above steps for testing:
./yolo.elf -model_type yolov5 -task_type detect -task_mode video -kmodel_path best.kmodel -labels_txt_filepath number_labels.txt -conf_thres 0.35 -nms_thres 0.65 -mask_thres 0.5 -debug_mode 0
YOLO Deployment#
YOLO is a commonly used model in vision tasks, supporting classification, detection, segmentation, rotated object detection and other tasks. We selected the classic YOLOv5, YOLOv8 and YOLO11 models from the YOLO series as the foundation, and encapsulated the deployment code for YOLOv5, YOLOv8 and YOLO11 to facilitate users in quickly deploying YOLO models. See the link for details: YOLO Application Guide.
YOLOv5 Cat and Dog Classification#
Deployment of a cat and dog classification model based on the YOLOv5 model on the K230.
YOLOv5 Source Code and Training Environment Setup#
For setting up the YOLOv5 training environment, please refer to ultralytics/yolov5: YOLOv5 🚀 in PyTorch > ONNX > CoreML > TFLite (github.com)
git clone https://github.com/ultralytics/yolov5.git
cd yolov5
pip install -r requirements.txt
If you have already set up the environment, please ignore this step.
Training Data Preparation#
Please download the provided sample dataset. The sample dataset uses cat and dog classification as the scenario, and is trained using YOLOv5.
cd yolov5
wget https://kendryte-download.canaan-creative.com/developer/k230/yolo_dataset/cat_dog.zip
unzip cat_dog.zip
⚠️ For Windows systems, please copy the link directly to the browser to download, and unzip it to the corresponding directory.
If you have already downloaded the data, please ignore this step.
Using YOLOv5 to Train the Cat and Dog Classification Model#
Execute the following command in the yolov5 directory to train the cat and dog classification model using yolov5:
python classify/train.py --model yolov5n-cls.pt --data cat_dog --epochs 100 --batch-size 8 --imgsz 224 --device '0'
Converting Cat and Dog Classification kmodel#
Model conversion requires installing the following libraries in the training environment:
# linux platform: nncase and nncase-kpu can be installed online, nncase-2.x requires installing dotnet-7
sudo apt-get install -y dotnet-sdk-7.0
pip install --upgrade pip
pip install nncase==2.11.0
pip install nncase-kpu==2.11.0
# windows platform: Please install dotnet-7 by yourself and add environment variables. Online installation of nncase via pip is supported, but the nncase-kpu library needs to be installed offline. Download nncase_kpu-2.*-py2.py3-none-win_amd64.whl from https://github.com/kendryte/nncase/releases
# Enter the corresponding python environment, and use pip to install in the directory where nncase_kpu-2.*-py2.py3-none-win_amd64.whl is downloaded
pip install nncase_kpu-2.*-py2.py3-none-win_amd64.whl
# In addition to nncase and nncase-kpu, other libraries used by the script include:
pip install onnx==1.15.0
pip install onnxruntime==1.19.0
pip install onnxsim==0.4.36
Download the script tool and unzip the model conversion script tool test_yolov5.zip to the yolov5 directory;
wget https://kendryte-download.canaan-creative.com/developer/k230/yolo_files/test_yolov5.zip
unzip test_yolov5.zip
According to the following commands, first export the pt model under runs/train-cls/exp/weights to an onnx model, and then convert it to a kmodel model:
# Export onnx, please choose the pt model path yourself
python export.py --weight runs/train-cls/exp/weights/best.pt --imgsz 224 --batch 1 --include onnx
cd test_yolov5/classify
# Replace the images in the test directory with part of your own training data, convert kmodel, please choose the onnx model path yourself, the generated kmodel is in the same directory as the onnx model
python to_kmodel.py --target k230 --model ../../runs/train-cls/exp/weights/best.onnx --dataset ../test --input_width 224 --input_height 224 --ptq_option 0
cd ../../
💡 Model Conversion Script (to_kmodel.py) Parameter Description:
Parameter Name |
Description |
Notes |
Type |
|---|---|---|---|
target |
Target Platform |
Options are k230/cpu, corresponding to k230 kpu and cpu; |
str |
model |
Model Path |
The ONNX model path to be converted; |
str |
dataset |
Calibration Image Set |
The image data used during model conversion, used in the quantization stage, can be taken from part of the training data |
str |
input_width |
Input Width |
The width of the model input |
int |
input_height |
Input Height |
The height of the model input |
int |
ptq_option |
Quantization Method |
The quantization strategies are |
0/1/2/3/4/5 |
Deploying the Model on k230 Using Linux#
Setting Up the Compilation Environment#
💡 Firmware Introduction: Please refer to the steps in the k230_linux documentation to set up the compilation environment and compile the firmware to ensure that the latest features are supported!
yolo Compilation#
Execute make menuconfig under k230_linux_sdk, select Target packages > canaan package > AI > yolo demo, then select Save->OK below, save and exit. In this way, the yolo example can be compiled into the firmware when compiling the firmware. After flashing the firmware, you can find the compiled yolo.elf executable file and test files under /app/yolo.
If you do not compile when compiling the firmware, you can also choose to compile the yolos example separately. Enter the k230_linux_sdk/buildroot-overlay/package/yolo directory and execute the ./build_app.sh script. The compiled yolo.elf executable file is in the k230_bin directory. You can copy the k230_bin directory to the development board on which the firmware has been flashed.
Model File Copy#
After the development board is flashed with the firmware and powered on, copy the converted kmodel, the compiled executable elf file, the test images, and the label file (txt format, each line represents one class) to a directory on the development board.
YOLO Module#
The YOLO module is deployment code developed based on YOLO series models, implementing the deployment process of such models. The supported situations are as follows:
Integrates four models:
YOLOv5, YOLOv8, YOLO11, YOLO26;Supports four types of tasks, among which
YOLOv5supports three types: classification (classify), detection (detect), and segmentation (segment).YOLOv8, YOLO11, YOLO26support five types: classification (classify), detection (detect), segmentation (segment), rotated object detection (obb), and keypoint detection (pose);Supports two inference modes, including image (image) and video stream (video);
Supports two display modes, including
LT9611 (hdmi,1920×1080),ST7701 (lcd screen,800×480);Parameter Description
Parameter Name |
Default Value |
Description |
|---|---|---|
|
yolov8 |
Set the model type, default value is yolov8, optional values: yolov5/yolov8/yolo11. |
|
detect |
Set the task type, default value is detect, optional values: classify/detect/segment/obb. |
|
video |
Set the task mode, default value is video, optional values: image/video |
|
test.jpg |
Set the image path, default value is test.jpg. |
|
yolov8n.kmodel |
Set the kmodel path, default value is yolov8n.kmodel. |
|
coco_labels.txt |
Set the label text file path, default value is coco_labels.txt, each label occupies one line. |
|
0.35 |
Set the confidence threshold, default value is 0.35. |
|
0.65 |
Set the non-maximum suppression threshold, default value is 0.65. |
|
0.5 |
Set the mask threshold, default value is 0.5. |
|
17 |
Set the number of keypoints, default value is 17 (human skeleton keypoint scenario). |
|
3 |
Set the keypoint dimension of the model, only supports 2/3, default value is 3 (human skeleton keypoint scenario). |
|
0 |
Set the debug mode, default value is 0, optional values: 0/1, 0 means no debugging, 1 means debug printing. |
Deploying the Model to Implement Image Inference#
For image inference, please refer to the following command, modify the parameters according to the actual situation;
./yolo.elf -model_type yolov5 -task_type classify -task_mode image -image_path test.jpg -kmodel_path yolov5n_cat_dog_cls.kmodel -labels_txt_filepath cat_dog_labels.txt -conf_thres 0.5 -debug_mode 0
Deploying the Model to Implement Video Inference#
For video inference, please refer to the following command, modify the parameters according to the actual situation;
./yolo.elf -model_type yolov5 -task_type classify -task_mode video -kmodel_path yolov5n_cat_dog_cls.kmodel -labels_txt_filepath cat_dog_labels.txt -conf_thres 0.5 -debug_mode 0
Deployment Effect#
Two cat and dog images are selected for classification using kmodel. The effect is shown in the figure below:
YOLOv8 Fall Detection#
Based on the YOLOv8 model, implement the deployment of the fall detection model on K230.
YOLOv8 Source Code and Training Environment Setup#
For setting up the YOLOv8 training environment, please refer to ultralytics/ultralytics: Ultralytics YOLO 🚀 (github.com)
# Pip install the ultralytics package including all requirements in a Python>=3.8 environment with PyTorch>=1.8.
pip install ultralytics
If you have already set up the environment, please ignore this step.
Training Data Preparation#
Download the provided fall detection dataset and extract it.
cd yolov8
wget https://kendryte-download.canaan-creative.com/developer/k230/yolo_dataset/fall_det.zip
unzip fall_det.zip
⚠️ For Windows systems, please copy the link directly to the browser to download, and extract it to the corresponding directory.
If you have already downloaded the data, please ignore this step.
Using YOLOv8 to Train the Fall Detection Model#
Execute the following command in the yolov8 directory to train the fall detection model using yolov8:
yolo detect train data=fall_det.yaml model=yolov8n.pt epochs=300 imgsz=320
Converting the Fall Detection kmodel#
The model conversion requires installing the following libraries in the training environment:
# linux platform: nncase and nncase-kpu can be installed online, nncase-2.x requires installing dotnet-7
sudo apt-get install -y dotnet-sdk-7.0
pip install --upgrade pip
pip install nncase==2.11.0
pip install nncase-kpu==2.11.0
# windows platform: please install dotnet-7 by yourself and add environment variables. nncase can be installed online using pip, but the nncase-kpu library needs to be installed offline. Download nncase_kpu-2.*-py2.py3-none-win_amd64.whl from https://github.com/kendryte/nncase/releases
# Enter the corresponding python environment, and use pip to install in the directory where nncase_kpu-2.*-py2.py3-none-win_amd64.whl is downloaded
pip install nncase_kpu-2.*-py2.py3-none-win_amd64.whl
# In addition to nncase and nncase-kpu, other libraries used in the script include:
pip install onnx==1.15.0
pip install onnxruntime==1.19.0
pip install onnxsim==0.4.36
Download the script tool and extract the model conversion script tool test_yolov8.zip to the yolov8 directory;
wget https://kendryte-download.canaan-creative.com/developer/k230/yolo_files/test_yolov8.zip
unzip test_yolov8.zip
According to the following commands, first export the pt model under runs/detect/train/weights to an onnx model, then convert it to a kmodel model:
# Export onnx, please choose the pt model path by yourself
yolo export model=runs/detect/train/weights/best.pt format=onnx imgsz=320
cd test_yolov8/detect
# Replace the images in the test directory with a portion of your own training data. To convert kmodel, please choose the onnx model path by yourself. The generated kmodel will be in the same directory as the onnx model
python to_kmodel.py --target k230 --model ../../runs/detect/train/weights/best.onnx --dataset ../test --input_width 320 --input_height 320 --ptq_option 1
cd ../../
💡 Description of the model conversion script (to_kmodel.py) parameters:
Parameter Name |
Description |
Notes |
Type |
|---|---|---|---|
target |
Target Platform |
Options are k230/cpu, corresponding to k230 kpu and cpu; |
str |
model |
Model Path |
Path of the ONNX model to be converted; |
str |
dataset |
Calibration Image Set |
Image data used in model conversion, used during the quantization stage. You can replace it with a portion of images from the training set |
str |
input_width |
Input Width |
The width of the model input |
int |
input_height |
Input Height |
The height of the model input |
int |
ptq_option |
Quantization Method |
Quantization strategies are |
0/1/2/3/4/5 |
Deploying the Model on k230 Using Linux#
Setting Up the Compilation Environment#
💡 Firmware Introduction: Please refer to the steps in the k230_linux documentation to set up the compilation environment and compile the firmware to ensure that the latest features are supported!
yolo Compilation#
Execute make menuconfig under k230_linux_sdk, select Target packages > canaan package > AI > yolo demo, select Save->OK below, and exit after saving. In this way, when compiling the firmware, the yolo example can be compiled into the firmware. After flashing the firmware, you can find the compiled yolo.elf executable file and test files under /app/yolo.
If you do not compile it when compiling the firmware, you can also choose to compile the yolos example separately. Enter the k230_linux_sdk/buildroot-overlay/package/yolo directory and execute the ./build_app.sh script. The yolo.elf executable file generated by compilation is in the k230_bin directory. You can copy the k230_bin directory to the development board that has already been flashed with firmware.
Copying Model Files#
After the development board is flashed with firmware and powered on, copy the converted kmodel, the compiled executable elf file, test images, and the label file (in txt format, with each line representing one class) to a directory on the development board.
YOLO Module#
The YOLO module is a deployment code developed based on YOLO series models, implementing the deployment process of such models. Its supported cases are as follows:
Integrates four models:
YOLOv5, YOLOv8, YOLO11, YOLO26;Supports four types of tasks.
YOLOv5supports classify, detect, and segment.YOLOv8, YOLO11, YOLO26support five types of tasks: classify, detect, segment, oriented bounding box (obb), and pose;Supports two inference modes, including image and video;
Supports two display modes, including
LT9611 (hdmi,1920×1080),ST7701 (lcd screen,800×480);Parameter Description
Parameter Name |
Default Value |
Description |
|---|---|---|
|
yolov8 |
Set the model type. The default value is yolov8. Optional values: yolov5/yolov8/yolo11. |
|
detect |
Set the task type. The default value is detect. Optional values: classify/detect/segment/obb. |
|
video |
Set the task mode. The default value is video. Optional values: image/video |
|
test.jpg |
Set the image path. The default value is test.jpg. |
|
yolov8n.kmodel |
Set the kmodel path. The default value is yolov8n.kmodel. |
|
coco_labels.txt |
Set the path of the label text file. The default value is coco_labels.txt, with each label on a separate line. |
|
0.35 |
Set the confidence threshold. The default value is 0.35. |
|
0.65 |
Set the non-maximum suppression threshold. The default value is 0.65. |
|
0.5 |
Set the mask threshold. The default value is 0.5. |
|
17 |
Set the number of keypoints. The default value is 17 (for the human skeleton keypoint scenario). |
|
3 |
Set the keypoint dimension of the model. Only 2/3 are supported. The default value is 3 (for the human skeleton keypoint scenario). |
|
0 |
Set the debug mode. The default value is 0. Optional values: 0/1, 0 for no debug, 1 for debug printing. |
Deploying the Model to Implement Image Inference#
For image inference, please refer to the following command, modify the parameters according to the actual situation;
./yolo.elf -model_type yolov8 -task_type detect -task_mode image -image_path test.jpg -kmodel_path yolov8n_fall_det.kmodel -labels_txt_filepath fall_labels.txt -conf_thres 0.5 -nms_thres 0.45 -debug_mode 0
Deploying the Model to Implement Video Inference#
For video inference, please refer to the following command, modify the parameters according to the actual situation;
./yolo.elf -model_type yolov8 -task_type detect -task_mode video -kmodel_path yolov8n_fall_det.kmodel -labels_txt_filepath fall_labels.txt -conf_thres 0.5 -nms_thres 0.45 -debug_mode 0
Deployment Effect#
Select a fall image and use kmodel for fall detection. The comparison between the original image and the inference result is shown in the figure below:
YOLO11 Fruit Segmentation#
YOLO11 Source Code and Training Environment Setup#
For setting up the YOLO11 training environment, please refer to ultralytics/ultralytics: Ultralytics YOLO 🚀 (github.com)
# Pip install the ultralytics package including all requirements in a Python>=3.8 environment with PyTorch>=1.8.
pip install ultralytics
If you have already set up the environment, please ignore this step.
Training Data Preparation#
Download the provided fruit segmentation dataset and extract it.
cd yolo11
wget https://kendryte-download.canaan-creative.com/developer/k230/yolo_dataset/fruit_seg.zip
unzip fruit_seg.zip
⚠️ For Windows systems, please copy the link directly to the browser to download, and extract to the corresponding directory.
If you have already downloaded the data, please ignore this step.
Using YOLO11 to Train the Fruit Segmentation Model#
Execute the command in the yolo11 directory to train a three-class fruit segmentation model using yolo11:
yolo segment train data=fruits_seg.yaml model=yolo11n-seg.pt epochs=100 imgsz=320
Converting the Fruit Segmentation kmodel#
Model conversion requires installing the following libraries in the training environment:
# linux platform: nncase and nncase-kpu can be installed online, nncase-2.x requires dotnet-7
sudo apt-get install -y dotnet-sdk-7.0
pip install --upgrade pip
pip install nncase==2.11.0
pip install nncase-kpu==2.11.0
# windows platform: please install dotnet-7 yourself and add environment variables. Online pip installation of nncase is supported, but the nncase-kpu library needs to be installed offline. Download nncase_kpu-2.*-py2.py3-none-win_amd64.whl from https://github.com/kendryte/nncase/releases
# Enter the corresponding python environment, and use pip to install in the directory where nncase_kpu-2.*-py2.py3-none-win_amd64.whl is downloaded
pip install nncase_kpu-2.*-py2.py3-none-win_amd64.whl
# In addition to nncase and nncase-kpu, the other libraries used by the script include:
pip install onnx==1.15.0
pip install onnxruntime==1.19.0
pip install onnxsim==0.4.36
Download the script tool and extract the model conversion script tool test_yolo11.zip to the yolo11 directory;
wget https://kendryte-download.canaan-creative.com/developer/k230/yolo_files/test_yolo11.zip
unzip test_yolo11.zip
According to the following commands, first export the pt model under runs/segment/train/weights to an onnx model, and then convert it to a kmodel model:
# Export onnx, please choose the pt model path yourself
yolo export model=runs/segment/train/weights/best.pt format=onnx imgsz=320
cd test_yolo11/segment
# The images in test can be selected from the training set to replace part of them. Convert kmodel, please choose the onnx model path yourself. The generated kmodel is in the same directory as the onnx model
python to_kmodel.py --target k230 --model ../../runs/segment/train/weights/best.onnx --dataset ../test --input_width 320 --input_height 320 --ptq_option 1
cd ../../
💡 Model conversion script (to_kmodel.py) parameter description:
Parameter Name |
Description |
Notes |
Type |
|---|---|---|---|
target |
Target platform |
Options are k230/cpu, corresponding to k230 kpu and cpu; |
str |
model |
Model path |
The path of the ONNX model to be converted; |
str |
dataset |
Calibration image set |
The image data used in model conversion, used in the quantization stage, can be taken from the training set to replace part of it |
str |
input_width |
Input width |
The width of the model input |
int |
input_height |
Input height |
The height of the model input |
int |
ptq_option |
Quantization method |
The quantization strategies are |
0/1/2/3/4/5 |
Deploying the Model on k230 Using Linux#
Building the Compilation Environment#
💡 Firmware Introduction: Please refer to the steps in the k230_linux documentation to build the compilation environment and compile the firmware, to ensure that the latest features are supported!
yolo Compilation#
Under k230_linux_sdk, execute make menuconfig, select Target packages > canaan package > AI > yolo demo, select Save->OK below, save and exit. In this way, the yolo example can be compiled into the firmware. After flashing the firmware, the compiled yolo.elf executable file and test files can be found under /app/yolo.
If you do not compile when building the firmware, you can also choose to compile the yolos example separately. Enter the k230_linux_sdk/buildroot-overlay/package/yolo directory and execute the ./build_app.sh script. The compiled yolo.elf executable file is in the k230_bin directory. You can copy the k230_bin directory to the development board that has already been flashed with the firmware.
Copying Model Files#
After the development board is flashed with the firmware and powered on, copy the converted kmodel, the compiled executable elf file, test images, and the label file (in txt format, one class per line) to a directory on the development board.
YOLO Module#
The YOLO module is a deployment code developed based on the YOLO series models, implementing the deployment process of such models. Its supported situations are as follows:
Integrates four models:
YOLOv5, YOLOv8, YOLO11, YOLO26;Supports four types of tasks. Among them,
YOLOv5supports three types: classify, detect, and segment.YOLOv8, YOLO11, YOLO26support five types: classify, detect, segment, obb (oriented object detection), and pose (keypoint detection);Supports two inference modes, including image and video stream;
Supports two display modes, including
LT9611 (hdmi,1920×1080)andST7701 (lcd screen,800×480);Parameter Description
Parameter Name |
Default Value |
Description |
|---|---|---|
|
yolov8 |
Set the model type. The default value is yolov8. Optional values: yolov5/yolov8/yolo11. |
|
detect |
Set the task type. The default value is detect. Optional values: classify/detect/segment/obb. |
|
video |
Set the task mode. The default value is video. Optional values: image/video |
|
test.jpg |
Set the image path. The default value is test.jpg. |
|
yolov8n.kmodel |
Set the kmodel path. The default value is yolov8n.kmodel. |
|
coco_labels.txt |
Set the label text file path. The default value is coco_labels.txt. Each label is on a separate line. |
|
0.35 |
Set the confidence threshold. The default value is 0.35. |
|
0.65 |
Set the non-maximum suppression threshold. The default value is 0.65. |
|
0.5 |
Set the mask threshold. The default value is 0.5. |
|
17 |
Set the number of keypoints. The default value is 17 (human skeleton keypoint scenario). |
|
3 |
Set the keypoint dimension of the model. Only 2/3 is supported. The default value is 3 (human skeleton keypoint scenario). |
|
0 |
Set the debug mode. The default value is 0. Optional values: 0/1. 0 means no debugging, 1 means debug printing. |
Deploying the Model to Implement Image Inference#
For image inference, please refer to the following command, modify the parameters according to the actual situation;
./yolo.elf -model_type yolo11 -task_type segment -task_mode image -image_path test.jpg -kmodel_path yolo11n_fruit_seg.kmodel -labels_txt_filepath fruit_labels.txt -conf_thres 0.5 -nms_thres 0.45 -mask_thres 0.5 -debug_mode 0
Deploying the Model to Implement Video Inference#
For video inference, please refer to the following command, modify the parameters according to the actual situation;
./yolo.elf -model_type yolo11 -task_type segment -task_mode video -kmodel_path yolo11n_fruit_seg.kmodel -labels_txt_filepath fruit_labels.txt -conf_thres 0.5 -nms_thres 0.45 -mask_thres 0.5 -debug_mode 0
Deployment Effect#
Select a fruit image and use the kmodel for fruit segmentation. The comparison between the original image and the inference result is shown in the figure below:
YOLO11 Rotated Object Detection#
YOLO11 Source Code and Training Environment Setup#
For setting up the YOLO11 training environment, please refer to ultralytics/ultralytics: Ultralytics YOLO 🚀 (github.com)
# Pip install the ultralytics package including all requirements in a Python>=3.8 environment with PyTorch>=1.8.
pip install ultralytics
If you have already set up the environment, please ignore this step.
Training Data Preparation#
Download the desktop pen rotated object detection dataset and unzip it.
cd yolo11
wget https://kendryte-download.canaan-creative.com/developer/k230/yolo_dataset/pen_obb.zip
unzip pen_obb.zip
⚠️ For Windows systems, please copy the link directly to the browser to download, and unzip to the corresponding directory.
If you have already downloaded the data, please ignore this step.
Using YOLO11 Rotated Object Detection Model#
Execute the command in the yolo11 directory to train a single-class rotated object detection model using yolo11:
yolo obb train data=pen_obb.yaml model=yolo11n-obb.pt epochs=100 imgsz=320
Converting Rotated Object Detection kmodel#
The model conversion requires installing the following libraries in the training environment:
# linux platform: nncase and nncase-kpu can be installed online, nncase-2.x requires installing dotnet-7
sudo apt-get install -y dotnet-sdk-7.0
pip install --upgrade pip
pip install nncase==2.11.0
pip install nncase-kpu==2.11.0
# windows platform: please install dotnet-7 by yourself and add environment variables, pip online installation of nncase is supported, but nncase-kpu library needs to be installed offline. Download nncase_kpu-2.*-py2.py3-none-win_amd64.whl at https://github.com/kendryte/nncase/releases
# Enter the corresponding python environment, and use pip to install in the directory where nncase_kpu-2.*-py2.py3-none-win_amd64.whl is downloaded
pip install nncase_kpu-2.*-py2.py3-none-win_amd64.whl
# In addition to nncase and nncase-kpu, other libraries used by the script include:
pip install onnx==1.15.0
pip install onnxruntime==1.19.0
pip install onnxsim==0.4.36
Download the script tool and unzip the model conversion script tool test_yolo11.zip into the yolo11 directory;
wget https://kendryte-download.canaan-creative.com/developer/k230/yolo_files/test_yolo11.zip
unzip test_yolo11.zip
According to the following commands, first export the pt model under runs/obb/train/weights to an onnx model, and then convert it to a kmodel model:
# Export onnx, please choose the pt model path by yourself
yolo export model=runs/obb/train/weights/best.pt format=onnx imgsz=320
cd test_yolo11/obb
# The images under test can be replaced by selecting a part from the training set, convert kmodel, please choose the onnx model path by yourself, the generated kmodel is in the same directory level as the onnx model
python to_kmodel.py --target k230 --model ../../runs/obb/train/weights/best.onnx --dataset ../test_obb --input_width 320 --input_height 320 --ptq_option 0
cd ../../
💡 Model conversion script (to_kmodel.py) parameter description:
Parameter Name |
Description |
Explanation |
Type |
|---|---|---|---|
target |
Target platform |
Options are k230/cpu, corresponding to k230 kpu and cpu; |
str |
model |
Model path |
The path of the ONNX model to be converted; |
str |
dataset |
Calibration image set |
The image data used during model conversion, used in the quantization stage, can be replaced by taking a part from the training set |
str |
input_width |
Input width |
The width of the model input |
int |
input_height |
Input height |
The height of the model input |
int |
ptq_option |
Quantization method |
The quantization strategies are |
0/1/2/3/4/5 |
Deploying the Model on k230 Using Linux#
Building the Compilation Environment#
💡 Firmware Introduction: Please refer to the steps in the k230_linux documentation to build the compilation environment and compile the firmware, to ensure that the latest features are supported!
yolo Compilation#
Execute make menuconfig under k230_linux_sdk, select Target packages > canaan package > AI > yolo demo, select Save->OK below, save and exit. In this way, when compiling the firmware, the yolo example can be compiled into the firmware. After flashing the firmware, you can find the compiled yolo.elf executable file and test files under /app/yolo.
If you do not compile when compiling the firmware, you can also choose to compile the yolos example separately. Enter the k230_linux_sdk/buildroot-overlay/package/yolo directory, execute the ./build_app.sh script, and the compiled yolo.elf executable file is in the k230_bin directory. You can copy the k230_bin directory to the development board where the firmware has been flashed.
Model File Copying#
After the development board is flashed with the firmware and powered on, copy the converted kmodel, the compiled executable elf file, test images, and label file (txt format, one line per class) to a directory on the development board.
YOLO Module#
The YOLO module is a deployment code developed based on the YOLO series models, which implements the deployment process of such models. Its supported situations are as follows:
Integrates four models of
YOLOv5, YOLOv8, YOLO11, YOLO26;Supports four types of tasks, of which
YOLOv5supports three types: classify, detect, and segment.YOLOv8, YOLO11, YOLO26support five types of tasks: classify, detect, segment, rotated object detection (obb), and keypoint detection (pose);Supports two inference modes, including image and video stream (video);
Supports two display modes, including
LT9611 (hdmi,1920×1080),ST7701 (lcd screen,800×480);Parameter Description
Parameter Name |
Default Value |
Description |
|---|---|---|
|
yolov8 |
Set the model type, the default value is yolov8, optional values: yolov5/yolov8/yolo11. |
|
detect |
Set the task type, the default value is detect, optional values: classify/detect/segment/obb. |
|
video |
Set the task mode, the default value is video, optional values: image/video |
|
test.jpg |
Set the image path, the default value is test.jpg. |
|
yolov8n.kmodel |
Set the kmodel path, the default value is yolov8n.kmodel. |
|
coco_labels.txt |
Set the label text file path, the default value is coco_labels.txt, each label occupies a line. |
|
0.35 |
Set the confidence threshold, the default value is 0.35. |
|
0.65 |
Set the non-maximum suppression threshold, the default value is 0.65. |
|
0.5 |
Set the mask threshold, the default value is 0.5. |
|
17 |
Set the number of keypoints, the default value is 17 (human skeleton keypoint scenario). |
|
3 |
Set the keypoint dimension of the model, only 2/3 is supported, the default value is 3 (human skeleton keypoint scenario). |
|
0 |
Set the debug mode, the default value is 0, optional values: 0/1, 0 is no debug, 1 is debug printing. |
Deploying the Model to Implement Image Inference#
For image inference, please refer to the following commands, modify the parameters according to the actual situation;
./yolo.elf -model_type yolo11 -task_type obb -task_mode image -image_path test.jpg -kmodel_path yolo11n_pen_obb.kmodel -labels_txt_filepath pen_labels.txt -conf_thres 0.5 -nms_thres 0.45 -debug_mode 0
Deploying the Model to Implement Video Inference#
For video inference, please refer to the following commands, modify the parameters according to the actual situation;
./yolo.elf -model_type yolo11 -task_type obb -task_mode video -kmodel_path yolo11n_pen_obb.kmodel -labels_txt_filepath pen_labels.txt -conf_thres 0.5 -nms_thres 0.45 -debug_mode 0
Deployment Effect#
Select a desktop pen image and use kmodel for rotated object detection. The comparison between the original image and the inference result is shown in the figure below:
Auxiliary Tools#
Online Training Platform#
Introduction to Cloud Training Platform#
The model training feature of the Canaan Developer Community is a training platform opened up to simplify the development process and improve development efficiency. This platform enables users to focus on the implementation of visual scenarios, and more quickly complete the process from data annotation to obtaining the KModel model in the deployment package, and deploy it on K230 and K230D chip development boards equipped with Canaan Technology’s Kendryte® series AIoT chips. Users only need to upload datasets and simply configure parameters to start training.
📌Platform address: Canaan Cloud Training Platform
📌Platform usage documentation reference: Canaan Cloud Training Platform Documentation Tutorial, please pay attention to the format of the dataset!
Supported Tasks Introduction#
There are 7 visual tasks supported for K230 series chips in the cloud training platform. The task introduction is shown in the table below:
💡 Task Introduction:
Task Name |
Task Description |
|---|---|
Image Classification |
Classify images and obtain the category result and score of the image. |
Object Detection |
Detect target objects in images, and provide object location information, category information, and scores. |
Semantic Segmentation |
Segment the target area in the image, cut out different label areas in the image, which is a pixel-level task. |
OCR Detection |
Detect text areas in images and provide location information of the text areas. |
OCR Recognition |
Recognize text content in images. |
Metric Learning |
Train a model that can characterize images, use this model to create a feature library, and through feature comparison, classify new categories without retraining the model, also known as self-learning. |
Multi-label Classification |
Perform multi-category classification on images. Some images may not only belong to a single category. Sky and sea can exist simultaneously, and obtain multi-label classification results of the image. |
Deployment Steps#
Deployment Package Description#
After training, the deployment package for the corresponding training task can be downloaded. After extracting the downloaded deployment zip package, the directory is as follows:
📦 task_name
├── 📁 **_result
│ ├── test_0.jpg
│ ├── test_1.jpg
│ └──...
├── mp_deployment_source
├── **_image_1_2_2.py
├── **_image_1_3.py
├── **_video_1_2_2.py
├── **_video_1_3.py
└── README.pdf
The content is shown in the figure:
Where mp_deployment_source is the code package deployed on the K230 image, which contains the deployment configuration file and the deployed KModel model. Note: The configuration files deploy_config.json and ***.kmodel generated by the platform are the key to deployment on k230, and they can be used in various SDKs.
File Copy#
✅ Firmware Selection: Please refer to the steps in the k230 Linux sdk documentation to set up the compilation environment and compile the firmware to ensure that the latest features are supported!
✅ Firmware Flashing: Flash the firmware according to the type of development board.
✅ Deployment Code: The deployment code is located in k230_linux_sdk/buildroot-overlay/package/cloudplat_deploy_code_linux under the compilation environment. You need to enter this directory and execute ./build.sh to complete the compilation. The compilation product is located in the k230_bin directory. For usage methods, please refer to the README.md in this directory.
Script Execution#
Parameter Configuration:
You can configure parameters in sensor_set.h in common_files. The analysis of parameter configuration is as follows, which is mainly used to configure screen display:
Macro Definition Parameter |
Description |
|---|---|
|
AI inference frame width |
|
AI inference frame height |
|
Number of AI inference frame channels |
Source Code Compilation:
Enter the k230_linux_sdk/buildroot-overlay/package/cloudplat_deploy_code_linux directory
# Enter the directory
cd cloudplat_deploy_code_linux
# Compile files, all task compiled elf files will be obtained in the k230_bin directory
./build.sh
# If you only want to compile the deployment file of a certain task, you can use ./build.sh <task_name>
./build.sh classification
./build.sh detection
...
The compilation product is in the k230_bin directory.
Board Deployment:
Copy the obtained elf files, font files, and the kmodel, deploy_config.json obtained from the Kendryte training platform, as well as test images to a certain directory on the development board, and run the command:
# Classification - Video inference, enter `q` and press Enter to exit video inference
./classification.elf deploy_config.json None 0
# Classification - Image inference
./classification.elf deploy_config.json test.jpg 0
# Detection - Video inference, enter `q` and press Enter to exit video inference
./detection.elf deploy_config.json None 0
# Detection - Image inference
./detection.elf deploy_config.json test.jpg 0
# Semantic Segmentation - Video inference, enter `q` and press Enter to exit video inference
./segmentation.elf deploy_config.json None 0
# Semantic Segmentation - Image inference
./segmentation.elf deploy_config.json test.jpg 0
# OCR Detection - Video inference, enter `q` and press Enter to exit video inference
./ocr_detection.elf deploy_config.json None 0
# OCR Detection - Image inference
./ocr_detection.elf deploy_config.json test.jpg 0
# OCR Recognition - Image inference, this task only supports image inference
./ocr_recognition.elf deploy_config.json test.jpg 0
# OCR - Video inference, enter `q` and press Enter to exit video inference
./ocr.elf ocrdet_deploy_config.json ocrrec_deploy_config.json None 0
# OCR - Image inference
./ocr.elf ocrdet_deploy_config.json ocrrec_deploy_config.json test.jpg 0
# Metric Learning - Video inference, enter `q` and press Enter to exit video inference
./metric_learning.elf deploy_config.json None 0
# Metric Learning - Image inference
./metric_learning.elf deploy_config.json test.jpg 0
# Multi-label Classification - Video inference, enter `q` and press Enter to exit video inference
./multilabel_classification.elf deploy_config.json None 0
# Multi-label Classification - Image inference
./multilabel_classification.elf deploy_config.json test.jpg 0
Deployment Notes#
📢 If the effect is not ideal when deploying the model, first adjust the threshold of the corresponding task and the resolution of the inference image to test whether the result can be improved!
📢 Learn to locate the problem, for example, check the test images in the
**_resultsdirectory of the deployment package. If the images are normal, the problem may be with the deployment code, model conversion, or threshold!📢 Adjust the parameters of model training, such as
epoch,learning_rate, etc., to prevent insufficient training!
AICube#
AICube Introduction#
AICube is an offline training tool provided by Canaan for developers, ensuring data security and enabling visualized local training. The platform supports 8 tasks: image classification, object detection, semantic segmentation, OCR detection, OCR recognition, metric learning, multi-label classification, and anomaly detection. Compared with the online training platform, it allows users to perform model training using local GPU and convert the model into kmodel for deployment on K230.
Environment Preparation and Software Installation#
Before installing AICube, please check whether the following prerequisites are met:
A device with NVIDIA GPU is required, with recommended memory of 8GB or above;
CUDA 11.7 or above and CUDNN have been installed on the computer;
dotnet 7.0 has been installed on the computer, and the installation path has been added to the environment variables;
Computer memory is recommended to be 8GB or above, with at least 20GB of remaining hard disk space;
If your computer meets the above conditions, you can download AICube and unzip it for use. AICube provides installation packages for both ubuntu and windows versions. Because the installation package contains the matching torch training environment, multiple pre-trained models, and example datasets, the installation package is relatively large. Please download it in a suitable network environment. For the download address, see: AICube Download. For usage steps, please refer to the user guide of the corresponding version.
📢 Please select the latest version when downloading.
Supported Tasks Introduction#
There are 8 visual tasks supported by AICube for K230 series chips, as shown in the table below:
💡 Task Introduction:
Task Name |
Task Description |
|---|---|
Image Classification |
Classify images to obtain the category results and scores of the images. |
Object Detection |
Detect target objects in images and provide their position information, category information, and scores. |
Semantic Segmentation |
Segment the target regions in images, cutting out different labeled regions, which is a pixel-level task. |
OCR Detection |
Detect text regions in images and provide position information of text regions. |
OCR Recognition |
Recognize text content in images. |
Metric Learning |
Train a model that can characterize images, use this model to create a feature library, and classify new categories without retraining the model through feature comparison, also known as self-learning. |
Multi-label Classification |
Perform multi-category classification on images. Some images may not only belong to a single category; sky and sea can exist simultaneously, obtaining multi-label classification results. |
Anomaly Detection |
Used to detect anomalous categories in certain products, commonly used in industrial quality inspection and other fields. |
Usage Instructions#
Function Page Introduction#
AI Cube contains 5 function pages. The “Project” page mainly implements project management functions, displaying current projects and recent projects; the “Image” page displays the dataset information of the current project, making it convenient for users to view dataset images; the “Split” page displays split information, statistics of split categories and images of different split sets; the “Train” page implements training parameter configuration, display of training information and training curves; the “Evaluate” page implements model evaluation and display of evaluation information, and can configure necessary deployment parameters to generate a deployment package.
🗂️ Project Page Illustration:
🗂️ Image Page Illustration:
🗂️ Split Page Illustration:
🗂️ Train Page Illustration:
🗂️ Evaluate Page Illustration:
Create Dataset#
Organize the dataset according to the dataset format of different tasks. The corresponding dataset format can be viewed by clicking New Project on the Project Page. We also provide example datasets for different tasks, located in the example_dataset directory; example projects created using these example datasets are located in the example_projects directory.
The corresponding relationships between example datasets and example tasks for different tasks are as follows:
Dataset Name |
Example Task |
Description |
|---|---|---|
vegetable_cls |
Image Classification |
Vegetable classification scenario |
insect |
Object Detection |
Insect detection scenario |
Ocular_lesions |
Semantic Segmentation |
Eye lesion area segmentation scenario |
dataset_td100 |
OCR Detection |
OCR text detection scenario |
ProductionDate |
OCR Recognition |
Production date recognition scenario |
drink |
Metric Learning |
Beverage bottle classification scenario |
multilabel2000 |
Multi-label Classification |
Natural scenery multi-label classification scenario |
bottle |
Anomaly Detection |
Bottle mouth anomaly detection scenario |
You can use the example datasets we provide, or organize your own datasets according to the corresponding task format on the New Project Interface. Most problems encountered in AICube are data-related. We only check the directory structure of the dataset, not the annotation information inside the data. Please handle the data with care.
Create Project#
Go to Project Page—>click the New Project button—>select task type—>import dataset—>select the project storage path—>add the project name—>create project.
The New Project interface is shown in the figure below:
After the project is created, it will automatically jump to the Image Page, where you can view your dataset details. Go to the Split Page, where you can split the dataset according to a custom ratio and view the statistics of the split sets.
Start Training#
Go to the Train Page and configure Model, Data Augmentation, and Training Parameters on the left side.
Common Parameter Analysis:
Platform Parameter Name |
Common Parameter Definition |
Parameter Meaning Analysis |
|---|---|---|
Model |
model |
Network models with different structures, used to implement different tasks; |
Backbone |
model backbone |
The feature extraction part of the network structure in the model, such as the model for detection and segmentation tasks; |
Pretrain |
pretrain |
Whether to load the pre-trained model provided by AICube; |
Pretrain Language |
pretrain language |
Specific task parameter for OCR Recognition, select the sample language for training the pre-trained model; other tasks ignore; |
Model Size |
model size |
n, s, m, l, x, variants of the same model, the difference is the model size, used to balance accuracy and speed; |
Model Width |
model width |
The larger the width, the larger the number of parameters; |
Image Size |
model input size |
Model input resolution, a single value indicates the input is [x,x], double values indicate the input is [x,y]; |
ASPP Dilation Rate |
ASPP dilation rate |
Specific task parameter for semantic segmentation, scales of different atrous convolutions and pooling operations; different dilation rates for atrous convolution can expand the receptive field and obtain broader contextual information; |
Embedding Length |
embedding length |
Specific task parameter for metric learning, the vector length to which samples are vectorized; |
Auto Data Augmentation |
TrivialAugment |
Parameter-free single-image random automatic data augmentation; |
Other Data Augmentation Methods |
— |
Brightness, contrast, saturation, hue, sharpness enhancement, flipping, rotation, random scaling, random cropping, perspective transformation, Gaussian blur, histogram equalization, gray world algorithm, CutOut, Random Erasing, Mask; |
Learning Rate |
learning rate |
Parameters of the optimization algorithm, the adjustment step size for each iteration; |
Epoch |
epoch |
An epoch is the process of the neural network training once using all training samples; |
Training Batch Size |
batchsize |
The number of samples used for each forward and backward propagation; |
Optimizer |
optimizer |
The optimization function used when optimizing the network, such as SGD, Adam, etc.; |
AutoAnchor |
autoanchor |
Anchor frame adaptation in object detection tasks; |
NMS Option |
nms option |
Non-maximum suppression option in object detection tasks to distinguish between intra-class and inter-class; |
Confidence Threshold |
confidience threshold |
Used for filtering prediction box categories, prediction boxes below this threshold will be deleted; |
IOU Threshold |
IOU threshold |
Perform maximum value filtering on multiple overlapping boxes, calculate the scores of all detection boxes, compare them with the highest-scoring detection box in turn, and detection boxes greater than this threshold will be deleted; the Box threshold in OCR detection is similar; |
Auto Mixed Precision |
AMP |
Adopt different data precision for different layers to save video memory and improve computation speed; |
Exponential Moving Average |
EMA |
Smoothing method to prevent the influence of outliers, with weights decaying exponentially over time; |
Early Stopping |
Early Stopping |
A method to increase model generalization and prevent overfitting; |
WarmUp Strategy |
WarmUp |
Operate the learning rate in the initial stage of training to make the model converge faster; |
Multi-Scale Training |
MST |
Implement training on input images of different scales to improve the detection generalization of the detection model for objects of different sizes; |
Loss Function |
loss function |
Used to evaluate the degree of difference between the model prediction and the true value, the smaller the loss, the better the model performance; |
Learning Rate Scheduler |
learning rate scheduler |
Learning rate adjustment strategy, dynamically adjusting the learning rate during training to adapt to the gradient descent process, including StepLR, CosineAnnealingLR, LinearLR, MultiStepLR, etc.; |
Loss Refresh Step |
loss refresh step |
The frequency of drawing the Loss curve on the interface, in batches; |
GPU Index |
gpu index |
Graphics card index; |
After configuring the corresponding parameters according to different tasks, you can click the Enhanced Sample Button to view some example samples after data augmentation; click Learning Rate Curve to view the learning rate changes caused by different learning rate strategies; click the Start Training Button, and the training information will be displayed on the upper right panel, the loss curve and metric curve will be drawn in the middle; the prediction results of the example samples will be iteratively displayed for each epoch in the lower right panel. The interface during training is shown in the figure below:
Model Test#
Go to the Evaluate Page, select the trained model, and then select the test method. The test methods are as follows:
Test Method |
Description |
|---|---|
Test Set Test |
Test and evaluate the test set obtained from the split, and output test metric data; |
Extra Data Test |
Use annotated data in the same format as the training dataset for testing, and output test metric data; |
Image Directory Test |
Only select to use the trained model and parameters to infer all unlabeled samples in the image directory, without test metrics; |
Click the “Start Test” button to perform the test. After the test is completed, check your model performance according to the evaluation metrics; double-click the entry in the test data list to view the large image of the inference result.
Model Deployment#
If the model performance meets your needs, you can configure deployment parameters in the chip adaptation panel, mainly the model’s input resolution and some basic parameters, and click the Deploy Button to generate the deployment package.
After the deployment artifacts are generated, you can find the following files in the root directory of the current project. We mainly use kmodel and the configuration file deploy_config.json:
📦 task_name
├── 📁 cpp_deployment_source
├── 📁 mp_deployment_source
└── README.md
The mp_deployment_source directory contains the resources deployed on the K230 solution, including the Kmodel file and the deployment configuration file! What we actually need is only the configuration file deploy_config.json and the model file ***.kmodel, copy this directory to the development board.
Deployment Steps#
Deployment Package Description#
After training is complete, the deployment artifacts for the corresponding training task can be obtained.
File Copy#
✅ Firmware Selection: Please refer to the steps in the k230 linux sdk documentation to set up the compilation environment and compile the firmware to ensure that the latest features are supported!
✅ Firmware Flashing: Flash the firmware according to the development board type.
✅ Deployment Code: The deployment code is located at k230_linux_sdk/buildroot-overlay/package/cloudplat_deploy_code_linux under the compilation environment. You need to enter this directory and execute ./build.sh to complete the compilation. The compilation output is located in the k230_bin directory. For usage, refer to the README.md in this directory.
Script Execution#
Parameter Configuration:
Source Code Compilation:
You can configure parameters in sensor_set.h in common_files. The description of the parameter configuration is as follows, mainly used to configure screen display:
Macro Definition Parameter |
Description |
|---|---|
|
AI inference frame width |
|
AI inference frame height |
|
AI inference frame channel count |
Source Code Compilation:
Enter the k230_linux_sdk/buildroot-overlay/package/cloudplat_deploy_code_linux directory
# Enter the directory
cd cloudplat_deploy_code_linux
# Compile the files, and all task compiled elf files will be obtained in the k230_bin directory
./build.sh
# If you only want to compile the deployment file for a specific task, you can use ./build.sh <task name>
./build.sh classification
./build.sh detection
...
The compilation output is in the k230_bin directory.
Board Deployment:
Copy the obtained elf file, font file, and kmodel and deploy_config.json obtained from the Kanzhi Training Platform, as well as test images, to a directory on the development board, and run the command:
# Classification - video inference, enter `q` and press Enter to exit video inference
./classification.elf deploy_config.json None 0
# Classification - image inference
./classification.elf deploy_config.json test.jpg 0
# Detection - video inference, enter `q` and press Enter to exit video inference
./detection.elf deploy_config.json None 0
# Detection - image inference
./detection.elf deploy_config.json test.jpg 0
# Semantic segmentation - video inference, enter `q` and press Enter to exit video inference
./segmentation.elf deploy_config.json None 0
# Semantic segmentation - image inference
./segmentation.elf deploy_config.json test.jpg 0
# OCR detection - video inference, enter `q` and press Enter to exit video inference
./ocr_detection.elf deploy_config.json None 0
# OCR detection - image inference
./ocr_detection.elf deploy_config.json test.jpg 0
# OCR recognition - image inference, this task only supports image inference
./ocr_recognition.elf deploy_config.json test.jpg 0
# OCR - video inference, enter `q` and press Enter to exit video inference
./ocr.elf ocrdet_deploy_config.json ocrrec_deploy_config.json None 0
# OCR - image inference
./ocr.elf ocrdet_deploy_config.json ocrrec_deploy_config.json test.jpg 0
# Metric learning - video inference, enter `q` and press Enter to exit video inference
./metric_learning.elf deploy_config.json None 0
# Metric learning - image inference
./metric_learning.elf deploy_config.json test.jpg 0
# Multi-label classification - video inference, enter `q` and press Enter to exit video inference
./multilabel_classification.elf deploy_config.json None 0
# Multi-label classification - image inference
./multilabel_classification.elf deploy_config.json test.jpg 0
Deployment Instructions#
📢 If the effect is not ideal when deploying the model, first adjust the threshold of the corresponding task and the resolution of the inference image to see if the test results can be improved!
📢 Learn to locate the problem, such as checking the AICube model evaluation results. If the image is normal, it may be a problem with the deployment code, model conversion, or threshold. You can choose to adjust the quantization method or adjust the deployment parameters for optimization!
📢 AICube has a large number of training parameters. Users who understand deep learning can adjust training parameters according to possible optimization directions, and adjust the model training parameters to achieve retraining and conversion!
FAQ#
How to find the problem during development?#
📝 First, take different approaches based on the stage and error:
If an error occurs during the model conversion stage, there may be an issue with the conversion code. You need to read the nncase usage instructions and adjust the conversion code;
If the model conversion is successful but the performance is not as expected, you can consider adjusting the threshold, changing the quantization method for model conversion, or adjusting training parameters during training;
If the model conversion is successful but the frame rate is low, you can consider using a lighter model or reducing the model input resolution;
If a deployment error occurs, please check the line number where the deployment code reports the error, refer to the API documentation to find the cause of the error, and adjust the code;
Which operators does nncase support?#
📝 For the onnx operators and tflite operators supported by nncase, see the links: onnx operators support and tflite operators support
Error “ImportError: DLL load failed while importing _nncase” when converting the model#
📝 Please refer to the solution at the following link: ImportError: DLL load failed while importing _nncase
Error “RuntimeError: Failed to initialize hostfxr” when converting the model#
📝 Please install dotnet-sdk-7.0. Do not install dotnet-sdk in an Anaconda virtual environment.
Linux:
sudo apt-get update
sudo apt-get install dotnet-sdk-7.0
If you still have problems after installation, maybe you install dotnet in a virtual enviroment, set the environment variables. dotnet error
export DOTNET_ROOT=/usr/share/dotnet
Windows: Please refer to the official Microsoft website.
What is the difference between the online training platform and AICube?#
📝 The online training platform uses cloud computing power, which requires queuing when resources are tight. Meanwhile, the parameter configuration is relatively simple with one-click training, resulting in lower flexibility. AICube uses local private computing power, with more complex environment and parameter configurations, but offers high flexibility. The purpose of both is to obtain kmodel and configuration files. Use the code in the firmware’s k230_linux_sdk/buildroot-overlay/package/cloudplat_deploy_code_linux to compile and achieve deployment.
Which tasks are supported in the YOLO library?#
📝 YOLOv5 supports three types of tasks: classification, detection, and segmentation. YOLOv8 and YOLO11 support four types of tasks: classification, detection, segmentation, and oriented object detection.
How to get support?#
📝 If you encounter problems during development, you can go to the Canaan Developer Community Q&A Forum to post your questions. Forum address: Canaan Q&A Forum.
Appendix#
API#
K230 Linux SDK documentation can be found at the following link: K230 Linux SDK Documentation
KTS#
K230_training_scripts（KTS） is an implemented end-to-end training process, but the code of this project is developed based on dual-system C++. You can use this tool to obtain kmodel. Project address: K230_training_scripts.