Note

This is the documentation for the latest development branch and may refer to features that are not available in released versions. If you are looking for the documentation for a specific release, use the drop-down menu on the left and select the desired version.

AI Demo User Guide

AI Demo

Overview

The K230 AI Demo integrates applications such as face, human body, hand, and license plate, encompassing a variety of functions including classification, detection, segmentation, recognition, tracking, and monocular ranging. It provides customers with a reference on how to develop AI-related applications using K230. The above-mentioned applications are used to verify the capabilities of K230 and enrich application scenarios. Actual application scenarios require targeted optimization to achieve better results. Reference optimization directions include adjusting thresholds, code optimization, quantization optimization, model optimization, training data optimization, etc.

Supported Development Boards

• CanMV-K230-V1.1 / CanMV-K230-V3.0 / 01Studio CanMV K230/ Bpi-CanMV-K230D-Zero/ lckfb-K230 / dongshanpi-k230

Source Code Description

File Tree

The source code path is located at k230_linux_sdk/buildroot-overlay/package/ai_demo, and the directory structure is as follows:

# AI Demo subdirectories (eg: bytetrack, face_detection, etc.) contain detailed Demo documentation
.
├── anomaly_det
├── bytetrack
├── common
├── crosswalk_detect
├── demo_mix
├── dynamic_gesture
├── eye_gaze
├── face_alignment
├── face_detection
├── face_emotion
├── face_gender
├── face_glasses
├── face_landmark
├── face_mask
├── face_mesh
├── face_parse
├── face_pose
├── face_verification
├── falldown_detect
├── finger_guessing
├── fitness
├── head_detection
├── helmet_detect
├── licence_det
├── licence_det_rec
├── llamac
├── object_detect_yolov8n
├── ocr
├── person_attr
├── person_detect
├── person_distance
├── pose_detect
├── pphumanseg
├── puzzle_game
├── segment_yolov8n
├── shell
├── smoke_detect
├── space_resize
├── sq_hand_det
├── sq_handkp_class
├── sq_handkp_det
├── sq_handkp_flower
├── sq_handkp_ocr
├── sq_handreco
├── traffic_light_detect
├── vehicle_attr
├── virtual_keyboard
├── yolop_lane_seg
├── CMakeLists.txt
├── ai_demo.mk
├── build.sh
└── Config.in

The files under the common_files directory are shared by all Demos. The file structure of this directory is as follows:

.
├── ai_base.cc        # Model inference wrapper class implementation, encapsulating the basic operations of nncase, including kmodel loading, setting inputs, getting outputs. Subsequent application development only needs to focus on pre- and post-processing of the model
├── ai_base.h         # Model inference wrapper class header file, defining the basic interfaces for model inference
├── utils.cc          # Utility methods and utility classes, providing implementations for getting color palette, saving images, and different preprocessing methods
├── utils.h           # Utility methods and utility classes header file, defining the interfaces for utility methods and utility classes
├── scoped_timing.hpp # Time measurement class, used to measure code execution time
├── setting.h         # Configuration header file, mainly implementing the setting of AI inference output image resolution
├── sensor_buf_manager.cc # Camera buffer data acquisition management implementation method, from which tensors used for inference can be obtained
├── sensor_buf_manager.h  # Camera buffer data acquisition management header file
├── ai_demo_cml_common  # When compiling with buildroot, the common compilation settings for all aidemos
├── ai_demo_mk_common   # When compiling with buildroot, the template for compilation configuration specific to all aidemos, related to the application directory name
├── ai_demo_commonConfig.cmake.in # CMake package configuration file. After the ai_demo_common library is compiled and installed, other applications can find this library through find_package
├── CMakeLists.txt    # ai_demo_common compilation configuration file. The ai_demo_common part is compiled into an independent library for other aidemos to use, and is a dependency for compiling other ai_demos
├── common.mk         # common part buildroot custom build fragment, implementing automatic download (or local sync) of model packages, setting up dependencies, and packaging/installing via CMake
└── Config.in         # Buildroot package configuration item

The kmodel and related dependency paths will be automatically downloaded to the current directory during compilation, providing the kmodel, test images, and other necessary files required for ai_demo to run. During the ai_demo compilation process, the build.sh script will automatically copy them to the output directory according to the demo name.

Demo Description

Demo Subdirectory	Scenario	Description	Link
anomaly_det	Anomaly Detection	The model provided by the anomaly detection example is trained using the patchcore anomaly detection method, and can identify whether there is an anomaly at the glass bottle mouth from the input image. Anomaly detection is usually applied in industrial image inspection, medical image analysis, security monitoring and other fields.	anomaly_det
bytetrack	Multi-Object Tracking	The ByteTrack multi-object tracking example uses YOLOv5 as the object detection algorithm, applies the Kalman filter algorithm for bounding box prediction, and applies the Hungarian algorithm for matching between targets and trajectories.	bytetrack
crosswalk_detect	Crosswalk Detection	The crosswalk detection uses the YOLOV5 network. This application detects crosswalks in images or videos, and can be used in scenarios such as assisted driving.	crosswalk_detect
demo_mix	Demo Collection	The demo collection integrates examples such as hand keypoint detection, gesture recognition, dynamic gesture recognition, face pose estimation, and object tracking in k230 to implement gesture-controlled application switching. When the “1” gesture is shown, it enters dynamic gesture recognition; when the “2” gesture is shown, it enters face pose estimation; when the “3” gesture is shown, it enters automatic face tracking. When the “love” gesture is shown, it exits the current task and enters the switching state.	demo_mix
dynamic_gesture	Visual Dynamic Gesture Recognition	Visual dynamic gesture recognition can recognize five actions: waving up, down, left, right, and five-finger pinching, used in mid-air operation control scenarios. The palm detection adopts the yolov5 network structure, with 1.0-mobilenetV2 as the backbone. The palm keypoint detection adopts the resnet50 network structure. The dynamic gesture recognition adopts the tsm structure, with mobilenetV2 as the backbone.	dynamic_gesture
eye_gaze	Gaze Estimation	The gaze estimation example predicts where a person is looking based on the face. For video frames or images, face detection is first performed, and then gaze estimation is performed on each face to predict the gaze vector, which is displayed on the screen in the form of an arrow. This application uses the retina-face network for face detection and L2CS-Net for gaze estimation. Gaze estimation can be applied in the field of automotive safety.	eye_gaze
face_alignment	Face Alignment	The face alignment example can obtain a depth map or a normalized projection coordinate coding map of each face in an image or video. The face detection adopts the retina-face network structure, with 0.25-mobilenet as the backbone. The face alignment network is implemented based on 3DDFA (3D Dense Face Alignment).	face_alignment
face_detection	Face Detection	The face detection example can obtain the detection box of each face in an image or video, as well as the positions of five key points of each face: left eye, right eye, nose tip, left mouth corner, and right mouth corner. The face detection adopts the retina-face network structure, with 0.25-mobilenet as the backbone.	face_detection
face_emotion	Facial Expression Recognition	The facial expression recognition uses two models to implement the function of recognizing the expression of each person in images/videos. The recognizable expression categories include Neutral, Happiness, Sadness, Anger, Disgust, Fear, and Surprise. The face detection uses the retina-face network structure; the expression classification uses mobilenet as the backbone for classification to obtain the person’s expression.	face_emotion
face_gender	Gender Classification	The face gender classification example uses two models to implement the function of judging the gender of each person in images/videos. The gender of each person is represented by M or F, where M stands for Male and F stands for Female. The face detection uses the retina-face network structure; the gender classification uses EfficientNetB3 as the backbone for classification to obtain the person’s gender.	face_gender
face_glasses	Glasses Wearing Classification	The glasses wearing classification example uses two models to implement the function of judging whether each person in images/videos is wearing glasses. The face detection model uses the retina-face network structure; the face glasses classification model uses SqueezeNet-1.1 as the backbone to judge the glasses wearing situation for each face box.	face_glasses
face_landmark	Dense Face Keypoints	The dense face keypoint detection application uses two models to implement the function of detecting 106 keypoints for each face in images/videos, and draws the contours of the face and facial features according to the 106 keypoints. Different contours are represented in different colors. The face detection uses the retina-face network structure; the dense keypoint detection uses 0.5-mobilenet as the backbone to detect 106 keypoints for each face. The 106 keypoints include the areas of the face, mouth, eyes, nose, and eyebrows.	face_landmark
face_mask	Mask Wearing Classification	The mask wearing classification application uses two models to implement the function of judging whether each person in images/videos is wearing a mask. In application scenarios where masks need to be worn, if someone is found not wearing a mask, relevant reminders can be made. The face detection model uses the retina-face network structure; the face mask classification model uses mobilenet-v2 as the backbone to judge the mask wearing situation for each face box.	face_mask
face_mesh	3D Face Mesh	The 3D face mesh can obtain the 3D mesh structure of each face in an image or video. The face detection adopts the retina-face network structure, with 0.25-mobilenet as the backbone. The face alignment network is implemented based on 3DDFA (3D Dense Face Alignment).	face_mesh
face_parse	Face Segmentation	The face segmentation example uses two models to implement the function of segmenting each face in images/videos. The face segmentation includes distinguishing the eyes, nose, mouth and other parts of the face at the pixel level, with different regions represented in different colors. The face detection adopts the retina-face network structure, and the face part segmentation uses the DeepNetV3 network structure, with mobilenet-1.0 as the backbone.	face_parse
face_pose	Face Pose Estimation	The face pose estimation uses two models to implement the function of estimating the angle of the face orientation of each person in images/videos. The face orientation is generally represented by Euler angles (roll/yaw/pitch), where roll represents the degree of the face shaking left and right; yaw represents the degree of the face rotating left and right; pitch represents the degree of the face looking down and up. The face detection adopts the retina-face model, and the face orientation estimation fits 98 2D keypoints.	face_pose
face_verification	Face Identity Verification	Face identity verification is an identity verification technology based on facial biological features, aiming to confirm whether an individual is the identity they claim to be. This technology verifies the identity by analyzing and comparing the user’s facial features, usually by comparing two images in a face verification system to determine whether the faces in the two images belong to the same person. The face detection adopts the retina-face model, and the face feature extraction uses ResNet50, outputting 512-dimensional features.	face_verification
falldown_detect	Fall Detection	Fall detection can detect the fall state of a person in an image or video. This example uses the yolov5n model.	falldown_detect
finger_guessing	Finger Guessing Game	The finger guessing game example distinguishes rock, chicken (fist), scissors, and cloth through hand gesture recognition, including two models: palm detection and hand 21-keypoint recognition. The gesture category is determined by the position constraints of the 21 keypoints. The palm detection part adopts the yolov5 network structure, with 1.0-mobilenetV2 as the backbone. The palm keypoint detection part adopts the resnet50 network structure.	finger_guessing
fitness	Squat-Up Counting	The squat-up counting example implements the function of counting the squat-up movements of a person in a video, suitable for scenarios such as fitness state detection. It uses the yolov8n-pose model.	fitness
head_detection	Head Detection and Counting	The head detection and counting example implements the function of obtaining the coordinates and quantity of heads appearing in an image or video. It uses the yolov8 model.	head_detection
helmet_detect	Helmet Detection	The helmet detection example implements the detection of whether a person appearing in an image or video is wearing a helmet, suitable for safety prevention scenarios in the construction and manufacturing industries. It uses the yolov5 model.	helmet_detect
kws	Keyword Wake-Up	Keyword wake-up detects whether the audio stream contains the keywords set during training through the audio recognition model. If the corresponding keyword is detected, a voice response is given. The model provided in this example is trained using WeNet. The positive and negative samples use the “xiaonan” audio collected on the k230 development board and the open-source dataset speech_commands respectively.	kws
licence_det	License Plate Detection	License plate detection can detect the license plates appearing in images or videos. The license plate detection adopts the retinanet network structure.	licence_det
licence_det_rec	License Plate Recognition	License plate recognition can recognize the position and license plate information of license plates appearing in images or videos. The license plate detection adopts the retinanet network structure, and the license plate recognition adopts the RLNet network structure with MobileNetV3 as the backbone.	licence_det_rec
object_detect_yolov8n	YOLOV8 Multi-Object Detection	The YOLOv8 multi-object detection example implements 80-category detection of the COCO dataset. It uses the yolov8n model.	object_detect_yolov8n
ocr	OCR Detection + Recognition	The OCR recognition example can detect the text position in an image or video as well as the corresponding text content. The OCR recognition task adopts the CRNN network structure, and the OCR detection task adopts the DBnet network structure.	ocr
person_attr	Human Attributes	Human attribute detection can identify the human position coordinates, gender, age, whether wearing glasses, and whether holding objects in an image or video. Human detection is implemented using the YOLOv5 model, and human attributes are implemented using the PULC human model.	person_attr
person_detect	Human Detection	Human detection can detect the human position coordinate information in an image or video, and mark it with a detection box. This example uses the yolov5 model.	person_detect
person_distance	Pedestrian Distance Measurement	Pedestrian distance measurement is to detect pedestrians through pedestrian detection, and then estimate the target distance by the size of the detection box in the image. The pedestrian detection adopts the yolov5n network structure. Using this application, the detection box of each pedestrian in an image or video and the estimated distance can be obtained. This technology can be applied in vehicle assisted driving systems, intelligent transportation and other fields. This application needs to adjust the calculation data according to the camera, and the existing example may not recognize accurately.	person_distance
pose_detect	Human Keypoint Detection	The output of the human keypoint detection model is a set of keypoints (17) representing the human body in an image or video, as well as the confidence score of each point. Lines of different colors are used to connect the keypoints into the shape of a human body. This example uses the yolov8n-pose model.	pose_detect
pphumanseg	Portrait Segmentation	Portrait segmentation refers to identifying the contour range of the human body in an image or video, separating it from the background, and returning the segmented binary map, grayscale map, foreground portrait map, etc., to achieve the replacement and synthesis of background images. It can be applied in scenarios such as portrait matting, photo synthesis, portrait special effects, and background special effects, greatly improving the efficiency of image and video tools. This example uses the pphumanseg model.	pphumanseg
puzzle_game	Puzzle Game	The puzzle game can obtain the positions of 21 bone keypoints of each palm in an image or video. It can also implement the puzzle game function: spread the thumb and middle finger, place the midpoint to a non-empty space next to the empty space, fit the two fingers, and the current non-empty space will move into the empty space. In the example, the palm detection adopts the yolov5 network structure, with 1.0-mobilenetV2 as the backbone. The palm keypoint detection adopts the resnet50 network structure.	puzzle_game
segment_yolov8n	YOLOV8 Multi-Object Segmentation	The YOLOv8 multi-object segmentation detection example implements 80-category segmentation masks of the COCO dataset. It uses the yolov8n-seg model.	segment_yolov8n
smoke_detect	Smoking Detection	Smoking detection performs real-time monitoring and recognition of smoking behaviors in images or videos. This example uses the yolov5 model.	smoke_detect
space_resize	Mid-Air Gesture Zoom	Mid-air gesture zoom can obtain the positions of 21 bone keypoints of each palm in an image or video, and we use the thumb and middle finger to implement mid-air image zooming. The palm detection adopts the yolov5 network structure, with 1.0-mobilenetV2 as the backbone. The palm keypoint detection adopts the resnet50 network structure.	space_resize
sq_hand_det	Palm Detection	Palm detection can obtain the detection box of each palm in an image or video. The palm detection adopts the yolov5 network structure, with 1.0-mobilenetV2 as the backbone.	sq_hand_det
sq_handkp_class	Palm Keypoint Gesture Classification	Palm keypoint gesture classification can obtain the positions of 21 bone keypoints of each palm in an image or video, and obtain static gestures according to the two-dimensional constraints of the keypoint positions. It supports a total of 9 gestures: fist, five-finger spread, one gesture, yeah gesture, three gesture, eight gesture, six gesture, thumbs up, and thumb-index-pinky spread. In this example, the palm detection adopts the yolov5 network structure, with 1.0-mobilenetV2 as the backbone. The palm keypoint detection adopts the resnet50 network structure.	sq_handkp_class
sq_handkp_det	Palm Keypoint Detection	The palm keypoint detection example can obtain the positions of 21 bone keypoints of each palm in an image or video. The palm detection adopts the yolov5 network structure, with 1.0-mobilenetV2 as the backbone. The palm keypoint detection adopts the resnet50 network structure.	sq_handkp_det
sq_handkp_flower	Fingertip Region Flower Classification	Fingertip region flower recognition can obtain the flower category within the area enclosed by the index fingertips of two palms in an image or video. It can support the recognition of 102 kinds of flowers. In this example, the palm detection adopts the yolov5 network structure, with 1.0-mobilenetV2 as the backbone. The palm keypoint detection adopts the resnet50 network structure. The flower classification uses 1.0-mobilenetV2 as the backbone.	sq_handkp_flower
sq_handkp_ocr	Finger Region OCR Recognition	Finger region OCR recognition can recognize the text within the upper left area of the index finger of each palm in an image or video. The palm detection adopts the yolov5 network structure, with 1.0-mobilenetV2 as the backbone. The palm keypoint detection adopts the resnet50 network structure. The text detection adopts the retinanet network structure, and the text recognition adopts the RLnet network structure with MobileNetV3 as the backbone.	sq_handkp_ocr
sq_handreco	Gesture Recognition	Gesture recognition can obtain the category of each gesture in an image or video. It only supports three gestures: five-finger spread, eight gesture, and yeah gesture. In this example, the palm detection adopts the yolov5 network structure, with 1.0-mobilenetV2 as the backbone. The gesture recognition uses 1.0-mobilenetV2 as the backbone.	sq_handreco
traffic_light_detect	Traffic Light Detection	It can detect the red, green, and yellow traffic lights in an image or video. This example uses the yolov5 model.	traffic_light_detect
tts_zh	Chinese Text-to-Speech	Chinese text-to-speech (text to chinese speech, tts_zh) uses three models. The user inputs text three times by default, and generates the wav file corresponding to the text. In this example, the FastSpeech2 model is split into two models: Encoder+Variance Adaptor is fastspeech1, and Decoder is fastspeech2. The vocoder selects hifigan. The duration features are added after fastspeech1.	tts_zh
vehicle_attr	Vehicle Attribute Recognition	Vehicle attribute recognition can identify each vehicle in an image or video, and return the position coordinates, vehicle type, and body color of the vehicle. This example uses the yolov5 network structure to implement vehicle detection, and uses the PULC model for vehicle attribute detection.	vehicle_attr
virtual_keyboard	Mid-Air Virtual Keyboard	The mid-air virtual keyboard can use the virtual keyboard on the screen to output characters. Pinching the thumb and index finger is the input action. In this example, the palm detection adopts the yolov5 network structure, with 1.0-mobilenetV2 as the backbone. The palm keypoint detection adopts the resnet50 network structure.	virtual_keyboard
yolop_lane_seg	Road Lane Line Segmentation	Road lane line segmentation can implement road surface segmentation in an image or video, that is, detect lane lines and drivable areas, and distinguish them with colors. This example uses the yolop model.	yolop_lane_seg

Compile and Run Programs

Setting Up the Build Environment

If you have already compiled the firmware, or choose not to use self-compiled firmware, you can skip this step.

• Download sdk source code

Refer to the following commands to download the sdk code

git clone git@github.com:kendryte/k230_linux_sdk.git
# git clone git@gitee.com:kendryte/k230_linux_sdk.git
cd k230_linux_sdk

The repository address on github is kendryte/k230_linux_sdk.git

The repository address on gitee is https://gitee.com/kendryte/k230_linux_sdk.git

• Install cross toolchain

Download the Xuantie-900-gcc-linux-6.6.0-glibc-x86_64-V3.0.2.tar.gz file (Download link 1: https://www.xrvm.cn/community/download?id=4433353576298909696 , Download link 2: https://kendryte-download.canaan-creative.com/k230/downloads/dl/gcc/Xuantie-900-gcc-linux-6.6.0-glibc-x86_64-V3.0.2-20250410.tar.gz), and extract it to the /opt/toolchain directory. Refer to the following commands:

mkdir -p /opt/toolchain;
tar -zxvf Xuantie-900-gcc-linux-6.6.0-glibc-x86_64-V3.0.2.tar.gz -C /opt/toolchain;

Install the new 32-bit cross toolchain (Download link: ruyisdk/riscv-gnu-toolchain-rv64ilp32) (optional, only required for k230d_canmv_ilp32_defconfig), refer to the following commands:

wget -c ruyisdk/riscv-gnu-toolchain-rv64ilp32 ;

mkdir -p /opt/toolchain/riscv64ilp32-elf-ubuntu-22.04-gcc-nightly-2024.06.25/ ;

tar -xvf riscv64ilp32-elf-ubuntu-22.04-gcc-nightly-2024.06.25-nightly.tar.gz -C /opt/toolchain/riscv64ilp32-elf-ubuntu-22.04-gcc-nightly-2024.06.25/

• Install dependencies

An ubuntu22.04 or ubuntu 24.04 system is required to install the following software (reference installation commands)

sudo apt-get install -y wget git sed make binutils build-essential diffutils gcc  g++ bash patch gzip bzip2 perl tar cpio unzip rsync file bc findutils wget libncurses-dev python3 libssl-dev  gawk cmake bison flex bash-completion  parted curl xz-utils

Dependency software packages can be found in the tools/docker/Dockerfile file. For instructions on building and entering the docker environment, refer to the following commands:

docker build -f tools/docker/Dockerfile -t wjx/d tools/docker #Build

docker run -it -h k230 -e uid=$(id -u) -e gid=$(id -g) -e user=${USER} -v ${HOME}:${HOME} -w $(pwd) wjx/d:latest #Use

• Compile firmware

Refer to the following commands to compile

make CONF=k230_canmv_01studio_defconfig

k230_canmv_01studio_defconfig is an example and needs to be replaced with the correct configuration file, for example, replace it with k230_canmv_defconfig

All configuration files supported by sdk can be found in the buildroot-overlay/configs directory

The meaning of make CONF=k230d_canmv_defconfig is to use the k230d_canmv_defconfig configuration file

• Compile output file

output/k230_canmv_01studio_defconfig/images/sysimage-sdcard.img.gz

This is the file downloaded from the Canaan official website. You need to extract it before flashing. For flashing methods, see below

k230_canmv_01studio_defconfig is an example. Please replace it with the correct name according to the compiled configuration file

Compile Firmware with AI Demo

In the K230 Linux SDK root directory, use make menuconfig to configure Target packages -> canaan package -> AI > AI demo to enable the corresponding aidemo. Save the configuration when exiting, and return to the root directory to re-execute the make command. After the firmware is generated, flash it. Connect to the serial port and you can see the compiled-in applications in the /root/app directory.

Compile AI Demo Separately

• Compile a single ai demo

cd buildroot-overlay/package/ai_demo
#Compile a single ai_demo (take face_detection as an example)
./build.sh face_detection

Build artifacts:

k230_bin/
├── face_detection
│   ├── 1024x624.jpg
│   ├── face_detect_image.sh
│   ├── face_detection_320.kmodel
│   ├── face_detection_640.kmodel
│   ├── face_detection.elf
│   └── face_detect_isp.sh

Copy the entire k230_bin/ folder to the board, and execute the sh script on the board to run the corresponding AI demo

#Enter the development board /app directory
scp -r username@ip:/xxx/k230_linux_sdk/buildroot-overlay/package/ai_demo/k230_bin .

#Execute the corresponding script to run face detection
#For detailed instructions on face detection, please refer to k230_linux_sdk/buildroot-overlay/package/ai_demo/face_detection/README.md
./face_detect_isp.sh

• Compile all AI Demos

cd buildroot-overlay/package/ai_demo
./build.sh

Generate the following files:

k230_bin/
......
├── face_detection
│   ├── 1024x624.jpg
│   ├── face_detect_image.sh
│   ├── face_detection_320.kmodel
│   ├── face_detection_640.kmodel
│   ├── face_detection.elf
│   └── face_detect_isp.sh
......
└── llamac
    ├── llama.bin
    ├── llama_build.sh
    ├── llama_run
    └── tokenizer.bin
......