# YOLO Application Guide

## Overview

K230 has encapsulated YOLOv5, YOLOv8, YOLO11, and YOLO26 in the YOLO model, supporting five types of tasks: classification (classify), detection (detect), segmentation (segment), rotated object detection (obb), and keypoint detection (pose). Users can flexibly use parameters to invoke different models and modify the input mode (video/image) according to scenario requirements; they can also freely modify the image resolution captured by the camera to tune the YOLO deployment.

## Model Conversion

For the process of training models and converting kmodel, please refer to: [YOLO Battle](https://www.kendryte.com/k230_canmv/zh/main/zh/example/ai/yolo_battle.html), **the models converted according to the linked documentation are all available within this documentation**.

## YOLO Support

The YOLO code is located in the `k230_linux_sdk/buildroot-overlay/package/yolo` directory. The code encapsulates the inference part of the YOLO model. Users only need to call the interface to obtain inference frames and feed them to the YOLO series models.

### Code Structure

The existing code structure is as follows:

``` shell
|yolo
├── src
│    ├── ai_base.cc
│    ├── ai_base.h
│    ├── main.cc
│    ├── scoped_timing.h
│    ├── sensor_buf_manager.cc
│    ├── sensor_buf_manager.h
│    ├── sensor_set.h
│    ├── utils.h
│    ├── utils.cc
│    ├── utils.h
│    ├── yolo26.cc
│    ├── yolo26.h
│    ├── yolo11.cc
│    ├── yolo11.h
│    ├── yolov5.cc
│    ├── yolov5.h
│    ├── yolov8.cc
│    └── yolov8.h
├── utils
├── CMakeLists.txt
└── build_app.sh
```

### Code Description

The code files are described below:

| File Name                    |         Function     |
|--------------------------- | -----------------|
|ai_base.h|Provides interfaces used during model inference|
|ai_bash.cc|Provides the interface implementation of the model inference methods defined in ai_bash.h|
|scoped_timing.h|Provides timing utilities to help with development and debugging|
|sensor_buf_manager.h|Interface for managing tensors in AI inference channels|
|sensor_buf_manager.cc|Implementation of the interfaces encapsulated in sensor_buf_manager.h|
|sensor_set.h|Parameter definitions for AI inference channels, setting the width and height of camera output images|
|utils.h|Provides common utility function interfaces such as binary data reading, image saving, and preprocessing configuration|
|utils.cc|Provides the implementation of utility functions defined in utils.h|
|yolo26.h|Provides interfaces for initialization, preprocessing, inference, postprocessing, and result drawing of the yolo26 model|
|yolo26.cc|Provides the interface implementation of the yolo26 model|
|yolo11.h|Provides interfaces for initialization, preprocessing, inference, postprocessing, and result drawing of the yolo11 model|
|yolo11.cc|Provides the interface implementation of the yolo11 model|
|yolov5.h|Provides interfaces for initialization, preprocessing, inference, postprocessing, and result drawing of the yolov5 model|
|yolov5.cc|Provides the interface implementation of the yolov5 model|
|yolov8.h|Provides interfaces for initialization, preprocessing, inference, postprocessing, and result drawing of the yolov8 model|
|yolov8.cc|Provides the interface implementation of the yolov8 model|
|main.cc|Main function implementation, which performs inference for different tasks (classify/detect/segment/obb/pose), different models (yolov5/yolov8/yolo11/yolo26), and different modes (image/video) of the YOLO series models|

Among them, the `utils` directory contains sample models and images used for board deployment, and `build_app.sh` is the compilation script.

## Application Steps

### Firmware Compilation

Execute `make menuconfig` under `k230_linux_sdk`, select `Target packages > canaan package > AI > yolo demo`, choose `Save`->`OK` below, save and exit. This way, when compiling the firmware, the yolo example can be compiled into the firmware. After flashing the firmware, you can find the compiled `yolo.elf` executable file and test files under `/app/yolo`.

### Code Compilation

If you do not compile during firmware compilation, you can also choose to compile the yolo example separately. Enter the `k230_linux_sdk/buildroot-overlay/package/yolo` directory, execute the `./build_app.sh` script, and the `yolo.elf` executable file generated by the compilation is in the `k230_bin` directory. You can copy the `k230_bin` directory to a development board that has already been flashed with the firmware.

### Running Parameters

The following describes the parameters when running on the board:

| Parameter Name        | Default Value   | Description                                                                 |
|-----------------------|-----------------|-----------------------------------------------------------------------------|
| `-model_type`         | yolov8          | Set the model type, the default value is yolov8, optional values: yolov5/yolov8/yolo11.          |
| `-task_type`          | detect          | Set the task type, the default value is detect, optional values: classify/detect/segment/obb.       |
| `-task_mode`          | video           | Set the task mode, the default value is video, optional values: image/video |
| `-image_path`         | test.jpg        | Set the image path, the default value is test.jpg.                                      |
| `-kmodel_path`        | yolov8n.kmodel  | Set the kmodel path, the default value is yolov8n.kmodel.                            |
| `-labels_txt_filepath`| coco_labels.txt | Set the label text file path, the default value is coco_labels.txt, each label occupies a separate line.                        |
| `-conf_thres`         | 0.35            | Set the confidence threshold, the default value is 0.35.                                        |
| `-nms_thres`          | 0.65            | Set the non-maximum suppression threshold, the default value is 0.65.                                  |
| `-mask_thres`         | 0.5             | Set the mask threshold, the default value is 0.5.                                           |
| `-kp_num`             | 17              | Set the number of keypoints, the default value is 17 (human skeleton keypoint scenario).                       |
| `-kp_dim`             | 3               | Set the model keypoint dimension, only 2/3 are supported, the default value is 3 (human skeleton keypoint scenario).       |
| `-debug_mode`         | 0               | Set the debug mode, the default value is 0, optional values: 0/1, 0 means no debug, 1 means debug print.                                |

### Running Example

Insert the flashed `TF` card into the `K230` development board, power it on, connect the development board using a serial port, and run the video inference command and image inference command respectively to see the inference results. You can execute `yolo.elf -help` to view the parameter configuration.

- Video Inference

```shell
#You can execute: ./video_run.sh
./yolo.elf -model_type yolov8 -task_type detect -task_mode video -kmodel_path yolov8n.kmodel -labels_txt_filepath coco_labels.txt -conf_thres 0.35 -nms_thres 0.65 -mask_thres 0.5 -debug_mode 0
```

- Image Inference

```shell
#You can execute: ./image_run.sh
./yolo.elf -model_type yolov8 -task_type detect -task_mode image -image_path test.jpg -kmodel_path yolov8n.kmodel -labels_txt_filepath coco_labels.txt -conf_thres 0.35 -nms_thres 0.65 -mask_thres 0.5 -debug_mode 0
```

During the deployment process, you can replace the model, task type, task mode, threshold parameters, etc. as needed, where **each label in the label text file occupies a separate line**.

## Notes

- Currently supported models are `yolov5`, `yolov8`, `yolo11` and `yolo26`.
- Currently supported task types: `yolov5` supports three tasks: `classify`, `detect`, and `segment`; `yolov8`, `yolo11`, and `yolo26` support five tasks: `classify`, `detect`, `segment`, `obb`, and `pose`.
- Currently supported task modes are `video` and `image`.
- During the tuning process, you can first adjust the threshold for tuning, and then modify the model quantization method and input resolution for tuning.
- If the AI frame resolution is set to the same value as the model input resolution, more optimized inference speed can be achieved. The AI frame resolution is defined in `sensor_set.h`.