`AIBase` Module API Manual#

Overview#

This manual is intended to guide developers in building a complete AI inference pipeline when developing AI Demos with MicroPython, implementing the full workflow from loading the model, preprocessing, inference, obtaining model output, to post-processing. This module encapsulates the inference process of a single model, wrapping preprocessing, inference, and output retrieval operations within the framework. Developers only need to focus on preprocessing configuration and post-processing when developing AI applications.

API Introduction#

init#

Description

AIBase constructor. Initializes the AI program to obtain the image resolution and display-related parameters.

Syntax

from libs.AIBase import AIBase

aibase=AIBase(kmodel_path="**.kmodel", model_input_size=[224,224], rgb888p_size=[1280,720], debug_mode=0)

Parameters

Parameter Name	Description	Input / Output	Notes
kmodel_path	kmodel model path	Input	Required
model_input_size	Model input resolution, list type, includes width and height, e.g. [512,512]	Input	Required
rgb888p_size	Input image resolution of the AI program, list type, includes width and height, e.g. [1280,720]	Input	Required
debug_mode	Debug timing mode, int type, 0 for no timing, 1 for timing	Input	Default is 0

Return Value

Return Value	Description	Notes
AIBase	AIBase instance	This class is generally inherited as a parent class by subclasses, and AI Demo classes for different scenarios are written based on this class

get_kmodel_inputs_num#

Description

Gets the number of inputs of the kmodel.

Syntax

aibase.get_kmodel_inputs_num()

Parameters

Parameter Name	Description	Input / Output	Notes
None

Return Value

Return Value	Description
inputs_num	Number of kmodel inputs

get_kmodel_outputs_num#

Description

Gets the number of outputs of the kmodel.

Syntax

aibase.get_kmodel_outputs_num()

Parameters

Parameter Name	Description	Input / Output	Notes
None

Return Value

Return Value	Description
outputs_num	Number of kmodel outputs

preprocess#

Description

Calls the ai2d preprocessing method defined in the subclass. If the preprocessing method cannot be implemented with ai2d or no preprocessing is needed, it needs to be overridden in the subclass.

Syntax

aibase.preprocess(input_np)

Parameters

Parameter Name	Description	Input / Output	Notes
input_np	Input data in `ulab.numpy.ndarray` format; the shape needs to match the configuration in `Ai2d.build`	Input	Required

Return Value

Return Value	Description
input_tensors	A list of length 1, whose element is the output tensor after ai2d preprocessing. This tensor is held by the `Ai2d` instance and reused across frames; the caller should not call `release()` on it themselves

inference#

Description

Performs inference using the kmodel and obtains the model output.

Syntax

results=aibase.inference(tensors)

Parameters

Parameter Name	Description	Input / Output	Notes
tensors	Input tensor list, arranged in the order of model inputs, list type	Input	Required. Ownership of the tensors still belongs to the caller; this method will not release them

Return Value

Return Value	Description
results	List of kmodel inference outputs; each output is in `ulab.numpy.ndarray` format

postprocess#

Description

Post-processing interface definition. Since different AI applications require different post-processing steps, this method needs to be overridden in the subclass.

Syntax

aibase.postprocess(results)

Parameters

Parameter Name	Description	Input / Output	Notes
results	List of model outputs in `ulab.numpy.ndarray`, arranged in output index order	Input	Required. The base class does no parsing; it is overridden by subclasses

Return Value

Return Value	Description
None	The base class implementation directly returns `None`; after being overridden by a subclass, it should return the post-processing result of the current task, which will be passed through as-is by `run()`

run#

Description

The entire process of single-model inference, including preprocessing, inference, obtaining outputs, and post-processing. Returns the post-processing output, which is used to draw results on the Display.

Syntax

res=aibase.run(input_np)

Parameters

Parameter Name	Description	Input / Output	Notes
input_np	Input data in `ulab.numpy.ndarray` format; the shape needs to match the configuration in `Ai2d.build`	Input	Required. This data will be saved as `self.cur_img` and must remain valid until this processing is complete

Return Value

Return Value	Description
res	The return value of the subclass’s `postprocess()`, i.e. the post-processing result of the current task, which can be passed directly to `draw_result()`

deinit#

Description

AIBase deinitialization method. Releases the kpu instance, input/output tensors, and Ai2d preprocessing resources in order, and returns the memory to the nncase memory pool. Must be called when the AI program exits or when switching models.

The specific approach is: first release the kpu (the interpreter holds references to the input/output tensors, which must be released before the tensors), then clear the tensor list and result list, then call ai2d.deinit(), and finally execute gc.collect() + nncase_runtime.shrink_memory_pool() + gc.collect().

Syntax

aibase.deinit()

Parameters

Parameter Name	Description	Input / Output	Notes
None

Return Value

Return Value	Description
None

Tip

No need to manually call gc.collect() and shrink_memory_pool() again; deinit() already does this internally.
It is safe to call repeatedly; the second call will return directly.
If the release of a certain native resource fails, AIBase will retain the reference to that resource for the next deinit() retry, and will raise the first exception that occurred after attempting to release all other resources.
If the subclass creates additional resources (for example, multiple Ai2d instances, or tensors created by nn.from_numpy() on its own), it needs to override deinit() to release these resources first, and then call super().deinit().

Subclass Override Example

class MyMultiStageApp(AIBase):
    def __init__(self, ...):
        super().__init__(kmodel_path, model_input_size, rgb888p_size, debug_mode)
        self.ai2d_pad = Ai2d(debug_mode)      # First-level preprocessing
        self.ai2d_crop = Ai2d(debug_mode)     # Second-level preprocessing
        self.ai2d = self.ai2d_pad             # Hand over one of them to the parent class for management

    # Override deinit to release the ai2d that the parent class doesn't know about
    def deinit(self):
        with ScopedTiming("deinit", self.debug_mode > 0):
            self.ai2d_crop.deinit()
            super().deinit()      # The parent class is responsible for releasing the kpu, tensors, and self.ai2d

Example Program#

Attention

The AIBase class is basically not used alone. It serves as the parent class for AI Demo application development, providing basic interfaces. Subclasses inherit from AIBase and override partial methods according to the task type to implement specific scenarios. During development, the draw_result method needs to be defined in the subclass to draw results according to the task.

Below is a face detection example program:

from libs.PipeLine import PipeLine
from libs.Utils import *
from libs.AIBase import AIBase
from libs.AI2D import Ai2d
import os,sys,gc,time,random,utime
import ujson
from media.media import *
from time import *
import nncase_runtime as nn
import ulab.numpy as np
import image
import aidemo

# 自定义人脸检测类，继承自AIBase基类
class FaceDetectionApp(AIBase):
    def __init__(self, kmodel_path, model_input_size, anchors, confidence_threshold=0.5, nms_threshold=0.2, rgb888p_size=[224,224], display_size=[1920,1080], debug_mode=0):
        super().__init__(kmodel_path, model_input_size, rgb888p_size, debug_mode)  # 调用基类的构造函数
        self.kmodel_path = kmodel_path  # 模型文件路径
        self.model_input_size = model_input_size  # 模型输入分辨率
        self.confidence_threshold = confidence_threshold  # 置信度阈值
        self.nms_threshold = nms_threshold  # NMS（非极大值抑制）阈值
        self.anchors = anchors  # 锚点数据，用于目标检测
        self.rgb888p_size = [ALIGN_UP(rgb888p_size[0], 16), rgb888p_size[1]]  # sensor给到AI的图像分辨率，并对宽度进行16的对齐
        self.display_size = [ALIGN_UP(display_size[0], 16), display_size[1]]  # 显示分辨率，并对宽度进行16的对齐
        self.debug_mode = debug_mode  # 是否开启调试模式
        self.ai2d = Ai2d(debug_mode)  # 实例化Ai2d，用于实现模型预处理
        self.ai2d.set_ai2d_dtype(nn.ai2d_format.NCHW_FMT, nn.ai2d_format.NCHW_FMT, np.uint8, np.uint8)  # 设置Ai2d的输入输出格式和类型

    # 配置预处理操作，这里使用了pad和resize，Ai2d支持crop/shift/pad/resize/affine，具体代码请打开/sdcard/libs/AI2D.py查看
    def config_preprocess(self, input_image_size=None):
        with ScopedTiming("set preprocess config", self.debug_mode > 0):  # 计时器，如果debug_mode大于0则开启
            ai2d_input_size = input_image_size if input_image_size else self.rgb888p_size  # 初始化ai2d预处理配置，默认为sensor给到AI的尺寸，可以通过设置input_image_size自行修改输入尺寸
            top, bottom, left, right = self.get_padding_param()  # 获取padding参数
            self.ai2d.pad([0, 0, 0, 0, top, bottom, left, right], 0, [104, 117, 123])  # 填充边缘
            self.ai2d.resize(nn.interp_method.tf_bilinear, nn.interp_mode.half_pixel)  # 缩放图像
            self.ai2d.build([1,3,ai2d_input_size[1],ai2d_input_size[0]],[1,3,self.model_input_size[1],self.model_input_size[0]])  # 构建预处理流程

    # 自定义当前任务的后处理，results是模型输出array列表，这里使用了aidemo库的face_det_post_process接口
    def postprocess(self, results):
        with ScopedTiming("postprocess", self.debug_mode > 0):
            post_ret = aidemo.face_det_post_process(self.confidence_threshold, self.nms_threshold, self.model_input_size[1], self.anchors, self.rgb888p_size, results)
            if len(post_ret) == 0:
                return post_ret
            else:
                return post_ret[0]

    # 绘制检测结果到画面上
    def draw_result(self, pl, dets):
        with ScopedTiming("display_draw", self.debug_mode > 0):
            if dets:
                pl.osd_img.clear()  # 清除OSD图像
                for det in dets:
                    # 将检测框的坐标转换为显示分辨率下的坐标
                    x, y, w, h = map(lambda x: int(round(x, 0)), det[:4])
                    x = x * self.display_size[0] // self.rgb888p_size[0]
                    y = y * self.display_size[1] // self.rgb888p_size[1]
                    w = w * self.display_size[0] // self.rgb888p_size[0]
                    h = h * self.display_size[1] // self.rgb888p_size[1]
                    pl.osd_img.draw_rectangle(x, y, w, h, color=(255, 255, 0, 255), thickness=2)  # 绘制矩形框
            else:
                pl.osd_img.clear()

    # 获取padding参数
    def get_padding_param(self):
        dst_w = self.model_input_size[0]  # 模型输入宽度
        dst_h = self.model_input_size[1]  # 模型输入高度
        ratio_w = dst_w / self.rgb888p_size[0]  # 宽度缩放比例
        ratio_h = dst_h / self.rgb888p_size[1]  # 高度缩放比例
        ratio = min(ratio_w, ratio_h)  # 取较小的缩放比例
        new_w = int(ratio * self.rgb888p_size[0])  # 新宽度
        new_h = int(ratio * self.rgb888p_size[1])  # 新高度
        dw = (dst_w - new_w) / 2  # 宽度差
        dh = (dst_h - new_h) / 2  # 高度差
        top = int(round(0))
        bottom = int(round(dh * 2 + 0.1))
        left = int(round(0))
        right = int(round(dw * 2 - 0.1))
        return top, bottom, left, right

if __name__ == "__main__":
    # 显示模式，"auto"按开发板型号自动选择屏驱动；也可显式指定"hdmi"/"lcd"/"st7701"等
    display_mode="auto"
    # None表示使用屏驱动的默认分辨率；需要指定时填[宽, 高]
    display_size=None
    # k230保持不变，k230d可调整为[640,360]
    rgb888p_size = [1920, 1080]
    # 设置模型路径和其他参数
    kmodel_path = "/sdcard/examples/kmodel/face_detection_320.kmodel"
    # 其它参数
    confidence_threshold = 0.5
    nms_threshold = 0.2
    anchor_len = 4200
    det_dim = 4
    anchors_path = "/sdcard/examples/utils/prior_data_320.bin"
    anchors = np.fromfile(anchors_path, dtype=np.float)
    anchors = anchors.reshape((anchor_len, det_dim))

    # 初始化PipeLine，rgb888p_size为传给AI的图像分辨率，display_size为显示分辨率
    pl = PipeLine(rgb888p_size=rgb888p_size, display_mode=display_mode, display_size=display_size)
    # 创建PipeLine，可按需传入sensor_id选择摄像头，例如pl.create(sensor_id=2)
    pl.create()  # 创建PipeLine实例
    display_size = pl.get_display_size()        # create之后再取实际显示分辨率
    # 初始化自定义人脸检测实例
    face_det = FaceDetectionApp(kmodel_path, model_input_size=[320, 320], anchors=anchors, confidence_threshold=confidence_threshold, nms_threshold=nms_threshold, rgb888p_size=rgb888p_size, display_size=display_size, debug_mode=0)
    face_det.config_preprocess()  # 配置预处理
    while True:
        with ScopedTiming("total",1):
            img = pl.get_frame()            # 获取当前帧数据
            res = face_det.run(img)         # 推理当前帧
            face_det.draw_result(pl, res)   # 绘制结果
            pl.show_image()                 # 显示结果
            gc.collect()                    # 垃圾回收
    face_det.deinit()                       # 反初始化
    pl.destroy()                            # 销毁PipeLine实例

Development Tips#

For common data type conversions during development, corresponding examples are provided here for reference.

Tips:

Image object to ulab.numpy.ndarray:

import image
img.to_rgb888().to_numpy_ref() #返回的array是HWC排布

ulab.numpy.ndarray to Image object:

import ulab.numpy as np
import image
img_np = np.zeros((height,width,4),dtype=np.uint8)
img = image.Image(width, height, image.ARGB8888, alloc=image.ALLOC_REF,data =img_np)

ulab.numpy.ndarray to tensor type:

import ulab.numpy as np
import nncase_runtime as nn
img_np = np.zeros((height,width,4),dtype=np.uint8)
tensor = nn.from_numpy(img_np)

tensor type to ulab.numpy.ndarray:

import ulab.numpy as np
import nncase_runtime as nn
img_np=tensor.to_numpy()

AIBase Module API Manual

Contents

`AIBase` Module API Manual#

Overview#

API Introduction#

init#

get_kmodel_inputs_num#

get_kmodel_outputs_num#

preprocess#

inference#

postprocess#

run#

deinit#

Example Program#

Development Tips#

AIBase Module API Manual

Contents

AIBase Module API Manual#

Overview#

API Introduction#

init#

get_kmodel_inputs_num#

get_kmodel_outputs_num#

preprocess#

inference#

postprocess#

run#

deinit#

Example Program#

Development Tips#

`AIBase` Module API Manual#