K230 Integrated Demo: Single-Channel Smart IPC#

Overview#

This integrated demo combines network streaming, local rendering, and face detection into a single smart IPC application. With one camera connected, the system can:

Display the live image on HDMI or LCD and overlay face-detection results
Stream video over the network through RTSP
Choose whether the streamed picture is the raw camera frame or the composited frame with AI overlays

Usage#

Source path: /src/rtsmart/examples/integrated_poc/smart_ipc

Enable the build in make menuconfig:

RT-Smart UserSpace Examples Configuration
Enable build integrated examples
Smart IPC examples

After a successful build, the executable is deployed to:

/sdcard/app/examples/integrated_poc/smart_ipc/smart_ipc.elf

Run on target:

/sdcard/app/examples/integrated_poc/smart_ipc/smart_ipc.elf [options]

Examples:

HDMI output:

./smart_ipc.elf -C 0

LCD output:

./smart_ipc.elf -C 1

Show help:

./smart_ipc.elf -H

Detailed options:

Option	Description	Default
`-H`	show help information	-
`-S`	Sensor type, auto-detected by default	`SENSOR_TYPE_MAX`
`-V`	Enable audio capture	`1`
`-a`	Audio sample rate	`8000`
`-c`	Audio channel count	`1`
`-t`	Video codec type: `h264` or `h265`	`h264`
`-w` / `-h`	Video encode width and height	`1280x720`
`-b`	Bitrate in kbps	`2000`
`-D`	Encode raw sensor output or VO WBC composited output	`0`
`-C`	Connector type: `0` HDMI, `1` LCD	HDMI
`-A` / `-I`	AI input width and height	`1280x720`
`-K`	`kmodel` path	`face_detection_320.kmodel`
`-T` / `-N`	Face detection threshold and NMS threshold	`0.6` / `0.4`
`-E`	Enable rendering	`1`
`-F`	Enable AI analysis	`1`
`-G`	Enable encoded network streaming	`1`

Code Structure#

Important files in the project include:

face_detection: face-detection module
media.cpp: media implementation; wraps MPI initialization and startup for camera, display, and codecs
media.h: media interface declarations
smart_ipc.cpp: main application flow
smart_ipc.h: MySmartIPC class and interfaces

MySmartIPC encapsulates all integrated smart-camera logic. In main, the app creates the object and calls init and start.

Inside MySmartIPC, wrapped MPI interfaces in KdMedia initialize VB buffers, camera/display, and codecs. The following callbacks are then used:

encoded video callback: OnVEncData
encoded audio callback: OnAEncData
AI input frame callback: OnAIFrameData

MySmartIPC also uses KdRtspServer and FaceDetection for RTSP streaming and face detection. In OnAIFrameData, detection results are drawn on the VO OSD layer. In OnVEncData and OnAEncData, encoded streams are pushed to network clients.

Data Pipeline#

The pipeline uses three camera channels for local display, AI input, and encoded streaming:

camera input captures real-time video and splits it into three paths
one path is displayed through HDMI/LCD
one path is sent to AI for face detection and overlay
one path is encoded and streamed:
- raw sensor frame path: encode directly from sensor channel (no AI overlays)
- composited frame path: get VO WBC-composited frame with AI overlays, then encode and stream

This pipeline supports local display, AI analysis, and network streaming simultaneously.

pipeline

Demo Walkthrough#

To view the live stream from a PC, make sure the board has a valid network IP.

run ifconfig on the board to confirm IP address
run ping <PC_IP> on the board to verify connectivity

For HDMI output, connect HDMI to a monitor. For LCD output, connect an LCD panel.

cd /sdcard/app/examples/integrated_poc/smart_ipc
./smart_ipc.elf -C 0 -D 0 # HDMI output, RTSP streams raw frame
./smart_ipc.elf -C 1 -D 1 # LCD output, RTSP streams composited AI-overlay frame

The serial log shows a prompt like: Play this stream using the URL rtsp://<IP>:8554/test. Open this URL in VLC on your PC to view the stream.

smart_ipc