Note

This is the documentation for the latest development branch and may refer to features that are not available in released versions. If you are looking for a specific release, use the drop-down menu on the left.

Multimedia Usage Guide

Overview

This guide aims to help users understand and use the K230 Linux multimedia features. The main content includes how to use the V4L2 VPU (Video Processing Unit) and ALSA audio interfaces. The guide provides example code for directly calling the V4L2 and ALSA interfaces, and introduces how to use the FFmpeg tool for multimedia processing, including detailed steps for calling the VPU and ALSA methods.

Through this guide, users will be able to:

• Master the use of V4L2 VPU, including video capture, encoding, and decoding, and be provided with v4l2 video codec example demonstrations

• Use V4L2 and DRM to complete single-channel and multi-channel camera previews and ISP scene switching

• Learn the use of the ALSA audio interface, including audio capture and playback, and be provided with alsa audio capture playback example demonstrations

• Use FFmpeg to convert and process multimedia files, and learn how to call the VPU and ALSA methods

• Understand the camera video stream transmission solutions provided by K230, supporting both RTSP and WebRTC protocols, with corresponding demo programs

V4L2 Camera Display Demo

VVCAM (V4L2 Camera) is K230’s camera driver framework. It provides three camera display examples based on V4L2 and DRM. The source code is located at k230_linux_sdk/buildroot-overlay/package/vvcam:

• v4l2-drm: Basic capture and display example, supporting cropping, mirroring, rotation, and Sensor output parameter configuration.

• v4l2-drm-three-camera: Captures up to three video streams simultaneously and displays them separately on three DRM video layers.

• v4l2-drm-scene: Captures a single video stream and switches between different ISP scene configurations while running.

The three programs are compiled and installed together with VVCAM in the /usr/bin directory on the development board. Before running, ensure that isp_media_server and the VVCAM-related kernel modules have been started, and confirm the device nodes corresponding to the cameras using the following command:

v4l2-ctl --list-devices

Linux dynamically assigns /dev/videoN numbers, so the camera path should not be determined based solely on the device number. It is recommended to identify the required device based on the vvcam-video.0.N name, and then pass the actual device number to the demo.

Basic Display Demo

v4l2-drm captures camera images through V4L2 and uses DMA-BUF to pass the images to DRM for display, avoiding CPU copies. Basic usage is as follows:

v4l2-drm -d 1 -w 1920 -h 1080 -f NV12 -n 5

Common parameters are as follows:

• -d N: Opens/dev/videoN. This parameter must be specified before the other parameters for this camera.

• -w, -h: Sets the output image width and height. The default is 640×480.

• -n: Sets the number of buffers.

• -f: Sets the pixel format. Common values areNV12orNV16, withNV12as the default.

• -s: Captures only, without using DRM for display.

• -x, -y, --crop-width, --crop-height: Sets the crop region.

• --hflip 0|1, --vflip 0|1: Sets horizontal or vertical Sensor mirroring.

• --rotation N: Sets the display rotation direction, where0, 1, 2, and 3respectively represent 0°, 90°, 180°, and 270°.

• --sw, --sh, --sfps: Sets the Sensor’s target output width, height, and frame rate, respectively. All three parameters must be used together, and only a single camera is supported.

• --ae 0|1: Sets automatic exposure before streaming starts. 0means manual, and1means automatic. If this parameter is omitted, the demo does not change the automatic exposure switch and retains the current ISP state.

• --exp US: Sets the manual exposure time in microseconds, not the 100-microsecond unit specified by the V4L2 standard. It must be a positive integer. It takes effect only in manual exposure mode.

• --again N: Sets the manual analog gain in thousandths, where1000means 1.0×, and2000means 2.0×. It must be a positive integer. It takes effect only in manual exposure mode.

To switch the Sensor’s target output mode before running, set the target width, height, and frame rate simultaneously:

v4l2-drm -d 1 -w 1280 -h 720 --sw 1280 --sh 720 --sfps 30

Manual exposure requires disabling automatic exposure at the same time. The following sets the exposure to 8000 microseconds and the analog gain to 2.0×:

v4l2-drm -d 1 -w 1920 -h 1080 --ae 0 --exp 8000 --again 2000

These three parameters, like-w, must be specified after the corresponding-d. 0, negative numbers, and non-numeric values cause an error and immediate exit. The program first writes the exposure and gain, then writes the automatic exposure switch, and only then starts streaming, allowing the ISP to use the newly written values in manual mode. Whenever any one of these parameters is set, the allowed range for the current Sensor mode and the values actually read back are printed at startup. Read-back values are rounded according to the Sensor’s register step and may differ from the written values by one step. With multiple cameras, each stream can be configured separately.

While the program is running, it outputs the capture and display frame rates to the terminal. After pressing a key, you need to press Enter again:

• d: Saves the current frame to the current directory.

• q: Ends the run.

• +, -: Doubles or halves the exposure time. When it falls below the minimum exposure for the current mode, the read-back value stops at one exposure-time step.

• ], [: Doubles or halves the analog gain, stopping at1000when it falls below 1.0×.

+, -, ], and[only remain effective in manual exposure mode. When automatic exposure is enabled, the ISP overrides the values just written. When multiple streams are open simultaneously, these four keys affect every stream.

Exposure Control Interface

The controls below are attached to the video node. v4l2-drm, the three cameras and the scene-switching demo, as well as other programs, all use the same interface. Programs access them through VIDIOC_S_CTRL / VIDIOC_G_CTRL. The --ae, --exp, and --again options of v4l2-drm correspond to the three items in the table.

Control	v4l2-ctl name	Value
V4L2_CID_EXPOSURE_AUTO	auto_exposure	0 for automatic, 1 for manual, and 3 is the same as automatic. 2 is not supported, and writing it returns EINVAL. It cannot be changed after streaming starts; otherwise, EBUSY is returned.
V4L2_CID_EXPOSURE	exposure	Microseconds. The static range is 1 to 1000000, with a default of 10000.
V4L2_CID_ANALOGUE_GAIN	analogue_gain	Thousandths of a multiple, where 1000 is 1.0x. The static range is 1000 to 256000, with a default of 1000.

The V4L2 specification defines the unit of V4L2_CID_EXPOSURE as 100 microseconds. Here, 1 microsecond is used because an exposure row can be shorter than 100 microseconds. The number displayed by v4l2-ctl -C exposure is the actual number of microseconds; do not multiply it by 100 again according to the specification’s 100-microsecond unit, as this would make the time appear to be 100 times the actual value. The thousandths-of-a-multiple unit for analogue gain is consistent with common practice.

The ranges returned by VIDIOC_QUERYCTRL (v4l2-ctl -l) are:

• Before this pipeline is established, the static ranges in the table above are returned. The pipeline is established during VIDIOC_G_FMT or when streaming starts.

• After it is established, the actual ranges for the current Sensor mode are returned and are retained after close. The exposure upper limit equals the current frame’s long exposure time and is updated when the resolution or frame rate changes. For gc2093 at 1080p30, it is approximately 28 to 33321 microseconds, with gain 1000 to 63984; at 1080p60, the exposure upper limit is approximately 16652 microseconds.

• Writes that exceed the actual range are truncated and do not return an error. The value read back is the value written to the Sensor register and rounded according to the step size.

Other conventions:

• It must be set on the fd that is about to stream. The pipeline is destroyed along with close. If v4l2-ctl --set-ctrl is opened and closed separately, the values remain in isp_media_server, but they do not affect another stream that is already running.

• In automatic mode, written exposure and gain values are only cached; the image remains controlled by automatic exposure, and they are not applied until switching to manual mode.

• After the video node is closed, the mode and values are still retained until isp_media_server is restarted.

• When entering manual mode, an unset manual exposure uses 10000 microseconds and 1.0x gain.

The current state is written to the ae_mode line in /run/isp_media_server/status. Manual mode is similar to:

ae_mode         : manual  exposure=9982us(req 10000)  again=2.000x

The value after req is the requested value, while the value before it is the actual value used. Automatic mode only has ae_mode : auto.

Three-Camera Display Demo

v4l2-drm-three-camera can open up to three V4L2 devices simultaneously. Each -d and the parameters following it constitute one camera configuration, allowing the display position and DRM video layer to be specified separately. The following example uses the MP outputs of three cameras corresponding to /dev/video1, /dev/video4, and /dev/video7:

v4l2-drm-three-camera \
    -d 1 -w 240 -h 240 -x 0   -y 0 -l 1 \
    -d 4 -w 240 -h 400 -x 240 -y 0 -l 2 \
    -d 7 -w 480 -h 800 -x 480 -y 0 -l 3

The actual device numbers may differ. Before running, replace the -d parameters above according to the output of v4l2-ctl --list-devices. For three cameras, the MP node of each camera should be selected in priority order. The MCM (multi-camera mode) layout means that the system registers video output nodes by camera group, with each camera registering the MP, SP1, and SP2 outputs sequentially. Therefore, the VVCAM entities of the three MP outputs are typically vvcam-video.0.0, vvcam-video.0.3, and vvcam-video.0.6. For a detailed description of the node layout, please refer to Sensor Switching Description.

The following parameters are supported for each camera:

• -d N: Video device number corresponding to /dev/videoN.

• -w, -h: Display width and height of the camera’s image.

• -n: Number of buffers.

• -f: Pixel format, supporting NV12 or NV16.

• -x, -y: Starting coordinates of the image on the screen.

• -l 1|2|3: Selects the video_1, video_2, or video_3 hardware layer of the K230.

• --rotation 0|1|2|3: Sets the display rotation direction for the camera.

• -s: Captures images from the camera without displaying them.

Different cameras cannot use the same DRM video layer. The program outputs the frame rate of each camera and the overall display frame rate in real time. Press the d key to save the current frame of each camera, and press the q key to exit.

ISP Scene Switching Demo

v4l2-drm-scene is used to verify runtime switching of a single camera between different ISP calibration parameters. The program always uses the NV12 format and 5 DMA-BUFs, and describes the available scenes through a JSON file:

{
    "scenes": [
        {
            "name": "day",
            "calib_dir": "/root/scenes/day",
            "mode": 0
        },
        {
            "name": "night",
            "calib_dir": "/root/scenes/night",
            "mode": 0
        }
    ]
}

Here, name is the scene name, calib_dir is the directory containing the ISP calibration files for the scene, and mode is the mode index in the Sensor driver. The configuration file uses simple line-by-line parsing and should maintain the one-field-per-line format shown in the example.

Use the following command to load the scene configuration:

v4l2-drm-scene -d 1 -w 1920 -h 1080 \
    -s /root/scenes/scene_config.json

The program supports the following keys at runtime:

• 1–9: Switch to the scene with the corresponding sequence number.

• g: Read and display the currently active calib_dir and mode.

• q: Exit the program.

Parameter descriptions:

• -d N: Video device number, corresponding to /dev/videoN, with a default value of 0.

• -w, -h: Capture and display resolution, with a default value of 640×480.

• -s: Path to the scene configuration JSON file. Note that -s here does not mean “disable display”.

Each calib_dir directory should provide XML, manual JSON, and auto JSON calibration files matching the current Sensor and resolution. isp_media_server locates these files according to the {calib_dir}/{sensor}-{width}x{height} rule.

Scene switching closes and reopens the video device, which usually results in a black screen lasting from several hundred milliseconds to approximately 1 second. Different scenes can only adjust internal parameters of ISP modules such as exposure, gain, white balance, gamma, noise reduction, and sharpening, and cannot change the enabled state of ISP submodules. The combinations of enabled and disabled ISP modules must remain consistent across all scenes; otherwise, repeated switching may cause libcam_engine anomalies.

V4L2 VPU Usage Guide

Video Encoding

In this section, we will demonstrate how to use the VPU (Video Processing Unit) to encode images in NV12 format. The image source can be frames captured from a camera, or other files serving as the encoder’s video input source.

Encoder Demo Introduction

mvx_encoder is a command-line video encoding tool that utilizes the VPU (Video Processing Unit) for hardware acceleration. It supports encoding formats such as H.264, H.265, and JPEG, and interacts with hardware devices through the V4L2 (Video for Linux 2) interface to achieve efficient encoding performance. The source code is located at k230_linux_sdk/buildroot-overlay/package/mvx_player.

Use the following command to encode raw data:

mvx_encoder --dev <vpu_device> -i <input_format> -w <width> -h <height> -o <output_format> <input_file> <output_file>

Where:

• --dev <vpu_device>: Specifies the vpu device path.

• -i <input_format>: Specifies the input image format.

• -w <width> -h <height>: Sets the image resolution.

• -o <output_format>: Specifies the output video format.

• <input_file>: The input image file.

• <output_file>: The output video file.

After executing the above command, the input image file <input_file> will be encoded into a video file <output_file> in the specified format.

In addition to basic usage, mvx_encoder also supports advanced features such as adjusting encoding quality, setting bitrate, and specifying encoding frame rate. Use the following command to view the complete parameter list:

mvx_encoder -help

Finding the VPU Device

Before using the VPU for video processing, you need to find the camera device and the VPU device. You can use the following command to list all video devices:

v4l2-ctl --list-devices

Example output:

Linlon Video device (platform:mvx):
    /dev/video0

Canaan nonai 2D (platform:nonai-2d):
    /dev/video5

vvcam-isp-subdev.0 (platform:vvcam-isp-subdev.0):
    /dev/v4l-subdev0

verisilicon_media (platform:vvcam-video.0):
    /dev/media0

vvcam-video.0.0 (platform:vvcam-video.0.0):
    /dev/video1

vvcam-video.0.1 (platform:vvcam-video.0.1):
    /dev/video2

vvcam-video.0.2 (platform:vvcam-video.0.2):
    /dev/video3

vvcam-video.0.3 (platform:vvcam-video.0.3):
    /dev/video4

In the above output:

• /dev/video0 is the codec device.

• /dev/video1 is the camera device.

With this information, you can determine the device path to use for subsequent video capture and encoding operations.

Use the following command to verify that the camera is working properly (an external display needs to be connected via the HDMI interface):

v4l2-drm -d 1 -w 480 -h 320

If you can see the real-time camera image on the display, the camera is working properly.

Using Camera to Capture Images and Encode

First, we will use the camera to capture images and save them in NV12 format. Then, we will encode these images using the VPU. Please ensure there is enough storage space to save the images. It is recommended to mount an external storage device via NFS.

Video Capture

Use the following command to capture camera images and save them in NV12 format at 1080p resolution, capturing 25 frames. Please ensure that the data write speed is faster than the camera’s capture speed; otherwise, please lower the camera’s resolution.

v4l2-ctl --device=/dev/video1 --set-fmt-video=width=1920,height=1080,pixelformat=NV12 --stream-mmap --stream-count=25 --stream-to=output.yuv

Where:

• --device=/dev/video1: Specifies the camera device, which can be adjusted according to actual conditions.

• --set-fmt-video=width=1920,height=1080,pixelformat=NV12: Sets the video format to 1080p resolution and NV12 pixel format.

• --stream-mmap: Uses memory-mapped mode for video stream processing.

• --stream-count=25: Specifies capturing 25 frames of images.

• --stream-to=output.yuv: Saves the captured images to the output.yuv file.

Video Encoding

Use the following command to encode the captured NV12 format images into an H.264 format video file via the VPU:

mvx_encoder --dev /dev/video0 -i yuv420_nv12 -w 1920 -h 1080 -o h264 ./output.yuv ./test.264
#mvx_encoder --dev /dev/video0 -i yuv420_nv12 -w 1920 -h 1080 -o h265 ./output.yuv ./test.265
#mvx_encoder --dev /dev/video0 -i yuv420_nv12 -w 1920 -h 1080 -o jpeg ./output.yuv ./test.jpg

Where:

• --dev /dev/video0: Specifies the VPU device.

• -i yuv420_nv12: Specifies the input image format as NV12.

• -w 1920 -h 1080: Sets the image resolution to 1920x1080.

• -o h264: Specifies the output video format as H.264.

• ./output.yuv: The input NV12 format image file.

• ./test.264: The output H.264 format video file.

After executing the above command, the 25 frames of NV12 format images captured by the camera will be encoded into the H.264 format video file test.264. This file can be copied to a PC and played with a player (such as PotPlayer) to verify the encoding effect.

Encoding Using Local Files

If you have local NV12 format image files, you can also use VPU for encoding. Simply replace the input file path in the above command with the local file path.

Through these steps, you can easily use VPU to encode NV12 format images, regardless of whether the image source is a camera or a local file.

Video Decoding

In this section, we will demonstrate how to use the VPU (Video Processing Unit) to decode NV12 format H.264 files. The image source can be files generated from the video encoding chapter, or files you have prepared yourself.

Decoder Demo Introduction

mvx_decoder is a command-line video decoding tool that leverages the VPU (Video Processing Unit) for hardware acceleration. It supports decoding formats such as H.264, H.265, and JPEG, and interacts with hardware devices through the V4L2 (Video for Linux 2) interface to achieve efficient decoding performance. The source code is located at k230_linux_sdk/buildroot-overlay/package/mvx_player.

Use the following command to decode a video file into images:

mvx_decoder --dev <vpu_device> -i <input_format> -o <output_format> <input_file> <output_file>

Where:

• --dev <vpu_device>: Specifies the VPU device path.

• -i <input_format>: Specifies the input image format.

• -o <output_format>: Specifies the output image format.

• <input_file>: The input video file.

• <output_file>: The output image file.

After executing the above command, the input video file <input_file> will be decoded into an image file <output_file> of the specified format.

In addition to basic usage, mvx_decoder also supports advanced features such as adjusting decoding parameters. Use the following command to view the complete parameter list:

mvx_decoder -help

Video Decoding

Before decoding, please first confirm the VPU device. For details, refer to the Finding VPU Device section.

Use the following command to decode an H.264 format video file into an NV12 format image file:

mvx_decoder --dev /dev/video0 -i h264 -o yuv420_nv12 test.264 output.yuv

Where:

• --dev /dev/video0: Specifies the VPU device.

• -i h264: Specifies the input video format as H.264.

• -o yuv420_nv12: Specifies the output image format as NV12.

• test.264: The input H.264 format video file.

• output.yuv: The output NV12 format image file.

After executing the above command, the input H.264 format video file test.264 will be decoded into an NV12 format image file output.yuv. You can copy this file to a PC and use a player that supports NV12 format (such as YUVView) to verify the decoding result.

ALSA Audio Usage Guide

audio demo Introduction

audio_demo is a sample program that runs on the K230 development board. It calls the ALSA (Advanced Linux Sound Architecture) API to implement audio recording and playback functions. This sample program not only demonstrates how to perform audio processing on the K230 development board, but also provides reference code for developers using the ALSA API interface to implement audio functions.

Recording WAV Files

Use the following command to record an audio file in WAV format:

audio_demo -type 2 -filename test.wav -channels 2 -samplerate 16000

Parameter description:

• -type 2: Specifies the operation type as recording.

• -filename: Sets the output file name.

• -channels: Number of recording channels, 2 for stereo.

• -samplerate: Sets the sample rate, in Hz.

After recording is complete, the audio file is saved as test.wav in the current directory.

Playing WAV Files

Use the following command to play the recorded WAV file:

audio_demo -type 0 -filename test.wav

Parameter description:

• -type 0: Specifies the operation type as playing a WAV file.

• -filename: The name of the file to play.

Playing MP3 Files

audio_demo also supports playing audio files in MP3 format:

audio_demo -type 1 -filename example.mp3

Parameter description:

• -type 1: Specifies the operation type as playing an MP3 file.

• -filename: The name of the MP3 file to play.

Voice Capture and Playback

audio_rec_play is a sample program for voice capture and playback. It simulates the voice call scenario on the K230 side and uses audio3a to enhance voice quality. During the first 30 seconds, it starts recording and plays a local file (which can be connected to an external speaker or amplifier device through the headphone interface). During the last 30 seconds, it plays back the recorded voice to check the sound effect.

Use the following command to run audio_rec_play:

audio_rec_play

By default, the program plays the /usr/bin/audio.pcm file.

The program execution flow is as follows:

1. First 30 seconds: Start recording and play the local file /usr/bin/audio.pcm.

2. Last 30 seconds: Stop playing the local file, and play back the voice recorded during the first 30 seconds to check the sound effect.

Through this sample program, users can experience the voice capture and playback functions on the K230 side, and verify the voice quality enhancement effect of audio3a.

Multimedia Processing Using FFmpeg

In this section, we will demonstrate how to use FFmpeg to invoke the VPU for hardware encoding and decoding, and how to use FFmpeg to invoke the ALSA interface for audio recording and playback.

Capturing Camera Video and Encoding

Use the following command to capture camera video via FFmpeg and invoke the VPU to encode it as an MP4 file:

ffmpeg -f v4l2 -input_format nv12 -r 30 -s 1920x1080 -i /dev/video1 -c:v h264_v4l2m2m -b:v 4M -vsync passthrough -y test.mp4

Where:

• -f v4l2: Specifies the video input format as V4L2.

• -input_format nv12: Specifies the input image format as NV12.

• -r 30: Sets the frame rate to 30fps.

• -s 1920x1080: Specifies the resolution as 1920x1080.

• -i /dev/video1: Specifies the camera device path.

• -c:v h264_v4l2m2m: Uses the VPU for H.264 encoding.

• -b:v 4M: Sets the video bitrate to 4Mbps.

• -vsync passthrough: Synchronizes video frames.

• -y test.mp4: The output file name is test.mp4.

h264_v4l2m2m/hevc_v4l2m2m/mjpeg_v4l2m2m is a hardware-accelerated video encoder in FFmpeg, which leverages the V4L2 (Video for Linux 2) and M2M (Memory-to-Memory) interfaces to achieve efficient video encoding. By interacting with hardware devices, it provides higher encoding performance and efficiency.

After executing the above command, the video captured by the camera will be encoded in H.264 format and saved as the test.mp4 file.

Capturing Camera Images

Use the following command to capture camera images via FFmpeg and invoke the VPU to encode them as JPEG files:

ffmpeg -f v4l2 -input_format nv12 -s 1920x1080 -i /dev/video1 -vframes 10 -vf format=nv12 -c:v mjpeg_v4l2m2m -vsync passthrough -y output_%03d.jpg

Where:

• -f v4l2: Specifies the video input format as V4L2.

• -input_format nv12: Specifies the input image format as NV12.

• -s 1920x1080: Specifies the resolution as 1920x1080.

• -i /dev/video1: Specifies the camera device path.

• -vframes 10: Sets the number of captured frames to 10.

• -vf format=nv12: Sets the image format to NV12.

• -c:v mjpeg_v4l2m2m: Uses the VPU for JPEG encoding.

• -vsync passthrough: Synchronizes video frames.

• -y output_%03d.jpg: The output file name format is output_001.jpg, output_002.jpg, etc.

After executing the above command, the images captured by the camera will be encoded in JPEG format and saved as multiple files, such as output_001.jpg, output_002.jpg, etc.

Capturing Camera Images and Saving as MJPEG Files

Use the following command to capture camera images via FFmpeg and invoke the VPU to encode them as MJPEG files:

ffmpeg -f v4l2 -input_format nv12 -s 1920x1080 -i /dev/video1 -vframes 100 -vf format=nv12 -c:v mjpeg_v4l2m2m -vsync passthrough -y output.mjpeg

Where:

• -f v4l2: Specifies the video input format as V4L2.

• -input_format nv12: Specifies the input image format as NV12.

• -s 1920x1080: Specifies the resolution as 1920x1080.

• -i /dev/video1: Specifies the camera device path.

• -vframes 100: Sets the number of captured frames to 100.

• -vf format=nv12: Sets the image format to NV12.

• -c:v mjpeg_v4l2m2m: Uses the VPU for MJPEG encoding.

• -vsync passthrough: Synchronizes video frames.

• -y output.mjpeg: The output file name is output.mjpeg.

After executing the above command, the images captured by the camera will be encoded in MJPEG format and saved as the output.mjpeg file.

Decoding Video and Saving as Images

Use the following command to decode an H.264 video file via FFmpeg and save it as NV12 format images:

ffmpeg -c:v h264_v4l2m2m -i test.mp4 -pix_fmt nv12 -f rawvideo -vsync passthrough -y output.yuv

Where:

• -c:v h264_v4l2m2m: Uses the VPU for H.264 decoding.

• -i input.mp4: Specifies the input video file.

• -pix_fmt nv12: Specifies the output image format as NV12.

• -f rawvideo: Specifies the output format as raw video stream.

• output.yuv: The output file name is output.yuv.

After executing the above command, the input H.264 video file input.mp4 will be decoded into an NV12 format image file output.yuv. This file can be copied to a PC and viewed using a player that supports the NV12 format (such as YUVView) to verify the decoding effect.

Recording Audio

Use the following command to invoke the ALSA interface via FFmpeg to record audio and save it as a WAV file:

ffmpeg -f alsa -i hw:0 -t 30 -ac 2 -ar 44100 -y output.wav

Where:

• -f alsa: Specifies the audio input format as ALSA.

• -i hw:0: Specifies the audio input device; hw:0 represents the default audio device.

• -t 30: Sets the recording duration to 30 seconds.

• -ac 2: Sets the number of recording channels to 2 (stereo).

• -ar 44100: Sets the sample rate to 44100Hz.

• -y output.wav: The output file name is output.wav.

After executing the above command, the recorded audio will be saved as a WAV format audio file output.wav.

Playing Audio

Use the following command to invoke the ALSA interface via FFmpeg to play the recorded WAV file:

ffmpeg -f wav -i output.wav -f alsa hw:0

Where:

• -f wav: Specifies the audio input format as WAV.

• -i output.wav: Specifies the input WAV file.

• -f alsa: Specifies the audio output format as ALSA.

• hw:0: Specifies the audio output device; hw:0 represents the default audio device.

After executing the above command, the recorded WAV file output.wav will be played through the ALSA interface.

Adjusting Volume

Adjusting Recording Volume

Use the following command to invoke the ALSA interface via FFmpeg to record audio and adjust the capture volume:

ffmpeg -f alsa -i hw:0 -t 30 -ac 2 -ar 44100 -filter:a "volume=2.0" -y output.wav

Where:

• -f alsa: Specifies the audio input format as ALSA.

• -i hw:0: Specifies the audio input device; hw:0 represents the default audio device.

• -t 30: Sets the recording duration to 30 seconds.

• -ac 2: Sets the number of recording channels to 2 (stereo).

• -ar 44100: Sets the sample rate to 44100Hz.

• -filter:a "volume=2.0": Adjusts the capture volume; 2.0 means doubling the volume.

• -y output.wav: The output file name is output.wav.

After executing the above command, the recorded audio will be saved as a WAV format audio file output.wav, and the volume will be doubled.

Adjusting Playback Volume

Use the following command to invoke the ALSA interface via FFmpeg to play audio and adjust the playback volume:

ffmpeg -f wav -i output.wav -filter:a "volume=2.0" -f alsa hw:0

Where:

• -f wav: Specifies the audio input format as WAV.

• -i output.wav: Specifies the input WAV file.

• -filter:a "volume=2.0": Adjusts the playback volume; 2.0 means doubling the volume.

• -f alsa: Specifies the audio output format as ALSA.

• hw:0: Specifies the audio output device; hw:0 represents the default audio device.

After executing the above command, the WAV file output.wav will be played through the ALSA interface, and the volume will be doubled.

Muting

The following command will use FFmpeg to invoke ALSA to play the audio file output.wav with muted output, using the hardware device hw:0.

ffmpeg -i output.wav -f alsa -ac 2 -ar 44100 -vol 0 hw:0

Where:

• -i output.wav: Specifies the input file as output.wav.

• -f alsa: Specifies the output format as ALSA.

• -ac 2: Specifies the number of audio channels as 2.

• -ar 44100: Specifies the audio sample rate as 44100 Hz.

• -vol 0: Sets the volume to 0, achieving muted playback.

• hw:0: Specifies the use of hardware device hw:0.

Camera Video Stream Transmission

K230 provides two camera video stream transmission solutions based on the RTSP protocol and the WebRTC protocol, along with corresponding demo programs:

Transmission Protocol	Demo Program	Source Code Path	Playback Method	Applicable Scenarios
RTSP	camera_rtsp_demo	k230_linux_sdk/buildroot-overlay/package/camera_rtsp_demo	Players supporting the RTSP protocol, such as VLC and FFplay	LAN monitoring, NVR system integration
WebRTC	camera_webrtc_demo	k230_linux_sdk/buildroot-overlay/package/camera_webrtc_demo	Open the web page directly in a browser	Low-latency real-time preview, Web application integration

Both solutions support camera capture, VPU hardware encoding (H.264/H.265), and OSD overlay. The following sections describe how to use each demo.

RTSP Streaming Demo

camera_rtsp_demo is a sample program for transmitting camera video streams via the RTSP protocol. It can encode camera-captured video into H.264 or H.265 format and transmit it in real time via the RTSP protocol. It demonstrates how to use the FFmpeg API to capture camera data, overlay OSD, and encode, as well as how to use the live555 API to output an RTSP stream.

Usage

Run camera_rtsp_demo using the following command:

./camera_rtsp_demo [-H] [-t <codec_type>] [-w <width>] [-h <height>] [-b <bitrate_kbps>] [-l <enable_log>] [-r <osd_region>]

Parameter descriptions:

• -H: Displays help information.

• -t <codec_type>: Specifies the video codec type. Available values are h264 or h265, with the default value being h264.

• -w <width>: Specifies the video encoding width, with the default value being 1280.

• -h <height>: Specifies the video encoding height, with the default value being 720.

• -b <bitrate_kbps>: Specifies the video encoding bitrate (kbps), with the default value being 2000.

• -l <enable_log>: Specifies whether to enable log output. Available values are 0 or 1, with the default value being 0.

• -r <osd_region>: Specifies the number of OSD overlay regions, with the default value being 0.

Example command:

./camera_rtsp_demo -t h264 -w 1920 -h 1080 -b 4000 -r 2

After executing the command above, the program will start the camera and encode the video stream in H.264 format at a resolution of 1920x1080 and a bitrate of 4000 kbps. It will overlay OSDs in 2 regions and transmit the stream using the RTSP protocol.

Find the RTSP URL in the serial port output, for example: “Play this stream using the URL: rtsp://<your_device_ip>:8554/test”, and open the URL with a player that supports the RTSP protocol (such as VLC) to view the video stream.

For the complete RTSP server API, including session management, audio and video data transmission, and the audio reverse channel, see the RTSP Server API Manual.

WebRTC Streaming Demo

camera_webrtc_demo is a sample program used to transmit camera video streams over the WebRTC protocol. The program encodes video captured from the camera in H.264 format and transmits it in real time over the WebRTC protocol. It includes an HTTP server for WebRTC SDP signaling exchange and provides a web playback page, allowing users to view the video stream directly in a browser. The underlying WebRTC communication is implemented using the libpeer library.

Compared with the RTSP solution, the main advantages of the WebRTC solution are that users do not need to install a dedicated player and can view real-time video streams directly through a browser, making deployment and use more convenient. WebRTC also inherently provides lower end-to-end latency.

Usage

Run camera_webrtc_demo using the following command:

./camera_webrtc_demo [-H] [-w <width>] [-h <height>] [-b <bitrate_kbps>] [-p <port>] [-l <enable_log>] [-r <osd_region>]

Parameter descriptions:

• -H: Display help information.

• -w <width>: Specify the video encoding width. The default value is 1280.

• -h <height>: Specify the video encoding height. The default value is 720.

• -b <bitrate_kbps>: Specify the video encoding bitrate (kbps). The default value is 2000.

• -p <port>: Specify the HTTP server port used for WebRTC signaling exchange and web page access. The default value is 8080.

• -l <enable_log>: Whether to enable log output. The available values are 0 or 1. The default value is 0.

• -r <osd_region>: Specify the number of OSD overlay regions. The default value is 0.

Example command:

./camera_webrtc_demo -w 1920 -h 1080 -b 4000 -p 8080 -r 2

After executing the command above, the program will start the camera, encode the video stream in H.264 format with a resolution of 1920x1080 and a bitrate of 4000 kbps, overlay OSDs in 2 regions, and transmit the stream over the WebRTC protocol.

Find the access URL in the serial output, for example:

========================================
  Camera WebRTC Demo
========================================
  Encode: 1920x1080 @ 4000kbps H264
  Open in browser:
  http://192.168.1.100:8080
========================================

Open this URL in the browser on a computer or mobile device on the same network to view the camera’s live video stream.

For the complete API interface of the underlying libpeer library for WebRTC (including PeerConnection management, signaling interaction, DataChannel communication, etc.), see the WebRTC API Manual.