Note

This is the documentation for the latest development branch and may refer to features that are not available in released versions. If you are looking for the documentation for a specific release, use the drop-down menu on the left and select the desired version.

Multimedia Usage Guide

Overview

This guide aims to help users understand and use the K230 Linux multimedia features. The main content includes how to use the V4L2 VPU (Video Processing Unit) and ALSA audio interfaces. The guide provides example code for directly calling the V4L2 and ALSA interfaces, and introduces how to use the FFmpeg tool for multimedia processing, including detailed steps for calling the VPU and ALSA methods.

Through this guide, users will be able to:

• Master the use of V4L2 VPU, including video capture, encoding, and decoding, and be provided with v4l2 video codec example demonstrations

• Learn the use of the ALSA audio interface, including audio capture and playback, and be provided with alsa audio capture playback example demonstrations

• Use FFmpeg to convert and process multimedia files, and learn how to call the VPU and ALSA methods

V4L2 VPU Usage Guide

Video Encoding

In this section, we will demonstrate how to use the VPU (Video Processing Unit) to encode images in NV12 format. The image source can be frames captured from a camera, or other files serving as the encoder’s video input source.

Encoder Demo Introduction

mvx_encoder is a command-line video encoding tool that utilizes the VPU (Video Processing Unit) for hardware acceleration. It supports encoding formats such as H.264, H.265, and JPEG, and interacts with hardware devices through the V4L2 (Video for Linux 2) interface to achieve efficient encoding performance. The source code is located at k230_linux_sdk/buildroot-overlay/package/mvx_player.

Use the following command to encode raw data:

mvx_encoder --dev <vpu_device> -i <input_format> -w <width> -h <height> -o <output_format> <input_file> <output_file>

Where:

• --dev <vpu_device>: Specifies the vpu device path.

• -i <input_format>: Specifies the input image format.

• -w <width> -h <height>: Sets the image resolution.

• -o <output_format>: Specifies the output video format.

• <input_file>: The input image file.

• <output_file>: The output video file.

After executing the above command, the input image file <input_file> will be encoded into a video file <output_file> in the specified format.

In addition to basic usage, mvx_encoder also supports advanced features such as adjusting encoding quality, setting bitrate, and specifying encoding frame rate. Use the following command to view the complete parameter list:

mvx_encoder -help

Finding the VPU Device

Before using the VPU for video processing, you need to find the camera device and the VPU device. You can use the following command to list all video devices:

v4l2-ctl --list-devices

Example output:

Linlon Video device (platform:mvx):
    /dev/video0

Canaan nonai 2D (platform:nonai-2d):
    /dev/video5

vvcam-isp-subdev.0 (platform:vvcam-isp-subdev.0):
    /dev/v4l-subdev0

verisilicon_media (platform:vvcam-video.0):
    /dev/media0

vvcam-video.0.0 (platform:vvcam-video.0.0):
    /dev/video1

vvcam-video.0.1 (platform:vvcam-video.0.1):
    /dev/video2

vvcam-video.0.2 (platform:vvcam-video.0.2):
    /dev/video3

vvcam-video.0.3 (platform:vvcam-video.0.3):
    /dev/video4

In the above output:

• /dev/video0 is the codec device.

• /dev/video1 is the camera device.

With this information, you can determine the device path to use for subsequent video capture and encoding operations.

Use the following command to verify that the camera is working properly (an external display needs to be connected via the HDMI interface):

v4l2-drm -d 1 -w 480 -h 320

If you can see the real-time camera image on the display, the camera is working properly.

Using Camera to Capture Images and Encode

First, we will use the camera to capture images and save them in NV12 format. Then, we will encode these images using the VPU. Please ensure there is enough storage space to save the images. It is recommended to mount an external storage device via NFS.

Video Capture

Use the following command to capture camera images and save them in NV12 format at 1080p resolution, capturing 25 frames. Please ensure that the data write speed is faster than the camera’s capture speed; otherwise, please lower the camera’s resolution.

v4l2-ctl --device=/dev/video1 --set-fmt-video=width=1920,height=1080,pixelformat=NV12 --stream-mmap --stream-count=25 --stream-to=output.yuv

Where:

• --device=/dev/video1: Specifies the camera device, which can be adjusted according to actual conditions.

• --set-fmt-video=width=1920,height=1080,pixelformat=NV12: Sets the video format to 1080p resolution and NV12 pixel format.

• --stream-mmap: Uses memory-mapped mode for video stream processing.

• --stream-count=25: Specifies capturing 25 frames of images.

• --stream-to=output.yuv: Saves the captured images to the output.yuv file.

Video Encoding

Use the following command to encode the captured NV12 format images into an H.264 format video file via the VPU:

mvx_encoder --dev /dev/video0 -i yuv420_nv12 -w 1920 -h 1080 -o h264 ./output.yuv ./test.264
#mvx_encoder --dev /dev/video0 -i yuv420_nv12 -w 1920 -h 1080 -o h265 ./output.yuv ./test.265
#mvx_encoder --dev /dev/video0 -i yuv420_nv12 -w 1920 -h 1080 -o jpeg ./output.yuv ./test.jpg

Where:

• --dev /dev/video0: Specifies the VPU device.

• -i yuv420_nv12: Specifies the input image format as NV12.

• -w 1920 -h 1080: Sets the image resolution to 1920x1080.

• -o h264: Specifies the output video format as H.264.

• ./output.yuv: The input NV12 format image file.

• ./test.264: The output H.264 format video file.

After executing the above command, the 25 frames of NV12 format images captured by the camera will be encoded into the H.264 format video file test.264. This file can be copied to a PC and played with a player (such as PotPlayer) to verify the encoding effect.

Encoding Using Local Files

If you have local NV12 format image files, you can also use VPU for encoding. Simply replace the input file path in the above command with the local file path.

Through these steps, you can easily use VPU to encode NV12 format images, regardless of whether the image source is a camera or a local file.

Video Decoding

In this section, we will demonstrate how to use the VPU (Video Processing Unit) to decode NV12 format H.264 files. The image source can be files generated from the video encoding chapter, or files you have prepared yourself.

Decoder Demo Introduction

mvx_decoder is a command-line video decoding tool that leverages the VPU (Video Processing Unit) for hardware acceleration. It supports decoding formats such as H.264, H.265, and JPEG, and interacts with hardware devices through the V4L2 (Video for Linux 2) interface to achieve efficient decoding performance. The source code is located at k230_linux_sdk/buildroot-overlay/package/mvx_player.

Use the following command to decode a video file into images:

mvx_decoder --dev <vpu_device> -i <input_format> -o <output_format> <input_file> <output_file>

Where:

• --dev <vpu_device>: Specifies the VPU device path.

• -i <input_format>: Specifies the input image format.

• -o <output_format>: Specifies the output image format.

• <input_file>: The input video file.

• <output_file>: The output image file.

After executing the above command, the input video file <input_file> will be decoded into an image file <output_file> of the specified format.

In addition to basic usage, mvx_decoder also supports advanced features such as adjusting decoding parameters. Use the following command to view the complete parameter list:

mvx_decoder -help

Video Decoding

Before decoding, please first confirm the VPU device. For details, refer to the Finding VPU Device section.

Use the following command to decode an H.264 format video file into an NV12 format image file:

mvx_decoder --dev /dev/video0 -i h264 -o yuv420_nv12 test.264 output.yuv

Where:

• --dev /dev/video0: Specifies the VPU device.

• -i h264: Specifies the input video format as H.264.

• -o yuv420_nv12: Specifies the output image format as NV12.

• test.264: The input H.264 format video file.

• output.yuv: The output NV12 format image file.

After executing the above command, the input H.264 format video file test.264 will be decoded into an NV12 format image file output.yuv. You can copy this file to a PC and use a player that supports NV12 format (such as YUVView) to verify the decoding result.

ALSA Audio Usage Guide

audio demo Introduction

audio_demo is a sample program that runs on the K230 development board. It calls the ALSA (Advanced Linux Sound Architecture) API to implement audio recording and playback functions. This sample program not only demonstrates how to perform audio processing on the K230 development board, but also provides reference code for developers using the ALSA API interface to implement audio functions.

Recording WAV Files

Use the following command to record an audio file in WAV format:

audio_demo -type 2 -filename test.wav -channels 2 -samplerate 16000

Parameter description:

• -type 2: Specifies the operation type as recording.

• -filename: Sets the output file name.

• -channels: Number of recording channels, 2 for stereo.

• -samplerate: Sets the sample rate, in Hz.

After recording is complete, the audio file is saved as test.wav in the current directory.

Playing WAV Files

Use the following command to play the recorded WAV file:

audio_demo -type 0 -filename test.wav

Parameter description:

• -type 0: Specifies the operation type as playing a WAV file.

• -filename: The name of the file to play.

Playing MP3 Files

audio_demo also supports playing audio files in MP3 format:

audio_demo -type 1 -filename example.mp3

Parameter description:

• -type 1: Specifies the operation type as playing an MP3 file.

• -filename: The name of the MP3 file to play.

Voice Capture and Playback

audio_rec_play is a sample program for voice capture and playback. It simulates the voice call scenario on the K230 side and uses audio3a to enhance voice quality. During the first 30 seconds, it starts recording and plays a local file (which can be connected to an external speaker or amplifier device through the headphone interface). During the last 30 seconds, it plays back the recorded voice to check the sound effect.

Use the following command to run audio_rec_play:

audio_rec_play

By default, the program plays the /usr/bin/audio.pcm file.

The program execution flow is as follows:

1. First 30 seconds: Start recording and play the local file /usr/bin/audio.pcm.

2. Last 30 seconds: Stop playing the local file, and play back the voice recorded during the first 30 seconds to check the sound effect.

Through this sample program, users can experience the voice capture and playback functions on the K230 side, and verify the voice quality enhancement effect of audio3a.

Multimedia Processing Using FFmpeg

In this section, we will demonstrate how to use FFmpeg to invoke the VPU for hardware encoding and decoding, and how to use FFmpeg to invoke the ALSA interface for audio recording and playback.

Capturing Camera Video and Encoding

Use the following command to capture camera video via FFmpeg and invoke the VPU to encode it as an MP4 file:

ffmpeg -f v4l2 -input_format nv12 -r 30 -s 1920x1080 -i /dev/video1 -c:v h264_v4l2m2m -b:v 4M -vsync passthrough -y test.mp4

Where:

• -f v4l2: Specifies the video input format as V4L2.

• -input_format nv12: Specifies the input image format as NV12.

• -r 30: Sets the frame rate to 30fps.

• -s 1920x1080: Specifies the resolution as 1920x1080.

• -i /dev/video1: Specifies the camera device path.

• -c:v h264_v4l2m2m: Uses the VPU for H.264 encoding.

• -b:v 4M: Sets the video bitrate to 4Mbps.

• -vsync passthrough: Synchronizes video frames.

• -y test.mp4: The output file name is test.mp4.

h264_v4l2m2m/hevc_v4l2m2m/mjpeg_v4l2m2m is a hardware-accelerated video encoder in FFmpeg, which leverages the V4L2 (Video for Linux 2) and M2M (Memory-to-Memory) interfaces to achieve efficient video encoding. By interacting with hardware devices, it provides higher encoding performance and efficiency.

After executing the above command, the video captured by the camera will be encoded in H.264 format and saved as the test.mp4 file.

Capturing Camera Images

Use the following command to capture camera images via FFmpeg and invoke the VPU to encode them as JPEG files:

ffmpeg -f v4l2 -input_format nv12 -s 1920x1080 -i /dev/video1 -vframes 10 -vf format=nv12 -c:v mjpeg_v4l2m2m -vsync passthrough -y output_%03d.jpg

Where:

• -f v4l2: Specifies the video input format as V4L2.

• -input_format nv12: Specifies the input image format as NV12.

• -s 1920x1080: Specifies the resolution as 1920x1080.

• -i /dev/video1: Specifies the camera device path.

• -vframes 10: Sets the number of captured frames to 10.

• -vf format=nv12: Sets the image format to NV12.

• -c:v mjpeg_v4l2m2m: Uses the VPU for JPEG encoding.

• -vsync passthrough: Synchronizes video frames.

• -y output_%03d.jpg: The output file name format is output_001.jpg, output_002.jpg, etc.

After executing the above command, the images captured by the camera will be encoded in JPEG format and saved as multiple files, such as output_001.jpg, output_002.jpg, etc.

Capturing Camera Images and Saving as MJPEG Files

Use the following command to capture camera images via FFmpeg and invoke the VPU to encode them as MJPEG files:

ffmpeg -f v4l2 -input_format nv12 -s 1920x1080 -i /dev/video1 -vframes 100 -vf format=nv12 -c:v mjpeg_v4l2m2m -vsync passthrough -y output.mjpeg

Where:

• -f v4l2: Specifies the video input format as V4L2.

• -input_format nv12: Specifies the input image format as NV12.

• -s 1920x1080: Specifies the resolution as 1920x1080.

• -i /dev/video1: Specifies the camera device path.

• -vframes 100: Sets the number of captured frames to 100.

• -vf format=nv12: Sets the image format to NV12.

• -c:v mjpeg_v4l2m2m: Uses the VPU for MJPEG encoding.

• -vsync passthrough: Synchronizes video frames.

• -y output.mjpeg: The output file name is output.mjpeg.

After executing the above command, the images captured by the camera will be encoded in MJPEG format and saved as the output.mjpeg file.

Decoding Video and Saving as Images

Use the following command to decode an H.264 video file via FFmpeg and save it as NV12 format images:

ffmpeg -c:v h264_v4l2m2m -i test.mp4 -pix_fmt nv12 -f rawvideo -vsync passthrough -y output.yuv

Where:

• -c:v h264_v4l2m2m: Uses the VPU for H.264 decoding.

• -i input.mp4: Specifies the input video file.

• -pix_fmt nv12: Specifies the output image format as NV12.

• -f rawvideo: Specifies the output format as raw video stream.

• output.yuv: The output file name is output.yuv.

After executing the above command, the input H.264 video file input.mp4 will be decoded into an NV12 format image file output.yuv. This file can be copied to a PC and viewed using a player that supports the NV12 format (such as YUVView) to verify the decoding effect.

Recording Audio

Use the following command to invoke the ALSA interface via FFmpeg to record audio and save it as a WAV file:

ffmpeg -f alsa -i hw:0 -t 30 -ac 2 -ar 44100 -y output.wav

Where:

• -f alsa: Specifies the audio input format as ALSA.

• -i hw:0: Specifies the audio input device; hw:0 represents the default audio device.

• -t 30: Sets the recording duration to 30 seconds.

• -ac 2: Sets the number of recording channels to 2 (stereo).

• -ar 44100: Sets the sample rate to 44100Hz.

• -y output.wav: The output file name is output.wav.

After executing the above command, the recorded audio will be saved as a WAV format audio file output.wav.

Playing Audio

Use the following command to invoke the ALSA interface via FFmpeg to play the recorded WAV file:

ffmpeg -f wav -i output.wav -f alsa hw:0

Where:

• -f wav: Specifies the audio input format as WAV.

• -i output.wav: Specifies the input WAV file.

• -f alsa: Specifies the audio output format as ALSA.

• hw:0: Specifies the audio output device; hw:0 represents the default audio device.

After executing the above command, the recorded WAV file output.wav will be played through the ALSA interface.

Adjusting Volume

Adjusting Recording Volume

Use the following command to invoke the ALSA interface via FFmpeg to record audio and adjust the capture volume:

ffmpeg -f alsa -i hw:0 -t 30 -ac 2 -ar 44100 -filter:a "volume=2.0" -y output.wav

Where:

• -f alsa: Specifies the audio input format as ALSA.

• -i hw:0: Specifies the audio input device; hw:0 represents the default audio device.

• -t 30: Sets the recording duration to 30 seconds.

• -ac 2: Sets the number of recording channels to 2 (stereo).

• -ar 44100: Sets the sample rate to 44100Hz.

• -filter:a "volume=2.0": Adjusts the capture volume; 2.0 means doubling the volume.

• -y output.wav: The output file name is output.wav.

After executing the above command, the recorded audio will be saved as a WAV format audio file output.wav, and the volume will be doubled.

Adjusting Playback Volume

Use the following command to invoke the ALSA interface via FFmpeg to play audio and adjust the playback volume:

ffmpeg -f wav -i output.wav -filter:a "volume=2.0" -f alsa hw:0

Where:

• -f wav: Specifies the audio input format as WAV.

• -i output.wav: Specifies the input WAV file.

• -filter:a "volume=2.0": Adjusts the playback volume; 2.0 means doubling the volume.

• -f alsa: Specifies the audio output format as ALSA.

• hw:0: Specifies the audio output device; hw:0 represents the default audio device.

After executing the above command, the WAV file output.wav will be played through the ALSA interface, and the volume will be doubled.

Muting

The following command will use FFmpeg to invoke ALSA to play the audio file output.wav with muted output, using the hardware device hw:0.

ffmpeg -i output.wav -f alsa -ac 2 -ar 44100 -vol 0 hw:0

Where:

• -i output.wav: Specifies the input file as output.wav.

• -f alsa: Specifies the output format as ALSA.

• -ac 2: Specifies the number of audio channels as 2.

• -ar 44100: Specifies the audio sample rate as 44100 Hz.

• -vol 0: Sets the volume to 0, achieving muted playback.

• hw:0: Specifies the use of hardware device hw:0.

Camera RTSP Streaming Demo

camera_rtsp_demo is a sample program used to transmit camera video streams via the RTSP protocol. This program can encode the video captured by the camera into H.264 or H.265 format and transmit it in real time via the RTSP protocol. It demonstrates how to use the FFmpeg API to obtain camera data, overlay OSD, and perform encoding, as well as how to use the live555 API to output an RTSP stream.

Usage

Use the following command to run camera_rtsp_demo:

./camera_rtsp_demo [-H] [-t <codec_type>] [-w <width>] [-h <height>] [-b <bitrate_kbps>]

Parameter description:

• -H: Display help information.

• -t <codec_type>: Specify the video encoding type, optional values are h264 or h265, the default value is h264.

• -w <width>: Specify the video encoding width, the default value is 1280.

• -h <height>: Specify the video encoding height, the default value is 720.

• -b <bitrate_kbps>: Specify the video encoding bitrate (kbps), the default value is 2000.

• -r <osd_region>: Specify the number of OSD overlay regions, the default value is 0.

Example command:

./camera_rtsp_demo -t h264 -w 1920 -h 1080 -b 4000 -r 2

After executing the above command, the program will start the camera and encode the video stream in H.264 format, with a resolution of 1920x1080, a bitrate of 4000 kbps, and 2 OSD overlay regions, and transmit it via the RTSP protocol.

Find the RTSP URL in the serial output, for example: “Play this stream using the URL: rtsp://<your_device_ip>:8554/test”, and use a player that supports the RTSP protocol (such as VLC) to open the URL to view the video stream.