# Multimedia Usage Guide ## Overview This guide aims to help users understand and use the K230 Linux multimedia features. The main content includes how to use the V4L2 VPU (Video Processing Unit) and ALSA audio interfaces. The guide provides example code for directly calling the V4L2 and ALSA interfaces, and introduces how to use the FFmpeg tool for multimedia processing, including detailed steps for calling the VPU and ALSA methods. Through this guide, users will be able to: - Master the use of V4L2 VPU, including video capture, encoding, and decoding, and be provided with v4l2 video codec example demonstrations - Learn the use of the ALSA audio interface, including audio capture and playback, and be provided with alsa audio capture playback example demonstrations - Use FFmpeg to convert and process multimedia files, and learn how to call the VPU and ALSA methods ## V4L2 VPU Usage Guide ### Video Encoding In this section, we will demonstrate how to use the VPU (Video Processing Unit) to encode images in NV12 format. The image source can be frames captured from a camera, or other files serving as the encoder's video input source. #### Encoder Demo Introduction `mvx_encoder` is a command-line video encoding tool that utilizes the VPU (Video Processing Unit) for hardware acceleration. It supports encoding formats such as H.264, H.265, and JPEG, and interacts with hardware devices through the V4L2 (Video for Linux 2) interface to achieve efficient encoding performance. The source code is located at `k230_linux_sdk/buildroot-overlay/package/mvx_player`. Use the following command to encode raw data: ```bash mvx_encoder --dev -i -w -h -o ``` Where: - `--dev `: Specifies the vpu device path. - `-i `: Specifies the input image format. - `-w -h `: Sets the image resolution. - `-o `: Specifies the output video format. - ``: The input image file. - ``: The output video file. After executing the above command, the input image file `` will be encoded into a video file `` in the specified format. In addition to basic usage, `mvx_encoder` also supports advanced features such as adjusting encoding quality, setting bitrate, and specifying encoding frame rate. Use the following command to view the complete parameter list: ```bash mvx_encoder -help ``` #### Finding the VPU Device Before using the VPU for video processing, you need to find the camera device and the VPU device. You can use the following command to list all video devices: ```bash v4l2-ctl --list-devices ``` Example output: ```bash Linlon Video device (platform:mvx): /dev/video0 Canaan nonai 2D (platform:nonai-2d): /dev/video5 vvcam-isp-subdev.0 (platform:vvcam-isp-subdev.0): /dev/v4l-subdev0 verisilicon_media (platform:vvcam-video.0): /dev/media0 vvcam-video.0.0 (platform:vvcam-video.0.0): /dev/video1 vvcam-video.0.1 (platform:vvcam-video.0.1): /dev/video2 vvcam-video.0.2 (platform:vvcam-video.0.2): /dev/video3 vvcam-video.0.3 (platform:vvcam-video.0.3): /dev/video4 ``` In the above output: - `/dev/video0` is the codec device. - `/dev/video1` is the camera device. With this information, you can determine the device path to use for subsequent video capture and encoding operations. Use the following command to verify that the camera is working properly (an external display needs to be connected via the HDMI interface): ```bash v4l2-drm -d 1 -w 480 -h 320 ``` If you can see the real-time camera image on the display, the camera is working properly. #### Using Camera to Capture Images and Encode First, we will use the camera to capture images and save them in NV12 format. Then, we will encode these images using the VPU. Please ensure there is enough storage space to save the images. It is recommended to mount an external storage device via NFS. ##### Video Capture Use the following command to capture camera images and save them in NV12 format at 1080p resolution, capturing 25 frames. Please ensure that the data write speed is faster than the camera's capture speed; otherwise, please lower the camera's resolution. ```bash v4l2-ctl --device=/dev/video1 --set-fmt-video=width=1920,height=1080,pixelformat=NV12 --stream-mmap --stream-count=25 --stream-to=output.yuv ``` Where: - `--device=/dev/video1`: Specifies the camera device, which can be adjusted according to actual conditions. - `--set-fmt-video=width=1920,height=1080,pixelformat=NV12`: Sets the video format to 1080p resolution and NV12 pixel format. - `--stream-mmap`: Uses memory-mapped mode for video stream processing. - `--stream-count=25`: Specifies capturing 25 frames of images. - `--stream-to=output.yuv`: Saves the captured images to the `output.yuv` file. ##### Video Encoding Use the following command to encode the captured NV12 format images into an H.264 format video file via the VPU: ```bash mvx_encoder --dev /dev/video0 -i yuv420_nv12 -w 1920 -h 1080 -o h264 ./output.yuv ./test.264 #mvx_encoder --dev /dev/video0 -i yuv420_nv12 -w 1920 -h 1080 -o h265 ./output.yuv ./test.265 #mvx_encoder --dev /dev/video0 -i yuv420_nv12 -w 1920 -h 1080 -o jpeg ./output.yuv ./test.jpg ``` Where: - `--dev /dev/video0`: Specifies the VPU device. - `-i yuv420_nv12`: Specifies the input image format as NV12. - `-w 1920 -h 1080`: Sets the image resolution to 1920x1080. - `-o h264`: Specifies the output video format as H.264. - `./output.yuv`: The input NV12 format image file. - `./test.264`: The output H.264 format video file. After executing the above command, the 25 frames of NV12 format images captured by the camera will be encoded into the H.264 format video file `test.264`. This file can be copied to a PC and played with a player (such as PotPlayer) to verify the encoding effect. #### Encoding Using Local Files If you have local NV12 format image files, you can also use VPU for encoding. Simply replace the input file path in the above command with the local file path. Through these steps, you can easily use VPU to encode NV12 format images, regardless of whether the image source is a camera or a local file. ### Video Decoding In this section, we will demonstrate how to use the VPU (Video Processing Unit) to decode NV12 format H.264 files. The image source can be files generated from the video encoding chapter, or files you have prepared yourself. #### Decoder Demo Introduction `mvx_decoder` is a command-line video decoding tool that leverages the VPU (Video Processing Unit) for hardware acceleration. It supports decoding formats such as H.264, H.265, and JPEG, and interacts with hardware devices through the V4L2 (Video for Linux 2) interface to achieve efficient decoding performance. The source code is located at `k230_linux_sdk/buildroot-overlay/package/mvx_player`. Use the following command to decode a video file into images: ```bash mvx_decoder --dev -i -o ``` Where: - `--dev `: Specifies the VPU device path. - `-i `: Specifies the input image format. - `-o `: Specifies the output image format. - ``: The input video file. - ``: The output image file. After executing the above command, the input video file `` will be decoded into an image file `` of the specified format. In addition to basic usage, `mvx_decoder` also supports advanced features such as adjusting decoding parameters. Use the following command to view the complete parameter list: ```bash mvx_decoder -help ``` #### Video Decoding Before decoding, please first confirm the VPU device. For details, refer to the [Finding VPU Device](#find-vpu-device) section. Use the following command to decode an H.264 format video file into an NV12 format image file: ```bash mvx_decoder --dev /dev/video0 -i h264 -o yuv420_nv12 test.264 output.yuv ``` Where: - `--dev /dev/video0`: Specifies the VPU device. - `-i h264`: Specifies the input video format as H.264. - `-o yuv420_nv12`: Specifies the output image format as NV12. - `test.264`: The input H.264 format video file. - `output.yuv`: The output NV12 format image file. After executing the above command, the input H.264 format video file `test.264` will be decoded into an NV12 format image file `output.yuv`. You can copy this file to a PC and use a player that supports NV12 format (such as YUVView) to verify the decoding result. ## ALSA Audio Usage Guide ### audio demo Introduction `audio_demo` is a sample program that runs on the K230 development board. It calls the ALSA (Advanced Linux Sound Architecture) API to implement audio recording and playback functions. This sample program not only demonstrates how to perform audio processing on the K230 development board, but also provides reference code for developers using the ALSA API interface to implement audio functions. #### Recording WAV Files Use the following command to record an audio file in WAV format: ```sh audio_demo -type 2 -filename test.wav -channels 2 -samplerate 16000 ``` Parameter description: - `-type 2`: Specifies the operation type as recording. - `-filename`: Sets the output file name. - `-channels`: Number of recording channels, `2` for stereo. - `-samplerate`: Sets the sample rate, in Hz. After recording is complete, the audio file is saved as `test.wav` in the current directory. #### Playing WAV Files Use the following command to play the recorded WAV file: ```sh audio_demo -type 0 -filename test.wav ``` Parameter description: - `-type 0`: Specifies the operation type as playing a WAV file. - `-filename`: The name of the file to play. #### Playing MP3 Files `audio_demo` also supports playing audio files in MP3 format: ```sh audio_demo -type 1 -filename example.mp3 ``` Parameter description: - `-type 1`: Specifies the operation type as playing an MP3 file. - `-filename`: The name of the MP3 file to play. #### Voice Capture and Playback `audio_rec_play` is a sample program for voice capture and playback. It simulates the voice call scenario on the K230 side and uses `audio3a` to enhance voice quality. During the first 30 seconds, it starts recording and plays a local file (which can be connected to an external speaker or amplifier device through the headphone interface). During the last 30 seconds, it plays back the recorded voice to check the sound effect. Use the following command to run `audio_rec_play`: ```sh audio_rec_play ``` By default, the program plays the `/usr/bin/audio.pcm` file. The program execution flow is as follows: 1. First 30 seconds: Start recording and play the local file `/usr/bin/audio.pcm`. 1. Last 30 seconds: Stop playing the local file, and play back the voice recorded during the first 30 seconds to check the sound effect. Through this sample program, users can experience the voice capture and playback functions on the K230 side, and verify the voice quality enhancement effect of `audio3a`. ## Multimedia Processing Using FFmpeg In this section, we will demonstrate how to use FFmpeg to invoke the VPU for hardware encoding and decoding, and how to use FFmpeg to invoke the ALSA interface for audio recording and playback. ### Capturing Camera Video and Encoding Use the following command to capture camera video via FFmpeg and invoke the VPU to encode it as an MP4 file: ```bash ffmpeg -f v4l2 -input_format nv12 -r 30 -s 1920x1080 -i /dev/video1 -c:v h264_v4l2m2m -b:v 4M -vsync passthrough -y test.mp4 ``` Where: - `-f v4l2`: Specifies the video input format as V4L2. - `-input_format nv12`: Specifies the input image format as NV12. - `-r 30`: Sets the frame rate to 30fps. - `-s 1920x1080`: Specifies the resolution as 1920x1080. - `-i /dev/video1`: Specifies the camera device path. - `-c:v h264_v4l2m2m`: Uses the VPU for H.264 encoding. - `-b:v 4M`: Sets the video bitrate to 4Mbps. - `-vsync passthrough`: Synchronizes video frames. - `-y test.mp4`: The output file name is `test.mp4`. `h264_v4l2m2m/hevc_v4l2m2m/mjpeg_v4l2m2m` is a hardware-accelerated video encoder in FFmpeg, which leverages the V4L2 (Video for Linux 2) and M2M (Memory-to-Memory) interfaces to achieve efficient video encoding. By interacting with hardware devices, it provides higher encoding performance and efficiency. After executing the above command, the video captured by the camera will be encoded in H.264 format and saved as the `test.mp4` file. ### Capturing Camera Images Use the following command to capture camera images via FFmpeg and invoke the VPU to encode them as JPEG files: ```bash ffmpeg -f v4l2 -input_format nv12 -s 1920x1080 -i /dev/video1 -vframes 10 -vf format=nv12 -c:v mjpeg_v4l2m2m -vsync passthrough -y output_%03d.jpg ``` Where: - `-f v4l2`: Specifies the video input format as V4L2. - `-input_format nv12`: Specifies the input image format as NV12. - `-s 1920x1080`: Specifies the resolution as 1920x1080. - `-i /dev/video1`: Specifies the camera device path. - `-vframes 10`: Sets the number of captured frames to 10. - `-vf format=nv12`: Sets the image format to NV12. - `-c:v mjpeg_v4l2m2m`: Uses the VPU for JPEG encoding. - `-vsync passthrough`: Synchronizes video frames. - `-y output_%03d.jpg`: The output file name format is `output_001.jpg`, `output_002.jpg`, etc. After executing the above command, the images captured by the camera will be encoded in JPEG format and saved as multiple files, such as `output_001.jpg`, `output_002.jpg`, etc. ### Capturing Camera Images and Saving as MJPEG Files Use the following command to capture camera images via FFmpeg and invoke the VPU to encode them as MJPEG files: ```bash ffmpeg -f v4l2 -input_format nv12 -s 1920x1080 -i /dev/video1 -vframes 100 -vf format=nv12 -c:v mjpeg_v4l2m2m -vsync passthrough -y output.mjpeg ``` Where: - `-f v4l2`: Specifies the video input format as V4L2. - `-input_format nv12`: Specifies the input image format as NV12. - `-s 1920x1080`: Specifies the resolution as 1920x1080. - `-i /dev/video1`: Specifies the camera device path. - `-vframes 100`: Sets the number of captured frames to 100. - `-vf format=nv12`: Sets the image format to NV12. - `-c:v mjpeg_v4l2m2m`: Uses the VPU for MJPEG encoding. - `-vsync passthrough`: Synchronizes video frames. - `-y output.mjpeg`: The output file name is `output.mjpeg`. After executing the above command, the images captured by the camera will be encoded in MJPEG format and saved as the `output.mjpeg` file. ### Decoding Video and Saving as Images Use the following command to decode an H.264 video file via FFmpeg and save it as NV12 format images: ```bash ffmpeg -c:v h264_v4l2m2m -i test.mp4 -pix_fmt nv12 -f rawvideo -vsync passthrough -y output.yuv ``` Where: - `-c:v h264_v4l2m2m`: Uses the VPU for H.264 decoding. - `-i input.mp4`: Specifies the input video file. - `-pix_fmt nv12`: Specifies the output image format as NV12. - `-f rawvideo`: Specifies the output format as raw video stream. - `output.yuv`: The output file name is `output.yuv`. After executing the above command, the input H.264 video file `input.mp4` will be decoded into an NV12 format image file `output.yuv`. This file can be copied to a PC and viewed using a player that supports the NV12 format (such as YUVView) to verify the decoding effect. ### Recording Audio Use the following command to invoke the ALSA interface via FFmpeg to record audio and save it as a WAV file: ```bash ffmpeg -f alsa -i hw:0 -t 30 -ac 2 -ar 44100 -y output.wav ``` Where: - `-f alsa`: Specifies the audio input format as ALSA. - `-i hw:0`: Specifies the audio input device; `hw:0` represents the default audio device. - `-t 30`: Sets the recording duration to 30 seconds. - `-ac 2`: Sets the number of recording channels to 2 (stereo). - `-ar 44100`: Sets the sample rate to 44100Hz. - `-y output.wav`: The output file name is `output.wav`. After executing the above command, the recorded audio will be saved as a WAV format audio file `output.wav`. ### Playing Audio Use the following command to invoke the ALSA interface via FFmpeg to play the recorded WAV file: ```bash ffmpeg -f wav -i output.wav -f alsa hw:0 ``` Where: - `-f wav`: Specifies the audio input format as WAV. - `-i output.wav`: Specifies the input WAV file. - `-f alsa`: Specifies the audio output format as ALSA. - `hw:0`: Specifies the audio output device; `hw:0` represents the default audio device. After executing the above command, the recorded WAV file `output.wav` will be played through the ALSA interface. ### Adjusting Volume #### Adjusting Recording Volume Use the following command to invoke the ALSA interface via FFmpeg to record audio and adjust the capture volume: ```bash ffmpeg -f alsa -i hw:0 -t 30 -ac 2 -ar 44100 -filter:a "volume=2.0" -y output.wav ``` Where: - `-f alsa`: Specifies the audio input format as ALSA. - `-i hw:0`: Specifies the audio input device; `hw:0` represents the default audio device. - `-t 30`: Sets the recording duration to 30 seconds. - `-ac 2`: Sets the number of recording channels to 2 (stereo). - `-ar 44100`: Sets the sample rate to 44100Hz. - `-filter:a "volume=2.0"`: Adjusts the capture volume; `2.0` means doubling the volume. - `-y output.wav`: The output file name is `output.wav`. After executing the above command, the recorded audio will be saved as a WAV format audio file `output.wav`, and the volume will be doubled. #### Adjusting Playback Volume Use the following command to invoke the ALSA interface via FFmpeg to play audio and adjust the playback volume: ```bash ffmpeg -f wav -i output.wav -filter:a "volume=2.0" -f alsa hw:0 ``` Where: - `-f wav`: Specifies the audio input format as WAV. - `-i output.wav`: Specifies the input WAV file. - `-filter:a "volume=2.0"`: Adjusts the playback volume; `2.0` means doubling the volume. - `-f alsa`: Specifies the audio output format as ALSA. - `hw:0`: Specifies the audio output device; `hw:0` represents the default audio device. After executing the above command, the WAV file `output.wav` will be played through the ALSA interface, and the volume will be doubled. #### Muting The following command will use FFmpeg to invoke ALSA to play the audio file `output.wav` with muted output, using the hardware device `hw:0`. ```sh ffmpeg -i output.wav -f alsa -ac 2 -ar 44100 -vol 0 hw:0 ``` Where: - `-i output.wav`: Specifies the input file as `output.wav`. - `-f alsa`: Specifies the output format as ALSA. - `-ac 2`: Specifies the number of audio channels as 2. - `-ar 44100`: Specifies the audio sample rate as 44100 Hz. - `-vol 0`: Sets the volume to 0, achieving muted playback. - `hw:0`: Specifies the use of hardware device `hw:0`. ## Camera RTSP Streaming Demo `camera_rtsp_demo` is a sample program used to transmit camera video streams via the RTSP protocol. This program can encode the video captured by the camera into H.264 or H.265 format and transmit it in real time via the RTSP protocol. It demonstrates how to use the FFmpeg API to obtain camera data, overlay OSD, and perform encoding, as well as how to use the live555 API to output an RTSP stream. ### Usage Use the following command to run `camera_rtsp_demo`: ```sh ./camera_rtsp_demo [-H] [-t ] [-w ] [-h ] [-b ] ``` Parameter description: - `-H`: Display help information. - `-t `: Specify the video encoding type, optional values are `h264` or `h265`, the default value is `h264`. - `-w `: Specify the video encoding width, the default value is `1280`. - `-h `: Specify the video encoding height, the default value is `720`. - `-b `: Specify the video encoding bitrate (kbps), the default value is `2000`. - `-r `: Specify the number of OSD overlay regions, the default value is `0`. Example command: ```sh ./camera_rtsp_demo -t h264 -w 1920 -h 1080 -b 4000 -r 2 ``` After executing the above command, the program will start the camera and encode the video stream in H.264 format, with a resolution of 1920x1080, a bitrate of 4000 kbps, and 2 OSD overlay regions, and transmit it via the RTSP protocol. Find the RTSP URL in the serial output, for example: "Play this stream using the URL: rtsp://:8554/test", and use a player that supports the RTSP protocol (such as VLC) to open the URL to view the video stream.