Individual cameras on CSI1 and CSI2 work fine in sample_vicap, but freeze if both used at the same time.

Viewed 83

Hello, I have two OVM6211 cameras connected to CSI1 and CSI2 of a CanMV-K230-V1.1 board. The sd card has the k230_sdk image most likely version 1.9, compiled from source with my custom camera drivers, and I followed the instructions for adding new camera sensors.

When I run

./sharefs/app/sample_vicap.elf -mode 1 -conn 1 -dev 0 -sensor 81 -chn 0 -ow 400 -oh 400

the output looks perfect. This is with the camera cofigured for 400x400 at 120fps. I also have tested the same command with sensor 82, which is the other CSI port, and the same result. However, when I run

./sharefs/app/sample_vicap.elf -mode 1 -conn 1 -dev 0 -sensor 81 -chn 0 -ow 400 -oh 400 -dev 1 -sensor 82 -chn 0 -ow 400 -oh 400

I get two frozen frames, with some visual artifacts.

Things I have tried:

  • Just in case it was a framerate issue, I added sensors 83 and 84 with 400x400 at 60fps, but the same result as above. Only difference is that now the two frozen frames have white noise and lots of artifacts
  • I checked for ISP bandwidth limitation, the specs say it can do 8MP@30fps, or 240MP/s. My two cameras are 2 * 400*400 * 120 = 38.4MP/s so I think it should be fine. 60fps should be even easier.
  • Not sure if it matters but all ISP stages are disabled and bypassed

I am not sure what to try next, can someone please suggest some options? Thank you.

5 Answers

能发一下你们测试出来的效果图吗,最好是提供单独一路的效果和两路运行的效果,我需要看一下现象来猜测是哪块出现的问题。
或者你可以先尝试关闭3dnr 试试,运行命令如下:
./sharefs/app/sample_vicap.elf -mode 1 -conn 1 -dev 0 -sensor 81 -dnr 0 -chn 0 -ow 400 -oh 400 -dev 1 -sensor 82 -dnr 0 -chn 0 -ow 400 -oh 400

Hello, thank you for the reply. Turning off 3dnr gave exactly the same output, but I think this is expected because my ISP tuning file should disable 3dnr.
Here is the single camera view pointed at my table surface (it responds to motion fine):

Next, here is dual camera normal and then also with a closeup of the second camera to show the pattern better (the screen immediately freezes and does not respond to motion, but sample_vicap is still responsive and I can press 'q' to quit):

I have been very carefully studying the datasheet for these cameras, and also researching the MIPI CSI protocol, and it seems that many SOCs have some limitations on the D-phy interface especially when running multiple cameras. I could not find any info about the K230 D-phy interface requirements, would you be able to share more details so I can make sure that the camera is configured to be compatible?

Also after enabling debug messages in the camera driver, I am noticing that at lower framerates the ISP is pulling maybe 4 or 5 frames instead of only one before freezing. This makes me wonder if maybe there is some clock configuration that is not correct for the SOC.

Please let me know if it would help for me to post the camera driver code, I am pretty sure it is fine because the OV7251 driver I wrote is able to stream two cameras at once, but I am offering just in case.

Here are some additional test results:
Part of the testing was to create a minimal implementation of sample_vicap streaming two cameras, the code is here:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <errno.h>
#include <signal.h>

#include "k_module.h"
#include "k_type.h"
#include "k_vb_comm.h"
#include "k_vicap_comm.h"
#include "k_video_comm.h"
#include "k_sys_comm.h"
#include "mpi_vb_api.h"
#include "mpi_vicap_api.h"
#include "mpi_isp_api.h"
#include "mpi_sys_api.h"
#include "k_vo_comm.h"
#include "mpi_vo_api.h"
#include "k_connector_comm.h"
#include "mpi_connector_api.h"
#include "mpi_sensor_api.h"

#define VICAP_OUTPUT_BUF_NUM 5
#define VICAP_INPUT_BUF_NUM 3
#define NUM_CAMERAS 2

static volatile int keep_running = 1;
void int_handler(int dummy) { keep_running = 0; }

static void vb_exit() {
    kd_mpi_vb_exit();
}

int main() {
    signal(SIGINT, int_handler);
    k_s32 ret = 0;

    k_connector_type connector_type = LT9611_MIPI_4LAN_1920X1080_60FPS;
    k_connector_info connector_info;
    memset(&connector_info, 0, sizeof(connector_info));
    ret = kd_mpi_get_connector_info(connector_type, &connector_info);
    if (ret) { printf("Failed to get connector info\n"); return ret; }
    k_s32 connector_fd = kd_mpi_connector_open(connector_info.connector_name);
    if (connector_fd < 0) { printf("Failed to open connector\n"); return 1; }
    kd_mpi_connector_power_set(connector_fd, 1);
    kd_mpi_connector_init(connector_fd, connector_info);

    // --- Camera/sensor setup ---
    k_vicap_sensor_type sensor_types[NUM_CAMERAS] = {81, 82};
    k_vicap_sensor_info sensor_infos[NUM_CAMERAS];
    k_vicap_dev_attr dev_attrs[NUM_CAMERAS];
    int dev_ids[NUM_CAMERAS] = {0, 1};
    int chn_ids[NUM_CAMERAS] = {0, 0}; // always using channel 0 on each dev

    memset(sensor_infos, 0, sizeof(sensor_infos));
    memset(dev_attrs, 0, sizeof(dev_attrs));

    for (int i = 0; i < NUM_CAMERAS; ++i) {
        sensor_infos[i].sensor_type = sensor_types[i];
        ret = kd_mpi_vicap_get_sensor_info(sensor_types[i], &sensor_infos[i]);
        if (ret) { printf("Failed to get sensor info for sensor %d\n", sensor_types[i]); return ret; }

        dev_attrs[i].input_type = VICAP_INPUT_TYPE_SENSOR;
        dev_attrs[i].acq_win.h_start = 0;
        dev_attrs[i].acq_win.v_start = 0;
        dev_attrs[i].acq_win.width = sensor_infos[i].width;
        dev_attrs[i].acq_win.height = sensor_infos[i].height;
        dev_attrs[i].mode = VICAP_WORK_OFFLINE_MODE;
        dev_attrs[i].buffer_num = VICAP_INPUT_BUF_NUM;
        dev_attrs[i].buffer_size = VICAP_ALIGN_UP((sensor_infos[i].width * sensor_infos[i].height * 2), VICAP_ALIGN_1K);
        // dev_attrs[i].pipe_ctrl.data = 0xFFFFFFFF;
        dev_attrs[i].pipe_ctrl.data = 0x00000000;
        dev_attrs[i].pipe_ctrl.bits.ae_enable = 0;
        dev_attrs[i].pipe_ctrl.bits.awb_enable = 0;
        dev_attrs[i].pipe_ctrl.bits.dnr3_enable = 0;
        dev_attrs[i].pipe_ctrl.bits.ahdr_enable = 0;
        dev_attrs[i].cpature_frame = 0;
        dev_attrs[i].dw_enable = 0;
        dev_attrs[i].mirror = VICAP_MIRROR_NONE;
        memcpy(&dev_attrs[i].sensor_info, &sensor_infos[i], sizeof(sensor_infos[i]));
        ret = kd_mpi_vicap_set_dev_attr(dev_ids[i], dev_attrs[i]);
        if (ret) { printf("Failed to set dev attr for dev %d\n", dev_ids[i]); return ret; }
    }

    // --- Video Buffer setup ---
    k_vb_config vb_config;
    memset(&vb_config, 0, sizeof(vb_config));
    vb_config.max_pool_cnt = NUM_CAMERAS * 2;
    for (int i = 0; i < NUM_CAMERAS; ++i) {
        // Input pool (for RAW)
        vb_config.comm_pool[i * NUM_CAMERAS].blk_cnt = VICAP_INPUT_BUF_NUM;
        vb_config.comm_pool[i * NUM_CAMERAS].mode = VB_REMAP_MODE_NOCACHE;
        vb_config.comm_pool[i * NUM_CAMERAS].blk_size = dev_attrs[i].buffer_size;

        // Output pool (for YUV)
        vb_config.comm_pool[i * NUM_CAMERAS + 1].blk_cnt = VICAP_OUTPUT_BUF_NUM;
        vb_config.comm_pool[i * NUM_CAMERAS + 1].mode = VB_REMAP_MODE_NOCACHE;
        vb_config.comm_pool[i * NUM_CAMERAS + 1].blk_size = VICAP_ALIGN_UP((sensor_infos[i].width * sensor_infos[i].height * 3 / 2), VICAP_ALIGN_1K);
    }
    ret = kd_mpi_vb_set_config(&vb_config);
    if (ret) { printf("Failed to set vb config\n"); return ret; }

    k_vb_supplement_config supplement_config;
    memset(&supplement_config, 0, sizeof(supplement_config));
    supplement_config.supplement_config |= VB_SUPPLEMENT_JPEG_MASK;
    ret = kd_mpi_vb_set_supplement_config(&supplement_config);
    if (ret) { printf("Failed to set vb supplement config\n"); return ret; }

    ret = kd_mpi_vb_init();
    if (ret) { printf("Failed to init vb\n"); return ret; }
    atexit(vb_exit);

    // --- VICAP channel attribute setup and binding ---
    k_vicap_chn_attr chn_attrs[NUM_CAMERAS];
    k_mpp_chn vicap_mpp_chn[NUM_CAMERAS], vo_mpp_chn[NUM_CAMERAS];
    k_vo_layer vo_layers[NUM_CAMERAS] = { K_VO_LAYER1, K_VO_LAYER2 };
    int vo_chn_ids[NUM_CAMERAS] = { K_VO_DISPLAY_CHN_ID1, K_VO_DISPLAY_CHN_ID2 };

    for (int i = 0; i < NUM_CAMERAS; ++i) {
        kd_mpi_vicap_set_dump_reserved(dev_ids[i], chn_ids[i], K_TRUE);
        memset(&chn_attrs[i], 0, sizeof(chn_attrs[i]));
        chn_attrs[i].out_win.width = sensor_infos[i].width;
        chn_attrs[i].out_win.height = sensor_infos[i].height;
        chn_attrs[i].crop_win.width = sensor_infos[i].width;
        chn_attrs[i].crop_win.height = sensor_infos[i].height;
        chn_attrs[i].scale_win = chn_attrs[i].out_win;
        chn_attrs[i].crop_enable = K_FALSE;
        chn_attrs[i].scale_enable = K_FALSE;
        chn_attrs[i].chn_enable = K_TRUE;
        chn_attrs[i].pix_format = PIXEL_FORMAT_YUV_SEMIPLANAR_420;
        chn_attrs[i].buffer_num = VICAP_OUTPUT_BUF_NUM;
        chn_attrs[i].buffer_size = vb_config.comm_pool[i * NUM_CAMERAS + 1].blk_size;
        chn_attrs[i].fps = 0;
        ret = kd_mpi_vicap_set_chn_attr(dev_ids[i], chn_ids[i], chn_attrs[i]);
        if (ret) { printf("Failed to set chn attr for dev %d\n", dev_ids[i]); return ret; }

        vicap_mpp_chn[i].mod_id = K_ID_VI;
        vicap_mpp_chn[i].dev_id = dev_ids[i];
        vicap_mpp_chn[i].chn_id = chn_ids[i];
        vo_mpp_chn[i].mod_id = K_ID_VO;
        vo_mpp_chn[i].dev_id = K_VO_DISPLAY_DEV_ID;
        vo_mpp_chn[i].chn_id = vo_chn_ids[i];
        ret = kd_mpi_sys_bind(&vicap_mpp_chn[i], &vo_mpp_chn[i]);
        if (ret) { printf("Failed to bind sys for dev %d\n", dev_ids[i]); return ret; }
    }

    // --- VO layer setup ---
    for (int i = 0; i < NUM_CAMERAS; ++i) {
        k_vo_video_layer_attr layer_attr;
        memset(&layer_attr, 0, sizeof(layer_attr));
        layer_attr.display_rect.x = i * 400;
        layer_attr.display_rect.y = 0;
        layer_attr.img_size.width = sensor_infos[i].width;
        layer_attr.img_size.height = sensor_infos[i].height;
        layer_attr.pixel_format = PIXEL_FORMAT_YVU_PLANAR_420;
        layer_attr.stride = (layer_attr.img_size.width / 8 - 1) + ((layer_attr.img_size.height - 1) << 16);
        layer_attr.func = K_ROTATION_0;
        ret = kd_mpi_vo_set_video_layer_attr(vo_layers[i], &layer_attr);
        if (ret) { printf("Failed to set vo layer attr %d\n", vo_layers[i]); return ret; }
        ret = kd_mpi_vo_enable_video_layer(vo_layers[i]);
        if (ret) { printf("Failed to enable vo layer %d\n", vo_layers[i]); return ret; }
    }
    ret = kd_mpi_vo_enable();
    if (ret) { printf("Failed to enable vo\n"); return ret; }

    // --- Start streaming for both cameras ---
    for (int i = 0; i < NUM_CAMERAS; ++i) {
        ret = kd_mpi_vicap_init(dev_ids[i]);
        if (ret) { printf("Failed to vicap init for dev %d\n", dev_ids[i]); return ret; }
    }
    
    for (int i = 0; i < NUM_CAMERAS; ++i) {
        ret = kd_mpi_vicap_start_stream(dev_ids[i]);
        if (ret) { printf("Failed to start stream for dev %d\n", dev_ids[i]); return ret; }
    }

    printf("Streaming 2 cameras. Press Ctrl-C to quit.\n");
    while (keep_running) {
        sleep(1);
    }

    // --- Cleanup ---
    for (int i = 0; i < NUM_CAMERAS; ++i) {
        kd_mpi_vicap_stop_stream(dev_ids[i]);
        kd_mpi_vicap_deinit(dev_ids[i]);
        kd_mpi_vo_disable_video_layer(vo_layers[i]);
        kd_mpi_sys_unbind(&vicap_mpp_chn[i], &vo_mpp_chn[i]);
    }

    return 0;
}

The cameras are configured for 400x400@120fps 10-bit raw mode, and the first time I run the program after power cycle I see the following result:

If I exit and run the program agian, I usually get two good frames, but they are still frozen:

nano,您好,了解下咱们这边有涉及到MIPI这块的测试验证吗

Hello, sorry but unfortunately I do not have access to the very specialized equipment to do MIPI CSI peripheral verification. I did acquire a 1GHz oscilloscope which has allowed me to inspect the waveform and compare it to a working camera in the attempt to find the difference, but I do not see any problems.

To give an update on the original thread, I have purchased some OV5647 camera modules to use for now so that project development can continue, since they are confirmed to work with simultaneous streaming. I would very much appreciate any assistance with the OVM6211 camera sensor integration though, as they are much more suitable for the application.