# K230 linux SENSOR Porting Guide This document mainly describes the basic framework of the Camera Sensor on the K230 platform and how to add a new Camera Sensor. The K230 platform only supports sensors with MIPI interface type. We will use the most commonly used MIPI CSI interface Sensor as an example for explanation. The hardware connection diagram between the Sensor and the main control platform is as follows: ![Text, letter description automatically generated](https://www.kendryte.com/api/post/attachment?id=464) The main control sends configuration registers through the I2C interface to control the working mode of the sensor, and the sensor sends image data to the main control SOC through the MIPI CSI interface. ## Sensor Adaptation Preparation Before adapting a new sensor, users need to do the following preparations: 1. Obtain the Sensor datasheet and initialization sequence from official channels. 1. Review the Sensor datasheet and related application manuals, paying special attention to the Sensor exposure and gain control methods, the limitations of exposure and gain in different modes, the implementation method of long exposure, the black level value, the bayer data output order, etc. 1. Request the required mode initialization sequences from the Sensor manufacturer, and understand the data rate of each sequence, the total output width and height of the Sensor, and the exact frame rate, etc. 1. Confirm whether the Sensor control interface is I2C, SPI, or other interfaces. The device address of the Sensor can be set through hardware. For scenarios with multiple cameras, try not to multiplex the I2C bus for sensors. The Sensor IO voltage level in different scenarios may be 1.8V or 3.3V. During design, it is necessary to ensure that the IO voltage level of the sensor is consistent with the GPIO power supply voltage of the corresponding SOC. If the sensor voltage level is 1.8V, the GPIO used for sensor control, such as Reset, I2C, PRDN, need to use 1.8V power supply. 1. Confirm the Sensor output data interface and protocol type. Currently, only MIPI CSI interface is supported. 1. Confirm whether the wdr mode is VC, DT, or DOL, etc. 1. Confirm whether image cropping is required for different timings. ## Sensor Adaptation Example This section will provide a detailed description of the steps to add support for a new camera sensor. Here we use the ov5647 driver as an example to illustrate. The corresponding driver file source code path is as follows: ```shell k230_linux_sdk/buildroot-overlay/package/vvcam/src/ ``` ### Define Supported Sensor Types View the default configuration of the current linux sensor ```shell [root@canaan /sharefs/isp_linux ]#cat /proc/vsi/isp_subdev0 /******sensor configuration******/ isp0 port0: sensor : ov5647 mode : 0 xml : /etc/vvcam/ov5647.xml manu_json: /etc/vvcam/ov5647.manual.json auto_json: /etc/vvcam/ov5647.auto.json ********************************* ``` It can be seen that the current default sensor is ov5647, and the corresponding configuration files are ov5647.manual.json, ov5647.xml, ov5647.auto.json in the /etc/vvcam/ directory. ### Sensor Driver Adaptation Sensor driver adaptation is the most important part of the entire process. Users can modify by copying the existing sensor driver files. The AE-related register configuration and calculation methods of the sensor need to refer to the corresponding manuals or seek professional assistance. #### Define the Sensor Register Configuration List The sensor register configuration is defined by the data type k_sensor_reg_list: ```c struct reg_list { uint16_t addr; uint8_t value; }; ``` The following is the register configuration list of ov5647 ```c static const reg_list ov5647_1920x1080_30fps[] = { //pixel_rate = 81666700 {0x0103, 0x01}, {0x0100, 0x00}, {0x3034, 0x1a}, {0x3035, 0x21}, ... {0x3501, 0x02}, {0x3502, 0xa0}, {0x3503, 0x07}, {0x350b, 0x10}, {0, 0x00}, }; ``` #### Define the Modes Supported by the Sensor The mode parameters of the sensor are defined by the data type ov5647_mode ```c struct ov5647_mode { struct vvcam_sensor_mode mode; struct reg_list* regs; }; ``` The following are the modes supported by ov5647 ```c static struct ov5647_mode modes[] = { { .mode = { .clk = 25000000, .width = 1920, .height = 1080, .lanes = VVCAM_SENSOR_2LANE, .freq = VVCAM_SENSOR_800M, .bayer = VVCAM_BAYER_PAT_GBRG, .bit_width = 10, .ae_info = { .frame_length = 1199, .cur_frame_length = 1199, .one_line_exp_time = 0.000027808, .gain_accuracy = 1024, .min_gain = 1.0, .max_gain = 8.0, .int_time_delay_frame = 2, .gain_delay_frame = 2, .color_type = 0, .integration_time_increment = 0.000027808, .gain_increment = (1.0f/16.0f), .max_long_integraion_line = 1199 - 12, .min_long_integraion_line = 2, .max_integraion_line = 1199 - 12, .min_integraion_line = 2, .max_long_integraion_time = 0.000027808 * (1199 - 12), .min_long_integraion_time = 0.000027808 * 2, .max_integraion_time = 0.000027808 * (1199 - 12), .min_integraion_time = 0.000027808 * 2, .cur_long_integration_time = 0.0, .cur_integration_time = 0.0, .cur_long_again = 0.0, .cur_long_dgain = 0.0, .cur_again = 0.0, .cur_dgain = 0.0, .a_long_gain.min = 1.0, .a_long_gain.max = 8.0, .a_long_gain.step = (1.0f/16.0f), .a_gain.min = 1.0, .a_gain.max = 8.0, .a_gain.step = (1.0f/16.0f), .d_long_gain.max = 1.0, .d_long_gain.min = 1.0, .d_long_gain.step = (1.0f/1024.0f), .d_gain.max = 1.0, .d_gain.min = 1.0, .d_gain.step = (1.0f/1024.0f), .cur_fps = 30, } }, .regs = ov5647_1920x1080_30fps } }; ``` #### Implement Sensor Operation Interfaces The sensor operation interfaces are defined by the data type k_sensor_function. Users implement the relevant operation interfaces according to the actual situation. Not all interfaces must be implemented. ```c struct vvcam_sensor_ctrl { int (*init)(void** ctx); void (*deinit)(void* ctx); int (*enum_mode)(void* ctx, uint32_t index, struct vvcam_sensor_mode* mode); int (*get_mode)(void* ctx, struct vvcam_sensor_mode* mode); int (*set_mode)(void* ctx, uint32_t index); int (*set_stream)(void* ctx, bool on); int (*set_analog_gain)(void* ctx, float gain); int (*set_digital_gain)(void* ctx, float gain); int (*set_int_time)(void* ctx, float time); }; ``` The following are the modes supported by ov5647 ```c struct vvcam_sensor vvcam_ov5647 = { .name = "ov5647", .ctrl = { .init = init, .deinit = deinit, .enum_mode = enum_mode, .get_mode = get_mode, .set_mode = set_mode, .set_stream = set_stream, .set_analog_gain = set_analog_gain, .set_digital_gain = set_digital_gain, .set_int_time = set_int_time } }; ``` #### Update the Sensor Driver List Add the sensor driver structure defined in the previous section to the vvcam_sensor_init function in lib.c. The list of sensors currently supported by the system is as follows: ```c void vvcam_sensor_init(void) { // get /dev/media0 printf("k230 builtin sensor driver, built %s %s, API version %lu\n", __DATE__, __TIME__, VVCAM_API_VERSION); vvcam_sensor_add(&vvcam_ov5647); vvcam_sensor_add(&vvcam_imx335); vvcam_sensor_add(&vvcam_gc2093); } ``` ## Compile sensor After reconfiguring the new sensor as described above, the sensor needs to be recompiled. The compilation command is as follows: ```shell make vvcam-reconfigure ``` After compilation is complete, a new `libvvcam.so` will be generated at the following path: ```shell k230_linux_sdk/output/k230_canmv_defconfig/target/usr/lib/libvvcam.so ``` ## Run sensor Replace the new `libvvcam.so` to the development board's `/usr/lib` directory, then modify the corresponding configuration files and sensor. The configuration commands are as follows. First, kill `isp_media_server` with the following command: ```shell killall isp_media_server ``` Check the default configuration. The default configuration is as follows: ```shell [root@canaan /sharefs/isp_linux ]#cat /proc/vsi/isp_subdev0 /******sensor configuration******/ isp0 port0: sensor : ov5647 mode : 0 xml : /etc/vvcam/ov5647.xml manu_json: /etc/vvcam/ov5647.manual.json auto_json: /etc/vvcam/ov5647.auto.json ********************************* ``` Configure the new sensor and configuration file: ```shell echo 0 sensor=imx335 > /proc/vsi/isp_subdev0 echo 0 mode=0 > /proc/vsi/isp_subdev0 echo 0 xml=/etc/vvcam/imx335.xml > /proc/vsi/isp_subdev0 echo 0 manu_json=/etc/vvcam/imx335.xml > /proc/vsi/isp_subdev0 echo 0 auto_json=/etc/vvcam/imx335.xml > /proc/vsi/isp_subdev0 ``` After configuration is complete, check the configuration file again: ```shell [root@canaan /sharefs/isp_linux ]#cat /proc/vsi/isp_subdev0 /******sensor configuration******/ isp0 port0: sensor : imx335 mode : 0 xml : /etc/vvcam/imx335.xml manu_json: /etc/vvcam/imx335.xml auto_json: /etc/vvcam/imx335.xml ********************************* ``` You can see that both the sensor type and configuration files have been switched to the configuration files and parameters we want. Re-execute `isp_media_server`. The command is as follows: ```shell ISP_MEDIA_SENSOR_DRIVER=/usr/lib/libvvcam.so /usr/bin/isp_media_server > /dev/null 2> /tmp/isp.err.log & ``` Check the video devices, and you will find multiple video devices: ```shell [root@canaan /sharefs/isp_linux ]#ls /dev/video video0 video1 video2 video3 video4 ``` Run the test command: ```shell v4l2-drm -d 1 -n 5 -w 640 -h 480 ``` The test result is as follows: ![Text, letter description automatically generated](https://www.kendryte.com/api/post/attachment?id=465)