Rockchip Linux 4.4 Camera Developer Guide
:::info 文档信息
文档标识：RK-KF-YF-347 · 发布版本：V2.0.0 · 发布日期：2020-03-18 · 页数：31 · 语言：英文
原始文档：Rockchip_Developer_Guide_Linux4.4_Camera_EN.pdf
:::
章节大纲
- Rockchip Linux 4.4 Camera Developer Guide（P1）
- Historical Version Introduction and Acronyms（P4）
- Acronyms（P4）
- Historical Versions of ISP and CIF Drivers（P5）
- FAQ Documents（P5）
- Sensor Driver Development and Porting（P5）
- Power-on Sequence（P6）
- Check Whether the Power-on Sequence is Correct（P7）
- Sensor Initialization Register List（P7）
- The v4l2_subdev_ops Callback Function（P7）
- V4l2 Controller（P8）
- Probe Function and Registration Media Entity and v4l2 subdev（P8）
- dts Example: MIPI Sensor（P8）
- dts Example: DVP Sensor（P9）
- Sensor Debugging（P9）
- Whether the Sensor is Registered Successfully（P9）
- Is there Any Output from Capture Image Tools（P9）
- Check Whether the Control Is Effective（P9）
- Power-on Sequence（P6）
- Debugging Tools and Commonly Used Commands（P10）
- v4l-utils（P10）
- Use media-ctl to Get Topology（P10）
- Display Topology（P10）
- Switch Sensor（P11）
- Modify the Format and Sze of Entity（P12）
- Commonly Used mbus-code Formats（P12）
- Find Video Devices（P13）
- Capture Pictures by v4l2-ctl（P14）
- Capture Frames by v4l2-ctl（P14）
- Set Controls like Exposure and Gain（P15）
- Capture Raw Images（P16）
- Commonly used FourCC Formats（P17）
- Display YUV Images by mplayer in Ubuntu（P18）
- GStreamer Usage（P18）
- Display Images with GStreamer（P18）
- Videos and Images Encoding with GStreamer（P19）
- Boards Debugging Without Screen（P19）
- Turn on Debugging Switch（P19）
- 3A Integration Method（P20）
- rkisp_3A_server（P20）
- Enable rkisp_3A_server log（P21）
- XML Loading Acceleration Function（P22）
- Execution Sequence of rkisp_3A_server（P22）
- Applications Development（P23）
- librkisp_api.so Interface Usage（P24）
- Share Memory by DMA Buffer（P25）
- Set Area Exposure Weight（P26）
- Simulate to an UVC Camera by librkuvc.so（P27）
- RKISP1 Driver Introduction（P28）
- Rkisp1 dts Board Configuration（P30）
- RKCIF Driver Introduction（P30）
- Historical Version Introduction and Acronyms（P4）
在线预览
若预览无法显示，请直接下载 PDF 后用本地阅读器打开。
1. Historical Version Introduction and Acronyms
1.1 Acronyms
3A, refers to auto focus (AF), auto exposure (AE) and auto white balance (AWB) algorithms, or algorithms.so library. Async Sub Device, refers to V4L2 sub-device under Media Controller structure,such as sensor,MIPI DPHY. Bayer Raw, or Raw Bayer, represents the data frame format of RGGB, BGGR, GBRG, GRBG output from device (sensor or ISP). Buildroot, refers to a series of Linux SDKs base on Buildroot released by Rockchip. Camera, refers to a complete system composed of VIP or ISP in Rockchip chipset and its connected sensor, and their drivers in this document. CIF, VIP module of RK chips which is used to receive data from sensor and save it to memory. Only used to transfer data, no ISP function. DVP, is a parallel data transfer interface, is the abbreviation of Digital Video Port. Entity, refers to each node under Media Controller framework. FCC or FourCC, that is Four Character (FCC) codes, and it refers to the image format represented by 4 characters in Linux Kernel. For details, see FourCC chapter. HSYNC, refers to the line synchronization signal of the DVP interface. ISP, Image Signal Processing, used to receive and process images. It refers to both hardware and ISP driver in general. IOMMU, Input-Output Memory Management Unit, in this document, it means iohmu module in Rockchip chips, it is used to map physically fragmented memory pages into visible contiguous memory of cif or isp. It refers to both hardware and IOMMU driver in general. IQ, Image Quality, in this document, it refers to the IQ xml debugged by Bayer Raw Camera. It is used for 3A tunning. Media Controller, a media framework for Linux kernel, mainly used for topology management. MIPI, refers to MIPI protocol in general in this document. MIPI-DPHY refers to MIPI-DPHY protocol, or the controller that complies with MIPI-DPHY protocol of Rockchip chips in this document. MP, Main Path, refers to an output node of Rockchip ISP, which can output full-resolution images and generally used for taking pictures and capturing raw images. PCLK is Pixel clock output from sensor. Pipeline refers to the link formed by the interconnection of each entity in media controller in this document. RKCIF, refers to the driver name of CIF. RKISP1, refers to the name of ISP driver. SP, Self Path, refers to an output node of Rockchip ISP, which can only output 1080p resolution. Userspace, that is Linux user space (relative to Linux kernel space). V4L2, Video4Linux2, the video processing module of Linux kernel. VIP refers to Video Input Processor in Rockchip chips, it once used as an alias for CIF. VSYNC, Field sync signal of DVP interface.
1.2 Historical Versions of ISP and CIF Drivers
The RKISP1 and RKCIF drivers described in this document are based on Media Controller, V4L2 Framework, VB2, and sensor is registered as an Async Sub Device asynchronously. Their codes are located in the
drivers/media/platform/rockchip/isp1/ and drivers/media/platform/rockchip/cif/
directories, respectively; sensor code is located in the drivers/media/i2c directory.
The previous versions are no longer updated continuously or no longer supported. The details are as follows:
| Drive name | Type | Kernel | Whether in the scope of this article | Code location |
|---|---|---|---|---|
| RKISP1 | ISP | 4.4 | Yes | drivers/media/platform/rockchip/isp1 |
| RKCIF | CIF | 4.4 | Yes | drivers/media/platform/rockchip/cif |
| RK-ISP10 | ISP | 4.4 | No | drivers/media/platform/rk-isp10/ |
| RK- CAMSYS | CIF | 4.4 | No | drivers/media/video |
1.3 FAQ Documents
The FAQ document named "Rockchip_Trouble_Shooting_Linux4.4_Camera_CN" which is generally located in the same directory as this document is recommended for customers debugging sensor faster and more conveniently.
2. Sensor Driver Development and Porting
Sensor drivers are located in the drivers/media/i2c directory. Note that the sensor drivers described in this chapter are with Media Controller attribute, so the drivers in the drivers/media/i2c/soc_camera directory are not applicable. Sensor driver and RKCIF or RKISP1 driver are independent to the greatest extent, the two are registered asynchronously, and the connection relationship is declared by the remote-endpoint in dts. Therefore, the sensor driver described in this chapter is applicable to both RKCIF and RKISP1. In the Media Controller structure, sensor is generally used as a Sub Device and linked with the Rkcif, Rkisp1 or Mipi Dphy driver through Pad. This chapter mainly introduces the sensor driver code, dts configuration, and how to debug when porting a sensor driver.
This chapter summarizes the development and porting of sensor driver into 5 parts,
Write the power-on sequence according to the datasheet, including vdd, reset, powerdown, clk, etc. Configure sensor register to output the required resolution and format. Write the callback functions needed by struct v4l2_subdev_ops, generally including set_fmt, get_fmt, s_stream, s_power. Added v4l2 controller for setting such as fps, exposure, gain, test pattern. Write the probe() function and add Media Control and Sub Device initialization code.
As a good habit, after completing the driver coding, you also need to add the corresponding documentation. You can refer to Documentation/devicetree/bindings/media/i2c/. In this way, the board-level dts can be quickly configured according to this document. Generally, in the board-level dts, the followings are necessary for sensor driver reference:
Configure the correct clk and io mux. Set the regulator and gpio required for the power-on sequence according to the schematic. Add the port sub-node to establish connection with cif or isp.
This chapter takes ov5695 and ov2685 as examples to analyze the sensor driver.
2.1 Power-on Sequence
Different sensors have different power-on sequence requirements. For example, most OV sensors may not have strict timing requirements. As long as mclk, vdd, reset, and powerdown states are correct, I2C communicate can be performed correctly and images can be output without worrying about the sequence and delay of electricity. However, there are still a few number of sensors that have very strict power-on requirements. For example, OV2685 must be powered on strictly follow the sequence.
In the datasheet provided by the sensor manufacturer, there is usually a power-on sequence diagram, which only needs to be configured in order. Take drivers/media/i2c/ov5695.c as an example, where __ov5695_power_on() is used to power on sensor. The code is as follows (with deletions).
| 1 static int ov5695 power on(struct ov5695 *ov5695) __ _ _ 2 { 3 int ret; 4 u32 delay us; _ 5 struct device *dev = &ov5695->client->dev; 6 7 ret = clk set rate(ov5695->xvclk, OV5695 XVCLK FREQ); _ _ _ _ 8 9 if (clk get rate(ov5695->xvclk) != OV5695 XVCLK FREQ) _ _ _ _ 10 dev warn(dev, "xvclk mismatched, modes are based on 24MHz\n"); _ 11 ret = clk prepare enable(ov5695->xvclk); _ _ 12 13 if (!IS ERR(ov5695->reset gpio)) _ _ 14 gpiod set value cansleep(ov5695->reset gpio, 1); _ _ _ _ 15 16 ret = regulator bulk enable(OV5695 NUM SUPPLIES, ov5695->supplies); _ _ _ _ 17 18 if (!IS ERR(ov5695->reset gpio)) _ _ 19 gpiod set value cansleep(ov5695->reset gpio, 0); _ _ _ _ 20 21 if (!IS ERR(ov5695->pwdn gpio)) _ _ 22 gpiod set value cansleep(ov5695->pwdn gpio, 1); _ _ _ _ 23 24 /* 8192 cycles prior to first SCCB transaction */ 25 delay us = ov5695 cal delay(8192); _ _ _ 26 usleep range(delay us, delay us * 2); _ _ _ 27 28 return 0; 29 } 30 | |
|---|---|
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 | static int ov5695 power on(struct ov5695 *ov5695) __ _ _ { int ret; u32 delay us; _ struct device *dev = &ov5695->client->dev; ret = clk set rate(ov5695->xvclk, OV5695 XVCLK FREQ); _ _ _ _ if (clk get rate(ov5695->xvclk) != OV5695 XVCLK FREQ) _ _ _ _ dev warn(dev, "xvclk mismatched, modes are based on 24MHz\n"); _ ret = clk prepare enable(ov5695->xvclk); _ _ if (!IS ERR(ov5695->reset gpio)) _ _ gpiod set value cansleep(ov5695->reset gpio, 1); _ _ _ _ ret = regulator bulk enable(OV5695 NUM SUPPLIES, ov5695->supplies); _ _ _ _ if (!IS ERR(ov5695->reset gpio)) _ _ gpiod set value cansleep(ov5695->reset gpio, 0); _ _ _ _ if (!IS ERR(ov5695->pwdn gpio)) _ _ gpiod set value cansleep(ov5695->pwdn gpio, 1); _ _ _ _ /* 8192 cycles prior to first SCCB transaction */ delay us = ov5695 cal delay(8192); _ _ _ usleep range(delay us, delay us * 2); _ _ _ return 0; } |
The power-on sequence of OV5695 is briefly described as follows,
Provide xvclk (that is mclk) first. Next, enable reset pin. Power on vdd of each channel. The regulator_bulk is used here, because vdd, vodd, and avdd have no strict order. If there are strict requirements between vdd, you have to be handled separately, please refer to OV2685 driver code. Set the reset and powerdown pin of sensor to work status. Reset or powerdown, may be one of them is needed . Configure according to the actual needs of the sensor package and hardware schematic. Finally, according to the timing requirements of ov5695, 8192 clk cycles are required to be delayed before power-on is completed.
Note that although many sensors can work normally without powering on as required by datasheet, it is undoubtedly that the most reliable to operate according to the sequence recommended by vendors.
Similarly, there will be a power down sequence in the datasheet, which also needs to be implemented as required.
2.1.1 Check Whether the Power-on Sequence is Correct
In the .probe() stage, it will try to read the chip id, such as ov5695_check_sensor_id() of ov5695. If the chip id can be read correctly, it is generally considered that the power-on sequence is correct and sensor can perform i2c communication normally.
2.2 Sensor Initialization Register List
Both of OV5695 and OV2685 define struct ov5695_mode and struct ov2685_mode , which are used to represent different initialization modes of sensors, that is, sensor can output images of different resolutions, different fps, etc. Mode can include such as resolution, Mbus Code, fps, register initialization list, etc.
Register initialization list, please fill in as it provided by the vendor directly. It should be noted that the list ends with REG_NULL at the end. Note that REG_NULL should not conflict with the register address.
2.3 The v4l2_subdev_ops Callback Function
The v4l2_subdev_ops callback function is the core of the logic control in the sensor driver. The callback function includes rich interfaces, please check the kernel code include/media/v4l2-subdev.h for details. It is recommended that the sensor driver include the following callback functions at least.
.open(), it will call the .open() function when opening the /dev/v4l-subdev? node through Userspace. When the application layer needs to set control of the sensor separately, .open() must be implemented. .s_power(), including power on and power off. .s_stream(), that is setting stream. Including stream on and stream off. Generally configure the register here to make it output images. .enum_mbus_code(), enumerate the mbus_code supported by driver. .enum_frame_size(), enumerate the resolutions supported by driver. .get_fmt(), returns the format/size selected by the current sensor. If .get_fmt() is missing, you cannot find the currently configured format of sensor entity by media-ctl tool. .set_fmt(), set the format/size of sensor.
In the above callbacks, .s_power() and .s_stream() will be more complicated. In the ov5695 driver code, pm_runtime is used to manage power. Use v4l2_ctrl_handler_setup() in .s_stream() to configure control information actually (v4l2 control may be updated when the sensor is powered off ) and write to the register.
2.4 V4l2 Controller
It is necessary to implement the v4l2 controller function to update exposure, gain, and blanking dynamically, especially for RAW BYAYER sensor. General Raw Bayer Sensors are required.
In the OV5695 driver code,
ov5695_initialize_controls(), used to declare which controls are supported and set the maximum and minimum value. struct v4l2_ctrl_ops, specifies the ov5695_set_ctrl() callback function, application can set value to each control through this function.
2.5 Probe Function and Registration Media Entity and v4l2 subdev
In the probe function, firstly, parse dts to obtain information such as regulator, gpio, clk and so on to power on and off the sensor. Next, register media entity, v4l2 subdev, and v4l2 controller information. Note that the registration of v4l2 subdev is asynchronous. Several key functions are as follows.
v4l2_i2c_subdev_init(), registered as a v4l2 subdev, callback function is provided in the parameter. ov5695_initialize_controls(), initialize v4l2 controls. media_entity_init(), registered as a media entity, OV5695 has only one output called Source Pad. v4l2_async_register_subdev(), declares that sensor needs to be registered asynchronously. This call is necessary, because both of RKISP1 and RKCIF register Sub Device asynchronously.
2.6 dts Example: MIPI Sensor
According to the hardware design, the key is to configure pinctl (iomux), clk, gpio, remote port. The following example is the OV5695 dts node in rk3326-evb-lp3-v10-linux.dts.
ov5695: ov5695@36 {
compatible = "ovti,ov5695";
reg = <0x36>;
clocks = <&cru SCLK_CIF_OUT>;
clock-names = "xvclk";
avdd-supply = <&vcc2v8_dvp>;
dovdd-supply = <&vcc1v8_dvp>;
dvdd-supply = <&vdd1v5_dvp>;
/*reset-gpios = <&gpio2 14 GPIO_ACTIVE_HIGH>;*/
pwdn-gpios = <&gpio2 14 GPIO_ACTIVE_HIGH>;
rockchip,camera-module-index = <0>;
rockchip,camera-module-facing = "back";
rockchip,camera-module-name = "TongJu";
rockchip,camera-module-lens-name = "CHT842-MD";
port {
ucam_out: endpoint {
remote-endpoint = <&mipi_in_ucam>;
data-lanes = <1 2>;
};
};
};
Note:
pinctrl, initialize the necessary pin iomux, this example includes reset pin initialization and clk iomux. clock, specify the name as xvclk (driver will read clock named xvclk ), that is, 24M clock. vdd supply, OV5695 need three power supplies. The port subnode defines an endpoint that declares a connection to mipi_in_wcam. Similarly, mipi dphy will reference wcam_out. data-lanes specifies that OV5695 uses two lanes. In the wcam_out node, data-lanes should match it.
2.7 dts Example: DVP Sensor
Compared with Mipi sensor, it has no configure data-lanes of DVP sensor's dts , the endpoint is connected to cif, and rest parts are the same.
Take gc2155 in arch/arm64/boot/dts/rockchip/rk3326-evb-lp3-v10-linux.dts as an example,
In the dts node of gc2155, the remote-endpoint points to cif_in. There's no need to configure data-lanes parameter.
2.8 Sensor Debugging
After completing sensor driver porting, you have to check whether it works normally.
If you encounter problems during the debugging process, please troubleshoot according to the FAQ document first.
2.8.1 Whether the Sensor is Registered Successfully
The first key point for sensor debugging is whether i2c can communicate successfully and check the chip id is correct or not. If it is correct, indicating that there is no problem with power-on sequence. The related log will usually be printed out in driver. Logs of different sensors are not the same, it will not take examples here. Second, check media-ctl topology to see if sensor is already registered as an entity. If so, sensor has been registered successfully.
2.8.2 Is there Any Output from Capture Image Tools
Please capture images through capture tools such as v4l2-ctl , gstreamer , camera app, etc.
2.8.3 Check Whether the Control Is Effective
Use v4l2-ctl to set relevant parameters, such as gain, exposure, blanking, and generate pictures to check whether controls of the sensor are effective. For example, increase gain or exposure whether the brightness of the picture increases; increase blanking whether the frame rate decreases.
3. Debugging Tools and Commonly Used Commands
This chapter mainly introduces commonly used capture tools. Because most of the commands are relatively long, in order to read more friendly, the escape character \ is used to split a line of commands into several lines. Users can directly copy and paste when using, but if users put commands on one line, please remove the escape character \ .
3.1 v4l-utils
In Linux SDKs released by Rockchip, the v4l-utils package has been integrated by default. Users can turn on or off the v4l-utils package through the building switch of buildroot. such as:
# grep -rn LIBV4L_UTILS - buildroot/configs/rockchip/camera.config
BR2_PACKAGE_LIBV4L_UTILS=y
Users can also get the source code for building on the official website of www.linuxtv.org.
The v4l-utils package can be installed directly through the apt tool under Ubuntu system, as follows:
| 1 # sudo apt-get install v4l-utils | |
|---|---|
| 1 | # sudo apt-get install v4l-utils |
3.2 Use media-ctl to Get Topology
The media-ctl is a tool in the v4l-utils package. It is mainly used to check and configure the information of each entity of Media Framework, such as format, crop, link enable, etc. This tool can expand camera functions more flexibly.
For regular application cases, users do not need to configure specific Entity information, just use the default.
3.2.1 Display Topology
The following command is used to display topology. Note: When both cif and isp are enabled, or there are multiple isps enabled, or USB camera is inserted, there may be multiple media devices, such as /dev/media0, /dev/media1, /dev/media2.
| 1 # media-ctl -p -d /dev/media0 | |
|---|---|
| 1 | # media-ctl -p -d /dev/media0 |
The main concern of developers is whether to find the Entity of sensor. If the Entity of sensor is not found, it means that there is a problem with sensor registration. Please check by FAQ document.
For example, after connecting an ov5695 camera to RK3326 SDK board, you can see the following output (with deletion).
| 1 # media-ctl -p -d /dev/media1 2 - entity 9: m00 b ov5695 2-0036 (1 pad, 1 link) _ _ 3 type V4L2 subdev subtype Sensor flags 0 4 device node name /dev/v4l-subdev2 5 pad0: Source 6 [fmt:SBGGR10 1X10/2592x1944@10000/300000 field:none] _ 7 -> "rockchip-mipi-dphy-rx":0 [ENABLED,DYNAMIC] | |
|---|---|
| 1 2 3 4 5 6 7 | # media-ctl -p -d /dev/media1 - entity 9: m00 b ov5695 2-0036 (1 pad, 1 link) _ _ type V4L2 subdev subtype Sensor flags 0 device node name /dev/v4l-subdev2 pad0: Source [fmt:SBGGR10 1X10/2592x1944@10000/300000 field:none] _ -> "rockchip-mipi-dphy-rx":0 [ENABLED,DYNAMIC] |
| You c |
The full name of the Entity is: m00_b_ov5695 2-0036 . It is a V4L2 subdev (Sub-Device) Sensor . Its corresponding node is /dev/v4l-subdev2 , applications (such as v4l2-ctl) can open it and configure. It has only one output ( Source ) node, that is pad0 . Its output format is [fmt:SBGGR10_1X10/2592x1944@10000/300000 field:none] , where SBGGR10_1X10 is the abbreviation of one mbus-code, the next section will list common mbus-code. Its Source pad0 is linked to ( -> ) pad0 of "rockchip-mipi-dphy-rx" , and its current status is ENABLED . DYNAMIC means that the status can be changed to DISABLED.
If the same ISP or CIF is connected to two sensors at the same time, only one of them is ENABLED, as shown in the following example (with deletion).
| 1 [root@rk3326 64:/]# media-ctl -p -d /dev/media1 _ 2 -entity 9: irs16x5c 1-003d (1 pad, 1 link) 3 type V4L2 subdev subtype Sensor flags 0 4 device node name /dev/v4l-subdev2 5 pad0: Source 6 [fmt:SBGGR12 1X12/224x1557@10000/300000 field:none] _ 7 -> "rockchip-mipi-dphy-rx":0 [ENABLED,DYNAMIC] 8 9 -entity 10: irs16x5c 2-003d (1 pad, 1 link) 10 type V4L2 subdev subtype Sensor flags 0 11 device node name /dev/v4l-subdev3 12 pad0: Source 13 [fmt:SBGGR12 1X12/224x1557@10000/300000 field:none] _ 14 -> "rockchip-mipi-dphy-rx":0 [DYNAMIC] 15 | |
|---|---|
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | [root@rk3326 64:/]# media-ctl -p -d /dev/media1 _ -entity 9: irs16x5c 1-003d (1 pad, 1 link) type V4L2 subdev subtype Sensor flags 0 device node name /dev/v4l-subdev2 pad0: Source [fmt:SBGGR12 1X12/224x1557@10000/300000 field:none] _ -> "rockchip-mipi-dphy-rx":0 [ENABLED,DYNAMIC] -entity 10: irs16x5c 2-003d (1 pad, 1 link) type V4L2 subdev subtype Sensor flags 0 device node name /dev/v4l-subdev3 pad0: Source [fmt:SBGGR12 1X12/224x1557@10000/300000 field:none] _ -> "rockchip-mipi-dphy-rx":0 [DYNAMIC] |
| In the a T |
You have to switch sensor only when the entire link stops working, that is: you cannot change the configuration of each Entity in the pipeline during capturing process.
3.2.2 Switch Sensor
If multiple sensors are connected, you can switch them with the following commands.
| 1 # media-ctl -d /dev/media0 \ 2 -l '"ov5695 7-0036":0->"rockchip-sy-mipi-dphy":0[0]' 3 # media-ctl -d /dev/media0 \ 4 -l '"ov2685 7-003c":0->"rockchip-sy-mipi-dphy":0[1]' | |
|---|---|
| 1 2 3 4 | # media-ctl -d /dev/media0 \ -l '"ov5695 7-0036":0->"rockchip-sy-mipi-dphy":0[0]' # media-ctl -d /dev/media0 \ -l '"ov2685 7-003c":0->"rockchip-sy-mipi-dphy":0[1]' |
The command format is media-ctl -l'"entity name":pad->"entity name":pad[Status]'. The entire link needs to use single quotes because of special characters: > [ ]. Entity name needs to use double quotes because there are spaces in between. Status can be 0 or 1 to indicate Active or In-Active.
3.2.3 Modify the Format and Sze of Entity
First example: OV5695 supports multiple resolution output, it is 2592x1944 by default. Now change the output resolution to 1920x1080.
# media-ctl -d /dev/media1 \
--set-v4l2 '"m00_b_ov5695 2-0036":0[fmt:SBGGR10_1X10/1920x1080]'
After modifying the output of OV5695, the size of rkisp1-isp-subdev and the video device crop should be modified accordingly. Because the size of the subsequent stage cannot be greater than the size of the previous stage.
| 1 # media-ctl -d /dev/media1 \ 2 --set-v4l2 '"rkisp1-isp-subdev":0[fmt:SBGGR10 1X10/1920x1080]' _ 3 # media-ctl -d /dev/media1 \ 4 --set-v4l2 '"rkisp1-isp-subdev":0[crop:(0,0)/1920x1080]' 5 # media-ctl -d /dev/media1 \ 6 --set-v4l2 '"rkisp1-isp-subdev":2[crop:(0,0)/1920x1080]' 7 # v4l2-ctl -d /dev/video1 \ 8 --set-selection=target=crop,top=0,left=0,width=1920,height=1080 | |
|---|---|
| 1 2 3 4 5 6 7 8 | # media-ctl -d /dev/media1 \ --set-v4l2 '"rkisp1-isp-subdev":0[fmt:SBGGR10 1X10/1920x1080]' _ # media-ctl -d /dev/media1 \ --set-v4l2 '"rkisp1-isp-subdev":0[crop:(0,0)/1920x1080]' # media-ctl -d /dev/media1 \ --set-v4l2 '"rkisp1-isp-subdev":2[crop:(0,0)/1920x1080]' # v4l2-ctl -d /dev/video1 \ --set-selection=target=crop,top=0,left=0,width=1920,height=1080 |
| Secon |
changed to the fmt of sensor, so that MP node can output raw images.
# media-ctl -d /dev/media1 \
--set-v4l2 '"rkisp1-isp-subdev":2[fmt:SBGGR10/2592x1944]'
In the above example, there are some points to be noted:
Pay attention to special characters, you need to use single or double quotes. Don't add or delete spaces in quotation marks. Please use media-ctl --help to view more detailed help.
3.2.4 Commonly Used mbus-code Formats
Mbus-code is short for Media Bus Pixel Codes, which describes the format used for transmission on the physical bus, such as the image format transmitted by sensor to isp through mipi dphy, or the format transmitted between sub-modules within ISP. In particular, it is necessary to distinguish Mbus-code from FourCC. The latter refers to the image format stored in Memory.
Mbus-code is defined in include/uapi/linux/media-bus-format.h of kernel.
The following table lists several Mbus-codes commonly used in this document.
| Macros defined in Kernel | Mubs-code abbreviation | Type | Bpp | Bus width | Samples |
|---|---|---|---|---|---|
| MEDIA BUS FMT SBGGR8 1X8 _ _ _ _ | SBGGR8 1X8 _ | Bayer Raw | 8 | 8 | 1 |
| MEDIA BUS FMT SRGGB8 1X8 _ _ _ _ | SRGGB8 1X8 _ | Bayer Raw | 8 | 8 | 1 |
| MEDIA BUS FMT SBGGR10 1X10 _ _ _ _ | SBGGR10 1X10 _ | Bayer Raw | 10 | 10 | 1 |
| MEDIA BUS FMT SRGGB10 1X10 _ _ _ _ | SRGGB10 1X10 _ | Bayer Raw | 10 | 10 | 1 |
| MEDIA BUS FMT SBGGR12 1X12 _ _ _ _ | SBGGR12 1X12 _ | Bayer Raw | 12 | 12 | 1 |
| MEDIA BUS FMT SRGGB12 1X12 _ _ _ _ | SRGGB12 1X12 _ | Bayer Raw | 12 | 12 | 1 |
| MEDIA BUS FMT YUYV8 2X8 _ _ _ _ | YVYU8 2X8 _ | YUV 422 | 16 | 8 | 2 |
| MEDIA BUS FMT UYUV8 2X8 _ _ _ _ | UYUV8 2X8 _ | YUV 422 | 16 | 8 | 2 |
| MEDIA BUS FMT Y8 1X8 _ _ _ _ | Y8 1X8 _ | YUV GREY | 8 | 8 | 1 |
| MEDIA BUS FMT RGB888 1X24 _ _ _ _ | RGB888 1X24 _ | RGB 888 | 24 | 24 | 1 |
The media-ctl can list the supported mbus codes.
| 1 # media-ctl --known-mbus-fmts | |
|---|---|
| 1 | # media-ctl --known-mbus-fmts |
3.2.5 Find Video Devices
There are multiple entities in the topology, some are sub devices and some are video devices. The device node corresponding to the former is /dev/v4l-subdev, and the latter corresponds to /dev/video. In the case of multiple video devices, users are most concerned about which device can output images.
| 1 # media-ctl -d /dev/media1 -e "rkisp1 selfpath" _ 2 /dev/video2 3 # media-ctl -d /dev/media1 -e "rkisp1 mainpath" _ 4 /dev/video1 | |
|---|---|
| 1 2 3 4 | # media-ctl -d /dev/media1 -e "rkisp1 selfpath" _ /dev/video2 # media-ctl -d /dev/media1 -e "rkisp1 mainpath" _ /dev/video1 |
The above two commands display the device path of SP and MP nodes of RKISP1 in the link /dev/media1 respectively. RKISP1 has two video output devices, both of which can output images.
It is similar to RKCIF:
# media-ctl -d /dev/media0 -e "stream_cif"
/dev/video0
The above command shows the path of RKCIF video device in the link /dev/media0. RKCIF has only one video output node.
| 1 # v4l2-ctl -d /dev/video1 --all | |
|---|---|
| 1 | # v4l2-ctl -d /dev/video1 --all |
The above command shows some main parameters of /dev/video1, such as crop, fmt, v4l2 controls, etc.
3.3 Capture Pictures by v4l2-ctl
The operation of the Media-ctl tool is through media devices such as /dev/medio0, and it manages format, size, and link of each node in the Media topology. The V4l2-ctl tool is for video devices such as /dev/video0 and /dev/video1. It performs a series of operations such as set_fmt, reqbuf, qbuf, dqbuf, stream_on, stream_off, etc. on video devices. This document mainly uses v4l2-ctl to collect frame data, set exposure, gain, VTS, etc. v4l2_control.
It is recommended to read the help file of v4l2-ctl first. The help document has rich contents and is divided into many parts. What we should pay more attention to are streaming and vidcap.
The summary of the help file is as follows:
| 1 # v4l2-ctl --help | |
|---|---|
| 1 | # v4l2-ctl --help |
To check the complete help document as follows, with rich content relatively.
| 1 # v4l2-ctl --help-all | |
|---|---|
| 1 | # v4l2-ctl --help-all |
To check parameters related to streaming as follows.
| 1 # v4l2-ctl --help-streaming | |
|---|---|
| 1 | # v4l2-ctl --help-streaming |
To check parameters related to vidcap as follows. It mainly includes get-fmt, set-fmt, etc.
| 1 # v4l2-ctl --help-vidcap | |
|---|---|
| 1 | # v4l2-ctl --help-vidcap |
3.3.1 Capture Frames by v4l2-ctl
Example 1: capture one frame of NV12 data output by RKCIF and save it to /tmp/nv12.bin with a resolution of 640x480. Before saving data, discard the first 3 frames (that is, although the first 3 frames are returned to userspace, they are not saved to the file).
| 1 # v4l2-ctl -d /dev/video0 \ 2 --set-fmt-video=width=640,height=480,pixelformat=NV12 \ 3 --stream-mmap=3 \ 4 --stream-skip=3 \ 5 --stream-to=/tmp/nv12.bin \ 6 --stream-count=1 \ 7 --stream-poll | |
|---|---|
| 1 2 3 4 5 6 7 | # v4l2-ctl -d /dev/video0 \ --set-fmt-video=width=640,height=480,pixelformat=NV12 \ --stream-mmap=3 \ --stream-skip=3 \ --stream-to=/tmp/nv12.bin \ --stream-count=1 \ --stream-poll |
Example 2: capture 10 frames of NV12 data output by RKISP and save it to /tmp/nv12.bin with a resolution of 1920x1080.
| 1 # v4l2-ctl -d /dev/video1 \ 2 --set-selection=target=crop,top=0,left=0,width=1920,height=1080 3 --set-fmt-video=width=1920,height=1080,pixelformat=NV12 \ 4 --stream-mmap=3 \ 5 --stream-to=/tmp/nv12.bin \ 6 --stream-count=10 \ 7 --stream-poll | |
|---|---|
| 1 2 3 4 5 6 7 | # v4l2-ctl -d /dev/video1 \ --set-selection=target=crop,top=0,left=0,width=1920,height=1080 --set-fmt-video=width=1920,height=1080,pixelformat=NV12 \ --stream-mmap=3 \ --stream-to=/tmp/nv12.bin \ --stream-count=10 \ --stream-poll |
| Note |
-d, specify the operation object as /dev/video0 device. --set-selection, specify to crop input images. Especially when the size of the previous stage of RKISP1 changes, make sure that the selection is not greater than the output size of the previous stage. The crop of RKCIF is set by the --set-crop parameter. --set-fmt-video, specifies the width and height and pxielformat (represented by FourCC). NV12 is the pixelformat represented by FourCC. --stream-mmap, specify the type of buffer as mmap, that is, a physically continuous or iommu mapped buffer allocated by kernel. --stream-skip, specify to discard (not save to file) the first 3 frames. --stream-to, specify the file path for saving frame data. --stream-count, specifies the number of frames captured, excluding the number of discarded by --stream- skip. --stream-poll, this option indicates v4l2-ctl use asynchronous IO, that is, use "select" to wait for frame data complete before dqbuf to ensure that dqbuf is not blocked. Otherwise, dqbuf will block until a data frame arrives.
3.3.2 Set Controls like Exposure and Gain
If the sensor driver implements v4l2 control, v4l2-ctl can be used to set the exposure, gain, etc. before capturing images. RKCIF or RKISP will inherit the control of sub device, so you can see the v4l2 control of sensor through /dev/video3.
The following are the relevant settings of OV5695 on RK3326 SDK device, including exposure, gain, blanking, test_pattern, etc.
# v4l2-ctl -d /dev/video1 -l
User Controls
exposure 0x00980911 (int) : min=4 max=2228 step=1 default=1104
value=1104
gain 0x00980913 (int) : min=0 max=16383 step=1 default=1024
value=1024
Image Source Controls
vertical_blanking 0x009e0901 (int) : min=1152 max=31687 step=1 default=1152
value=1152
analogue_gain 0x009e0903 (int) : min=16 max=248 step=1 default=248
value=248
Image Processing Controls
test_pattern 0x009f0903 (menu): min=0 max=4 default=0 value=0
These controls can be modified by v4l2-ctl. For example, modify exposure and analogue_gain as follows.
| 1 # v4l2-ctl -d /dev/video3 --set-ctrl'exposure=1216,analogue gain=10' _ | |
|---|---|
| 1 | # v4l2-ctl -d /dev/video3 --set-ctrl'exposure=1216,analogue gain=10' _ |
3.3.3 Capture Raw Images
The MP node of RKISP1 can capture raw images. At this time, ISP is in the by-pass state and does not tune data.
For example, capture Raw Bayer data output from sensor OV5695. The format is SBGGR10_1X10 and the size is 2592x1944.
| 1 # media-ctl -d /dev/media0 \ 2 --set-v4l2 '"ov5695 7-0036":0[fmt:SBGGR10 1X10/2592x1944]' _ 3 # media-ctl -d /dev/media0 \ 4 --set-v4l2 '"rkisp1-isp-subdev":0[fmt:SBGGR10 1X10/2592x1944]' _ 5 # media-ctl -d /dev/media0 \ 6 --set-v4l2 '"rkisp1-isp-subdev":0[crop:(0,0)/2592x1944]' 7 # media-ctl -d /dev/media0 \ 8 --set-v4l2 '"rkisp1-isp-subdev":2[fmt:SBGGR10 1X10/2592x1944]' _ 9 # media-ctl -d /dev/media0 \ 10 --set-v4l2 '"rkisp1-isp-subdev":2[crop:(0,0)/2592x1944]' 11 # v4l2-ctl -d /dev/video4 \ 12 --set-ctrl 'exposure=1216,analogue gain=10' \ _ 13 --set-selection=target=crop,top=0,left=0,width=2592,height=1944 \ 14 --set-fmt-video=width=2592,height=1944,pixelformat=BG10 \ 15 --stream-mmap=3 \ 16 --stream-to=/tmp/mp.raw.out \ 17 --stream-count=1 \ 18 --stream-poll | |
|---|---|
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | # media-ctl -d /dev/media0 \ --set-v4l2 '"ov5695 7-0036":0[fmt:SBGGR10 1X10/2592x1944]' _ # media-ctl -d /dev/media0 \ --set-v4l2 '"rkisp1-isp-subdev":0[fmt:SBGGR10 1X10/2592x1944]' _ # media-ctl -d /dev/media0 \ --set-v4l2 '"rkisp1-isp-subdev":0[crop:(0,0)/2592x1944]' # media-ctl -d /dev/media0 \ --set-v4l2 '"rkisp1-isp-subdev":2[fmt:SBGGR10 1X10/2592x1944]' _ # media-ctl -d /dev/media0 \ --set-v4l2 '"rkisp1-isp-subdev":2[crop:(0,0)/2592x1944]' # v4l2-ctl -d /dev/video4 \ --set-ctrl 'exposure=1216,analogue gain=10' \ _ --set-selection=target=crop,top=0,left=0,width=2592,height=1944 \ --set-fmt-video=width=2592,height=1944,pixelformat=BG10 \ --stream-mmap=3 \ --stream-to=/tmp/mp.raw.out \ --stream-count=1 \ --stream-poll |
| Note: M L |
In line 6, if the picture is too dark, you can adjust expo or gain to increase brightness optionally. And sensor driver needs to implement v4l2 control. In line 7 and line 8, v4l2-ctl sets the selection without cropping and the output pixel format FourCC is BG10. It should be noted that although ISP does not process raw images, it still fills the low bits of 10bit data with 0 into 16bit. Regardless of the sensor input is 10bit or 12bit, the final upper layer is 16bit per pixel.
Add PGM symbol to the header of Bayer Raw file, and it will be converted into pgm images that can be directly opened and viewed in Ubuntu. Just add three lines of PGM header.
For example, convert raw images to pgm format:
| 1 # cat > /tmp/raw.pgm << EOF 2 P5 3 2592 1944 4 65535 5 EOF 6 7 # cat /tmp/mp.raw.out >> /tmp/raw.pgm | |
|---|---|
| 1 2 3 4 5 6 7 | # cat > /tmp/raw.pgm << EOF P5 2592 1944 65535 EOF # cat /tmp/mp.raw.out >> /tmp/raw.pgm |
| Note: |
Line 2, P5 is the fixed identifier. Line 3 indicates the resolution of raw images, that is, the length and width, separated by a space character. Line 4 indicates the depth, 65535 is 16bit. If it is 8bit, change it to 255 accordingly. Line 7, add the raw data to the end of the pgm file header. Note that there are only three lines in the pgm header, do not add extra blank lines.
3.3.4 Commonly used FourCC Formats
FourCC is short for Four Character Codes, which uses 4 characters (that is 32bit) to name the image format. In Linux Kernel, the macro definition is as follows:
#define v4l2_fourcc(a,b,c,d) \
(((__u32)(a)<<0)|((__u32)(b)<<8)|((__u32)(c)<<16)|((__u32)(d)<<24))
The format defined by FourCC is the format in which images and videos are stored in the memory. This should be distinguished from mbus-code.
Several formats commonly used in this document are listed below. For more detailed definition, please refer to videodev2.h of the kernel code.
| Macro defined in kernel | FourCC |
|---|---|
| V4L2 PIX FMT NV12 _ _ _ | NV12 |
| V4L2 PIX FMT NV21 _ _ _ | NV21 |
| V4L2 PIX FMT NV16 _ _ _ | NV16 |
| V4L2 PIX FMT NV61 _ _ _ | NV61 |
| V4L2 PIX FMT NV12M _ _ _ | NM12 |
| V4L2 PIX FMT YUYV _ _ _ | YUYV |
| V4L2 PIX FMT YUV420 _ _ _ | YU12 |
| V4L2 PIX FMT SBGGR10 _ _ _ | BG10 |
| V4L2 PIX FMT SGBRG10 _ _ _ | GB10 |
| V4L2 PIX FMT SGRBG10 _ _ _ | BA10 |
| V4L2 PIX FMT SRGGB10 _ _ _ | RG10 |
| V4L2 PIX FMT GREY _ _ _ | GREY |
3.4 Display YUV Images by mplayer in Ubuntu
Some commands in the previous section capture frame data and save them as a file, which can be parsed by mplayer in the Ubuntu environment.
The mplayer can be installed via apt as follows.
| 1 # sudo apt-get install mplayer | |
|---|---|
| 1 | # sudo apt-get install mplayer |
Play NV12 images with 640x480 size as follows.
# W=640; H=480; mplayer /tmp/nv12.bin -loop 0 -demuxer rawvideo -fps 30 \
-rawvideo w=${W}:h=${H}:size=$((${W}*${H}*3/2)):format=NV12
Play YUYV images with 640x480 size, as follows.
# W=640; H=480; mplayer /tmp/yuyv.bin -loop 0 -demuxer rawvideo -fps 30 \
-rawvideo w=${W}:h=${H}:size=$((${W}*${H}*2)):format=YUY2
In the above example:
W and H are variables, specify the width and height to facilitate subsequent reference. fps specifies playback rate. If fps is 1, then one frame will be played in 1 second. size refers to the size of each frame. format refers to format, mplayer -rawvideo format=help can display all supported formats.
In Windows, tools such as 7yuv can be used to parse images.
3.5 GStreamer Usage
In Linux SDKs released by Rockchip, you can use GStreamer to preview Camera's images and encoding.
The v4l2src plugin can be used to get images from video device. By default, rkisp_3A_server will also start tunning, so that images with normal brightness and color can be obtained.
3.5.1 Display Images with GStreamer
The following command will display camera images on screen.
| 1 # export LD LIBRARY PATH=$LD LIBRARY PATH:/usr/lib/gstreamer-1.0 _ _ _ _ 2 # export XDG RUNTIME DIR=/tmp/.xdg _ _ 3 # gst-launch-1.0 v4l2src device=/dev/video1 ! \ 4 video/x-raw,format=NV12,width=2592,height=1944,framerate=30/1 ! kmssink | |
|---|---|
| 1 2 3 4 | # export LD LIBRARY PATH=$LD LIBRARY PATH:/usr/lib/gstreamer-1.0 _ _ _ _ # export XDG RUNTIME DIR=/tmp/.xdg _ _ # gst-launch-1.0 v4l2src device=/dev/video1 ! \ video/x-raw,format=NV12,width=2592,height=1944,framerate=30/1 ! kmssink |
3.5.2 Videos and Images Encoding with GStreamer
Linux SDKs also have hardware encoding. The following commands can encode and save camera data stream into a file.
| 1 # gst-launch-1.0 v4l2src device=/dev/video1 num-buffers=100 ! \ 2 video/x-raw,format=NV12,width=1920,height=1088,framerate=30/1 ! \ 3 videoconvert ! mpph264enc ! h264parse ! mp4mux ! \ 4 filesink location=/tmp/h264.mp4 | |
|---|---|
| 1 2 3 4 | # gst-launch-1.0 v4l2src device=/dev/video1 num-buffers=100 ! \ video/x-raw,format=NV12,width=1920,height=1088,framerate=30/1 ! \ videoconvert ! mpph264enc ! h264parse ! mp4mux ! \ filesink location=/tmp/h264.mp4 |
| 1 # gst-launch-1.0 -v v4l2src device=/dev/video1 num-buffers=10 ! \ 2 video/x-raw,format=NV12,width=1920,height=1080 ! mppjpegenc ! \ 3 multifilesink location=/tmp/test%05d.jpg | |
| 1 2 3 | # gst-launch-1.0 -v v4l2src device=/dev/video1 num-buffers=10 ! \ video/x-raw,format=NV12,width=1920,height=1080 ! mppjpegenc ! \ multifilesink location=/tmp/test%05d.jpg |
Note:
mppjpegenc and mpph264enc encoder are hardware encoding provided by rockchipmpp plugin. The mpp encoder needs height with 16 alignment, so the kernel needs to include this patch:
fea937b015e7 media: rockchip: isp1/cif: set height alignment to 16 in
queue_setup .
3.6 Boards Debugging Without Screen
Rockchip Linux SDKs provide librkuvc.so, by this interface, boards can be used as an uvc camera and connect to a PC via a USB OTG cable.
In the chapter of application development, sample codes are given for reference.
3.7 Turn on Debugging Switch
Both RKISP1 or RKCIF drivers contain some v4l2_dbg() logs, which can be turned on by commands, as follows.
| 1 # echo 1 > /sys/module/video rkcif/parameters/debug _ | |
|---|---|
| 1 | # echo 1 > /sys/module/video rkcif/parameters/debug _ |
Or
| 1 # echo 1 > /sys/module/video rkisp1/parameters/debug _ | |
|---|---|
| 1 | # echo 1 > /sys/module/video rkisp1/parameters/debug _ |
In addition, VB2 and V4L2 also have debugging switches.
Open VB2 related log as follows.
| 1 # echo 7 > /sys/module/videobuf2 core/parameters/debug _ | |
|---|---|
| 1 | # echo 7 > /sys/module/videobuf2 core/parameters/debug _ |
VB2 log mainly includes buffer rotation, such as reqbuf, qbuf, dqbuf and buffer status changes. It should be noted that the vb2 module switch is a general switch, and other related logs that use vb2(such as VPU/ISP, etc.) will also enable output.
Open v4l2 related log, such as ioctl call. The following command will open all v4l2 related logs.
| 1 # echo 0x1f > /sys/class/video4linux/video0/dev debug _ | |
|---|---|
| 1 | # echo 0x1f > /sys/class/video4linux/video0/dev debug _ |
You can also open only a small part of log. The following Kernel macro defines which log will be enabled by each bit. Just turn on the bit corresponding to the required log. These macros are defined in the kernel header file
include/media/v4l2-ioctl.h .
| 1 /* Just log the ioctl name + error code */ 2 #define V4L2 DEV DEBUG IOCTL 0x01 _ _ _ 3 /* Log the ioctl name arguments + error code */ 4 #define V4L2 DEV DEBUG IOCTL ARG 0x02 _ _ _ _ 5 /* Log the file operations open, release, mmap and get unmapped area */ _ _ 6 #define V4L2 DEV DEBUG FOP 0x04 _ _ _ 7 /* Log the read and write file operations and the VIDIOC (D)QBUF ioctls */ _ 8 #define V4L2 DEV DEBUG STREAMING 0x08 _ _ _ 9 /* Log poll() */ 10 #define V4L2 DEV DEBUG POLL 0x10 _ _ _ | |
|---|---|
| 1 2 3 4 5 6 7 8 9 10 | /* Just log the ioctl name + error code */ #define V4L2 DEV DEBUG IOCTL 0x01 _ _ _ /* Log the ioctl name arguments + error code */ #define V4L2 DEV DEBUG IOCTL ARG 0x02 _ _ _ _ /* Log the file operations open, release, mmap and get unmapped area */ _ _ #define V4L2 DEV DEBUG FOP 0x04 _ _ _ /* Log the read and write file operations and the VIDIOC (D)QBUF ioctls */ _ #define V4L2 DEV DEBUG STREAMING 0x08 _ _ _ /* Log poll() */ #define V4L2 DEV DEBUG POLL 0x10 _ _ _ |
4. 3A Integration Method
The contents of this section only applies to RKISP1. RKCIF is without ISP function and does not require 3A.
In order to support 3A, many ways have been added in the past version to support it, such as:
Provide application program link of librkisp.so library. Write rkisp, rkv4l2src gstreamer plug-ins to support GStreamer.
However, the above ways are troublesome for applications and does not effectively adapt to existing programs, such as VLC and Chrome browsers.
After the RKISP1 v0.1.5 driver and camera_engine_rkisp v2.2.0 version, the rkisp_3A_server process has been added, which will trigger and complete 3A tunning automatically. In this way,
The application no longer needs 3A library librkisp.so, but is only responsible for data flow. The v4l2src plug-in of GStreamer can be used directly, open source tools such as vlc can also be used directly, and normal 3A images can be obtained.
4.1 rkisp_3A_server
Including Kernel's RKISP1 driver, camera_engine_rkisp, and related startup scripts. If you have updated to the latest Rockchip Linux SDK, 3A is already integrated by default, including the following three main parts:
The driver version of RKISP1 is v0.1.5 or later versions. The version number is defined in:
kernel/drivers/media/platform/rockchip/isp1/version.h
The camera_engine_rkisp package is updated to v2.2.0 or later. The path is in
external/camera_engine_rkisp .
The path of camera_engine_rkisp building script and self-starting script are in:
buildroot/package/rockchip/camera_engine_rkisp
The file built by camera_engine_rkisp is as follows:
| 1 /usr/bin/rkisp 3A server _ _ 2 /usr/lib/librkisp.so 3 /usr/lib/librkisp api.so _ 4 /usr/lib/rkisp/ 5 ├── ae 6 │ └── librkisp aec.so _ 7 ├── af 8 │ └── librkisp af.so _ 9 └── awb 10 └── librkisp awb.so _ 11 /etc/iqfiles/ 12 /etc/init.d/S40rkisp 3A _ | |
|---|---|
| 1 2 3 4 5 6 7 8 9 10 11 12 | /usr/bin/rkisp 3A server _ _ /usr/lib/librkisp.so /usr/lib/librkisp api.so _ /usr/lib/rkisp/ ├── ae │ └── librkisp aec.so _ ├── af │ └── librkisp af.so _ └── awb └── librkisp awb.so _ /etc/iqfiles/ /etc/init.d/S40rkisp 3A _ |
| Note: |
librkisp.so, which implements the main interface of 3A, and calls aec, af, awb link libraries respectively. The latter is not open source iqfiles directory, storing iq parameters and xml files of sensors S40rkisp_3A is the boot script of rkisp_3A_server so that it can be started with boot
Except different booting way, the other logic of Debian system provided by Rockchip is the same as Linux SDKs.
4.2 Enable rkisp_3A_server log
Enable log by declaring environment variables, for example, open AEC log as follows:
| 1 # export persist camera engine log=0x40 _ _ _ | |
|---|---|
| 1 | # export persist camera engine log=0x40 _ _ _ |
Add the above line to /etc/init.d/S40rkisp_3A, the log will be saved to /var/log/messages. If there are many logs, the messages file will be split into multiple files, such as /var/log/messages.0 and /var/log/messages.1. When users pack logs, remember to pack all logs. such as:
| 1 # tar zcvf /tmp/camera-log.tar.gz /var/log/messages* | |
|---|---|
| 1 | # tar zcvf /tmp/camera-log.tar.gz /var/log/messages* |
Logs can be opened according to modules as follows,
| 1 bits: 31-28 27-24 23-20 19-16 15-12 11-8 7-4 3-0 2 module: [u] [u] [xcore] [ISP] [AF] [AWB] [AEC] [NO] 3 *[u] means unused now. 4 each module log has following ascending levels: 5 0: error 6 1: warning 7 2: info 8 3: verbose 9 4: debug 10 5: low1 11 6-7: unused, now the same as debug 12 | |
|---|---|
| 1 2 3 4 5 6 7 8 9 10 11 12 | bits: 31-28 27-24 23-20 19-16 15-12 11-8 7-4 3-0 module: [u] [u] [xcore] [ISP] [AF] [AWB] [AEC] [NO] *[u] means unused now. each module log has following ascending levels: 0: error 1: warning 2: info 3: verbose 4: debug 5: low1 6-7: unused, now the same as debug |
| Specia to redi |
serial port, shell default output).
# /etc/init.d/S40rkisp_3A stop
# export persist_camera_engine_log=0x40
# /usr/bin/rkisp_3A_server --mmedia=/dev/media0 > /tmp/log 2>&1
As shown above, first stop service, set environment variables, and then manually execute the command to start rkisp_3A_server, and redirect logs to /tmp/log file.
4.3 XML Loading Acceleration Function
For some products that need to boot up and display images faster, we provide xml loading acceleration. By opening the macro definition
# make menuconfig
│ │ --- Rockchip BSP packages │
│
│ │ [*] Rockchip Camera Engine for linux │
│
│ │ [*] Rockchip Camera Engine 3A service run in booting │
│
│ │ Specify a directory to store xml speed up bin (disabled) ---> │
│
│ │ () Rockchip Camera Engine IQ xml file │
│
According to whether rootfs is writable, select the option Specify a directory to store xml speed up bin .
│ │ (X) disabled │
│
│ │ ( ) /etc/iqfiles-db │
│
│ │ ( ) /userdata/iqfiles-db │
│
If rootfs is read only, you can only choose /userdata/iqfiles-db . This option specifies a directory for storing xml bin, and the file system where it is located needs to be writable.
4.4 Execution Sequence of rkisp_3A_server
The following shows the sequence diagram of rkisp_3A_server. userspace_app kernel_rkisp1_driver rkisp_3A_server 3A_Tunning_Thread
start to capture images
Sensor output Raw data, need to start 3A
| Tunning thread started suc | |
|---|---|
| cessfully | |
| Ready to exit | Continuo Exit tunning thread |
| Tunning has ended |
Continuous tunning
userspace_app kernel_rkisp1_driver rkisp_3A_server 3A_Tunning_Thread
5. Applications Development
In addition to directly capturing and previewing images through v4l2-ctl, GStreamer, VLC, etc., users can also write programs based on the V4L2 interface to capture images and then process them.
Applications can also link librkisp.so library by itself to get richer image information, such as:
Manually control exposure time and gain. Set exposure weight and spot metering for a certain area. Obtain brightness statistics, such as the average brightness of each area, the histogram of the entire picture. Control the maximum fps and gain to improve image quality. All of the above functions can be obtained after applications are linked to librkisp.so. Therefore, the rkisp_3A_server process needs to be stopped first, and 3A initialization and start are completed by applications.
| 1 # /etc/init.d/S40rkisp 3A stop _ | |
|---|---|
| 1 | # /etc/init.d/S40rkisp 3A stop _ |
Or delete S40rkisp_3A directly.
| 1 # rm /etc/init.d/S40rkisp 3A _ | |
|---|---|
| 1 | # rm /etc/init.d/S40rkisp 3A _ |
5.1 librkisp_api.so Interface Usage
In order to use it faster, camera_engine_rkisp provides the rkisp_api interface, which is built into librkisp_api.so and can be called directly. It can also be used as a reference. Its main API declaration is in the rkisp_api.h header file.
The first example, open and get a 1920x1080 image.
| 1 int test capture mmap quick() _ _ _ 2 { 3 const struct rkisp api ctx *ctx; _ _ 4 const struct rkisp api buf *buf; _ _ 5 int count = 10; 6 7 ctx = rkisp open device("/dev/video1", 0); _ _ 8 if (ctx == NULL) 9 return -1; 10 11 rkisp set fmt(ctx, 1920, 1080, ctx->fcc); _ _ 12 13 if (rkisp start capture(ctx)) _ _ 14 return -1; 15 16 do { 17 buf = rkisp get frame(ctx, 0); _ _ 18 19 /* Deal with the buffer */ 20 21 rkisp put frame(ctx, buf); _ _ 22 } while (count--); 23 24 rkisp stop capture(ctx); _ _ 25 rkisp close device(ctx); _ _ 26 27 return 0; 28 } | |
|---|---|
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | int test capture mmap quick() _ _ _ { const struct rkisp api ctx *ctx; _ _ const struct rkisp api buf *buf; _ _ int count = 10; ctx = rkisp open device("/dev/video1", 0); _ _ if (ctx == NULL) return -1; rkisp set fmt(ctx, 1920, 1080, ctx->fcc); _ _ if (rkisp start capture(ctx)) _ _ return -1; do { buf = rkisp get frame(ctx, 0); _ _ /* Deal with the buffer */ rkisp put frame(ctx, buf); _ _ } while (count--); rkisp stop capture(ctx); _ _ rkisp close device(ctx); _ _ return 0; } |
| In the a related | |
| ctx = rkisp open device("/dev/video1", 1); _ _ | |
| 1 | ctx = rkisp open device("/dev/video1", 1); _ _ |
| The ob |
The second example, modify the resolution of sensor, for example, modify the default output resolution of ov5695 to 1920x1080.
| 1 const struct rkisp api ctx *ctx; _ _ 2 3 ctx = rkisp open device("/dev/video1", 0); _ _ 4 if (ctx == NULL) 5 return -1; 6 rkisp set sensor fmt(ctx, 1920, 1080, MEDIA BUS FMT SBGGR10 1X10); _ _ _ _ _ _ _ | |
|---|---|
| 1 2 3 4 5 6 | const struct rkisp api ctx *ctx; _ _ ctx = rkisp open device("/dev/video1", 0); _ _ if (ctx == NULL) return -1; rkisp set sensor fmt(ctx, 1920, 1080, MEDIA BUS FMT SBGGR10 1X10); _ _ _ _ _ _ _ |
5.2 Share Memory by DMA Buffer
Sharing Buffer between multiple modules can reduce memory copy and improve efficiency. DMA Buffer is a detailed method. DMA Buffer can share memory between camera, RGA (image processing module), DRM (display), and MPP (coding).
By default, rkisp_api uses mmap to allocate memory from kernel. It can export memory to userspace and use it as a DMA Buffer for other modules. Conversely, it can also accept DMA Buffers from other modules and write directly to the target Buffer when Kernel collects data frames, so as to avoid once copy.
The first example, use MMAP to export DMA Buffer to other modules.
| 1 const struct rkisp api ctx *ctx; _ _ 2 const struct rkisp api buf *buf; _ _ 3 4 ctx = rkisp open device("/dev/video1", 0); _ _ 5 if (ctx == NULL || rkisp start capture(ctx)) _ _ 6 return -1; 7 8 rkisp get frame(ctx, 0); _ _ 9 printf("size: %d, dmabuf fd: %d\n", buf->size, buf->fd); | |
|---|---|
| 1 2 3 4 5 6 7 8 9 | const struct rkisp api ctx *ctx; _ _ const struct rkisp api buf *buf; _ _ ctx = rkisp open device("/dev/video1", 0); _ _ if (ctx == NULL || rkisp start capture(ctx)) _ _ return -1; rkisp get frame(ctx, 0); _ _ printf("size: %d, dmabuf fd: %d\n", buf->size, buf->fd); |
| In the |
you have to pay attention to that:
Before the buffer is used up, rkisp_put_frame() cannot be called. After using the buffer, don’t forget to call rkisp_put_frame().
The second example: using DMA Buffer of other modules, the camera data collected by kernel is directly filled into the target Buffer.
int test_capture_ext_dmabuf()
{
ctx = rkisp_open_device("/dev/video1", 0);
if (ctx == NULL)
return -1;
for (i = 0, ret = 0; i < buf_count; i++) {
if (drmGetBuffer(dev.drm_fd, width, height, FORMAT, &buf[i]))
goto out;
dmabuf_fd[i] = buf[i].dmabuf_fd;
}
rkisp_set_fmt(ctx, width, height, ctx->fcc);
rkisp_set_buf(ctx, buf_count, dmabuf_fd, buf[0].size);
if (rkisp_start_capture(ctx))
goto out;
buf = rkisp_get_frame(ctx, 0);
printf("The ext buf fd is: %d\n", buf->fd);
rkisp_put_frame(ctx, buf);
rkisp_stop_capture(ctx);
out:
while (--i >= 0)
drmPutBuffer(dev.drm_fd, &buf[i]);
rkisp_close_device(ctx);
}
In the above example, rkisp_set_buf() directly give DRM DMA Buffer to camera for use, and Buffer is allocated through the drm interface. Here we mainly introduce the usage of rkisp_api interface. The functions such as drmGetBuffer() and drmPutBuffer() will not be listed in details.
5.3 Set Area Exposure Weight
In some special cases, the default global exposure cannot get the best effect. Users can change the key area of metering and increase the weight value of this area to get the best brightness of the area. Of course, other areas may be overexposed or dark at this time. For example, the following two cases:
Face recognition under backlight. When there is no HDR, backlight makes the face area darker and is not beneficial recognition; at this time, after the face area is detected, the weight of the area can be increased to
make the face clearer.
The sweeper recognizes objects on the ground. At this time, the lower part of the image is concerned by the application, and due to the difference in object reflection, the brightness of the upper and lower parts is often different; at this time, the metering weight of the lower part of the image can be increased.
For example, call the setting weight interface:
| 1 unsigned char weights[] = { 2 1, 1, 5, 9, 15, 31, 15, 5, 1, 3 1, 1, 5, 9, 15, 31, 15, 5, 1, 4 1, 1, 5, 9, 15, 31, 15, 5, 1, 5 1, 1, 5, 9, 15, 31, 15, 5, 1, 6 1, 1, 5, 9, 15, 31, 15, 5, 1, 7 1, 1, 5, 9, 15, 31, 15, 5, 1, 8 1, 1, 5, 9, 15, 31, 15, 5, 1, 9 1, 1, 5, 9, 15, 31, 15, 5, 1, 10 1, 1, 5, 9, 15, 31, 15, 5, 1, 11 }; 12 rkisp set expo weights(ctx, weights, 81); _ _ _ | |
|---|---|
| 1 2 3 4 5 6 7 8 9 10 11 12 | unsigned char weights[] = { 1, 1, 5, 9, 15, 31, 15, 5, 1, 1, 1, 5, 9, 15, 31, 15, 5, 1, 1, 1, 5, 9, 15, 31, 15, 5, 1, 1, 1, 5, 9, 15, 31, 15, 5, 1, 1, 1, 5, 9, 15, 31, 15, 5, 1, 1, 1, 5, 9, 15, 31, 15, 5, 1, 1, 1, 5, 9, 15, 31, 15, 5, 1, 1, 1, 5, 9, 15, 31, 15, 5, 1, 1, 1, 5, 9, 15, 31, 15, 5, 1, }; rkisp set expo weights(ctx, weights, 81); _ _ _ |
| In the a regions |
5.4 Simulate to an UVC Camera by librkuvc.so
Rockchip Linux SDKs integrate librkuvc.so library, the source code of which is located in external/uvc_app . After obtaining camera images, it can be compressed and transmitted to PC via librkuvc.so.
For example, call librkuvc.so:
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 | #include "uvc control.h" _ #include "uvc video.h" _ int test uvc mmap() _ _ { const struct rkisp api ctx *ctx; _ _ const struct rkisp api buf *buf; _ _ uint32 t flags = 0; _ int extra cnt = 0; _ ctx = rkisp open device("/dev/video1", 0); _ _ if (ctx == NULL) return -1; if (rkisp set fmt(ctx, 640, 480, V4L2 PIX FMT NV12)) _ _ _ _ _ return -1; if (rkisp start capture(ctx)) _ _ return -1; flags = UVC CONTROL LOOP ONCE; _ _ _ uvc control run(flags); _ _ do { buf = rkisp get frame(ctx, 0); _ _ extra cnt++; _ uvc read camera buffer(buf->buf, buf->fd, buf->size, &extra cnt, _ _ _ _ sizeof(extra cnt)); _ rkisp put frame(ctx, buf); _ _ } while (1); uvc control join(flags); _ _ rkisp stop capture(ctx); _ _ rkisp close device(ctx); _ _ return 0; } |
|---|---|
| In the a |
| 1 # GCC=buildroot/output/rockchip rk3326 64/host/bin/aarch64-buildroot-linux- _ _ gnu-gcc 2 # SYSROOT=buildroot/output/rockchip rk3326 64/staging _ _ 3 # ENGINE=external/camera engine rkisp _ _ 4 # $GCC --sysroot=$SYSROOT camera uvc.c \ _ 5 -L$ENGINE/build/lib/ -lrkisp -lrkisp api \ _ 6 -L $SYSROOT/usr/lib/ -lrkuvc \ 7 -I $SYSROOT/usr/include/uvc/ \ 8 -I./ \ 9 -o camera uvc _ | |
|---|---|
| 1 2 3 4 5 6 7 8 9 | # GCC=buildroot/output/rockchip rk3326 64/host/bin/aarch64-buildroot-linux- _ _ gnu-gcc # SYSROOT=buildroot/output/rockchip rk3326 64/staging _ _ # ENGINE=external/camera engine rkisp _ _ # $GCC --sysroot=$SYSROOT camera uvc.c \ _ -L$ENGINE/build/lib/ -lrkisp -lrkisp api \ _ -L $SYSROOT/usr/lib/ -lrkuvc \ -I $SYSROOT/usr/include/uvc/ \ -I./ \ -o camera uvc _ |
Build and copy camera_uvc to the /usr/bin directory, and call it on the development board as follows:
# /usr/bin/uvc_MJPEG.sh
# /usr/bin/camera_uvc
A few points to note:
uvc_MJPEG.sh only needs to be initialized once after booting. Support resolutions of 640x480, 1280x720, 1920x1080, 2560x1440, please check /usr/bin/uvc_MJPEG.sh for updates.
Hardware requirements: mpp encoding requires the height of buffer to be aligned to 16, otherwise the memory access overrun will occur.
6. RKISP1 Driver Introduction
The driver code of RKISP1 is located in the drivers/media/platform/rockchip/isp1 directory. It is mainly based on v4l2 / media framework to implement hardware configuration, interrupt processing, control buffer rotation, control subdevice (such as mipi dphy and sensor) power on and off functions.
A brief introduction to the contents of each file in the driver is as follows:
| 1 # tree drivers/media/platform/rockchip/isp1/ 2 drivers/media/platform/rockchip/isp1/ 3 ├── capture.c #Contains the configuration of mp/sp and vb2, frame interrupt processing 4 ├── dev.c #Include probe, sensor registration, clock, pipeline, iommu 5 ├── isp params.c #3A Related parameter settings _ 6 ├── isp stats.c #3A related statistics _ 7 ├── regs.c #register related read and write operations 8 ├── rkisp1.c #corresponding to rkisp-isp-sd entity, including receiving data from mipi/dvp and crop function | |
|---|---|
| 1 2 3 4 5 6 7 8 | # tree drivers/media/platform/rockchip/isp1/ drivers/media/platform/rockchip/isp1/ ├── capture.c #Contains the configuration of mp/sp and vb2, frame interrupt processing ├── dev.c #Include probe, sensor registration, clock, pipeline, iommu ├── isp params.c #3A Related parameter settings _ ├── isp stats.c #3A related statistics _ ├── regs.c #register related read and write operations ├── rkisp1.c #corresponding to rkisp-isp-sd entity, including receiving data from mipi/dvp and crop function |
| The c |
The nodes in the figure below are the connection diagrams that users often encounter during development and usage, from sensor to ISP output.
Userspace Application
SP: Self Path Video Device
MP: Main Path Video Device
Memory Interface
Memory Interface
SpOutResize
MpOutResize
Self Path In Crop
Main Path In Crop
Isp Sub Device
ISP Out Crop
ISP In Crop
MIPI DPHY
DVP Sensor
MIPI Sensor A
MIPI Sensor B
As shown above, RKISP1 has the following features:
It can be adapted to MIPI or DVP interface, MIPI DPHY is required when connecting to MIPI sensor. Multiple sensors can be connected, but only one is active at the same time. After an image is input to ISP, it can be divided into two outputs of MP and SP. After MP and SP are processed based on the same raw image, the two channels can be output simultaneously. MP stands for Main Path. it can output full resolution images, up to 4416x3312. MP can output yuv or raw images, and only MP can output raw images. SP stands for Self Path. Supports up to 1920x1080 resolution. SP can output yuv or rgb images, but not raw images. Both MP and SP have crop and resize functions, which do not affect each other.
The comparison of MP and SP output functions is as follows:
| Output Device | Max Resolution | Supported Format | Crop/Resize |
|---|---|---|---|
| SP | 1920x1080 | YUV, RGB | Support |
| MP | 4416x3312 | YUV, RAW | Support |
RKISP1 also has some other nodes such as rkisp1-input-params, rkisp1-statistics, which are specially used for 3A tunning; rkisp1_rawpath and rkisp1_dmapath are used in special cases, and generally will not be used during App development.
6.1 Rkisp1 dts Board Configuration
When RK Linux SDKs are released, if the chip supports ISP, the rkisp1 node has been defined in its dtsi, such as the isp node in rk3288-rkisp1.dtsi, and the rkisp1_0 and rkisp1_1 nodes in rk3399.dtsi. The following table describes ISP information of each chip.
| Chip name | dts node name | corresponding mipi dphy | corresponding iommu |
|---|---|---|---|
| RK3399 | rkisp1 0 _ | mipi dphy rx0 _ _ | isp0 mmu _ |
| RK3399 | rkisp1 1 _ | mipi dphy tx1rx1 _ _ | isp1 mmu _ |
| RK3288 | rkisp1 | mipi phy rx0 or mipi phy tx1rx1 _ _ _ _ | isp mmu _ |
| PX30/RK3326 | rkisp1 | mipi dphy rx0 _ _ | isp mmu _ |
| RK1808 | rkisp1 | mipi dphy rx _ _ | isp mmu _ |
From the above table:
RK3399 has two isps corresponding to different dphy and mmu respectively RK3288 has only one isp, but hardware dphy can choose rx0 or tx1rx1
During board-level configuration, you only have to enable the corresponding nodes separately and establish a remote-endpoint link relationship. Please refer to the existing configuration of kernel, such as: arch/arm64/boot/dts/rockchip/rk3326-evb-lp3-v10-linux.dts , enable sensor, mipi_dphy_rx0, rkisp1, isp_mmu respectively, and set remote-endpoint associate node.
It is especially important to note that for MIPI sensors, both of the data-lane parameter sensor and mipi_dphy_rx0 need to be configured correctly with the same value.
7. RKCIF Driver Introduction
TODO.