SoC camera

To address point 4 from this post I have written and posted an RFC and a series of patches to introduce an Imagebus API for negotiation of the data format on the video bus, i.e., on the bus between a client (e.g., a camera sensor) and a host (an SoC). In the patch series I added the necessary minimal API and ported all soc-camera drivers to Imagebus.

As the discussion of other RFCs progressed, the video data format negotiation has been merged with the bus configuration RFC in the latest version, so, the actual implementation might eventually diverge from my original proposal.

The bulk of the soc-camera to v4l2-subdev conversion, consisting of 35 patches, has been pushed to v4l hg:

01/35: ARM: convert pcm990 to soc-camera as platform-device
02/35: soc-camera: prepare soc_camera_platform.c and its users for conversion
03/35: soc_camera_platform: pass device pointer from soc-camera core on .add_device()
04/35: soc-camera: convert to platform device
05/35: sh: soc-camera updates
06/35: soc-camera: remove unused .iface from struct soc_camera_platform_info
07/35: sh: prepare board-ap325rxa.c for v4l2-subdev conversion
08/35: soc-camera: (partially) convert to v4l2-(sub)dev API
09/35: V4L2: add a new V4L2_CID_BAND_STOP_FILTER integer control
10/35: ov772x: implement a band-stop filter support
11/35: soc-camera: add support for camera-host controls
12/35: sh_mobile_ceu_camera: add a control for the camera low-pass filter
13/35: soc-camera: use struct v4l2_rect in struct soc_camera_device
14/35: ov772x: successful S_FMT and S_CROP must update user-provided rectangle
15/35: mt9t031: improve rectangle placement in invalid S_CROP
16/35: ov772x: S_CROP must return actually configured geometry
17/35: soc-camera: put pixel format initialisation back in probe, add .put_formats()
18/35: sh-mobile-ceu-camera: do not wait for interrupt when releasing buffers
19/35: soc-camera: use .s_std() from struct v4l2_subdev_core_ops
20/35: sh-mobile-ceu-camera: implement host-side cropping
21/35: tw9910: return updated geometry on successful S_FMT and S_CROP
22/35: soc-camera: S_CROP V4L2 API compliance fix
23/35: soc-camera: prohibit geometry change with initialised buffers
24/35: ov772x: do not use scaling for cropping
25/35: tw9910: do not lie about cropping abilities
26/35: sh_mobile_ceu_camera: implement host-side image scaling
27/35: soc-camera: switch to s_crop v4l2-subdev video operation
28/35: soc-camera: switch to using v4l2_subdev_call()
29/35: soc-camera: Use I2C device for dev_{dbg,info,...} output in all clients
30/35: soc-camera: Use camera device object for core output
31/35: soc-camera: Use video device object for output in host drivers
32/35: soc-camera: V4L2 API compliant scaling (S_FMT) and cropping (S_CROP)
33/35: soc-camera: remove .init() and .release() methods from struct soc_camera_ops
34/35: soc-camera: remove .gain and .exposure struct soc_camera_device members
35/35: soc-camera: fix recently introduced overlong lines

aiming at 2.6.32, a few more patches should follow.

On the 22 of August 2009 an unofficial meeting took place with the purpose to discuss a new RFC for a new V4L API. Participants were (alphabetically sorted): Hans de Goede, Laurent Pinchart, Hans Verkuil and myself. The main discussion of this API will take place during the Linux Plumbers Conference, but since not everyone, interested in this development, including the author of these pages, will be able to attend the conference, it has been decided to hold a small unofficial preliminary meeting on this topic. The following implications on the soc-camera migration to v4l2-subdev have been decided upon:

1. We will have to remove bus-parameter auto-negotiation and force every board to specify a complete list of bus parameters, like signal polarities and sensing edge.
   
2. Will consider using the .notify() member of struct v4l2_device in sensor drivers, that change their output pixel clock frequency dynamically, e.g., depending on the frame format, instead of using the current sense object.
   
3. Maybe use a list of video subdevices as a parameter to soc-camera host platform device to avoid allocating otherwise "useless" per-source platform devices.
   
4. Use a new enumeration for various pixel data representations on the video source - video host data bus, similar to the current fourcc codes for the in-memory presentation of pixel formats.

We will have to give a bit more thinking to items 2 and 3 though.

All soc-camera drivers have been updated to comply with the V4L2 API scaling and cropping definitions. The main differences to the original soc-camera implementation are:

1. VIDIOC_S_CROP ioctl now accepts rectangle parameters in sensor input window pixel units
   
2. VIDIOC_S_CROP ioctl now tries to preserve scales, and not the output window
   
3. VIDIOC_S_FMT ioctl now preserves the sensor input window
   

The patch stack is uploaded here and is based on the 2.6.31-rc4 kernel.

A short announcement: my talk about the soc-camera framework has been accepted for the ELC-E 2009 conference in October 2009 in Grenoble, France. So, maybe, see you there.

While adding more hardware support and functionality to soc-camera, it became apparent, that its cropping / scaling concept was wrong. Trying to fix the situation by reading the API specs and trying to understand it revealed, that the documentation is indeed quite cumbersome and that even experienced V4L kernel developers do not understand completely how VIDIOC_S_CROP and VIDIOC_S_FMT ioctls shall configure cropping and scaling in camera drivers. As the situation slowly clarifies all soc-camera drivers and the core have to be modified to comply with an interpretation of the V4L2 API, which will hopefully result from the currently running on the mailing list discussion.

Porting soc-camera with all its drivers and platforms to the v4l2-subdev framework confirmed to be a complicated task. Having prepared the patches in a way to make changes as isolated as possible, they now have to be applied to the mainline in a certain order avoiding any intermittent breakage. After the 2.6.30 kernel has been released earlier this week, two first conversion steps shall be pushed upstream during the 2.6.31 merge window: a preparatory forward-compatibility patch to the soc-camera core, and multiple patches to convert all platforms currently in the mainline to the future platform-binding scheme. The actual incomplete conversion patches will then have to wait until 2.6.32. Then we will be completing the conversion some time in the 2.6.33 timeframe. For details see my post at the linux-media mailing list.

Following the "01/01: soc-camera: prepare for the platform driver conversion" patch from 07.05 6 further patches have been submitted to ARM and SuperH trees to convert all 5 currently in the mainline soc-camera platforms to the new scheme.

After that the platform-driver and v4l2-subdev conversion patches have been updated, posted to the linux-media mailing list on 15.05 and uploaded to http://download.open-technology.de/soc-camera/20090515/. See this thread for a detailed description.

Following patches have been pushed to v4l-dvb hg:

01/02: soc-camera: Fix section mismatch warning
02/02: soc_camera: depends on I2C

Following patch has been pushed to v4l-dvb hg:

01/01: soc-camera: prepare for the platform driver conversion

Following patches have been pushed to v4l-dvb hg:

01/08: ARM: update pcm990-baseboard.c to match mainline
02/08: soc-camera: add a free_bus method to struct soc_camera_link
03/08: soc-camera: host-driver cleanup
04/08: soc-camera: remove an extra device generation from struct soc_camera_host
05/08: soc-camera: simplify register access routines in multiple sensor drivers
06/08: soc-camera: link host drivers after clients
07/08: mx3_camera: Fix compilation with CONFIG_PM
08/08: pxa_camera: Documentation of the FSM

Completed step 1 of soc-camera conversion to the v4l2 framework. This means, all drivers and platforms have been converted to a scheme, where camera instances are registered by platforms as platform devices, and the soc-camera registers I2C devices later (see also my earlier posts). I also removed struct device from struct soc_camera_host, making camera instances direct children of host platform devices. In the meantime soc-camera framework covers four architectures: PXA270, SH7722, i.MX31, and i.MX1, is used by 6 in-tree platforms: pcm037, pcm990, em-x270, mioa701, ap325rxa, migor, and supports 6 client devices: mt9m001, mt9m111, mt9t031, mt9v022, ov772x, tw9910.

Following patches have been pushed:

01/09: pxa_camera: Enforce YUV422P frame sizes to be 16 multiples
02/09: pxa_camera: Remove YUV planar formats hole
03/09: pxa_camera: Redesign DMA handling
04/09: pxa_camera: Fix overrun condition on last buffer
05/09: pxa-camera: simplify the .buf_queue path by merging two loops
06/09: ov772x: wrong pointer for soc_camera_link is modified
07/09: soc-camera: fix breakage caused by 1fa5ae857bb14f6046205171d98506d8112dd74e
08/09: mt9m001: fix advertised pixel clock polarity
09/09: ov772x: add edge contrl support

The first step of the soc-camera framework conversion to the v4l2-device API is the transformation of the soc-camera core driver to a platform driver. This way the bond between soc-camera and single (sensor, decoder,...) I2C device drivers is becoming weaker and the double probing of those I2C drivers is eliminated.

The reason for the double probing previously has been, that many camera sensors have their I2C interfaces disabled in the absence of the master clock. Therefore the actual I2C probing routine could not access the chip and only registered the device with the soc-camera framework and reported success unconditionally. Then, when the soc-camera framework found a match between a camera device and a host, it registered respective devices and the second stage probing has been initiated, which switched the master clock of the camera interface on and made the actual I2C probing possible.

With the new platform-device approach first soc-camera core probe method is called, which switches the master clock on, and when the camera device driver becomes available, its I2C probe method is called and it can immediately access device I2C registers and perform the actual device identification and initialisation.

The first part of this conversion has been to convert a specific configuration to this scheme. The chosen configuration is the i.MX31 host and a MT9T031 camera, which have been successfully converted and an RFC patch has been posted to the V4L mailing list.

Following patches have been pushed:

01/12: soc-camera: separate S_FMT and S_CROP operations
02/12: soc-camera: configure drivers with a default format on open
03/12: sh-mobile-ceu-camera: set field to the value, configured at open()
04/12: soc-camera: configure drivers with a default format at probe time
05/12: ov772x: use soft sleep mode in stop_capture
06/12: video: use videobuf_waiton() in sh_mobile_ceu free_buffer()
07/12: soc-camera: add board hook to specify the buswidth for camera sensors
08/12: pcm990 baseboard: add camera bus width switch setting
09/12: mt9m001: allow setting of bus width from board code
10/12: mt9v022: allow setting of bus width from board code
11/12: soc-camera: remove now unused gpio member of struct soc_camera_link
12/12: mt9t031 bugfix