Linux kernel

V4L2 & soc-camera WiP: pad API wrappers, scaling-cropping helpers

Friday, May 3. 2013, 12:42 PM

1. V4L2 is developing at a fast pace to match new faster, higher resolution, more capable hardware. Mobile phones with multiple 1GHz+ CPU cores, gigabytes of RAM, hardware accelerated graphics, capable of driving HD displays, multiple hardware multimedia engines for video processing, compression and decompression, format conversion, capture and streaming is just one product category. This development requires not only new drivers to be developed, but also new core APIs to be designed and implemented. Soc-camera was one of the first APIs to cleanly separate camera interface drivers from sensor drivers. Then the V4L2 subdevice API followed, which has also been adopted by soc-camera. Next Media Controller API has been developed, which is still not supported by soc-camera. With the MC API various hardware blocks of video data processing pipelines like sensors, converters, scalers, video DMA engines implement "entity" objects. Data flow between those entities is performed via connected "pads." Traditionally the whole pipeline has been represented in the user-space by a single video device node. With MC, entities, pads and links, connecting them, more device nodes have been added in the user-space, users can now configure each hardware block in the pipeline independently. To support this a set of pad operations has been added to the subdevice API. Pad operations duplicate many classical subdevice operations like configuring data formats, scaling, cropping, etc., but do it in a more precisely targeted manner. Since multiple user-space applications can open the same device node simultaneously, pad operations use a notion of a file-handle to separate user contexts. Users get their own "playgrounds," where they can try various configurations without actually reconfiguring the hardware. Of course, once a user decides to push a configuration to the hardware, it becomes global.

Redundancy between the classical subdevice and the pad-level APIs make it impractical to implement both of them in subdevice drivers. Currently both kinds of subdevice drivers can be found in the kernel. This makes working with them difficult: camera interface (bridge) drivers, supporting the classical subdevice API cannot use subdevice drivers, implementing the pad-level API and vice versa. To alleviate this problem a wrapper has been proposed and posted to the linux-media mailing list. This is just the first shot and will definitely require a few more revisions, before it can be pushed to the mainline.

2. Since soc-camera doesn't (yet) support pad-level operations, video pipeline configuration from the user-space is performed in the traditional way - via a single video device node. E.g. the user has only a possibility to request a global scaling, without being able to tell, which unit in the pipeline should perform the scaling. If both the camera sensor and the camera SoC interface can scale video, the decision who should do this can easily become non-trivial. Same applies to cropping. The sh_mobile_ceu_camera soc-camera driver for r-/sh-mobile ARM and SuperH SoCs implements a rather advanced algorithm to make such decisions and to perform the required configuration. A large part of that algorithm is, however, hardware-neutral and can be made to a generic scaling-cropping helper library. The first version of such a library has been posted to the mailing list. It is hoped, that at least the emerging VIN driver will be using it instead of duplicating the functions.

Posted by Guennadi Liakhovetski in SoC camera
Comments (0) | Trackbacks (0)

Ongoing V4L2 development: Device Tree, clock helpers, asynchronous probing

Friday, May 3. 2013, 11:39 AM

1. V4L2 Device Tree support
Most of SoCs, supported by V4L2 use ARM CPU cores. Information about board hardware configuration on ARM is currently specified using Flattened Device Tree images. This made DT support in V4L2 necessary. Initial DT binding RFC has been submitted to the linux-media mailing list in July 2012. The first parser implementation with usage examples followed in September. Unfortunately, later I have been distracted by other work, and Sylwester Nawrocki has kindly and courageously picked up this development. The last version of his documentation and parser have been included in the mainline and are appearing in the 3.10 Linux kernel. Sylwester deserves every imaginable credit for this work. Unfortunately, there is no (official) way to specify multiple patch authors in the kernel at the moment.

2. V4L2 clock helpers
A typical SoC camera implementation consists of a camera sensor connected to an SoC camera interface. Camera sensors as well as other video data sources like TV decoders, usually require a "master clock" input. It is also common for such sensors to even not respond to I2C commands as long as the master clock is off. Many SoC camera interfaces have a dedicated pin to provide this clock, alternatively it can be supplied by a stand-alone oscillator. This clock has to be actively managed by the sensor driver, preferably using the standard Linux kernel clock API. This would be especially beneficial with a DT hardware description. The implementation of clock providers by generic drivers is, however, only possible if the platform implements the Common Clock Framework. Since not all ARM platforms support CCF yet, it has been decided to implement temporary simple V4L2 clock helpers, which would be replaced by proper CCF implementations as it becomes available. The last version of V4L2 clock helpers has been submitted to the mailing list in April 2013 and is currently under review. It is hoped, that this work will be included in the 3.11 kernel.

3. V4L2 asynchronous device probing
Due to the above mentioned clock dependency it is usually only possible to probe (typically I2C, less commonly SPI, etc.) devices like camera sensors after the SoC camera interface has successfully been probed and has turned on the master clock to the sensor. To achieve this currently V4L2 doesn't allow traditional I2C device registration from platform data (or DT). Instead SoC camera interfaces have to trigger dependent (I2C) device registration once they are ready to supply clock to them. This doesn't fit the DT concept very well, where we want to let camera sensors be listed just like all other I2C devices as subnodes of their respective I2C adapter DT nodes. To allow this the V4L2 subsystem has to be extended to support asynchronous device probing. Patches to implement that have been developed in parallel with V4L2 DT support and clock helpers, their last version along with usage examples has been published in the same thread with V4L2 clock helpers above. This work is also targeted to the 3.11 kernel.

All this work benefits a lot from thorough reviews of fellow V4L2 developers.

Posted by Guennadi Liakhovetski in SoC camera
Comments (0) | Trackbacks (0)

Pull request of 12 April 2013: updates

Friday, May 3. 2013, 10:17 AM

Two more soc-camera patches for 3.10:

Guennadi Liakhovetski (1):
soc-camera: fix typos in the default format-conversion table

Laurent Pinchart (1):
MAINTAINERS: Mark the SH VOU driver as Odd Fixes