Open Technology

Since 2014 I've been working at Intel, developing software for Intel(R) RealSense(TM) UVC cameras. A small part of that work resulted in several patches to the UVC driver. So far only a small number of cosmetic patches have been merged. Several more patches, adding support for metadata nodes and asynchronous status reporting, are currently under review. Their mainlining is taking a very long time because until recently the Video4Linux subsystem had no support for camera metadata. It has only been merged in 4.12, which now allows the UVC support for metadata nodes to be merged too.

The soc-camera framework in the Linux kernel has been deprecated. Since a while now new drivers are explicitly banned from using it. Most existing drivers host drivers have been patched to become stand-alone V4L2 camera host drivers. The only host driver, still using soc-camera is sh_mobile_ceu_camera.c. Soc-camera camera sensor drivers are kept in their present form until they are needed for any standard V4L2 host drivers, at which point they should be converted to stand-alone V4L2 subdevice drivers too. Respectively, the flow of soc.camera patches has almost completely ceased.
This is indeed a positive development, since more powerful and generic APIs appeared in recent years, making soc-camera really unfit for modern media pipelines.

Conversion of soc-camera camera sensor drivers to the V4L2 clock API happened to break the em28xx USB video driver, when used with an 0v2640 sensor. A lengthy discussion followed to find a suitable solution with a minimum risk of breaking numerous other em28xx configurations, which ended up with the below patch series. Additionally the pull request contained two more soc-camera related patches:

Guennadi Liakhovetski (9):
V4L2: (cosmetic) remove redundant use of unlikely()
imx074: fix error handling for failed async subdevice registration
V4L2: add a common V4L2 subdevice platform data type
soc-camera: switch to using the new struct v4l2_subdev_platform_data
V4L2: add v4l2-clock helpers to register and unregister a fixed-rate clock
V4L2: add a v4l2-clk helper macro to produce an I2C device ID
V4L2: em28xx: register a V4L2 clock source
V4L2: soc-camera: work around unbalanced calls to .s_power()
V4L2: em28xx: tell the ov2640 driver to balance clock enabling internally

Michael Opdenacker (1):
sh_mobile_ceu_camera: remove deprecated IRQF_DISABLED

Valentine Barshak (1):
media: rcar_vin: Add preliminary r8a7790 support

Two fixes for the sh_vou video output and the mx3_camera drivers:

Dan Carpenter (2):
sh_vou: almost forever loop in sh_vou_try_fmt_vid_out()
mx3-camera: locking cleanup in mx3_videobuf_queue()

A new soc-camera host driver for a VIN unit on r-car SoCs from Renesas and extensions for ms3_camera and a couple of camera sensor drivers, related to asynchronous subdevice probing, have been pushed to 3.12

Guennadi Liakhovetski (6):
V4L2: soc-camera: fix requesting regulators in synchronous case
V4L2: mx3_camera: convert to managed resource allocation
V4L2: mx3_camera: print V4L2_MBUS_FMT_* codes in hexadecimal format
V4L2: mx3_camera: add support for asynchronous subdevice registration
V4L2: mt9t031: don't Oops if asynchronous probing is attempted
V4L2: mt9m111: switch to asynchronous subdevice probing

Hans Verkuil (1):
soc_camera: fix compiler warning

Vladimir Barinov (1):
V4L2: soc_camera: Renesas R-Car VIN driver

Several improvements and fixes, related to the V4L2 clock and asynchronous probing APIs and documentation for the same:

Guennadi Liakhovetski (7):
V4L2: sh_vou: add I2C build dependency
V4L2: fix compilation if CONFIG_I2C is undefined
V4L2: soc-camera: fix uninitialised use compiler warning
V4L2: add documentation for V4L2 clock helpers and asynchronous probing
V4L2: sh_mobile_ceu_camera: remove CEU specific data from generic functions
V4L2: soc-camera: move generic functions into a separate file
V4L2: soc-camera: remove several CEU references in the generic scaler

At last V4L2 clock and asynchronous probing APIs, that have been mentioned here before have been pushed in the second set of 3.11 updates:

Guennadi Liakhovetski (20):
soc-camera: move common code to soc_camera.c
soc-camera: add host clock callbacks to start and stop the master clock
pxa-camera: move interface activation and deactivation to clock callbacks
omap1-camera: move interface activation and deactivation to clock callbacks
atmel-isi: move interface activation and deactivation to clock callbacks
mx3-camera: move interface activation and deactivation to clock callbacks
mx2-camera: move interface activation and deactivation to clock callbacks
mx1-camera: move interface activation and deactivation to clock callbacks
sh-mobile-ceu-camera: move interface activation and deactivation to clock callbacks
soc-camera: make .clock_{start,stop} compulsory, .add / .remove optional
soc-camera: don't attach the client to the host during probing
sh-mobile-ceu-camera: add primitive OF support
sh-mobile-ceu-driver: support max width and height in DT
V4L2: add temporary clock helpers
V4L2: add a device pointer to struct v4l2_subdev
V4L2: support asynchronous subdevice registration
soc-camera: switch I2C subdevice drivers to use v4l2-clk
soc-camera: add V4L2-async support
sh_mobile_ceu_camera: add asynchronous subdevice probing support
imx074: support asynchronous probing

The following nine clean up patches have been pushed in the first round of 3.11 updates:

Guennadi Liakhovetski (1):
V4L2: soc-camera: remove unneeded include path

Paul Bolle (1):
soc-camera: remove two unused configs

Sachin Kamat (6):
soc_camera: Constify dev_pm_ops in mt9t031.c
soc_camera: Fix checkpatch warning in ov9640.c
soc_camera/sh_mobile_csi2: Remove redundant platform_set_drvdata()
soc_camera_platform: Remove redundant platform_set_drvdata()
soc_camera: mt9t112: Remove empty function
soc_camera: tw9910: Remove empty function

Thomas Meyer (1):
media: Cocci spatch "ptr_ret.spatch"

Three fixes for the sh_veu mem2mem driver and a fix for the soc-camera core for 3.10:

Katsuya Matsubara (3):
sh_veu: invoke v4l2_m2m_job_finish() even if a job has been aborted
sh_veu: keep power supply until the m2m context is released
sh_veu: fix the buffer size calculation

Wenbing Wang (1):
soc_camera: error dev remove and v4l2 call

As of 3.11-rc1 V4L2 clock and asynchronous probing APIs are in the mainline. This makes it possible to initialise the V4L2 subsystem from the Device Tree without having to impose artificial requirements on drivers' probing order. The V4L2 clock API also moves us closer to supporting the Common Clock Framework universally under V4L2, while also allowing platforms, not yet implementing CCF to use the same API. This also makes subdevice driver re-use between soc-camera and the rest of V4L2 easier.

1. V4L2 is developing at a fast pace to match new faster, higher resolution, more capable hardware. Mobile phones with multiple 1GHz+ CPU cores, gigabytes of RAM, hardware accelerated graphics, capable of driving HD displays, multiple hardware multimedia engines for video processing, compression and decompression, format conversion, capture and streaming is just one product category. This development requires not only new drivers to be developed, but also new core APIs to be designed and implemented. Soc-camera was one of the first APIs to cleanly separate camera interface drivers from sensor drivers. Then the V4L2 subdevice API followed, which has also been adopted by soc-camera. Next Media Controller API has been developed, which is still not supported by soc-camera. With the MC API various hardware blocks of video data processing pipelines like sensors, converters, scalers, video DMA engines implement "entity" objects. Data flow between those entities is performed via connected "pads." Traditionally the whole pipeline has been represented in the user-space by a single video device node. With MC, entities, pads and links, connecting them, more device nodes have been added in the user-space, users can now configure each hardware block in the pipeline independently. To support this a set of pad operations has been added to the subdevice API. Pad operations duplicate many classical subdevice operations like configuring data formats, scaling, cropping, etc., but do it in a more precisely targeted manner. Since multiple user-space applications can open the same device node simultaneously, pad operations use a notion of a file-handle to separate user contexts. Users get their own "playgrounds," where they can try various configurations without actually reconfiguring the hardware. Of course, once a user decides to push a configuration to the hardware, it becomes global.

Redundancy between the classical subdevice and the pad-level APIs make it impractical to implement both of them in subdevice drivers. Currently both kinds of subdevice drivers can be found in the kernel. This makes working with them difficult: camera interface (bridge) drivers, supporting the classical subdevice API cannot use subdevice drivers, implementing the pad-level API and vice versa. To alleviate this problem a wrapper has been proposed and posted to the linux-media mailing list. This is just the first shot and will definitely require a few more revisions, before it can be pushed to the mainline.

2. Since soc-camera doesn't (yet) support pad-level operations, video pipeline configuration from the user-space is performed in the traditional way - via a single video device node. E.g. the user has only a possibility to request a global scaling, without being able to tell, which unit in the pipeline should perform the scaling. If both the camera sensor and the camera SoC interface can scale video, the decision who should do this can easily become non-trivial. Same applies to cropping. The sh_mobile_ceu_camera soc-camera driver for r-/sh-mobile ARM and SuperH SoCs implements a rather advanced algorithm to make such decisions and to perform the required configuration. A large part of that algorithm is, however, hardware-neutral and can be made to a generic scaling-cropping helper library. The first version of such a library has been posted to the mailing list. It is hoped, that at least the emerging VIN driver will be using it instead of duplicating the functions.

1. V4L2 Device Tree support
Most of SoCs, supported by V4L2 use ARM CPU cores. Information about board hardware configuration on ARM is currently specified using Flattened Device Tree images. This made DT support in V4L2 necessary. Initial DT binding RFC has been submitted to the linux-media mailing list in July 2012. The first parser implementation with usage examples followed in September. Unfortunately, later I have been distracted by other work, and Sylwester Nawrocki has kindly and courageously picked up this development. The last version of his documentation and parser have been included in the mainline and are appearing in the 3.10 Linux kernel. Sylwester deserves every imaginable credit for this work. Unfortunately, there is no (official) way to specify multiple patch authors in the kernel at the moment.

2. V4L2 clock helpers
A typical SoC camera implementation consists of a camera sensor connected to an SoC camera interface. Camera sensors as well as other video data sources like TV decoders, usually require a "master clock" input. It is also common for such sensors to even not respond to I2C commands as long as the master clock is off. Many SoC camera interfaces have a dedicated pin to provide this clock, alternatively it can be supplied by a stand-alone oscillator. This clock has to be actively managed by the sensor driver, preferably using the standard Linux kernel clock API. This would be especially beneficial with a DT hardware description. The implementation of clock providers by generic drivers is, however, only possible if the platform implements the Common Clock Framework. Since not all ARM platforms support CCF yet, it has been decided to implement temporary simple V4L2 clock helpers, which would be replaced by proper CCF implementations as it becomes available. The last version of V4L2 clock helpers has been submitted to the mailing list in April 2013 and is currently under review. It is hoped, that this work will be included in the 3.11 kernel.

3. V4L2 asynchronous device probing
Due to the above mentioned clock dependency it is usually only possible to probe (typically I2C, less commonly SPI, etc.) devices like camera sensors after the SoC camera interface has successfully been probed and has turned on the master clock to the sensor. To achieve this currently V4L2 doesn't allow traditional I2C device registration from platform data (or DT). Instead SoC camera interfaces have to trigger dependent (I2C) device registration once they are ready to supply clock to them. This doesn't fit the DT concept very well, where we want to let camera sensors be listed just like all other I2C devices as subnodes of their respective I2C adapter DT nodes. To allow this the V4L2 subsystem has to be extended to support asynchronous device probing. Patches to implement that have been developed in parallel with V4L2 DT support and clock helpers, their last version along with usage examples has been published in the same thread with V4L2 clock helpers above. This work is also targeted to the 3.11 kernel.

All this work benefits a lot from thorough reviews of fellow V4L2 developers.

Two more soc-camera patches for 3.10:

Guennadi Liakhovetski (1):
soc-camera: fix typos in the default format-conversion table

Laurent Pinchart (1):
MAINTAINERS: Mark the SH VOU driver as Odd Fixes

The second set of 3.10 soc-camera updates:

Fabio Porcedda (2):
drivers: media: use module_platform_driver_probe()
mx2_camera: use module_platform_driver_probe()

Guennadi Liakhovetski (1):
soc-camera: protect against racing open(2) and rmmod

Phil Edworthy (1):
soc_camera: Add RGB666 & RGB888 formats

Sachin Kamat (7):
soc_camera/mx1_camera: Fix warnings related to spacing
soc_camera/mx2_camera: Fix warnings related to spacing
soc_camera/mx3_camera: Fix warning related to spacing
soc_camera/pxa_camera: Fix warning related to spacing
soc_camera/pxa_camera: Constify struct dev_pm_ops
soc_camera/sh_mobile_ceu_camera: Fix warning related to spacing
soc_camera/soc_camera_platform: Fix warning related to spacing

Minor clean up and API updates:

Paul Bolle (1):
soc_camera: remove two outdated selects

Sachin Kamat (7):
soc_camera/sh_mobile_ceu_camera: Convert to devm_ioremap_resource()
soc_camera/sh_mobile_csi2: Convert to devm_ioremap_resource()
soc_camera/pxa_camera: Convert to devm_ioremap_resource()
sh_veu.c: Convert to devm_ioremap_resource()
soc_camera/mx1_camera: Use module_platform_driver_probe macro
sh_veu: Use module_platform_driver_probe macro
sh_vou: Use module_platform_driver_probe macro