Open Technology

1. V4L2 is developing at a fast pace to match new faster, higher resolution, more capable hardware. Mobile phones with multiple 1GHz+ CPU cores, gigabytes of RAM, hardware accelerated graphics, capable of driving HD displays, multiple hardware multimedia engines for video processing, compression and decompression, format conversion, capture and streaming is just one product category. This development requires not only new drivers to be developed, but also new core APIs to be designed and implemented. Soc-camera was one of the first APIs to cleanly separate camera interface drivers from sensor drivers. Then the V4L2 subdevice API followed, which has also been adopted by soc-camera. Next Media Controller API has been developed, which is still not supported by soc-camera. With the MC API various hardware blocks of video data processing pipelines like sensors, converters, scalers, video DMA engines implement "entity" objects. Data flow between those entities is performed via connected "pads." Traditionally the whole pipeline has been represented in the user-space by a single video device node. With MC, entities, pads and links, connecting them, more device nodes have been added in the user-space, users can now configure each hardware block in the pipeline independently. To support this a set of pad operations has been added to the subdevice API. Pad operations duplicate many classical subdevice operations like configuring data formats, scaling, cropping, etc., but do it in a more precisely targeted manner. Since multiple user-space applications can open the same device node simultaneously, pad operations use a notion of a file-handle to separate user contexts. Users get their own "playgrounds," where they can try various configurations without actually reconfiguring the hardware. Of course, once a user decides to push a configuration to the hardware, it becomes global.

Redundancy between the classical subdevice and the pad-level APIs make it impractical to implement both of them in subdevice drivers. Currently both kinds of subdevice drivers can be found in the kernel. This makes working with them difficult: camera interface (bridge) drivers, supporting the classical subdevice API cannot use subdevice drivers, implementing the pad-level API and vice versa. To alleviate this problem a wrapper has been proposed and posted to the linux-media mailing list. This is just the first shot and will definitely require a few more revisions, before it can be pushed to the mainline.

2. Since soc-camera doesn't (yet) support pad-level operations, video pipeline configuration from the user-space is performed in the traditional way - via a single video device node. E.g. the user has only a possibility to request a global scaling, without being able to tell, which unit in the pipeline should perform the scaling. If both the camera sensor and the camera SoC interface can scale video, the decision who should do this can easily become non-trivial. Same applies to cropping. The sh_mobile_ceu_camera soc-camera driver for r-/sh-mobile ARM and SuperH SoCs implements a rather advanced algorithm to make such decisions and to perform the required configuration. A large part of that algorithm is, however, hardware-neutral and can be made to a generic scaling-cropping helper library. The first version of such a library has been posted to the mailing list. It is hoped, that at least the emerging VIN driver will be using it instead of duplicating the functions.

1. V4L2 Device Tree support
Most of SoCs, supported by V4L2 use ARM CPU cores. Information about board hardware configuration on ARM is currently specified using Flattened Device Tree images. This made DT support in V4L2 necessary. Initial DT binding RFC has been submitted to the linux-media mailing list in July 2012. The first parser implementation with usage examples followed in September. Unfortunately, later I have been distracted by other work, and Sylwester Nawrocki has kindly and courageously picked up this development. The last version of his documentation and parser have been included in the mainline and are appearing in the 3.10 Linux kernel. Sylwester deserves every imaginable credit for this work. Unfortunately, there is no (official) way to specify multiple patch authors in the kernel at the moment.

2. V4L2 clock helpers
A typical SoC camera implementation consists of a camera sensor connected to an SoC camera interface. Camera sensors as well as other video data sources like TV decoders, usually require a "master clock" input. It is also common for such sensors to even not respond to I2C commands as long as the master clock is off. Many SoC camera interfaces have a dedicated pin to provide this clock, alternatively it can be supplied by a stand-alone oscillator. This clock has to be actively managed by the sensor driver, preferably using the standard Linux kernel clock API. This would be especially beneficial with a DT hardware description. The implementation of clock providers by generic drivers is, however, only possible if the platform implements the Common Clock Framework. Since not all ARM platforms support CCF yet, it has been decided to implement temporary simple V4L2 clock helpers, which would be replaced by proper CCF implementations as it becomes available. The last version of V4L2 clock helpers has been submitted to the mailing list in April 2013 and is currently under review. It is hoped, that this work will be included in the 3.11 kernel.

3. V4L2 asynchronous device probing
Due to the above mentioned clock dependency it is usually only possible to probe (typically I2C, less commonly SPI, etc.) devices like camera sensors after the SoC camera interface has successfully been probed and has turned on the master clock to the sensor. To achieve this currently V4L2 doesn't allow traditional I2C device registration from platform data (or DT). Instead SoC camera interfaces have to trigger dependent (I2C) device registration once they are ready to supply clock to them. This doesn't fit the DT concept very well, where we want to let camera sensors be listed just like all other I2C devices as subnodes of their respective I2C adapter DT nodes. To allow this the V4L2 subsystem has to be extended to support asynchronous device probing. Patches to implement that have been developed in parallel with V4L2 DT support and clock helpers, their last version along with usage examples has been published in the same thread with V4L2 clock helpers above. This work is also targeted to the 3.11 kernel.

All this work benefits a lot from thorough reviews of fellow V4L2 developers.

Two more soc-camera patches for 3.10:

Guennadi Liakhovetski (1):
soc-camera: fix typos in the default format-conversion table

Laurent Pinchart (1):
MAINTAINERS: Mark the SH VOU driver as Odd Fixes

The second set of 3.10 soc-camera updates:

Fabio Porcedda (2):
drivers: media: use module_platform_driver_probe()
mx2_camera: use module_platform_driver_probe()

Guennadi Liakhovetski (1):
soc-camera: protect against racing open(2) and rmmod

Phil Edworthy (1):
soc_camera: Add RGB666 & RGB888 formats

Sachin Kamat (7):
soc_camera/mx1_camera: Fix warnings related to spacing
soc_camera/mx2_camera: Fix warnings related to spacing
soc_camera/mx3_camera: Fix warning related to spacing
soc_camera/pxa_camera: Fix warning related to spacing
soc_camera/pxa_camera: Constify struct dev_pm_ops
soc_camera/sh_mobile_ceu_camera: Fix warning related to spacing
soc_camera/soc_camera_platform: Fix warning related to spacing

Minor clean up and API updates:

Paul Bolle (1):
soc_camera: remove two outdated selects

Sachin Kamat (7):
soc_camera/sh_mobile_ceu_camera: Convert to devm_ioremap_resource()
soc_camera/sh_mobile_csi2: Convert to devm_ioremap_resource()
soc_camera/pxa_camera: Convert to devm_ioremap_resource()
sh_veu.c: Convert to devm_ioremap_resource()
soc_camera/mx1_camera: Use module_platform_driver_probe macro
sh_veu: Use module_platform_driver_probe macro
sh_vou: Use module_platform_driver_probe macro

Several sensor-driver bug-fix patches, targeted at 3.9 and stable, unfortunately, didn't make it in 3.9, will be applied to 3.10:

Andrei Andreyanau (1):
mt9v022 driver: send valid HORIZONTAL_BLANKING values to mt9v024 soc camera

Benoît Thébaudeau (1):
soc-camera: mt9m111: Fix auto-exposure control

Guennadi Liakhovetski (1):
mt9m111: fix Oops - initialise context before dereferencing

Several minor driver improvements, no real functionality changes.

Julia Lawall (1):
drivers/media/platform/soc_camera/pxa_camera.c: use devm_ functions

Laurent Pinchart (2):
sh_vou: Use video_drvdata()
sh_vou: Use vou_dev instead of vou_file wherever possible

Two driver fixes:

Guennadi Liakhovetski (2):
sh-mobile-ceu-camera: fix SHARPNESS control default
mt9t112: mt9t111 format set up differs from mt9t112

Several updates and fixes for the 3.9 series, including - after a long battle - the removal of i.MX25 support from the mx2_camera driver, it now only supports i.MX27.

Fabio Estevam (1):
mx2_camera: Convert it to platform driver

Guennadi Liakhovetski (5):
soc-camera: properly fix camera probing races
soc-camera: fix repeated regulator requesting
soc-camera: remove struct soc_camera_device::video_lock
soc-camera: split struct soc_camera_link into host and subdevice parts
soc-camera: use devm_kzalloc in subdevice drivers

Javier Martin (3):
mx2_camera: Remove i.mx25 support.
mx2_camera: Remove 'buf_cleanup' callback.
mx2_camera: Remove buffer states.

Several commits ended up not making it into the V4L tree from my previous pull requests, so, they are repeated here. Additionally a long overdue soc-camera documentation update, several managed resource allocation optimisations, other minor updates.

Guennadi Liakhovetski (9):
media: mem2mem: make reference to struct m2m_ops in the core const
media: add a VEU MEM2MEM format conversion and scaling driver
media: soc-camera: use managed devm_regulator_bulk_get()
media: sh-mobile-ceu-camera: runtime PM suspending doesn't have to be synchronous
media: soc-camera: update documentation
media: soc-camera: remove superfluous JPEG checking
media: sh_mobile_csi2: use managed memory and resource allocations
media: sh_mobile_ceu_camera: use managed memory and resource allocations
MAINTAINERS: add entries for sh_veu and sh_vou V4L2 drivers

Javier Martin (1):
media: mx2_camera: Add image size HW limits.

Several fixes, which, however, somehow ended up being delayed a lot:

Anatolij Gustschin (1):
soc_camera: fix VIDIOC_S_CROP ioctl

Cyril Roelandt (1):
mx2_camera: use GFP_ATOMIC under spin lock.

Guennadi Liakhovetski (1):
media: sh-vou: fix compiler warnings

One new (not soc-camera) driver for a mem2mem video processing device, one driver (mx1_camera) declared broken, several driver updates.

Anatolij Gustschin (4):
V4L: soc_camera: allow reading from video device if supported
mt9v022: add v4l2 controls for blanking
mt9v022: support required register settings in snapshot mode
mt9v022: set y_skip_top field to zero as default

Frank SchÀfer (1):
ov2640: add support for V4L2_MBUS_FMT_YUYV8_2X8, V4L2_MBUS_FMT_RGB565_2X8_BE

Guennadi Liakhovetski (1):
media: add a VEU MEM2MEM format conversion and scaling driver

Shawn Guo (1):
media: mx1_camera: mark the driver BROKEN

Mostly compiler warning fixes for 3.7 after the struct v4l2_ioctl_ops::vidioc_s_crop prototype change, additionally a V4L2 control implementation fix for mt9v022

Anatolij Gustschin (1):
mt9v022: fix the V4L2_CID_EXPOSURE control

Guennadi Liakhovetski (7):
media: sh_vou: fix const cropping related warnings
media: sh_mobile_ceu_camera: fix const cropping related warnings
media: pxa_camera: fix const cropping related warnings
media: mx3_camera: fix const cropping related warnings
media: mx2_camera: fix const cropping related warnings
media: mx1_camera: use the default .set_crop() implementation
media: omap1_camera: fix const cropping related warnings

My first version of the Android generic V4L2 camera HAL had 2 CPU-intensive stages in the camera preview pipeline. The first of them was converting video data from the camera output format (YUV) to an RGB format, suitable for displaying on the framebuffer. The second one is surface compositing onto the framebuffer. I tried to address the former by utilising a VEU (Video Engine Unit) block on SH7372. I used an existing vidix driver for MPlayer as a basis and implemented a Mem2Mem V4L2 VEU driver. Tests under Linux were successful and showed good performance with reduced CPU load. However, making Android to use the driver to convert data from camera output buffers directly into the preview layer buffer turned out to be difficult, because Android uses system memory to allocate virtually-continuous surface buffers. VEU, however, requires physically-continuous buffers for its DMA engine. In recent (4.0-4.1) Android versions this should be doable, using the ION memory manager on top of the Linux dmabuf subsystem. However, ION hasn't yet been integrated in the Android 4.0 version, that I've been using and switching to it would be a major development. As a proof of concept a short-cut has been implemented to force Android to use user-supplied buffers for the camera preview surface. These buffers are obtained from the VEU driver, using the MMAP method and are thus physically-continuous. This indeed increased the capture framerate from about 4 to the full 27 frames per second. However, this still hasn't produces any visible improvements, because software compositing still kept the display framerate below 5 fps. A proper solution would involve using ION for preview buffer management and implementing a hardware-accelerated compositing solution.

A last minute 3.6 fixes pull request has been sent, but, unfortunately, hasn't managed it in before the release, they will have to be pushed to 3.6-stable:

Frank Schäfer (1):
ov2640: select sensor register bank before applying h/v-flip settings

Peter Senna Tschudin (2):
drivers/media/video/mx2_camera.c: fix error return code
drivers/media/video/soc_camera.c: fix error return code