Open Technology - Linux kernel

Video4Linux/DVB Workshop at the Kernel Summit 2011 in Prague

nospam@example.com (Guennadi Liakhovetski) — Wed, 09 Nov 2011 23:54:59 +0100

Several workshops, including one for Video4Linux and DVB, took place during the Kernel Summit 2011 in Prague. Unlike during regular V4L meetings, where we usually use 3 full days, we only had incomplete 2 days for our discussions this time, and our agenda was as full as usual. Apart from the above two links, there are also multiple other online sources, describing the events. For a general coverage of the Kernel Summit LWN has done a great job as usual. As for the V4L/DVB workshop itself, unfortunately, I don't see a long-term description of our work there. You can refer to this page for now, but as the address suggests, it will only be pointing to this event until the next one arrives. There is also this thread and its continuation, but since that mailing list has only lived as long as the workshop has been discussed, I am not sure, how long it will remain in the linuxtv.org archives. One of the outcomes of the workshops was this small experiment, the discussion continues.
As usual - many thanks go to organisers, to sponsors, and to speakers for a well-orginised and rich with high-level contents conference.

Pull request of 29 September 2011

nospam@example.com (Guennadi Liakhovetski) — Wed, 09 Nov 2011 23:24:57 +0100

This was my biggest pull request so far. Which, as it later turned out, wasn't a very good idea, instead I should have broken it into several smaller pull requests. Anyway, it seems to have worked in the end - they all are in 3.2-rc1. The list of patches is way too long for this blog (101 patches), so, I'll just provide a link to the post on the linux-media mailing list. Below are some highlights from this patch set:

1. Two new ioctl()s: VIDIOC_CREATE_BUFS and VIDIOC_PREPARE_BUF, designed to support multi-size buffers on a single queue for fast switching between different frame-formats.
2. Two new subdevice video operations: .g_mbus_config() and .s_mbus_config()
3. Soc-camera conversion to the control framework
4. Soc-camera sensor driver conversion to standard V4L2 subdevice APIs, allowing their reuse outside of the soc-camera framework
5. lots of core and individual driver updates and fixes

soc-camera 3.2 and beyond roadmap

nospam@example.com (Guennadi Liakhovetski) — Tue, 13 Sep 2011 12:46:15 +0200

As mentioned in multiple earlier posts to the linux-media mailing list, the soc-camera framework in its present state, as published in this git branch, has converted all its sensor and TV-decoder drivers, except ov6650 and mt9t031, which require a little more work, to be usable with generic V4L2 bridge drivers (thanks again to Hans Verkuil for V4L2 control framework conversion). Next on our roadmap is support for the Media Controller and pad-level APIs. Below are a couple of ideas, how this could be done, without any supporting code yet. The purpose of this post is to formalise my ideas a bit and to give you all a chance to point out any flaws in my concept. Since I haven't so far worked too closely with MC, such flaws are quite possible.

At the moment the soc-camera framework is mostly designed around a model, in which the video subsystem consists of a video capture interface on the SoC, handled as a single block, and one external capture device, like a camera sensor or a TV-decoder, connected to the above interface and additionally controlled over I2C or by some other means. Some extensions to this model, like an addition of further video processing units on the SoC, or further external modules in the video signal path are possible, but are not very well integrated into soc-camera. Examples of such extensions are the CSI-2 controller on sh-mobile and I2C lane shifters, residing on certain mt9m001 and mt9v022 camera modules and used to switch between 10- and 8-bit operation modes, when these modules are used with the PXA270 SoC.

Extending this model to better support multi-entity configurations is also on my TODO list, but is a separate task, therefore in this first step of the MC conversion I will initially address this simplistic 2-point scheme, but try to make design decisions, that would make supporting more complex configurations in the future simple enough.

The actual idea for this first step is to add an ability to support client (sensor and decoder) drivers, implementing the pad-level API, to soc-camera in a native way. This means, without wrapping subdev pad operations in standard video and core subdev operations, but by building a minimum MC implementation on top of existing soc-camera SoC (host / bridge) drivers, ideally, without having to modify them at all. That way, if a standard subdev driver is attached to your SoC, you get a standard V4L2 user-space interface, if your subdev driver implements pad-level operations, you can get a functional MC interface in user space with the same SoC driver.

Such a minimal MC implementation would create two entities: one for the actual video device (for the DMA engine), and one for the external video interface. In this simple case they shall be connected by an immutable link. The external pad will then be linked to the external video device, at least in the typical simple case of only one source pad on it.

Additionally, it should be possible for SoC drivers to implement advanced MC support, while still using parts of the soc-camera infrastructure.

In the most generic case, when both the SoC and the client drivers implement their own MC API support, the soc-camera framework should be made aware of the fact, that the configuration of all the entities in the system can change at any time, bypassing soc-camera interfaces, which means no cached values can be used by the soc-camera core.

Soc-camera July 2011 status update

nospam@example.com (Guennadi Liakhovetski) — Mon, 18 Jul 2011 12:08:39 +0200

One of present soc-camera goals is to allow re-use of client (sensor, decoder, etc.) drivers with standard V4L2 bridge (host) drivers, ideally, even with gspca;-) The soc-camera framework has been using the V4L2 subdevice API since a long time, but even until now there are several methods, that soc-camera client drivers have to provide, that are not a part of the V4L2 subdevice API. These methods are collected in struct soc_camera_ops and include two calls to configure the media bus parameters. Besides, that struct contains a list of V4L2 control descriptors, simplifying their implementation in specific drivers. After a lengthy discussion on the linux-media mailing list a patch has been accepted, that adds two V4L2 subdevice compatibility operations to allow a smooth soc-camera driver porting.
The V4L2 control support in soc-camera also has to be migrated to the new V4L2 control framework. Hans Verkuil has posted a patch series back in January 2011 to perform this migration. Unfortunately, since then he hasn't found the time to update it. When updated, this patch series will finally allow supporting of the full driver functionality both within and outside of the soc-camera subsystem.
Another less externally visible, but also very significant change in the soc-camera framework is the patch, removing the soc-camera bus type and device nodes, that used to reside on that bus. This bus has been a nice abstraction in the past to represent client devices in sysfs and to maintain object life-cycle. However, with the advancements of the V4L2 subdevice, Media Controller and media pad-level APIs, these devices have become redundant. This has been one of those nasty huge patches, simultaneously modifying the API and its users, but changes to specific drivers are really mostly very small and trivial, and doing the conversion "the right way" would have taken too much work. So, I decided to bite the bullet and pull this change through.
As for more remote tasks, soc-camera is heading towards the pad-level API, multi-size buffers, snapshot mode support and many more exciting features, as well as new wonderful and truly amazing drivers, so, stay tuned:-)

Pull request of 17 July 2011

nospam@example.com (Guennadi Liakhovetski) — Mon, 18 Jul 2011 11:59:23 +0200

This one is more exciting. On the routine side of things it includes a new camera driver for OV5640 / OV5642 OmniVision CMOS sensors. On the more experimental side this pull includes 2 new V4L2 subdevice compatibility operations for existing soc-camera drivers, removal of soc-camera PM client methods, and removal of the soc-camera bus and device nodes on it. A more detailed description of these and other forthcoming changes to the soc-camera API will be posted separately.

Bastian Hecht (1):
V4L: initial driver for ov5642 CMOS sensor

Guennadi Liakhovetski (8):
V4L: pxa-camera: switch to using standard PM hooks
V4L: soc-camera: remove now unused soc-camera specific PM hooks
V4L: soc-camera: group struct field initialisations together
V4L: add media bus configuration subdev operations
V4L: sh_mobile_csi2: switch away from using the soc-camera bus notifier
V4L: soc-camera: un-export the soc-camera bus
V4L: soc-camera: remove soc-camera bus and devices on it
V4L: sh_mobile_ceu_camera: fix Oops when USERPTR mapping fails

Michael Grzeschik (2):
V4L: mt9m111: fix missing return value check mt9m111_reg_clear
V4L: mt9m111: rewrite set_pixfmt

Pull request of 29 June 2011

nospam@example.com (Guennadi Liakhovetski) — Mon, 18 Jul 2011 11:23:40 +0200

The first pull request for the 3.1 kernel finally includes a driver for the ISI camera interface from Atmel. So far it only supports at91 ARM SoCs, maybe the time, when I'll be able to use my MT9M112 (I think), is no longer far away... We also continue to make the soc-camera client API thinner to move closer to sensor driver reuse with standard v4l2 drivers:

Andrew Chew (6):
V4L: ov9740: Cleanup hex casing inconsistencies
V4L: ov9740: Correct print in ov9740_reg_rmw()
V4L: ov9740: Fixed some settings
V4L: ov9740: Remove hardcoded resolution regs
V4L: ov9740: Reorder video and core ops
V4L: ov9740: Add suspend/resume

Guennadi Liakhovetski (11):
V4L: mx3_camera: remove redundant calculations
V4L: pxa_camera: remove redundant calculations
V4L: pxa-camera: try to force progressive video format
V4L: pxa-camera: switch to using subdev .s_power() core operation
V4L: mx2_camera: .try_fmt shouldn't fail
V4L: sh_mobile_ceu_camera: remove redundant calculations
V4L: tw9910: remove bogus ENUMINPUT implementation
V4L: soc-camera: MIPI flags are not sensor flags
V4L: mt9m111: propagate higher level abstraction down in functions
V4L: mt9m111: switch to v4l2-subdev .s_power() method
V4L: soc-camera: remove several now unused soc-camera client operations

Josh Wu (1):
V4L: at91: add Atmel Image Sensor Interface (ISI) support

Pull request of 23 June 2011

nospam@example.com (Guennadi Liakhovetski) — Mon, 18 Jul 2011 11:16:18 +0200

We only needed a single bug-fix for 3.0, so, we're nearly perfect!

Andre Bartke (1):
V4L: mx1-camera: fix uninitialized variable

Pull request of 20 May 2011

nospam@example.com (Guennadi Liakhovetski) — Sat, 21 May 2011 19:29:30 +0200

The bulk of soc-camera patches for the 2.6.40 development cycle:

Guennadi Liakhovetski (9):
V4L: sh_mobile_ceu_camera: implement .stop_streaming()
V4L: mx3_camera: implement .stop_streaming()
V4L: soc-camera: add a livecrop host operation
V4L: sh_mobile_ceu_camera: implement live cropping
V4L: soc-camera: avoid huge arrays, caused by changed format codes
V4L: omap1-camera: fix huge lookup array
V4L: soc-camera: add a new packing for YUV 4:2:0 type formats
V4L: soc-camera: add more format look-up entries
V4L: soc-camera: a missing mediabus code -> fourcc translation is not critical

Kassey Li (2):
V4L: soc-camera: add JPEG support
V4L: soc-camera: add MIPI bus flags

Teresa GÃ¡mez (2):
V4L: mt9v022: fix pixel clock
V4L: mt9m111: fix pixel clock

Pull request of 3 May 2011

nospam@example.com (Guennadi Liakhovetski) — Sat, 21 May 2011 19:27:32 +0200

One more regression fix has been target at 2.6.39, but has not managed it in before the release, has been applied to 2.6.40.

Sergio Aguirre (1):
V4L: soc-camera: regression fix: calculate .sizeimage in soc_camera.c

Pull request of 7 April 2011

nospam@example.com (Guennadi Liakhovetski) — Sat, 21 May 2011 19:23:26 +0200

Eight fixes for old and new bugs have been pushed to 2.6.39:

Guennadi Liakhovetski (8):
V4L: fix videobuf2 to correctly identify allocation failures
V4L: fix an error message
V4L: fix a macro definition
V4L: soc-camera: fix a recent multi-camera breakage on sh-mobile
V4L: imx074: return a meaningful error code instead of -1
V4L: soc-camera: don't dereference I2C client after it has been removed
V4L: sh_mobile_csi2: fix module reloading
V4L: sh_mobile_ceu_camera: fix typos in documentation

Pull request of 23 March 2011

nospam@example.com (Guennadi Liakhovetski) — Sat, 21 May 2011 19:22:14 +0200

This patch didn't make it into 2.6.39, it has been postponed until 2.6.40:

Guennadi Liakhovetski (1):
V4L: soc_camera_platform: add helper functions to manage device instances

Pull request of 22 March 2011

nospam@example.com (Guennadi Liakhovetski) — Sat, 21 May 2011 19:17:50 +0200

Four more enhancements for 2.6.39:

Guennadi Liakhovetski (1):
V4L: soc-camera: explicitly require V4L2_BUF_TYPE_VIDEO_CAPTURE

Pawel Osciak (2):
sh_mobile_ceu_camera: Do not call vb2's mem_ops directly
videobuf2-dma-contig: make cookie() return a pointer to dma_addr_t

Sergio Aguirre (1):
v4l: soc-camera: Store negotiated buffer settings

V4L2 brainstorming meeting, Warsaw, 2011

nospam@example.com (Guennadi Liakhovetski) — Mon, 21 Mar 2011 14:56:35 +0100

The 2011 V4L2 brainstorming meeting in Warsaw has been hosted by the Samsung Poland RnD Center. A wide range of topics has been discussed from 16th to 18th of March. The meeting was very well organised, for which special thanks are in order to the moderator Hans Verkuil and to Marek Szyprowski and others from Samsung Poland.

Of all the topics, discussed at the meeting the following have been specifically interesting to the author (preserved numbers from the Announcement Agenda):

3. Pipeline configuration.
Many modern video set ups consist of an input device like an image sensor and an SoC. The sensor itself can be rather complex, it can contain standard cropping, binning, skipping, flipping functions, besides, some sensors also include a DSP for colorspace conversions, precise scaling, compression, etc. The SoC also often contains multiple data processing blocks for resizing, format conversion, DMA transport into RAM or direct streaming to an output device. The new Media Controller API allows configuring each of these units separately. This ensures the greatest possible flexibility, but also presents a challenge: how to guarantee consistency of all specified parameters. Since parameters on various units are interdependent (source output data format must match sink input data format, supported framerates may depend on the data format, etc.), it is often impossible (or impractical) to require, that all intermediate states are valid. This can be solved by using a transaction-based approach.
Further, an application, communicating with the user in case of an interactive system only has a very high level information about the mode, that the user would like to configure, e.g., the input window sizes and location, output window sizes, exposure, gain, framerate. But even if the application has been specially developed for this specific platform, it cannot know how to configure each unit in the pipeline for any such possible end configuration, how to calculate "the best" pipeline set up. Therefore the only solution, that the author sees for this problem is to hard-code in user-space all configurations, that the system designer would like to support. The drivers then do not evaluate and try to fix each dubious parameter, like is currently done, if the VIDIOC_S_FMT ioctl() is issued with unsupported parameter values. The drivers in such complex set ups would then only verify, if the final (at the end of the complete transaction) configuration is valid. If yes - it will be used. If not - the transaction either fails or falls back to a single preselected default configuration, possibly provided in the platform data.

5. Sensor / Flash / Snapshot.
This topic is based on the following two RFCs from the author: [RFC] snapshot mode, flash capabilities and control and [RFC] video-buffer management optimizations.
Preview and snapshot modes are typical functions of most digital photo-camera systems. On them the user is presented with a live preview from the sensor on a relatively small size viewfinder, when the release button is pressed, a snapshot should be taken with a minimum possible delay of a significantly larger size. This short delay is currently problematic with V4L2. The biggest part of this delay comes from video-buffer management. Ways to reduce this time have been extensively discussed during days 2 and 3 of the meeting.
To further improve support for the snapshot mode, V4L2 could make use of the native single-shot mode, present on many video sensors. In this mode exposure is controlled manually by either toggling an input pin, or by writing to a trigger register. Many sensors also support the flash strobe function in the single-shot mode. Due to the lack of the time, only very general principles of the required new APIs have been discussed at the meeting. It has been agreed to present and discuss a "flash API" proposal on the mailing list.
Regarding the video-buffer management, instead of supporting multiple buffer queues per device, as has been initially proposed, it has been decided to support different size buffers on the same queue and to implement an ability to prequeue buffers. Such prequeued buffers would be fully prepared, but kept inactive on the queue. After the user switches from preview to snapshot mode, a new frame format will be set and the application will start using prequeued buffers, which would minimize the delay by avoiding expensive allocation and cache-invalidation operations. The author will develop and present an RFC with, hopefully, an example implementation to the mailing list.

24 Mar 2011 update: here is a meeting results report from Hans Verkuil.

V4L2 development

nospam@example.com (Guennadi Liakhovetski) — Mon, 14 Mar 2011 14:13:32 +0100

An interesting time for the V4L subsystem is approaching once again:-) The most visible, and, probably, significant novelty is the upstreaming of the Media Controller framework and the omap3isp driver, that is using it. This new framework significantly extends the "Video for Linux version 2" API with a comprehensive support for complex multimedia set ups, like the ones, currently implemented on many SoCs. Such set ups include Systems on a Chip, containing multiple units for media (video and audio) data processing, that can be dynamically configured and connected to form data processing pipelines. OMAP3 family of SoCs is just one example of such systems and the first one to use the new API.

Apart from that a few new ideas are currently emerging and are being discussed on the mailing list. To speed up the discussion a V4L2 brainstorming meeting has been proposed and is starting next Wednesday 16 March 2011 in Warsaw. After the meeting a report will be published on the V4L mailing list, and I will also try to collect and report at least some of my personal impressions from the meeting here.

Pull request of 21.02.2011

nospam@example.com (Guennadi Liakhovetski) — Mon, 14 Mar 2011 13:54:37 +0100

Not too many patches, but several rather significant changes with regression potential: (1) first queue videobuffers and start streaming afterwards, (2) port mx3 and sh_mobile_ceu camera host drivers to videobuf2. There also have been a couple of improvements, bug-fixes and enhancements and a new sensor driver for the OV9740 camera:

Alberto Panizzo (2):
V4L: soc_mediabus: add a method to obtain the number of samples per pixel
V4L: mx3_camera: fix capture issues for non 8-bit per pixel formats

Anatolij Gustschin (2):
V4L: soc-camera: start stream after queueing the buffers
V4L: mx3_camera: correct 'sizeimage' value reporting

Andrew Chew (1):
V4L: Initial submit of OV9740 driver.

Guennadi Liakhovetski (7):
V4L: omap1_camera: join split format lines
V4L: add missing EXPORT_SYMBOL* statements to vb2
V4L: soc-camera: extend to also support videobuf2
V4L: soc-camera: add helper functions for videobuf queue handling
V4L: sh_mobile_ceu_camera: convert to videobuf2
V4L: mx3_camera: convert to videobuf2
V4l: sh_mobile_ceu_camera: fix cropping offset calculation

Mathias Krause (1):
V4L: omap1_camera: fix use after free

Qing Xu (2):
V4L: add enum_mbus_fsizes video operation
V4L: soc-camera: add enum-frame-size ioctl