The 2011 V4L2 brainstorming meeting in Warsaw has been hosted by the
Samsung Poland RnD Center. A wide range of
topics has been discussed from 16th to 18th of March. The meeting was very well organised, for which special thanks are in order to the moderator Hans Verkuil and to Marek Szyprowski and others from Samsung Poland.
Of all the topics, discussed at the meeting the following have been specifically interesting to the author (preserved numbers from the Announcement Agenda):
3. Pipeline configuration.
Many modern video set ups consist of an input device like an image sensor and an SoC. The sensor itself can be rather complex, it can contain standard cropping, binning, skipping, flipping functions, besides, some sensors also include a DSP for colorspace conversions, precise scaling, compression, etc. The SoC also often contains multiple data processing blocks for resizing, format conversion, DMA transport into RAM or direct streaming to an output device. The new
Media Controller API allows configuring each of these units separately. This ensures the greatest possible flexibility, but also presents a
challenge: how to guarantee consistency of all specified parameters. Since parameters on various units are interdependent (source output data format must match sink input data format, supported framerates may depend on the data format, etc.), it is often impossible (or impractical) to require, that all intermediate states are valid. This can be solved by using a transaction-based approach.
Further, an application, communicating with the user in case of an interactive system only has a very high level information about the mode, that the user would like to configure, e.g., the input window sizes and location, output window sizes, exposure, gain, framerate. But even if the application has been specially developed for this specific platform, it cannot know how to configure each unit in the pipeline for any such possible end configuration, how to calculate "the best" pipeline set up. Therefore the only solution, that the author sees for this problem is to hard-code in user-space all configurations, that the system designer would like to support. The drivers then do not evaluate and try to fix each dubious parameter, like is currently done, if the VIDIOC_S_FMT ioctl() is issued with unsupported parameter values. The drivers in such complex set ups would then only verify, if the final (at the end of the complete transaction) configuration is valid. If yes - it will be used. If not - the transaction either fails or falls back to a single preselected default configuration, possibly provided in the platform data.
5. Sensor / Flash / Snapshot.
This topic is based on the following two RFCs from the author:
[RFC] snapshot mode, flash capabilities and control and
[RFC] video-buffer management optimizations.
Preview and snapshot modes are typical functions of most digital photo-camera systems. On them the user is presented with a live preview from the sensor on a relatively small size viewfinder, when the release button is pressed, a snapshot should be taken with a minimum possible delay of a significantly larger size. This short delay is currently problematic with V4L2. The biggest part of this delay comes from video-buffer management. Ways to reduce this time have been extensively discussed during days 2 and 3 of the meeting.
To further improve support for the snapshot mode, V4L2 could make use of the native single-shot mode, present on many video sensors. In this mode exposure is controlled manually by either toggling an input pin, or by writing to a trigger register. Many sensors also support the flash strobe function in the single-shot mode. Due to the lack of the time, only very general principles of the required new APIs have been discussed at the meeting. It has been agreed to present and discuss a "flash API" proposal on the mailing list.
Regarding the video-buffer management, instead of supporting multiple buffer queues per device, as has been initially proposed, it has been decided to support different size buffers on the same queue and to implement an ability to prequeue buffers. Such prequeued buffers would be fully prepared, but kept inactive on the queue. After the user switches from preview to snapshot mode, a new frame format will be set and the application will start using prequeued buffers, which would minimize the delay by avoiding expensive allocation and cache-invalidation operations. The author will develop and present an RFC with, hopefully, an example implementation to the mailing list.
24 Mar 2011 update: here is a meeting results
report from Hans Verkuil.
