Open Technology

To address point 4 from this post I have written and posted an RFC and a series of patches to introduce an Imagebus API for negotiation of the data format on the video bus, i.e., on the bus between a client (e.g., a camera sensor) and a host (an SoC). In the patch series I added the necessary minimal API and ported all soc-camera drivers to Imagebus.

As the discussion of other RFCs progressed, the video data format negotiation has been merged with the bus configuration RFC in the latest version, so, the actual implementation might eventually diverge from my original proposal.

The bulk of the soc-camera to v4l2-subdev conversion, consisting of 35 patches, has been pushed to v4l hg:

01/35: ARM: convert pcm990 to soc-camera as platform-device
02/35: soc-camera: prepare soc_camera_platform.c and its users for conversion
03/35: soc_camera_platform: pass device pointer from soc-camera core on .add_device()
04/35: soc-camera: convert to platform device
05/35: sh: soc-camera updates
06/35: soc-camera: remove unused .iface from struct soc_camera_platform_info
07/35: sh: prepare board-ap325rxa.c for v4l2-subdev conversion
08/35: soc-camera: (partially) convert to v4l2-(sub)dev API
09/35: V4L2: add a new V4L2_CID_BAND_STOP_FILTER integer control
10/35: ov772x: implement a band-stop filter support
11/35: soc-camera: add support for camera-host controls
12/35: sh_mobile_ceu_camera: add a control for the camera low-pass filter
13/35: soc-camera: use struct v4l2_rect in struct soc_camera_device
14/35: ov772x: successful S_FMT and S_CROP must update user-provided rectangle
15/35: mt9t031: improve rectangle placement in invalid S_CROP
16/35: ov772x: S_CROP must return actually configured geometry
17/35: soc-camera: put pixel format initialisation back in probe, add .put_formats()
18/35: sh-mobile-ceu-camera: do not wait for interrupt when releasing buffers
19/35: soc-camera: use .s_std() from struct v4l2_subdev_core_ops
20/35: sh-mobile-ceu-camera: implement host-side cropping
21/35: tw9910: return updated geometry on successful S_FMT and S_CROP
22/35: soc-camera: S_CROP V4L2 API compliance fix
23/35: soc-camera: prohibit geometry change with initialised buffers
24/35: ov772x: do not use scaling for cropping
25/35: tw9910: do not lie about cropping abilities
26/35: sh_mobile_ceu_camera: implement host-side image scaling
27/35: soc-camera: switch to s_crop v4l2-subdev video operation
28/35: soc-camera: switch to using v4l2_subdev_call()
29/35: soc-camera: Use I2C device for dev_{dbg,info,...} output in all clients
30/35: soc-camera: Use camera device object for core output
31/35: soc-camera: Use video device object for output in host drivers
32/35: soc-camera: V4L2 API compliant scaling (S_FMT) and cropping (S_CROP)
33/35: soc-camera: remove .init() and .release() methods from struct soc_camera_ops
34/35: soc-camera: remove .gain and .exposure struct soc_camera_device members
35/35: soc-camera: fix recently introduced overlong lines

aiming at 2.6.32, a few more patches should follow.

On the 22 of August 2009 an unofficial meeting took place with the purpose to discuss a new RFC for a new V4L API. Participants were (alphabetically sorted): Hans de Goede, Laurent Pinchart, Hans Verkuil and myself. The main discussion of this API will take place during the Linux Plumbers Conference, but since not everyone, interested in this development, including the author of these pages, will be able to attend the conference, it has been decided to hold a small unofficial preliminary meeting on this topic. The following implications on the soc-camera migration to v4l2-subdev have been decided upon:

1. We will have to remove bus-parameter auto-negotiation and force every board to specify a complete list of bus parameters, like signal polarities and sensing edge.
   
2. Will consider using the .notify() member of struct v4l2_device in sensor drivers, that change their output pixel clock frequency dynamically, e.g., depending on the frame format, instead of using the current sense object.
   
3. Maybe use a list of video subdevices as a parameter to soc-camera host platform device to avoid allocating otherwise "useless" per-source platform devices.
   
4. Use a new enumeration for various pixel data representations on the video source - video host data bus, similar to the current fourcc codes for the in-memory presentation of pixel formats.

We will have to give a bit more thinking to items 2 and 3 though.

All soc-camera drivers have been updated to comply with the V4L2 API scaling and cropping definitions. The main differences to the original soc-camera implementation are:

1. VIDIOC_S_CROP ioctl now accepts rectangle parameters in sensor input window pixel units
   
2. VIDIOC_S_CROP ioctl now tries to preserve scales, and not the output window
   
3. VIDIOC_S_FMT ioctl now preserves the sensor input window
   

The patch stack is uploaded here and is based on the 2.6.31-rc4 kernel.

A short announcement: my talk about the soc-camera framework has been accepted for the ELC-E 2009 conference in October 2009 in Grenoble, France. So, maybe, see you there.

While adding more hardware support and functionality to soc-camera, it became apparent, that its cropping / scaling concept was wrong. Trying to fix the situation by reading the API specs and trying to understand it revealed, that the documentation is indeed quite cumbersome and that even experienced V4L kernel developers do not understand completely how VIDIOC_S_CROP and VIDIOC_S_FMT ioctls shall configure cropping and scaling in camera drivers. As the situation slowly clarifies all soc-camera drivers and the core have to be modified to comply with an interpretation of the V4L2 API, which will hopefully result from the currently running on the mailing list discussion.

As of previous post, we still had a number of problems with optimised builds. Specifically, Java "long" type variables have been handled wrongly with optimisation beyond "-O." We now have identified and fixed all currently known problems, ranging from missing assembly code clobber flags to unsafe 64-bit variable manipulation. While working on this we have written a couple of Java "long" type tests, which, after adding all missing long bytecode trap handlers, now all work. Jamvm can now be successfully built and used with "-O," "-O2," "-O3," and "-Os" optimisation flags. Interestingly, we have not been able to detect any performance improvements when switching from "-O" to any deeper optimisation. All relevant changes committed upstream.

Added a TABLESWITCH bytecode implementation, fixed a couple more object references, and fixed two bugs, that caused segfaults when built with "-O" gcc optimisation flag. This brings a nice speed-up of 0.6 seconds on "Hello, World!" (around 33%), but the software version benefits from it too, when similarly fixed, and still keeps us just above 10% behind on this test. All fixes committed to the git repository.

Porting soc-camera with all its drivers and platforms to the v4l2-subdev framework confirmed to be a complicated task. Having prepared the patches in a way to make changes as isolated as possible, they now have to be applied to the mainline in a certain order avoiding any intermittent breakage. After the 2.6.30 kernel has been released earlier this week, two first conversion steps shall be pushed upstream during the 2.6.31 merge window: a preparatory forward-compatibility patch to the soc-camera core, and multiple patches to convert all platforms currently in the mainline to the future platform-binding scheme. The actual incomplete conversion patches will then have to wait until 2.6.32. Then we will be completing the conversion some time in the 2.6.33 timeframe. For details see my post at the linux-media mailing list.

I submitted two talk proposals for the ELC Europe 2009, as is easy to guess, one about soc-camera and one about JEM. I've got two reviews so far, both positive and both indicate, that they like soc-camera better, but find both interesting:-) Will see what will be approved.

Following the "01/01: soc-camera: prepare for the platform driver conversion" patch from 07.05 6 further patches have been submitted to ARM and SuperH trees to convert all 5 currently in the mainline soc-camera platforms to the new scheme.

After that the platform-driver and v4l2-subdev conversion patches have been updated, posted to the linux-media mailing list on 15.05 and uploaded to http://download.open-technology.de/soc-camera/20090515/. See this thread for a detailed description.

Following patches have been pushed to v4l-dvb hg:

01/02: soc-camera: Fix section mismatch warning
02/02: soc_camera: depends on I2C

Following patch has been pushed to v4l-dvb hg:

01/01: soc-camera: prepare for the platform driver conversion

Windows 7 uses variants of PTP operations, that previously testes OS's didn't use, therefore implementations of those operations were untested and unsurprisingly contained bugs. As of commit ef7ef29b5cdcf8630c901362a66905b956242e0d these bugs are fixed and ptp-gadget is now also compatible with Windows 7 and works in both "camera" and "Windows explorer" modes.

We have two positive results to report:

1. We managed to further improve performance of our hardware-accelerated JVM using some of "quick" bytecodes, supported by JEM, optimizing memory allocations for code segments, eliminating multiple redundant calls to pthread_getspecific(). This brings us down to 1.83s on our "Hello, World!" example, which is now "just" 17% slower than the pure software version. One further optimization we have not committed yet is using (MAP_PRIVATE | MAP_POPULATE) instead of (MAP_SHARED), which saves us another 40ms on the same test (15% slower than the soft version), but we are not quite certain about implications of using a private mapping yet.
   
2. Our first threaded example works! Having implemented kernel support for JEM context saving and restoring (see earlier posts in this category) we now have fixed a few more object reference on the heap cases and a couple of other buglets, and now the test successfully runs through.
   

As usual, the code is available for download from our git repository and marked with "jem-0.2" tag.