|
|
|||||||||||
|
|||||||||||||
 |
||||
News2015-11-12 Report of the Linux Kernel Summit Media Workshop in Seoul, South Corea, Oct, 26 2015This is the report for the Linux Kernel Summit Media Workshop in Seoul, South Corea, Oct, 26 2015. Linux Kernel Summit Media Workshop – Seoul 2015 Kernel Summit Media Workshop Group Photo Attendees list:
1. Codec APIStream APIThe original V4L2 codec API was developed along with the Exynos codec driver. As the device implements high-level operations in hardware the resulting API was high-level as well with drivers accepting unprocessed raw streams. This matches older ARM SoCs where CPU power wasn’t deemed enough to implement stream parsing. Drivers implement two V4L2 buffer queues, one on the uncompressed side and one on the compressed side. The two queues operate independently, without a 1:1 correspondence between consumed and produced buffers (for instance reference frames need to be accumulated when starting video encoding before producing any output or bitstream header parsed before CAPTURE buffers can be allocated). The mem2mem V4L2 kernel framework thus can’t be used to implement such codec drivers as it hardcodes this 1:1 correspondence. This is a kernel framework issue, not a V4L2 userspace API issue. (For stream API fundamentals see Pawel’s slides) Frame API (Slice API)CPUs are getting faster in the ARM world. The trend is to implement lower-level hardware codecs that require stream parsing on the CPU. CPU code needs to slice the stream, extract information, process them and pass the result to a shader-like device. This is the model used on Intel platforms and implemented in the VA API library. Drivers still implement two V4L2 buffer queues, but the encoded stream is split into frames of slices, and a large number of codec-specific controls need to be set from parsed stream information. Stream parsing and parameters calculation is better done in userspace. Userspace is responsible for managing reference frames and their life time, and for passing data to the codec in such a way that an input buffer will always produce an output buffer. The two queues operate together with a 1:1 correspondence between buffers. The mem2mem framework is thus usable. Sources buffer contain only slice data (macroblocks + coefficient data). Controls contain information extracted from stream parsing, list of reference frames and DPB (Decoded Picture Buffer). The request API can be used to associate controls with source buffers. Keeping references to reference frames is one of the remaining problems (especially with DMABUF, and possibly with MMAP in the future when we’ll have the ability to destroy MMAP buffers while streaming). This problem also exists for the stream API. More discussion is needed to design a solution. Pawel promised to upstream the ChromeOS codec drivers code this year (and you should be also nagging Tomasz to do it…). For encoders header generation should probably be done in userspace as well as the code is complex and doesn’t require a kernel implementation. Userspace code should be implemented as libv4l2 plugins to interface between the frame API exposed by the kernel and a stream API exposed to applications. StatusSee Pawel’s slides. Discussion pointsFurther discussions should happen during the Workshop week (scheduled to happen on Tuesday at 11:30). References:Intel libVA SDK:
https://bugs.freedesktop.org/show_bug.cgi?id=92533 2. kABIThe presentation slides are at the Samsung OSG slideshare. The API status is that the DVB documentation API for DVR/demux and CA are outdated and needs some work. The level of documentation at V4L2 is superior than the ones provided for the other media APIs. The kABI status is that just the DVB Demux and CA documentation is OK. The remaining kABI core functions/structs/enum/… are incomplete. Also, non-core V4L2 headers are mixed with the core ones, making harder to identify the gaps. It was agreed to move some documentation from Documentation/{video4linux,dvb} to the kAPI DocBook. It was pointed kABI needs more review by developers and sub-maintainers. It was decided that all new kernel kABI need to be documented, but it was agreed that adding e.g. new fields to undocumented structures doesn’t require the submitter to document the full struct (although it obviously would be very much appreciated). Laurent suggested to use kernel doc for ‘top-level’ documentation, as found at drivers/gpu/drm/drm_atomic_helper.c (look for “DOC: overview”) and 3. uAPI improvements for DVB streamingvideobuf2Junghak explained the status of the current videobuf2 DVB patches:
SoC pipelinesSamsung SoCs have full hardware pipelines from the tuner output to the screen that, once set, won’t require any intervention at runtime. This requires connecting a DVB device to a V4L2 and/or DRM device. Hardware pipelines include hardware-configured scalers for seamless resolution change, in order to achieve real-time operation when the input resolution changes, as there are some requirements on changing the resolution without causing any visible image glimpses. It is currently hard to make this work with the current DVB framework, without the Media Controller. 4. v4l2_buffer & Y2038PS.: Slides for this topic and others below are at v4l2-workshop-2015.odp. Should we create a new structure and fix several other issues in one go? This could be an opportunity to start supporting sub-streams. As Arnd pointed, even if we do it, a backward compatible code will be needed in order to handle the changes on the size of the structure between 32 and 64 bits calls. It sounds reasonable, though that it is time to create a v4l2_buffer_v2 now that we’ll need to increase the size of the struct anyway. There might be a cache-related problem due to the way DMABUF is handled. DMABUF buffers are only unmapped when applications queue a different DMABUF fd on the same V4L2 buffer index. By that time the driver doesn’t own access to the buffer but still tries to clean the cache. The root cause needs to be investigated. A related issue is how V4L2 keeps DMABUF mappings around for performance improvement. This prevents buffer eviction by the GPU. This should be discussed with DRM developers and dma-buf maintainers. Should we add support for sub-streams in a buffer ? Possibly for 3D capture, for other cases there’s not so much interest. Use cases would be needed. Formats are specified for buffers and makes it difficult to transport meta-data in a plane along with image planes. Redesign of v4l2_buffer and v4l2_plane should take such use case into account. 5. poll() for output streamsThe current poll() behaviour is not usable for codecs. As mem2mem is considered as the main use case for output devices we need to optimize for that, and implement write emulation specific code in poll() as a special case. We’ll revert to the old implementation and update the documentation accordingly. 6. poll for capture streamsq->waiting_for_buffers is v4l2 specific, but after the vb2 split it is now part of the vb2 core: should this be moved to the v4l2 part or be kept in the vb2 core? 7. Hardware Color Space Conversion (CSC) status updateFor video output this already works: v4l2_pix_format(_mplane) contains all the information of the memory format including colorspace information and the driver can enable CSC hardware as needed. However, for video capture the colorspace information (colorspace, ycbcr_enc, quantization and xfer_func fields) is documented as being returned by the driver and the application cannot set it to a desired colorspace. 8. HDMI-CEC status updateLast patch series is at v9: http://www.spinics.net/lists/linux-media/msg93362.html. The main to do is to rewrite the documentation for the CEC framework internals and to address some cleanup requests from Russell King. 9. VIDIOC_CREATE_BUFS and v4l2_format validationHow to validate the ‘struct v4l2_format format;’ field? It is documented as “Filled in by the application, preserved by the driver” and “Unsupported formats will result in an error”. However, most drivers use just the sizeimage field, some do partial validation (usually the pixelformat and resolution). Validation is done in the vb2 queue_setup() callback: leads to a void * argument in vb2 core. The original idea was that format contains a format different from the current and that the created buffers would be valid for the requested format. In practice it’s the different buffer size that is of interest. That means that there is an inconsistent behavior. The question is how to fix? It was proposed to handle it the same way as REQUEST_BUFS: expect that userspace calls TRY_FMT or similar to fill the format field and only use sizeimage field in drivers. 10. Media Controller Next Generation status updateIt was briefly discussed the MC next gen status, focusing on the discussions on how this will work with ALSA. It was pointed by Mark and Vinod that the current ALSA patches are barely touching the needs; an aSoC pipeline would be a way more complex than what was done so far on the ALSA patchset. Vinod and Mark are looking for a way to be able to set a topology using ‘cat file >/dev/mediaX’. This is currently provided via request_firmware logic on aSoC. We need to find a way to keep a core MC support that would be consistent with the current topology setup on aSoC. Both ALSA and media developers need to be involved on such development. Atomic configuration updates across subsystems (MC for links, V4L2/ALSA for other parameters, …) needs to be taken into account. For audio devices routing changes ordering matters unless the hardware can apply routing changes atomically. 11. VIDIOC_SUBDEV_QUERYCAPQUERYCAP is desired for v4l-subdev devices, because there are currently no ioctls that are always available for v4l-subdev devices, and having such an ioctl will make v4l2-compliance easier to write. It would also be nice to find the media controller from a device node. It was proposed two solutions:
It was suggested to check if implementing MEDIA_IOC_INTERFACE_INFO is easy to be added on the various subsystems (dvb, alsa, drm, iio). 12. DELETE_BUFFERSSuggested by Laurent to create a new ioctl to be able to dynamically delete some buffers created via CREATE_BUFFERS. A related issue is to allow for more than 32 buffers (already requested in the past). One of the issues is if the buffer numbers will have “holes” or if the Kernel will dynamically re-allocate bufs array in vb2_queue. The problem with having holes is the need for an efficient index-to-vb2_buffer lookup. It was agreed that we’d like to have it. Ditto for more than 32 buffers (don’t limit this). 13. Workshop formatIt is currently very technical, hard for non-core developers to follow. What would be the alternatives? One alternative would be to split into two parts:
How do other subsystems do it? It was proposed to do that on the media workshops that there are at least two days afailable. If we have two days available, then post a CFP to various mailing lists (including apps like gstreamer). It was suggested that we should do one of the media workshops next year co-located with alsa workshop, in order to address the Media Controller related questions. |
||||
Home