Difference between revisions of "V4L2 Userspace Library"

From LinuxTVWiki
Jump to: navigation, search
(New page: The need What do we need in userspace, only for v4l (dvb is not concerned): * particular video decompression algorithm like stk11xx or usbvision (the decompression algorithm is in ke...)
 
m (rename file names (prep'ing for image merge))
 
(7 intermediate revisions by 3 users not shown)
Line 1: Line 1:
The need
+
== The need ==
 +
 
  
 
What do we need in userspace, only for v4l (dvb is not concerned):
 
What do we need in userspace, only for v4l (dvb is not concerned):
 +
* particular video decompression algorithm like stk11xx or usbvision (the decompression algorithm is in kernelspace since it is of linear complexity but shall be moved to userspace)
 +
* colorspace translations
 +
* filters that be done in hardware if the selected hardware can, otherwise software plugin
 +
 +
 +
== The general constraints ==
 +
  
    * particular video decompression algorithm like stk11xx or usbvision (the decompression algorithm is in kernelspace since it is of linear complexity but shall be moved to userspace)
+
* Application shall still have access to the basic kernel driver. Thus, applications will have access to the 'raw' input data without software post-processing. Application developers love to develop optimized codecs :)
    * colorspace translations
+
* The original v4l driver infrastructure should not change. We do not want to break any drivers. However, the extended v4l driver will use common functionalities like video_ioctl2 if this contributes to simplify the architecture.
    * filters that be done in hardware if the selected hardware can, otherwise software plugin
+
* As a consequence, the v4l API does not change for applications.
 +
* v4l driver and extended v4l driver shall mutually exclusive since they share the same hardware
 +
* Limit as much as possible kernel memory and CPU consumption.
 +
* Elegant, flexible.This is why I specify before coding :)
  
The general constraints
+
== The ideal solution ==
 +
The ideal solution would be to create a v4l2 library provided to applications.
 +
They would access the v4l2 drivers via this library that would manage decompression if needed.
 +
If this library existed from the v4l project start, applications would have linked to it forcibly.
 +
The v4l project has a long history and it is more and more difficult to break compatibility now and force applications to link to a new library.
  
    * Application shall still have access to the basic kernel driver. Thus, applications will have access to the 'raw' input data without software post-processing. Application developers love to develop optimized codecs :)
+
== A solution ==
    * The original v4l driver infrastructure should not change. We do not want to break any drivers. However, the extended v4l driver will use common functionalities like video_ioctl2 if this contributes to simplify the architecture.
+
    * As a consequence, the v4l API does not change for applications.
+
    * v4l driver and extended v4l driver shall mutually exclusive since they share the same hardware
+
    * Limit as much as possible kernel memory and CPU consumption.
+
    * Elegant, flexible.This is why I specify before coding :)
+
  
A solution
 
  
 
I thought it would be funny to create userspace drivers like FUSE based projects do.
 
I thought it would be funny to create userspace drivers like FUSE based projects do.
Line 25: Line 34:
 
For v4l drivers for hardware that can fullfill all application requested formats, this changes... nothing too.
 
For v4l drivers for hardware that can fullfill all application requested formats, this changes... nothing too.
 
This architecture is interesting for v4l drivers that interface hardware that provide proprietary compressed video pictures or that are not able to fullfill the application needs by hardware.
 
This architecture is interesting for v4l drivers that interface hardware that provide proprietary compressed video pictures or that are not able to fullfill the application needs by hardware.
 +
 
A general overview of the thing:
 
A general overview of the thing:
 +
 +
[[Image:V4L2_Userspace_Library_General.png]]
  
 
As a consequence:
 
As a consequence:
 +
 
The only change for the application is only a new device to access (/dev/video1) so there is no extension of the v4l API/ABI.
 
The only change for the application is only a new device to access (/dev/video1) so there is no extension of the v4l API/ABI.
 
The only change for the user is to launch another daemon in the starting scripts (/etc/rc.d/...). The v4l driver does not change, only extension points will be implemented if necessary.
 
The only change for the user is to launch another daemon in the starting scripts (/etc/rc.d/...). The v4l driver does not change, only extension points will be implemented if necessary.
 +
 
The extended v4l driver is an instance of a v4l driver with some entry points for the helper daemon
 
The extended v4l driver is an instance of a v4l driver with some entry points for the helper daemon
Specification
+
 
 +
 
 +
== Specification ==
 +
 
 
First, some obvious assumptions:
 
First, some obvious assumptions:
  
    * The API available to the applications will be named v4l public API (headers in path /usr/include/linux/)
+
* The API available to the applications will be named v4l public API (headers in path /usr/include/linux/)
    * All extensions will be made by internal APIs. I will name it v4l extension private API (headers in path /usr/src/linux/drivers/media/video/...)
+
* All extensions will be made by internal APIs. I will name it v4l extension private API (headers in path /usr/src/linux/drivers/media/video/...)
    * I will name uncompressed frames formats the ones defined in the v4l public API.
+
* I will name uncompressed frames formats the ones defined in the v4l public API.
    * I will name compressed frames formats the ones not defined in the v4l public API.
+
* I will name compressed frames formats the ones not defined in the v4l public API.
  
  
General use case view of the extended v4l driver
+
General use case view of the extended v4l driver:
 +
 
 +
[[Image:V4L2_Userspace_Library_Diagramme1.png]]
 +
 
 +
=== Open/close access the hardware ===
 +
 
 +
'''Summary'''
  
Open/close access the hardware
 
Summary
 
 
This step corresponds to the first and the last access to hardware via the extended driver.
 
This step corresponds to the first and the last access to hardware via the extended driver.
 
Context
 
Context
 
The application opens the extended v4l driver. The scenario will assume that /dev/video0 is the base device entry point and /dev/video1 is the extended entry point.
 
The application opens the extended v4l driver. The scenario will assume that /dev/video0 is the base device entry point and /dev/video1 is the extended entry point.
Pre-conditions
 
Description
 
  
  1. The application opens /dev/video1.
+
'''Description'''
  2. The extended device driver asks the base driver to lock the device.
+
 
  3. The application controls/gets incoming frames from the hardware (see further).
+
# The application opens /dev/video1.
  4. The application closes /dev/video1.
+
# The extended device driver asks the base driver to lock the device.
  5. The extended device driver wipes out all internal allocations and asks the v4l helper to do so...
+
# The application controls/gets incoming frames from the hardware (see further).
  6. The extended device driver asks the base driver to unlock the device.
+
# The application closes /dev/video1.
 +
# The extended device driver wipes out all internal allocations and asks the v4l helper to do so...
 +
# The extended device driver asks the base driver to unlock the device.
 +
 
 +
[[Image:V4L2_Userspace_Library_OpenCloseAccessTheHardware.png]]
 +
 
 +
'''Exceptions'''
 +
 
 +
* At open, the base v4l driver is already locked. The extended v4l driver returns -EBUSY.
 +
 
  
Post-conditions
+
=== Control the hardware ===
Exceptions
+
  
    * At open, the base v4l driver is already locked. The extended v4l driver returns -EBUSY.
+
'''Summary'''
  
Control the hardware
 
Summary
 
 
This step corresponds mainly to the ioctl() entry points' behaviour.
 
This step corresponds mainly to the ioctl() entry points' behaviour.
 
The extended v4l driver will have to interact with the v4l base driver and with the v4l helper daemon with specific ioctls.
 
The extended v4l driver will have to interact with the v4l base driver and with the v4l helper daemon with specific ioctls.
 
The scenario will show the typical controls the application will address to the extended v4l driver.
 
The scenario will show the typical controls the application will address to the extended v4l driver.
 
In most cases, the extended v4l driver will ask the base v4l driver to do the job. The v4l helper daemon will be used on specific control steps.
 
In most cases, the extended v4l driver will ask the base v4l driver to do the job. The v4l helper daemon will be used on specific control steps.
Context
+
 
 +
'''Context'''
 +
 
 
The application needs to get frames from the v4l peripheral, this task is fulfilled by the extended v4l driver with the help of the base v4l driver.
 
The application needs to get frames from the v4l peripheral, this task is fulfilled by the extended v4l driver with the help of the base v4l driver.
Pre-conditions
+
'''Pre-conditions'''
 +
 
 
The extended driver is opened properly on /dev/video1.
 
The extended driver is opened properly on /dev/video1.
Description
 
  
  1. The application gets the device hardware capabilities with VIDIOC_QUERYCAP.
+
'''Description'''
  2. The application enumerates inputs with VIDIOC_ENUMINPUT.
+
  3. The application sets desired input with VIDIOC_S_INPUT.
+
  4. The application enumerates supported video standards with VIDIOC_ENUMSTD.
+
  
      For all these previous steps, the extended video driver transmits the request to the base driver without interaction.
+
# The application gets the device hardware capabilities with VIDIOC_QUERYCAP.
 +
# The application enumerates inputs with VIDIOC_ENUMINPUT.
 +
# The application sets desired input with VIDIOC_S_INPUT.
 +
# The application enumerates supported video standards with VIDIOC_ENUMSTD.
  
  5. The application negociates with the extended v4l driver the video data format, with VIDIOC_ENUM_FMT, VIDIOC_S_FMT.
+
For all these previous steps, the extended video driver transmits the request to the base driver without interaction.
      If the v4l helper is here, see the extension point named "Provide compatible video formats".
+
 
      Otherwise, see alternative A1
+
# The application negociates with the extended v4l driver the video data format, with VIDIOC_ENUM_FMT, VIDIOC_S_FMT.
 +
# If the v4l helper is here, see the extension point named "Provide compatible video formats". Otherwise, see alternative A1
 +
 
 +
[[Image:V4L2_Userspace_Library_ControlTheHardware.png]]
 +
 
 +
'''Alternatives'''
  
Post-conditions
 
Alternatives
 
 
A1 - the v4l helper daemon is not running (the /dev/v4lctl driver is not opened by the helper daemon)
 
A1 - the v4l helper daemon is not running (the /dev/v4lctl driver is not opened by the helper daemon)
 
the extended v4l driver transmits the request to the base driver without interaction...
 
the extended v4l driver transmits the request to the base driver without interaction...
Exceptions
+
 
Get video frames
+
 
Summary
+
=== Get video frames ===
 +
 
 +
'''Summary'''
 +
 
 
As the driver is correctly configured, the hardware captures and streams data from the device.
 
As the driver is correctly configured, the hardware captures and streams data from the device.
 
This step will show how to managed the incoming data and provide them to the Application.
 
This step will show how to managed the incoming data and provide them to the Application.
 
This is the most important part to understand from this specification.
 
This is the most important part to understand from this specification.
Context
+
 
Pre-conditions
+
'''Pre-conditions'''
 +
 
 
The extended v4l driver is properly configured on /dev/video1.
 
The extended v4l driver is properly configured on /dev/video1.
Description
 
  
  1. The application requests the driver to allocate frames with VIDIOC_REQBUFS.
+
'''Description'''
  2. The extended v4l driver requests the v4l helper daemon to allocate video buffers
+
 
  3. The application requests each allocated buffer characteristics with VIDIOC_QUERYBUF.
+
# The application requests the driver to allocate frames with VIDIOC_REQBUFS.
  4. The application requests the driver to enqueue each frame with VIDIOC_QBUF.
+
# The extended v4l driver requests the v4l helper daemon to allocate video buffers
  5. The application starts the acquisition chain with VIDIOC_STREAMON.
+
# The application requests each allocated buffer characteristics with VIDIOC_QUERYBUF.
  6. The application waits and dequeues a frame with VIDIOC_DQBUF.
+
# The application requests the driver to enqueue each frame with VIDIOC_QBUF.
  7. The extended v4l driver waits and dequeues a frame from the base v4l driver. If the frame needs to be uncompressed, see the extension point named "Uncompress frame".
+
# The application starts the acquisition chain with VIDIOC_STREAMON.
      Otherwise the extended v4l driver transmits the request to the base driver without interaction.
+
# The application waits and dequeues a frame with VIDIOC_DQBUF.
  8. As soon as the frame is processed, the application calls VIDIOC_QBUF to re-enqueue this buffer.
+
# The extended v4l driver waits and dequeues a frame from the base v4l driver. If the frame needs to be uncompressed, see the extension point named "Uncompress frame".
  9. The VIDIOC_DQBUF/QBUF mechanism is repeated to get each frame sequentially.
+
Otherwise the extended v4l driver transmits the request to the base driver without interaction.
  10. The application stops the acquisition with VIDIOC_STREAMOFF.
+
# As soon as the frame is processed, the application calls VIDIOC_QBUF to re-enqueue this buffer.
 +
# The VIDIOC_DQBUF/QBUF mechanism is repeated to get each frame sequentially.
 +
# The application stops the acquisition with VIDIOC_STREAMOFF.
 +
 
 +
[[Image:V4L2_Userspace_Library_GetVideoFrames.png]]
 +
 
 +
'''Exceptions'''
  
Post-conditions
 
Exceptions
 
 
EX1 The v4l helper daemon is not active or does not respond correctly to the extended v4l device driver request
 
EX1 The v4l helper daemon is not active or does not respond correctly to the extended v4l device driver request
 
The extended v4l device driver returns -EFAULT (internal error) to the application, this ends the process.
 
The extended v4l device driver returns -EFAULT (internal error) to the application, this ends the process.
Provide compatible video data formats
+
 
Summary
+
=== Provide compatible video data formats ===
 +
 
 +
'''Summary'''
 +
 
 
This step specifies how the extended v4l driver is able to provide to the application the hardware and software compatible video data formats
 
This step specifies how the extended v4l driver is able to provide to the application the hardware and software compatible video data formats
Context
+
 
Pre-conditions
+
'''Pre-conditions'''
 +
 
 
The v4l helper daemon has been registered
 
The v4l helper daemon has been registered
Description
 
  
  1. Either the application has requested to enumerate all supported video formats,
+
'''Description'''
        1. The extended v4l driver ask the v4l helper daemon all possible formats for this hardware.
+
 
        2. The v4l helper daemon provides all additional software video format.
+
# Either the application has requested to enumerate all supported video formats,
  2. Or the applications has requested to set a specific video format,
+
## The extended v4l driver ask the v4l helper daemon all possible formats for this hardware.
        1. The extended v4l driver asks the v4l helper daemon if the requested video format is possible for this hardware
+
## The v4l helper daemon provides all additional software video format.
        2. If the helper daemon is able to process by software the video format, it acknowledges with the negociated video format. Otherwise it returns a NACK
+
# Or the applications has requested to set a specific video format,
 +
## The extended v4l driver asks the v4l helper daemon if the requested video format is possible for this hardware
 +
## If the helper daemon is able to process by software the video format, it acknowledges with the negociated video format. Otherwise it returns a NACK
 +
 
 +
 
 +
=== Uncompress frames ===
 +
 
 +
'''Summary'''
  
Post-conditions
 
Exceptions
 
Uncompress frames
 
Summary
 
 
This part concerns the interactions with the userland v4l helper in order to uncompress frames.
 
This part concerns the interactions with the userland v4l helper in order to uncompress frames.
Context
+
 
 +
'''Context'''
 +
 
 
In order to precise the context of a frame decompression, here is a collaboration overview of all components involved.
 
In order to precise the context of a frame decompression, here is a collaboration overview of all components involved.
Pre-conditions
+
 
 +
[[Image:V4L2_Userspace_Library_UncompressFrames.png]]
 +
 
 +
'''Pre-conditions'''
 +
 
 
The extended v4l driver is properly configured on /dev/video1.
 
The extended v4l driver is properly configured on /dev/video1.
 
The v4l helper is present and available (has answered correctly to all steps before)
 
The v4l helper is present and available (has answered correctly to all steps before)
 
The extended v4l driver received a new compressed frame from the base v4l driver
 
The extended v4l driver received a new compressed frame from the base v4l driver
Description
 
  
  1. The extended v4l driver prepares data for the v4l helper daemon. This implies some generic preprocessing such as eventual memcopies.
+
'''Description'''
  2. The extended v4l driver asks the v4l helper daemon to uncompress the newly available frame
+
 
  3. The helper daemon answers as soon as the processing is done
+
# The extended v4l driver prepares data for the v4l helper daemon. This implies some generic preprocessing such as eventual memcopies.
 +
# The extended v4l driver asks the v4l helper daemon to uncompress the newly available frame
 +
# The helper daemon answers as soon as the processing is done
  
Post-conditions
+
[[Category:Development]]
Exceptions
+

Latest revision as of 18:37, 7 June 2009

The need

What do we need in userspace, only for v4l (dvb is not concerned):

  • particular video decompression algorithm like stk11xx or usbvision (the decompression algorithm is in kernelspace since it is of linear complexity but shall be moved to userspace)
  • colorspace translations
  • filters that be done in hardware if the selected hardware can, otherwise software plugin


The general constraints

  • Application shall still have access to the basic kernel driver. Thus, applications will have access to the 'raw' input data without software post-processing. Application developers love to develop optimized codecs :)
  • The original v4l driver infrastructure should not change. We do not want to break any drivers. However, the extended v4l driver will use common functionalities like video_ioctl2 if this contributes to simplify the architecture.
  • As a consequence, the v4l API does not change for applications.
  • v4l driver and extended v4l driver shall mutually exclusive since they share the same hardware
  • Limit as much as possible kernel memory and CPU consumption.
  • Elegant, flexible.This is why I specify before coding :)

The ideal solution

The ideal solution would be to create a v4l2 library provided to applications. They would access the v4l2 drivers via this library that would manage decompression if needed. If this library existed from the v4l project start, applications would have linked to it forcibly. The v4l project has a long history and it is more and more difficult to break compatibility now and force applications to link to a new library.

A solution

I thought it would be funny to create userspace drivers like FUSE based projects do. Applications would see nothing but a new device driver to access, all decompression algorithm would be in userspace. The path from the application to the v4l2 kernel driver: [Application]<->[/dev/video1 (created by FUSE-like driver)]<->[/dev/video0 (created by kernel v4l2 driver)] For applications, this changes nothing... For v4l drivers for hardware that can fullfill all application requested formats, this changes... nothing too. This architecture is interesting for v4l drivers that interface hardware that provide proprietary compressed video pictures or that are not able to fullfill the application needs by hardware.

A general overview of the thing:

V4L2 Userspace Library General.png

As a consequence:

The only change for the application is only a new device to access (/dev/video1) so there is no extension of the v4l API/ABI. The only change for the user is to launch another daemon in the starting scripts (/etc/rc.d/...). The v4l driver does not change, only extension points will be implemented if necessary.

The extended v4l driver is an instance of a v4l driver with some entry points for the helper daemon


Specification

First, some obvious assumptions:

  • The API available to the applications will be named v4l public API (headers in path /usr/include/linux/)
  • All extensions will be made by internal APIs. I will name it v4l extension private API (headers in path /usr/src/linux/drivers/media/video/...)
  • I will name uncompressed frames formats the ones defined in the v4l public API.
  • I will name compressed frames formats the ones not defined in the v4l public API.


General use case view of the extended v4l driver:

V4L2 Userspace Library Diagramme1.png

Open/close access the hardware

Summary

This step corresponds to the first and the last access to hardware via the extended driver. Context The application opens the extended v4l driver. The scenario will assume that /dev/video0 is the base device entry point and /dev/video1 is the extended entry point.

Description

  1. The application opens /dev/video1.
  2. The extended device driver asks the base driver to lock the device.
  3. The application controls/gets incoming frames from the hardware (see further).
  4. The application closes /dev/video1.
  5. The extended device driver wipes out all internal allocations and asks the v4l helper to do so...
  6. The extended device driver asks the base driver to unlock the device.

V4L2 Userspace Library OpenCloseAccessTheHardware.png

Exceptions

  • At open, the base v4l driver is already locked. The extended v4l driver returns -EBUSY.


Control the hardware

Summary

This step corresponds mainly to the ioctl() entry points' behaviour. The extended v4l driver will have to interact with the v4l base driver and with the v4l helper daemon with specific ioctls. The scenario will show the typical controls the application will address to the extended v4l driver. In most cases, the extended v4l driver will ask the base v4l driver to do the job. The v4l helper daemon will be used on specific control steps.

Context

The application needs to get frames from the v4l peripheral, this task is fulfilled by the extended v4l driver with the help of the base v4l driver. Pre-conditions

The extended driver is opened properly on /dev/video1.

Description

  1. The application gets the device hardware capabilities with VIDIOC_QUERYCAP.
  2. The application enumerates inputs with VIDIOC_ENUMINPUT.
  3. The application sets desired input with VIDIOC_S_INPUT.
  4. The application enumerates supported video standards with VIDIOC_ENUMSTD.

For all these previous steps, the extended video driver transmits the request to the base driver without interaction.

  1. The application negociates with the extended v4l driver the video data format, with VIDIOC_ENUM_FMT, VIDIOC_S_FMT.
  2. If the v4l helper is here, see the extension point named "Provide compatible video formats". Otherwise, see alternative A1

V4L2 Userspace Library ControlTheHardware.png

Alternatives

A1 - the v4l helper daemon is not running (the /dev/v4lctl driver is not opened by the helper daemon) the extended v4l driver transmits the request to the base driver without interaction...


Get video frames

Summary

As the driver is correctly configured, the hardware captures and streams data from the device. This step will show how to managed the incoming data and provide them to the Application. This is the most important part to understand from this specification.

Pre-conditions

The extended v4l driver is properly configured on /dev/video1.

Description

  1. The application requests the driver to allocate frames with VIDIOC_REQBUFS.
  2. The extended v4l driver requests the v4l helper daemon to allocate video buffers
  3. The application requests each allocated buffer characteristics with VIDIOC_QUERYBUF.
  4. The application requests the driver to enqueue each frame with VIDIOC_QBUF.
  5. The application starts the acquisition chain with VIDIOC_STREAMON.
  6. The application waits and dequeues a frame with VIDIOC_DQBUF.
  7. The extended v4l driver waits and dequeues a frame from the base v4l driver. If the frame needs to be uncompressed, see the extension point named "Uncompress frame".

Otherwise the extended v4l driver transmits the request to the base driver without interaction.

  1. As soon as the frame is processed, the application calls VIDIOC_QBUF to re-enqueue this buffer.
  2. The VIDIOC_DQBUF/QBUF mechanism is repeated to get each frame sequentially.
  3. The application stops the acquisition with VIDIOC_STREAMOFF.

V4L2 Userspace Library GetVideoFrames.png

Exceptions

EX1 The v4l helper daemon is not active or does not respond correctly to the extended v4l device driver request The extended v4l device driver returns -EFAULT (internal error) to the application, this ends the process.

Provide compatible video data formats

Summary

This step specifies how the extended v4l driver is able to provide to the application the hardware and software compatible video data formats

Pre-conditions

The v4l helper daemon has been registered

Description

  1. Either the application has requested to enumerate all supported video formats,
    1. The extended v4l driver ask the v4l helper daemon all possible formats for this hardware.
    2. The v4l helper daemon provides all additional software video format.
  2. Or the applications has requested to set a specific video format,
    1. The extended v4l driver asks the v4l helper daemon if the requested video format is possible for this hardware
    2. If the helper daemon is able to process by software the video format, it acknowledges with the negociated video format. Otherwise it returns a NACK


Uncompress frames

Summary

This part concerns the interactions with the userland v4l helper in order to uncompress frames.

Context

In order to precise the context of a frame decompression, here is a collaboration overview of all components involved.

V4L2 Userspace Library UncompressFrames.png

Pre-conditions

The extended v4l driver is properly configured on /dev/video1. The v4l helper is present and available (has answered correctly to all steps before) The extended v4l driver received a new compressed frame from the base v4l driver

Description

  1. The extended v4l driver prepares data for the v4l helper daemon. This implies some generic preprocessing such as eventual memcopies.
  2. The extended v4l driver asks the v4l helper daemon to uncompress the newly available frame
  3. The helper daemon answers as soon as the processing is done