Documentation/media-framework.txt - linux - Git at Google

 Linux kernel media framework
 ============================

 This document describes the Linux kernel media framework, its data structures,
 functions and their usage.


 Introduction
 ------------

 The media controller API is documented in DocBook format in
 Documentation/DocBook/v4l/media-controller.xml. This document will focus on
 the kernel-side implementation of the media framework.


 Abstract media device model
 ---------------------------

 Discovering a device internal topology, and configuring it at runtime, is one
 of the goals of the media framework. To achieve this, hardware devices are
 modeled as an oriented graph of building blocks called entities connected
 through pads.

 An entity is a basic media hardware building block. It can correspond to
 a large variety of logical blocks such as physical hardware devices
 (CMOS sensor for instance), logical hardware devices (a building block
 in a System-on-Chip image processing pipeline), DMA channels or physical
 connectors.

 A pad is a connection endpoint through which an entity can interact with
 other entities. Data (not restricted to video) produced by an entity
 flows from the entity's output to one or more entity inputs. Pads should
 not be confused with physical pins at chip boundaries.

 A link is a point-to-point oriented connection between two pads, either
 on the same entity or on different entities. Data flows from a source
 pad to a sink pad.


 Media device
 ------------

 A media device is represented by a struct media_device instance, defined in
 include/media/media-device.h. Allocation of the structure is handled by the
 media device driver, usually by embedding the media_device instance in a
 larger driver-specific structure.

 Drivers register media device instances by calling

 	media_device_register(struct media_device *mdev);

 The caller is responsible for initializing the media_device structure before
 registration. The following fields must be set:

  - dev must point to the parent device (usually a pci_dev, usb_interface or
    platform_device instance).

  - model must be filled with the device model name as a NUL-terminated UTF-8
    string. The device/model revision must not be stored in this field.

 The following fields are optional:

  - serial is a unique serial number stored as a NUL-terminated ASCII string.
    The field is big enough to store a GUID in text form. If the hardware
    doesn't provide a unique serial number this field must be left empty.

  - bus_info represents the location of the device in the system as a
    NUL-terminated ASCII string. For PCI/PCIe devices bus_info must be set to
    "PCI:" (or "PCIe:") followed by the value of pci_name(). For USB devices,
    the usb_make_path() function must be used. This field is used by
    applications to distinguish between otherwise identical devices that don't
    provide a serial number.

  - hw_revision is the hardware device revision in a driver-specific format.
    When possible the revision should be formatted with the KERNEL_VERSION
    macro.

  - driver_version is formatted with the KERNEL_VERSION macro. The version
    minor must be incremented when new features are added to the userspace API
    without breaking binary compatibility. The version major must be
    incremented when binary compatibility is broken.

 Upon successful registration a character device named media[0-9]+ is created.
 The device major and minor numbers are dynamic. The model name is exported as
 a sysfs attribute.

 Drivers unregister media device instances by calling

 	media_device_unregister(struct media_device *mdev);

 Unregistering a media device that hasn't been registered is *NOT* safe.


 Entities, pads and links
 ------------------------

 - Entities

 Entities are represented by a struct media_entity instance, defined in
 include/media/media-entity.h. The structure is usually embedded into a
 higher-level structure, such as a v4l2_subdev or video_device instance,
 although drivers can allocate entities directly.

 Drivers initialize entities by calling

 	media_entity_init(struct media_entity *entity, u16 num_pads,
 			  struct media_pad *pads, u16 extra_links);

 The media_entity name, type, flags, revision and group_id fields can be
 initialized before or after calling media_entity_init. Entities embedded in
 higher-level standard structures can have some of those fields set by the
 higher-level framework.

 As the number of pads is known in advance, the pads array is not allocated
 dynamically but is managed by the entity driver. Most drivers will embed the
 pads array in a driver-specific structure, avoiding dynamic allocation.

 Drivers must set the direction of every pad in the pads array before calling
 media_entity_init. The function will initialize the other pads fields.

 Unlike the number of pads, the total number of links isn't always known in
 advance by the entity driver. As an initial estimate, media_entity_init
 pre-allocates a number of links equal to the number of pads plus an optional
 number of extra links. The links array will be reallocated if it grows beyond
 the initial estimate.

 Drivers register entities with a media device by calling

 	media_device_register_entity(struct media_device *mdev,
 				     struct media_entity *entity);

 Entities are identified by a unique positive integer ID. Drivers can provide an
 ID by filling the media_entity id field prior to registration, or request the
 media controller framework to assign an ID automatically. Drivers that provide
 IDs manually must ensure that all IDs are unique. IDs are not guaranteed to be
 contiguous even when they are all assigned automatically by the framework.

 Drivers unregister entities by calling

 	media_device_unregister_entity(struct media_entity *entity);

 Unregistering an entity will not change the IDs of the other entities, and the
 ID will never be reused for a newly registered entity.

 When a media device is unregistered, all its entities are unregistered
 automatically. No manual entities unregistration is then required.

 Drivers free resources associated with an entity by calling

 	media_entity_cleanup(struct media_entity *entity);

 This function must be called during the cleanup phase after unregistering the
 entity. Note that the media_entity instance itself must be freed explicitly by
 the driver if required.

 Entities have flags that describe the entity capabilities and state.

 	MEDIA_ENT_FL_DEFAULT indicates the default entity for a given type.
 	This can be used to report the default audio and video devices or the
 	default camera sensor.

 Logical entity groups can be defined by setting the group ID of all member
 entities to the same non-zero value. An entity group serves no purpose in the
 kernel, but is reported to userspace during entities enumeration. The group_id
 field belongs to the media device driver and must not by touched by entity
 drivers.

 Media device drivers should define groups if several entities are logically
 bound together. Example usages include reporting

 	- ALSA, VBI and video nodes that carry the same media stream
 	- lens and flash controllers associated with a sensor

 - Pads

 Pads are represented by a struct media_pad instance, defined in
 include/media/media-entity.h. Each entity stores its pads in a pads array
 managed by the entity driver. Drivers usually embed the array in a
 driver-specific structure.

 Pads are identified by their entity and their 0-based index in the pads array.
 Both information are stored in the media_pad structure, making the media_pad
 pointer the canonical way to store and pass link references.

 Pads have flags that describe the pad capabilities and state.

 	MEDIA_PAD_FL_SINK indicates that the pad supports sinking data.
 	MEDIA_PAD_FL_SOURCE indicates that the pad supports sourcing data.

 One and only one of MEDIA_PAD_FL_SINK and MEDIA_PAD_FL_SOURCE must be set for
 each pad.

 - Links

 Links are represented by a struct media_link instance, defined in
 include/media/media-entity.h. Each entity stores all links originating at or
 targetting any of its pads in a links array. A given link is thus stored
 twice, once in the source entity and once in the target entity. The array is
 pre-allocated and grows dynamically as needed.

 Drivers create links by calling

 	media_entity_create_link(struct media_entity *source, u16 source_pad,
 				 struct media_entity *sink,   u16 sink_pad,
 				 u32 flags);

 An entry in the link array of each entity is allocated and stores pointers
 to source and sink pads.

 Links have flags that describe the link capabilities and state.

 	MEDIA_LNK_FL_ENABLED indicates that the link is enabled and can be used
 	to transfer media data. When two or more links target a sink pad, only
 	one of them can be enabled at a time.
 	MEDIA_LNK_FL_IMMUTABLE indicates that the link enabled state can't be
 	modified at runtime. If MEDIA_LNK_FL_IMMUTABLE is set, then
 	MEDIA_LNK_FL_ENABLED must also be set since an immutable link is always
 	enabled.
	Linux kernel media framework
	============================

	This document describes the Linux kernel media framework, its data structures,
	functions and their usage.


	Introduction
	------------

	The media controller API is documented in DocBook format in
	Documentation/DocBook/v4l/media-controller.xml. This document will focus on
	the kernel-side implementation of the media framework.


	Abstract media device model
	---------------------------

	Discovering a device internal topology, and configuring it at runtime, is one
	of the goals of the media framework. To achieve this, hardware devices are
	modeled as an oriented graph of building blocks called entities connected
	through pads.

	An entity is a basic media hardware building block. It can correspond to
	a large variety of logical blocks such as physical hardware devices
	(CMOS sensor for instance), logical hardware devices (a building block
	in a System-on-Chip image processing pipeline), DMA channels or physical
	connectors.

	A pad is a connection endpoint through which an entity can interact with
	other entities. Data (not restricted to video) produced by an entity
	flows from the entity's output to one or more entity inputs. Pads should
	not be confused with physical pins at chip boundaries.

	A link is a point-to-point oriented connection between two pads, either
	on the same entity or on different entities. Data flows from a source
	pad to a sink pad.


	Media device
	------------

	A media device is represented by a struct media_device instance, defined in
	include/media/media-device.h. Allocation of the structure is handled by the
	media device driver, usually by embedding the media_device instance in a
	larger driver-specific structure.

	Drivers register media device instances by calling

	media_device_register(struct media_device *mdev);

	The caller is responsible for initializing the media_device structure before
	registration. The following fields must be set:

	- dev must point to the parent device (usually a pci_dev, usb_interface or
	platform_device instance).

	- model must be filled with the device model name as a NUL-terminated UTF-8
	string. The device/model revision must not be stored in this field.

	The following fields are optional:

	- serial is a unique serial number stored as a NUL-terminated ASCII string.
	The field is big enough to store a GUID in text form. If the hardware
	doesn't provide a unique serial number this field must be left empty.

	- bus_info represents the location of the device in the system as a
	NUL-terminated ASCII string. For PCI/PCIe devices bus_info must be set to
	"PCI:" (or "PCIe:") followed by the value of pci_name(). For USB devices,
	the usb_make_path() function must be used. This field is used by
	applications to distinguish between otherwise identical devices that don't
	provide a serial number.

	- hw_revision is the hardware device revision in a driver-specific format.
	When possible the revision should be formatted with the KERNEL_VERSION
	macro.

	- driver_version is formatted with the KERNEL_VERSION macro. The version
	minor must be incremented when new features are added to the userspace API
	without breaking binary compatibility. The version major must be
	incremented when binary compatibility is broken.

	Upon successful registration a character device named media[0-9]+ is created.
	The device major and minor numbers are dynamic. The model name is exported as
	a sysfs attribute.

	Drivers unregister media device instances by calling

	media_device_unregister(struct media_device *mdev);

	Unregistering a media device that hasn't been registered is NOT safe.


	Entities, pads and links
	------------------------

	- Entities

	Entities are represented by a struct media_entity instance, defined in
	include/media/media-entity.h. The structure is usually embedded into a
	higher-level structure, such as a v4l2_subdev or video_device instance,
	although drivers can allocate entities directly.

	Drivers initialize entities by calling

	media_entity_init(struct media_entity *entity, u16 num_pads,
	struct media_pad *pads, u16 extra_links);

	The media_entity name, type, flags, revision and group_id fields can be
	initialized before or after calling media_entity_init. Entities embedded in
	higher-level standard structures can have some of those fields set by the
	higher-level framework.

	As the number of pads is known in advance, the pads array is not allocated
	dynamically but is managed by the entity driver. Most drivers will embed the
	pads array in a driver-specific structure, avoiding dynamic allocation.

	Drivers must set the direction of every pad in the pads array before calling
	media_entity_init. The function will initialize the other pads fields.

	Unlike the number of pads, the total number of links isn't always known in
	advance by the entity driver. As an initial estimate, media_entity_init
	pre-allocates a number of links equal to the number of pads plus an optional
	number of extra links. The links array will be reallocated if it grows beyond
	the initial estimate.

	Drivers register entities with a media device by calling

	media_device_register_entity(struct media_device *mdev,
	struct media_entity *entity);

	Entities are identified by a unique positive integer ID. Drivers can provide an
	ID by filling the media_entity id field prior to registration, or request the
	media controller framework to assign an ID automatically. Drivers that provide
	IDs manually must ensure that all IDs are unique. IDs are not guaranteed to be
	contiguous even when they are all assigned automatically by the framework.

	Drivers unregister entities by calling

	media_device_unregister_entity(struct media_entity *entity);

	Unregistering an entity will not change the IDs of the other entities, and the
	ID will never be reused for a newly registered entity.

	When a media device is unregistered, all its entities are unregistered
	automatically. No manual entities unregistration is then required.

	Drivers free resources associated with an entity by calling

	media_entity_cleanup(struct media_entity *entity);

	This function must be called during the cleanup phase after unregistering the
	entity. Note that the media_entity instance itself must be freed explicitly by
	the driver if required.

	Entities have flags that describe the entity capabilities and state.

	MEDIA_ENT_FL_DEFAULT indicates the default entity for a given type.
	This can be used to report the default audio and video devices or the
	default camera sensor.

	Logical entity groups can be defined by setting the group ID of all member
	entities to the same non-zero value. An entity group serves no purpose in the
	kernel, but is reported to userspace during entities enumeration. The group_id
	field belongs to the media device driver and must not by touched by entity
	drivers.

	Media device drivers should define groups if several entities are logically
	bound together. Example usages include reporting

	- ALSA, VBI and video nodes that carry the same media stream
	- lens and flash controllers associated with a sensor

	- Pads

	Pads are represented by a struct media_pad instance, defined in
	include/media/media-entity.h. Each entity stores its pads in a pads array
	managed by the entity driver. Drivers usually embed the array in a
	driver-specific structure.

	Pads are identified by their entity and their 0-based index in the pads array.
	Both information are stored in the media_pad structure, making the media_pad
	pointer the canonical way to store and pass link references.

	Pads have flags that describe the pad capabilities and state.

	MEDIA_PAD_FL_SINK indicates that the pad supports sinking data.
	MEDIA_PAD_FL_SOURCE indicates that the pad supports sourcing data.

	One and only one of MEDIA_PAD_FL_SINK and MEDIA_PAD_FL_SOURCE must be set for
	each pad.

	- Links

	Links are represented by a struct media_link instance, defined in
	include/media/media-entity.h. Each entity stores all links originating at or
	targetting any of its pads in a links array. A given link is thus stored
	twice, once in the source entity and once in the target entity. The array is
	pre-allocated and grows dynamically as needed.

	Drivers create links by calling

	media_entity_create_link(struct media_entity *source, u16 source_pad,
	struct media_entity *sink, u16 sink_pad,
	u32 flags);

	An entry in the link array of each entity is allocated and stores pointers
	to source and sink pads.

	Links have flags that describe the link capabilities and state.

	MEDIA_LNK_FL_ENABLED indicates that the link is enabled and can be used
	to transfer media data. When two or more links target a sink pad, only
	one of them can be enabled at a time.
	MEDIA_LNK_FL_IMMUTABLE indicates that the link enabled state can't be
	modified at runtime. If MEDIA_LNK_FL_IMMUTABLE is set, then
	MEDIA_LNK_FL_ENABLED must also be set since an immutable link is always
	enabled.