Documentation/driver-api/media/camera-sensor.rst - linux - Git at Google


            
            .. SPDX-License-Identifier: GPL-2.0


            Writing camera sensor drivers

            
            =============================


            CSI-2 and parallel (BT.601 and BT.656) busses

            
            ---------------------------------------------


            Please see :ref:`transmitter-receiver`.


            Handling clocks

            
            ---------------


            Camera sensors have an internal clock tree including a PLL and a number of

            
            divisors. The clock tree is generally configured by the driver based on a few

            
            input parameters that are specific to the hardware:: the external clock frequency

            
            and the link frequency. The two parameters generally are obtained from system

            
            firmware. **No other frequencies should be used in any circumstances.**


            The reason why the clock frequencies are so important is that the clock signals

            
            come out of the SoC, and in many cases a specific frequency is designed to be

            
            used in the system. Using another frequency may cause harmful effects

            
            elsewhere. Therefore only the pre-determined frequencies are configurable by the

            
            user.


            ACPI

            
            ~~~~


            Read the ``clock-frequency`` _DSD property to denote the frequency. The driver

            
            can rely on this frequency being used.


            Devicetree

            
            ~~~~~~~~~~


            The currently preferred way to achieve this is using ``assigned-clocks``,

            
            ``assigned-clock-parents`` and ``assigned-clock-rates`` properties. See

            
            ``Documentation/devicetree/bindings/clock/clock-bindings.txt`` for more

            
            information. The driver then gets the frequency using ``clk_get_rate()``.


            This approach has the drawback that there's no guarantee that the frequency

            
            hasn't been modified directly or indirectly by another driver, or supported by

            
            the board's clock tree to begin with. Changes to the Common Clock Framework API

            
            are required to ensure reliability.


            Frame size

            
            ----------


            There are two distinct ways to configure the frame size produced by camera

            
            sensors.


            Freely configurable camera sensor drivers

            
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


            Freely configurable camera sensor drivers expose the device's internal

            
            processing pipeline as one or more sub-devices with different cropping and

            
            scaling configurations. The output size of the device is the result of a series

            
            of cropping and scaling operations from the device's pixel array's size.


            An example of such a driver is the CCS driver (see ``drivers/media/i2c/ccs``).


            Register list based drivers

            
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~


            Register list based drivers generally, instead of able to configure the device

            
            they control based on user requests, are limited to a number of preset

            
            configurations that combine a number of different parameters that on hardware

            
            level are independent. How a driver picks such configuration is based on the

            
            format set on a source pad at the end of the device's internal pipeline.


            Most sensor drivers are implemented this way, see e.g.

            
            ``drivers/media/i2c/imx319.c`` for an example.


            Frame interval configuration

            
            ----------------------------


            There are two different methods for obtaining possibilities for different frame

            
            intervals as well as configuring the frame interval. Which one to implement

            
            depends on the type of the device.


            Raw camera sensors

            
            ~~~~~~~~~~~~~~~~~~


            Instead of a high level parameter such as frame interval, the frame interval is

            
            a result of the configuration of a number of camera sensor implementation

            
            specific parameters. Luckily, these parameters tend to be the same for more or

            
            less all modern raw camera sensors.


            The frame interval is calculated using the following equation::


            	frame interval = (analogue crop width + horizontal blanking) *

            
            			 (analogue crop height + vertical blanking) / pixel rate


            The formula is bus independent and is applicable for raw timing parameters on

            
            large variety of devices beyond camera sensors. Devices that have no analogue

            
            crop, use the full source image size, i.e. pixel array size.


            Horizontal and vertical blanking are specified by ``V4L2_CID_HBLANK`` and

            
            ``V4L2_CID_VBLANK``, respectively. The unit of the ``V4L2_CID_HBLANK`` control

            
            is pixels and the unit of the ``V4L2_CID_VBLANK`` is lines. The pixel rate in

            
            the sensor's **pixel array** is specified by ``V4L2_CID_PIXEL_RATE`` in the same

            
            sub-device. The unit of that control is pixels per second.


            Register list based drivers need to implement read-only sub-device nodes for the

            
            purpose. Devices that are not register list based need these to configure the

            
            device's internal processing pipeline.


            The first entity in the linear pipeline is the pixel array. The pixel array may

            
            be followed by other entities that are there to allow configuring binning,

            
            skipping, scaling or digital crop :ref:`v4l2-subdev-selections`.


            USB cameras etc. devices

            
            ~~~~~~~~~~~~~~~~~~~~~~~~


            USB video class hardware, as well as many cameras offering a similar higher

            
            level interface natively, generally use the concept of frame interval (or frame

            
            rate) on device level in firmware or hardware. This means lower level controls

            
            implemented by raw cameras may not be used on uAPI (or even kAPI) to control the

            
            frame interval on these devices.


            Power management

            
            ----------------


            Always use runtime PM to manage the power states of your device. Camera sensor

            
            drivers are in no way special in this respect: they are responsible for

            
            controlling the power state of the device they otherwise control as well. In

            
            general, the device must be powered on at least when its registers are being

            
            accessed and when it is streaming.


            Existing camera sensor drivers may rely on the old

            
            struct v4l2_subdev_core_ops->s_power() callback for bridge or ISP drivers to

            
            manage their power state. This is however **deprecated**. If you feel you need

            
            to begin calling an s_power from an ISP or a bridge driver, instead please add

            
            runtime PM support to the sensor driver you are using. Likewise, new drivers

            
            should not use s_power.


            Please see examples in e.g. ``drivers/media/i2c/ov8856.c`` and

            
            ``drivers/media/i2c/ccs/ccs-core.c``. The two drivers work in both ACPI

            
            and DT based systems.


            Control framework

            
            ~~~~~~~~~~~~~~~~~


            ``v4l2_ctrl_handler_setup()`` function may not be used in the device's runtime

            
            PM ``runtime_resume`` callback, as it has no way to figure out the power state

            
            of the device. This is because the power state of the device is only changed

            
            after the power state transition has taken place. The ``s_ctrl`` callback can be

            
            used to obtain device's power state after the power state transition:


            .. c:function:: int pm_runtime_get_if_in_use(struct device *dev);


            The function returns a non-zero value if it succeeded getting the power count or

            
            runtime PM was disabled, in either of which cases the driver may proceed to

            
            access the device.


            Rotation, orientation and flipping

            
            ----------------------------------


            Some systems have the camera sensor mounted upside down compared to its natural

            
            mounting rotation. In such cases, drivers shall expose the information to

            
            userspace with the :ref:`V4L2_CID_CAMERA_SENSOR_ROTATION

            
            <v4l2-camera-sensor-rotation>` control.


            Sensor drivers shall also report the sensor's mounting orientation with the

            
            :ref:`V4L2_CID_CAMERA_SENSOR_ORIENTATION <v4l2-camera-sensor-orientation>`.


            Use ``v4l2_fwnode_device_parse()`` to obtain rotation and orientation

            
            information from system firmware and ``v4l2_ctrl_new_fwnode_properties()`` to

            
            register the appropriate controls.


            Sensor drivers that have any vertical or horizontal flips embedded in the

            
            register programming sequences shall initialize the V4L2_CID_HFLIP and

            
            V4L2_CID_VFLIP controls with the values programmed by the register sequences.

            
            The default values of these controls shall be 0 (disabled). Especially these

            
            controls shall not be inverted, independently of the sensor's mounting

            
            rotation.
	.. SPDX-License-Identifier: GPL-2.0

	Writing camera sensor drivers
	=============================

	CSI-2 and parallel (BT.601 and BT.656) busses
	---------------------------------------------

	Please see :ref:`transmitter-receiver`.

	Handling clocks
	---------------

	Camera sensors have an internal clock tree including a PLL and a number of
	divisors. The clock tree is generally configured by the driver based on a few
	input parameters that are specific to the hardware:: the external clock frequency
	and the link frequency. The two parameters generally are obtained from system
	firmware. No other frequencies should be used in any circumstances.

	The reason why the clock frequencies are so important is that the clock signals
	come out of the SoC, and in many cases a specific frequency is designed to be
	used in the system. Using another frequency may cause harmful effects
	elsewhere. Therefore only the pre-determined frequencies are configurable by the
	user.

	ACPI
	~~~~

	Read the ``clock-frequency`` _DSD property to denote the frequency. The driver
	can rely on this frequency being used.

	Devicetree
	~~~~~~~~~~

	The currently preferred way to achieve this is using ``assigned-clocks``,
	``assigned-clock-parents`` and ``assigned-clock-rates`` properties. See
	``Documentation/devicetree/bindings/clock/clock-bindings.txt`` for more
	information. The driver then gets the frequency using ``clk_get_rate()``.

	This approach has the drawback that there's no guarantee that the frequency
	hasn't been modified directly or indirectly by another driver, or supported by
	the board's clock tree to begin with. Changes to the Common Clock Framework API
	are required to ensure reliability.

	Frame size
	----------

	There are two distinct ways to configure the frame size produced by camera
	sensors.

	Freely configurable camera sensor drivers
	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

	Freely configurable camera sensor drivers expose the device's internal
	processing pipeline as one or more sub-devices with different cropping and
	scaling configurations. The output size of the device is the result of a series
	of cropping and scaling operations from the device's pixel array's size.

	An example of such a driver is the CCS driver (see ``drivers/media/i2c/ccs``).

	Register list based drivers
	~~~~~~~~~~~~~~~~~~~~~~~~~~~

	Register list based drivers generally, instead of able to configure the device
	they control based on user requests, are limited to a number of preset
	configurations that combine a number of different parameters that on hardware
	level are independent. How a driver picks such configuration is based on the
	format set on a source pad at the end of the device's internal pipeline.

	Most sensor drivers are implemented this way, see e.g.
	``drivers/media/i2c/imx319.c`` for an example.

	Frame interval configuration
	----------------------------

	There are two different methods for obtaining possibilities for different frame
	intervals as well as configuring the frame interval. Which one to implement
	depends on the type of the device.

	Raw camera sensors
	~~~~~~~~~~~~~~~~~~

	Instead of a high level parameter such as frame interval, the frame interval is
	a result of the configuration of a number of camera sensor implementation
	specific parameters. Luckily, these parameters tend to be the same for more or
	less all modern raw camera sensors.

	The frame interval is calculated using the following equation::

	frame interval = (analogue crop width + horizontal blanking) *
	(analogue crop height + vertical blanking) / pixel rate

	The formula is bus independent and is applicable for raw timing parameters on
	large variety of devices beyond camera sensors. Devices that have no analogue
	crop, use the full source image size, i.e. pixel array size.

	Horizontal and vertical blanking are specified by ``V4L2_CID_HBLANK`` and
	``V4L2_CID_VBLANK``, respectively. The unit of the ``V4L2_CID_HBLANK`` control
	is pixels and the unit of the ``V4L2_CID_VBLANK`` is lines. The pixel rate in
	the sensor's pixel array is specified by ``V4L2_CID_PIXEL_RATE`` in the same
	sub-device. The unit of that control is pixels per second.

	Register list based drivers need to implement read-only sub-device nodes for the
	purpose. Devices that are not register list based need these to configure the
	device's internal processing pipeline.

	The first entity in the linear pipeline is the pixel array. The pixel array may
	be followed by other entities that are there to allow configuring binning,
	skipping, scaling or digital crop :ref:`v4l2-subdev-selections`.

	USB cameras etc. devices
	~~~~~~~~~~~~~~~~~~~~~~~~

	USB video class hardware, as well as many cameras offering a similar higher
	level interface natively, generally use the concept of frame interval (or frame
	rate) on device level in firmware or hardware. This means lower level controls
	implemented by raw cameras may not be used on uAPI (or even kAPI) to control the
	frame interval on these devices.

	Power management
	----------------

	Always use runtime PM to manage the power states of your device. Camera sensor
	drivers are in no way special in this respect: they are responsible for
	controlling the power state of the device they otherwise control as well. In
	general, the device must be powered on at least when its registers are being
	accessed and when it is streaming.

	Existing camera sensor drivers may rely on the old
	struct v4l2_subdev_core_ops->s_power() callback for bridge or ISP drivers to
	manage their power state. This is however deprecated. If you feel you need
	to begin calling an s_power from an ISP or a bridge driver, instead please add
	runtime PM support to the sensor driver you are using. Likewise, new drivers
	should not use s_power.

	Please see examples in e.g. ``drivers/media/i2c/ov8856.c`` and
	``drivers/media/i2c/ccs/ccs-core.c``. The two drivers work in both ACPI
	and DT based systems.

	Control framework
	~~~~~~~~~~~~~~~~~

	``v4l2_ctrl_handler_setup()`` function may not be used in the device's runtime
	PM ``runtime_resume`` callback, as it has no way to figure out the power state
	of the device. This is because the power state of the device is only changed
	after the power state transition has taken place. The ``s_ctrl`` callback can be
	used to obtain device's power state after the power state transition:

	.. c:function:: int pm_runtime_get_if_in_use(struct device *dev);

	The function returns a non-zero value if it succeeded getting the power count or
	runtime PM was disabled, in either of which cases the driver may proceed to
	access the device.

	Rotation, orientation and flipping
	----------------------------------

	Some systems have the camera sensor mounted upside down compared to its natural
	mounting rotation. In such cases, drivers shall expose the information to
	userspace with the :ref:`V4L2_CID_CAMERA_SENSOR_ROTATION
	<v4l2-camera-sensor-rotation>` control.

	Sensor drivers shall also report the sensor's mounting orientation with the
	:ref:`V4L2_CID_CAMERA_SENSOR_ORIENTATION <v4l2-camera-sensor-orientation>`.

	Use ``v4l2_fwnode_device_parse()`` to obtain rotation and orientation
	information from system firmware and ``v4l2_ctrl_new_fwnode_properties()`` to
	register the appropriate controls.

	Sensor drivers that have any vertical or horizontal flips embedded in the
	register programming sequences shall initialize the V4L2_CID_HFLIP and
	V4L2_CID_VFLIP controls with the values programmed by the register sequences.
	The default values of these controls shall be 0 (disabled). Especially these
	controls shall not be inverted, independently of the sensor's mounting
	rotation.