| .. include:: <isonum.txt> |
| |
| ===================== |
| VFIO Mediated devices |
| ===================== |
| |
| :Copyright: |copy| 2016, NVIDIA CORPORATION. All rights reserved. |
| :Author: Neo Jia <cjia@nvidia.com> |
| :Author: Kirti Wankhede <kwankhede@nvidia.com> |
| |
| This program is free software; you can redistribute it and/or modify |
| it under the terms of the GNU General Public License version 2 as |
| published by the Free Software Foundation. |
| |
| |
| Virtual Function I/O (VFIO) Mediated devices[1] |
| =============================================== |
| |
| The number of use cases for virtualizing DMA devices that do not have built-in |
| SR_IOV capability is increasing. Previously, to virtualize such devices, |
| developers had to create their own management interfaces and APIs, and then |
| integrate them with user space software. To simplify integration with user space |
| software, we have identified common requirements and a unified management |
| interface for such devices. |
| |
| The VFIO driver framework provides unified APIs for direct device access. It is |
| an IOMMU/device-agnostic framework for exposing direct device access to user |
| space in a secure, IOMMU-protected environment. This framework is used for |
| multiple devices, such as GPUs, network adapters, and compute accelerators. With |
| direct device access, virtual machines or user space applications have direct |
| access to the physical device. This framework is reused for mediated devices. |
| |
| The mediated core driver provides a common interface for mediated device |
| management that can be used by drivers of different devices. This module |
| provides a generic interface to perform these operations: |
| |
| * Create and destroy a mediated device |
| * Add a mediated device to and remove it from a mediated bus driver |
| * Add a mediated device to and remove it from an IOMMU group |
| |
| The mediated core driver also provides an interface to register a bus driver. |
| For example, the mediated VFIO mdev driver is designed for mediated devices and |
| supports VFIO APIs. The mediated bus driver adds a mediated device to and |
| removes it from a VFIO group. |
| |
| The following high-level block diagram shows the main components and interfaces |
| in the VFIO mediated driver framework. The diagram shows NVIDIA, Intel, and IBM |
| devices as examples, as these devices are the first devices to use this module:: |
| |
| +---------------+ |
| | | |
| | +-----------+ | mdev_register_driver() +--------------+ |
| | | | +<------------------------+ | |
| | | mdev | | | | |
| | | bus | +------------------------>+ vfio_mdev.ko |<-> VFIO user |
| | | driver | | probe()/remove() | | APIs |
| | | | | +--------------+ |
| | +-----------+ | |
| | | |
| | MDEV CORE | |
| | MODULE | |
| | mdev.ko | |
| | +-----------+ | mdev_register_device() +--------------+ |
| | | | +<------------------------+ | |
| | | | | | nvidia.ko |<-> physical |
| | | | +------------------------>+ | device |
| | | | | callbacks +--------------+ |
| | | Physical | | |
| | | device | | mdev_register_device() +--------------+ |
| | | interface | |<------------------------+ | |
| | | | | | i915.ko |<-> physical |
| | | | +------------------------>+ | device |
| | | | | callbacks +--------------+ |
| | | | | |
| | | | | mdev_register_device() +--------------+ |
| | | | +<------------------------+ | |
| | | | | | ccw_device.ko|<-> physical |
| | | | +------------------------>+ | device |
| | | | | callbacks +--------------+ |
| | +-----------+ | |
| +---------------+ |
| |
| |
| Registration Interfaces |
| ======================= |
| |
| The mediated core driver provides the following types of registration |
| interfaces: |
| |
| * Registration interface for a mediated bus driver |
| * Physical device driver interface |
| |
| Registration Interface for a Mediated Bus Driver |
| ------------------------------------------------ |
| |
| The registration interface for a mediated bus driver provides the following |
| structure to represent a mediated device's driver:: |
| |
| /* |
| * struct mdev_driver [2] - Mediated device's driver |
| * @name: driver name |
| * @probe: called when new device created |
| * @remove: called when device removed |
| * @driver: device driver structure |
| */ |
| struct mdev_driver { |
| const char *name; |
| int (*probe) (struct device *dev); |
| void (*remove) (struct device *dev); |
| struct device_driver driver; |
| }; |
| |
| A mediated bus driver for mdev should use this structure in the function calls |
| to register and unregister itself with the core driver: |
| |
| * Register:: |
| |
| extern int mdev_register_driver(struct mdev_driver *drv, |
| struct module *owner); |
| |
| * Unregister:: |
| |
| extern void mdev_unregister_driver(struct mdev_driver *drv); |
| |
| The mediated bus driver is responsible for adding mediated devices to the VFIO |
| group when devices are bound to the driver and removing mediated devices from |
| the VFIO when devices are unbound from the driver. |
| |
| |
| Physical Device Driver Interface |
| -------------------------------- |
| |
| The physical device driver interface provides the mdev_parent_ops[3] structure |
| to define the APIs to manage work in the mediated core driver that is related |
| to the physical device. |
| |
| The structures in the mdev_parent_ops structure are as follows: |
| |
| * dev_attr_groups: attributes of the parent device |
| * mdev_attr_groups: attributes of the mediated device |
| * supported_config: attributes to define supported configurations |
| |
| The functions in the mdev_parent_ops structure are as follows: |
| |
| * create: allocate basic resources in a driver for a mediated device |
| * remove: free resources in a driver when a mediated device is destroyed |
| |
| The callbacks in the mdev_parent_ops structure are as follows: |
| |
| * open: open callback of mediated device |
| * close: close callback of mediated device |
| * ioctl: ioctl callback of mediated device |
| * read : read emulation callback |
| * write: write emulation callback |
| * mmap: mmap emulation callback |
| |
| A driver should use the mdev_parent_ops structure in the function call to |
| register itself with the mdev core driver:: |
| |
| extern int mdev_register_device(struct device *dev, |
| const struct mdev_parent_ops *ops); |
| |
| However, the mdev_parent_ops structure is not required in the function call |
| that a driver should use to unregister itself with the mdev core driver:: |
| |
| extern void mdev_unregister_device(struct device *dev); |
| |
| |
| Mediated Device Management Interface Through sysfs |
| ================================================== |
| |
| The management interface through sysfs enables user space software, such as |
| libvirt, to query and configure mediated devices in a hardware-agnostic fashion. |
| This management interface provides flexibility to the underlying physical |
| device's driver to support features such as: |
| |
| * Mediated device hot plug |
| * Multiple mediated devices in a single virtual machine |
| * Multiple mediated devices from different physical devices |
| |
| Links in the mdev_bus Class Directory |
| ------------------------------------- |
| The /sys/class/mdev_bus/ directory contains links to devices that are registered |
| with the mdev core driver. |
| |
| Directories and files under the sysfs for Each Physical Device |
| -------------------------------------------------------------- |
| |
| :: |
| |
| |- [parent physical device] |
| |--- Vendor-specific-attributes [optional] |
| |--- [mdev_supported_types] |
| | |--- [<type-id>] |
| | | |--- create |
| | | |--- name |
| | | |--- available_instances |
| | | |--- device_api |
| | | |--- description |
| | | |--- [devices] |
| | |--- [<type-id>] |
| | | |--- create |
| | | |--- name |
| | | |--- available_instances |
| | | |--- device_api |
| | | |--- description |
| | | |--- [devices] |
| | |--- [<type-id>] |
| | |--- create |
| | |--- name |
| | |--- available_instances |
| | |--- device_api |
| | |--- description |
| | |--- [devices] |
| |
| * [mdev_supported_types] |
| |
| The list of currently supported mediated device types and their details. |
| |
| [<type-id>], device_api, and available_instances are mandatory attributes |
| that should be provided by vendor driver. |
| |
| * [<type-id>] |
| |
| The [<type-id>] name is created by adding the device driver string as a prefix |
| to the string provided by the vendor driver. This format of this name is as |
| follows:: |
| |
| sprintf(buf, "%s-%s", dev_driver_string(parent->dev), group->name); |
| |
| (or using mdev_parent_dev(mdev) to arrive at the parent device outside |
| of the core mdev code) |
| |
| * device_api |
| |
| This attribute should show which device API is being created, for example, |
| "vfio-pci" for a PCI device. |
| |
| * available_instances |
| |
| This attribute should show the number of devices of type <type-id> that can be |
| created. |
| |
| * [device] |
| |
| This directory contains links to the devices of type <type-id> that have been |
| created. |
| |
| * name |
| |
| This attribute should show human readable name. This is optional attribute. |
| |
| * description |
| |
| This attribute should show brief features/description of the type. This is |
| optional attribute. |
| |
| Directories and Files Under the sysfs for Each mdev Device |
| ---------------------------------------------------------- |
| |
| :: |
| |
| |- [parent phy device] |
| |--- [$MDEV_UUID] |
| |--- remove |
| |--- mdev_type {link to its type} |
| |--- vendor-specific-attributes [optional] |
| |
| * remove (write only) |
| |
| Writing '1' to the 'remove' file destroys the mdev device. The vendor driver can |
| fail the remove() callback if that device is active and the vendor driver |
| doesn't support hot unplug. |
| |
| Example:: |
| |
| # echo 1 > /sys/bus/mdev/devices/$mdev_UUID/remove |
| |
| Mediated device Hot plug |
| ------------------------ |
| |
| Mediated devices can be created and assigned at runtime. The procedure to hot |
| plug a mediated device is the same as the procedure to hot plug a PCI device. |
| |
| Translation APIs for Mediated Devices |
| ===================================== |
| |
| The following APIs are provided for translating user pfn to host pfn in a VFIO |
| driver:: |
| |
| extern int vfio_pin_pages(struct device *dev, unsigned long *user_pfn, |
| int npage, int prot, unsigned long *phys_pfn); |
| |
| extern int vfio_unpin_pages(struct device *dev, unsigned long *user_pfn, |
| int npage); |
| |
| These functions call back into the back-end IOMMU module by using the pin_pages |
| and unpin_pages callbacks of the struct vfio_iommu_driver_ops[4]. Currently |
| these callbacks are supported in the TYPE1 IOMMU module. To enable them for |
| other IOMMU backend modules, such as PPC64 sPAPR module, they need to provide |
| these two callback functions. |
| |
| Using the Sample Code |
| ===================== |
| |
| mtty.c in samples/vfio-mdev/ directory is a sample driver program to |
| demonstrate how to use the mediated device framework. |
| |
| The sample driver creates an mdev device that simulates a serial port over a PCI |
| card. |
| |
| 1. Build and load the mtty.ko module. |
| |
| This step creates a dummy device, /sys/devices/virtual/mtty/mtty/ |
| |
| Files in this device directory in sysfs are similar to the following:: |
| |
| # tree /sys/devices/virtual/mtty/mtty/ |
| /sys/devices/virtual/mtty/mtty/ |
| |-- mdev_supported_types |
| | |-- mtty-1 |
| | | |-- available_instances |
| | | |-- create |
| | | |-- device_api |
| | | |-- devices |
| | | `-- name |
| | `-- mtty-2 |
| | |-- available_instances |
| | |-- create |
| | |-- device_api |
| | |-- devices |
| | `-- name |
| |-- mtty_dev |
| | `-- sample_mtty_dev |
| |-- power |
| | |-- autosuspend_delay_ms |
| | |-- control |
| | |-- runtime_active_time |
| | |-- runtime_status |
| | `-- runtime_suspended_time |
| |-- subsystem -> ../../../../class/mtty |
| `-- uevent |
| |
| 2. Create a mediated device by using the dummy device that you created in the |
| previous step:: |
| |
| # echo "83b8f4f2-509f-382f-3c1e-e6bfe0fa1001" > \ |
| /sys/devices/virtual/mtty/mtty/mdev_supported_types/mtty-2/create |
| |
| 3. Add parameters to qemu-kvm:: |
| |
| -device vfio-pci,\ |
| sysfsdev=/sys/bus/mdev/devices/83b8f4f2-509f-382f-3c1e-e6bfe0fa1001 |
| |
| 4. Boot the VM. |
| |
| In the Linux guest VM, with no hardware on the host, the device appears |
| as follows:: |
| |
| # lspci -s 00:05.0 -xxvv |
| 00:05.0 Serial controller: Device 4348:3253 (rev 10) (prog-if 02 [16550]) |
| Subsystem: Device 4348:3253 |
| Physical Slot: 5 |
| Control: I/O+ Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- |
| Stepping- SERR- FastB2B- DisINTx- |
| Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- |
| <TAbort- <MAbort- >SERR- <PERR- INTx- |
| Interrupt: pin A routed to IRQ 10 |
| Region 0: I/O ports at c150 [size=8] |
| Region 1: I/O ports at c158 [size=8] |
| Kernel driver in use: serial |
| 00: 48 43 53 32 01 00 00 02 10 02 00 07 00 00 00 00 |
| 10: 51 c1 00 00 59 c1 00 00 00 00 00 00 00 00 00 00 |
| 20: 00 00 00 00 00 00 00 00 00 00 00 00 48 43 53 32 |
| 30: 00 00 00 00 00 00 00 00 00 00 00 00 0a 01 00 00 |
| |
| In the Linux guest VM, dmesg output for the device is as follows: |
| |
| serial 0000:00:05.0: PCI INT A -> Link[LNKA] -> GSI 10 (level, high) -> IRQ 10 |
| 0000:00:05.0: ttyS1 at I/O 0xc150 (irq = 10) is a 16550A |
| 0000:00:05.0: ttyS2 at I/O 0xc158 (irq = 10) is a 16550A |
| |
| |
| 5. In the Linux guest VM, check the serial ports:: |
| |
| # setserial -g /dev/ttyS* |
| /dev/ttyS0, UART: 16550A, Port: 0x03f8, IRQ: 4 |
| /dev/ttyS1, UART: 16550A, Port: 0xc150, IRQ: 10 |
| /dev/ttyS2, UART: 16550A, Port: 0xc158, IRQ: 10 |
| |
| 6. Using minicom or any terminal emulation program, open port /dev/ttyS1 or |
| /dev/ttyS2 with hardware flow control disabled. |
| |
| 7. Type data on the minicom terminal or send data to the terminal emulation |
| program and read the data. |
| |
| Data is loop backed from hosts mtty driver. |
| |
| 8. Destroy the mediated device that you created:: |
| |
| # echo 1 > /sys/bus/mdev/devices/83b8f4f2-509f-382f-3c1e-e6bfe0fa1001/remove |
| |
| References |
| ========== |
| |
| 1. See Documentation/vfio.txt for more information on VFIO. |
| 2. struct mdev_driver in include/linux/mdev.h |
| 3. struct mdev_parent_ops in include/linux/mdev.h |
| 4. struct vfio_iommu_driver_ops in include/linux/vfio.h |
