diff options
Diffstat (limited to 'Documentation/vfio-mediated-device.txt')
| -rw-r--r-- | Documentation/vfio-mediated-device.txt | 414 |
1 files changed, 0 insertions, 414 deletions
diff --git a/Documentation/vfio-mediated-device.txt b/Documentation/vfio-mediated-device.txt deleted file mode 100644 index c3f69bcaf96e..000000000000 --- a/Documentation/vfio-mediated-device.txt +++ /dev/null @@ -1,414 +0,0 @@ -.. include:: <isonum.txt> - -===================== -VFIO Mediated devices -===================== - -:Copyright: |copy| 2016, NVIDIA CORPORATION. All rights reserved. -:Author: Neo Jia <cjia@nvidia.com> -:Author: Kirti Wankhede <kwankhede@nvidia.com> - -This program is free software; you can redistribute it and/or modify -it under the terms of the GNU General Public License version 2 as -published by the Free Software Foundation. - - -Virtual Function I/O (VFIO) Mediated devices[1] -=============================================== - -The number of use cases for virtualizing DMA devices that do not have built-in -SR_IOV capability is increasing. Previously, to virtualize such devices, -developers had to create their own management interfaces and APIs, and then -integrate them with user space software. To simplify integration with user space -software, we have identified common requirements and a unified management -interface for such devices. - -The VFIO driver framework provides unified APIs for direct device access. It is -an IOMMU/device-agnostic framework for exposing direct device access to user -space in a secure, IOMMU-protected environment. This framework is used for -multiple devices, such as GPUs, network adapters, and compute accelerators. With -direct device access, virtual machines or user space applications have direct -access to the physical device. This framework is reused for mediated devices. - -The mediated core driver provides a common interface for mediated device -management that can be used by drivers of different devices. This module -provides a generic interface to perform these operations: - -* Create and destroy a mediated device -* Add a mediated device to and remove it from a mediated bus driver -* Add a mediated device to and remove it from an IOMMU group - -The mediated core driver also provides an interface to register a bus driver. -For example, the mediated VFIO mdev driver is designed for mediated devices and -supports VFIO APIs. The mediated bus driver adds a mediated device to and -removes it from a VFIO group. - -The following high-level block diagram shows the main components and interfaces -in the VFIO mediated driver framework. The diagram shows NVIDIA, Intel, and IBM -devices as examples, as these devices are the first devices to use this module:: - - +---------------+ - | | - | +-----------+ | mdev_register_driver() +--------------+ - | | | +<------------------------+ | - | | mdev | | | | - | | bus | +------------------------>+ vfio_mdev.ko |<-> VFIO user - | | driver | | probe()/remove() | | APIs - | | | | +--------------+ - | +-----------+ | - | | - | MDEV CORE | - | MODULE | - | mdev.ko | - | +-----------+ | mdev_register_device() +--------------+ - | | | +<------------------------+ | - | | | | | nvidia.ko |<-> physical - | | | +------------------------>+ | device - | | | | callbacks +--------------+ - | | Physical | | - | | device | | mdev_register_device() +--------------+ - | | interface | |<------------------------+ | - | | | | | i915.ko |<-> physical - | | | +------------------------>+ | device - | | | | callbacks +--------------+ - | | | | - | | | | mdev_register_device() +--------------+ - | | | +<------------------------+ | - | | | | | ccw_device.ko|<-> physical - | | | +------------------------>+ | device - | | | | callbacks +--------------+ - | +-----------+ | - +---------------+ - - -Registration Interfaces -======================= - -The mediated core driver provides the following types of registration -interfaces: - -* Registration interface for a mediated bus driver -* Physical device driver interface - -Registration Interface for a Mediated Bus Driver ------------------------------------------------- - -The registration interface for a mediated bus driver provides the following -structure to represent a mediated device's driver:: - - /* - * struct mdev_driver [2] - Mediated device's driver - * @name: driver name - * @probe: called when new device created - * @remove: called when device removed - * @driver: device driver structure - */ - struct mdev_driver { - const char *name; - int (*probe) (struct device *dev); - void (*remove) (struct device *dev); - struct device_driver driver; - }; - -A mediated bus driver for mdev should use this structure in the function calls -to register and unregister itself with the core driver: - -* Register:: - - extern int mdev_register_driver(struct mdev_driver *drv, - struct module *owner); - -* Unregister:: - - extern void mdev_unregister_driver(struct mdev_driver *drv); - -The mediated bus driver is responsible for adding mediated devices to the VFIO -group when devices are bound to the driver and removing mediated devices from -the VFIO when devices are unbound from the driver. - - -Physical Device Driver Interface --------------------------------- - -The physical device driver interface provides the mdev_parent_ops[3] structure -to define the APIs to manage work in the mediated core driver that is related -to the physical device. - -The structures in the mdev_parent_ops structure are as follows: - -* dev_attr_groups: attributes of the parent device -* mdev_attr_groups: attributes of the mediated device -* supported_config: attributes to define supported configurations - -The functions in the mdev_parent_ops structure are as follows: - -* create: allocate basic resources in a driver for a mediated device -* remove: free resources in a driver when a mediated device is destroyed - -(Note that mdev-core provides no implicit serialization of create/remove -callbacks per mdev parent device, per mdev type, or any other categorization. -Vendor drivers are expected to be fully asynchronous in this respect or -provide their own internal resource protection.) - -The callbacks in the mdev_parent_ops structure are as follows: - -* open: open callback of mediated device -* close: close callback of mediated device -* ioctl: ioctl callback of mediated device -* read : read emulation callback -* write: write emulation callback -* mmap: mmap emulation callback - -A driver should use the mdev_parent_ops structure in the function call to -register itself with the mdev core driver:: - - extern int mdev_register_device(struct device *dev, - const struct mdev_parent_ops *ops); - -However, the mdev_parent_ops structure is not required in the function call -that a driver should use to unregister itself with the mdev core driver:: - - extern void mdev_unregister_device(struct device *dev); - - -Mediated Device Management Interface Through sysfs -================================================== - -The management interface through sysfs enables user space software, such as -libvirt, to query and configure mediated devices in a hardware-agnostic fashion. -This management interface provides flexibility to the underlying physical -device's driver to support features such as: - -* Mediated device hot plug -* Multiple mediated devices in a single virtual machine -* Multiple mediated devices from different physical devices - -Links in the mdev_bus Class Directory -------------------------------------- -The /sys/class/mdev_bus/ directory contains links to devices that are registered -with the mdev core driver. - -Directories and files under the sysfs for Each Physical Device --------------------------------------------------------------- - -:: - - |- [parent physical device] - |--- Vendor-specific-attributes [optional] - |--- [mdev_supported_types] - | |--- [<type-id>] - | | |--- create - | | |--- name - | | |--- available_instances - | | |--- device_api - | | |--- description - | | |--- [devices] - | |--- [<type-id>] - | | |--- create - | | |--- name - | | |--- available_instances - | | |--- device_api - | | |--- description - | | |--- [devices] - | |--- [<type-id>] - | |--- create - | |--- name - | |--- available_instances - | |--- device_api - | |--- description - | |--- [devices] - -* [mdev_supported_types] - - The list of currently supported mediated device types and their details. - - [<type-id>], device_api, and available_instances are mandatory attributes - that should be provided by vendor driver. - -* [<type-id>] - - The [<type-id>] name is created by adding the device driver string as a prefix - to the string provided by the vendor driver. This format of this name is as - follows:: - - sprintf(buf, "%s-%s", dev_driver_string(parent->dev), group->name); - - (or using mdev_parent_dev(mdev) to arrive at the parent device outside - of the core mdev code) - -* device_api - - This attribute should show which device API is being created, for example, - "vfio-pci" for a PCI device. - -* available_instances - - This attribute should show the number of devices of type <type-id> that can be - created. - -* [device] - - This directory contains links to the devices of type <type-id> that have been - created. - -* name - - This attribute should show human readable name. This is optional attribute. - -* description - - This attribute should show brief features/description of the type. This is - optional attribute. - -Directories and Files Under the sysfs for Each mdev Device ----------------------------------------------------------- - -:: - - |- [parent phy device] - |--- [$MDEV_UUID] - |--- remove - |--- mdev_type {link to its type} - |--- vendor-specific-attributes [optional] - -* remove (write only) - -Writing '1' to the 'remove' file destroys the mdev device. The vendor driver can -fail the remove() callback if that device is active and the vendor driver -doesn't support hot unplug. - -Example:: - - # echo 1 > /sys/bus/mdev/devices/$mdev_UUID/remove - -Mediated device Hot plug ------------------------- - -Mediated devices can be created and assigned at runtime. The procedure to hot -plug a mediated device is the same as the procedure to hot plug a PCI device. - -Translation APIs for Mediated Devices -===================================== - -The following APIs are provided for translating user pfn to host pfn in a VFIO -driver:: - - extern int vfio_pin_pages(struct device *dev, unsigned long *user_pfn, - int npage, int prot, unsigned long *phys_pfn); - - extern int vfio_unpin_pages(struct device *dev, unsigned long *user_pfn, - int npage); - -These functions call back into the back-end IOMMU module by using the pin_pages -and unpin_pages callbacks of the struct vfio_iommu_driver_ops[4]. Currently -these callbacks are supported in the TYPE1 IOMMU module. To enable them for -other IOMMU backend modules, such as PPC64 sPAPR module, they need to provide -these two callback functions. - -Using the Sample Code -===================== - -mtty.c in samples/vfio-mdev/ directory is a sample driver program to -demonstrate how to use the mediated device framework. - -The sample driver creates an mdev device that simulates a serial port over a PCI -card. - -1. Build and load the mtty.ko module. - - This step creates a dummy device, /sys/devices/virtual/mtty/mtty/ - - Files in this device directory in sysfs are similar to the following:: - - # tree /sys/devices/virtual/mtty/mtty/ - /sys/devices/virtual/mtty/mtty/ - |-- mdev_supported_types - | |-- mtty-1 - | | |-- available_instances - | | |-- create - | | |-- device_api - | | |-- devices - | | `-- name - | `-- mtty-2 - | |-- available_instances - | |-- create - | |-- device_api - | |-- devices - | `-- name - |-- mtty_dev - | `-- sample_mtty_dev - |-- power - | |-- autosuspend_delay_ms - | |-- control - | |-- runtime_active_time - | |-- runtime_status - | `-- runtime_suspended_time - |-- subsystem -> ../../../../class/mtty - `-- uevent - -2. Create a mediated device by using the dummy device that you created in the - previous step:: - - # echo "83b8f4f2-509f-382f-3c1e-e6bfe0fa1001" > \ - /sys/devices/virtual/mtty/mtty/mdev_supported_types/mtty-2/create - -3. Add parameters to qemu-kvm:: - - -device vfio-pci,\ - sysfsdev=/sys/bus/mdev/devices/83b8f4f2-509f-382f-3c1e-e6bfe0fa1001 - -4. Boot the VM. - - In the Linux guest VM, with no hardware on the host, the device appears - as follows:: - - # lspci -s 00:05.0 -xxvv - 00:05.0 Serial controller: Device 4348:3253 (rev 10) (prog-if 02 [16550]) - Subsystem: Device 4348:3253 - Physical Slot: 5 - Control: I/O+ Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- - Stepping- SERR- FastB2B- DisINTx- - Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- - <TAbort- <MAbort- >SERR- <PERR- INTx- - Interrupt: pin A routed to IRQ 10 - Region 0: I/O ports at c150 [size=8] - Region 1: I/O ports at c158 [size=8] - Kernel driver in use: serial - 00: 48 43 53 32 01 00 00 02 10 02 00 07 00 00 00 00 - 10: 51 c1 00 00 59 c1 00 00 00 00 00 00 00 00 00 00 - 20: 00 00 00 00 00 00 00 00 00 00 00 00 48 43 53 32 - 30: 00 00 00 00 00 00 00 00 00 00 00 00 0a 01 00 00 - - In the Linux guest VM, dmesg output for the device is as follows: - - serial 0000:00:05.0: PCI INT A -> Link[LNKA] -> GSI 10 (level, high) -> IRQ 10 - 0000:00:05.0: ttyS1 at I/O 0xc150 (irq = 10) is a 16550A - 0000:00:05.0: ttyS2 at I/O 0xc158 (irq = 10) is a 16550A - - -5. In the Linux guest VM, check the serial ports:: - - # setserial -g /dev/ttyS* - /dev/ttyS0, UART: 16550A, Port: 0x03f8, IRQ: 4 - /dev/ttyS1, UART: 16550A, Port: 0xc150, IRQ: 10 - /dev/ttyS2, UART: 16550A, Port: 0xc158, IRQ: 10 - -6. Using minicom or any terminal emulation program, open port /dev/ttyS1 or - /dev/ttyS2 with hardware flow control disabled. - -7. Type data on the minicom terminal or send data to the terminal emulation - program and read the data. - - Data is loop backed from hosts mtty driver. - -8. Destroy the mediated device that you created:: - - # echo 1 > /sys/bus/mdev/devices/83b8f4f2-509f-382f-3c1e-e6bfe0fa1001/remove - -References -========== - -1. See Documentation/vfio.txt for more information on VFIO. -2. struct mdev_driver in include/linux/mdev.h -3. struct mdev_parent_ops in include/linux/mdev.h -4. struct vfio_iommu_driver_ops in include/linux/vfio.h |