diff options
Diffstat (limited to 'Documentation/fpga/dfl.rst')
| -rw-r--r-- | Documentation/fpga/dfl.rst | 172 |
1 files changed, 167 insertions, 5 deletions
diff --git a/Documentation/fpga/dfl.rst b/Documentation/fpga/dfl.rst index 094fc8aacd8e..15b670926084 100644 --- a/Documentation/fpga/dfl.rst +++ b/Documentation/fpga/dfl.rst @@ -7,9 +7,10 @@ Authors: - Enno Luebbers <enno.luebbers@intel.com> - Xiao Guangrong <guangrong.xiao@linux.intel.com> - Wu Hao <hao.wu@intel.com> +- Xu Yilun <yilun.xu@intel.com> -The Device Feature List (DFL) FPGA framework (and drivers according to this -this framework) hides the very details of low layer hardwares and provides +The Device Feature List (DFL) FPGA framework (and drivers according to +this framework) hides the very details of low layer hardware and provides unified interfaces to userspace. Applications could use these interfaces to configure, enumerate, open and access FPGA accelerators on platforms which implement the DFL in the device memory. Besides this, the DFL framework @@ -56,7 +57,7 @@ FPGA Interface Unit (FIU) represents a standalone functional unit for the interface to FPGA, e.g. the FPGA Management Engine (FME) and Port (more descriptions on FME and Port in later sections). -Accelerated Function Unit (AFU) represents a FPGA programmable region and +Accelerated Function Unit (AFU) represents an FPGA programmable region and always connects to a FIU (e.g. a Port) as its child as illustrated above. Private Features represent sub features of the FIU and AFU. They could be @@ -89,6 +90,8 @@ The following functions are exposed through ioctls: - Program bitstream (DFL_FPGA_FME_PORT_PR) - Assign port to PF (DFL_FPGA_FME_PORT_ASSIGN) - Release port from PF (DFL_FPGA_FME_PORT_RELEASE) +- Get number of irqs of FME global error (DFL_FPGA_FME_ERR_GET_IRQ_NUM) +- Set interrupt trigger for FME error (DFL_FPGA_FME_ERR_SET_IRQ) More functions are exposed through sysfs (/sys/class/fpga_region/regionX/dfl-fme.n/): @@ -118,6 +121,11 @@ More functions are exposed through sysfs management information (current temperature, thresholds, threshold status, etc.). + Performance reporting + performance counters are exposed through perf PMU APIs. Standard perf tool + can be used to monitor all available perf events. Please see performance + counter section below for more detailed information. + FIU - PORT ========== @@ -144,6 +152,10 @@ The following functions are exposed through ioctls: - Map DMA buffer (DFL_FPGA_PORT_DMA_MAP) - Unmap DMA buffer (DFL_FPGA_PORT_DMA_UNMAP) - Reset AFU (DFL_FPGA_PORT_RESET) +- Get number of irqs of port error (DFL_FPGA_PORT_ERR_GET_IRQ_NUM) +- Set interrupt trigger for port error (DFL_FPGA_PORT_ERR_SET_IRQ) +- Get number of irqs of UINT (DFL_FPGA_PORT_UINT_GET_IRQ_NUM) +- Set interrupt trigger for UINT (DFL_FPGA_PORT_UINT_SET_IRQ) DFL_FPGA_PORT_RESET: reset the FPGA Port and its AFU. Userspace can do Port @@ -193,7 +205,7 @@ given Device Feature Lists and create platform devices for feature devices also abstracts operations for the private features and exposes common ops to feature device drivers. -The FPGA DFL Device could be different hardwares, e.g. PCIe device, platform +The FPGA DFL Device could be different hardware, e.g. PCIe device, platform device and etc. Its driver module is always loaded first once the device is created by the system. This driver plays an infrastructural role in the driver architecture. It locates the DFLs in the device memory, handles them @@ -299,7 +311,7 @@ The driver organization in virtualization case is illustrated below: | PCI PF Device | | | PCI VF Device | +---------------+ | +---------------+ -FPGA PCIe device driver is always loaded first once a FPGA PCIe PF or VF device +FPGA PCIe device driver is always loaded first once an FPGA PCIe PF or VF device is detected. It: * Finishes enumeration on both FPGA PCIe PF and VF device using common @@ -378,6 +390,98 @@ The device nodes used for ioctl() or mmap() can be referenced through:: /sys/class/fpga_region/<regionX>/<dfl-port.n>/dev +Performance Counters +==================== +Performance reporting is one private feature implemented in FME. It could +supports several independent, system-wide, device counter sets in hardware to +monitor and count for performance events, including "basic", "cache", "fabric", +"vtd" and "vtd_sip" counters. Users could use standard perf tool to monitor +FPGA cache hit/miss rate, transaction number, interface clock counter of AFU +and other FPGA performance events. + +Different FPGA devices may have different counter sets, depending on hardware +implementation. E.g., some discrete FPGA cards don't have any cache. User could +use "perf list" to check which perf events are supported by target hardware. + +In order to allow user to use standard perf API to access these performance +counters, driver creates a perf PMU, and related sysfs interfaces in +/sys/bus/event_source/devices/dfl_fme* to describe available perf events and +configuration options. + +The "format" directory describes the format of the config field of struct +perf_event_attr. There are 3 bitfields for config: "evtype" defines which type +the perf event belongs to; "event" is the identity of the event within its +category; "portid" is introduced to decide counters set to monitor on FPGA +overall data or a specific port. + +The "events" directory describes the configuration templates for all available +events which can be used with perf tool directly. For example, fab_mmio_read +has the configuration "event=0x06,evtype=0x02,portid=0xff", which shows this +event belongs to fabric type (0x02), the local event id is 0x06 and it is for +overall monitoring (portid=0xff). + +Example usage of perf:: + + $# perf list |grep dfl_fme + + dfl_fme0/fab_mmio_read/ [Kernel PMU event] + <...> + dfl_fme0/fab_port_mmio_read,portid=?/ [Kernel PMU event] + <...> + + $# perf stat -a -e dfl_fme0/fab_mmio_read/ <command> + or + $# perf stat -a -e dfl_fme0/event=0x06,evtype=0x02,portid=0xff/ <command> + or + $# perf stat -a -e dfl_fme0/config=0xff2006/ <command> + +Another example, fab_port_mmio_read monitors mmio read of a specific port. So +its configuration template is "event=0x06,evtype=0x01,portid=?". The portid +should be explicitly set. + +Its usage of perf:: + + $# perf stat -a -e dfl_fme0/fab_port_mmio_read,portid=0x0/ <command> + or + $# perf stat -a -e dfl_fme0/event=0x06,evtype=0x02,portid=0x0/ <command> + or + $# perf stat -a -e dfl_fme0/config=0x2006/ <command> + +Please note for fabric counters, overall perf events (fab_*) and port perf +events (fab_port_*) actually share one set of counters in hardware, so it can't +monitor both at the same time. If this set of counters is configured to monitor +overall data, then per port perf data is not supported. See below example:: + + $# perf stat -e dfl_fme0/fab_mmio_read/,dfl_fme0/fab_port_mmio_write,\ + portid=0/ sleep 1 + + Performance counter stats for 'system wide': + + 3 dfl_fme0/fab_mmio_read/ + <not supported> dfl_fme0/fab_port_mmio_write,portid=0x0/ + + 1.001750904 seconds time elapsed + +The driver also provides a "cpumask" sysfs attribute, which contains only one +CPU id used to access these perf events. Counting on multiple CPU is not allowed +since they are system-wide counters on FPGA device. + +The current driver does not support sampling. So "perf record" is unsupported. + + +Interrupt support +================= +Some FME and AFU private features are able to generate interrupts. As mentioned +above, users could call ioctl (DFL_FPGA_*_GET_IRQ_NUM) to know whether or how +many interrupts are supported for this private feature. Drivers also implement +an eventfd based interrupt handling mechanism for users to get notified when +interrupt happens. Users could set eventfds to driver via +ioctl (DFL_FPGA_*_SET_IRQ), and then poll/select on these eventfds waiting for +notification. +In Current DFL, 3 sub features (Port error, FME global error and AFU interrupt) +support interrupts. + + Add new FIUs support ==================== It's possible that developers made some new function blocks (FIUs) under this @@ -398,6 +502,64 @@ Developer only needs to provide a sub feature driver with matched feature id. FME Partial Reconfiguration Sub Feature driver (see drivers/fpga/dfl-fme-pr.c) could be a reference. +Please refer to below link to existing feature id table and guide for new feature +ids application. +https://github.com/OPAE/dfl-feature-id + + +Location of DFLs on a PCI Device +================================ +The original method for finding a DFL on a PCI device assumed the start of the +first DFL to offset 0 of bar 0. If the first node of the DFL is an FME, +then further DFLs in the port(s) are specified in FME header registers. +Alternatively, a PCIe vendor specific capability structure can be used to +specify the location of all the DFLs on the device, providing flexibility +for the type of starting node in the DFL. Intel has reserved the +VSEC ID of 0x43 for this purpose. The vendor specific +data begins with a 4 byte vendor specific register for the number of DFLs followed 4 byte +Offset/BIR vendor specific registers for each DFL. Bits 2:0 of Offset/BIR register +indicates the BAR, and bits 31:3 form the 8 byte aligned offset where bits 2:0 are +zero. +:: + + +----------------------------+ + |31 Number of DFLS 0| + +----------------------------+ + |31 Offset 3|2 BIR 0| + +----------------------------+ + . . . + +----------------------------+ + |31 Offset 3|2 BIR 0| + +----------------------------+ + +Being able to specify more than one DFL per BAR has been considered, but it +was determined the use case did not provide value. Specifying a single DFL +per BAR simplifies the implementation and allows for extra error checking. + + +Userspace driver support for DFL devices +======================================== +The purpose of an FPGA is to be reprogrammed with newly developed hardware +components. New hardware can instantiate a new private feature in the DFL, and +then present a DFL device in the system. In some cases users may need a +userspace driver for the DFL device: + +* Users may need to run some diagnostic test for their hardware. +* Users may prototype the kernel driver in user space. +* Some hardware is designed for specific purposes and does not fit into one of + the standard kernel subsystems. + +This requires direct access to MMIO space and interrupt handling from +userspace. The uio_dfl module exposes the UIO device interfaces for this +purpose. + +Currently the uio_dfl driver only supports the Ether Group sub feature, which +has no irq in hardware. So the interrupt handling is not added in this driver. + +UIO_DFL should be selected to enable the uio_dfl module driver. To support a +new DFL feature via UIO direct access, its feature id should be added to the +driver's id_table. + Open discussion =============== |