From affa48de8417ccdde467b075577f6e5154ff9943 Mon Sep 17 00:00:00 2001 From: Ashutosh Dixit Date: Wed, 3 Feb 2016 14:33:06 -0800 Subject: staging/rdma/hfi1: Add support for enabling/disabling PCIe ASPM hfi1 HW has a high PCIe ASPM L1 exit latency and also advertises an acceptable latency less than actual ASPM latencies. Additional mechanisms than those provided by BIOS/OS are therefore required to enable/disable ASPM for hfi1 to provide acceptable power/performance trade offs. This patch adds this support. By means of a module parameter ASPM can be either (a) always enabled (power save mode) (b) always disabled (performance mode) (c) enabled/disabled dynamically. The dynamic mode implements two heuristics to alleviate possible problems with high ASPM L1 exit latency. ASPM is normally enabled but is disabled if (a) there are any active user space PSM contexts, or (b) for verbs, ASPM is disabled as interrupt activity for a context starts to increase. A few more points about the verbs implementation. In order to reduce lock/cache contention between multiple verbs contexts, some processing is done at the context layer before contending for device layer locks. ASPM is disabled when two interrupts for a context happen within 1 millisec. A timer is scheduled which will re-enable ASPM after 1 second should the interrupt activity cease. Normally, every interrupt, or interrupt-pair should push the timer out further. However, since this might increase the processing load per interrupt, pushing the timer out is postponed for half a second. If after half a second we get two interrupts within 1 millisec the timer is pushed out by another second. Finally, the kernel ASPM API is not used in this patch. This is because this patch does several non-standard things as SW workarounds for HW issues. As mentioned above, it enables ASPM even when advertised actual latencies are greater than acceptable latencies. Also, whereas the kernel API only allows drivers to disable ASPM from driver probe, this patch enables/disables ASPM directly from interrupt context. Due to these reasons the kernel ASPM API was not used. Reviewed-by: Mike Marciniszyn Reviewed-by: Dean Luick Reviewed-by: Ira Weiny Signed-off-by: Ashutosh Dixit Signed-off-by: Doug Ledford --- drivers/staging/rdma/hfi1/init.c | 7 +++++++ 1 file changed, 7 insertions(+) (limited to 'drivers/staging/rdma/hfi1/init.c') diff --git a/drivers/staging/rdma/hfi1/init.c b/drivers/staging/rdma/hfi1/init.c index fc3d40a58cf3..ba52ee38c1f2 100644 --- a/drivers/staging/rdma/hfi1/init.c +++ b/drivers/staging/rdma/hfi1/init.c @@ -66,6 +66,7 @@ #include "sdma.h" #include "debugfs.h" #include "verbs.h" +#include "aspm.h" #undef pr_fmt #define pr_fmt(fmt) DRIVER_NAME ": " fmt @@ -190,6 +191,12 @@ int hfi1_create_ctxts(struct hfi1_devdata *dd) } } + /* + * Initialize aspm, to be done after gen3 transition and setting up + * contexts and before enabling interrupts + */ + aspm_init(dd); + return 0; nomem: ret = -ENOMEM; -- cgit v1.2.3-59-g8ed1b