IB/hfi1: Rework fault injection machinery

The packet fault injection code present in the HFI1 driver had some
issues which not only fragment the code but also created user
confusion. Furthermore, it suffered from the following issues:

  1. The fault_packet method only worked for received packets. This
     meant that the only fault injection mode available for sent
     packets is fault_opcode, which did not allow for random packet
     drops on all egressing packets.
  2. The mask available for the fault_opcode mode did not really work
     due to the fact that the opcode values are not bits in a bitmask but
     rather sequential integer values. Creating a opcode/mask pair that
     would successfully capture a set of packets was nearly impossible.
  3. The code was fragmented and used too many debugfs entries to
     operate and control. This was confusing to users.
  4. It did not allow filtering fault injection on a per direction basis -
     egress vs. ingress.

In order to improve or fix the above issues, the following changes have
been made:

   1. The fault injection methods have been combined into a single fault
      injection facility. As such, the fault injection has been plugged
      into both the send and receive code paths. Regardless of method used
      the fault injection will operate on both egress and ingress packets.
   2. The type of fault injection - by packet or by opcode - is now controlled
      by changing the boolean value of the file "opcode_mode". When the value
      is set to True, fault injection is done by opcode. Otherwise, by
      packet.
   2. The masking ability has been removed in favor of a bitmap that holds
      opcodes of interest (one bit per opcode, a total of 256 bits). This
      works in tandem with the "opcode_mode" value. When the value of
      "opcode_mode" is False, this bitmap is ignored. When the value is
      True, the bitmap lists all opcodes to be considered for fault injection.
      By default, the bitmap is empty. When the user wants to filter by opcode,
      the user sets the corresponding bit in the bitmap by echo'ing the bit
      position into the 'opcodes' file. This gets around the issue that the set
      of opcodes does not lend itself to effective masks and allow for extremely
      fine-grained filtering by opcode.
   4. fault_packet and fault_opcode methods have been combined. Hence, there
      is only one debugfs directory controlling the entire operation of the
      fault injection machinery. This reduces the number of debugfs entries
      and provides a more unified user experience.
   5. A new control files - "direction" - is provided to allow the user to
      control the direction of packets, which are subject to fault injection.
   6. A new control file - "skip_usec" - is added that would allow the user
      to specify a "timeout" during which no fault injection will occur.

In addition, the following bug fixes have been applied:

   1. The fault injection code has been split into its own header and source
      files. This was done to better organize the code and support conditional
      compilation without littering the code with #ifdef's.
   2. The method by which the TX PIO packets were being marked for drop
      conflicted with the way send contexts were being setup. As a result,
      the send context was repeatedly being reset.
   3. The fault injection only makes sense when the user can control it
      through the debugfs entries. However, a kernel configuration can
      enable fault injection but keep fault injection debugfs entries
      disabled. Therefore, it makes sense that the HFI fault injection
      code depends on both.
   4. Error suppression did not take into account the method by which PIO
      packets were being dropped. Therefore, even with error suppression
      turned on, errors would still be displayed to the screen. A larger
      enough packet drop percentage would case the kernel to crash because
      the driver would be stuck printing errors.

Reviewed-by: Dennis Dalessandro <dennis.dalessandro@intel.com>
Reviewed-by: Don Hiatt <don.hiatt@intel.com>
Reviewed-by: Mike Marciniszyn <mike.marciniszyn@intel.com>
Signed-off-by: Mitko Haralanov <mitko.haralanov@intel.com>
Signed-off-by: Dennis Dalessandro <dennis.dalessandro@intel.com>
Signed-off-by: Doug Ledford <dledford@redhat.com>

1 files changed, 11 insertions, 9 deletions

diff --git a/drivers/infiniband/hw/hfi1/driver.c b/drivers/infiniband/hw/hfi1/driver.c
index bd837a048bf4..e5a57ebd8da4 100644
--- a/drivers/infiniband/hw/hfi1/driver.c
+++ b/drivers/infiniband/hw/hfi1/driver.c
@@ -1,5 +1,5 @@
 /*
- * Copyright(c) 2015-2017 Intel Corporation.
+ * Copyright(c) 2015-2018 Intel Corporation.
  *
  * This file is provided under a dual BSD/GPLv2 license.  When using or
  * redistributing this file, you may do so under either license.
@@ -61,6 +61,7 @@
 #include "sdma.h"
 #include "debugfs.h"
 #include "vnic.h"
+#include "fault.h"
 
 #undef pr_fmt
 #define pr_fmt(fmt) DRIVER_NAME ": " fmt
@@ -1565,10 +1566,10 @@ void handle_eflags(struct hfi1_packet *packet)
  */
 int process_receive_ib(struct hfi1_packet *packet)
 {
-	if (unlikely(hfi1_dbg_fault_packet(packet)))
+	if (hfi1_setup_9B_packet(packet))
 		return RHF_RCV_CONTINUE;
 
-	if (hfi1_setup_9B_packet(packet))
+	if (unlikely(hfi1_dbg_should_fault_rx(packet)))
 		return RHF_RCV_CONTINUE;
 
 	trace_hfi1_rcvhdr(packet);
@@ -1642,7 +1643,8 @@ int process_receive_error(struct hfi1_packet *packet)
 	/* KHdrHCRCErr -- KDETH packet with a bad HCRC */
 	if (unlikely(
 		 hfi1_dbg_fault_suppress_err(&packet->rcd->dd->verbs_dev) &&
-		 rhf_rcv_type_err(packet->rhf) == 3))
+		 (rhf_rcv_type_err(packet->rhf) == RHF_RCV_TYPE_ERROR ||
+		  packet->rhf & RHF_DC_ERR)))
 		return RHF_RCV_CONTINUE;
 
 	hfi1_setup_ib_header(packet);
@@ -1657,10 +1659,10 @@ int process_receive_error(struct hfi1_packet *packet)
 
 int kdeth_process_expected(struct hfi1_packet *packet)
 {
-	if (unlikely(hfi1_dbg_fault_packet(packet)))
+	hfi1_setup_9B_packet(packet);
+	if (unlikely(hfi1_dbg_should_fault_rx(packet)))
 		return RHF_RCV_CONTINUE;
 
-	hfi1_setup_ib_header(packet);
 	if (unlikely(rhf_err_flags(packet->rhf)))
 		handle_eflags(packet);
 
@@ -1671,11 +1673,11 @@ int kdeth_process_expected(struct hfi1_packet *packet)
 
 int kdeth_process_eager(struct hfi1_packet *packet)
 {
-	hfi1_setup_ib_header(packet);
+	hfi1_setup_9B_packet(packet);
+	if (unlikely(hfi1_dbg_should_fault_rx(packet)))
+		return RHF_RCV_CONTINUE;
 	if (unlikely(rhf_err_flags(packet->rhf)))
 		handle_eflags(packet);
-	if (unlikely(hfi1_dbg_fault_packet(packet)))
-		return RHF_RCV_CONTINUE;
 
 	dd_dev_err(packet->rcd->dd,
 		   "Unhandled eager packet received. Dropping.\n");