diff options
Diffstat (limited to 'Documentation/admin-guide/ras.rst')
| -rw-r--r-- | Documentation/admin-guide/ras.rst | 1219 |
1 files changed, 0 insertions, 1219 deletions
diff --git a/Documentation/admin-guide/ras.rst b/Documentation/admin-guide/ras.rst deleted file mode 100644 index 7b481b2a368e..000000000000 --- a/Documentation/admin-guide/ras.rst +++ /dev/null @@ -1,1219 +0,0 @@ -.. include:: <isonum.txt> - -============================================ -Reliability, Availability and Serviceability -============================================ - -RAS concepts -************ - -Reliability, Availability and Serviceability (RAS) is a concept used on -servers meant to measure their robustness. - -Reliability - is the probability that a system will produce correct outputs. - - * Generally measured as Mean Time Between Failures (MTBF) - * Enhanced by features that help to avoid, detect and repair hardware faults - -Availability - is the probability that a system is operational at a given time - - * Generally measured as a percentage of downtime per a period of time - * Often uses mechanisms to detect and correct hardware faults in - runtime; - -Serviceability (or maintainability) - is the simplicity and speed with which a system can be repaired or - maintained - - * Generally measured on Mean Time Between Repair (MTBR) - -Improving RAS -------------- - -In order to reduce systems downtime, a system should be capable of detecting -hardware errors, and, when possible correcting them in runtime. It should -also provide mechanisms to detect hardware degradation, in order to warn -the system administrator to take the action of replacing a component before -it causes data loss or system downtime. - -Among the monitoring measures, the most usual ones include: - -* CPU – detect errors at instruction execution and at L1/L2/L3 caches; -* Memory – add error correction logic (ECC) to detect and correct errors; -* I/O – add CRC checksums for transferred data; -* Storage – RAID, journal file systems, checksums, - Self-Monitoring, Analysis and Reporting Technology (SMART). - -By monitoring the number of occurrences of error detections, it is possible -to identify if the probability of hardware errors is increasing, and, on such -case, do a preventive maintenance to replace a degraded component while -those errors are correctable. - -Types of errors ---------------- - -Most mechanisms used on modern systems use technologies like Hamming -Codes that allow error correction when the number of errors on a bit packet -is below a threshold. If the number of errors is above, those mechanisms -can indicate with a high degree of confidence that an error happened, but -they can't correct. - -Also, sometimes an error occur on a component that it is not used. For -example, a part of the memory that it is not currently allocated. - -That defines some categories of errors: - -* **Correctable Error (CE)** - the error detection mechanism detected and - corrected the error. Such errors are usually not fatal, although some - Kernel mechanisms allow the system administrator to consider them as fatal. - -* **Uncorrected Error (UE)** - the amount of errors happened above the error - correction threshold, and the system was unable to auto-correct. - -* **Fatal Error** - when an UE error happens on a critical component of the - system (for example, a piece of the Kernel got corrupted by an UE), the - only reliable way to avoid data corruption is to hang or reboot the machine. - -* **Non-fatal Error** - when an UE error happens on an unused component, - like a CPU in power down state or an unused memory bank, the system may - still run, eventually replacing the affected hardware by a hot spare, - if available. - - Also, when an error happens on a userspace process, it is also possible to - kill such process and let userspace restart it. - -The mechanism for handling non-fatal errors is usually complex and may -require the help of some userspace application, in order to apply the -policy desired by the system administrator. - -Identifying a bad hardware component ------------------------------------- - -Just detecting a hardware flaw is usually not enough, as the system needs -to pinpoint to the minimal replaceable unit (MRU) that should be exchanged -to make the hardware reliable again. - -So, it requires not only error logging facilities, but also mechanisms that -will translate the error message to the silkscreen or component label for -the MRU. - -Typically, it is very complex for memory, as modern CPUs interlace memory -from different memory modules, in order to provide a better performance. The -DMI BIOS usually have a list of memory module labels, with can be obtained -using the ``dmidecode`` tool. For example, on a desktop machine, it shows:: - - Memory Device - Total Width: 64 bits - Data Width: 64 bits - Size: 16384 MB - Form Factor: SODIMM - Set: None - Locator: ChannelA-DIMM0 - Bank Locator: BANK 0 - Type: DDR4 - Type Detail: Synchronous - Speed: 2133 MHz - Rank: 2 - Configured Clock Speed: 2133 MHz - -On the above example, a DDR4 SO-DIMM memory module is located at the -system's memory labeled as "BANK 0", as given by the *bank locator* field. -Please notice that, on such system, the *total width* is equal to the -*data width*. It means that such memory module doesn't have error -detection/correction mechanisms. - -Unfortunately, not all systems use the same field to specify the memory -bank. On this example, from an older server, ``dmidecode`` shows:: - - Memory Device - Array Handle: 0x1000 - Error Information Handle: Not Provided - Total Width: 72 bits - Data Width: 64 bits - Size: 8192 MB - Form Factor: DIMM - Set: 1 - Locator: DIMM_A1 - Bank Locator: Not Specified - Type: DDR3 - Type Detail: Synchronous Registered (Buffered) - Speed: 1600 MHz - Rank: 2 - Configured Clock Speed: 1600 MHz - -There, the DDR3 RDIMM memory module is located at the system's memory labeled -as "DIMM_A1", as given by the *locator* field. Please notice that this -memory module has 64 bits of *data width* and 72 bits of *total width*. So, -it has 8 extra bits to be used by error detection and correction mechanisms. -Such kind of memory is called Error-correcting code memory (ECC memory). - -To make things even worse, it is not uncommon that systems with different -labels on their system's board to use exactly the same BIOS, meaning that -the labels provided by the BIOS won't match the real ones. - -ECC memory ----------- - -As mentioned in the previous section, ECC memory has extra bits to be -used for error correction. In the above example, a memory module has -64 bits of *data width*, and 72 bits of *total width*. The extra 8 -bits which are used for the error detection and correction mechanisms -are referred to as the *syndrome*\ [#f1]_\ [#f2]_. - -So, when the cpu requests the memory controller to write a word with -*data width*, the memory controller calculates the *syndrome* in real time, -using Hamming code, or some other error correction code, like SECDED+, -producing a code with *total width* size. Such code is then written -on the memory modules. - -At read, the *total width* bits code is converted back, using the same -ECC code used on write, producing a word with *data width* and a *syndrome*. -The word with *data width* is sent to the CPU, even when errors happen. - -The memory controller also looks at the *syndrome* in order to check if -there was an error, and if the ECC code was able to fix such error. -If the error was corrected, a Corrected Error (CE) happened. If not, an -Uncorrected Error (UE) happened. - -The information about the CE/UE errors is stored on some special registers -at the memory controller and can be accessed by reading such registers, -either by BIOS, by some special CPUs or by Linux EDAC driver. On x86 64 -bit CPUs, such errors can also be retrieved via the Machine Check -Architecture (MCA)\ [#f3]_. - -.. [#f1] Please notice that several memory controllers allow operation on a - mode called "Lock-Step", where it groups two memory modules together, - doing 128-bit reads/writes. That gives 16 bits for error correction, with - significantly improves the error correction mechanism, at the expense - that, when an error happens, there's no way to know what memory module is - to blame. So, it has to blame both memory modules. - -.. [#f2] Some memory controllers also allow using memory in mirror mode. - On such mode, the same data is written to two memory modules. At read, - the system checks both memory modules, in order to check if both provide - identical data. On such configuration, when an error happens, there's no - way to know what memory module is to blame. So, it has to blame both - memory modules (or 4 memory modules, if the system is also on Lock-step - mode). - -.. [#f3] For more details about the Machine Check Architecture (MCA), - please read Documentation/x86/x86_64/machinecheck.rst at the Kernel tree. - -EDAC - Error Detection And Correction -************************************* - -.. note:: - - "bluesmoke" was the name for this device driver subsystem when it - was "out-of-tree" and maintained at http://bluesmoke.sourceforge.net. - That site is mostly archaic now and can be used only for historical - purposes. - - When the subsystem was pushed upstream for the first time, on - Kernel 2.6.16, it was renamed to ``EDAC``. - -Purpose -------- - -The ``edac`` kernel module's goal is to detect and report hardware errors -that occur within the computer system running under linux. - -Memory ------- - -Memory Correctable Errors (CE) and Uncorrectable Errors (UE) are the -primary errors being harvested. These types of errors are harvested by -the ``edac_mc`` device. - -Detecting CE events, then harvesting those events and reporting them, -**can** but must not necessarily be a predictor of future UE events. With -CE events only, the system can and will continue to operate as no data -has been damaged yet. - -However, preventive maintenance and proactive part replacement of memory -modules exhibiting CEs can reduce the likelihood of the dreaded UE events -and system panics. - -Other hardware elements ------------------------ - -A new feature for EDAC, the ``edac_device`` class of device, was added in -the 2.6.23 version of the kernel. - -This new device type allows for non-memory type of ECC hardware detectors -to have their states harvested and presented to userspace via the sysfs -interface. - -Some architectures have ECC detectors for L1, L2 and L3 caches, -along with DMA engines, fabric switches, main data path switches, -interconnections, and various other hardware data paths. If the hardware -reports it, then a edac_device device probably can be constructed to -harvest and present that to userspace. - - -PCI bus scanning ----------------- - -In addition, PCI devices are scanned for PCI Bus Parity and SERR Errors -in order to determine if errors are occurring during data transfers. - -The presence of PCI Parity errors must be examined with a grain of salt. -There are several add-in adapters that do **not** follow the PCI specification -with regards to Parity generation and reporting. The specification says -the vendor should tie the parity status bits to 0 if they do not intend -to generate parity. Some vendors do not do this, and thus the parity bit -can "float" giving false positives. - -There is a PCI device attribute located in sysfs that is checked by -the EDAC PCI scanning code. If that attribute is set, PCI parity/error -scanning is skipped for that device. The attribute is:: - - broken_parity_status - -and is located in ``/sys/devices/pci<XXX>/0000:XX:YY.Z`` directories for -PCI devices. - - -Versioning ----------- - -EDAC is composed of a "core" module (``edac_core.ko``) and several Memory -Controller (MC) driver modules. On a given system, the CORE is loaded -and one MC driver will be loaded. Both the CORE and the MC driver (or -``edac_device`` driver) have individual versions that reflect current -release level of their respective modules. - -Thus, to "report" on what version a system is running, one must report -both the CORE's and the MC driver's versions. - - -Loading -------- - -If ``edac`` was statically linked with the kernel then no loading -is necessary. If ``edac`` was built as modules then simply modprobe -the ``edac`` pieces that you need. You should be able to modprobe -hardware-specific modules and have the dependencies load the necessary -core modules. - -Example:: - - $ modprobe amd76x_edac - -loads both the ``amd76x_edac.ko`` memory controller module and the -``edac_mc.ko`` core module. - - -Sysfs interface ---------------- - -EDAC presents a ``sysfs`` interface for control and reporting purposes. It -lives in the /sys/devices/system/edac directory. - -Within this directory there currently reside 2 components: - - ======= ============================== - mc memory controller(s) system - pci PCI control and status system - ======= ============================== - - - -Memory Controller (mc) Model ----------------------------- - -Each ``mc`` device controls a set of memory modules [#f4]_. These modules -are laid out in a Chip-Select Row (``csrowX``) and Channel table (``chX``). -There can be multiple csrows and multiple channels. - -.. [#f4] Nowadays, the term DIMM (Dual In-line Memory Module) is widely - used to refer to a memory module, although there are other memory - packaging alternatives, like SO-DIMM, SIMM, etc. The UEFI - specification (Version 2.7) defines a memory module in the Common - Platform Error Record (CPER) section to be an SMBIOS Memory Device - (Type 17). Along this document, and inside the EDAC subsystem, the term - "dimm" is used for all memory modules, even when they use a - different kind of packaging. - -Memory controllers allow for several csrows, with 8 csrows being a -typical value. Yet, the actual number of csrows depends on the layout of -a given motherboard, memory controller and memory module characteristics. - -Dual channels allow for dual data length (e. g. 128 bits, on 64 bit systems) -data transfers to/from the CPU from/to memory. Some newer chipsets allow -for more than 2 channels, like Fully Buffered DIMMs (FB-DIMMs) memory -controllers. The following example will assume 2 channels: - - +------------+-----------------------+ - | CS Rows | Channels | - +------------+-----------+-----------+ - | | ``ch0`` | ``ch1`` | - +============+===========+===========+ - | |**DIMM_A0**|**DIMM_B0**| - +------------+-----------+-----------+ - | ``csrow0`` | rank0 | rank0 | - +------------+-----------+-----------+ - | ``csrow1`` | rank1 | rank1 | - +------------+-----------+-----------+ - | |**DIMM_A1**|**DIMM_B1**| - +------------+-----------+-----------+ - | ``csrow2`` | rank0 | rank0 | - +------------+-----------+-----------+ - | ``csrow3`` | rank1 | rank1 | - +------------+-----------+-----------+ - -In the above example, there are 4 physical slots on the motherboard -for memory DIMMs: - - +---------+---------+ - | DIMM_A0 | DIMM_B0 | - +---------+---------+ - | DIMM_A1 | DIMM_B1 | - +---------+---------+ - -Labels for these slots are usually silk-screened on the motherboard. -Slots labeled ``A`` are channel 0 in this example. Slots labeled ``B`` are -channel 1. Notice that there are two csrows possible on a physical DIMM. -These csrows are allocated their csrow assignment based on the slot into -which the memory DIMM is placed. Thus, when 1 DIMM is placed in each -Channel, the csrows cross both DIMMs. - -Memory DIMMs come single or dual "ranked". A rank is a populated csrow. -In the example above 2 dual ranked DIMMs are similarly placed. Thus, -both csrow0 and csrow1 are populated. On the other hand, when 2 single -ranked DIMMs are placed in slots DIMM_A0 and DIMM_B0, then they will -have just one csrow (csrow0) and csrow1 will be empty. The pattern -repeats itself for csrow2 and csrow3. Also note that some memory -controllers don't have any logic to identify the memory module, see -``rankX`` directories below. - -The representation of the above is reflected in the directory -tree in EDAC's sysfs interface. Starting in directory -``/sys/devices/system/edac/mc``, each memory controller will be -represented by its own ``mcX`` directory, where ``X`` is the -index of the MC:: - - ..../edac/mc/ - | - |->mc0 - |->mc1 - |->mc2 - .... - -Under each ``mcX`` directory each ``csrowX`` is again represented by a -``csrowX``, where ``X`` is the csrow index:: - - .../mc/mc0/ - | - |->csrow0 - |->csrow2 - |->csrow3 - .... - -Notice that there is no csrow1, which indicates that csrow0 is composed -of a single ranked DIMMs. This should also apply in both Channels, in -order to have dual-channel mode be operational. Since both csrow2 and -csrow3 are populated, this indicates a dual ranked set of DIMMs for -channels 0 and 1. - -Within each of the ``mcX`` and ``csrowX`` directories are several EDAC -control and attribute files. - -``mcX`` directories -------------------- - -In ``mcX`` directories are EDAC control and attribute files for -this ``X`` instance of the memory controllers. - -For a description of the sysfs API, please see: - - Documentation/ABI/testing/sysfs-devices-edac - - -``dimmX`` or ``rankX`` directories ----------------------------------- - -The recommended way to use the EDAC subsystem is to look at the information -provided by the ``dimmX`` or ``rankX`` directories [#f5]_. - -A typical EDAC system has the following structure under -``/sys/devices/system/edac/``\ [#f6]_:: - - /sys/devices/system/edac/ - ├── mc - │ ├── mc0 - │ │ ├── ce_count - │ │ ├── ce_noinfo_count - │ │ ├── dimm0 - │ │ │ ├── dimm_ce_count - │ │ │ ├── dimm_dev_type - │ │ │ ├── dimm_edac_mode - │ │ │ ├── dimm_label - │ │ │ ├── dimm_location - │ │ │ ├── dimm_mem_type - │ │ │ ├── dimm_ue_count - │ │ │ ├── size - │ │ │ └── uevent - │ │ ├── max_location - │ │ ├── mc_name - │ │ ├── reset_counters - │ │ ├── seconds_since_reset - │ │ ├── size_mb - │ │ ├── ue_count - │ │ ├── ue_noinfo_count - │ │ └── uevent - │ ├── mc1 - │ │ ├── ce_count - │ │ ├── ce_noinfo_count - │ │ ├── dimm0 - │ │ │ ├── dimm_ce_count - │ │ │ ├── dimm_dev_type - │ │ │ ├── dimm_edac_mode - │ │ │ ├── dimm_label - │ │ │ ├── dimm_location - │ │ │ ├── dimm_mem_type - │ │ │ ├── dimm_ue_count - │ │ │ ├── size - │ │ │ └── uevent - │ │ ├── max_location - │ │ ├── mc_name - │ │ ├── reset_counters - │ │ ├── seconds_since_reset - │ │ ├── size_mb - │ │ ├── ue_count - │ │ ├── ue_noinfo_count - │ │ └── uevent - │ └── uevent - └── uevent - -In the ``dimmX`` directories are EDAC control and attribute files for -this ``X`` memory module: - -- ``size`` - Total memory managed by this csrow attribute file - - This attribute file displays, in count of megabytes, the memory - that this csrow contains. - -- ``dimm_ue_count`` - Uncorrectable Errors count attribute file - - This attribute file displays the total count of uncorrectable - errors that have occurred on this DIMM. If panic_on_ue is set - this counter will not have a chance to increment, since EDAC - will panic the system. - -- ``dimm_ce_count`` - Correctable Errors count attribute file - - This attribute file displays the total count of correctable - errors that have occurred on this DIMM. This count is very - important to examine. CEs provide early indications that a - DIMM is beginning to fail. This count field should be - monitored for non-zero values and report such information - to the system administrator. - -- ``dimm_dev_type`` - Device type attribute file - - This attribute file will display what type of DRAM device is - being utilized on this DIMM. - Examples: - - - x1 - - x2 - - x4 - - x8 - -- ``dimm_edac_mode`` - EDAC Mode of operation attribute file - - This attribute file will display what type of Error detection - and correction is being utilized. - -- ``dimm_label`` - memory module label control file - - This control file allows this DIMM to have a label assigned - to it. With this label in the module, when errors occur - the output can provide the DIMM label in the system log. - This becomes vital for panic events to isolate the - cause of the UE event. - - DIMM Labels must be assigned after booting, with information - that correctly identifies the physical slot with its - silk screen label. This information is currently very - motherboard specific and determination of this information - must occur in userland at this time. - -- ``dimm_location`` - location of the memory module - - The location can have up to 3 levels, and describe how the - memory controller identifies the location of a memory module. - Depending on the type of memory and memory controller, it - can be: - - - *csrow* and *channel* - used when the memory controller - doesn't identify a single DIMM - e. g. in ``rankX`` dir; - - *branch*, *channel*, *slot* - typically used on FB-DIMM memory - controllers; - - *channel*, *slot* - used on Nehalem and newer Intel drivers. - -- ``dimm_mem_type`` - Memory Type attribute file - - This attribute file will display what type of memory is currently - on this csrow. Normally, either buffered or unbuffered memory. - Examples: - - - Registered-DDR - - Unbuffered-DDR - -.. [#f5] On some systems, the memory controller doesn't have any logic - to identify the memory module. On such systems, the directory is called ``rankX`` and works on a similar way as the ``csrowX`` directories. - On modern Intel memory controllers, the memory controller identifies the - memory modules directly. On such systems, the directory is called ``dimmX``. - -.. [#f6] There are also some ``power`` directories and ``subsystem`` - symlinks inside the sysfs mapping that are automatically created by - the sysfs subsystem. Currently, they serve no purpose. - -``csrowX`` directories ----------------------- - -When CONFIG_EDAC_LEGACY_SYSFS is enabled, sysfs will contain the ``csrowX`` -directories. As this API doesn't work properly for Rambus, FB-DIMMs and -modern Intel Memory Controllers, this is being deprecated in favor of -``dimmX`` directories. - -In the ``csrowX`` directories are EDAC control and attribute files for -this ``X`` instance of csrow: - - -- ``ue_count`` - Total Uncorrectable Errors count attribute file - - This attribute file displays the total count of uncorrectable - errors that have occurred on this csrow. If panic_on_ue is set - this counter will not have a chance to increment, since EDAC - will panic the system. - - -- ``ce_count`` - Total Correctable Errors count attribute file - - This attribute file displays the total count of correctable - errors that have occurred on this csrow. This count is very - important to examine. CEs provide early indications that a - DIMM is beginning to fail. This count field should be - monitored for non-zero values and report such information - to the system administrator. - - -- ``size_mb`` - Total memory managed by this csrow attribute file - - This attribute file displays, in count of megabytes, the memory - that this csrow contains. - - -- ``mem_type`` - Memory Type attribute file - - This attribute file will display what type of memory is currently - on this csrow. Normally, either buffered or unbuffered memory. - Examples: - - - Registered-DDR - - Unbuffered-DDR - - -- ``edac_mode`` - EDAC Mode of operation attribute file - - This attribute file will display what type of Error detection - and correction is being utilized. - - -- ``dev_type`` - Device type attribute file - - This attribute file will display what type of DRAM device is - being utilized on this DIMM. - Examples: - - - x1 - - x2 - - x4 - - x8 - - -- ``ch0_ce_count`` - Channel 0 CE Count attribute file - - This attribute file will display the count of CEs on this - DIMM located in channel 0. - - -- ``ch0_ue_count`` - Channel 0 UE Count attribute file - - This attribute file will display the count of UEs on this - DIMM located in channel 0. - - -- ``ch0_dimm_label`` - Channel 0 DIMM Label control file - - - This control file allows this DIMM to have a label assigned - to it. With this label in the module, when errors occur - the output can provide the DIMM label in the system log. - This becomes vital for panic events to isolate the - cause of the UE event. - - DIMM Labels must be assigned after booting, with information - that correctly identifies the physical slot with its - silk screen label. This information is currently very - motherboard specific and determination of this information - must occur in userland at this time. - - -- ``ch1_ce_count`` - Channel 1 CE Count attribute file - - - This attribute file will display the count of CEs on this - DIMM located in channel 1. - - -- ``ch1_ue_count`` - Channel 1 UE Count attribute file - - - This attribute file will display the count of UEs on this - DIMM located in channel 0. - - -- ``ch1_dimm_label`` - Channel 1 DIMM Label control file - - This control file allows this DIMM to have a label assigned - to it. With this label in the module, when errors occur - the output can provide the DIMM label in the system log. - This becomes vital for panic events to isolate the - cause of the UE event. - - DIMM Labels must be assigned after booting, with information - that correctly identifies the physical slot with its - silk screen label. This information is currently very - motherboard specific and determination of this information - must occur in userland at this time. - - -System Logging --------------- - -If logging for UEs and CEs is enabled, then system logs will contain -information indicating that errors have been detected:: - - EDAC MC0: CE page 0x283, offset 0xce0, grain 8, syndrome 0x6ec3, row 0, channel 1 "DIMM_B1": amd76x_edac - EDAC MC0: CE page 0x1e5, offset 0xfb0, grain 8, syndrome 0xb741, row 0, channel 1 "DIMM_B1": amd76x_edac - - -The structure of the message is: - - +---------------------------------------+-------------+ - | Content | Example | - +=======================================+=============+ - | The memory controller | MC0 | - +---------------------------------------+-------------+ - | Error type | CE | - +---------------------------------------+-------------+ - | Memory page | 0x283 | - +---------------------------------------+-------------+ - | Offset in the page | 0xce0 | - +---------------------------------------+-------------+ - | The byte granularity | grain 8 | - | or resolution of the error | | - +---------------------------------------+-------------+ - | The error syndrome | 0xb741 | - +---------------------------------------+-------------+ - | Memory row | row 0 | - +---------------------------------------+-------------+ - | Memory channel | channel 1 | - +---------------------------------------+-------------+ - | DIMM label, if set prior | DIMM B1 | - +---------------------------------------+-------------+ - | And then an optional, driver-specific | | - | message that may have additional | | - | information. | | - +---------------------------------------+-------------+ - -Both UEs and CEs with no info will lack all but memory controller, error -type, a notice of "no info" and then an optional, driver-specific error -message. - - -PCI Bus Parity Detection ------------------------- - -On Header Type 00 devices, the primary status is looked at for any -parity error regardless of whether parity is enabled on the device or -not. (The spec indicates parity is generated in some cases). On Header -Type 01 bridges, the secondary status register is also looked at to see -if parity occurred on the bus on the other side of the bridge. - - -Sysfs configuration -------------------- - -Under ``/sys/devices/system/edac/pci`` are control and attribute files as -follows: - - -- ``check_pci_parity`` - Enable/Disable PCI Parity checking control file - - This control file enables or disables the PCI Bus Parity scanning - operation. Writing a 1 to this file enables the scanning. Writing - a 0 to this file disables the scanning. - - Enable:: - - echo "1" >/sys/devices/system/edac/pci/check_pci_parity - - Disable:: - - echo "0" >/sys/devices/system/edac/pci/check_pci_parity - - -- ``pci_parity_count`` - Parity Count - - This attribute file will display the number of parity errors that - have been detected. - - -Module parameters ------------------ - -- ``edac_mc_panic_on_ue`` - Panic on UE control file - - An uncorrectable error will cause a machine panic. This is usually - desirable. It is a bad idea to continue when an uncorrectable error - occurs - it is indeterminate what was uncorrected and the operating - system context might be so mangled that continuing will lead to further - corruption. If the kernel has MCE configured, then EDAC will never - notice the UE. - - LOAD TIME:: - - module/kernel parameter: edac_mc_panic_on_ue=[0|1] - - RUN TIME:: - - echo "1" > /sys/module/edac_core/parameters/edac_mc_panic_on_ue - - -- ``edac_mc_log_ue`` - Log UE control file - - - Generate kernel messages describing uncorrectable errors. These errors - are reported through the system message log system. UE statistics - will be accumulated even when UE logging is disabled. - - LOAD TIME:: - - module/kernel parameter: edac_mc_log_ue=[0|1] - - RUN TIME:: - - echo "1" > /sys/module/edac_core/parameters/edac_mc_log_ue - - -- ``edac_mc_log_ce`` - Log CE control file - - - Generate kernel messages describing correctable errors. These - errors are reported through the system message log system. - CE statistics will be accumulated even when CE logging is disabled. - - LOAD TIME:: - - module/kernel parameter: edac_mc_log_ce=[0|1] - - RUN TIME:: - - echo "1" > /sys/module/edac_core/parameters/edac_mc_log_ce - - -- ``edac_mc_poll_msec`` - Polling period control file - - - The time period, in milliseconds, for polling for error information. - Too small a value wastes resources. Too large a value might delay - necessary handling of errors and might loose valuable information for - locating the error. 1000 milliseconds (once each second) is the current - default. Systems which require all the bandwidth they can get, may - increase this. - - LOAD TIME:: - - module/kernel parameter: edac_mc_poll_msec=[0|1] - - RUN TIME:: - - echo "1000" > /sys/module/edac_core/parameters/edac_mc_poll_msec - - -- ``panic_on_pci_parity`` - Panic on PCI PARITY Error - - - This control file enables or disables panicking when a parity - error has been detected. - - - module/kernel parameter:: - - edac_panic_on_pci_pe=[0|1] - - Enable:: - - echo "1" > /sys/module/edac_core/parameters/edac_panic_on_pci_pe - - Disable:: - - echo "0" > /sys/module/edac_core/parameters/edac_panic_on_pci_pe - - - -EDAC device type ----------------- - -In the header file, edac_pci.h, there is a series of edac_device structures -and APIs for the EDAC_DEVICE. - -User space access to an edac_device is through the sysfs interface. - -At the location ``/sys/devices/system/edac`` (sysfs) new edac_device devices -will appear. - -There is a three level tree beneath the above ``edac`` directory. For example, -the ``test_device_edac`` device (found at the http://bluesmoke.sourceforget.net -website) installs itself as:: - - /sys/devices/system/edac/test-instance - -in this directory are various controls, a symlink and one or more ``instance`` -directories. - -The standard default controls are: - - ============== ======================================================= - log_ce boolean to log CE events - log_ue boolean to log UE events - panic_on_ue boolean to ``panic`` the system if an UE is encountered - (default off, can be set true via startup script) - poll_msec time period between POLL cycles for events - ============== ======================================================= - -The test_device_edac device adds at least one of its own custom control: - - ============== ================================================== - test_bits which in the current test driver does nothing but - show how it is installed. A ported driver can - add one or more such controls and/or attributes - for specific uses. - One out-of-tree driver uses controls here to allow - for ERROR INJECTION operations to hardware - injection registers - ============== ================================================== - -The symlink points to the 'struct dev' that is registered for this edac_device. - -Instances ---------- - -One or more instance directories are present. For the ``test_device_edac`` -case: - - +----------------+ - | test-instance0 | - +----------------+ - - -In this directory there are two default counter attributes, which are totals of -counter in deeper subdirectories. - - ============== ==================================== - ce_count total of CE events of subdirectories - ue_count total of UE events of subdirectories - ============== ==================================== - -Blocks ------- - -At the lowest directory level is the ``block`` directory. There can be 0, 1 -or more blocks specified in each instance: - - +-------------+ - | test-block0 | - +-------------+ - -In this directory the default attributes are: - - ============== ================================================ - ce_count which is counter of CE events for this ``block`` - of hardware being monitored - ue_count which is counter of UE events for this ``block`` - of hardware being monitored - ============== ================================================ - - -The ``test_device_edac`` device adds 4 attributes and 1 control: - - ================== ==================================================== - test-block-bits-0 for every POLL cycle this counter - is incremented - test-block-bits-1 every 10 cycles, this counter is bumped once, - and test-block-bits-0 is set to 0 - test-block-bits-2 every 100 cycles, this counter is bumped once, - and test-block-bits-1 is set to 0 - test-block-bits-3 every 1000 cycles, this counter is bumped once, - and test-block-bits-2 is set to 0 - ================== ==================================================== - - - ================== ==================================================== - reset-counters writing ANY thing to this control will - reset all the above counters. - ================== ==================================================== - - -Use of the ``test_device_edac`` driver should enable any others to create their own -unique drivers for their hardware systems. - -The ``test_device_edac`` sample driver is located at the -http://bluesmoke.sourceforge.net project site for EDAC. - - -Usage of EDAC APIs on Nehalem and newer Intel CPUs --------------------------------------------------- - -On older Intel architectures, the memory controller was part of the North -Bridge chipset. Nehalem, Sandy Bridge, Ivy Bridge, Haswell, Sky Lake and -newer Intel architectures integrated an enhanced version of the memory -controller (MC) inside the CPUs. - -This chapter will cover the differences of the enhanced memory controllers -found on newer Intel CPUs, such as ``i7core_edac``, ``sb_edac`` and -``sbx_edac`` drivers. - -.. note:: - - The Xeon E7 processor families use a separate chip for the memory - controller, called Intel Scalable Memory Buffer. This section doesn't - apply for such families. - -1) There is one Memory Controller per Quick Patch Interconnect - (QPI). At the driver, the term "socket" means one QPI. This is - associated with a physical CPU socket. - - Each MC have 3 physical read channels, 3 physical write channels and - 3 logic channels. The driver currently sees it as just 3 channels. - Each channel can have up to 3 DIMMs. - - The minimum known unity is DIMMs. There are no information about csrows. - As EDAC API maps the minimum unity is csrows, the driver sequentially - maps channel/DIMM into different csrows. - - For example, supposing the following layout:: - - Ch0 phy rd0, wr0 (0x063f4031): 2 ranks, UDIMMs - dimm 0 1024 Mb offset: 0, bank: 8, rank: 1, row: 0x4000, col: 0x400 - dimm 1 1024 Mb offset: 4, bank: 8, rank: 1, row: 0x4000, col: 0x400 - Ch1 phy rd1, wr1 (0x063f4031): 2 ranks, UDIMMs - dimm 0 1024 Mb offset: 0, bank: 8, rank: 1, row: 0x4000, col: 0x400 - Ch2 phy rd3, wr3 (0x063f4031): 2 ranks, UDIMMs - dimm 0 1024 Mb offset: 0, bank: 8, rank: 1, row: 0x4000, col: 0x400 - - The driver will map it as:: - - csrow0: channel 0, dimm0 - csrow1: channel 0, dimm1 - csrow2: channel 1, dimm0 - csrow3: channel 2, dimm0 - - exports one DIMM per csrow. - - Each QPI is exported as a different memory controller. - -2) The MC has the ability to inject errors to test drivers. The drivers - implement this functionality via some error injection nodes: - - For injecting a memory error, there are some sysfs nodes, under - ``/sys/devices/system/edac/mc/mc?/``: - - - ``inject_addrmatch/*``: - Controls the error injection mask register. It is possible to specify - several characteristics of the address to match an error code:: - - dimm = the affected dimm. Numbers are relative to a channel; - rank = the memory rank; - channel = the channel that will generate an error; - bank = the affected bank; - page = the page address; - column (or col) = the address column. - - each of the above values can be set to "any" to match any valid value. - - At driver init, all values are set to any. - - For example, to generate an error at rank 1 of dimm 2, for any channel, - any bank, any page, any column:: - - echo 2 >/sys/devices/system/edac/mc/mc0/inject_addrmatch/dimm - echo 1 >/sys/devices/system/edac/mc/mc0/inject_addrmatch/rank - - To return to the default behaviour of matching any, you can do:: - - echo any >/sys/devices/system/edac/mc/mc0/inject_addrmatch/dimm - echo any >/sys/devices/system/edac/mc/mc0/inject_addrmatch/rank - - - ``inject_eccmask``: - specifies what bits will have troubles, - - - ``inject_section``: - specifies what ECC cache section will get the error:: - - 3 for both - 2 for the highest - 1 for the lowest - - - ``inject_type``: - specifies the type of error, being a combination of the following bits:: - - bit 0 - repeat - bit 1 - ecc - bit 2 - parity - - - ``inject_enable``: - starts the error generation when something different than 0 is written. - - All inject vars can be read. root permission is needed for write. - - Datasheet states that the error will only be generated after a write on an - address that matches inject_addrmatch. It seems, however, that reading will - also produce an error. - - For example, the following code will generate an error for any write access - at socket 0, on any DIMM/address on channel 2:: - - echo 2 >/sys/devices/system/edac/mc/mc0/inject_addrmatch/channel - echo 2 >/sys/devices/system/edac/mc/mc0/inject_type - echo 64 >/sys/devices/system/edac/mc/mc0/inject_eccmask - echo 3 >/sys/devices/system/edac/mc/mc0/inject_section - echo 1 >/sys/devices/system/edac/mc/mc0/inject_enable - dd if=/dev/mem of=/dev/null seek=16k bs=4k count=1 >& /dev/null - - For socket 1, it is needed to replace "mc0" by "mc1" at the above - commands. - - The generated error message will look like:: - - EDAC MC0: UE row 0, channel-a= 0 channel-b= 0 labels "-": NON_FATAL (addr = 0x0075b980, socket=0, Dimm=0, Channel=2, syndrome=0x00000040, count=1, Err=8c0000400001009f:4000080482 (read error: read ECC error)) - -3) Corrected Error memory register counters - - Those newer MCs have some registers to count memory errors. The driver - uses those registers to report Corrected Errors on devices with Registered - DIMMs. - - However, those counters don't work with Unregistered DIMM. As the chipset - offers some counters that also work with UDIMMs (but with a worse level of - granularity than the default ones), the driver exposes those registers for - UDIMM memories. - - They can be read by looking at the contents of ``all_channel_counts/``:: - - $ for i in /sys/devices/system/edac/mc/mc0/all_channel_counts/*; do echo $i; cat $i; done - /sys/devices/system/edac/mc/mc0/all_channel_counts/udimm0 - 0 - /sys/devices/system/edac/mc/mc0/all_channel_counts/udimm1 - 0 - /sys/devices/system/edac/mc/mc0/all_channel_counts/udimm2 - 0 - - What happens here is that errors on different csrows, but at the same - dimm number will increment the same counter. - So, in this memory mapping:: - - csrow0: channel 0, dimm0 - csrow1: channel 0, dimm1 - csrow2: channel 1, dimm0 - csrow3: channel 2, dimm0 - - The hardware will increment udimm0 for an error at the first dimm at either - csrow0, csrow2 or csrow3; - - The hardware will increment udimm1 for an error at the second dimm at either - csrow0, csrow2 or csrow3; - - The hardware will increment udimm2 for an error at the third dimm at either - csrow0, csrow2 or csrow3; - -4) Standard error counters - - The standard error counters are generated when an mcelog error is received - by the driver. Since, with UDIMM, this is counted by software, it is - possible that some errors could be lost. With RDIMM's, they display the - contents of the registers - -Reference documents used on ``amd64_edac`` ------------------------------------------- - -``amd64_edac`` module is based on the following documents -(available from http://support.amd.com/en-us/search/tech-docs): - -1. :Title: BIOS and Kernel Developer's Guide for AMD Athlon 64 and AMD - Opteron Processors - :AMD publication #: 26094 - :Revision: 3.26 - :Link: http://support.amd.com/TechDocs/26094.PDF - -2. :Title: BIOS and Kernel Developer's Guide for AMD NPT Family 0Fh - Processors - :AMD publication #: 32559 - :Revision: 3.00 - :Issue Date: May 2006 - :Link: http://support.amd.com/TechDocs/32559.pdf - -3. :Title: BIOS and Kernel Developer's Guide (BKDG) For AMD Family 10h - Processors - :AMD publication #: 31116 - :Revision: 3.00 - :Issue Date: September 07, 2007 - :Link: http://support.amd.com/TechDocs/31116.pdf - -4. :Title: BIOS and Kernel Developer's Guide (BKDG) for AMD Family 15h - Models 30h-3Fh Processors - :AMD publication #: 49125 - :Revision: 3.06 - :Issue Date: 2/12/2015 (latest release) - :Link: http://support.amd.com/TechDocs/49125_15h_Models_30h-3Fh_BKDG.pdf - -5. :Title: BIOS and Kernel Developer's Guide (BKDG) for AMD Family 15h - Models 60h-6Fh Processors - :AMD publication #: 50742 - :Revision: 3.01 - :Issue Date: 7/23/2015 (latest release) - :Link: http://support.amd.com/TechDocs/50742_15h_Models_60h-6Fh_BKDG.pdf - -6. :Title: BIOS and Kernel Developer's Guide (BKDG) for AMD Family 16h - Models 00h-0Fh Processors - :AMD publication #: 48751 - :Revision: 3.03 - :Issue Date: 2/23/2015 (latest release) - :Link: http://support.amd.com/TechDocs/48751_16h_bkdg.pdf - -Credits -======= - -* Written by Doug Thompson <dougthompson@xmission.com> - - - 7 Dec 2005 - - 17 Jul 2007 Updated - -* |copy| Mauro Carvalho Chehab - - - 05 Aug 2009 Nehalem interface - - 26 Oct 2016 Converted to ReST and cleanups at the Nehalem section - -* EDAC authors/maintainers: - - - Doug Thompson, Dave Jiang, Dave Peterson et al, - - Mauro Carvalho Chehab - - Borislav Petkov - - original author: Thayne Harbaugh |