当前位置：网站首页>Server memory failure prediction can actually do this!

Server memory failure prediction can actually do this!

2022-07-26 09:11:00 【InfoQ】

author ：vivo Internet server team -  Hao Chan

With the rapid development of Internet business , The availability of infrastructure is also increasingly concerned by the industry . The failure rate of memory failure is high 、 Many times 、 Great influence , These are unacceptable for upper tier businesses .

This paper mainly introduces EDAC（Error Detection And Correction） Application of framework in memory prediction . First, it introduces EDAC The background of the application , Next is EDAC Introduction to the principle of , And then through EDAC install —— To configure —— The testing process is introduced in detail EDAC stay vivo Applications on the server , Finally, memory prediction is proposed EDAC Scheme summary and server RAS（Reliability, Availability and Serviceability） Application prospects for reducing the impact of hardware failures on the system .

One 、 Background introduction

With the rapid development of Internet business , The availability of infrastructure is also increasingly concerned by the industry . However, hardware failure has always been a common phenomenon , The loss caused by hardware failure is often huge . In all parts of the server , Except for hard disk failure , Memory failure is the second most common type of hardware failure . And the amount of server memory is large ,vivo The amount of memory reaches 40w+ strip , The most serious consequence of memory failure is that it will directly lead to system crash , Server down , These are unacceptable for upper tier businesses .

Memory failure can be divided into UCE（Uncorrectable Error） and CE（Correctable Error）. When the hardware detects an error , It will be reported to CPU Of . One way is to interrupt , If this is the case UCE That is, uncorrectable mistakes , It may cause the server to shut down immediately . If it is CE, You can correct the error , The hardware will use some resources to repair the error , And when memory CE Accumulated too much , When self repair is not possible , Will produce UCE, Cause system downtime and restart . therefore , We need to find out as soon as possible CE Too many memory modules , Replace it in time , Avoid causing heavy losses .

In the past, memory failures were mostly through MCE（Machine Check Exception）log and BMC Records of the SEL （System Error Log） Combine logs to find and locate faults , The biggest problem is that you can't find memory problems in advance , It is often found passively after the server is down and restarted . In addition, there are also the following problems ：

MCE It is difficult to locate the log directly to the failed memory slot .
There is no intuitive CE/UCE Error count .
Cannot be based on the memory module CE/UCE The amount of determines the health of memory .

For the above problems , We need to find other solutions . At this time EDAC It appears in our vision , It can perfectly solve all the problems mentioned above , And can realize memory CE Active fault discovery , Discover memory problems in advance .

This article will mainly introduce EDAC And how to achieve fault prediction through it .

Two 、EDAC Principle introduction

EDAC（Error Detection And Correction） yes Linux System error detection and correction framework , Its purpose is to linux During system operation , When errors occur, hardware errors can be found and reported .EDAC By a core （edac_core.ko） And multiple memory controller driver modules , Its subsystems are edac_mc、edac_device、PCI bus scanning, They are responsible for collecting memory controllers , Other controllers （ such as L3 Cache controller ） as well as PCI Errors reported by the device .

This is mainly about EDAC Subsystem edac_mc How to collect memory controller errors . Memory CE as well as UCE yes edac_mc class The main error types obtained , It mainly involves the following functions ：

【edac_mc_alloc()】
： Use structure mem_ctl_info To describe the memory controller , Only EDAC The core of can touch it , adopt edac_mc_alloc() This function allocates the contents of the filling structure .
【edac_device_handle_ce()】
： Mark CE error .
【edac_device_handle_ue()】
： sign UCE error .
【edac_mc_handle_error()】
： Report memory events to user space , Its parameters include the hierarchy of fault points and fault types , Cumulative Correlation UCE/CE Error count statistics .
【edac_raw_mc_handle_error()】
： Report memory events to user space , But don't do anything to find its location , Only when the hardware error comes from BIOS when , Will be edac_mc_handle_error() Call directly .

that EDAC How to control and report equipment failure ？ How does it locate and record the fault to the corresponding memory module ？

Linux It's through sysfs File system to show the hierarchical relationship of kernel devices ,EDAC It is used to control and report equipment faults .EDAC It is through the abstract memory controller model , Locate the fault on the corresponding memory module , This is mainly related to the arrangement structure of memory in the system .CPU For each of them MC（memory controller） The device controls a group DIMM Memory module , These modules are selected by a chip (Chip-Select Row,csrowX) And channel (Channel,chX) Arrange in a way , There can be multiple in the system csrow And multiple channels .

You can view relevant files through the following path ：

# ls /sys/devices/system/edac/mc/mc0/csrow0/
ce_count ch0_ce_count ch0_dimm_label ch1_ce_count ch1_dimm_label dev_type edac_mode mem_type power size_mb subsystem ue_count uevent

The purpose of some documents is shown in the following table ：

EDAC If the hardware device controller is found to report UE event , And the controller requires UE Namely shutdown , Will restart the system . The controller has detected CE After the event , It can be seen as a contribution to the future UCE Prediction of events . We can use some shielding methods or replace the memory module to reduce UE Events and the possibility of system downtime .

3、 ... and 、EDAC Application

EDAC stay vivo The application process in the current network is mainly divided into the following steps ：

（1）EDAC stay Linux Support in the system

EDAC stay Linux 2.6.16 The above kernels and system distributions have been supported , But in the kernel edac There are many driver modules , Different system versions support different driver modules , You can check which driver modules the system supports in the following ways .

# ls /lib/modules/3.10.0-693.el7.x86_64/kernel/drivers/edac/
amd64_edac_mod.ko.xz edac_core.ko.xz i3000_edac.ko.xz i5000_edac.ko.xz i5400_edac.ko.xz i7core_edac.ko.xz ie31200_edac.ko.xz skx_edac.ko.xz
e752x_edac.ko.xz edac_mce_amd.ko.xz i3200_edac.ko.xz i5100_edac.ko.xz i7300_edac.ko.xz i82975x_edac.ko.xz sb_edac.ko.xz x38_edac.ko.xz

So what is the difference between these driver modules ？ How should we choose ？ take sb_edac And skx_edac To illustrate , Let's first look at their descriptions .

# modinfo sb_edac
filename: /lib/modules/3.10.0-693.el7.x86_64/kernel/drivers/edac/sb_edac.ko.xz
description: MC Driver for Intel Sandy Bridge and Ivy Bridge memory controllers - Ver: 1.1.1
...

# modinfo skx_edac
filename: /lib/modules/3.10.0-693.el7.x86_64/kernel/drivers/edac/skx_edac.ko.xz
description: MC Driver for Intel Skylake server processors
...

By looking at the description, we find , The original driver module is and CPU Product architecture , Installing mismatched modules will appear edac-util: Error: No memory controller data found Such a report is wrong . After our test, we found that , generally speaking , If CPU If the driver module supported by the product architecture exists , The system will install supported drivers by default .

（2） Configure the corresponding relationship between memory slot and physical slot

adopt sysfs File system we can see which CPU Which memory controls which channel and which memory CE Count , But which memory does it correspond to in the system , After all, the daily operation and maintenance of our servers , What you often see is the system slot name , So what is their relationship ？

After check edac-util Source code structure discovery , It provides labels.db This configuration file , To store the corresponding relationship between the system slot and the physical slot of the server memory .

# cat /etc/edac/labels.db
# EDAC Motherboard DIMM labels Database file.
#
# $Id: labels.db 102 2008-09-25 15:52:07Z grondo $
#
# Vendor-name and model-name are found from the program 'dmidecode'
# labels are found from the silk screen on the motherboard.
#
#Vendor: <vendor-name>
# Model: <model-name>
# <label>: <mc>.<row>.<channel>

When I wrote this document , We need to know how the memory is inserted on the server , And know that it corresponds to the slot name in the system , Different server models have different names of system slots . Generally, the interpolation method that can maximize the memory performance , To sum up, it is symmetrical interpolation , And insert it first CPU Far passage , Insert it into each channel first CPU Far slot .

When the configuration is complete , How to check whether the configuration is correct , It is mainly divided into two steps ：

① Use edac-ctl see SYSFS CONTETS Whether the number of pieces is correct
② use dmidecode -t memory Check whether the memory names are consistent

Here we also encounter a rpm Package problem ： For the motherboard of the manufacturer model name There are multiple spaces before and after ,edac-ctl Unable to recognize the motherboard model name,lables.db Unable to register successfully . Finally, we modified edac-utils Package source code , Repackaged .

（3） Testing and verification

After the installation configuration is complete , It's time to test and verify , How to verify EDAC The correctness of the , Guarantee CE The error is recorded on the corresponding memory module ？ We can use APEI Error inject Do some error injection tests .

APEI Error inject Its principle is to rely on APEI（ACPI Platform Error Interface）, There are four tables in its structure ：

BERT（Boot Error Record Table）
： It is mainly used to record errors during startup
ERST（Error Record Serialization Table) 
： An abstract interface for storing errors permanently , Store errors related to various hardware or platforms , Error types include Corrected Error（CE）,Uncorrected Recoverable Error（UCR）, as well as Uncorrected Non-Recoverable Error, Or say Fatal Error.
EINJ（Error Injection Table）
： The main function is to inject errors and trigger errors , Is a table for testing
HEST（Hardware Error Source Table）
： Many error sources and types are defined . The purpose of defining these hardware error sources is to standardize the implementation of software and hardware error interfaces .

This is through debugfs To the kernel APEI The structure of the EINJ Table injection memory error to test ,debugfs Is a virtual file system for kernel debugging , In short, it can be passed through debugfs Map kernel data to user space , Enable users to modify some data for debugging .

The method steps are as follows ：

#  See if it exists EINJ surface 
# ls /sys/firmware/acpi/tables/EINJ 

# grep < The following fields > /boot/config-3.10.0-693.el7.x86_64

CONFIG_DEBUG_FS=y 
CONFIG_ACPI_APEI=y
CONFIG_ACPI_APEI_EINJ=m

#  install einj
# modprobe einj


#  Check the memory address range , This step is because /proc/iomem This file records the allocation of physical addresses , Some memory addresses are reserved by the system and occupied by other devices , Unable to inject errors .
# cat /proc/iomem | grep &quot;System RAM&quot;
00001000-000997ff
00100000-69f79fff
6c867000-6c9e6fff
6f345000-6f7fffff
100000000-407fffffff

#  Check the memory page size 
# getconf PAGESIZE
4096  namely 4KB

#  Get into edac Error injection directory 

# cat /proc/mounts | grep debugfs
debugfs /sys/kernel/debug debugfs rw,relatime 0 0

# cd /sys/kernel/debug/apei/einj/

#  Check the error types that support injection 
# cat available_error_type
0x00000008 Memory Correctable
0x00000010 Memory Uncorrectable non-fatal
0x00000020 Memory Uncorrectable fatal

#  Write the type of error to inject 
echo 0x8 > error_type 

#  Write memory address mask 
echo 0xfffffffffffff000 > param2

#  Write memory address 
echo 0x32dec000 > param1

#  write in 0x0, if 1, Will skip the trigger link 
echo 0x0 > notrigger

#  Writing any integer triggers error injection , This is the last step of error injection 
echo 1 > error_inject

#  Check the log 
# tail /var/log/message
xxxxxx xxxxxxxx kernel: [2258720.203422] EDAC MC0: 1 CE memory read error on CPU_SrcID#0_MC#0_Chan#0_DIMM#1 (channel:0 slot:1 page:0x32dec offset:0x0 grain:32 syndrome:0x0 - err_code:0101:0090 socket:0 imc:0 rank:0 bg:0 ba:3 row:327 col:300)

#  Use edac-util -v see , You can see that the corresponding memory module has added CE Count

Four 、 Summary and prospect

EDAC You can definitely get the data on each memory of the server CE Count , We can go through CE Count to set the threshold , analysis CE Counting curve, etc , Combine with other MCE log 、SEL Wait for the health evaluation of the memory , Make memory prediction .EDAC stay vivo Since the server was fully launched , Cumulative discovery in advance 450+ case Of memory CE problem , The number of server downtime is significantly reduced . Migrate the server business that meets the repair standard , And replace the corresponding memory module , Avoid business instability caused by sudden server downtime , Even the losses caused by this .

EDAC Is the server RAS（Reliability, Availability and Serviceability） A small part of memory applications .RAS It refers to some technical means , The combination of software and hardware ensures the three capabilities of the server .RAS There are still many optimizations in memory , for example MCA（Machine Check Architecture）recovery wait . In the future, we will also introduce RAS To mitigate the impact of hardware failure on the system .

Reference material ：