Generation and use of system level defect tables for main memory

[0001] 1. Field of the Invention

[0002] Embodiments of the present invention generally relate to techniques for increasing the overall reliability of memory systems that utilize modular memory modules.

[0003] 2. Description of the Related Art

[0004] Computer system performance can be increased by increasing computing power, for example, by utilizing more powerful processors or a larger number of processors to form a multi-processor system. It is well established, however, that increasing memory in computing systems can often have a greater effect on overall system performance than increasing computing power. This result holds true from personal computers (PCs) to massively parallel supercomputers.

[0005] High performance computing platforms, such as the Altix systems available from Silicon Graphics, Inc. may include several tera-bytes (TBs) of memory. To provide such a large amount of memory, such configurations may include many thousands of modular memory modules, such as dual inline memory modules (DIMMs). Unfortunately, with such a large number of modules in use (each having a number of memory chips), at least some amount of memory failures can be expected. While factory testing at the device (IC) level can catch many defects and, in some cases, replace defective cells with redundant cells (e.g., via fusing), some defects may develop over time after factory testing.

[0006] In conventional systems, a zero defect tolerance is typically employed. If a memory failure is detected, the entire module will be replaced, even if the failure is limited to a relatively small portion of the module. Replacing modules in this manner is inefficient in a number of ways, in addition to the possible interruption of computing and loss of data. On the one hand, the replacement may be performed by repair personnel of the system vendor at substantial cost to the vendor. On the other hand, the replacement may be performed by dedicated personnel of the customer, at substantial cost to the customer.

[0007] A costly solution to increase fault tolerance is through redundancy. For example, some systems may utilize some type of system memory mirroring whereby the same data is stored in multiple “mirrored” memory devices. However, this solution can be cost prohibitive, particularly as the overall memory space increases. Another alternative is to simply avoid allocating an entire defective device or DIMM from allocation. However, this approach may significantly impact performance by reducing the available memory by an entire device or DIMM, regardless of the amount of memory locations found to be defective.

[0008] Accordingly, what is needed is a technique to increase the overall reliability of memory systems that utilize modular memory modules.

[0009] Embodiments of the present invention provide techniques for increasing the overall reliability of memory systems that utilize modular memory modules.

[0010] One embodiment provides a method for maintaining and utilizing memory defect tables in a computing system. The memory defect tables may store entries indicating defective memory locations of memory modules (such as DIMMs) which may then, at the system level, be mapped to non-defective memory locations. The memory defect tables may be maintained across reset (e.g., power-up) cycles of the computing system.

[0011] Another embodiment provides a computing system configured to maintain and utilize memory defect tables. The memory defect tables may store entries indicating defective memory locations of memory modules (such as DIMMs) which may then, at the system level, be mapped to non-defective memory locations. The memory defect tables may be maintained across reset cycles of the computing system.

[0012] Another embodiment provides a computer readable medium containing a program which, when executed by a processor of a computing system, performs operations to maintain and utilizes memory defect tables. The operations may include detecting defective memory locations of memory modules and storing entries in the defect tables indicating the defective memory locations. The operations may also include mapping the defective memory locations to non-defective memory locations. The operations may also include accessing the memory defect tables upon power up, or other reset, and allocating memory to avoid defective locations identified in the tables.

[0013] Another embodiment provides a memory module comprising a plurality of volatile memory devices and at least one non-volatile memory device. A defect table identifying defective locations within the volatile memory devices is contained in the non-volatile memory device. For some embodiments, the non-volatile memory device may also include information for mapping the defective memory locations to non-defective memory locations.

[0020] Embodiments of the present invention generally provide methods and apparatus for maintaining and utilizing system memory defect tables that store information identifying defective memory locations in memory modules. For some embodiments, the defect tables may be utilized to identify and re-map defective memory locations to non-defective replacement (spare) memory locations as an alternative to replacing an entire memory module. For some embodiments, some portion of the overall capacity of the memory module may be allocated for such replacement. As a result, memory modules may be made more fault-tolerant and systems more reliable. Further, this increased fault-tolerance may significantly reduce service requirements (e.g., repair/replacement) and overall operating cost to the owner and/or vendor.

[0021] Embodiments of the present invention will be described below with reference to defect table to identify defective memory locations in dual inline memory modules (DIMMs). However, those skilled in the art will recognize that embodiments of the invention may also be used to advantage to identify defective memory locations in any other type of memory module or arrangement of memory devices, including cache memory structures. Thus, reference to DIMMs should be understood as a particular, but not limiting, example of a type of memory module.

[0022] Further, while embodiments will be described with reference to operations performed by executing code (e.g., by a processor or CPU), it may also be possible to perform similar operations with dedicated or modified hardware. Further, those skilled in the art will recognize that the techniques described herein may be used to advantage in any type of system in which multiple memory devices are utilized.

[0023] FIG. 1 illustrates an exemplary computing system 100 including one or more processors 110 coupled to a memory system 120 via an interface 122. While not shown, the system 100 may also include any other suitable components, such as graphics processor units (GPUs), network interface modules, or any other type of input/output (I/O) interface. The interface 122 may include any suitable type of bus (e.g., a front side bus) and corresponding components to allow the processors 110 to communicate with the memory system 120 and with each other.

[0024] As illustrated, the memory system 120 may include a plurality of memory modules, illustratively shown as DIMMs 130. As previously described, the number of DIMMs 130 may grow quite large, for example, into the thousands for large scale memory systems of several terabytes (TBs). Each DIMM 130 may include a plurality of volatile memory devices, illustratively shown as dynamic random access memory (DRAM) devices 132. As previously described, as the number of DIMMs 130 increases, thereby increasing the number of DRAM devices 132, the likelihood of a defect developing in a memory location over time increases.

[0025] In an effort to monitor the defective memory locations, one or more memory defect tables 150 may be maintained to store defective memory locations. For some embodiments, identification of a defective block or page within a particular DIMM and/or DRAM device 132 may be stored (e.g., regardless of the number of bits that failed). Contents of the defect tables 150 may be maintained across reset cycles, for example, by storing the defect tables 150 in non-volatile memory. As will be described in greater detail below, for some embodiments, defect tables 150 may be stored on the DIMMs, for example in non-volatile memory, such as EEPROMs 134, allowing defect information to travel with the DIMMs.

[0026] Defects may be detected via any suitable means, including conventional error checking algorithms utilizing checksums, error correction codes (ECC) bits, and the like. Regardless, upon detection of a defective memory location, an entry may be made in the defect table 150 and the defective memory location may be remapped to a different (non-defective) location. In some cases, system level software, such as a process 142 running as part of an operating system (O/S) 140 may detect defective memory locations, maintain, and/or utilize the defect table 150, by performing operations described herein.

[0027] For some embodiments, a predetermined amount of memory locations in the DIMM may be allocated and used to replace defective memory locations when detected. In some cases, replacement may be allowed in specified minimal sizes. For example, for some embodiments, entire pages of memory may need to be replaced, regardless of the number of bits found to be defective in that page.

[0028] For some embodiments, some number of spare pages may be allocated and used for replacement by remapping defective memory locations. As an example, for a DIMM with 1 GB total capacity, if n pages were allocated for replacement, the total usable capacity may be 1 GB−n*page_size. The reduction in usable capacity may be acceptable given the offsetting benefit in increased fault tolerance and reduced service/repair. Further, in certain systems, such as cache-coherent non-uniform memory access (ccNUMA) systems, some portion of memory is dedicated to overhead, such as maintaining cache coherency, such that a relatively small (e.g., 1-2%) reduction in overall usable capacity would be transparent to the user.

[0029] For some embodiments, a single defect table 150 may be used to store defective locations for multiple DIMMs 130. For other embodiments, a different defect table 150 may be maintained for each DIMM 130. As illustrated in FIG. 2, for some embodiments, processors and memory may be logically partitioned into a plurality of nodes 210 (as shown 210₀-210_M). Each node 210 may include one or more processors 110 that access DIMMs 130 via an interface hub (SHUB 220). As illustrated, the collective memory of each node 210 may be considered shared memory and accessible by processors 110 on each node. As illustrated, in such configurations, each node 210 may maintain one or more defect tables 150 to store information regarding defective memory locations for their respective DIMMs 130.

[0030] FIG. 3A illustrates exemplary operations for maintaining system memory defect table, in accordance with one embodiment of the present invention. The operations may be performed, for example, as operating system code, or application code. The operations begin, at step 302, by detecting defective memory locations. At step 304, the defective memory locations are stored in the memory defect table 150, which may be maintained across reset cycles. These defective memory locations may be subsequently avoided, for example, by mapping the defective locations to spare (non-defective) locations, at step 306.

[0031] As illustrated, for some embodiments, the defect table 150 may include entries that include a defective location and a corresponding replacement location. For example, the defective location may be identified by page, DIMM, and/or device. Similarly, the replacement location (to which the defective location is mapped) may be identified by page, DIMM, and/or device.

[0032] Referring to FIG. 3B, by maintaining the defect table 150 across reset cycles, the defect table 150 may be utilized upon a subsequent reset to avoid allocating defective memory locations. For example, upon a subsequent reset, at step 308, defective memory locations may be read from the defect memory table 150, at step 310. These memory locations may be avoided during allocation, at step 312, for example, by mapping the defective locations to replacement locations, which may also identified in the defect table 150.

[0033] Defective memory locations may be mapped to replacement memory locations, for example, by manipulating virtual-to-physical address translation. For some embodiments, page table entries may be generated that translate internal virtual addresses to the physical addresses of the replacement memory locations rather than the defective memory locations. For some embodiments, defective memory locations on one DIMM may be replaced by replacement memory locations on another DIMM.

[0034] In any case, by avoiding the allocation of defective memory locations, it may be possible to avoid replacing entire DIMMs 130 and replace defective memory locations in a manner that is transparent to a user. As a result, a much greater utilization of memory resources may be achieved, particularly if only a small percentage of memory locations exhibit defects (e.g., some limited number of pages encounter defective cells). For some embodiments, however, an entire DIMM may be effectively removed (e.g., by de-allocation) if the total number of defective locations exceeds some threshold number or percentage. In other words, the occurrence of a certain number of defective locations may be a good predictor of pending device failure.

[0035] As illustrated in FIG. 4, for some embodiments, defective locations of DRAM devices 432 may be stored on a DIMM 430. For example, one or more defect tables 450 may be stored in non-volatile memory, such as a EEPROM 434. DIMMs typically contain a EEPROM to comply with the JEDEC Serial Present Detect (SPD) standard. According to this standard, DIMM manufactures may store necessary operating parameters (e.g., describing the DRAM devices, operating parameters, etc.), for use by the OS during system reset in configuring memory controllers. If the size of these EEPROMs were sufficient, defect tables 450 could be stored with this SPD data. In fact, it may be beneficial that the storage and use of such defect tables may be incorporated into such a standard.

[0036] Further, in some cases, some memory cells may be more prone to failure, for example, due to marginal storage capacitance in DRAM cells. For some embodiments, such marginal locations may be proactively detected to avoid errors and increase reliability. As an example, FIG. 4 illustrates a technique whereby extended refresh cycles may be utilized for identifying weak DRAM memory cells. Such operations may be performed periodically, or upon reset, as part of an initialization procedure by the operating system. For some embodiments, to prevent uncorrectable errors occurring during testing with extended refresh cycles, user data may be offloaded from a DIMM under test (e.g., to another DIMM or a disk drive).

[0037] In any case, the operations may begin, for example, by setting an initial refresh rate, which may be the same as that typically utilized during normal operation. As is well known, rows of memory cells may be refreshed automatically by issuing a refresh command. DRAM devices may internally increment a row address such that a new row (or set of rows in different banks) is refreshed with each issued command. The rate at which these commands are issued during normal operation is controlled to ensure each row is refreshed within a defined retention time specified by the device manufacturer.

[0038] However, to test for marginal cells, this rate may be incrementally decreased at step 504 (e.g., by increasing the interval between refresh commands) until each row is not refreshed within the specified retention time. At step 506, memory locations exhibiting defects at the lower refresh rate are detected. This detection may be performed in any suitable manner, such as preloading the memory devices with known data, reading the data back, and comparing the data read to the known data. A mismatch indicated a memory cell that fails to maintain data at the lowered refresh rate. Such defective (marginal) memory locations may be stored in a defect table, at step 508.

[0039] As illustrated, the operations may repeat, with successively reduced refresh rates, until the refresh rate reaches a predetermined minimum amount (which may be well below the device specified operating range), as determined at step 510. By storing these defective locations in the defect table, these locations may be avoided during allocation, as described above. For some embodiments, memory defect tables may be populated with both marginal locations detected during testing with extended refresh periods and defective locations detected during normal operation.

[0040] By generating and maintaining system memory defect tables memory module reliability may be significantly increased. As a result, the number of service/repair operations and overall operating costs of systems utilizing memory modules may be reduced accordingly.

[0041] While the foregoing is directed to embodiments of the present invention, other and further embodiments of the invention may be devised without departing from the basic scope thereof, and the scope thereof is determined by the claims that follow.