Method and system for handling multiple bit errors to enhance system reliability

[0001] 1. Technical Field

[0002] The present invention relates generally to an improved data processing system, and in particular to a method and apparatus for handling multiple bit errors in a data processing system. The present invention also provides a set of computer-implemented instruction for serving multiple bit errors.

[0003] 2. Description of Related Art

[0004] In a large memory system, errors from memory device may cause a catastrophic system error. A hard error means that a cell within the memory device is permanently defective. On the other hand, a soft error is a temporary fault such as when a data bit is lost. With soft errors, the memory device still functions correctly after the data is rewritten into the memory cell. There are many causes of soft errors, such as alpha particles, noise on power or control signals, temperature extremes, marginal timing, or the like.

[0005] Today computer systems with high availability requirements use error detection logic and parity to ensure data integrity and system reliability. For computer hardware with high failure rates (e.g. system memory, cache, etc.), error correction code (ECC) logic is used to correct single bit error. Such ECC logic helps to prevent an immediate failure of the system and improve overall system availability.

[0006] A system memory is the central storage in the computer where program and data reside and waiting to be processed by the processor. A cache is a temporary storage area close to or internal to the processor that allows speedy access to program or data. An array is a term generally refer to as smaller arrangements of temporary memory storage, including cache. A cache or memory address is a reference to a physical location within the cache or memory storage which store one or several bytes of computer instruction or data. A cache line is a block of addresses or physical locations within the cache, usually a group of 128, 256 or 512 bytes of data. Such architecture of line addressing may also apply to any memory system.

[0007] A cache or memory address with repeating single bit errors indicates a hard error condition which requires continuous error correction by ECC logic. A cache or memory with a single hard error, if left in the system for an extended period of time, may lead to an incorrectable error condition and system outage due to an occurrence of second hard error within the same or adjacent physical address location. A typical ECC logic can only handle single bit error. To prevent potential system failure for a computer system with high availability requirement, it is a general practice to replace the cache or memory with single bit hard error. However, frequent replacement of parts can lead to high service cost for the computer manufacturer and poor system reliability perception by the customer.

[0008] Therefore, it would be advantageous to have an improved system to minimize service cost and to improve system reliability by having the capability to continuously run the system without replacing the part with single-bit hard error and even when a second hard error occurs. It would further be beneficial to have an apparatus and method that allows scheduling of maintenance time after a second hard error occurrence but before a catastrophic error or system down time.

[0009] The present invention provides a method and system for handling bit errors on a data processing system. The method first determines whether the bit error is a soft or hard error. The error correction code logic corrects and handles soft errors. When a hard error exists on a line for the first time on the system, the bit error is removed from the system. Specifically, a bit error is removed by deleting the line that correspond to the bit error and permanently damaged memory device, thereby the line is no longer used by the system. As a result, the system is running with “N−1” lines wherein “N” constitutes the total number of existing lines.

[0010] When multiple hard errors exist, the system will prevent any down time by logging in the parts to be replaced by an operator. The error correction code logic corrects the hard errors until that part is replaced.

[0017] With reference to the figures, in particular, FIG. 1 depicts a block diagram of a data processing system (“DPS”) according to an embodiment of the present invention. In this example, data processing system 100 may be a symmetric multiprocessor (“SMP”) system including a plurality of processors 101, 102, 103, and 104 connected to system bus 106. For example, data processing system 100 may be an IBM eServer pSeries (formerly known as RS/6000), a product of International Business Machines Corporation in Armonk, N.Y. Alternatively, a single processor system may be employed. Also connected to system bus 106 is a memory controller, 108 which provides an interface to a plurality of local memories 160–163. I/O bus bridge 110 is connected to system bus 106 and provides an interface to I/O bus 112. Memory controller 108 and I/O bus bridge 110 may be integrated as depicted.

[0018] Peripheral component interconnect (PCI) Host bridge 114 connected to I/O bus 112 provides an interface to PCI bus 115. A number of terminal bridges 116–117 may be connected to PCI bus 115. Typical PCI bus implementations will support four terminal bridges for providing expansion slots or add-in connectors. Each of terminal bridges 116–117 is connected to a PCI I/O adapter 120–121 through PCI Bus 118–119. Each I/O adapter 120–121 provides an interface between data processing system 100 and input/output devices such as, for example, other network computers, which are clients to server 100.

[0019] Additional PCI host bridges 122, 130, and 140 provide interfaces for additional PCI buses 123, 131, and 141. Each of additional PCI buses 123, 131, and 141 are connected to a plurality of terminal bridges 124–125, 132–133, and 142–143, which are connected to a PCI I/O adapter 128–129, 136–137, and 146–147 by a PCI bus 126–127, 134–135, and 144–145. Thus, additional I/O devices, such as modems or network adapters may be supported through each of PCI I/O adapters 128–129, 136–137, and 146–147. In this manner, server 100 allows connections to multiple network computers. A memory mapped graphics adapter 148 and hard disk 150 may also be connected to I/O bus 112 as depicted, either directly or indirectly.

[0020] Those of ordinary skill in the art will appreciate that the hardware depicted in FIG. 1 may vary. For example, other peripheral devices, such as optical disk drives and the like, also may be used in addition to or in place of the hardware depicted. The depicted example is not meant to imply architectural limitations with respect to the present invention.

[0021] FIG. 2 depicts an example of a processor as described in FIG. 1 according to a preferred embodiment of the present invention. Processor 101 includes caches, such as a level 1 (L1) and a level 2 (L2) cache. A cache is a memory area close to or in the processor 101 that allows speedy access of information. It is used to speed up data transfer and is not part of the local memory 160–163. Instructions and data are transferred to the cache in blocks using a look-ahead algorithm. The more sequential the instructions in the routine being accessed, and the more sequential the order of the data being read, the more chance the next desired item will still be in the cache, and the greater improvement in performance.

[0022] In this example, the L1 and L2 cache are permanently located within processor 101. A level 3 (“L3”) cache 201 exists outside of the processor, but is nearby. In this example, each L3 cache consists of 32 Mega-byte of storage. Additional L3 cache is required as more processors are added into system configuration. L3 cache improves system performance by minimizing the time required for the processor to read data from system memory. Each L3 cache line has 512 bytes. L3 cache hardware is designed with error correction code (ECC) to correct single bit error and with a system capable of operating with “N−1” cache line. “N” represents the total number of cache lines for a particular cache level in the system. An embodiment of the present invention may be implemented with a system running at “N−x” where “x” is a smaller value than “N”.

[0023] The L3 cache includes supporting storages: L3 Fault Address storage 203 and L3 Fault Isolation storage 204. L3 Fault Address storage 203 contains the address of the physical location in the cache that is faulty. L3 Fault Isolation storage 204 contains captured error signatures including the single bit correctable error condition as to the cause of the error.

[0024] When single-bit error condition occurs in L3 cache array or line, the L3 cache hardware detects the error during a read or write cycle to the cache. L3 ECC hardware logic corrects the error. Information about the error is then recorded. A single-bit correctable error (CE) condition is set in L3 Fault Isolation storage 204. The address of the physical location of the error is saved in L3 Fault Address storage 203. Next, L3 cache hardware asserts “Attention” signal to service processor 205 to indicate an error condition. Processor 101 also includes L3 cache controller 207. L3 cache controller 207 contains L3 Purge Logic 208, L3 Mode storage 209, and L3 Line Delete register 210. The process utilizes these registers and logic in handling bit errors in the system.

[0025] The L3 cache line purge operation 208 ensures that if the data in L3 cache line is modified by any processor, the modified data is saved in system memory for future reference by other software program. Before the data is saved in memory, the single bit error is corrected by ECC logic when the data is accessed. The L3 cache line purge operation 208 also removes the soft error from L3 cache 201.

[0026] The present invention may be implemented as hardware or software, or a combination thereof. In a preferred embodiment, it is implemented from a firmware 206 of a service processor (SP) 205. SP firmware 206 is a set of computer instructions produced by computer hardware manufacturer to implement a specific computer function. In this case, the SP firmware 206 is written specifically for handling bit error. The firmware is executing processor instructions on the service processor. Those skilled in the art will understand that a set of computer instructions may be implemented from any processor in the system. The present invention is not limited to use by or in a service processor.

[0027] FIG. 3 depicts a flow chart of a process for distinguishing between soft and hard errors in the system. Initially, the system runs without any bit errors. There is no existing saved address for a non-existing bit error. The saved address corresponds to a prior physical location within the L3 cache of the data processing system that had the error. When a new error occurred and detected (step 301) for the first time in the system (step 302), the process receives an error indication (step 301). It checks for the source of the error. This first time error is assumed to be a soft error (step 303). The corresponding address of the error is saved (step 304) into a service processor storage location which is compared to latter address error and continually saves the latest address error. The soft error is handled accordingly.

[0028] When another error occurs, the process again receives an error indication, it checks for the source of the bit error (step 301). Then, it detects that an address is previously saved indicating that this is not a first time error (step 302). Next, it compares the error address to the previously saved address (step 305). A soft error is further detected if the addresses are not the same when compared (step 306). The address of this later bit error is then saved (step 307) ready to for another comparison when the next latter error occurs. The soft error is handled accordingly and is described in greater detail in FIG. 5.

[0029] When the process receives a bit error which turns out to be a hard error, it again checks for the source of the bit error (step 301). Then, it detects that an address is previously saved in the storage indicating that this is not a first time error (step 302). Next, it compares the error address to the previously saved address (step 305). The process detects a hard error if the compared addresses (step 306) are the same. The cache address of this bit error is then saved into the storage (step 308) ready to for another comparison when the next error occurs. The hard error is then handled accordingly as first or second hard error which is described in FIG. 4 and FIG. 5.

[0030] FIG. 4 is a flowchart generally outlining a process for handling cache with multiple single-bit errors of the present invention. As shown in FIG. 4, the process starts with the receipt of an attention to an error condition (step 402). The error condition for detection and correction includes bit errors. These bit errors include soft errors and hard errors. Next, the process determines the type of error conditions, soft error, serviceable hard, or last allowable hard error (step 404). If a soft error is detected, error is handled by a cache line purge operation (step 406).

[0031] If a serviceable hard error is detected, it is serviced accordingly step (408). The number of serviceable hard error that can be processed is predetermined by the capability of the L3 hardware logic design, the capability of the system to operate with “N−x” cache line, capability to exchange cache lines, or the combination thereof. Each time a hard error occurs, the number of hard error is incremented. In a preferred embodiment, the serviceable hard error that can be processed is one (1) because the system is designed with the capability to run with “N−1” cache line. The present invention may be implemented with a system running at “N−x” where “x” is a smaller value than “N” or by exchangeable cache line.

[0032] Exchangeable cache line is permissible when the system provides for spare cache line(s). A spare cache line is designed to replace a damaged one by exchanging or rerouting cache line to the spare while the damaged memory device becomes unused. A damaged one is replaced or exchanged without any system down time.

[0033] If a last allowable hard error is detected, the error is processed and the system continues to run with continuous error correction (step 410). Last allowable hard error is predetermined by the maximum number of serviceable hard error plus one. Last allowable hard error constitutes the last bit error the can be handled without system down time. The part(s) to be replaced is determined and logged. An operator is scheduled to replace these parts before system down time.

[0034] FIG. 5 is an exemplary flowchart of a process for handling bit errors on a data processing system. The elements shown in FIG. 5 may be implemented as hardware, software, firmware or a combination thereof. The operation illustrated in FIG. 5 may be implemented as instructions in system firmware executing on a computer by one or more processors. In these examples, steps 501 through 510 are implemented as firmware executing computer instructions from a service processor supporting a computer system with errors in L3 cache.

[0035] The process begins when SP 205 receives an Attention signal (step 502). The operation checks for the source of Attention signal (step 503) or error information. Step 503 includes reading L3 Fault Isolation register 204 and L3 Fault Address register 203. Determining the cause of the Attention signal involves the reading and deciphering L3 Fault Isolation 204. Reading the L3 Fault Address register 203 determines the location or address of a single bit error in the cache.

[0036] Next, in step 504, the process decides whether the error condition is a soft error or a hard error. First, the current address of the error is compared with a previously saved address of another error. If there is no previously saved address or the compared addresses are not the same, then the error is assumed as a soft error because a single bit error is present in this address location for the first time. In step 505, the SP firmware 206 handles the soft error by saving the current address of error in its storage and performing a single L3 cache line purge operation 208 by utilizing hardware logic in L3 controller 207. Next, the cache line is marked available for future usage (step 511). The handling of the soft error is terminated. Throughout this process, data processing system 100 continues to run. However, if the compared addresses are the same, then a second occurrence of a single bit error in this address location is present (step 504). In accordance with the probability and statistic of projected soft error failure rate, the second occurrence of a single bit error in the same address location indicates that this is a hard single bit error condition as noted in step 506. The SP firmware 206 then reads the Line Delete Valid Bit 211 to determine whether a cache line delete operation has been performed (step 507) on this L3 slice. A slice is a sub-section of a cache. In this example, L3 cache 201 is logically divided into two slices, slice 0 and slice 1.

[0037] If Line Delete Valid Bit 211 designated for this L3 slice is equal 0, then a line has not been previously deleted. This condition indicates a first hard error or a single-bit hard error for this L3 slice (step 508). First, an L3 cache line delete operation (step 508) is performed. Thereafter, the system runs with “N−1” cache line (step 512) for this L3 slice. Only one cache line delete operation (step 508) can be performed for each L3 slice for this embodiment of the present invention. Cache line delete operation prevents (step 508) the L3 cache line with single-bit hard error from future usage by the system. The impact to the system performance is extremely small. After step 508, the handling of a first hard error is terminated. At which time, the system continues to run with “N−1” cache line. Line delete operations (step 508) include the following steps:

[0038] (1) setting the corresponding address value field in the L3 Line Delete register 210 to equal the L3 Fault Address 203 captured above preventing future usage of the deleted cache line;

[0039] (2) setting the mode field of the L3 Mode Register 209 to “cache line delete” mode;

[0040] (3) performing a single L3 cache line purge operation 208 by using hardware logic in L3 controller 207;

[0041] (4) setting the corresponding cache set value field in the L3 Line Delete register 210 to this cache set preventing the L3 controller from snooping new requests against this deleted cache line entry in this cache set; and

[0042] (5) setting to 1 the Line Delete Valid Bit 211 for this L3 slice preventing another cache line delete operation on the L3 slice.

[0043] According to step 507, if Line Delete Valid Bit 211 designated for this L3 slice is equal to 1, a cache line has already been deleted and no further cache line delete operation for this L3 slice is possible since only one line delete operation can be performed for each L3 slice. The second hard error is then handled by logging and reporting the bit error to indicate that an L3 cache part replacement is required (step 509). Consequently, a service personnel has the opportunity to replace the defective part(s) before an occurrence of system down time. The system continues to run with continuous error correction by the ECC (step 510). Second single-bit hard error is a type of hard error that occurs on another address other than the address of the first hard error. More importantly, the system has handled distinctive bit errors occurring on distinct addresses, thereby preventing any system down time.

[0044] Thus, the present invention provides an improved method, an apparatus, and a set of computer-implemented instructions for handling cache with multiple single-bit hard errors. The mechanism of the present invention includes a routine or a process that handles accordingly a soft error, a first hard error and a second hard error in which the system will not have down time by logging and reporting the error for part replacement by a service personnel.

[0045] It is important to note that while the present invention has been described in the context L3 cache, those of ordinary skill in the art will appreciate that the processes of the present system are desirable for any memory system with ECC logic.

[0046] It also is important to note that while the present invention has been described in the context of a fully functioning data processing system, those of ordinary skill in the art will appreciate that the processes of the present invention are capable of being distributed in the form of a computer readable medium of instructions and a variety of forms and that the present invention applies equally regardless of the particular type of signal bearing media actually used to carry out the distribution. Examples of computer readable media include recordable-type media, such as a floppy disk, a hard disk drive, a RAM, a CD-ROM, and DVD-ROM; and transmission-type media, such as digital and analog communications links, and wired or wireless communications links using transmission forms, such as radio frequency and light wave transmissions. The computer readable media may take the form of coded formats that are decoded for actual use in a particular data processing system.

[0047] The description of the present invention has been presented for purposes of illustration and description, and is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiment was chosen and described in order to best explain the principles of the invention, the practical application, and to enable others of ordinary skill in the art to understand the invention for various embodiments with various modifications as are suited to the particular use contemplated.