Memory Controller With Error Logging

The patent application of Raymond D. Bowden, III, Edward R. Salas, Marc E. Sanfacon and Jeffrey S. Somers entitled, "System for Determining Status of Errors in a Memory Subsystem," Ser. No. 07/287,928, filed on Dec. 21, 1988, which is assigned to the same assignee as this patent application.

The present invention relates to a memory controller having error logging capabilities and, more particularly, to a memory controller that logs errors and determines the locations thereof.

In any digital system where data is stored, transmitted or received, one or more of the data bytes may be received and stored in error. In fact, a data value may even be corrupted spontaneously due to impingement of the memory device by alpha particles or due to other random, unpredictable events. This has been a problem from the time data processing systems were first invented.

As more sophisticated data processing operations are performed, involving more complex equipment, there is a greater need for systems to detect and correct stored errors. For example, operations such as merging files, sorting of data within files, numerical/statistical analyses, complex data handling procedures and word processing operations require increased reliability in data transfer and storage. If data errors occur and are undetected, valuable information and system operation itself may be affected. Thus, error detecting and correcting features are not only advantageous, they are required to improve system integrity.

In response to the problem of error generation, systems have been developed to detect such errors. One of the earliest methods for detecting errors was the parity check code. A binary code word has odd parity if an odd number of its digits are 1's. For example, the number 1011 has three 1 digits and therefore has odd parity. Similarly, the binary code word 1100 has an even number of 1 digits and therefore has even parity.

A single parity check code is characterized by an additional check bit added to each data word to generate either odd or even parity. An error in a single digit or bit in a data word would be discernible since the parity check bit associated with that data word would then be reversed from what is expected. Typically, a parity generator adds the parity check bit to each word before transmission. This technique is called padding the data word. At the data requesting device or receiver, the digits in the word are tested and if the parity is incorrect, one of the bits in the data word is considered to be in error. When an error is detected at a receiver, a request for a repeat transmission can be given to memory so that the error can be corrected. Only errors in an odd number of digits can be detected with a single parity check, since an even number of errors results in the parity expected for a correct transmission. Moreover, the specific bit in error cannot be identified by the parity check procedure as hereinabove described.

A more sophisticated error detection system was later devised. Data words of a fixed length of bits were grouped into blocks of a fixed number of data words each. Parity checks were then performed between different data words as well as for each individual data word. The block parity code detected many patterns of errors and could be used not only for error detection, but also for error correction when an isolated error occurred in a given row and column of the matrix. While these geometric codes were an improvement over parity check bits per se, they still could not be used to detect errors that were even in number and symmetrical in two dimensions.

After parity check codes and geometric codes were devised, a code was invented by R. W. Hamming, after whom it is named. The Hamming code is a system of multiple parity checks that encodes data words in a logical manner so that single errors can be not only detected but also identified for correction. A transmitted data word used in the Hamming code consists of the original data word and parity check digits appended thereto. Each of the required parity checks is performed upon specific bit positions of the transmitted word. The system enables the isolation of an erroneous digit, whether it is in one of the original data word bits or in one of the added parity check bits.

If all the parity check operations are performed successfully, the data word is assumed to be error free. If one or more of the check operations is unsuccessful, however, the single bit in error is uniquely determined by decoding so-called syndrome bits, which are derived from the parity check bits. It should be noted once again that only single bit errors are detected and corrected by use of the conventional Hamming code. Double bit errors, although detectable by the Hamming code, are not correctable.

The Hamming code is only one of a number of codes, generically called error correcting codes (ECC's). Codes are usually described in mathematics as closed sets of values that comprise all the allowed number sequences in the code. In data communications and processing, transmitted or transferred numbers are essentially random data patterns which are not related to any predetermined code set. The sequence of data, then, is forced into compliance with the code set by adding to it at the transmitter, as hereinabove mentioned. A scheme has heretofore been developed to determine what precise extra string to append to the original data stream to make the concatenation of transmitted data a valid code. There is a consistent way of extracting the original data from the code value at the receiver and to deliver the actual data to the location where it is ultimately used. For the code scheme to be effective, it must contain allowed values sufficiently different from one another so that expected errors do not alter an allowed value such that it becomes a different allowed value of the code.

A cyclic redundancy code (CRC) consists of string of binary data evenly divisible by a generator polynomial, which is a selected number that results in a code set of values different enough from one another to achieve a low probability of an undetected error. To determine what to append to the string of original data, the original string is divided as it is being transmitted. When the last data bit is passed, the remainder from the division is the required string that is added since the string including the remainder is evenly divisible by the generator polynomial. Because the generator polynomial is of a known length, the remainder added to the original string is also of fixed length.

At the receiver, the incoming string is divided by the generator polynomial. If the incoming string does not divide without remainder, an error is assumed to have occurred. If the incoming string is divided by the generator polynomial without remainder, the data delivered to the ultimate destination is the incoming data with the fixed length remainder field removed.

A so-called soft error (i.e., one that is correctable) could be the result of an alpha particle destroying the charge on a memory cell. This is known as a soft error because the data can be rewritten and the cell can perform properly again. A soft error is correctable via an Error Detection and Correction (EDAC) chip.

If the error is a result of a memory cell stuck at logic one or zero, the memory cell can no longer function properly and should be removed. This is known as a "hard" or uncorrectable error.

Both types of errors are symptomatic of potential memory cell degradation in DRAMs. While distinguishing between soft and hard errors is helpful in determining the effects of system operation, such distinction by itself does not identify the cause of the error.

In systems that utilize a great deal of memory, banks or matrices of integrated circuit chips store correspondingly great amounts of data. The probability of a soft error occurring is the same for any one memory storage device, of course; but the probability of at least one error occurring during a predetermined time interval in a general system increases in direct proportion to the number of such memory devices or size of overall memory. Thus, larger, more complex computer systems are especially susceptible to soft errors due to random anomalies and are therefore also susceptible to defectively stored data.

U.S. Pat. Nos. 4,371,930 and 4,375,664 both issued to Kim, teach the use of error detection, correction and selective logging apparatus for single bit memory errors. Only single bit errors which the system determines most likely to be solid (hardware-related) errors are logged. The solid error situation occurs when two single bit errors have consecutive memory addresses.

U.S. Pat. Nos. 4,523,313 and 4,527,251, both issued to Chester M. Nibby, Jr., et al, teach the use of partially good memory devices in a semiconductor memory system. The defective portions of memory devices in the aforementioned references, however, are hard, uncorrectable portions, while the good portions are consistently reliable and identified as such before they are incorporated into the computer system. Identifying errors that may be correctable (soft errors) and determining the chips in which they occur is beyond the scope of these references.

U.S. Pat. No. 4,506,362 issued to Morley teaches the use of apparatus for periodically reading data from each addressable memory location in a single memory device and for determining the presence of an error. Circuitry can also be provided for logging areas of memory in which errors have been detected, according to this patent, but such circuitry is not specifically disclosed therein.

U.S. Pat. No. 4,535,455 issued to Peterson teaches a system for correcting and logging transient errors that occurred in a block of memory locations in a memory device. A microprocessor under the control of a program accesses and reaccesses each memory location, rewrites each word of readout information, again reads each word, and finally logs only transient errors in an error rate table.

U.S. Pat. No. 4,570,217 issued to Allen et al teaches a system for interfacing a machine capable of performing industrial processes with a human operator. Multi-chip memory locations, divided into banks, are refreshed periodically and an error log is kept, but only selected errors that occur at predetermined times are logged.

It would be advantageous to detect and isolate errors.

It would also be advantageous to identify the specific memory chip, in a multi-chip group, responsible for data errors.

It would also be advantageous to log the occurrence and location of errors for future analysis.

It would also be advantageous to generate syndrome bits as a function of check bits associated with suspect data so as to facilitate location identification.

In accordance with the present invention, there is provided a system for logging an error that occurred in a multi-chip memory storage device in a data processing system. The system has a mechanism for detecting an error and for receiving data and check bits associated therewith. The mechanism for detecting an error generates syndrome bits as a function of the data and of the check bits. Connected to the error detecting mechanism is an error logging mechanism which is adapted to receive the syndrome bits and to determine the chip in which the error occurred.

A complete understanding of the present invention may be obtained by reference to the accompanying drawings, when taken in conjunction with the detailed description thereof and in which:

FIG. 1 is a block diagram of the memory controller of the present invention;

FIG. 2 is a block diagram of the error logging subsystem in greater detail;

FIG. 3 is a block diagram of the scrubbing mechanism in greater detail;

FIGS. 4a-4f and 4g-4h taken together represent a schematic diagram of error flag RAMs and associated circuitry for use in the error logging subsystem; and

FIG. 5a and b is a schematic diagram of the translator for use in the error logging subsystem.

Referring now to FIG. 1, there is shown the memory controller of the present invention. A bank or matrix of DRAM chips 10 contains a plurality of 1 megabit memory storage chips. It should be understood that any number of DRAM chips of any suitable capacity and any suitable configuration can be used in this multi-chip memory subsection. In the preferred embodiment, 704 DRAM chips are divided into four memory pacs, each containing 176 chips. Connected to DRAM chips 10 is error detection and correction (EDAC) logic 12, such as is available on an IC chip from Advanced Micro Devices, Inc. as Model No. Am2960.

EDAC 12 is connected to DRAM chips 10 by means of a 16-bit parallel data bus 14. Also connecting EDAC 12 to DRAM chips 10 is a 6-bit parallel bus 16 for transferring check bits between EDAC 12 and DRAM chips 10. The check bits, not shown, are generated by EDAC 12 during write operations to DRAM chips 10. Check bits are read back and decoded during read operations from DRAM chips 10. EDAC 12 can generate a 2-bit error signal, when required, over a line 18 connected thereto and provided expressly for that purpose.

EDAC 12 also generates syndrome bits over a 6-bit parallel syndrome bit bus 20. Syndrome bits are generated in the course of error processing, as mentioned hereinabove.

Connected to syndrome bit bus 20 and to error line 18 is a DRAM error or failure log device 22. Failure log 22 includes a bank of error flag RAMs (EFRs), not shown in this Figure, but discussed in greater detail hereinbelow, and a microprocessor, not shown.

An address bus 34 is connected to DRAM chips 10 and to DRAM failure log 22. Also connected to address bus 34 by means of an extension bus 34a is an alpha particle scrub device 32, used to scrub (read data, correct if necessary, then write to) memory locations in DRAM chips 10. Alpha scrub device 32 initiates a scrub cycle over a line 35, which is connected between alpha scrub device 32 and EDAC 12. Alpha scrub device 32 generates an effective 26 bit address over address bus 34, which address is applied to failure log 22.

Referring now also to FIG. 2, there is shown a block diagram of the DRAM failure log device 22 (FIG. 1) in greater detail. A processor 26 is provided with a microprocessor 40, such as is available from Motorola as Model No. MC68000. Microprocessor 40 handles power up initialization and testing, diagnostic testing, and error logging. Connected to microprocessor 40 is a ROM 42 that contains instructions for microprocessor 40. Random access memory 44 is connected by means of a bidirectional line 46 to microprocessor 40.

A translator PROM 50, the function of which is described in greater detail below, is connected to processor 26 by means of a 6-bit parallel address bus 52. The output of translator PROM 50 is applied to error flag RAMs (EFRs) address multiplexers 54. The outputs of EFR multiplexers 54, in turn, are applied to EFRs 24. EFR data multiplexers 56 are connected to the data output ports 62 of processor 26 over line 58.

Data is read from EFRs 24 by means of data output ports 60 and applied to data ports 62 of processor 26 over line 59, provided for that function.

A flip-flop device 64 is connected to EFR address and EFR data multiplexers 54 and 56, respectively, for allowing one of the multiplexers (e.g., data 56) to receive data while the other (e.g., address 54) is being unloaded.

So called "glue logic" 66 is provided to reformat signals generated by flip-flop 64, EDAC 12 and microprocessor 40 into protocol suitable for address and data multiplexers 54 and 56 and EFR 24.

Referring now also to FIG. 3, there is shown a block diagram of alpha scrub device 32 (FIG. 1) in greater detail. A REFRESH signal is generated over REFRESH line 33 and applied to a scrub cycle initiator 36, which generates a SCRUB CYCLE instruction over line 35. The REFRESH signal is also applied to address counters 38, which generate signals that are transferred to address bus 34a.

In the technique known as alpha-scrubbing, every 0.96 msec one memory location in DRAM 10 (FIG. 1) is read by the alpha scrub circuitry. If an error is found, EDAC 12 corrects it and the corrected data is written back into memory 10. Therefore, every memory location on a controller with a capacity of 256 Megabytes is scrubbed every 17.9 hours. Furthermore, one memory row of DRAM chips 10 is scrubbed every 67.11 minutes. The monitoring function of the present invention can determine at the soonest possible moment when a given DRAM 10 has been scrubbed. For each error, 6 syndrome bits and 4 memory row selects are generated which identify to the DRAM or row of DRAMs where the error occurred. These bits are used to record each hard and soft error in EFR 24 (FIG. 2). Monitoring processor 26 periodically interrogates EFRs 24 and records the errors in failure log 22.

As mentioned previously, EDAC 12 generates syndrome bits 20 which are fed into failure log 22. These syndrome bits 20 are encoded by EDAC 12 such that the data bit 14 in error on a single bit error can be determined by processor 26 with the use of a lookup table stored in processor's ROM 42. Additionally, the four row selects contained in address bus 34 are used to complete the error isolation to one of 704 DRAM chips 10.

In the case of multiple bit errors, syndrome bits 20 cannot isolate which data bits 14 are in error. The only isolation that can be accomplished is the determination of the DRAM chip row in which the error occurred. This determination can be made from the four row selects contained in address bus 34.

Processor 26 transfers the contents of failure log 22 to the system bus, not shown, using a handshake scheme. The transferred failure log 22 can contain up to 704 single bit errors and up to 32 double bit errors. Each single bit error corresponds to a single DRAM chip 10 and each double bit error corresponds to 22 DRAM chips.

A scrub error occurs when:

(1) an error is detected in the given DRAM 10,

(2) the DRAM is subsequently scrubbed, and

(3) another error is subsequently detected in the given DRAM 10.

Scrub error detection is used to distinguish between a DRAM which experiences a one time error and a DRAM which experiences recurring soft errors or hard errors. In essence, this technique eliminates counting multiple detections of a soft error before the DRAM has been scrubbed.

Referring now also to FIGS. 4a-4f, there is shown a schematic diagram of error flag RAMs (EFRs) 24 (FIG. 2) and associated circuitry for rendering them operational. FIGS. 4a-4c and 4d-4g are very similar. FIGS. 4a-4c show the connection of address multiplexers 54a and 54b and data multiplexer 56a (which includes certain address bits) to EFRs 24 for first EFR odd and even sections, 24a and 24b, respectively. Similarly, FIGS. 4d-4f show multiplexers 54c, 54d and 56b for second EFR odd and even sections 24c and 24d, respectively.

Address multiplexers 54a and 54b are connected to EFRs 24a and 24b as shown in the FIGURE. EFR data multiplexer 56a is connected to first portion error flag RAMs 24a and 24b.

Connected to the data outputs of EFRs 24a and 24b is a tri-state driver 66a. Also connected to driver 66a is the output of a NOR gate 68a.

The major distinction between the two sets of FIGS. 4a-4c and FIGS. 4d-4f, besides relating to the first EFRs section 24a, 24b and the second EFRs section 24c, 24d, is that NOR gate 68a is replaced in FIGS. 4d-4f by an OR gate 68b. Accordingly, for the sake of simplicity only the operation of FIGS. 4a-4c is discussed herein since the subsystem shown in FIGS. 4d-4f and the operation thereof is a mirror image of that shown in FIGS. 4a-4c.

Another distinction between operation of the subsystem shown in FIGS. 4a-4c and that shown in FIGS. 4d-4f is that the control signals applied to the Write Enable (WE) and Chip Select (CS) ports of EFRs 24a and 24b differ from those applied to ports of EFRs 24c and 24d and are distinguished by unique signal nomenclature. In FIGS. 4a-4c, signals input to WE and CS of EFR 24a are EFWE10-00 and EFCS10-00, respectively, as opposed to corresponding input signals for EFR 24c (FIGS. 4d-4f) which are, respectively, EFWE20-00 and EFCS20-00 (emphasis added). Similarly, signals input to EFR 24b (FIGS. 4a-4c) are EFWE1E-00 and EFCS1E-00, respectively, as opposed to corresponding signals input to EFR 24d (FIGS. 4d-4f), which are EFWE2E-00 and EFCS2E-00.

The Select inputs of multiplexers 54a, 54b, 56a and of multiplexers 54c, 54d, 56b are individually addressed by flip-flop 64 (FIG. 2).

Referring now also to FIG. 5, translator PROM 50 and associated circuitry are shown in greater detail. In particular, an EFR control programmable array logic chip 72 generates control input signals for EFRs 24 (FIG. 2). Translator PROM 50 translates addresses from microprocessor 40 to appropriate addresses for EFRs 24. Flip flops along with the AND gate select which EFR 24 is gathering data and which EFR is accessible by microprocessor 40.

Two categories of DRAM 10 failures must be isolated. The first and most obvious is the hard error. If a memory cell, not shown, is stuck in one state, it is inoperative and the DRAM containing the cell should be replaced. The second category of failures is a DRAM which experiences an abnormal sensitivity to soft errors. A DRAM which exhibits this condition is a system hazard for two reasons: first, two DRAMs failing in the same word read from memory results in an uncorrectable error and a DRAM having a high frequency of soft errors increases the chance of such occurrence; second, a DRAM exhibiting this condition is degenerating and is likely to begin exhibiting hard errors.

In DRAMs 10, normal soft errors are likely to occur 5 times per million operating hours. This category of failure is normal and is not considered a problem.

In operation, processor 26 reserves 16 bits of data for each DRAM 10 and for each row of DRAMs on the controller. The first time an error is found on a DRAM 10, it is recorded in EFRs 24 and transferred to a compiled error history in the processor's RAM 44 (FIG. 2). Until DRAM 10 has been completely scrubbed by alpha-scrubbing mechanism 32, no further soft errors are transferred to this error history. (Otherwise, if the failing DRAM were read 100 times before it could be scrubbed, 100 failures would be recorded when only one error occurred.) By waiting until after the error has been scrubbed, a soft error is logged only once.

After DRAM locations 10 have been scrubbed, DRAM logging is once again enabled. If another error occurs in the same DRAM location 10, an error count is incremented in the processor's RAM 44 (FIG. 2) and logging is again disabled until DRAM 10 has been scrubbed. By monitoring these counts a faulty DRAM can be discovered.

The system logs all errors in failure log 22 on any read access from memory 10, including any errors encountered during alpha scrubbing. This technique provides a way of checking the validity of memory 10 not being used. Errors even in seldom used memory locations 10 are logged when the alpha scrubbing mechanism 32 attempts to scrub them. When an error is encountered during alpha scrubbing, it is recorded in the usual manner. When errors are encountered during subsequent alpha scrubs for a DRAM 10, the error count in the processor's RAM 44 for that DRAM location 10 increments, indicating improper operation.

A typical controller consists of a memory controller mother board and from 1 to 4 attached memory pacs. Each memory pac can contain from 4 to 8 rows of DRAM chips, each row consisting of 22 DRAM chips. Thus a memory controller board can have up to 704 DRAM chips.

The present invention monitors EDAC 12. Any error detected by EDAC 12 causes an error log to be updated with the location of the DRAM responsible for the error.

Failure log 22 lists both hard and soft types of errors.

System software can use failure log 22 to bypass failed memory, as is well known in the art; and field service personnel can use failure log 22 to find and replace a failed memory pac quickly. A memory pac/board need not be replaced due to a soft error, since a memory location with a soft error may be reusable after scrubbing, thus avoiding cost to return it to the factory needlessly. Factory replacement time is reduced since the problem chips are preidentifiable, thus eliminating further testing to locate defective chips.

Since other modifications and changes varied to fit particular operating requirements and environments will be apparent to the skilled in the art, the invention is not considered limited to the example chosen for purposes of disclosure, and covers all changes and modifications which do not constitute departures from the true spirit and scope of this invention.