Chipkill for a low end server or workstation

[0002] The applications entitled “Self-Healing Memory System for High Availability Server”, identified by HP Docket Number 10991629 and the inventor Michael B. Raynham and James G. Mathios, filed Oct. 28, 1999 and “Radial Arm Memory Bus for a High Availability Computer System” identified by HP Docket Number 10991678 and the inventors Michael B. Raynham and Hans Wiggers, filed Oct. 28, 1999 include subject matter related to the copending application.

[0003] Available memory systems are constantly expanding in size with time, with current server memory systems often being in the range of up to 64 Gbytes (approximately half a trillion storage bits) or larger for high end servers. Dependent on the customer requirements, the customer may choose to purchase a low end or high end server. Current low end server systems typically include one to four dual in line memory modules (DIMMs) while a high end servers typically include four or more DMMs. Typically, high end servers also include high availability features such as memory chip redundancy, hot swapping, and the ability to do chipkill error correction.

[0004] Referring to FIG. 1A shows a side view of a memory system for a low end server system 100. The memory system includes a CPU or memory controller 102 affixed to a motherboard 106 and two dual in line memory modules 108aand 108b. The two memory modules 108a-bshown each includes N memory devices 112 connected in parallel. Assuming for purposes of discussion that N is equal to eighteen, the eighteen memory devices 112a-N on each memory module 108aand 108bare connected to the memory controller 102 by a data bus 114, which includes board trace portions 116, a connectors 118 and a module trace portions 120.

[0005] FIG. 1B shows a block diagram of the memory structure of the memory modules of the low end server shown in FIG. 1A. In the embodiment shown in FIG. 1B, the data bus is 72 bits wide where 64 bits are used for data and 8 bits are used for error correction. Each of the eighteen memory devices on the memory module 108a-bis 4 bits wide and for a 256 Mbyte system each of the eighteen×4 SDRAMs is 32 Mbyte. The eighteen×4 memory devices are connected in parallel so that for each memory operation, the output onto the data bus 114 is 72 bits wide.

[0006] Referring to FIG. 1C shows a clock pulse for reading or writing to a memory location of the low end server shown in FIG. 1A. The memory controller reads a single word or memory location from a single memory module at a time. Assuming a single data rate (SDR) system and a read operation, the memory location in memory module 108 having the address 000000 is read at the clock edge t₁. The contents of the memory location is 72 bits wide. No memory operation occurs at clock edge t₂. A second memory location having the address location 000001 in memory module 108 is read at the clock edge t3.

[0007] FIG. 2A shows a side view of a memory system for a conventional high end server system 200 having eight DIMM modules. Similar to the low end server configuration shown In FIG. 1A, the memory system shown in FIG. 2A includes a CPU or memory controller 202 affixed to a motherboard 206. However, the high end configuration Includes eight dual in line memory modules 208a-hinstead of the two DIMMs 108a-bshown in FIG. 1A. The eight memory modules 208ashown each includes N memory devices 212, The memory controller 202 is connected to the eight memory modules 208a-hby a data bus 214, which includes board trace portions 216, connectors 218 and module trace portions 220.

[0008] FIG. 2B shows a block diagram of the memory structure of the high end server shown in FIG. 2A. In the high end server shown, the data bus is 144 bits wide where 128 bits are used for data and 16 bits are used for error correction. Preferably each memory module Includes eighteen memory devices (N=18), each memory device being 4 bits wide. For each memory module, the eighteen×4 memory devices are connected in parallel. Data is read from two memory modules simultaneously, so that for each memory operation, the output onto the data bus 214 is 144 bits wide.

[0009] Referring to FIG. 2C shows a clock pulse for reading or writing to a memory location of the high end server shown in FIG. 2A. The memory controller reads a single word or memory location from a single memory module at a time. Assuming a double data rate (DDR) system and a memory read operation, the memory location in memory module 208ahaving the address 000000 and the memory location in memory module 208ehaving the address 000000 are both read simultaneously at the clock edge t₁. A second memory location in memory module 208ahaving the address location 000001 and a second memory location having the address 000001 in memory module 208dare both read simultaneously at the clock edge t₂.

[0010] The current trend of increasing memory size is likely to continue. Microprocessor suppliers continue to supply higher speed CPUs. With increases in CPU speed come increased speed in the CPU bus and supporting I/O systems and a corresponding increase in server memory size per CPU since more users per CPU can be supported. As the size of memory systems increases, the probability of a memory bit failing, and thus the memory system failing, increases. Customers are demanding improved error correction features to deal with these increases in memory failures even for low end systems.

[0011] One error correction feature that was traditionally not supported in low servers is what is known in the industry as chipkill. The term chipkill traditionally refers to the ability to correct multiple bit errors in memory, where the multiple bit error is the width of the memory device. For example, for a 32 Mbit SDRAM that is 4 bits wide, a system that supports the chipkill function would be able to correct a 4 bit wide error in the memory device. Thus, the failure of an entire SDRAM chip organized in a×4 configuration in a system that supports chipkill would not cause the system to fail.

[0012] Chipkill is provided in high end chipsets, for example, by combining two DIMMs into a 144 bit bus that includes 128 data bits and 16 ECC bits where ECC stands for error correcting or error checking and correcting codes. The number of bits that can be corrected, typically depends on the number of ECC bits supported by the system. ECC or error correction code refers to a commonly used error detection and correction process that is typically based on a CRC (cyclic redundancy code) algorithm. CRC algorithms work so that when data is received, the complete data sequence (which includes CRC bits appended to the end of the data field) are read by a CRC checker. The complete data sequence should be exactly divisible by a CRC polynomial. If the complete data sequence is not divisible by a CRC polynomial, an error is deemed to have occurred.

[0013] Supporting the chipkill function based on an ECC process typically requires additional error correction bits where the number of bits corrected depends on the number of ECC bits supported by the system. For example, typically the CRC algorithm used to correct for a 4 bit wide memory organization requires more than the eight error correction bits that are provided by the low end server shown in FIG. 1A. Thus, to perform the chipkill function for a×4 organization, the low end server would require additional memory modules or devices to provide the additional ECC bits necessary to perform the required CRC algorithms. However, because the CRC algorithm typically used to correct for a 4 bit wide memory organization does not require more than the 16 error correction bits supported by the high end server shown in FIG. 2A, the high end server could support the chipkill function.

[0014] An alternative implementation available to low end server systems that wish to provide chip kill error correction is to provide a custom ASIC, such as that currently made commercially available from IBM Corporation, that provides specialized logic. The custom ASIC uses a specialized controller, and a memory wider than 72 bits, i.e. 80 bits to provide 16 ECC bits and uses fast SDRAM. This is twice the number of ECC memory normally required which limits the maximum memory size, since additional memory is required for ECC memory. Although the custom ASIC provides chipkill functionality, it is comparatively expensive and it would be preferable to use standard components if available. Further, currently available ASICs have limited memory sizes available and may not work at the 133 MHz speed required for next generation bus systems.

[0015] A memory system configuration and process that provides chipkill error detection in a low end server system that does not require costly custom ASIC chips or additional memory modules or devices to provide the additional ECC bits necessary to perform the required CRC algorithms is needed.

[0016] The present invention provides a process and memory configuration for providing chipkill error detection in a low end server without requiring non-standard components or additional memory devices. Typically for semiconductor memory devices, error detection and correction is performed using ECC error correction. To perform ECC correction, a predetermined number of bits in a specified memory device must be available for error correction. Typically, the number of error correction bits required for chipkill functionality is dependent on the width of the memory chip to be corrected and the CRC algorithm that is used. If the required number of error correction bits is not provided by the memory system, another option according to the present invention, is to perform doubleword operations for consecutive memory locations in a single DIMM module. Reading the memory module in double words basically provides the system with double its previous number of available ECC bits per data transfer. Since these low end systems would typically not have the number of error correction bits necessary to do the chipkill operation, this double word technique allows chipkill to be supported in low end systems which would normally not be able to perform the chipkill operation.

[0017] Using doubleword read and/or writes in combination with the described memory configuration allows the chipkill operation to be performed with minimal system performance impact and without increasing system costs for the cost sensitive low end server or workstation applications. The implementation according to the present invention does not require a custom memory module and is not limited certain memory module sizes as are today's commercially available custom ASICs.

[0018] For server applications, the double word implementation uses the DDR extra bandwidth that would otherwise be wasted, to “hide” the overhead for the chipkill operation. When used in the low end server applications, the performance penalty of reading double words (16 B data, 2 B ECC) is minimized by the bandwidth of the memory system being two times the system bus bandwidth. In a PC or workstation, this bandwidth is used for non-cache coherent AGP accesses to memory, not applicable in a server. Therefore the overhead for R-M-W used in byte write operation is hidden and bandwidth that would otherwise be wasted is used. Further, the present invention can provides the same chipkill functionality that it provides in a server system in a PC chipset with minor memory controller changes without requiring additional parts or custom parts.

[0019] In determining whether to implement the doubleword read, the system user implementing the system must first determine whether or not chipkill is a desired function. If the chipkill function is desired there should be a determination if the number of error correction bits need to support the target memory width of the chip is already provided by the error correction bits in a single memory address. If the number of error bits required is already provided, there is no need to a doubleword operation. Otherwise, the memory locations in a single memory module can be read or written in double words. Reading the memory module in double words basically provides the system with double its previous number of ECC per data transfer. This increased number of error correction bits may be adequate to perform a chip kill operation for the targeted chip width.

[0020] A further understanding of the nature and advantages of the present invention may be realized with reference to the remaining portions of the specification and the attached drawings.

[0021] FIG. 1A shows a side view of a memory system for a low end server system that includes two DIMM modules.

[0022] FIG. 1B shows a block diagram of the memory structure of the memory modules of the low end server shown in FIG. 1A.

[0023] FIG. 1C shows a clock signal for reading or writing to a memory location of the low end server shown in FIG. 1A.

[0024] FIG. 2A shows a side view of a memory system for a conventional high end server system 200 having eight DIMM modules.

[0025] FIG. 2B shows a block diagram of the memory structure of the high end server shown in FIG. 2A.

[0026] FIG. 2C shows a clock signal for reading or writing to a memory location of the low end server shown in FIG. 2A.

[0027] FIG. 3A shows a side view of a memory system for a low end server that includes two memory modules according to the present invention.

[0028] FIG. 3B shows a block diagram of the memory structure of the memory modules of the low end server shown in FIG. 3A.

[0029] FIG. 3C shows a clock signal for reading or writing to a memory location of the low end server shown in FIG. 3A.

[0030] FIG. 4A shows a flowchart of the steps for determining whether to implement chipkill in a low end server.

[0031] FIG. 4B shows a flowchart of the steps for implementing chipkill in a low end server which does not support chipkill when utilizing single word memory operations.

[0032] The present invention provides a method for providing chipkill functionality to a memory configuration that includes a memory controller and at least one memory module. The memory module preferably includes a plurality of memory devices having a target width. According to the present invention, double word memory operations are initiated by the memory controller in order to provide an adequate number of error correction bits to provide chipkill functionality.

[0033] Referring to FIG. 3A shows a side view of a memory system for a low end server system 300 that includes two memory modules. The memory system includes a CPU or memory controller 302 affixed to a motherboard 306 and two dual in line memory modules 308aand 308b. The two memory modules 308a-beach includes N memory devices 312 preferably connected in series. Assuming for purposes of discussion that N is equal to eighteen, the eighteen memory devices 312a-N on the memory module 308a-bare connected to the memory controller 302 by a data bus 314, which includes board trace portions 316, a connectors 318 and a module trace portions 320.

[0034] FIG. 3B shows a block diagram of the memory structure of the memory modules of the low end server shown in FIG. 3A. In the embodiment shown in FIG. 3B, although each DIMM modules is 72 bits wide, double word operations result in a data bus that has an effective width of 144 bits wide where 128 bits are used for data and 16 bits are used for error correction. Each of the eighteen memory devices on the memory modules 308a-bis 4 bits wide and for a 256 Mbit system each of the eighteen×4 SDRAMs is 32 Mbyte.

[0035] Referring to FIG. 3C shows a clock pulse for reading or writing to a memory location of the low end server shown in FIG. 3A. The memory controller 302 simultaneously reads a double word from a single memory module. Assuming a double data rate (DDR) system and a read operation, the memory location in memory module 308 having the address 000000 is read at the clock edge t₁. The contents of memory address 000000 is 72 bits wide. Because the present invention operates on double words, the contents of a second memory location having the address location 00000001 in the same memory module 308ais then read at the clock edge t₂. The contents of memory location 00000001 are also 72 bits wide. Thus, at clock edge t₂a total of 144 bits are read from the data bus (72 bits from memory location 00000 and 72 from memory location 000001.)

[0036] Using doubleword read and/or writes in combination with the described memory configuration allows the chipkill operation to be performed with minimal system performance impact and without increasing system costs for the cost sensitive low end server or workstation applications. The implementation according to the present invention does not require a custom memory module and is not limited certain memory module sizes as are today's commercially available custom ASICs.

[0037] For server applications, the double word implementation uses the DDR extra bandwidth that would otherwise be wasted, to “hide” the overhead for the chipkill operation. When used in the low end server applications, the performance penalty of reading double words (16 B data, 2 B ECC) is minimized by the bandwidth of the memory system being two times the system bus bandwidth. In a PC or workstation, this bandwidth is used for non-cache coherent AGP accesses to memory, not applicable in a server. Therefore the overhead for R-M-W used in byte write operation is hidden and bandwidth that would otherwise be wasted is used. Further, the present invention can provide the same chipkill functionality that it provides in a server system in a PC chipset with minor memory controller changes without requiring additional parts or custom parts.

[0038] Overall the chipkill features provided are similar to the high end system without incurring a performance or cost penalty. However, although the doubleword implementation is substantially transparent to the system with respect to performance and system costs, the doubleword implementation will modify system output in some cases compared to cases where single word operation occurs. For example, in the case of a DIMM connector data pin failure, the double read operation will cause a double bit error in the implementation according to the present invention but this can be corrected with CRC error correction methods.

[0039] With respect to replacement of DIMM modules, typically for low end systems it is not realistic to provide a hot spare modules and basic error correction techniques and the basic failure immunity can be relied upon until a service downtime interval to replace a failed DIMM or a hot spare can be provided. Depending on the CRC polynomial and SDRAM organization used, failures in the memory could be corrected and then rebuilt on the hot spare or simply corrected and then the DIMM replaced at the next annual server downtime period. In a 3-4 DIMM PC server system could use a DIMM for a hot spare if the complexity of handling “rebuild” is minimized.

[0040] FIG. 4A shows a flowchart 400 of the steps for determining whether to implement chipkill in a low end server. In determining whether to implement the doubleword read, it should first be determined by the system implementor determine whether or not chipkill is a desired function (step 404). If chipkill function is not desired (step 406), then single word operations that are used in prior art systems would be acceptable (step 408).

[0041] If chipkill functionality is desired (step 410), then there should be a determination if the number of error correction bits provided by a single memory location of the memory module is sufficient to support chipkill for desired width of memory device (step 414). Different algorithms for performing chipkill are well known and the particular algorithm chosen determines the number of error correction bits that are required. If the number of error correction bits required is already provided (step 418), then there Is no need to a doubleword operation as the single word operation would support chipkill functionality. If the number of bits required is not already provided (step 418), a check can also be performed to determine whether double number the bits would provide a sufficient number of error correction bits (step 420), if not (step 422) chipkill Is not possible using the double word implementation of the present Invention (step 24). Otherwise, the memory locations in a single memory module can be read or written in double words (step 428). Reading the memory module in double words basically provides the system with double its previous number of ECC per data transfer.

[0042] The method recited in FIG. 4A could also be used by the memory controller to determine whether to implement single word memory operations or a double word memory operations. In the preferred embodiment, the memory controller would determine whether a single word or double word implement would performed during system initialization. In one embodiment, the implementation chosen is be based upon a switch indicator where the switch indicator indicates whether or not enough bits are present in the single word configuration to perform the chipkill algorithm. Alternatively, the memory controller could switch automatically between single word memory operations and double word memory operations dependent upon a switch indicator.

[0043] FIG. 4B shows a flowchart of the steps for implementing chipkill in a low end server which does not support chipkill when utilizing single word memory operations. First, a determination is made that a doubleword memory implementation is desired (step 450). Preferably this step is performed before system initialization using steps 404-426 of the method shown in FIG. 4A. Next, if a double word memory implementation is desired a double word memory operation is performed (step 452). After the double word memory operation is performed to gather the required data, the chipkill algorithm is performed (step 454).

[0044] It is understood that the above description is intended to be illustrative and not restrictive. For example, the same principles recited in this application could be applied to mid range systems having a 144 bit bus (128 bits data, 16 bits ECC) where the system implementation requires a 288 bit system with 32 ECC bits and a cache line of 128 bytes compared to the described implementation. The double words in this case would be 256 bits data and 32 bits ECC with a burst length of 8. Further, the present invention may be used in combination with the configuration or processes disclosed in the related applications which are herein incorporated by reference. The scope of the invention should therefore not be determined with reference to the above description but instead should be determined with reference to the appended claims along with the full scope of equivalents to which such claims are entitled.