NAND flash module replacement for DRAM module

[0001] The invention relates to the field of memory module design and more particularly to the design of memory modules using non-volatile flash memory.

[0002] Many memory systems including Storage Subsystem Controllers commonly use DRAM chips mounted on small cards known as DIMMs as common memory modules. The DIMM modules can include several DRAM units, as well as buffers, clocking and a memory controller. The DIMMs are interfaced to the host CPU and Memory controllers via a variety of standard interface buses such as DDR, DDR2, DDR3 and FB-DIMM, which specify the addressing, data, clock, power and ground connections. DRAMs are volatile, so when used in systems such as storage subsystems that require non-volatility, additional batteries and battery monitoring and recharging circuitry must be added to DRAMs to make them non-volatile.

[0003] Flash memory is a specific type of EEPROM that is erased and programmed in relatively large blocks. NAND flash memory is becoming cheaper than DRAM on a cost per bit basis and is inherently nonvolatile so battery backup is not required. However, NAND flash operation differs from DRAM, for example, in that erase-operations, which are required before new data can be written, can only be performed on entire blocks of data. In this way NAND flash is similar to a disk drive. NAND flash is also typically slower than DRAM, and NAND flash has different and frequently proprietary chip level interfaces. Typical commercially available NAND flash memory chips require multiplexed input/output (I/O) pins for handling data and commands. A NAND flash memory device is typically accessed with bursts of data, for example, 512 bytes of data might be written to or read from the device in one burst. Thus, NAND flash cannot directly replace DRAM or any other devices that allow byte or word level random access.

[0004] Another limitation is that flash memory is subject to faster wear-out than DRAM, because it has a limited number of times that it can be erased and rewritten. This limitation can be offset in part by wear-leveling techniques which involve remapping blocks of the flash in order to prevent the erase-write operations from being concentrated in certain blocks.

[0005] FB-DIMM is a daisy chain architecture using a high speed serial interface with fully buffered dual inline memory modules. The architecture includes separate unidirectional read and write channels. All memory control for the DRAM devices resides in the host, including memory request initiation, timing, refresh, scrubbing, sparing, configuration access, and power management. Commercially available Advanced Memory Buffer (AMB) integrated circuits, for example those made by Intel Corporation, are compliant with the FB-DIMM Architecture and Protocol Specification. The AMB interface is responsible for handling the FB-DIMM channel and memory requests to and from the local DIMM and for forwarding requests to other DIMMs on the FB-DIMM channel. Fully Buffered DIMMs use commodity DRAMs isolated from the channel behind a buffer on the DIMM.

[0006] There is a need for nonvolatile memory card designs that are sufficiently similar to those using DRAM so that non-volatile memory can be added to existing systems with minimal changes in the system design.

[0007] An electronic memory module according to the invention provides non-volatile memory that can be used in place of a DRAM module without battery backup. One embodiment of the invention is an electronic memory module (device) with a NAND flash memory that can be used in place of a DRAM memory module in certain applications without requiring wear-leveling methods to be implemented elsewhere in the system. Alternatively a memory module according to the invention uses NAND flash but is largely compatible with the FB-DIMM standards and generally performs like a DRAM based FB-DIMM although at a slower speed. This embodiment of the invention can thereby provide non-volatile memory that can be incorporated into existing system designs that use FB-DIMMs with minimal design changes.

[0008] An embodiment of the invention includes embedded microprocessor with microcode that translates the FB-DIMM address and control signals from the system into appropriate address and control signals for the NAND flash. Wear-leveling, bad block management, garbage collection are preferably implemented by microcode executed by the microprocessor. The microprocessor, additional logic, and embedded memory provide the functions of a flash memory controller. The microprocessor memory preferably contains address translation (mapping) tables, bad block mapping table, free page queue and garbage collection information. Preferably error correction circuitry (ECC) is also included in the module to encode redundant information along with the data that is written to the NAND flash memory.

[0009] Read requests for DRAM addresses are mapped to the correct area in flash using the address translation tables which includes the page and offset in the flash memory. The data is read from the flash, and sent in serial form over the data out bus. Write requests are implemented by an update sequence in which the current page(s) containing the data to be updated are read into RAM, the new data is inserted into the page(s) at the predetermined offset point, the old page(s) is marked for subsequent garbage collection, and the new data block is programmed (written) to a new page in the flash. The wear-leveling algorithm results in the new data page being written to a different location. Write requests are mapped to “clean” pages in the flash that have been erased as part of the garbage collection routine. In one embodiment a queue of free pages is maintained in order of last use so that the least recently used page is selected when a new page is needed. Background garbage collection is preferably performed by microcode routines. When sufficient pages in a block are marked as dirty, pages still in use are moved to other blocks, the whole block gets erased and all of the pages are placed in the queue for reuse.

[0010] The address translation tables are preferably kept in RAM but must be recoverable from non-volatile memory after power has been removed or lost. Alternative embodiments include saving the tables in non-volatile memory when imminent loss of power is detected. In another embodiment the tables are rebuilt after a power loss from metadata saved in non-volatile memory whenever pages are written in the flash memory.

[0011] A memory module according to the invention will typically perform faster than data can be read from the disk in a disk drive, but slower than DRAM. The module according to the invention is most efficient when used in applications that do not require small block transfers or vast amounts of writing.

[0014] A preferred embodiment of the invention is a NAND flash based memory module that is plug compatible with existing DRAM and DIMM modules to enable NAND flash to be included in systems where DRAMs and DIMMs were previously used. Because NAND flash is slower, application workload may preclude its use in some systems. However, in large storage controllers for disk drives, the DRAM memory arrays are accessed in large blocks, frequently doing 16K-64K transfers at a time which makes the use of flash memory feasible. Likewise, most printer controllers do not require high data rates, so a flash memory module according to the invention could be used effectively.

[0015] Buffering architecture allows the output to be delayed while the appropriate block of data is read from the flash at less than DRAM speed. In general, where asynchronous interfaces are used from the host to the memory target, the system according to the invention can supply data from the slower NAND flash without causing system problems. Preferably the system has internal RAM so that it can prefetch entire pages from NAND flash for reads, and likewise, do read-modify-write operations for host driven write commands (preferably keeping the updated page so that each small write does not become a separate NAND page write).

[0016] FIG. 1 is a block diagram illustrating an embodiment of the invention in an FB-DIMM controller module 40 with external behavior similar to Advanced Memory Buffer (AMB) devices except as described below. The invention uses DRAM-NAND flash controller 43 to translate requests for DRAM read/write operations into NAND flash operations to allow the NAND flash array 60 to be used according to the invention with FB-DIMM module 40. The DRAM-NAND flash controller 43 supplies the control and address signals needed by the NAND flash array 60. These signals are different than those used by DRAM devices and may vary with different designs. For example, NAND Flash devices typically do not contain dedicated address pins. Data, commands and addresses are multiplexed onto the same pins and received by I/O control circuits and must be latched into registers. Addresses might be loaded using a 5-cycle sequence, for example. A NAND flash memory array 60 is typically programmed (written) in page-based operations and is erased in blocks comprising multiple pages. For example, a page is typically 2 or 4 KB which is only 1/32nd of a block.

[0017] Elements from prior art AMB devices included are pass-through and merging logic 41; pass-through logic 46; serializer 42; deserializer and decode logic 44; and Serial Presence Detect (SPD) EEPROM 61. The device interfaces to the DIMM bus and provides data transfers in the same way that an FB-DIMM module with DRAM memory does. The DRAM-NAND flash controller 43 will be described in more detail below.

[0018] While DRAM modules may incorporate ECC, some types of flash (especially MLC) require a higher rate of ECC coverage than DRAMs and without wear-leveling and garbage collection, repeated reads or writes to one location would eventually cause loss of data. The memory module according to the invention could also include extra (invisible to the system) capacity to provide for some blocks to go bad during the life of the device. The invention uses ECC encoder 47 to encode the data written to NAND flash 60 using prior art techniques which typically involve adding special redundant ECC information along with the original data as it is written out. The input for the ECC encoder 47 is supplied by deserializer and decode logic 44.

[0019] When the data is read back from the NAND flash 60, reciprocal ECC decoder 48 recalculates redundant ECC information using the data as read and compares the calculated ECC information with what was read from the memory. If a mismatch occurs the redundant ECC information read from memory is used to correct errors up to the capability of the ECC encoding. Errors that cannot be corrected by ECC are permanent and must be handled accordingly. The output from the ECC decoder 48 is fed to the serializer 42.

[0020] FIG. 2 is a block diagram illustrating an embodiment of a DRAM-NAND flash controller 43 for use with NAND flash chip array 60 according to an embodiment of the invention. The controller 43 can be implemented as a “system on a chip” with an embedded microprocessor 51, program code 52 and embedded RAM memory 53. The system responds to DRAM commands using NAND flash as the memory. Program code 52 must originate in non-volatile memory, but the instructions can be copied into RAM for execution. Embedded RAM memory 53 includes address translation tables 55 that are used to translate addresses in DRAM read/write requests into NAND flash addresses. The DRAM addresses are treated as logical addresses that are mapped into physical NAND flash addresses.

[0021] Addresses in read requests are mapped to the correct block in NAND flash using the address translation tables 55. The address for the NAND flash will typically include at least a page and offset portion to reference a particular starting point in the flash memory. In general the length of data to be read from NAND flash can be any number of bytes or words, including a single byte, up to the maximum size allowed in the particular design. The appropriate data from the flash is read, and the result is serialized over the data out bus as described above.

[0022] By providing dynamic logical to physical address translation, the use of the address translation tables 55 allows the system to achieve the important functions of bad block mapping and wear-leveling. When bad blocks are detected they will be permanently eliminated from being accessed preferably by removing them from the address translation tables. NAND flash chips may have bad blocks when they are shipped from the factory; therefore, special provision should preferably be made for initially mapping the bad blocks as part of the system setup. This mapping is not typically required for DRAM devices, so the FB-DIMM module 40 and in turn the controller 43 should be designed to have a special pin(s), jumpers, switches, commands, or initial power-up functions on that can be used to trigger the initial bad block mapping. The manufacturers of NAND flash typically mark the bad blocks in some arbitrary way such as writing particular data in designated locations to indicate that a block is bad. Once a device is being used for random data the arbitrary initial marking may no longer be valid; thus the initial processing of factory-marked bad blocks may only be done prior to actual use of the NAND flash for data.

[0023] Wear-leveling requires that the address mapping be changed frequently as the NAND flash is used. Any prior art wear-leveling algorithm can be used. In this embodiment the device preferably maintains a garbage collection table 56 that records each page of NAND flash memory that has been previously programmed with data, but is no longer being used. As pages of memory are updated, the pages are marked in the garbage collection table 56, but actual garbage collection typically only takes place when triggered by conditions that can be determined by the prior art. This process is dictated by the fact that NAND flash memory is designed to be erased at the block level. The simplest case occurs when every page in a block is marked as dirty which allows the entire block to be erased. However, it is typically desirable to consolidate or defragment partially used blocks when selected conditions arise. For example, if only one page out of 32 pages in a block is being used and the other 31 are dirty pages, the one page in use can be copied to another block, the old block erased and returned to a free list, and the address mapping tables updated accordingly.

[0024] After a block is erased in the garbage collection process, it is again available for each page to be re-written with new data. The new data is typically programmed into the memory one page at a time. In the embodiment shown in FIG. 2 the system maintains free page queue 57 which preferably records the order in which the pages were last used. This allows the microprocessor to select the least recently used page each time a new page is written to implement wear-leveling. By moving each page of memory to the least recently used page that is available when updating data in a page, wear-leveling is accomplished. The mapping in the address translation tables 55 is changed to point to the new physical flash page. Whatever wear-leveling is performed is transparent to the host system which does not know of or care what physical locations in the NAND flash are actually used for any particular unit of data.

[0025] The DRAM to NAND address translation tables 55 are preferably kept in RAM as a performance enhancement so that the microprocessor does not have to scan the NAND Flash array to determine the current mapping when each access command is processed. In order to preserve the mapping information across power-down cycles, in one alternative the complete table could be written to NAND flash memory when an impending power loss is sensed. Any prior art method of detecting power loss can be used. For example, one method is to provide power failure detection logic and a capacitor connected to the power input of the device to briefly provide power for the shutdown procedures. In another alternative the mapping data is recorded with each page of flash memory whenever a write occurs. In this scenario the corresponding (logical) host DRAM address for each NAND page of data is written in the extra bytes (metadata) associated with each page when the page is written. When the device is powered-up after power is lost or removed, in addition to reading the bad block information, the initialization procedures in the microcode can include rebuilding the translation table in RAM by reading the mapping metadata for each page in the NAND flash memory.

[0026] DRAM-type write requests are implemented by an update sequence in which the current page(s) contents are read into a RAM memory buffer 54, the new data is inserted into the page(s) in RAM at the appropriate offset point, the old page(s) is marked in garbage collection table 56 for subsequent processing, and the new data page(s) is programmed (written) in the flash memory at a new page(s) selected from the free page queue 57. The address translation table 55 will initially provide the page address and offset for the old data.

[0027] As noted the wear-leveling algorithm will result in the new data page being written to a different location in the NAND flash which requires updating the mapping tables. Write requests must be mapped to “clean” pages in the flash, and background garbage collection is performed by microcode routines. The old page is marked dirty and is eventually erased and marked for reuse by the garbage collection process by adding the pages to the free page queue 57.

[0028] The invention has been described with reference to specific embodiments, but one of ordinary skill in the art will readily recognize variations to the embodiments and those variations are within the spirit and scope of the present invention.