Memory interleaving

[0001] A channel generally refers to a pathway between a computer system and other computing systems and/or other devices. Each of a computing system's channels is an independent unit that can transfer data at the same time as other channels. Each channel is typically assigned a segment of memory address space and can transfer data corresponding to its assigned memory address space. In this way, the computing system's processor may access different segments of memory via different channels without idling while the memory completes an access to one segment before beginning another memory access. This type of memory access is generally called interleaving.

[0002] FIG. 1 is a block diagram of an example channel control system.

[0003] FIG. 2 is a flowchart of an example process of memory interleaving.

[0004] FIG. 3 is a flowchart of an example process of determining whether a region is within a channel.

[0005] FIG. 4 is a flowchart of an example process of reducing an address.

[0006] FIG. 5 is a block diagram of an example address reduction.

[0007] FIG. 6 is a flowchart of an example process of adjusting an address.

[0008] FIG. 7 is a flowchart of an example process of address remapping.

[0009] FIG. 8 is a block diagram of an example machine system.

[0010] Referring to FIG. 1, an example channel control system 100 can interleave access to a memory using channel controllers 102(1)-102(X) each associated with one channel. X can equal any positive whole number greater than one, including non-power of two positive whole numbers (three, five, six, etc.). Whatever the value of X, interleaving may be performed without using one or more bits of a memory address to indicate which channel to use in accessing the memory. Because no address bits need to be used in channel selection, the number of channels for interleaving is not restricted to a power of two number of channels as in traditional channel interleaving. Addresses may be mapped to the channel controllers 102(1)-102(X), and accesses to the memory may be concurrently made for addresses mapped to different channels.

[0011] Each of the channel controllers 102(1)-102(X) includes one of X match detect mechanisms 104(1)-104(X) and one of X address and count remapping mechanisms 106(1)-106(X). Each of the channel controllers 102(1)-102(X) receives information regarding a region to access in a memory and determines whether its associated channel is mapped to access data included in that region. The channel controllers 102(1)-102(X) typically make their determinations in parallel although they could process the information according to some pre-programmed priority or ordering scheme.

[0012] The information received by the channel controllers 102(1)-102(X) can include a data pair including a start address indicating where to begin accessing data in the memory and a count indicating how much data to access starting at the start address. The count is typically provided in bytes although any data measurement or size indicator may be used. The start address and the count define a region beginning at the start address and extending for a number of bytes indicated by the count (or as otherwise indicated by the count depending on how the count is provided).

[0013] Each of the match detect mechanisms 104(1)-104(X) determines if its associated channel is mapped to any of the addresses included in the region. Addresses may be mapped to the channels using a traditional scheme that assigned segments of address space to channels and/or using a scheme described further below that can spread adjacent addresses over the channels such that adjacent or nearby addresses may be spread over multiple channels. The channel may be mapped to access no addresses included in the region, all addresses included in the region, or some of the addresses included in the region. If a channel is mapped to some of the addresses included in the region, then part of the region lies within the channel and at least two channels are mapped to that region and may both access the region.

[0014] If one of the match detect mechanism 104(1)-104(X) determines that part of the region lies within its associated channel, then the one of the address and count remapping mechanisms 106(1)-106(X) associated with that channel determines a remapped start address and a remapped count indicating a part of the access for its associated one of the channel controllers 102(1)-102(X) to complete. The address and count remapping mechanisms 106(1)-106(X) may determine remapped addresses and remapped counts while the match detect mechanisms 104(1)-104(X) determine which channel or channels (if any) map to the region to save on processing time.

[0015] Once the channel controllers 102(1)-102(X) have determined if their associated channels map to the region, the appropriate one or ones of the channels may access the data in indicated region. In this way, addresses may be mapped to channels such that multiple channels may be used to access data at relatively close addresses. Furthermore, an interleave scheme can include using two or more channels, including a non-power of two number of channels.

[0016] In other examples, the match detect mechanisms 104(1)-104(X) and/or the address and count remapping mechanisms 106(1)-106(X) may be external to the channel controllers 102(1)-102(X). Furthermore, some or all of the channel controllers 102(1)-102(X) may use the same one or ones of the match detect mechanisms 104(1)-104(X) and/or the address and count remapping mechanisms 106(1)-106(X).

[0017] FIG. 2 shows an example process 200 of memory interleaving. In the process 200, the channel controller 102 receives 202 address and count information about a region to access in memory. (Each of the channel controllers 102(1)-102(X) receives the same information.)

[0018] The channel controllers 102(1)-102(X), the match detect mechanisms 104(1)-104(X), and the address and count remapping mechanisms 106(1)-106(X) each function similar to their like-named counterparts. For simplicity, the match detect mechanism 104(1) (match detect 104) and the address and count remapping mechanism 106(1) (remap 106) included in the channel controller 102(1) (channel controller 102) are used as representative examples.

[0019] The match detect 104 determines 204 how much of the region lies with the channel associated with the channel controller 102, channel one in this example. An example of how the match detect 104 may make such a determination is discussed further below.

[0020] If none of the region lies within the channel, then the process 200 ends 206 because the data to be accessed cannot be accessed through that channel.

[0021] If the region lies entirely within the channel, then the channel controller 102 triggers 208 an access of an amount of data equal to the count starting at the start address through the channel. The channel controller 102 may itself retrieve the data.

[0022] If the region lies partially within the channel, then the remap 106 adjusts 210 the address and the count to an adjusted address and adjusted count so that the channel only accesses data in its assigned region. The channel controller 102 may then trigger 212 an access of an amount of data equal to the adjusted count starting at the adjusted start address through the channel. The channel controller 102 may itself retrieve the data. For example, the start address may be indexed within one channel but the count extends the region into an area mapped to another channel, and the other channel needs to adjust the start address to reflect where the region begins in its indexed area. The adjusted start address may be the same as the start address while the adjusted count differs from the count in the case where the count extends beyond a channel's mapped area.

[0023] FIG. 3 shows an example determination process 300 of how the match detect 104 may determine whether and how much of a region lies within a channel. In the determination process 300, the match detect 104 calculates 302 an upper address of the region corresponding to the end of the data included in the region. The match detect 104 may calculate the upper address as the start address plus the count minus one. The one is subtracted to account for the data at the start address.

[0024] The match detect 104 reduces 304 the upper address and the start address to two bits each. With the addresses reduced to two bits each, the match detect 104 can determine if the region lies at least partially within the channel associated with the match detect. The addresses may be reduced to two bits each because in an example including three channels (X equals three), each of the channels can be represented by a different two bit combination (e.g., 01 for channel one, 10 for channel two, and 11 for channel three) and the two bit representation of the channel including the address can be used in reducing the address to help determine whether the channel associated with the match detect 104 is mapped to the region. If the system includes more than three channels, the addresses may be reduced to more than two bits because more than two bits may be necessary to represent each of the different channels. An example of how the match detect 104 may perform a reduction is discussed further below.

[0025] The match detect 104 determines 306 if the channel number associated with the match detect 104 (one in this example) matches either the reduced upper address or the reduced start address. If not, then none of the region lies within the channel.

[0026] If either reduced address matches the channel number, then the match detect 104 determines 308 if both the reduced upper address and the reduced start address match the channel number. If so, then the entire region lies within the channel. If not, then only one of the reduced addresses matches the channel number and only part of the region lies within the channel. The match detect 104 determines 310 if the reduced start address matches the channel number. If so, then a lower part of the data transfer (access), a part starting at the start address, lies within the channel. If not, then the match detect 104 concludes that an upper part of the data transfer, a part starting at an address higher than the start address and continuing through the upper address (unless the starting address of the upper part is the upper address in which case the upper part of the data transfer includes only the upper address), lies within the channel. The match detect 104 in this example checks for a match with the start address and makes an assumption about an upper address match based on that determination, but the match detect 104 may check the upper address and make an assumption about the start address in other examples.

[0027] FIG. 4 shows an example reduction process 400 that the match detect 104 may use to reduce an address. In the reduction process 400 generally, as shown in an example address reduction 500 in FIG. 5, the match detect 104 takes five levels of gating to reduce a thirty-one bit address 502 to a two bit output number 504. The match detect 104 may ignore one or more bits included in the address 502 that are not part of the start or upper address. In this example showing a start address, the match detect 104 ignores the seven bits included in the address that represent a byte offset (the interleave byte size) and considers the twenty-four bit start address. One of the gating levels takes as an input a channel number 506 of the channel that holds the address.

[0028] Referring back to FIG. 4, the match detect 104 recodes 402 the address using 2-bit to 2-bit recoding to produce a first number of bits including as many bits as the address (twenty-four bits) according to the following table:

	Input		Output
	n 1	n	2	1
	0	0	0	0
	0	1	0	1
	1	0	1	0
	1	1	0	0

[0029] The match detect 104 reduces 404 the first number of bits using a 4-bit to 2-bit reducer to produce a second number of bits including the half as many bits as the first number of bits (twenty-four bits reduced to twelve bits) according to the following table:

		00	01	10
	00	00	01	10
		(0)	(1)	(2)
	01	01	10	00
		(1)	(2)	(0)
	10	10	00	01
		(2)	(0)	(1)

[0030] The match detect 104 reduces 406 the second number of bits according to the above 4-bit to 2-bit reducer table to produce a third number of bits including half as many bits as the second number of bits (twelve bits reduced to six bits).

[0031] The match detect 104 reduces 408 the third number of bits plus the channel number including the address according to the above 4-bit to 2-bit reducer table to produce a fourth number of bits (eight bits reduced to four bits). If the match detect 104 did not include the channel number as an input in this reduction, there would be two unused inputs. The match detect 104 may receive the channel number from the remap 106, may look it up in an address mapping table that indexes addresses and their corresponding channel numbers, or may otherwise acquire the channel number.

[0032] The match detect 104 adds 410 the two resulting 2-bit recoded numbers. Sums of zero, one, and two do not change while a sum of four adds to zero. This addition produces the final two-bit output of the reduction.

[0033] The match detect 104 determines 412 if the final output indicates a match with the channel associated with the match detect 104. A final output of zero (00) indicates a match while a one (01) or a two (10) indicates a non-match. Including the channel number associated with the address (the start address or the upper address) as an input in the reduction process allows the match detect 104 to make a match determination from the final bit output of the reduction process.

[0034] FIG. 6 shows an example adjustment process 600 that the remap 106 may use to adjust the start address and the count if the match detect 104 determines that only part of the region lies within its associated channel (see FIG. 2).

[0035] In the adjustment process 600, the remap 106 calculates 602 a boundary address representing an address an interleaved amount beyond the start address. The interleaved amount is typically included in the address that includes the start address (see, e.g., the address 502 in FIG. 5), and in this example equals one hundred twenty-eight bytes. The remap 106 may calculate the boundary address by performing a logical AND operation on the start address and 0xFFFFFF80 and adding 0x80 (the interleaved amount).

[0036] The remap 106 also calculates 604 a lower count representing a number of bytes between the start address and the boundary address. The remap 106 may calculate the lower count by subtracting the start address from the boundary address.

[0037] The remap 106 also calculates 606 an upper count representing a number of bytes between the boundary address and a stop address (the start address plus the count). The remap 106 may calculate the upper count by subtracting the boundary address from the stop address and adding one (the one accounts for the fact that the lower count includes the boundary address).

[0038] If the channel associated with the remap 106 owns a lower part of the transfer (e.g., as determined through the determination process 300), then the remap 106 considers 608 the adjusted start address as the start address and the adjusted count as the lower count. If not (i.e., if the channel associated with the remap 106 owns an upper part of the transfer), then the remap 106 considers 610 the adjusted start address as the boundary address and the adjusted count as the upper count.

[0039] FIG. 7 shows an example remapping process 700 that the remap 106 may use to index an address within a channel. The remap 106 finds 702 a longest string of consecutive address bits with a value of one in the address to be indexed. The remap 106 may begin searching for the longest string with the least significant bit in the address. Once found, the remap 106 drops 704 the longest string from the address, right justifies 706 the remaining bits in the address. If the channel includes a power of two number of memory locations (e.g., addresses), then the remap 106 fills 708 the vacated bit positions (the most significant bits) with ones. This filling essentially adds three quarters to the remaining space of channels, e.g., starts the filling three quarters over from the address. The resulting bits form the remapped address.

[0040] If the channel is of a non-power of two, then a shifting of three quarters is not typically the same size as for a power of two channels (e.g., may not be two bit positions but some other number). The remap 106 determines 710 where to start adding ones in the vacated positions. The remap 106 may make such a determination by consulting one or more lookup tables including constants that indicate start addresses where the remap 106 may start filling 712 in ones. Each of the tables may be included in the remap 106 or be otherwise accessible to the remap 106 and may include constants for any number of channels and for any number of shifted bits. The resulting bits form the remapped address.

[0041] For example, in a three channel system, the lookup tables may be as follows, where the constant values are shown in hexadecimal format and where KO represents three quarters of a channel, K1 represents three quarters plus three quarters of a channel K2 represents three quarters plus three quarters plus three quarters of a channel, etc. The tables show constant values up to 768 MBytes, but the values in the table may be scaled as appropriate for a larger number of MBytes.

	MBytes
	32	64	96	128	160	192	224	256
K11	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A
K10	N/A	N/A	N/A	N/A	13FFFF	17FFFF	1BFFFF	1FFFFF
K9	N/A	7FFFF	BFFFF	FFFFF	13FFFE	17FFFE	1BFFFE	1FFFFE
K8	3FFFF	7FFFE	BFFFD	FFFFC	13FFFB	17FFFA	1BFFF9	1FFFF8
K7	3FFFC	7FFF8	BFFF4	FFFF0	13FFEC	17FFE8	1BFFE4	1FFFE0
K6	3FFF0	7FFE0	BFFD0	FFFC0	13FFB0	17FFA0	1BFF90	1FFF80
K5	3FFC0	7FF80	BFF40	FFF00	13FEC0	17FE80	1BFE40	1FFE00
K4	3FF00	7FE00	BFD00	FFC00	13FB00	17FA00	1BF900	1FF800
K3	3FC00	7F800	BF400	FF000	13EC00	17E800	1BE400	1FE000
K2	3F000	7E000	BD000	FC000	13B000	17A000	1BE9000	1F8000
K1	3C000	78000	B4000	F0000	12C000	168000	1A4000	1E000
K0	30000	60000	9000	C0000	F0000	120000	150000	180000
	MBytes
	288	320	352	384	416	448	480	512
K11	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A
K10	23FFFF	27FFFF	2BFFFF	2FFFFF	33FFFF	37FFFF	3BFFFF	3FFFFF
K9	23FFFD	27FFFD	2BFFFD	2FFFFD	33FFFC	37FFFC	3BFFFC	3FFFFC
K8	23FFF7	27FFF6	2BFFF6	2FFFF4	33FFF3	37FFF2	3BFFF1	3FFFF0
K7	23FFDC	27FFD8	2BFFD4	2FFFD0	33FFCC	37FFC8	3BFFC4	3FFFC0
K6	23FF70	27FF60	2BFF50	2FFF40	33FF30	37FF20	3BFF10	3FFF00
K5	23FDC0	27FD80	2BFD40	2FFD00	33FCC0	37FC80	3BFC40	3FFC00
K4	23F700	27F600	2BF500	2FF400	33F300	37F200	3BF100	3FF000
K3	23DC00	27D800	2BD400	2FD000	33CC00	37C800	3BC400	3FC000
K2	237000	276000	2B5000	2F4000	333000	372000	3B1000	3F0000
K1	21C000	258000	294000	2D0000	30C000	348000	384000	3C0000
K0	1B0000	1E0000	210000	240000	270000	2A0000	2D0000	300000
	MBytes
	544	576	608	640	672	704	736	768
K11	43FFFF	47FFFF	4BFFFF	4FFFFF	53FFFF	57FFFF	5BFFFF	5FFFFF
K10	43FFFE	47FFFE	4BFFFE	4FFFFE	53FFFE	57FFFE	5BFFFE	5FFFFE
K9	43FFFB	47FFFB	4BFFFB	4FFFFB	53FFFA	57FFFA	5BFFFA	5FFFFA
K8	43FFEF	47FFEE	4BFFED	4FFFEC	53FFEB	57FFEA	5BFFE9	5FFFE8
K7	43FFBC	47FFB8	4BFFB4	4FFF80	53FFAC	57FFA8	5BFFA4	5FFFA0
K6	43FEF0	47FEE0	4BFED0	4FFEC0	53FEB0	57FEA0	5BFE90	5FFE80
K5	43FBCO	47FB80	4BFB40	4FFB00	53FAC0	57FA80	5BFA40	5FFA00
K4	43EF00	47EE00	4BED00	4FEC00	53EB00	57EA00	5BE900	5FE800
K3	43BC00	47B800	4BB400	4FB000	53AC00	57A800	5BA400	5FA000
K2	43BC00	47B800	4BB400	4FB000	53AC00	57A800	5BA400	5FA000
K1	3FC000	438000	474000	4B0000	4EC000	528000	564000	5A0000
K0	330000	360000	390000	3C0000	3F0000	420000	450000	480000

[0042] The remap 106 indexes all available addresses, e.g., all addresses handled by the channel control system 100 (see FIG. 1). As a simple example, in a system including three channels with eight addresses each and twenty-four addresses, address locations in the channels would be remapped as shown:

	Index in
	Channel	Channel 0	Channel 1	Channel 2
	0	0	1	2
	1	6	4	5
	2	9	10	8
	3	12	13	14
	4	18	16	17
	5	21	22	20
	6	3	7	11
	7	15	19	23

[0043] Although the processes in FIGS. 2, 3, 4, 6, and 7 are each described with reference to the elements included in the example channel control system 100 of FIG. 1, these or similar processes, each including the same, more, or fewer elements, reorganized or not, may be performed in the channel control system 100 or in another, similar system. Additionally, the processes in FIGS. 2, 3, 4, 6, and 7 are described systems using 128-byte interleaving and including three channels and thirty-one bit addresses, but the processes may be used (with any appropriate modifications) for any size interleaving, for any number of channels, and for addresses of any size. Furthermore, the processes in FIGS. 2, 3, 4, 6, and 7 need not all be performed together in the same system but may find applicability alone or in partial combination with two or more of the other processes.

[0044] Referring to FIG. 8, a machine 800 includes a processing system 802 including a memory controller 804 that may include or be configured similar to the channel control system 100 (see FIG. 100). The elements described with reference to FIG. 8 can be implemented in a variety of ways.

[0045] A consuming device 806 may need information stored at a location in a main memory 808. The consuming device 806 typically connects to the machine 800 via input/output (I/O) ports, bays, and/or slots 810 and requests data from the main memory 808 through a chipset 812 and a processor 814.

[0046] The memory controller 804 may control access to and map addresses in the main memory 808 as described above, interleaving reads/writes using multiple memory channels. The main memory 808 can include any memory mechanism capable of storing data. Examples of the main memory 808 include random access memory (RAM) such as dynamic RAM or static RAM, read only memory (ROM), flash memory, tapes, disks, buffers, and other types of similar storage mechanisms. The main memory 808 may include one storage mechanism, e.g., one RAM chip, or any combination of storage mechanisms, e.g., multiple RAM chips. For example, memory may include SDRAM. SDRAM generally refers to a type of DRAM that can run at much higher clock speeds than conventional memory. SDRAM can synchronize itself with a bus associated with a processor included in the computing system (e.g., the processor 814). DDR-SDRAM generally refers to a type of SDRAM that supports data transfers on both edges of each clock cycle (the rising and falling edges), effectively doubling the memory's data throughput.

[0047] The machine 800 can include any mechanism or device capable of processing data. Examples of the machine 800 include workstations, stationary personal computers, mobile personal computers, servers, personal digital assistants, pagers, telephones, and other similar mechanisms and devices.

[0048] The consuming device 806 can include an I/O device, network interface, or other mechanism that may communicate with or be included in the machine 800. I/O devices generally include devices used to transfer data into and/or out of a computer system. Examples of I/O devices include mice, keyboards, printers, display devices such as monitors, disk drives, graphics devices, joysticks, paddles, Zip drives, scanners, CD drives, DVD drives, modems, cameras, video devices, microphones, and other similar types of internal, external, and internal/external devices. One consuming device is shown but the machine 800 may communicate with more than consuming device.

[0049] The I/O ports, bays, and/or slots 810 may include any mechanisms or interfaces capable of connecting one or more consuming devices to the machine 800. For example, the I/O ports, bays, and/or slots 810 may include peripheral component interconnect (PCI) slots, parallel ports, serial bus ports, disk drive bays, and other similar types of mechanisms and interfaces.

[0050] The processor 814 can include any processing mechanism such as a microprocessor or a central processing unit (CPU). The processor 814 may include one or more individual processors. The processor 814 may include a network processor, a general purpose embedded processor, or other similar type of processor.

[0051] The chipset 812 can include any number of chips/integrated circuits that can provide interfaces between the machine's subsystems.

[0052] Instructions and data are typically communicated to and from the main memory 808 in blocks. A block generally refers to a collection of bits or bytes communicated or processed as a group. A block may include any number of words, and a word may include any number of bits or bytes.

[0053] Data can be communicated between elements on communication links. The communication links can include any kind and any combination of communication links such as buses (of any type and size), physical ports, wireless links, and other similar links. For bus communication links, the buses can have any width, e.g., sixteen bits, thirty-two bits, sixty-four bits, etc, and may run at any speed, e.g., thirty-three Mega Hertz (MHz), 100 MHz, etc. A bus may have a sideband feature in which the bus includes parallel channels that can each simultaneously carry data and/or address information. Additionally, each of the communication links may include one or more individual communication links.

[0054] The memory controller 804 generally includes any mechanism capable of communicating with and managing the main memory 808. The memory controller 804 may include one or more chips and may be included in the chipset 812 or may be a mechanism independent from the chipset 812. The memory controller 804 may include any number of and any type of instructions, routines, applications, and/or programs.

[0055] Furthermore, the machine 800 is simplified for ease of explanation. The machine 800 may include more or fewer additional elements such as communication links, processors, storage mechanisms (buffers, caches, memories, databases, etc.), display mechanisms, consuming devices, bridges, chips, and other similar types of machine elements.

[0056] The techniques described here are not limited to any particular hardware or software configuration; they may find applicability in any computing or processing environment. The techniques may be implemented in hardware, software, or a combination of the two. The techniques may be implemented in programs executing on programmable machines such as mobile computers, stationary computers, personal digital assistants, and similar devices that each include a processor, a storage medium readable by the processor (including volatile and non-volatile memory and/or storage elements), at least one input device, and one or more output devices. Program code is applied to data entered using the input device to perform the functions described and to generate output information. The output information is applied to one or more output devices.

[0057] Each program may be implemented in a high level procedural or object oriented programming language to communicate with a machine system. However, the programs can be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or interpreted language.

[0058] Each such program may be stored on a storage medium or device, e.g., compact disc read only memory (CD-ROM), hard disk, magnetic diskette, or similar medium or device, that is readable by a general or special purpose programmable machine for configuring and operating the machine when the storage medium or device is read by the computer to perform the procedures described in this document. The system may also be considered to be implemented as a machine-readable storage medium, configured with a program, where the storage medium so configured causes a machine to operate in a specific and predefined manner.

[0059] Other embodiments are within the scope of the following claims.