Method to improve system DMA mapping while substantially reducing memory fragmentation

[0001] 1. Technical Field

[0002] The present invention relates generally to computer systems and in particular to memory allocation in a computer system. Still more particularly, the present invention relates to a method, system and computer program product for proving improved DMA mapping.

[0003] 2. Description of the Related Art

[0004] Computer systems comprise a limited physical memory resource that is dynamically allocated to executing applications and input/output (IO) devices (or associated adapters) on request. Memory is accessed via a virtual address translated into a real (or physical) address that corresponds to the physical location within the memory. One method of completing these allocations and/or accesses to memory address space is via a direct memory access (DMA) operation issued from an IO adapter.

[0005] In many of today's computer systems, the system's physical memory address space is typically greater than the IO address space. With these computer systems, in order for the IO adapter(s) to access the entire system physical memory, some translation mechanisms is required. For example, a 32 bit IO address subsystem requires some kind of memory mapping to allow the IO adapter to access system memory addresses that are greater than 4 GB. Currently, most Operating Systems (OSes) set the maximum page size (in memory) to 4 Kbytes (4K), and thus each mapping page is 4 Kbytes. Table 1 below illustrates an example of an address mapping table, which shows the translation between system memory address and IO DMA (direct memory access) address for a given 4K page base address.

[0006]

	TABLE I
	System Memory Address	IO DMA address
	9000000E 00120000	F1000000
	9000000E 00221000	F1001000
	.	.
	.	.
	.	.
	9000000E 01010000	F10AF000
	9000000E 21002100	F11B0000
	.	.
	.	.
	.	.

[0007] To satisfy new requirements of high performance IO adapters, the data buffer mapping size needs to be greater than 4 Kbytes, particularly to take advantage of Ethernet jumbo frame and large TCP segmentation offload (TSO) send, for example. To enable support of this larger data buffer mapping size, the developers of the OSes have improved the OS' mapping methods to allow the address mapping of more than 4 Kbytes of continuous IO DMA address.

[0008] FIG. 1 is a flow chart illustrating the prior art methods by which the device driver maps a system physical address to an IO DMA address. The illustrated method describes the AIX OS function. However, the presented flow chart is provided as one example and may apply to other OSes as well.

[0009] As shown, the process begins at block 102 at which the computer system (or IO adapter) is initialized. During initializing of the IO adapter, the device driver makes a system call to register the size of the IO address space the driver needs for the operation, as shown at block 104. D_MAP_INIT is an example of this system call. Following, at block 106, the device driver calls the memory allocation routine to allocate system memory (buffer). Then, the device driver calls the system mapping routine to map the system memory to an IO DMA address at block 108. An example of this system call is D_MAP_PAGE/D_MAP_LIST.

[0010] Once this call is made, the device driver monitors when the IO DMA address is no longer needed, as indicated at decision block 110. If the IO DMA address is still needed, then the adapter maintains the space, as shown at block 116. However, when the IO DMA address is no longer needed, the device driver calls the kernel unmap routines to return the IO DMA address back to the kernel, as shown at block 112. Then, the device driver/OS frees the allocated memory back to the kernel, as provided at block 114. Examples of these system calls that complete the return of the allocated memory back to the kernel are D_UNMAP_PAGE/D_UNMAP_LIST and D_MAP_CLEAR.

[0011] As more and more adapters request memory address space, over time, the IO DMA address space becomes more and more fragmented. When this fragmentation surpasses a threshold point, as multiple portions of the large IO DMA space are assigned to DMA requests, the contiguity of available space decreases, and contiguous space becomes more difficult to find for assigning to new DMA requests.

[0012] Additionally, as the level of fragmentation increases, the latency in obtaining an assignment of an IO DMA address (for a contiguous address space) from the OS increases as well. This increase latency may cause measurable delays in processing and thus have a substantial negative impact on the overall system performance. While these delays are very common in the operation of most computer systems today, they are not desirable. Thus, a system that initially performs DMA address allocations at a relatively fast speed, eventually loses substantial performance speed (i.e., requires increased latency) after a period of operation. These systems thus do not perform as well as when the system initially starts up.

[0013] As described with the above and other types of conventional DMA mapping (e.g., using application programming interfaces (APIs)), several limitations exist. Among these limitations are the following: (1) while the driver is able to pre-register the size of the I/O DMA address the driver needs during the IPL time, the pre-registration does not guarantee address mapping operation success all the time. That is, the mapping may fail if the memory space runs out of the amount of continuous address space that is requested by the driver; and (2) the longer the system up time, the more fragmented the IO DMA address space will become and the longer the time required (i.e., latency of operation) to obtain a large contiguous address space for an IO DMA mapping. As noted above, this increased latency negatively impacts the overall system performance.

[0014] Disclosed is a method, system and computer program product for increasing the efficiency of IO DMA operations in a computer system by eliminating the latency in searching for contiguous memory space by an IO DMA request of a device driver when the search occurs sometime after the initial program loader (IPL). Three new application programming interfaces (APIs) are provided within the operating system (OS) code that allows the device driver(s) to (1) pre-request and pre-allocate the IO DMA address range from the OS during the IPL and maintain control of the address, (2) map a system (virtual/physical) address range to a specific pre-allocated IO DMA address range, and (3) free the pre-allocated IO DMA address space back to the kernel when the space is no longer required. Utilizing these APIs enables advanced IO DMA address mapping techniques maintained by the device drivers, and the assigned/allocated IO DMA address space is no longer fragmented, and the latency of completing the IO DMA mapping is substantially reduced/eliminated.

[0015] The above as well as additional objectives, features, and advantages of the present invention will become apparent in the following detailed written description.

[0020] The present invention provides a method, system and computer program product for increasing the efficiency of IO DMA operations in a computer system by eliminating the latency in searching for contiguous memory space by an IO DMA request of a device driver when the search occurs sometime after the initial program loader (IPL). Three new application programming interfaces (APIs) are provided within the operating system (OS) code that allows the device driver(s) to (1) pre-request and pre-allocate the IO DMA address range from the OS during the IPL and maintain control of the address, (2) map a system (virtual/physical) address range to a specific pre-allocated IO DMA address range, and (3) free the pre-allocated IO DMA address space back to the kernel when the space is no longer required. Utilizing these APIs enables advanced IO DMA address mapping techniques maintained by the device drivers, and the assigned/allocated IO DMA address space is no longer fragmented, and the latency of completing the IO DMA mapping is substantially reduced/eliminated.

[0021] With reference now to FIG. 2, a block diagram of a data processing system is shown in which the present invention may be implemented. Data processing system 200 is an example of a computer in which code or instructions implementing the processes of the present invention may be located. Data processing system 200 employs a peripheral component interconnect (PCI) local bus architecture. Although the depicted example employs a PCI bus, other bus architectures such as Accelerated Graphics Port (AGP) and Industry Standard Architecture (ISA) may be used. Processor 202 and main memory 204 are connected to PCI local bus 206 through PCI bridge 208. PCI bridge 208 also may include an integrated memory controller and cache memory for processor 202. Additional connections to PCI local bus 206 may be made through direct component interconnection or through add-in boards.

[0022] In the depicted example, local area network (LAN) adapter 210, small computer system interface (SCSI) host bus adapter 212, and expansion bus interface 214 are connected to PCI local bus 206 by direct component connection. In contrast, audio adapter 216, graphics adapter 218, and audio/video adapter 219 are connected to PCI local bus 206 by add-in boards inserted into expansion slots. Expansion bus interface 214 provides a connection for a keyboard and mouse adapter 220, modem 222, and additional memory 224. SCSI host bus adapter 212 provides a connection for hard disk drive 226, tape drive 228, and CD-ROM drive 230. Typical PCI local bus implementations will support three or four PCI expansion slots or add-in connectors.

[0023] An operating system (OS) runs on processor 202 and is used to coordinate and provide control of various components within data processing system 200 in FIG. 2. The operating system may be a commercially available operating system such as Windows XP, which is available from Microsoft Corporation. An object oriented programming system such as Java may run in conjunction with the operating system and provides calls to the operating system from Java programs or applications executing on data processing system 200. “Java” is a trademark of Sun Microsystems, Inc. Instructions for the operating system, the object-oriented programming system, and applications or programs are located on storage devices, such as hard disk drive 226, and may be loaded into main memory 204 for execution by processor 202. The data processing system depicted in FIG. 2 may be, for example, an IBM eServer pSeries system, a product of International Business Machines Corporation in Armonk, N.Y., running the Advanced Interactive Executive (AIX) operating system or LINUX operating system.

[0024] Those of ordinary skill in the art will appreciate that the hardware in FIG. 2 may vary depending on the implementation. Other internal hardware or peripheral devices, such as flash read-only memory (ROM), equivalent nonvolatile memory, or optical disk drives and the like, may be used in addition to or in place of the hardware depicted in FIG. 2. Also, the processes of the present invention may be applied to a multiprocessor data processing system.

[0025] For example, data processing system 200, if optionally configured as a network computer, may not include SCSI host bus adapter 212, hard disk drive 226, tape drive 228, and CD-ROM 230. In that case, the computer, to be properly called a client computer, includes some type of network communication interface, such as network adapter 210, modem 222, or the like.

[0026] The depicted example in FIG. 2 and above-described examples are not meant to imply architectural limitations. For example, data processing system 200 also may be a notebook computer or hand held computer in addition to taking the form of a PDA. Data processing system 200 also may be a kiosk or a Web appliance.

[0027] The processes of the present invention are performed by processor 202 using computer implemented instructions, which may be located in a memory such as, for example, main memory 204, memory 224, or in one or more peripheral devices 226-230.

[0028] Additional devices may also be provided for accessing memory 224, for example a Direct Memory Access (DMA) controller 207. DMA controller 207 is shown connected to system bus 206; However it is understood that DMA controller (or perhaps a second DMA controller) may be connected to an external bus, which connects to other IO master devices, such as a digital signal processor (DSP). DMA controller 207 accesses memory utilizing standard page frame tables and the like, according to established memory access techniques.

[0029] The features of the present invention are implemented in a data processing system such as the data processing system 200 of FIG. 2. The described embodiment of the invention provides a new set of application programming interfaces (APIs) that allow the IO adapter device driver to maintain its own IO DMA address mapping. Implementation of these new APIs enables avoidance of the IO DMA address fragmentation and also reduces the CPU cycles required (i.e., the latency) to perform the address mapping. Thus, these new APIs measurably improve the system's performance.

[0030] According to one embodiment, the new APIs are implemented as commands having formats provided below. These commands and their respective descriptions or definitions of the parameters utilized, include:

[0031] (API_1) IO address=IO DMA_ADDRESS REQUEST (size), where “IO address” is a return value that is the beginning of IO DMA address space which the kernel (OS) has assigned to the caller, and “size” is the size of contiguous address space requested by the caller;

[0032] (API_2) VIRT_TO_DMA_MAPPING (virt, DMA, size), where “Virt” is the virtual memory address needed to be mapped to IO DMA address space, “DMA” is the DMA address that the driver requests the kernel map to the virtual address, and “Size” is again the size of the mapping; and

[0033] (API_3) IO_DMA_ADDRESS_FREE (IO address, size), where IO address is the beginning of the IO DMA address which the driver wishes to free back to the kernel and “size” is the size of continuous address space that the caller wishes to give back to the kernel, which may be re-allocated to a later address request.

[0034] With these new APIs, the device driver is able to provide several advanced functions, which functions are illustrated by the process illustrated within the flow chart of FIG. 3, described below. The specific addresses (numbers) and/or sizes described herein are provided merely for illustration and not meant to imply any limitations on the invention. The process begins at block 302, which shows the initiation of a DMA address request during the IPL call of the IO adapter. As an example, the following device driver call (API_1) is made:

[0035] IO_address=IO_DMA_ADDRESS_REQUEST (0x100000).

[0036] According to the invention, the above API call occurs during adapter IPL time. After the API call, the IO address of that adapter receives a value of 0xF0001000. The kernel further assigns the IO_DMA address space 0xE0001000 to 0xF0101000 to the device driver, as shown at block 304. During run time, the device driver allocates a system memory (virtual address) as shown at block 306, and at block 308, the device driver calls the new API (API_2) to map to the specific IO DMA address which the driver requested during IPL time and which address is maintained by the driver. This function is provided by an API_2 call, such as:

[0037] VIRT_TO_DMA_MAPPING (0x56780000, 0xF0001000, 0x4000)

With this call, the driver requests the kernel to map the system memory from 0x56780000 . . . 0x56784000 to the IO DMA address space 0xF0001000 . . . 0xF0005000.

[0038] As shown at block 310, the device driver is able to request any number of other mappings using an API_2 call, such as:

[0039] VIRT_TO_DMA_MAPPING (0x77780000, 0xF0005000, 0x1000)

With this call, the driver requests the kernel to map the system memory from 0x77780000 . . . 0x77781000 to IO DMA address space 0xF0005000 . . . 0xF0006000.

[0040] Finally, the driver monitors at block 312 for an indication of when the pre-allocated the IO DMA address is no longer needed. Following this determination at block 312, the driver returns the IO DMA address to the kernel as shown at block 314. In one embodiment, the return of the IO DMA address occurs only when the driver is going to close the device. This process may (for example) involve the API_3 function call of:

[0041] IO_DMA_ADDRESS_FREE (0XF0001000, 0x100000)

With this call, the device driver releases the IO DMA address space 0xF0001000 . . . 0xF0101000 to the kernel.

[0042] With the above described new APIs, the device driver is able to maintain the IO DMA address space from the IPL rather than being made to incur the latency of attempting to find contiguous address space when the space is later requested/needed. The invention guarantees the contiguous size of IO DMA address space will always be available for the adapter to later utilize, when the adapter wishes to utilize the contiguous space. Further, the invention enables the kernel to substantially eliminate the need to search for free IO DMA address space within a fragmented memory space. The mapping of IO DMA address is provided by the device driver rather than the kernel. Finally, the invention enables the reduction of the number of CPU cycles (latency) needed to map the IO DMA address space, and helps to prevent IO DMA address fragmentation in the kernel.

[0043] As a final matter, it is important that while an illustrative embodiment of the present invention has been, and will continue to be, described in the context of a fully functional computer system with installed management software, those skilled in the art will appreciate that the software aspects of an illustrative embodiment of the present invention are capable of being distributed as a program product in a variety of forms, and that an illustrative embodiment of the present invention applies equally regardless of the particular type of signal bearing media used to actually carry out the distribution. Examples of signal bearing media include recordable type media such as floppy disks, hard disk drives, CD ROMs, and transmission type media such as digital and analogue communication links.

[0044] While the invention has been particularly shown and described with reference to a preferred embodiment, it will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the invention.