US 5,930,830 AGrant
System and Method for Concatenating Discontiguous Memory Pages
Issue Date:1999-07-27
•12 Claims
•3 Drawing Sheets
Abstract
A system and method are provided which significantly speed up the ability to reassemble network message transfer units (MTUs) using existing virtual memory systems. Discontiguous physical pages are rearranged in a continuous format in virtual memory by manipulating virtual page pointers in a hardware memory page table. The hardware memory page table provides any necessary virtual-to-real address translations during the execution of a process.
Metadata
Assignee
- International Business Machines Corporation
Inventors
- Abraham Mendelson
- Ronald Mraz
- Lucas Aaron Womack
Application Information
Application Number:US 783599&
Filing Date:1997-01-13
Priority Date:1997-01-13
Art Unit:272
Classifications
IPC:
G06F 1202
Field of Search:
711395707153;156;165;170;171;172;173;201;203;205;206;207;208;209;210;221;6200.01;200.02;200.13;200.16;200.17;885;894205;206
Patent Drawings (3 sheets)
Description
BACKGROUND OF THE INVENTION
1. Field of the Invention
The present invention relates generally to segmenting and reassembling message transfer units (MTU) and placing them in a memory, and more particularly to a system and method for concatenating and reassembling discontiguous MTUs via virtual memory address reassignment.
2. Description of Prior Art
One way for improving the performance of computer networks is through the use of DMA (Direct Memory Access) based protocols which automatically segment and reassemble ATM (Asynchronous Transfer Mode) cells and place them, in sequence, in memory segments or pages which are typically discontiguous in a host computer's memory. This technique eliminates the need for system software to copy data to and from the host memory to the communications card during data transmissions, yielding effectively a zero-copy interface.
However, this technique is ineffective when the system software receives network message transfer units (MTU) that exceed the size of a single hardware page. One adhoc solution is to use pre-allocated MTU-sized buffers of contiguous pages to store incoming data from an I/O card, e.g., a communications card. This solution, however, puts a strain on the operating system for memory resources, since it has to allocate contiguous physical memory for the incoming data. A resource-efficient solution is to reassemble discontiguous pages by copying them into a contiguous virtual region of memory. This technique solves the resource problem, but is plagued by being a slow memory copy (memcopy) operation, which limits its performance.
Accordingly, a need exists for a system and method which uses existing virtual memory systems to allow for a zero-copy transfer for MTUs larger than the memory page size and which is faster than traditional memcopy re-assembly.
SUMMARY OF THE INVENTION
The present invention provides a system and method for Virtual Memory MTU Reassembly (VMMR) which avoids the resource demands of pre-allocated contiguous buffers and the slow performance of memcopy operations. In the VMMR procedure, discontiguous pages are concatenated and reassembled via virtual memory address reassignment. The present invention includes a method for concatenating discontiguous memory pages in a processor. The method includes the steps of receiving a block of data that exceeds the size of a system page; writing portions of the block of data to two or more system memory pages to store the entire data block using two or more system memory pages; and assigning contiguous virtual addresses to the two or more system memory pages.
The present disclosure also relates to a system for concatenating discontiguous memory pages in a processor. The system includes a receiver for receiving a block of data that exceeds the size of a memory page within the processor; a physical storage area having a plurality of physical addresses each identifying the location of a physical memory page, where the block of data is stored within two or more physical memory pages; a virtual storage area having a plurality of virtual addresses each identifying the location of a corresponding virtual memory page, where the block of data is stored within two or more virtual memory pages; and a page table having virtual-to-real address mappings for tracking the location of the block of data in the virtual storage area.
Advantageously, the system and method of the present invention as mentioned above is faster than the traditional memcopy approach. The system and method of the present invention can also be used to allocate large contiguous buffers on demand, since the hardware page table can be used to locate a block of unused virtual memory to serve as the contiguous buffers.
Many other objects and purposes of the invention will be clear from the following detailed description of the drawings.
BRIEF DESCRIPTION OF THE DRAWINGS
In the drawings:
FIG. 1 is a graph which compares a traditional memcopy and a Virtual Memory MTU Reassembly (VMMR) procedure of the present invention.
FIG. 2 is a flowchart illustrating an implementation of VMMR within an operating system.
FIG. 3 is a graph of a system implementation in which an illustrative VMMR procedure according to the present invention is used.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
The system and method of the present invention rearranges discontiguous physical pages by manipulating virtual page pointers in a hardware page table. Although physical pages remain discontiguous in physical memory, they are made contiguous in virtual memory. The page table provides any necessary virtual-to-real address translation. A system describing a page table configuration is disclosed in U.S. Pat. No. 5,109,496 issued on Apr. 28, 1992 to Beausoleil et al. the contents of which are incorporated herein by a reference.
FIG. 1 depicts an illustrative chart comparing times required by the traditional memcopy approach and the VMMR procedure to concatenate, for example, from 1 to 6,144 hardware pages (4 kB each). For the memcopy approach the times rise exponentially (shown linearly on the log-log graph in FIG. 1) from approximately 200 usec (1 page) to 1 sec (6,144 pages). The VMMR curve reports the times required to re-assemble a non-contiguous MTU of 1 to 6,144 pages. At small message sizes (32 pages or less), VMMR speedup over memcopy ranges between 6 and 80. For larger message sizes (greater than 32 pages), VMMR speedup is constant at approximately 82. Therefore, the VMMR procedure of the present invention is faster over traditional memcopy operations, since it utilizes the host computer's virtual memory and a hardware page table to map the real address of an MTU to its virtual address without copying the MTU to the virtual region.
With reference to FIG. 2 there is shown a flowchart of an illustrative implementation of a Virtual Memory MTU Reassembly (VMMR) process of the present invention within an operating system. When an application or user process makes a system call to the operating system notifying the operating system to attach the application to a network device, a kernel locates a set of free pages within physical memory (Block 1). These free pages need not be contiguous and/or adjacent. The operating system allocates these pages for the user process requesting them (Block 2), and the operating system notifies the communications card of available physical pages that can be used by the operating system to receive or store the arriving MTUs into memory (Block 3). The process then polls the communications card or awaits an interrupt notifying that one or multiple fragmented MTUs have arrived and have been written to memory (Block 4). When this step has been completed, the kernel invokes the VMMR routine to reassemble each MTU in virtual memory (Block 5) and the physical pages are left untouched. The VMMR then returns a virtual memory pointer which represents the starting location of a contiguous MTU that is passed on to a protocol stack (see FIG. 3) for further processing. In a high-performance parallel processing interface, VMMR reassembles MTUs received as discontiguous pages and forwards them to an awaiting application, bypassing the protocol stack.
FIG. 3 is a graph of a system implementation in which an illustrative VMMR procedure according to the present invention is used in conjunction with an operating system. The operating system manages a number of user processes at any one time, and each process has a set of virtual pages 10 which translate to appropriate physical pages 12 in the main memory 14 through the page table 16. The page table 16 may include mappings for the currently executing process. When an MTU is received at 18 and stored into physical memory 14 from the communications card 22, the memory 14 storing the MTU is usually discontiguous 20. An area of unused virtual memory 24 for the process awaiting the MTU is then declared and used by the VMMR routines to remap the physical pages 12. The process reads the MTU by reading the virtual memory pages 10 in a sequential order. It is completely unaware that the VMMR has updated the page table 16 to provide mappings from virtual memory space 24 to physical memory space 14.
During context switching, the table is usually paged out to memory or swapped to a disk or hard drive and a new page table for the scheduled process is placed in the system area before the switch is completed. In many traditional UNIX-based operating systems, a VMMR may be done after a context switch has occurred. This is because the process that was waiting for an MTU has usually been placed on an operating system sleep queue by the time the MTU arrives. The operating system may then swap out the currently executing process, brings in the sleeping one (and its page table) from the appropriate disk location, and subsequently performs the VMMR.
For a VMMR to work, an operating system has to maintain some pool of free contiguous virtual pages. In the illustrative implementation described herein, a bitmap 26 is used to represent pages in some reserved area of virtual memory set aside by the operating system, for example during bootup or at application runtime. Generally, the executing process receives a set of physical pages as an operating parameter. The executing process then searches for a starting bit within the bitmap 26 of a contiguous set of unallocated bits equal to the number of pages requiring re-assembly. Once this bit has been found, it is converted to a starting virtual memory address (this is later returned to the application), and appropriate page table entries are added or modified to provide a transparent re-assembly. An example is provided and compared to memcopy further below.
The VMMR procedure can be optimized by reducing the bit search time. This is done by managing several bitmaps 26 of various sizes; each bitmap 26 has bits representing some fixed number of pages. Therefore, one bitmap 26 may include bits, each of which represent one page; another bitmap 26 includes bits, each of which represent two pages; and so on. When the optimized VMMR routines are called, the number of physical pages requiring re-assembly is determined, and an appropriately sized bitmap 26 is used. The free set search time drops dramatically because a set could be represented by a single bit. FIG. 1 illustrates a performance curve derived from this optimization technique.
An example illustrating and comparing the memcopy approach and the VMMR procedure follows, where a communications card has filled four pages of 4096 bytes each in the main memory in the following sequence: 1000.sub.16, 7000.sub.16, 5000.sub.16 and 3000.sub.16. The memcopy approach and the VMMR procedure relocate this 16 kB MTU in a single block starting at hex address 100000.sub.16. It is noted that steps 3 to 6 of the VMMR procedure are faster than steps 3 to 6 of the memcopy approach.
Memcopy
1. Wait for an MTU to arrive (1000.sub.16, 7000.sub.16, 5000.sub.16 and 3000.sub.16).
2. Find a free address region for reassembly (100000.sub.16).
3. Copy the region 1000.sub.16 -1FFF.sub.16 to 100000.sub.16 -100FFF.sub.16.
4. Copy the region 7000.sub.16 -7FFF.sub.16 to 101000.sub.16 -101FFF.sub.16.
5. Copy the region 5000.sub.16 -5FFF.sub.16 to 102000.sub.16 -102FFF.sub.16.
6. Copy the region 3000.sub.16 -3FFF.sub.16 to 103000.sub.16 -103FFF.sub.16.
7. Inform application that a new MTU has arrived and is located at starting location 100000.sub.16.
VMMR
1. Wait for an MTU to arrive (1000.sub.16, 7000.sub.16, 5000.sub.16 and 3000.sub.16).
2. Get a 16 kB area of contiguous virtual address space for reassembly (100000.sub.16).
3. Assign virtual address 100000.sub.16 to reference memory at the region 1000.sub.16 -1FFF.sub.16.
4. Assign virtual address 101000.sub.16 to reference memory at the region 7000.sub.16 -7FFF.sub.16.
5. Assign virtual address 102000.sub.16 to reference memory at the region 5000.sub.16 -5FFF.sub.16.
6. Assign virtual address 103000.sub.16 to reference memory at the region 3000.sub.16 -3FFF.sub.16.
7. Inform application that a new MTU has arrived and is located at starting location 100000.sub.16.
The appendix attached hereto includes source code for implementing a VMMR procedure according to the present disclosure.
Many changes and modifications in the above-described embodiment of the invention can of course, be carried out without departing from the scope thereof. Accordingly, that scope is intended to be limited only by the scope of the appended claims.
Claims
What is claimed is:
1. A method for concatenating discontiguous memory pages, said method comprising the steps: receiving into a transfer buffer a block of data that exceeds the size of a system page; storing the block of data using a processor to two or more system memory pages of a memory, each system memory page having a location identified by a physical address; identifying at least one bitmap having bits representing a fixed number of contiguous virtual memory pages of an area of unused virtual memory, the fixed number of contiguous virtual memory pages being equal to or greater than the number of virtual memory pages required to store the block of data during the duration of the concatenating method; storing the block of data using the processor to two or more of the fixed number of contiguous virtual memory pages; assigning virtual addresses to the two or more system memory pages where each virtual address identifies a location in the memory of a virtual memory page; and determining the location of the block of data stored in the two or more contiguous virtual memory pages by mapping the two or more system memory pages to their assigned contiguous virtual addresses.
2. The method of claim 1, wherein the two or more system memory pages are physically disjunct.
3. The method of claim 1, wherein a hardware memory page table includes virtual-to-real address mappings for mapping the two or more system memory pages to their assigned contiguous virtual addresses.
4. A system for concatenating discontiguous memory pages in a processor comprising: a receiver for receiving at least one block of data that exceeds the size of a memory page within said processor; a physical storage area having a plurality of physical addresses each identifying the location of a physical memory page, said at least one block of data is stored within two or more physical memory pages; a virtual storage area having a plurality of virtual addresses each identifying the location of a corresponding virtual memory page; at least one bitmap having bits representing a fixed number of contiguous virtual memory pages of the virtual storage area during the duration of a concatenating method, said at least one block of data is stored within two or more of the fixed number of contiguous virtual memory pages; and at least one memory page table having virtual-to-real address mappings for tracking the location of the at least one block of data in the virtual storage area.
5. The system of claim 4, wherein the two or more system memory pages are physically disjunct.
6. The system of claim 4, wherein the block of data is a network message transfer unit (MTU).
7. The system of claim 4, wherein the processor updates the at least one memory page table during the execution of a process.
8. The system of claim 4, wherein said at least one bitmap represents said fixed number of contiguous virtual memory pages in some reserved area of the virtual storage area set aside by an operating system.
9. The system of claim 4, wherein a protocol stack is used to temporarily store said at least one block of data before said at least one block of data is stored in said virtual storage area.
10. A system for concatenating and reassembling discontiguous pages via virtual memory address reassignment, said system comprising: means for receiving a block of data that exceeds a size of the processor's memory page; means for writing portions of the block of data to two or more system memory pages so that the entire data block is stored using two or more system memory pages; and means for assigning contiguous virtual addresses to the two or more system memory pages wherein the contiguous virtual addresses belong to a group, the group includes a fixed number of contiguous virtual addresses which is equal to or greater than the number of system memory pages used to write portions of the block of data during the duration of a concatenating method.
11. The system of claim 10, wherein the two or more system memory pages are physically disjunct.
12. The system of claim 10, wherein a hardware memory page table contains virtual-to-real address mappings for mapping the two or more system memory pages to their assigned contiguous virtual addresses.
Patent Citations (9)
| Patent | Date | Inventor | Cited By |
|---|---|---|---|
| US4037215 | 1977-07-01 | Birney et al. | |
| US4660130 | 1987-04-01 | Bartley et al. | |
| US4775932 | 1988-10-01 | Oxley et al. | |
| US5058003 | 1991-10-01 | White | |
| US5109496 | 1992-04-01 | Beausoliel et al. | |
| US5386524 | 1995-01-01 | Lary et al. | |
| US5426748 | 1995-06-01 | Brenza et al. | |
| US5465343 | 1995-11-01 | Henson et al. | |
| US5539899 | 1996-07-01 | Huynh et al. |
Non-Patent Literature (1)
- Doyle et al., ed., Microsoft Press Computer Dictionary, 2.sup.nd ed, Microsoft Press, 1994, pp. 371 and 412.