Run formation in large scale sorting using batched replacement selection

[0002] This invention relates to systems and methods for large scale sorting of data records. More particularly, this invention relates to run formation techniques for producing sorted subsets or “runs” of data records that will be subsequently merged together to produce the final sorted set of data records.

[0003] Database systems store data records on disk arrays or other forms of non-volatile memory. Operations performed on databases can be easily accommodated if all of the data records are first loaded into volatile main memory (i.e., RAM) for processing. However, databases often contain large numbers of data records, which far surpass the volatile memory resources. As a result, subsets of data records are processed in the main memory and then written back to disk after they are processed.

[0004] Large scale sorting is one type of operation performed by a database system on a large set of data records that is too large to fit in main memory. “Mergesort” is the standard technique used for sorting large sets of records. Mergesort has two phases: (1) a run formation phase that creates sorted subsets, called “runs”, and (2) a merge phase that repeatedly merges runs into larger and larger runs, until a single run has been created. This invention pertains to the run formation phase of a mergesort operation.

[0005] In general, there are two types of run formation techniques: load-sort-store and replacement selection. Most sort implementations use the load-sort-store algorithm for run formation. This invention is directed to an improved replacement selection technique. Load-sort-store is briefly described below, followed by a more detailed look at replacement selection.

[0006] FIG. 1 shows the load-sort-store process implemented on a database system 20 having main memory 22 and persistent data storage 24. As the name implies, the load-sort-store process has three main steps: loading, sorting, and storing. During a “load” step 1, a subset of records is read from the data storage 24 into the main memory 22. Next, during a “sort” step 2, the process extracts pointers to all records into an array and sorts the entries in the array according to a designated sort key to form the run. Any in-memory sorting algorithm can be used, with “Quicksort” being the most popular choice. Afterwards, during a “store” step 3, the run is written back to the data storage 24 and all records in main memory 22 are erased.

[0007] These three steps are repeated until all records have been processed. All runs created will be of the same length, except possibly the last run. After multiple runs have been created and stored during the run formation phase, the sorting process transitions to the merge phase in which the runs are merged into one large run.

[0008] One drawback with the load-sort-store process is that CPU processing and I/O (input/output) cannot be overlapped. As a result, the computing resources of the database system are not fully utilized.

[0009] Replacement selection is an alternative technique that produces longer runs and completely overlaps CPU processing and I/O. It is based on the observation that by tracking the highest key output thus far, the process can easily determine whether an incoming record can still be made part of the current run or should be deferred to the next run. Any record tacked on to the current run in this way increases the run length. Adding a record to the current run is expensive if a (sorted) pointer array is used, so replacement selection often uses a (binary) selection tree instead.

[0010] FIG. 2 shows a replacement selection process implemented in the database system 20. The process described below is the traditional version of the replacement selection algorithm as described, for example, by Donald Knuth in “The Art of Computer Programming, Volume. 3: Sorting and Searching”, Second Edition 1998. This version of the algorithm assumes that records are of fixed length but, in practice, variable-length records are more common.

[0011] Initially, a subset of records is read into the main memory 22. When records are of fixed length, the main loop of the algorithm has three steps: selection, output, and get new record. In the “selection” step 1, the process selects from among the records in memory, a record with the lowest key greater than or equal to the last key output. Then, at the “output” step 2, the selected record is written out to the data storage 24 and the key value is remembered. Afterwards, at the “get” step 3, the process retrieves a new record from the data storage 24 and stores it in the slot previously occupied by the record just output.

[0012] While load-sort-store is more widely used, replacement selection has many benefits. First, replacement selection produces runs larger than the memory used, which reduces the number of runs and the subsequent merge effort. Second, replacement selection exhibits steady I/O behavior during run generation rather than I/O in great bursts. This improves the utilization of I/O devices. Third, it very effectively exploits pre-sorting to produce longer runs, i.e., input sequences that are not random but somewhat correlated to the desired sorted output sequence. In particular, if the input is already sorted, a single run will be generated. Fourth, sorting is often done as part of grouping with aggregation or for duplicate removal. A technique called early aggregation can then be applied, which reduces I/O significantly. Early aggregation achieves much higher I/O reduction if runs are created by replacement selection.

[0013] The following sections examine the selection step of replacement selection using a binary selection tree. The first section considers a tree formed with fixed length records, and a subsequent section examines the tree with variable length records.

[0014] The selection step is the most expensive part of the replacement selection operation. The selection step can be performed efficiently with the help of a selection tree, using the run number and the record key as the (compound) selection key. A selection tree for N records is a left-complete binary tree with N external nodes and N−1 internal nodes stored in an array without pointers. Logically, each external node or tree “leaf” stores a data record with a combined sort key consisting of a run number and the original sort key of the record. The combined sort key is here called a run-key pair and represented by a notation “run number, key” such as “1,12”. Each internal node stores the lesser of the sort keys of its two children nodes plus a reference to the source node of the key. Physically, the selection tree and nodes can be implemented in several different ways.

[0015] FIGS. 3aand 3bshow a selection tree 30 before and after a, selection replacement operation for replacing the lowest key on the first run (i.e., key 12 for run 1). FIG. 3ashows the initial state of the selection tree 30 with five keys (i.e., 5, 12, 15, 22, and 32). The key of the last record output was 8. External nodes or tree “leafs” are drawn as rectangles and internal tree nodes are shown as ellipses. The number to the left of a node indicates its position in the array.

[0016] Initially, the record represented by external node (1,12) is selected and output. Its key value “12” is now the last key value output, and this value is recorded. Outputting the record frees up the external node at position 9. A new record is input and stored in this node. The key of the new record is 30, which is higher than the last key value output (12) so the new record “fits” in the current run. Its run number is set to 1, resulting in the pair (1,30) shown in position 9. Next, a new minimum key is found by traversing the path from the external node at position 9 up to the root at position 1. At each node, the sort keys of two sibling nodes are compared and the node with the lower sort key is promoted to the parent node. For example, for external nodes (1,32) and (1,30) at positions 8 and 9, the key (1,30) at position 9 is less than the key (1,32) and is promoted to the parent node at position 4. Again, comparison of the internal nodes at positions 4 and 5 reveals that node (1,15) is less than (1,30) and is therefore promoted to parent node at position 2. The number of comparisons is always either floor(log2(N)) or ceil(log2(N)).

[0017] FIG. 4 shows one implementation of the selection tree 30 stored in an array 40, although the selection tree can be implemented in several different ways. A node consists only of a pointer 41 to a corresponding record, in a record data structure 42 in order to keep the tree itself of minimal size. The beginning of the array is offset so that each pair of sibling nodes occupies a common cache line. The type of a node is determined by its position in the array so the type need not be explicitly marked. Here, the internal nodes are at the top of the array and the external nodes are at the bottom. There are approximately twice as many nodes as records (i.e., 2N−1 nodes to N records), so the selection tree array adds approximately eight bytes of overhead per record.

[0018] A record slot 43 in the record data structure 42 contains a record consisting of a record key 44 and possibly some other fields 45. Each record slot also has two additional fields needed by the run formation algorithm, namely, a run number field 46 (e.g., 4 bytes), and a home node field 48 recording the position in the selection tree array of the external node that owns the record (e.g., 4 bytes). The last two fields 46 and 48 add another eight bytes of overhead per record, bringing the total overhead to sixteen bytes per record. The home node field 48 tracks the external node occupied by a record so that, when a record has been output, the new record can be placed in that tree node. Knuth's version of the algorithm handles this problem by adding a level of indirection: an internal node points to the external node owning the record, which in turn points to the actual record. The indirection increases cache misses during traversal because to locate a record the external node must first be accessed.

[0019] When records are of fixed length, the home node field can actually be eliminated—a records home node can be deduced from which record slot it occupies. However, this is not possible when records are of variable length.

[0020] Knuth's version uses two additional refinements: storing the loser instead of the winner in each internal node and packing one internal node and one external node together into a combined physical node. These changes make the algorithm harder to understand, but have little effect on its performance.

[0021] When records are of fixed length and the amount of memory available is known before run formation starts, the size of the selection tree can be determined before run formation begins. The tree is pre-allocated and filled with a fictitious run zero that is never output. Processing of an incoming record then consists of the following steps:

[0022] 1. Record the key value and run number of the top record.

[0023] 2. Copy the top record to an output buffer (provided it is not from the fictitious run zero).

[0024] 3. If the output buffer is full, write it to the run file.

[0025] 4. Copy the incoming record into the vacated slot in memory.

[0026] 5. Determine its run number by comparing it with the recorded key value and set the run number field in the record.

[0027] 6. Copy a pointer to the new record into the appropriate node in the selection tree. Call this node T.

[0028] 7. Set the home node field in the record to point to T.

[0029] 8. Fix up the path from node T to the root in the selection tree.

[0030] When there are no more input records, the memory is once again filled with fictitious records that are never output. The only operation performed is replacement, consisting mainly of copying two records and traversing a path from a leaf to the root.

[0031] The above discussion of tree selection assumes that the records are of fixed length. When records are of variable length, two complications arise. First, managing the space reserved for storing records becomes more complex. Second, records are no longer just replaced in the selection tree. A record may be output and deleted from the tree without being replaced (if there is no free slot large enough for the replacement record). Similarly, records may be input and added to the tree without outputting any existing records. Consequently, the tree is no longer of constant size.

[0032] In the case of variable length records, it is better to view replacement selection as consisting of two processes: (1) an input process that fills memory with new records and adds them to the selection tree and (2) an output process that repeatedly deletes the top record from the tree and outputs it. The input process drives the processing. Whenever it fails to find memory space for an incoming record, it resumes the output process, which runs until it has created a free slot at least as large as needed by the new record. To purge all records when reaching the end of input, the input process requests space for an infinitely large record.

[0033] FIG. 5 illustrates the input process in which a new node (1,14) is added to the selection tree 30 from FIG. 3b. The new node (1,14) is added to the end of the tree after the last external node, as represented by insertion of node (1,14) into position 10 in the tree 30 (step 1 in FIG. 5). Because the tree is complete, its parent is always the first external node, which in this case is node (1,15) at position 5 (see FIG. 3b). The content of the parent node is copied into the next free node to become the right sibling node of the new element, as represented by new node (1,15) in position 11 (step 2 in FIG. 5).

[0034] A different external node now owns the record, so the home node field in the record has to be updated from position 5 to position 11. In addition, the parent node changes its role from an external node to an internal node. The sort keys of the two new external sibling nodes (1,14) and (1,15) are compared and the lower one promoted to the parent node (step 3 in FIG. 5). In this case, the new node (1,14) contains the lower key and it is promoted to position 5. The walk up the tree continues until there is no change; that is, the record that gets promoted is the same as the one already stored in the parent node (step 4 in FIG. 5). In this case, the node (1,14) is promoted all the way to the root node at position 1. However, in other cases, the traversal may not continue all the way to the root.

[0035] FIGS. 6aand 6bshow the output process in which a node is deleted from the selection tree. The node with the lowest key is always the one deleted. Using the tree in FIG. 3b, suppose the output process deletes the external node (1,15) from the selection tree 30. First, the target node (1,15) is exchanged with the last node (1,30) of the tree 30 and the home node fields of both records are updated (step 1 in FIG. 6a).

[0036] The two tree paths affected by the exchange are then fixed up (step 2 in FIG. 6a). As a result of the exchange, the last node will always contain the lowest key. Accordingly, to fix up the path from the last node, every internal node on the path to the root is updated to point to this record without the need for key comparisons. The other path affected by the exchange is updated in the normal way by comparing keys. In this case, the internal node (1,15) at position 4 has a lower key value than the external node (1,30) at position 5, and hence the traversal stops here and does not continue all the way to the root. The result is a valid selection tree with the lowest run-key pair (1,15) as the last external node of the tree.

[0037] Following the path update, the target node (1,15) in the last position is deleted from array 40 (step 3 in FIG. 6b). This deletion causes external node (1,32) to move to its parent node at position 4 (step 4 in FIG. 6b). The path from there is then fixed up all the way to the root (step 5 in FIG. 6b), causing node (1,22) to be promoted to the top.

[0038] When records are of variable length, managing the space reserved for storing records becomes an issue. In an article by Per-Åke Larson and Goetz Graefe, entitled “Memory Management during Run Generation in External Sorting”, SIGMOD, 1998: 472-483, the authors showed that a version of best-fit allocation solves this problem efficiently, resulting in a memory utilization of 90% is or better. The basic idea of best-fit allocation is to always allocate space from the smallest free block large enough for the record being stored. If the selected free block is larger than required, the unused space is returned to the pool as a smaller free block. Immediate coalescing of adjacent free blocks and the use of boundary tags is recommended for efficiency. Best-fit allocation depends on being able to locate the smallest block larger than a given size efficiently. One technique is to store a collection of free blocks in a (balanced or unbalanced) binary tree with the block size as the key. If the tree is kept balanced (e.g., an AVL-tree), the best-fitting free block can always be located in logarithmic time. An alternative technique is to maintain multiple free lists segregated by block size. For example, we may use 256 free list and keep all free blocks of size 16 bytes or less on list 1, all free blocks between 17 and 32 bytes on list 2, and so on. In practice this technique is much faster and, consequently, is the preferred implementation.

[0039] Modern CPUs rely heavily on caches to hide memory latency and increase overall performance. As a result, it is increasingly important to design algorithms that generate few cache misses. Unfortunately, the classical replacement.selection algorithm has poor cache behavior when the number of records in memory is large. The main loop of the algorithm consists of traversing a path of the selection tree from a leaf node to the root, while comparing sort keys of each sibling pair. Which path is traversed is unrelated to previously used paths. The nodes in the top part of the tree, and their associated sort keys, are touched frequently and are likely to remain in the cache but not the ones lower down in the tree.

[0040] Accordingly, there is a need for a cache-conscious version of replacement selection, which reduces the number of cache misses significantly.

[0041] This invention concerns large scale sorting technology for forming runs of data records using an improved cache-friendly replacement selection process that reduces cache misses.

[0042] In one implementation, a database system has a memory subsystem with persistent data storage (e.g., disks, RAID, etc.) and a main memory. The system also has at least one processing unit with a processor and cache memory (e.g., L1 and L2 caches). A selection tree is formed in the main memory as an array of pointers to records that are being sorted. The system also has a mini-run assembly data structure stored in the main memory to create mini-runs of sorted records prior to adding the records to the selection tree.

[0043] The system runs a large-scale sorting program having a batched replacement selection module for forming runs of sorted data records. The batched replacement selection module reads multiple records from the persistent data storage into main memory, adds them to the mini-run assembly data structure and, whenever the mini-run assembly structure becomes full, sorts its records to form a mini-run of multiple sorted data records. As an example, a mini-run might consist of 500 records sorted according to their key values. After formation, the mini-run is added to the selection tree by inserting a pointer to a first record in the mini-run into the array of pointers. The first record is linked to remaining records in the mini-run.

[0044] The batched replacement selection module selects a record for output from a node in the selection tree. If the mini-run associated with the node is not empty, the module replaces the selected record with a next record from the mini-run. Once the mini-run becomes empty, the module deletes the node. In this manner, the selection tree grows and shrinks as mini-runs are added and deleted.

[0045] When main memory becomes full, the batched replacement selection module switches to output mode. It selects some number of records for output using the selection tree and copies the selected records into an output buffer in main memory. Whenever the output buffer becomes full, the module writes contents of the buffer back to the persistent data storage at the end of the current run. Exactly how long the output process runs is a matter of policy. Two possible policies are: stop after a predetermined number of records have been output or stop after a predetermined number of output buffers have been filled.

[0046] The combination of using pre-sorted mini-runs as tree nodes and batch processing input/output records significantly reduces the number of cache misses (particularly, L2 cache misses).

[0047] FIG. 1 is a diagrammatic illustration of a database system implementing a conventional load-sort-store process for run formation in large-scale mergesort operations.

[0048] FIG. 2 is a diagrammatic illustration of a database system implementing a conventional replacement selection process for run formation in large-scale mergesort operations.

[0049] FIGS. 3aand 3bshow a selection tree before and after a selection replacement operation is performed on a leaf node.

[0050] FIG. 4 is a diagrammatic illustration of a data array used to implement the selection tree of FIG. 3a.

[0051] FIG. 5 is a diagrammatic illustration of a selection tree and illustrates the addition of a node to the selection tree.

[0052] FIGS. 6aand 6bare diagrammatic illustrations of a selection tree and illustrate the deletion of a node from the selection tree.

[0053] FIG. 7 shows a database storage system that implements a batched replacement selection process.

[0054] FIG. 8 is a diagrammatic illustration of a selection tree with mini-runs attached at the external leaf nodes.

[0055] FIG. 9 is a diagrammatic illustration of a data structure used to assemble mini-runs.

[0056] FIG. 10 is a diagrammatic, illustration of a data array used to implement the selection tree of FIG. 9.

[0057] FIG. 11 is a flow diagram for a batched replacement selection process.

[0058] FIG. 12 shows the overall performance of the four methods—Quicksort, Alphasort, classic replacement selection, and batched replacement selection—as a function of memory size.

[0059] FIG. 13 shows results of experiments on four methods—Quicksort, Alphasort, classic replacement selection, and batched replacement selection—run with a single disk.

[0060] FIG. 14 shows results of experiments on four methods—Quicksort, Alphasort, classic replacement selection, and batched replacement selection—run with two disks.

[0061] This invention concerns a cache-conscious replacement selection methodology that employs local batch processing to improve performance on modern-day CPUs. For discussion purposes, the invention is described in the context of a uni-processor system although the invention may be employed in a shared-memory multiprocessor system (SMP).

[0062] FIG. 7 shows a database storage system 50 implemented on a computer having a processor 52 coupled to a memory subsystem 54. A system bus (or other interconnection network) 56 interconnects processor 52 and memory subsystem 54.

[0063] The processor 52 has a chip 60 that holds a CPU (central processing unit) 62 and a first level (or L1) cache 64. The processor 52 also has a second level (or L2) cache 68 that may be on the same or a different chip (as represented by chip 66). The L1 cache 64 is considerably smaller than the L2 cache 68. For example, the L1 cache 64 typically ranges in size from 16 Kbytes to 256 Kbytes, whereas the L2 cache 68 typically varies from 128 Kbytes up to 4 Mbytes. Due to its proximity to the CPU 62, however, the L1 cache 64 has a significantly lower access time in comparison to the L2 cache 68. Typically, the access time to remote L2 cache 68 is five times longer than the access time to L1 cache 64.

[0064] The memory subsystem 54 has a persistent storage 70 that provides the bulk of the data storage. The persistent storage 70 is stable and persists across system crashes. Examples of such storage include disk, RAID (redundant array of independent/inexpensive disks) storage systems, tape systems, and so forth. Data is stored in storage 70 as records 72.

[0065] The memory subsystem 54 also has a main memory 74 that comprises volatile memory, non-volatile memory, or a combination of both. During program execution, a subset of record 72′ is read from the persistent memory 70 into the main memory 74. The computer system 50 thus forms a hierarchical memory system having tiers ranging from small, high-speed memory (e.g., L1 cache 64) to large, slow access memory (e.g., stable storage 70).

[0066] The system 50 runs an application program that performs large-scale sorts, such as the mergesort operation. The application program may be a database program, such as a relational database program (e.g., SQL Server program from Microsoft Corporation). It is also noted that aspects of this invention may be implemented in other types of programs that may employ database concepts, such as spreadsheet programs, accounting software, workflow management software, and the like.

[0067] As part of the sort operation, the application program forms runs of data records using a replacement selection process. As noted in the Background, the traditional version of replacement selection has poor locality of reference and generates frequent cache misses (primarily, L2 cache misses). According to an aspect of this invention, the CPU runs a batched replacement selection (RS) process 80 that generates significantly fewer L2 cache misses. The batched RS process 80 is illustrated as a software module executed on the microprocessor 52. The module may further be stored in main memory 74.

[0068] The batched selection replacement process uses a selection tree to sort the records into runs. Entries in the selection tree are loaded into L2 cache 68, and the sort keys of few records from the data in L2 cache 68 can be further loaded into L1 cache 64 for highest access speed.

[0069] Traditionally, the selection tree has as many external nodes as there are records. That is, each external node consists of one record. With batched replacement selection, the selection tree is formed with a small batch of sorted records attached to each external node. This reduces the size of the tree. The small batches of sorted records are referred to as “mini-runs” to distinguish them from the eventual output runs.

[0070] The selection tree is implemented in FIG. 7 as a data array 82 that contains pointers to the records 72′ in main memory. The external nodes contain pointers to the first record in a mini-run. The first record then references the next sorted record, and so on. Accordingly, the linked list of records forms the mini-run associated with each external node. Another data structure 84 is employed to assemble the mini-runs as records are input to the selection replacement process. When the data structure is full, the assembled mini-run is added to the selection tree as an external node. The data array 82 and data structure 84 are described below in more detail with reference to FIGS. 9 and 10.

[0071] The batched replacement selection process facilitates a cache-friendly; methodology that reduces cache misses. The process implements two techniques: (1) adding sorted mini-runs to the selection tree; and (2) batched processing of the input and output records. The combination of these techniques significantly reduces cache misses at the L2 cache level. As an example, during experimentation, the inventors observed a 50% reduction of L2 cache misses for 100 byte records and 20 MB of memory.

[0072] I. Mini-Run Utilization

[0073] A first part of the batched replacement selection process is to create sorted mini-runs and add them as nodes to the selection tree. The mini-runs replace single record nodes at the external leaf nodes of the selection tree. This helps compact the size of the tree. FIGS., 8-10 illustrate an exemplary selection tree and associated data structures to implement the tree and form the mini-runs.

[0074] FIG. 8 shows an exemplary selection tree 90 that is similar to tree 30 in FIG. 3b, except that the external nodes are formed of mini-runs 92(1)-92(5). The mini-runs 92(1)-92(5) contain a short set of sorted records. Typically, the mini-runs; consist of 500-1000 sorted records. For illustration purposes, the mini-runs are shown with only a few records. For easier readability, the figure shows the mini-runs as arrays of records even though the preferred implementation is a linked list. For example, mini-run 92(1) at node position 8 contains four records: record (1,32); record (1,36); record (1,48); and record (1,52). These records are sorted according to their run number (all“1” in this example) and key value “32”, “36”, “48”, and “52”. It is noted that the tree may also be configured to store key prefixes.

[0075] When a record is output, the next record in its mini-run replaces it and the selection tree is updated by propagating the lowest key up the tree. For instance, if record (1,15) is output, the next record (1,17) in mini-run 92(3) is placed in the external node 5. The record keys of nodes 4 and 5 are compared (i.e., record (1,17) has a lower key than record (1,30)), and hence record (1,17) is propagated up to node 2, and eventually to root node 1.

[0076] When a mini-run becomes empty, there is no replacement record. The empty node is removed in the manner described earlier, thereby shrinking the tree. For example, when the last record (1,43) in mini-run 92(3) is output, there are no more records in this mini-run. When this happens, the target node (1,43) is exchanged with the mini-run 92(2) at the last node 9 of the tree 90 and the tree paths affected by the exchange are fixed up. Following the path update, the target node (1,43) in the last position is deleted. However, note that the use of mini-runs significantly reduces the number of times the selection tree shrinks and grows.

[0077] Incoming records are placed in memory but are not added individually to the tree. Instead, the incoming records are assembled into mini-runs in data structure 84 (FIG. 7). FIG. 9 shows the mini-run assembly data structure 84 as having an array of record pointers 94 and a counter 96 to track the number of records in the array. For illustration purposes, the mini-run assembly data structure 84 holds the sorted mini-run 92(4) for the external node at position 6 (FIG. 8). When the current mini-run becomes full, the run number of each record in the batch is determined and the records are sorted (e.g., using Quicksort). The records in the mini-run are then linked together in sorted order. The first record in the mini-run is then added to the selection tree. In this manner, the size of the selection tree varies as mini-runs are added and deleted.

[0078] FIG. 10 shows the selection tree 90 implemented in array 82. A node in the selection tree contains a pointer 100 to a corresponding record 102 in a record data structure 104. Each record 102 contains a record key 106 and possibly some other fields (not shown). Each record 102 also has three fields for replacement. selection: a run number field 110 to identify the run within which the record belongs, and a home node field 112 to index the record's home slot in the selection tree, and a next record field 114 to hold a pointer to the next record in the mini-run. All records in a mini-run contain the same fields but for easier readability only the first record in each mini-run is shown in detail.

[0079] The record slots for the external nodes reference the first remaining record in a mini-run. As an example, slot 8 in array 82 references record (1,32) as the first record in run 92(1). The mini-runs are linked together so that the previous record references a next record. Here, record (1,32) references the next record (1,36), which in turn references the next record (1,48), and so on. Linking requires less space than representing a mini-run by a pointer array (although this is possible).

[0080] Collecting records into mini-runs reduces the size of the selection tree and the set of keys participating in comparisons when traversing the tree. It can be shown that mini-runs are on average half full, meaning that a mini-run size of B records reduces the tree size by a factor of B/2 and thus the number of tree levels by log2(B/2). For example, using mini-runs of 512 records reduces the selection tree by a factor of 256 or 8 levels. Each level may generate two or three L2 cache misses during traversal so the potential saving is 16 to 24 misses.

[0081] The selection tree is now much smaller in comparison to trees, employed in conventional replacement selection processes. It might make sense to use an indirection scheme in which an internal node points to an external node which, in turn, points to the actual record. If indirection is adopted, the home node field in each record slot can be eliminated, thereby saving a significant amount of memory. Another possibility is to include both a pointer to the record and a pointer to the external node in each internal node.

[0082] II. Batched Processing of I/O Records

[0083] A second part of the batched replacement selection process is the batch processing of input and output records. Recall that replacement selection can be viewed as consisting of an input process in which records are read into main memory and an output process in which records are written out to the data storage. Traditional selection replacement inputs/outputs, on average, one record at a time. With batch processing, the input and output processes are performed for many records at a time.

[0084] In general, the I/O process is implemented as two short loops:

[0085] 1. An input loop fills the available free memory space with new records and adds them to mini-run assembly data structure 84 in main memory 74. For instance, the input loop might read in 500 records to fill the available free memory. Whenever the current mini-run structure 84 is full, the records are sorted into a mini-run and the mini-run is added to the selection tree. Whenever there is no more free space for an incoming record, the process switches to the output loop.

[0086] 2. An output loop selects and outputs a batch of, say, 1000 records and then switches to input mode. Whenever a mini-run in the tree becomes empty, the corresponding external node is deleted and the tree shrinks. As soon a record has been copied to an output buffer, its memory space is returned to the free pool. Output buffers are written to disk as soon as they become full.

[0087] Batch processing reduces the working set and number of cache misses. For example, when switching to the output loop, the selection tree and related keys have probably been expelled from the cache. There is first a flurry of cache misses but pretty soon large parts of the tree and the active keys will have been loaded into the cache and most activity takes place in the cache. Similar arguments can be made regarding the effects of input batching.

[0088] Switching from input mode to output mode is governed by whether an incoming record can be placed in memory. Switching back from output mode to input mode is governed by how large of an output batch is desired and supportable. There is no reason to switch back to the input mode until a free slot large enough to store the incoming record has been created. Beyond that, several rules are possible: (1) output a fixed number of records; (2) fill one output buffer, issue a write and switch; and (3) output until a certain percentage of the memory space has been freed up. In the experiments described below, the first rule is employed.

[0089] FIG. 11 shows a batched replacement selection process. It consists of two cooperating process loops: (1) batch input, including mini-run creation and addition to the selection tree; (2) batch output, including record selection using the selection tree. The process loops may execute simultaneously such that, for example, mini-runs are being created for input to the selection tree at the same time that records in the tree are being processed. Alternatively, as illustrated in FIG. 11, the batch input and batch output loops may execute in alternating fashion, switching back-and-forth between the input and output modes. The batched replacement selection process described in FIG. 11 is performed in software, preferably by the batched RS module 80 executing on the processor 52.

[0090] Beginning with the batch input process, multiple records are read from storage 70 into the main memory 74 until memory becomes full (steps 200, 202 and 206). At the same time, the mini-run assembly data structure 84 is filled (steps 206 and 208). Once the data structure 84 is fill, the records in the data structure 84 are sorted to form a mini-run. At step 210, the run number of each record in the full data structure 84 is determined. The records are then sorted in the data structure 84 to form a mini-run (step 212). The mini-run is added to the selection tree by inserting its first record as the last external node in the tree and the tree is updated to reflect the addition of the new node (step 214).

[0091] Meanwhile and perhaps concurrently with the mini-run creation process, a tree process is underway extracting, in sorted order, the records contained in the selection tree. The tree process selects a record with the lowest key for output (step 216). If this is not the last record in the mini-run (i.e., the “no” branch from step 218), the next record in the mini-run is inserted into the external node (step 220). When a mini-run becomes empty, however, there is no replacement record. Accordingly, the node is removed to shrink the selection tree (step 222).

[0092] The selected records are collected and held in a buffer in main memory 74 for batch output to the data storage 70 (step 224). When the buffer becomes fill, the process writes it out to the data storage 70 (step 228). After a predetermined number of records have been collected and written out, the process switches back to the input mode for batch input of data records (step 232).

[0093] A series of experiments were conducted to evaluate the performance of batched replacement selection in comparison with other run formation techniques. All experiments were run on 300 MHz Pentium II processor with a 512 KB unified L2 cache and two 16 KB L1 caches, one for data and one for instructions. The machine had 128 MB of memory and two 9GB SCSI disks (Seagate, Barracuda 9, Ultra SCSI).

[0094] Unfortunately, the cache miss penalty on Pentium II processors is not a simple number; it depends on the circumstances. Pentium II is a superscalar processor with out-of-order execution. This means that when one instruction stalls on a cache miss, the processor attempts to continue executing other instructions. If so, this masks some of the cache penalty. Furthermore, reading or writing to memory is not a simple fixed-duration operation; a read/write request may have to wait for the bus and/or memory module to serve other requests. Accordingly, there is an assumption that an L1 cache miss costs 5 cycles and an L2 cache miss costs 50 cycles. Though probably not entirely correct, these estimates are off by at most a factor of two. On the latest 500 MHz Pentium III processors, 10 cycles and 100 cycles, respectively, are better estimates.

[0095] In the experiments, four methods were compared: classic replacement selection, batched replacement selection, load-sort-store using Quicksort, and load-sort-store using AlphaSort. AlphaSort is a cache-conscious in-memory sorting algorithm that is well suited for run formation. In all experiments, records were generated randomly in memory (not read from disk). The key of each record was a string randomly selected from a list of 25,084 English words, extended with a filler character to a total of 30 characters.

[0096] The first series of experiments focused on the CPU behavior of the various techniques without I/O. Records were copied into output buffers but the actual writes were omitted. Two series of experiments are reported here: one with 100-byte fixed-length records and one with variable length records with an average length of 100 bytes.

[0097] FIG. 12 shows the overall performance of the four methods—Quicksort, Alphasort, classic replacement selection, and batched replacement selection—as a function of memory size. Performance is measured by throughput in MB per second (i.e., the total amount of sorted run data output per second by each method). The first observation is that throughput decreases as memory size increases. Note though that as memory size increases, fewer runs are generated and more of the sort work has been done. Second, the two cache-conscious methods (AlphaSort and batched replacement selection) are faster than their cache-oblivious counterparts (Quicksort and classic replacement selection). Third, AlphaSort is the fastest overall, but it produces shorter runs than replacement selection (i.e., it does a smaller share of the overall sort work).

[0098] Table 1 provides more detailed information on elapsed time in CPU cycles, instructions executed and cache misses for memory size 20MB. All figures are per record processed. First, the batched replacement selection generates about 50%. fewer L2 cache misses than classical replacement selection. The number of L1 cache misses also drops but significantly less. Second, reducing L2 cache misses pays off. For fixed-size record, batched replacement selection actually executes 7.7% more instructions than the classical algorithm (2688 versus 2494) while running 16.7% faster (4776 versus 5737 cycles). For variable-size records, batched replacement selection is as much as 33.5% faster. Finally, the number of clock cycles per instructions is high. The Pentium II is a superscalar processor capable of retiring two instructions per cycle, which translates to 0.5 cycles per instructions. Here, between 1.78 and 2.30 cycles per instructions were observed, meaning that the processor runs at about a quarter of its peak performance.

[0099] Table 2 compares the number of key comparisons performed by the various methods. A large part of the total time is consumed by key comparisons so it is important to keep the number of comparisons low. The number of comparisons for batched replacement is slightly lower when records are of variable size because fewer records fit in memory and runs are slightly shorter.

[0100] Table 3 lists the number of runs output and average run length (relative to memory size) for different memory sizes and an input size of 50× the memory size. The experiments show that batched replacement selection produces slightly longer runs than the traditional replacement selection, mostly because the selection tree requires less space. Compared with Quicksort and AlphaSort, batched replacement selection produces about 45% fewer runs for fixed-length records and about 40% fewer for variable-length records.

[0101] Two series of experiments were run with I/O included: one series writing to a single disk and one series writing to two disks in parallel. (The input records were generated on the fly so no input disks were required in these experiments.) All experiments used output buffers of size 64 KB. To allow concurrent writing to disk and CPU processing, the number of buffers was set to the number of disks plus one. Writes were asynchronous, allowing full overlap of I/O and CPU processing. All writes bypassed the file system cache completely, thereby. avoiding one round of copying.

[0102] FIG. 13 plots the throughput for one disk and FIG. 14 for two disks. Replacement selection overlaps CPU processing and writing completely so the total throughput is limited by the slower of the two activities.

[0103] FIG. 13 shows that the maximum output rate for a single disk is around 4.5 MB/s. For fixed-sized records, both versions of replacement selection are able to keep up with the disk. When records are of variable length, more time is spent on memory management. Batched replacement selection is still able to keep up with the disk, but the traditional version cannot and its throughput drops as memory size increases. When writing runs to a single disk, batched replacement selection was the fastest and produced the fewest runs.

[0104] In FIG. 14, the situation is different when the output bandwidth increases. When writing to two disks, the maximum output rate is around 8 MB/s. When memory is small and records are of fixed size, replacement selection is fast enough to keep both disks busy but the output rate drops when memory size increases or records are of variable length.

[0105] Although the invention has been described in language specific to structural features and/or methodological steps, it is to be understood that the invention defined in the appended claims is not necessarily limited to the specific features or steps described. Rather, the specific features and steps are disclosed as preferred forms of implementing the claimed invention.