Split directory-based cache coherency technique for a multi-processor computer system

[0002] The present invention is related to the subject matter disclosed in U.S. patent applications Ser. No. 08/992,763 filed Dec. 17, 1997 for “Multiprocessor Computer Architecture Incorporating a Plurality of Memory Algorithm Processors in the Memory Subsystem”; U.S. Ser. No. 09/008,871 filed Jan. 20, 1998 for “Scalable Single System Image Operating Software Architecture for a Multi-Processing Computer System”; and U.S. Ser. No. 09/018,032 filed Feb. 3, 1998 for “System and Method for Dynamic Priority Conflict Resolution in a Multi-Processor Computer System Having Shared Memory Resources” all assigned to SRC Computers, Inc., Colorado Springs, Colo., assignee of the present invention, the disclosures of which are herein specifically incorporated by this reference.

[0003] The present invention relates, in general, to the field of multi-processor computer systems. In particular, the present invention relates to a split directory-based cache coherency technique for a multi-processor computer system.

[0004] The advent of low-cost high-performance microprocessors has made large-scale multiprocessor computers feasible. In general, these microprocessors are cache-oriented; that is, they maintain a subset of the contents of main memory in high-speed storage close to the processor to improve the access latency and bandwidth of frequently-used memory data. This local memory can become inconsistent if one processor changes an element of memory by modifying its local cache and then the change is not propagated to all processors that share that memory. The precise structure of such caches varies greatly depending on the system design.

[0005] This caching problem has led to two basic architectures sometimes known as “shared memory” and “partitioned memory”. In a shared memory system, algorithms are used to maintain the consistency of the shared data. Typically, in commercially successful systems, the consistency is implemented by hardware and is invisible to the software. Such systems are called “cache-consistent” and form the basis of almost all multiprocessor computer systems produced. On the other hand, the partitioned memory approach disallows sharing of memory altogether or allows sharing by only a small number of processors, thereby simplifying the problem greatly. In such computer systems, larger configurations are created by connecting groups of computer systems with a network and using a message-passing paradigm that is most often made visible to the application software running on the system.

[0006] The development of cache coherent systems has led to some fundamental design problems. For large-scale systems, the data transmission and speed limitations make cache coherency difficult to achieve successfully. Coherency operations transmitted across the communications channel have traditionally been limited by low bandwidths, thus reducing overall system speed. Large-scale systems containing a high number of processors require accurate and high-speed cache coherency implementations.

[0007] With this in mind, some fundamental issues must be resolved in order to maintain a consistent view of memory across processors. First, processors must follow an arbitration protocol that grants permission to a processor to read or modify memory contents. To perform this function, coherency protocols divide memory into fixed “lines”, (subsections of memory, typically 32, 64, or 128 bytes in size) that are treated as an atomic unit. Typically, each line is allocated to a single processor in “exclusive mode”, which allows writing, to one or more processors in “read-only mode”, or that line is currently not cached. A processor is required to request a line in exclusive or read-only mode when loading it from the memory. In order to support this, the cache must allow the memory subsystem to delay completion of a request while the state of the line is analyzed and operations are performed on the processor cache while the system is waiting for an operation to complete.

[0008] The process of moving a line from one processor to another, when that is required, can be done in many ways. One of these approaches is termed “invalidation based” and is the technique most frequently used in existing multi-processor computer systems. In such systems, lines are removed from other processors' caches when the contents of a line are to be changed. Another approach allows for updating all caches containing the line when that line is changed.

[0009] The most common method of providing cache coherence is by using a “snoopy bus” approach. In such systems, all processors can monitor all memory transactions because they are all performed over a small number of buses, usually one or two. This approach cannot be used for large-scale systems because buses cannot supply the required data bandwidth from memory to the processors.

[0010] In such cases, most commonly a “directory” approach is used. Such systems use a database to record the processors to which lines are allocated. Transactions on memory require that the directory be examined to determine what coherency operations are required to allocate the line in question. The method of keeping the directory varies.

[0011] Many schemes have been proposed to record the contents of the directory. Most either require time-expensive searches when a directory inquiry is made or use broadcasting when the precise set of caches containing the line is too large to be recorded in the directory hardware. “Broadcasting”, in this context, means sending a message to all processors in the system, often by the use of special hardware features to support this style of communication. The difficulty with broadcasting is that switch-based networks do not easily support such operations, and the cost of interrupting processors with requests that do not involve their cache contents can be high.

[0012] In order to invalidate a line that is to be updated, all caches that contain the line must be contacted, which requires a decision as to which processors to contact. Once a list of processors that have allocated the line has been made from the directory, each processor must be sent a message instructing it to remove the line from the cache and to send any changes to the memory. This operation must be supported by the microprocessor cache hardware.

[0013] In order to provide processors with a cache-coherent view of shared memory resources, all of the processors in a multi-processor computer system must view all memory changes in a useful, predefined order. For the class of microprocessors disclosed in a preferred embodiment described in greater detail hereinafter (e.g. the Deschutes™ microprocessor developed by Intel Corporation, Santa Clara, Calif.), the coherency model is called “total store order”. This means that all memory changes made by a given processor are visible in the order in which they are made by that particular processor and are visible in that order to all processors in the system. Likewise, read operations do not cross conflicting write operations.

[0014] Nevertheless, the cache coherency technique disclosed herein is not limited to this particular coherency model and, in fact, can support all current models through the connection of memory to the processors with a cache communication network.

[0015] However, since the processors cannot view all transactions in such a system, the present invention contemplates the inclusion of reasonable-cost, complete directories with low-complexity directory lookup. This approach can be extended to allow even smaller directories with some broadcasting if desired, for a given application.

[0016] In order to provide coherency, the technique of the present invention requires extra data storage associated with each line of memory (a “coherency tag”) to hold parts of the directory. In addition, a secondary directory area is used for each memory controller. This secondary directory consists of entries that are used for widely-shared lines. In the embodiment disclosed, it is assumed that each such entry contains a bit for every processor on the system, which bit indicates whether that processor holds the line in question. In addition to the bit mask, in certain applications it may be desirable to keep a count of the number of bits that are set in the mask.

[0017] What is disclosed herein is a split directory-based cache coherency technique which utilizes a secondary directory in memory to implement a bit mask used to indicate when more than one processor cache in a multi-processor computer system contains the same line of memory. This technique thereby reduces the search complexity required to perform the coherency operations and the overall size of the memory needed to support the coherency system. The technique includes the attachment of a “coherency tag” to a line of memory so that its status can be tracked without having to read each processor's cache to see if the line of memory is contained within that cache. The technique disclosed may be further expanded to incorporate the “bus lock” capability of bus-based systems compatible with the requirements for multi-processor synchronization.

[0018] Particularly disclosed herein is a multi-processor computer system which comprises a plurality of processors, each having an associated data cache. The system further includes main memory and a communication network bidirectionally coupling the plurality of processors with the main memory to enable communication of data therebetween. The main memory comprises at least one directory having a number of entries therein, each of the entries containing a bit location for each of the plurality of processors in the computer system.

[0019] Further disclosed herein is a method for maintaining cache coherency in a multi-processor computer system which includes a main memory and wherein each of a plurality of processors has an associated data cache. The method comprises the steps of providing a communication network coupling the plurality of processors to the main memory; also providing at least one directory in the main memory having a number of entries therein and designating a bit location in the directory for each of the plurality of processors in the computer system.

[0020] The aforementioned and other features and objects of the present invention and the manner of attaining them will become more apparent and the invention itself will be best understood by reference to the following description of a preferred embodiment taken in conjunction with the accompanying drawings, wherein:

[0021] FIG. 1A is a functional block diagram of an exemplary multi-processor computer system implementing the cache coherency technique of the present invention;

[0022] FIG. 1B is a functional block diagram of an alternative embodiment of a multi-processor computer system implementing the cache coherency technique of the present invention wherein the cache communication network transmits the cache coherency commands between the processors and memory utilizing a Sebring ring interconnect strategy;

[0023] FIG. 2 is an exemplary illustration of the multi-processor computer system of the preceding FIGURES wherein two processor caches contain the same line of memory data;

[0024] FIG. 3 is an additional exemplary illustration of the multi-processor computer system of FIG. 2 wherein a single processor cache contains a given line of memory data;

[0025] FIGS. 4A and 4B are a functional block system overview illustrating a computer system in accordance with an embodiment of the present invention comprising between 1 and 16 segments coupled together by a like number of trunk lines, each segment containing a number of computational and service processors in addition to memory and a crossbar switch assembly; and

[0026] FIG. 5 is a simplified functional block diagram for the interconnect strategy for the computer system of FIGS. 4A and 4B allowing a total of three hundred twenty processors to access any of the two hundred fifty six memory banks comprising a shared memory resource (or main memory).

[0027] With reference now to FIG. 1A, a representative computer system 60 architecture for possible implementation of the split directory-based cache coherency technique of the present invention is shown. It should be noted that the interconnections shown in the representative embodiment illustrated can be provided in a wide variety of alternative implementations as the cache coherency technique disclosed is not ultimately dependent on how communications between the various system elements are implemented.

[0028] The particular computer system 60 shown includes a plurality of processors 16₀through 16_Nwhich are interconnected to a plurality of memory banks 50₀through 50_N(comprising a shared memory resource, or system main memory) through a communication network 68, comprising single or multiple communications resources, as will be more fully described hereinafter. The processors 16₀through 16_Nare also interconnected to the memory banks 50₀through 50_Nby means of an interconnect switch 70 comprising a number of memory switches 52₀through 52_N, processor switches 54₀through 54_Nand interconnecting trunk lines 14₀through 14_Nas described and shown with respect to the preceding FIG. 2. Each memory bank 50₀through 50_Nincludes portions 62₀through 62_Ncontaining coherency information corresponding to data portions 64₀through 64_Nas well as a secondary directory 66₀through 66_Nrespectively.

[0029] As an example only, the hypothetical computer system 60 may contain sixteen processors 16₀through 16₁₅and sixteen memory banks (or units) 50₀through 50₁₅interleaved at the line level to form a global main memory or shared memory resources. It should be noted that the cache coherency technique disclosed herein can be applied to any number of processors and any number of memory units of any size. The cache associated with each processor 16 in the exemplary computer system 60 is modeled as containing 4096 lines in a 4-way set-associative manner. Thus, each memory unit 50 can have at most 256 lines present in any one processor 16 cache at any given point in time because each memory unit 50 maps {fraction (1/16)} of the global memory Given that there are sixteen processors 16 in the computer system 60, the directory 66 for each memory unit 50 need not exceed 4096 processor-line pairs.

[0030] With reference additionally now to FIG. 1B, another exemplary computer system 80 architecture is shown wherein the cache communication network 68¹is implemented using a Sebring ring interconnect. Structure corresponding to that previously described with respect to FIG. 3 is like numbered and the foregoing description thereof shall suffice herefor.

[0031] The cache communication network 68¹provides a medium in which cache coherency commands can be transmitted between the memory 50 and the processors 16. By using the ring as designed, shared reads through main data paths are not delayed. This provides system performance similar to that which would be the case with no cache coherency checking. This ring also provides a high data bandwidth for cache coherency commands to be transferred.

[0032] In the computer system 80, the cache communication network 68¹transmits the cache coherency commands between the processors 16 and memory 50 while data transfers between the processors 16 and memory 50 are transmitted using the interconnect switch 70. The computer system 80, utilizing a Sebring ring interconnect provides significant benefits in terms of superior bandwidth, scalability, ease of use, and relatively low cost implementation.

[0033] With reference additionally now to FIG. 2 a portion 100 of the multi-processor computer system of the preceding figures is shown to illustrate the cache coherency technique of the present invention in those circumstances wherein two processor caches contain the same line of memory data. As shown, each of the two processors 16₁, 16₂illustrated include an associated cache 102₁, and 102₂each including the same memory line (“Line A”) 104. This same memory line 104 is also maintained in the computer system main memory 50 and has with it an associated coherency tag 106. The memory 50 also includes a secondary directory 108 comprising, in the exemplary implementation shown, 2048 sixteen bit entries with a corresponding count 110.

[0034] The coherency tag 106 of each memory line 104 may be twelve bits in length with the first (or any other bit) constituting a “flag” and the remaining bits serving as a pointer to a particular entry in the secondary directory 108. The secondary directory 108 is as wide as the maximum number of processors 16 for the computer system, which in the example described, is sixteen.

[0035] In operation, associated with each line in memory 50 are twelve bits of additional storage denominated a “coherency tag” 106 which is used for implementation of the cache coherency technique of the present invention. If the flag (in this case, bit 1) is set, then two processor caches 102 contain the same memory line 104 (Line A) and thus, the line 104 is allocated in “read-only mode”. In this case, the lower 11 bits of the coherency tag 106 point to an entry in the secondary directory 108.

[0036] Each set bit in the mask in the secondary directory 108 corresponds to a processor 16 holding the memory line 104. Thus, each secondary directory 108 entry contains sixteen bits (one bit for each processor 16 in this example). Because such an entry is required only when at least two processors 16 hold the line 104, the secondary directory 108 needs to contain only 2048 entries.

[0037] In addition, a count 110 of the number of set processors 16 holding the line 104 may also be provided. This count 110 may be kept in an array parallel to the secondary directory 108 or, alternatively, if space is available, the count can be kept in the coherency tag 106.

[0038] With reference additionally now to FIG. 3, a corresponding portion 120 of the multi-processor computer system of FIG. 2 is shown to illustrate the situation wherein only a single processor 16 cache 102₁, contains a given memory line 104 (Line A) of memory data and, for example, cache 102₂does not contain Line A. In this figure, like structure to that previously described and shown with respect to FIG. 5 is like numbered and the foregoing description thereof shall suffice herefor.

[0039] In this instance, the coherency tag 106 includes a “0” in the bit 1 “flag” position as well as a “1” in both the bit 2 and bit 3 positions. A “1” in the bit 2 position may be used to indicate whether the memory line is held in one of the processor 16 caches 102 while a “1” in the bit 3 position indicates whether the line is in either a “read-only” or “exclusive” mode. Bits 4 and 5 may correspond to a “bus lock” and “busy flag” indication. Bits 6 through 8 inclusive are unused if the memory line 104 is held in only a single cache 102 while bits 9 through 12 inclusive identify the single processor 16 that contains the memory line 104, in this exemplary illustration, processor 16₁(bits 9-12=“0001”).

[0040] In this illustration, if the flag (i.e. the first bit) is not set, the second bit indicates whether the line 104 is held in exactly one cache 102 or not. If the line is held in a cache 102, the third bit is used to specify whether the line is held by the processor 16 in “read-only” or “exclusive” mode. Another four bits are used to identify the processor 16 that holds the line 104. Thus, the state of a line 104 can be fully described using the coherency tag 106 and the secondary directory 108 and no search needs to be made other than the lookup of the coherency tag 106 and possibly the secondary directory 108 entry.

[0041] In the embodiment of the present invention illustrated and described, the coherency tag is a tuple:

[0042] <sde, active, exclusive, procid, addr>

[0043] Where:

[0044] Although a particular implementation of the cache coherency technique of the present invention has been shown and described in detail, many ways of storing and reorganizing this information may be utilized instead. In any reasonable implementation however, the coherency tag 106 is much smaller than a full directory entry in the case of large systems. This coherency tag 106 can be used to point to a directory entry if required and can be used for the full coherency information if the memory line 104 is located in at most one cache 102. In addition, as will be discussed more fully hereinafter, other information can be stored in the coherency tag 106 when the line 104 is in transition or held in at most one cache 102.

[0045] Extensions to the coherency tag 106 may be useful in those transactions in which one processor 16 requests exclusive access to a memory line 104 shared by more than one processor 16 and other transactions requiring invalidation operations. In this case, the data for the request can be returned across the interconnect switch 70 (FIGS. 3 and 4) immediately, but the processor 16 must be prevented from proceeding until the memory line 104 is removed from all caches 102 holding that memory line 104. The coherency tag 106 as described above can be augmented with a busy flag (i.e. bit 5) so that any request for the line 104 can be told to wait, without requiring a comparison against a list of lines being processed. When the invalidations have all been received, the new coherency tag 106 can be written and the memory line 104 moved out of the busy state.

[0046] Other uses of the coherency tag 106 are possible and quite advantageous. For example, multi-processor computer systems require synchronization primitives. The Intel Deschutes™ processor 16 architecture implements a “bus lock” capability which allows a processor 16 to obtain exclusive access to memory 50 in order to perform a sequence of read and write operations in an atomic fashion. In a large network-based system, implementing such a lock would be difficult and costly in performance. The bus lock capability is used primarily to read, test and then optionally modify one memory line or two contiguous memory lines. To simulate the bus lock, a bit in the coherency tag 106 (i.e. bit 4) can be used as a “flag” for a locked line. Once the lines for an atomic operation are read and locked, the processor 16 can test the values, issue the write operations and then release the lock. Because Intel locking primitives touch at most two lines, (and those two lines must be contiguous) no “deadlock” or “livelock” condition is possible in a practical program.

[0047] As previously described, most prior schemes which have been used to perform cache coherency do not lend themselves towards large multi-processor computer system architectures. The cache coherency technique of the present invention overcomes these limitations. The technique of the present invention uses a directory approach which creates a listing of processor caches that contain the line of memory.

[0048] In the specific embodiment of the present invention disclosed herein, it is assumed that a total store order is desired and that the cache coherency system provides sufficient control of the processor to allow modeling of the cache contents. Many other methods of accomplishing this are possible and the technique disclosed is applicable to all of them.

[0049] With reference additionally now to FIGS. 4A and 4B, a symmetric multi-processing computer system 10 in accordance with the present invention is shown. The exemplary computer system 10 comprises, in pertinent part, any number of interconnected segments 12₀through 12₁₅, although the principles of the present invention are likewise applicable to any scalable system having large numbers of processors. The various segments 12₀through 12₁₅are coupled through a number of trunk lines 14₀through 14₁₅as will be more fully described hereinafter.

[0050] Each of the segments 12 comprises a number of functionally differentiated processing elements in the form of service processors 16₀through 16₃(service processor 16₀functions additionally as a master boot device) and computational processors 18₀through 18₁₅. The service processors 16 are coupled to a number of peripheral component interconnect (“PCI”) interface modules 20, and in the embodiment shown, each service processor is coupled to two such modules 20 to enable the service processors 16 to carry out all of the I/O functionality of the segment 12.

[0051] The service processor 16 further includes a serial interface 22 for coupling a system console 24 to at least one of the segments 12 of the computer system 10. The system console 24 is operational for enabling a user of the computer system 10 to download boot information to the computer system 10, configure devices, monitor status, and perform diagnostic functions. Regardless of how many segments 12 are configured in the computer system 10, only one system console 24 is required.

[0052] The boot device 26 (for example, a JAZ® removable disk computer mass storage device available from Iomega Corporation, Roy UT)is also coupled to the master boot service processor 16₀through one of the PCI modules 20. The PCI modules 20 coupled to service processors 16₁through 16₃are utilized to couple the segment 12 to all other peripheral devices such as, for example, disk arrays 28₀through 28₅, any one or more of which may be replaced by, for example, an Ethernet connection.

[0053] The computer system 10 comprises sophisticated hardware and building blocks which are commodity based, with some enhancements to accommodate the uniqueness of high-performance computing (“HPC”). On the hardware side, the base unit for the computer system 10 is a segment 12. Each segment 12 contains computation and service processor 18, 16 elements, memory, power supplies, and a crossbar switch assembly. The computer system 10 is “scalable” in that an end user can configure a system that consists of from 1 to 16 interconnected segments 12. Each segment 12 contains 20 total processors: sixteen computational processors 18 and four service processors 16. In a preferred embodiment, the computational processors 18 may reside on an individual assembly that contains four processors (e.g. the Deschutes™ microprocessor) and twelve interface chips (i.e. three per computational processor 18). Each computational processor 18 has an internal processor clock rate greater than 300 MHz and a system clock speed greater than 100 MHz, and the interface chips provide the connection between the computational processors 18 and the memory switches that connect to memory as will be described and shown in greater detail hereafter.

[0054] The service processors 16 may be contained on a service processor assembly, which is responsible for all input and output for the computer system 10. Each of the service processor assemblies contain a processor (the same type as the computational processor 18), three interface chips, a 128 Mbyte I/O buffer, and two bi-directional PCI buses. Each PCI bus has a single connector. All I/O ports have DMA capability with equal priority to processors. The PCI modules 20 serve dual purposes, depending upon which service processor 16 with which they are used. The PCI connectors on the master boot service processor 16₀are used to connect to the boot device 26. The serial port 22 is connected to the system console 24. The PCI modules 20 on the regular service processors 16₁through 163 are used for all other peripherals. Some of the supported PCI-based interconnects include small computer systems interface (“SCSI”), fiber distributed data interface (“FDDI”), high performance parallel interface (“HIPPI”) and others. Each PCI bus has a corresponding commodity-based host adapter.

[0055] The separation of service functions from computing functions allows for concurrent execution of numeric processing and the servicing of operating system duties and external peripherals as is disclosed in greater detail in the aforementioned patent applications.

[0056] With reference additionally now to FIG. 5, the interconnect strategy for the computer system 10 of FIGS. 1A and 1B is shown in greater detail in an implementation employing sixteen segments 12₀through 12₁₅interconnected by means of sixteen trunk lines 14₀through 14₁₅. As shown, a number of memory banks 50₀through 50₁₅, each allocated to a respective one of the computational processors 18₀through 18₁₅(resulting in sixteen memory banks 50 per segment 12 and two hundred fifty six memory banks 50 in total for a sixteen segment 12 computer system 10) form a portion of the computer system 10 and are respectively coupled to the trunk lines 14₀through 14₁₅through a like number of memory switches 52₀through 52₁₅. The memory utilized in the memory banks 50₀through 50₁₅may be synchronous static random access memory (“SSRAM”) or other suitable high speed memory devices. Also as shown, each of the segments 12₀through 12₁₅includes, for example, twenty processors (four service processors 16₀through 16₃and sixteen computational processors 18₀through 18₁₅) coupled to the trunk lines 14₀through 14₁₅through a corresponding one of a like number of processor switches 54₀through 54₁₅.

[0057] Each segment 12 interconnects to all other segments 12 through the crossbar switch. The computer system 10 crossbar switch technology enables segments 12 to have uniform memory access times across segment boundaries, as well as within the individual segment 12. It also enables the computer system 10 to employ a single memory access protocol for all the memory in the system. The crossbar switch may utilize high-speed Field Programmable Gate Arrays (“FPGAs”)to provide interconnect paths between memory and the processors, regardless of where the processors and memory are physically located. This crossbar switch interconnects every segment 12 and enables the processors and memory located in different segments 12 to communicate with a uniform latency. In a preferred embodiment, each crossbar switch has a 1 clock latency per tier, which includes reconfiguration time. For a sixteen segment 12 computer system 10 utilizing three hundred and twenty processors 16, 18 only two crossbar tiers are required.

[0058] As mentioned previously, the computer system 10 may preferably utilize SSRAM for the memory banks 50 since it presents a component cycle time of 6 nanoseconds. Each memory bank 50 supports from 64 to 256 Mbytes of memory. Each computational processor 18 supports one memory bank 50 , with each memory bank 50 being 256 bits wide, plus 32 parity bits for a total width of 288 bits. In addition, the memory bank 50 size may be designed to match the cache line size, resulting in a single bank access for a full cache line. Read and write memory error correction may be provided by completing parity checks on address and data packets.

[0059] The parity check for address packets may be the same for both read and write functions wherein new and old parity bits are compared to determine whether or not the memory read or write should continue or abort. When a memory “write” occurs, a parity check may be done on each of the data packets arriving in memory. Each of these data packets has an 8-bit parity code appended to it. As the data packet arrives in memory, a new 8-bit parity code is generated for the data packet and the old and new parity codes are compared. The comparison results in one of two types of codes: single bit error (“SBE”) or double-bit or multi-bit error (“DBE”). The single-bit error may be corrected on the data packet before it is entered in memory. In the case of a double-bit or multi-bit error, the data packet is not written to memory, but is reported back to the processor, which retries the data packet reference. When a memory “read” occurs, each of the data packets read from memory generates an 8-bit parity code. This parity code is forwarded with the data to the processor. The processor performs single error correction and double error detection (“SECDED”) on each data packet.

[0060] While there have been described above the principles of the present invention in conjunction with specific multi-processor computer system architectures, directory structures and coherency tag contents it is to be clearly understood that the foregoing description is made only by way of example and not as a limitation to the scope of the invention. Particularly, it is recognized that the teachings of the foregoing disclosure will suggest other modifications to those persons skilled in the relevant art. Such modifications may involve other features which are already known per se and which may be used instead of or in addition to features already described herein. Although claims have been formulated in this application to particular combinations of features, it should be understood that the scope of the disclosure herein also includes any novel feature or any novel combination of features disclosed either explicitly or implicitly or any generalization or modification thereof which would be apparent to persons skilled in the relevant art, whether or not such relates to the same invention as presently claimed in any claim and whether or not it mitigates any or all of the same technical problems as confronted by the present invention. The applicants hereby reserve the right to formulate new claims to such features and/or combinations of such features during the prosecution of the present application or of any further application derived therefrom.