US 6,785,776 B2Grant
DMA exclusive cache state providing a fully pipelined input/output DMA write mechanism
Issue Date:2004-08-31
•21 Claims
•5 Drawing Sheets
Abstract
A data processing system that provides a DMA Exclusive state that enables pipelining of Input/Output (I/O) DMA Write transactions. The data processing system includes a system processor, a system bus, a memory, a plurality of I/O components and an I/O processor. The data processing system further comprises operational protocol providing a pair of instructions/commands that are utilized to complete a DMA Write operation. The pair of instructions is DMA_Write_No_Data and DMA_Write With_Data. DMA_Write_No_Data is an address-only operation on the system bus that is utilized to acquire “DMA ownership” of a cache line that is to be written. The initial ownership of the cache line is marked by a weak DMA state (D1), which indicates that the cache line is being held for writing to the memory, but that the cache line cannot yet force a retry of snooped operations. When each preceding DMA Write operation has completed or each corresponding DMA_Write_No_Data operation has been placed in a DMA Exclusive (D2) state, then the next cache line in a weak DMA state transitions to a DMA Exclusive (D2) state, which forces a retry of snooped operations until the DMA_Write_With_Data transaction to memory is completed. Accordingly, DMA_Write_No_Data operations that are provided sequentially may be completed in a parallel manner on the system bus although the corresponding DMA_Write_With_Data is held until a DMA Exclusive state attaches to the cache line. Also, the DMA_Write_With_Data may be completed out of order and once the write of the cache line is completed, the coherency state transitions from the DMA Exclusive state to the MESI Exclusive (E) state or the MESI Invalidate (I) state depending on processor operating characteristics.
Metadata
Assignee
- International Business Machines Corporation
Inventors
- Ravi Kumar Arimilli
- George William Daly, Jr.
- Paul K. Umbarger
Application Information
Application Number:US 09/915,669
Filing Date:2001-07-26
Priority Date:2001-07-26
Art Unit:7
Classifications
IPC:
G06F 1200
Field of Search:
711144711146711140711145711141710 22
Patent Drawings (5 sheets)
Description
Related Applications
[0002] The present invention is related to the subject matter of commonly assigned U.S. patent application Ser. No. 09/915,432, entitled “Microprocessor System Bus Protocol Providing Fully Pipelined Input/Output DMA Write Mechanisms,” filed concurrently herewith.
Background of the Invention
[0003] 1. Technical Field
[0004] The present invention relates generally to data processing systems and in particular to input/output (I/O) mechanisms of a data processing system. Still more particularly, the present invention relates to a method and system for providing fully pipelined I/O Direct Memory Access (DMA) write operations via utilization of a DMA Exclusive cache state.
[0005] 2. Description of the Related Art
[0006] A standard data processing system comprises one or more central processing units (CPU), one or more levels of caches, one or more memory, and input/output (I/O) mechanisms all interconnected via an interconnect. Traditionally, the interconnects utilized consisted primarily of a system bus and an I/O bus. In newer processing systems, however, particularly those with large numbers of CPUs and distributed memory, a switch is often utilized as the interconnecting mechanism.
[0007] In addition to the major components, data processing systems today are often equipped with an I/O controller, which controls I/O operations for the various I/O devices. More than one I/O controller may be utilized, each supporting particular I/O devices via an I/O channel, and the I/O controllers may be coupled to the interconnect via an I/O bus. Further, new processing systems typically comprise a plurality of paths (buses) for routing transactions between the I/O controller and the memory or distributed memory. Each path includes a series of latches, etc., and may each have different transmit times/latency based on the distance to/from the memory and number of latches, etc. Data is transmitted along these paths in a packet-like manner and each data packet may have different access latencies. Thus, in operation, data A written to a first memory or memory location may have a different access latency than data B written to a second memory or memory location if data A travels on a different path than data B.
[0008] Computer systems typically provide at least one system bus and a system memory area that is predominantly used by one or more processors for computation and data manipulation. I/O is sometimes performed by the processor. However, utilization of the CPU to perform input/output (I/O) transfers for these peripheral devices and subsystems places a burden on the CPU and negatively affects the CPU's efficiency. Thus, Direct Memory Access (DMA) controllers have been provided in computer systems for off-loading transaction work from the CPU to a dedicated controller, in order to increase the availability of the CPU to perform computational and other tasks.
[0009] Each DMA operation is a specialized processor operation that transfers data between memory and I/O devices. The DMA transaction operates as a master on the I/O bus and is frequently a part of the I/O controller. When, the I/O controller completes the DMA task, the I/O controller signals (i.e., sends an interrupt to) the processor that the task specified is complete.
[0010] The DMA controllers free the processor from I/O tasks and usually perform transfers more efficiently. DMA I/O transfers can also be performed by the devices themselves. This type of device is referred to as a “bus master” because it is capable of acquiring a bus and transferring data directly to and from memory or devices located on the bus.
[0011] The application software or device driver performs data communication with the device by writing or reading the data to or from memory and signaling the device or DMA controller to perform the transfer. A DMA transfer can also be performed from one device to another device using two discrete DMA transfers, one writing to memory, i.e., a DMA Write, and the second reading from memory, i.e., a DMA read. With a DMA Write, the input device data is transferred to memory from the input device by a DMA controller or by the input device if it is a bus master and the data is written to system memory.
[0012] The I/O channels provide input and output commands to and from peripheral components, respectively. Standard, logical operation of current processing systems requires that operations to memory be completed in the order in which they are received (i.e., sequential program order). Thus, the I/O channels operate as a First In First Out (FIFO) devices because the I/O writes to system memory from a device must be “ordered” to the system memory. That is, for example, an I/O DMA Write command of a 128 Byte cache line A that is sequentially followed by an I/O DMA Write command of a 4 Byte cache line B has to be completed (i.e., data written) before the write of cache line B can begin execution. The write data B request is placed in the FIFO queue at the I/O controller and waits on the receipt of a completion signal from the write data A operation. The processor begins execution of write data B command only after receipt of a completion signal.
[0013] FIG. 2A illustrates a sample timing diagram by which the writes of data A and data B are completed according to the prior art. As shown, DMA Write A 201 is issued at time 0 (measured in clock cycles) and a corresponding snoop response 203 is generated and received several cycles later. When the clean snoop response 203 is received, often after several retries of DMA Write A 201, the acquisition and transmission of data A to the memory block is undertaken over the next few cycles. Then, the actual writing (storage) of data A 205 is completed over several cycles. Following the completion of the write data A 205, an acknowledgment 207 is sent to the processor to indicate the completion of the write data A operation. Once the acknowledgment 207 is received, the DMA Write B data 209 commences and takes several cycles to complete (see snoop response 211 and B data to storage 213). Data B is then stored in memory. Since no operation is issued to the I/O bus while the DMA Write data A operation is completing, the bus remains idle for several cycles and write data B 209 is held in the FIFO queue.
[0014] Once the write A command is issued, the processor waits for the return of a tag or interrupt generated by the successful completion of the previous write data A operation. When the tag or interrupt returns, this indicates that data A storage to memory is completed, and the CPU can then issue the read data B command.
[0015] The logical structure of processing systems requires that I/O operations be ordered in the I/O channel. Thus, the I/O channel must write the data to memory “in-order” and also must wait until the successful completion of the previous operation before issuing the next operation. This waiting/polling is required because, as in the above example, if write B is issued prior to the completion of write A in current systems, write B would be completed before write A because of the smaller size of data B. This would then cause corruption of data and the corrupted data would propagate throughout the execution of the application resulting in incorrect results being generated and/or possibly a stall in the processor's execution.
[0016] The long latency in completing some write operations, particularly those for large data such as data A, coupled with the requirement that the next operation cannot begin until after the completion of the previous write operation, significantly reduces overall processor efficiency. The present architectural and operation guidelines for processing systems that require the maintenance of the order when completing operations is proving to be a significant hurdle in development of more efficient I/O mechanisms. Currently, system developers are looking for ways to streamline the write process for I/O operations. Pipelining, for example, one of the key implementation techniques utilized to make CPUs faster, has not been successfully extended to I/O transactions because of the requirement that the previous data operation be completed prior to the next operation beginning. Current DMA transactions operate as single threaded transactions (or in a serialized manner), and there is currently no known way to extend the benefits of pipelining to DMA operations. One method suggested to reduce the latency is to move the I/O controllers closer to the I/O device thereby reducing the transmission time for acquisition of the data on the bus. However, because most of the latency in I/O transactions is tied to the wait for completion requirement and not the actual transmission of the data, these methods do not solve the problem of long latencies for I/O DMA operations.
[0017] The present invention recognizes that it would be desirable to provide a method, system and I/O processor coherency protocol that enables pipelining of I/O DMA Write operations. A method, system, and processor logic that enables a cache state that allows pipelining of serially provided DMA Writes in an I/O subsystem would be a welcomed improvement. These and other benefits are provided by the invention described herein.
Summary of the Invention
[0018] Disclosed is a method and data processing system that provides pipelining of Input/Output (I/O) DMA Write transactions. An I/O processor's operational protocol is provided with a pair of instructions/commands that are utilized to complete a DMA Write operation. The instructions are DMA_Write_No_Data and DMA_Write_With_Data. DMA_Write_No_Data is an address-only operation on the system bus that is utilized to acquire ownership of a cache line that is to be written. The ownership of the cache line is marked by a weak DMA ownership state, which indicates that the cache line is being held for writing to the memory, but that the cache line cannot force a retry of snooped operations. When all preceding DMA Write operations complete or each corresponding DMA_Write_No_Data operation has acquired the cache line exclusively for the DMA operation, then the weak DMA ownership state is changed to a DMA Exclusive state. The DMA Exclusive state causes a retry of snooped operations until the write transaction to memory is completed. In this way, DMA Writes that are provided sequentially may be issued so that their respective operations occur in a parallel manner on the system bus and their corresponding DMA_Write_No_Data operations may be completed in any order, but cannot be made DMA Exclusive unless the above conditions are satisfied.
[0019] Further, once a DMA Exclusive state is acquired, a DMA_Write_With_Data may be issued for each of the sequential DMA Write operations in the DMA Exclusive state. The DMA_Write_With_Data may then be completed out-of-order with respect to each other. However, the system processor is sent the completion messages of each DMA_Write_With_Data operation in the sequential order in which the DMA Write operations were received, thus adhering to the I/O processor's requirements for ordered operations, while providing fully-pipelined (parallel) execution of the DMA transactions.
[0020] According to a preferred embodiment, weak DMA ownership is indicated by an affiliated cache state (D1). Likewise, DMA Exclusive is also indicated by an affiliate cache state (D2). A cache line transitions from D1 to D2 once DMA Exclusive ownership is acquired by the requesting process. After the cache line is written to memory, the D2 state transitions to either MESI Invalid or Exclusive states dependent on the system's operational requirements.
[0021] The above as well as additional objectives, features, and advantages of the present invention will become apparent in the following detailed written description.
Brief Description of the Drawings
[0022] The novel features believed characteristic of the invention are set forth in the appended claims. The invention itself, however, as well as a preferred mode of use, further objectives, and advantages thereof, will best be understood by reference to the following detailed description of an illustrative embodiment when read in conjunction with the accompanying drawings, wherein like numbers correspond to like elements and further:
[0023] FIG. 1 is a block diagram illustrating the core components and peripheral devices of a data processing system in which the preferred embodiment of the invention may be implemented;
[0024] FIG. 2A is a timing diagram of a sample DMA Write transaction according to the prior art;
[0025] FIG. 2B is a timing diagram of multiple “pipelined” DMA transactions according to a preferred embodiment of the invention;
[0026] FIG. 3 illustrates a FIFO queue of several DMA Writes according to one implementation of the current invention;
[0027] FIG. 4 illustrates the operation of the invention with multiple DMA Write transactions pipelined in accordance with a preferred embodiment of the invention;
[0028] FIG. 5 is a flow chart of the process of pipelining DMA Write transactions according to a preferred embodiment of the present invention; and
[0029] FIG. 6 is a diagram of the coherency state transitions with a DMA Exclusive state according to a preferred embodiment of the invention.
Description of the Illustrative Embodiments
[0030] With reference now to the figures, and in particular with reference to FIG. 1, the core components of a computer system in which the present invention may be practiced are depicted in block diagram form. Processor 100 is coupled to system global memory 116, (which may include cache memories as well as planar memory arrays), by system global bus 112. Optional DMA controller 128B is coupled to system global bus 112, and optional DMA Controller 128A is coupled to local bus 122. Bus bridge 114 couples local bus 122 to system global bus 112, using bus bridge or Input/Output Channel Controller (IOCC) 114. Local bus 122 may be a local Peripheral Component Interconnect (PCI) or Personal Computer Memory Card International Association (PCMCIA) bus that connects peripheral device A 124, device B 126, and local memory 118.
[0031] The hardware architecture supports direct DMA transfers from device to device and device to local memory 118 transfers. In addition, a peripheral connected to local bus 122 is able to transfer data via direct memory access (DMA) to system global memory 116 through IOCC 114, which is designed to translate transactions on local bus 122 to transactions on system global bus 112.
[0032] The devices 124, and 126 are preferably bus masters, i.e., they generate the transactions locally. Otherwise, IOCC 114 is provided with a DMA controller capability. In one embodiment, a DMA Controller 128A is coupled to local bus 122, and a DMA transaction can be made between non-bus master devices 124, and 126 and local memory 118 without causing transactions to occur on system global bus 112. DMA controllers 128 are preferably utilized for embodiments wherein the devices exchanging data with memory are not bus masters and therefore need an external controller to perform DMA transfers.
[0033] According to the preferred embodiment of the invention, computer system of FIG. 1 provides multiple different address and communication paths between IOCC 114, DMA Controller 128A, DMA controller 128B and system global memory 116. System global memory 116 may be a distributed memory or single memory supporting multiple transfer paths to and from IOCC 114. Each path comprises latches and staging components, and each path may exhibit different access/transmission latencies.
[0034] The present invention provides a method and system of I/O operation by which DMA Writes are executed in a fully pipelined manner thereby decreasing latency between individual write operations. DMA Writes that are received in a serial order are executed parallel to each other (i.e., their respective operations may be issued sequentially in a pipe lined manner but execute on the system at the same time), and the actual writes to memory may be completed out of order without disturbing the logic sequence of the processor. The preferred embodiment of the invention comprises introduction of two new executable instructions (or commands) and modifications to the processor/cache logic (or IOCC) to support the two new instructions. Additionally, the invention introduces a “DMA Exclusive” (D) state for a DMA request of a cache line that is given weak ownership of the cache line. As utilized herein, the term “exclusive” means that the DClaim operation owns the line and can prevent others from accessing it. It is not synonymous with the exclusive (E) state of a cache line as provided by the MESI protocol. The E state of the MESI protocol contains valid memory data, while the DMA Exclusive state does not contain memory data and just provides (or indicates) cache line “ownership.”
[0035] Two new coupled instruction executables (commands) are provided:
[0036] (1) “DMA_Write_No_Data” and (2) “DMA_Write_With_Data.” The two operate as a pair of instructions that perform the operation of a single DMA Write request that is executed. DMA_Write_No_Data is an address only transaction for I/O (i.e., no data is transmitted). DMA_Write_No_Data is similar to a DClaim operation, which is utilized to gain ownership of the cache line to be written later. When a DMA_Write_No_Data is issued, the cache responds by forcing a retry or providing the DMA Write request with weak exclusive ownership of the cache line. For illustrative purposes, the invention is described with reference to DClaim operations, although any operation that provides the functional features of a DMA_Write_No_Data is applicable.
[0037] When the DMA_Write_No_Data is issued, the cache's snooper treats it like a DMA Write and responds accordingly, except that no data is provided even if the data is present in the cache and is clean. Thus, the DMA_Write_No_Data is issued, and when a snoop response indicates that the cache line is clean, i.e., it is in a valid state or there is no other processor arbitrating for the cache line, the DMA_Write_No_Data is provided weak ownership of the cache line. (The I/O DMA Controller is messaged that the cache line is available for a write operation). If the snoop response does not indicate a clean state for the cache line, the DMA Write No Data (or DClaim) may be reissued.
[0038] DMA_Write_With_Data is an actual write command that writes (stores) the data acquired from the cache line to memory. Unlike the DMA_Write_No_Data, which is issued as soon as a DMA Write request is received, the corresponding DMA_Write_With_Data is not completed until the DMA Write request gains protection (i.e., full/exclusive ownership) of the cache line, and the completion of the DMA_WRITE_WITH_DATA operation is not messaged to the IOCC until all preceding sequentially-ordered DMA_WRITE_WITH_DATA instructions are completed and in the idle state.
[0039] Utilizing the above DMA commands and the operational logic associated with each command, the DMA Write request transitions in a state-machine fashion from weak ownership of the cache line, to exclusive ownership, to a released state. Progression between states is controlled by the IOCC, which monitors the operations within its multiple state machines.
[0040] FIG. 3 illustrates a FIFO queue 303 of a DMA controllable I/O device 301 connected to several memory devices 307 via an I/O channel 305. Within FIFO queue 303 are several write commands, labeled alphabetically from A through K, which may each be completed on different memory blocks. According to the present invention, the write transactions are serially received within the FIFO queue but may be completed in a parallel fashion, wherein later writes to memory may be completed before earlier writes to memory (i.e., out-of-order write transactions).
[0041] FIG. 4 illustrates an example process by which the write transactions in the FIFO queue of FIG. 3 are handled according to the features provided by the invention. Serialized DMA Write requests are sent to FIFO 401 by processor or other components that generate the DMA Write requests. Placement of the DMA Write requests in FIFO 401 triggers the issuing of a DMA Exclusive claim (DClaim) operation, i.e., a DMA_Write_No_Data is issued on the bus. FIG. 4 illustrates the DMA_Write_No_Data 403 within latches. Thus, the DMA_Write_No_Data 403 are issued on the bus in parallel manner, and all propagate on the bus to snoop their respective cache lines.
[0042] A snoop response 405 is generated for each issued DMA_Write_No_Data 403. According to the illustrated embodiment, each issued DMA_Write_No_Data operation is assigned a single bit value indicating the snoop response received for the particular operation. As utilized herein, a bit value of 1 indicates a clean response, i.e., the DMA_Write_No_Data operation is provided with weak ownership of the requested cache line, and a bit value of 0 indicates a dirty response, i.e., no ownership of the cache line provided. According to the values indicated in FIG. 4, no ownership was provided to DMA_Write_No_Data for cache line D.
[0043] The invention thus involves the introduction of two bus transactions to complete each DMA Write, the first, “DMA_Write_No_Data”, and the second “DMA_Write_With_Data.” DMA controller logic generates the two DMA Write commands and controls the checking and monitoring of previous DMA operations. The instruction set and bus protocols are modified to support these two commands.
[0044] Returning now to FIG. 4, in one embodiment, the 1 value may indicate that the write transactions are fully DMA Exclusive while the 0 value indicates that a cache line is retried. Following the return of the snoop response, the DMA Write requests that received the clean responses are later protected, i.e., provided full DMA Exclusive ownership of the cache lines, and the actual write commands are issued as DMA_Write_With_Data 407 that may be completed in any order. According to the invention, DMA Exclusive ownership of the cache line simply provides the go ahead for the DMA_Write_With_Data operation even when intervening actions occur, for example, another processor seeking to modify the cache line. Thus, the cache line corresponding to DClaim (D) of FIG. 3 is not DMA Exclusive.
[0045] Ideally, all cache lines are DMA Exclusive, as this allows for pipelining of all the DMA Writes (i.e., the DMA_Write_With_Data). With a DMA Exclusive, the IOCC ignores other requests to provide ownership of the cache line other than to the DClaim, and each cache line stands on its own. Operations having DMA Exclusive state are coupled together and will each force a retry of any snoops against their protected cache line. The coupling effects of multiple DMA Exclusive states is thus provided, thus allowing writes to be completed to memory out-of-order.
[0046] In order to implement the DMA Exclusive state, rules are coded within the IOCC about how and when to give up these exclusive lines. According to the above example, assuming a clean response is referenced by a 1 and a retry is referenced by a 0, the illustrated stack of DMA_Write_No_Data operations indicate a 11101 snoop response. If a processor desires to access the cache line, which is weakly owned by the DClaim of DMA Write C, DMA Write C is forced to give up the line unless C has acquired a protection of the line. DClaim (C) may have to be retried to get the cache line or DMA Write C may simply issue as a normal DMA Write. When the cache line is in the protected state, however, i.e., exclusive ownership of the cache line is acquired, a processor request for the data within the cache line that is propagated on the bus has to wait until the data is stored to memory.
[0047] The actual DMA Write process 409 is completed in any order. Thus write A, B, and C operations may be completed in parallel and/or almost simultaneously. The completion acknowledgments 411 of the write operations are, however, messaged back to the IOCC in order. As indicated, once A, B, and C become fully exclusive (protected), the data is written to system global memory in “any” order, thus allowing fully pipeline transactions within the I/O channel. Completion in any order is allowed because each write may be issued to its respective memory block via a different path to memory.
[0048] Each write transaction is released (i.e., completion messaged to the IOCC) in order to ensure that the IOCC requirements for in-order completion of DMA Writes is complied with. Further, any attempt to read the memory block of a DMA_Write_With_Data prior to a completion of all other DMA_Write_With_Data sequentially ahead will be retried until the completion signal is messaged to the IOCC. A retry of DClaim (D) 413 is sent to the cache and when D finally receives ownership of the cache line, DMA_Write_With_Data for D is completed. However, no DMA_Write_With_Data operation for E is allowed until the operation for D is completed and there is no pipelining effect due to the un-surety of gaining ownership of the cache line requested by DClaim(D).
[0049] DMA_Write_Without_Data operations may be issued sequentially, i.e., the addresses are sent out on the bus and completed in parallel. The DMA operations want all cache lines to be DMA Exclusive, i.e., the DMA operations are each arbitrating for DMA Exclusive command/ownership of a particular cache line. When the DMA operation acquires the cache line in a fully DMA Exclusive state, then a write with data is issued. Otherwise, all processing of later write operations are placed on hold. The DMA_Write_With_Data commands are grouped and issued to the memory simultaneously or in a parallel fashion. Thus according to the example, A, B, and C are DMA Exclusive so they can be grouped and written to system memory in any order, allowing the I/O channel to fully pipeline the transactions.
[0050] With reference again to FIG. 4, DMA Write A, B, and C are in a strongly ordered mode and can be grouped for completion of the write operations. This is because all the snoop responses are clean and thus each write has ownership (albeit weak) of its cache line and may write the cache line once it gains protection for the cache line. The order of writing the cache line is not dependent on where the DMA Write request falls in the sequential order received from the IOCC. Thus A, B, and C form a strongly enforced grouping, while D and consecutive write commands form a weakly enforced grouping.
[0051] DMA Write D cannot be completed until DMA Write A, B, and C are completed. Thus, according to the invention the DClaims are immediately issued and parallely executed, but the actual writes are only issued when the DMA Write acquires exclusive ownership of the cache line and knows, based on the established rules, that it can write the data (although a DMA Write at the head of the DMA stack may proceed to write data without issuing a DMA_Write_No_Data as explained above). Once DMA Exclusive ownership of the cache line is acquired however, the data can be written in any order. According to the preferred embodiment, only the addresses of the cache lines are transmitted in the DClaim transaction and the data itself is only transferred during the actual DMA_Write_With_Data operation.
[0052] If another processor attempts to get ownership of data block E, then the processor is able to do so because the issuing IOCC cannot protect the data cache line since the previous DMA Exclusive request (on D) has to be retried on D. Therefore, the write request for E 415 has to be polled until a DMA Exclusive is received.
[0053] Thus, each of the issued operation has a response, which is monitored by the DMA controller logic to determine what actions to take with respect to the DMA operation or IOCC interaction. If a clean response is not received or if no response is received by a time-out period, a snoop is retried until an appropriate response is received. In one embodiment, if the DClaim is lost or the snoop returns with a dirty response, then the DMA Write is issued as a normal/regular DMA Write and no retry of the DClaim occurs.
[0054] DMA controller logic contains specific rules for the correct implementation of the DMA Write transactions and functionality. According to the preferred embodiment, whenever DMA Writes are encountered, DClaims are immediately issued and commence a state machine operation. When a clean response is received, the DClaim is given weak ownership of the cache line. Whenever a previous DMA_Write_No_Data has not gotten a clean snoop response, the corresponding DMA_Write_With_Data is not launched until all previous DMA Writes ahead of it have launched. However, for the first DClaim operation in a series of DClaim operations (e.g., DClaim of A), the DClaim write may be immediately launched. Also, while in a weakly own state, if another request is received for the cache line, for example, a read-with-intent-to-modify, then the DClaim is forced to give up the cache line.
[0055] Notably, in the preferred embodiment, no limitations exist on the DMA Write that is at the head of the sequential stack of DMA Write requests. Thus, DMA Write A does not have to issue a DMA Write No Data because it is the first DMA Write in the stack. IOCC logic checks for previous DMA Writes in the instruction sequence and, when there is no preceding DMA Write (i.e., the current DMA Write operation is the head of the stack), that DMA Write is “exempted” from having to first issue a DMA Write No Data and wait for a DMA Exclusive. Rather the DMA Write operation is sent directly to memory and the cache line is written to memory. Accordingly, at worst case, if each previous DMA Write completes before a subsequent DMA Write is placed in the FIFO, no performance degradation is seen from current systems, but substantial performance benefit is possible whenever multiple DMA Writes are sequentially issued/placed within the FIFO and pipelined.
[0056] Turning now to FIG. 5, there is illustrated the process of pipelining DMA Write transactions according to one implementation of the invention. The process begins at block 501 and thereafter proceeds to block 503 where a DMA_Write (N) is received. The process simplifies an operation in which several DMA Writes (e.g., DMA_Write (N−2), DMA_Write (N−1), DMA_Write(N), DMA_Write (N+1)) are provided in a pipelined fashion. DClaims are issued for each DMA transactions to secure ownership of the cache line that holds the data. DClaim (N−1) is sent independent of the state of DClaim (N−2) or DClaim (N), etc. If a clean response is received for a DClaim, then the corresponding DMA Write is provided exclusive ownership of the cache line. If a clean response is not received, the DMA Write retries its DClaim operation.
[0057] Returning now to FIG. 5, following the receipt of DMA Write(N), a DClaim (N) is issued for the cache line associated with DMA Write(N) as shown at block 505. DClaim (N) is issued immediately after the DMA Write(N) is received from the IOCC. In response to the issuing of the DClaim (N), a snoop response is received with status information as shown at block 507. A determination is made at block 509 whether the snoop response indicates a clean cache line. If the snoop response does not indicate a clean cache line, the DClaim (N) is retried as indicated at block 511. If a clean response is received, however, a next determination is made whether the DMA Write(N) is the first DMA Write of the application/program or whether the DMA Write(N−1) has received exclusive ownership of its cache line as shown at block 513. If neither conditions are true, the DClaim (N) is given weak ownership of the cache line as indicated at block 515. DClaim (N) weakly owns the line if it gets a clean response but DMA Write(N−1) not yet reaches protection point and must release ownership of the line if another request for that line is received.
[0058] Following, a determination is made at block 517, whether the DMA Write (N−1) has acquired exclusive ownership of its cache line, i.e., if the cache line is protected. If the cache line is not protected a next determination (or a monitoring function) is completed as shown at block 519, whether a request for the weakly owned cache line has been received. If no request has been received for the cache line or if DMA_Wr(N−1) has received protection or if the DMA Write(N) was the first write of the application, then DMA Write(N) is placed in the protected state and the DMA Write(N)_With_Data is issued to commence writing the data to memory as illustrated at block 523. DMA Write (N)_With_Data is executed because it is in the protected state (i.e., exclusive ownership of the cache line has been acquired). The protection of DMA Write must be released in the order in which the DMA Writes were received from the IOCC and a completion acknowledgment is issued once the previous writes have been completed as shown at block 525. Then, the process ends as illustrated at block 531.
[0059] Returning now to block 519, if a request for the cache line is received while the DClaim has weak ownership, the cache line is released to the requesting process as shown at block 521. Then the DMA Write(N) is forced to wait until DMA Write(N−1) is idle as depicted at block 527. Once DMA Write (N−1) (that is all previous write operations) is idle, then DMA Write (N) is issued as shown at block 529. Then the process ends as shown at block 531.
[0060] According to one embodiment, no change to the coherency protocol is required because the new DMA Exclusive (D2) state applies solely to DClaim operations and communication between the DMA and the processor holding the cache line. The D state is held by the IOCC and the IOCC will not give up the data to another snooper if the data is being held in that state.
[0061] In another embodiment, however, the invention introduces two new I/O coherency states to supplement the coherency protocol being utilized for I/O transactions. FIG. 6 illustrates the coherency states and their transitioning characteristics. Notably, the Invalid (I) and Exclusive (E) states of the MESI protocol provide the initial and final states of the transition. The D1 state indicates weak DMA ownership of a cache line and the D2 state indicates DMA Exclusive ownership of the cache line. As illustrated, a cache line's coherency state transitions from I to D1 when a DMA Write_No_Data is snooped that acquires ownership of the cache line. The coherency state transitions from D1 to D2 when the conditions described herein for DMA Exclusive ownership occur. From D2, the coherency state transitions either to E or to the I state depending on the operating characteristics of the data processing system. In the preferred embodiment, the D2 transitions to I once the cache line is written to memory. In an alternate embodiment, D2 transitions to E to allow other processor operations that wish to gain access to the line to do so. Transition from D1 to I is also possible and occurs whenever an operation is snooped for the cache line while the cache line is in the DMA weak ownership state.
[0062] The state transitions are provided within the I/O directory associated with the I/O cache according to the system configuration and operational protocols. The invention further contemplates the extension of the utilization of the D1 and D2 states to a processor cache and cache directory for other operations that may require temporary holding states.
[0063] Each DMA Write process represents a state machine that cycles from beginning, when the DMA Write first issues, to end, when the IOCC is messaged with the completion of the writing/storage of the data. The state machine is set to idle once it completes the storage of the data if special conditions are met. According to the invention, the logic forces the state machine to stay active (i.e., not go idle) if there are any state machines associated with previously issued DMA Writes that have not yet gone idle. Once the state machine goes idle, the IOCC is signaled that the corresponding DMA Write operation is completed. The commands may be executed in parallel, but must be seen as completed in order by the IOCC. Thus, a background kill operation on a previous write command has to be completed before the state machine goes to an idle state.
[0064] FIG. 2B illustrates the performance benefits of the present invention over the presently available methods. Similar operations are tracked as those provided in FIG. 2A and numbered alike. Unlike FIG. 2A, however, DMA Write B (i.e., DClaim and DMA_Write_With_Data) is issued in parallel to corresponding operations of DMA Write A. Ownership of the associated cache line is acquired while the DMA Write A operations are proceeding. After DMA Write A acquires full ownership of its cache line 221, DMA Write B also acquires full ownership 223 and issues DMA Write(B)_With_Data 213 whereby data B is written to memory prior to the completion of storage of data A to memory. When data A has completed being stored, an acknowledgment 207 is sent to the IOCC indicating completion of the DMA Write A transaction, and immediately following, an acknowledgment 208 is sent to the IOCC indicating completion of DMA Write B transaction. Thus the order of completion is preserved from the viewpoint of the IOCC and substantial time savings is recognized.
[0065] Another related embodiment extends the features of the invention to bus transactions with special architectural mechanism. For example, when applied to pre-fetching operations, the features of the invention allows looking ahead or speculating with I/O and looking ahead to writes in parallel. Thus, an assumption that the DMA Writes occurs sequentially in the instruction stream is made and a DClaim for a next cache line is issued in parallel to the previous DClaim before the address is actually received. In another embodiment, the features of the invention may be further utilized to determine what processes may be needed at a future time and keeping the processes in a separate directory or history of each set of transactions. This further allows extending the features of the invention to completion of write without data by priming the IOCC even before requesting information or data.
[0066] Although the invention has been described with reference to specific embodiments, this description should not be construed in a limiting sense. Various modifications of the disclosed embodiments, as well as alternative embodiments of the invention, will become apparent to persons skilled in the art upon reference to the description of the invention. It is therefore contemplated that such modifications can be made without departing from the spirit or scope of the present invention as defined in the appended claims.
Claims
What is claimed is:
1. A method for providing pipelined DMA Write operations in a data processing system having a processor, a system memory, a cache memory, an Input/Output (I/O) subsystem and an I/O Channel Controller (IOCC), said method comprising:
responsive to a snoop by said IOCC of a DMA Write_No_Data request for a cache line, setting a tag state of said cache line to indicate a weak DMA ownership;
monitoring when all cache lines requested by previously issued DMA Write_No_Data requests are in a DMA Exclusive (D2) state;
responsive to said monitoring, changing said tag state of said cache line to DMA Exclusive; and
writing data from said cache line to said system memory responsive to said cache line being tagged as DMA Exclusive.
2. The method of claim 1, wherein responsive to a snoop of an operation for said cache line,
releasing ownership of said cache line by said DMA Write operation when said cache line has weak DMA ownership state; and
forcing a retry of said operation when said cache line is DMA Exclusive until a completion of said write of said cache line to memory.
3. The method of claim 2, wherein responsive to said releasing of ownership of said cache line, said method comprises transitioning said cache line coherency state to Invalid (I).
4. The method of claim 3, further comprising transitioning said coherency state from D2 to I when said writing data to said system memory has been completed.
5. The method of claim 4, further comprising transitioning said coherency state from D2 to Exclusive when said DMA_Write_With_Data operation has completed and said cache line has been written to memory and said processor still requires access to said cache line.
6. A system for providing pipelined DMA Write operations in a data processing system having a processor, a system memory, a cache memory, an IOCC with an I/O cache, I/O directory and cache coherency protocol, comprising:
responsive to a snoop by said IOCC of a DMA Write_No_Data request for a cache line, means for setting a tag state of said cache line to indicate a weak DMA ownership;
means for monitoring when all cache lines of previously issued DMA Writes are in a DMA Exclusive state;
means, responsive to said cache lines being in said DMA Exclusive, for providing DMA Exclusive to said cache line; and
means for writing data from said cache line to said system memory responsive to said cache line being tagged as DMA Exclusive.
7. The system of claim 6, wherein responsive to a snoop of an operation for said cache line, said system provides
means for giving up ownership of said cache line when said cache line has said weak DMA ownership state; and
means for forcing a retry of said operation when said cache line is DMA Exclusive until a completion of said write of said cache line to memory.
8. The system of claim 7, wherein said means for giving up ownership of said cache line further includes means for transitioning said cache line coherency state to invalid.
9. The system of claim 8, further comprising means for transitioning said coherency state from D2 to I when said writing data to said system memory has been completed.
10. The system of claim 9, further comprising means for transitioning from D2 to Exclusive when said DMA_Write_With_Data operation has completed and said cache line has been written to memory and said processor still requires access to said cache line.
11. A data processing system having a processor, a memory interconnected to said processor via a system bus, at least one Input/Output (I/O) device, a cache memory, an I/O Channel Controller (IOCC) connected to said system bus and to said at least one I/O device, and further comprising:
a Direct Memory Access (DMA) Write response protocol that tags a cache line of said memory cache as DMA Exclusive, responsive to a receipt of a request for ownership of said cache line when all other cache lines associated with previously issued requests for ownership of said other cache lines are DMA Exclusive; and
means for issuing a corresponding write operation to write said cache line to said memory when said cache line is DMA Exclusive.
12. The processing system of claim 11, wherein said DMA Exclusive state further forces an operation snooped on said system bus for said cache line to be retried until said write operation of said cache line to said memory is completed.
13. The data processing system of claim 12, said response protocol further comprising a weak ownership state that is grouped with consecutive responses of said first operation that also provides a weak ownership state, wherein said weak ownership state of a latter one of said first operation automatically transitions to said DMA Exclusive state when all previous weak ownership states of former ones of said first operation have transitioned to a DMA Exclusive state.
14. The data processing system of claim 13, wherein said weak ownership state is released whenever an operation for said cache line is snooped on the system bus.
15. The data processing system of claim 14, further comprising:
a directory associated with said cache; and
wherein, said response protocol comprises a coherency protocol that includes an Exclusive (E) state, an Invalid (I) state, a DMA weak ownership (D1) state, and a DMA Exclusive (D2) state, wherein when a DMA Write is received for a cache line in said I state, said coherency protocol provides:
a first transition from I to D1 responsive to snoop of a DMA_Write_No_Data request for said cache line that results in transfer of ownership of said cache line to said operation;
a second transition from D1 to D2 responsive to an IOCC determination that all D1 states of previously issued DMA Write requests have transitioned to D2.
16. The data processing system of claim 15, wherein said coherency protocol provides a third transition from D2 to I when said Write operation has completed writing said cache line to memory.
17. The data processing system of claim 15, wherein said coherency protocol provides a third transition from D2 to E when said Write operation has completed writing said cache line to memory, and said cache line is being utilized by another operation.
18. A method for providing pipelined operations in a data processing system having a processor, a memory, and cache, said method comprising:
responsive to a snoop of a first-type operation arbitrating for ownership of a cache line, setting a coherency state of said cache line to indicate weak ownership of said cache line when said ownership is provided;
monitoring when all cache lines requested by previously issued first-type operations are in a cache state indicating exclusive ownership of said cache line by a requesting operation;
responsive to said monitoring, setting said coherency state of said cache line to indicate said exclusive ownership by a requesting operation; and
immediately forwarding said cache line to said requesting operation for completion of said operation once said cache line is tagged with said exclusive ownership state.
19. The method of claim 1, further comprising, responsive to a snoop of an operation for said cache line,
releasing ownership of said cache line by said requesting operation when said cache line has said weak ownership state; and
forcing a retry of said operation when said cache line has an exclusive ownership state until a completion of said write of said cache line to memory.
20. The method of claim 19, further comprising transitioning said coherency state from exclusive ownership state to an Invalid state when said requesting operation has completed.
21. The method of claim 19, further comprising transitioning said coherency state from said exclusive ownership state to an Exclusive state when said requesting operation has completed and said cache line is required by another processor operation.
Patent Citations (3)
Non-Patent Literature (2)
- IBM TDB-NN9211140,Asynchronous Pipeline for Queueing Synchronous DMA Cache, IBM Technical Bulletin, vol. 35, Issue No. 6, p. No. 140-141, Nov. 1, 1992.Applicant
- IBM TDB-NA9406241,DEMI Cache Management Policy for a Coherent DMA Cache on a Snooping Memory Bus, IBM Technical Bulletin, vol. 37, Issue No. 6A, p. No. 241-242, Jun. 1, 1994.Examiner