Computer System and Arbitrator Utilizing a Bus Bridge That Avoids Livelock

The present invention contains subject matter which may be related to patent applications Ser. Nos. 08/606,913 and 08/606,912, and docket numbers AT9-95-142 and AT9-96-007, filed Feb. 26, 1996.

1. Field of the Invention

The present invention relates to incompatibility between a host bus bridge and one or more bus bridges and, more particularly, but without limitation, to avoiding a livelock condition between a host bus bridge and a PCI-to-PCI bus bridge.

2. Background Information and Description of the Related Art

FIG. 1 illustrates a schematic diagram of a conventional architecture including CPU 100, which could be, for example, PowerPC or x86 based, a host bus bridge (e.g., memory controller) 102 connected to CPU 100 via 64 bit host bus 134, and system memory 102 (e.g., DRAM) connected to host bus bridge 102. Thirty two bit primary PCI bus 130 interconnects host bus bridge 102 to a variety of adapters (e.g., SCSIs 114 and 116), peripherals (not shown) and/or adapter slots 118 and 120.

Primary PCI bus 130 is limited on how many electrical loads it can drive. However, PCI bridges may be connected to primary PCI bus 130 so that additional loads can be added. Illustratively, PCI-to-PCI bridge (herein PCI bridge) 112 connects adapters 122, 124, and 126 via secondary PCI bus 132 to primary PCI bus 130, while PCI-to-ISA bridge (herein ISA bridge) 104 connects peripherals, such as floppy disk 110, to primary PCI bus 130.

However, because ISA direct memory access (DMA) cannot be suspended once initiated, ISA bridge 104 must guarantee that host bus bridge 102 will not suspend access to system memory 105 after ISA bridge 104 attempts a DMA. To do so, ISA bridge 104 asserts a PCI sideband signal FLUSHREQ* on connection 106 which, when asserted, requests that host bus bridge 102 flush any PCI bus transactions in progress. In response, host bus bridge 102 asserts a PCI sideband signal MEMACK* on connection 108 after it has flushed any PCI transactions in progress, and, will suspend any further requested PCI transactions while it asserts MEMACK*. In this manner, host bus bridge 102 guarantees ISA bridge 104 that it will not suspend access to system memory 105 once ISA bridge 104 begins a DMA.

Unfortunately, no such guarantee can be made from host bus bridge 102 to PCI bridge 112. The PCI-to-PCI Bridge Architecture Specification 1.0 and the PCI Local Bus Specification 2.0 delineate two requirements that can cause a livelock condition. First, PCI bridge 112 must flush its internal write buffer 134 before it can allow transfer of data to/from secondary PCI bus 132. Second, host bus bridge 102 must complete in process I/Os before allowing PCI bridge 112 to store memory write data into system memory 105. Hypothetically, a livelock condition may occur, for example, when SCSI adapter 122 generates a memory write transaction (e.g., MEMORY WRITE or MEMORY WRITE AND INVALIDATE) targeting system memory 105. In response, PCI bridge 112 completes the transaction with SCSI adapter 122 and its peripheral (not shown) and posts the memory write data in internal write buffer 134. However, in this hypothetical, before the posted memory write data can be flushed (i.e., written to system memory 105), CPU 100 initiates a PCI non-postable transaction that targets a device on secondary PCI bus 132 and, therefore, must cross PCI bridge 112.

Non-postable PCI transactions include, for example, MEMORY READ, MEMORY READ LINE, MEMORY READ MULTIPLE, I/O READ, I/O WRITE, CONFIGURATION READ, and CONFIGURATION WRITE. When this sequence occurs, the conditions triggering the livelock have been established. The PCI-to-PCI Bridge Architecture Specification 1.0 requires that PCI bridge 112 terminate the non-posted access from CPU 100 by signalling retry because the posted memory write data remains in its internal write buffer 134. This requirement preserves the x86 and PowerPC processor architecture ordering rules necessary for correct operation of device drivers and hardware. Therefore, the posted memory write data in internal write buffer 134 must be flushed before PCI bridge 112 will allow the non-posted transaction to proceed. However, host bus bridge 102 will not allow access to system memory 105 by PCI bridge 112 until the PCI transaction initiated by CPU 100 completes. Host bus bridge 102 will simply repeat the retry transaction continuously. In response, PCI bridge 112 will terminate the transaction by signalling retry because it has been unable to flush its internal write buffer 134. This will repeat continuously, resulting in a PCI bus livelock condition.

Accordingly, there would be great demand for a technique that allows host bus bridge 102 and PCI bridge 112 to conform to the requirements delineated in the PCI-to-PCI Bridge Architecture Specification 1.0 and the PCI Local Bus Specification 2.0, while avoiding a livelock condition. This technique should delay secondary PCI bus 132 device write accesses to system memory 105 until a guarantee can be made that host bus bridge 102 has flushed any PCI transactions in progress and will postpone any further PCI transactions until all memory write data has been stored in system memory 105.

In accordance with the present invention, a computer system and arbitrator prevent a livelock condition from occurring between a host bus bridge and a PCI bridge, where the host bus bridge and PCI bridge conform to the specification delineated in the PCI-to-PCI Bridge Architecture Specification 1.0 and PCI Local Bus Specification 2.0. Specifically, the system prevents a livelock condition from occurring between a host bus bridge that must complete all in process I/O requests before allowing access to system memory and a PCI bridge that must flush its internal write buffer before allowing transfer of data to/from a secondary PCI bus. The system includes an arbitrator for masking from the PCI bridge a request (REQ.sub.--) generated by a device on a second bus. The arbitrator requests that the host bus bridge flush all existing I/O requests (FLUSHREQ.sub.--) and postpone any future I/O requests from a central processing unit. The third step includes, in response to a notification from the host bus bridge that all I/O requests have been flushed and that any future I/O requests from the central processing unit will be postponed (MEMACK.sub.--), the arbitrator unmasks the request to the PCI bridge (GREQ.sub.--). In response to unmasking the request to the PCI bridge, the PCI bridge grants control of the second bus to the device (GNT.sub.--). This system guarantees that any data posted in an internal write buffer of the bus bridge can be delivered to system memory.

FIG. 1 illustrates a schematic diagram of a conventional architecture subject to a livelock condition.

FIG. 2 illustrates a schematic diagram of an architecture having a unique bus negotiator for preventing a livelock condition in accordance with the present invention.

FIG. 3 illustrates a timing diagram of a hypothetical transaction for the architecture of FIG. 2.

FIG. 4 illustrates the arbitrator in accordance with the present invention.

FIG. 2 illustrates a schematic diagram of architecture 200 having a unique bus negotiator in accordance with the preferred embodiment. Specifically, architecture 200 includes bus negotiator 240, having PCI-to-PCI bridge (herein PCI bridge) 206 and arbitrator 204. In the preferred embodiment, PCI bridge 206 is implemented within arbitrator 204 (collectively referred to as bus negotiator 240), although arbitrator 204 could be implemented separately from PCI bridge 206.

Architecture 200 further includes a number of interconnected components, such as any suitable CPU 234 (in the preferred embodiment, a PowerPC 604), any suitable host bus bridge 232 (e.g., memory controller) connected to host bus 226, any suitable bus master device (e.g., SCSI adapter 212), and any suitable system memory 202. As previously described, PCI bridge 206 and host bus bridge 232 conform to the specifications delineated in the PCI-to-PCI Bridge Architecture Specification 1.0 and PCI Local Bus Specification 2.0.

Primary PCI bus 228 connects a variety of peripherals, such as PCI-to-ISA bridge 208, SCSI adapter 260, and communications adapter 262, to host bus bridge 232 and PCI bridge 206. However, unlike the conventional architecture described in FIG. 1, ISA bridge 208 asserts a request signal (i.e., ISA REQ.sub.-- 258, ".sub.-- " indicating active LOW) to arbitrator 204, and does not assert a FLUSHREQ.sub.-- sideband signal to host bus bridge 232 (described in more detail herein). Secondary PCI bus 230 connects a variety of peripherals and adapters, such as SCSI adapters 212 and 214 and disks 236, 238, and 239, to PCI bridge 206 and arbitrator 204. Alternatively, other arrangements of peripherals could be made and/or additional stages of bridges added without departing from the scope and spirit of the preferred embodiment.

FIG. 3 illustrates a timing diagram for use with architecture 200 shown in FIG. 2. Referring to FIGS. 2 and 3, at clock 0, primary and secondary PCI buses 228 and 230 are initially idle, waiting for a primary, secondary, or another stage (not shown) PCI bus device to assert its REQ.sub.-- signal. Next, for sake of illustration, at clock 1, a PCI device on secondary PCI bus 230 (e.g., SCSI adapter 212) asserts REQ.sub.-- 322 until it is granted secondary PCI bus access from PCI bridge 206 to initiate a WRITE, MEMORY WRITE or MEMORY WRITE AND INVALIDATE transaction. Arbitrator 204 temporarily masks off the asserted REQ.sub.-- from PCI bridge 206. Alternatively, arbitrator 204 could have received an asserted ISA REQ.sub.-- 258 from ISA bridge 208.

In response to a received REQ.sub.-- 322 or 258, at clock 1, arbitrator 204 asserts FLUSHREQ.sub.-- Q 224 to host bus bridge 232. Arbitrator 204 continues to assert FLUSHREQ.sub.-- 224 until it detects that data from SCSI adapter 212 has been successfully delivered to system memory 202 (described in more detail herein). When host bus bridge 232 has flushed all PCI transactions, it asserts MEMACK.sub.-- 222, which, in this example, occurs at clock 2. Host bus bridge 232 continues to assert MEMACK.sub.-- 222 while arbitrator 204 asserts FLUSHREQ.sub.-- 224, thereby postponing any further host bus bridge 232 initiated PCI transactions. Similarly, if ISA bridge 208 asserted REQ.sub.-- 258, it requests and receives control of primary PCI bus 228 to perform DMA access to system memory 202.

In response to host bus bridge 232 asserting MEMACK.sub.13 222, at clock 2, arbitrator 204 forwards the asserted SCSI adapter 212 REQ.sub.-- to PCI bridge 206 on G.sub.-- REQ.sub.-- 220, thereby allowing PCI bridge 206 to detect the request. In response, PCI bridge 206 asserts GNT.sub.-- to the requesting device (e.g., SCSI adapter 212) at clock 3, thereby relinquishing control of secondary PCI bus 230 to the requesting device (e.g., SCSI adapter 212). In response, the requesting device will initiate a secondary PCI bus transaction by asserting PCI signal FRAME* at clock 4. If the initiated PCI transaction is a postable PCI transaction targeting system memory 202 (e.g., MEMORY WRITE or MEMORY WRITE AND INVALIDATE), the requesting device writes data into internal write buffer 252 of PCI bridge 206. Otherwise, arbitrator 204 de-asserts FLUSHREQ.sub.-- and control returns to the idle state of clock 1.

In response to the requesting device de-asserting FRAME* at clock 8, indicating it has finished transmitting data to internal write buffer 252, arbitrator 204 asserts DISPST* 218 for two clock cycles at clock 8 to determine the status of internal write buffer 252. PCI bridge 206 responds within one clock cycle (i.e., clock 9) by asserting BUFNE* 216, indicating that the posted internal write buffer 252 is not empty. Otherwise, PCI bridge 206 fails to assert BUFNE* 216, indicating that the posted internal write buffer 252 is empty. In BUFNE* 216 is asserted, at clock A, arbitrator 204 de-asserts DISPST* 218 and waits for PCI bridge 206 to de-assert BUFNE* 216, indicating that posted internal write buffer 252 has been flushed to system memory 202. In this example, PCI bridge 206 de-asserts BUNFE* 216 at clock B and, in response, arbitrator 204 de-asserts FLUSHREQ.sub.-- Q 224 at clock B. In response to FLUSHREQ.sub.-- Q 224 being de-asserted, host bus bridge 232 de-asserts MEMACK.sub.-- 222 at clock C.

Accordingly, arbitrator 204 allows host bus bridge 232 and PCI bridge 206 to conform to the requirements delineated in the PCI-to-PCI Bridge Architecture Specification 1.0 and the PCI Level Bus Specification 2.0, while avoiding a livelock condition. To do so, arbitrator 204 delays secondary PCI bus 230 device write accesses to system memory 202 until a guarantee is made from host bus bridge 232 that it has flushed any PCI transactions in progress and will postpone any further PCI transactions until all memory write data has been stored in system memory 202.

FIG. 4 illustrates a detailed view of arbitrator 204. Arbitrator 204 includes two 24 pin logic chips, namely first circuit 400 and second circuit 450. However, one skilled in the art readily recognizes that additional chips or other hardware configurations could be utilized to perform the described functions. Pin designations and descriptions are provided below. An ".sub.-- " after any pin name indicates active LOW:

For ease in explanation, detailed logic for first circuit 400 is described below. A "*" represents a logical AND, a "+" represents a logical OR, a "/" before a name represents a logical NOT, and a ":" before "=" represents "latched" (i.e., state can only change on rising edge of clock). Again, a ".sub.-- " after a name represents active LOW.

1. Latched FRAME--Indicates FRAME is latched on PCI clock edge transitioning from HIGH to LOW. ##EQU1## 2. As indicated by the circuit below, first circuit 400 asserts NEXT.sub.-- STATE.sub.-- on the first clock of FRAME (i.e., LSFRAME.sub.-- not asserted) to inform second circuit 450 that a secondary PCI bus cycle utilizing memory 202 (i.e., DMA R/W, indicated by asserted SFRAME, SAD31 and SCBE2) has started and that the STATE (i.e., determined by three state machine bits STATE0, STATE1, and STATE2, described in more detail herein) should now be advanced to the next state because there is at least one REQ.sub.-- waiting, which will be indicated by SERIALREQ.sub.-- being asserted. NEXT.sub.-- STATE.sub.-- de-asserts once it detects that SERIALREQ.sub.-- has de-asserted (i.e., no more REQs.sub.-- or MULTIPLE.sub.-- REQs.sub.-- asserted), or that PCI bus 230 is idle (i.e., SFRAME and SIRDY de-assert). ##EQU2## 3. First circuit 400 asserts DMAW.sub.-- on the first clock edge of SFRAME if it detects a direct memory access (DMA) write operation (i.e., SCBE0, SAD31, and SCBE2 all asserted). DMAW.sub.-- remains asserted as long as a DMA write operation is asserted, or until secondary PCI bus 230 is idle. ##EQU3## 4. First circuit 400 asserts DMAR.sub.-- on the first clock edge of SFRAMED.sub.-- if a DMA read operation is detected. DMAR.sub.-- remains asserted until the first data transfer is detected, or PCI bus is idle. ##EQU4## 5. LDISPST.sub.-- asserts on the first data transfer of a DMA write. When it asserts, PCI bridge 206 will respond by asserting BUFNE.sub.-- if it has posted data in its internal write buffer 252. Once the assertion of BUFNE.sub.-- is detected, LDISPST.sub.-- de-asserts. ##EQU5## 6. First circuit 400 transmits LOCKOUT.sub.-- to second circuit 450 to prevent the detection of a re-asserted SERIALREQ.sub.-- for the duration of a PCI bus cycle and until delivery is made of any posted write data to memory. That is, once SERIALREQ.sub.-- de-asserts, LOCKOUT.sub.-- asserts and will remain asserted until the PCI bus cycle completes and any data in internal write buffer 252 flushes. Therefore, if REQ2.sub.-- is asserted while REQ3.sub.-- is being processed, the assertion of SERIALREQ will not effect CPU.sub.-- FLUSHREQ.sub.-- until LOCKOUT.sub.-- is de-asserted. Also, REQ2.sub.-- will not be gated to PCI bridge 206 as GREG2.sub.-- until LOCKOUT.sub.-- de-asserts (described herein). ##EQU6## 7. LSERIALREQ.sub.-- is a latched version of SERIALREQ.sub.--. ##EQU7## 8. Circuit 400 asserts CPU.sub.-- FLUSHREQ.sub.-- to alert host bus bridge 232 that a valid DMA request to read or write data in memory is desirous, and that host bus bridge 232 must flush all PCI buffers (not shown) and hold off PCI cycles. CPU.sub.-- FLUSHREQ.sub.-- asserts when SERIALREQ.sub.-- is or was recently asserted, and LOCKOUT.sub.-- is not asserted. CPU.sub.-- FLUSHREQ.sub.-- remains asserted during the assertion of SFRAME for memory R/W, LDISPST, BUFNE, or ISA.sub.-- FLUSHREQ.sub.--. Host bridge 232 responds with MEMACK.sub.--. ##EQU8## 9. Latched CPU.sub.-- FLUSHREQ.sub.--. ##EQU9## 10. Circuit 400 asserts gated MEMACK.sub.-- to the GREQ.sub.-- circuit in circuit 450 (described herein). G.sub.-- MEMACK will assert if FLUSHREQ.sub.-- was asserted last clock cycle (i.e., /LCPU.sub.-- FLUSHREQ.sub.--) and FLUSHREQ.sub.-- continues to be asserted (i.e., /CPU.sub.-- FLUSHREQ.sub.--). Once asserted, G.sub.-- MEMACK remains asserted until CPU.sub.-- FLUSHREQ de-asserts. ##EQU10## Detailed logic for second circuit 450 is described below. 1. Second circuit 450 asserts MULTIPLE.sub.-- REQS.sub.-- when more than one REQ is pending. ##EQU11## 2. Second circuit 450 asserts SERIALREQ if the asserted REQn.sub.-- corresponds to the current state, e.g., REQ2 is asserted and STATE=010 (i.e., high order bit STATE2 is not asserted, middle order bit STATE1 is asserted, and low order bit STATE0 is not asserted). SERIAL REQ.sub.-- is de-asserted if both MULTIPLE.sub.-- REQS.sub.-- and NEXT.sub.-- STATE.sub.-- are asserted simultaneously (which forces the STATE to change to the next state), there are no more REQs, or the asserted REQ de-asserts. ##EQU12## 3. Second circuit generates STATE MACHINE terms to define the state transitions. The state transitions occur if SERIALREQ.sub.-- is not asserted, and another REQn is pending or more than one REQn are pending. States 0, 1, 2, 3, & 4 directly correspond to the input reqs REQ0.sub.--, REQ1.sub.--, REQ2.sub.--, REQ3.sub.--, & REQ4.sub.--. ##EQU13## 4. GREQn.sub.-- is used to present only one REQn.sub.-- at a time to PCI bridge 206. REQn.sub.-- will only be forwarded to GREQn.sub.-- if: REQn.sub.-- is asserted, STATEn.sub.-- is in the corresponding state (e.g. REQ1.sub.-- & STATE=001), G.sub.-- MEMACK.sub.-- is asserted indicating that circuit 400 is ready, LOCKOUT.sub.-- is de-asserted indicating circuit 400 is not busy, and MULTIPLE.sub.-- REQS.sub.-- and NEXT.sub.-- STATE.sub.-- are not both asserted (indicating that SERIALREQ.sub.-- has not been forced inactive so that the STATE can advance to next state). ##EQU14##

While the invention has been shown and described with reference to particular embodiments thereof, it will be understood by those skilled in the art that the foregoing and other changes in form and detail may be made therein without departing from the spirit and scope of the invention.