Packet Switch Network Protocol

1. Field of the Invention

This invention relates to multiprocessor systems and more particularly to a data link protocol for transmitting data packets between nodes in a multiprocessor system.

2. Background Information

Closely-coupled multiprocessor systems require high performance from the underlying communication network. It is not unusual to expect a constant level of performance from the communication network as the load generated by a plurality of attached processors within the system peaks. Traditionally, this demand has been met with backplane networks composed of buses of parallel wires controlled by high speed hardware units. While this interconnection methodology facilitates communication of parallel data at very high speeds, it is inherently unreliable because of the large number of wires and connectors involved in such a network. Moreover, the buses introduce crosstalk noise, signal skew and signal level shifting into the system which degrade its integrity and reliability. Additionally, the bus connectors introduce substantial reliability problems as a result of poor mechanical contacts between the bus and the connection target. These problems are increased by the repeated insertion and removal of connectors and movement of the system.

Alternately, the use of long haul networks have proven to be highly reliable. In long haul networks, a data link, such as a wire or optical fiber, which interconnects the communicating systems, facilitates the serial transmission of data packets from one system to another system. Long haul networks generally are characterized by fewer wires or fibers which actually connect the communicating systems, the feasibility of shielding wires to minimize crosstalk noise, reliable serial connectors and the use of sophisticated protocols which facilitate error detection and recovery on serial data links. Conceptually, backplane networks modeled after long haul networks would be an attractive solution to the reliability problem. Unfortunately, the model fails to provide the performance required by closely-coupled systems. Similarly, local area networks, which are capable of comparatively high speed data transmission over short distances, fail to provide the required performance. Local area networks and long haul networks perform the major portion of data transmission or network function with software which is resident in each of a plurality of communicating processors or host. This transmission methodology while adding flexibility to the system severely degrades the effective data transmission rate.

Therefore it is an object of the invention to provide an improved method of transmitting data packets within a multiprocessor system.

It is another object of the invention to provide a method of transmitting a data packet from a sending node to a receiving node only when the receiving node is available.

It is still another object of the invention to detect and correct errors during the transmission of data packets.

It is a further object of the invention to provide a method of transmitting a data packet from a first node through a second node to a third node without storing the data packet in the second node.

In accordance with these objects, features and advantages of the invention are accomplished by an asynchronous data link protocol described herein. A first of a plurality of nodes in a multiprocessor system transmits a data packet to a second of the plurality of nodes over a data channel when the first node detects a predetermined sequence of bits on a control channel connected therebetween. The data packet is preceded on the data channel by a header which includes a plurality of fields informing the second node of the beginning of the data packet, the length of the data packet and the sequence number assigned to the packet. The data packet is followed by a field which enables the second node to determine whether the data packet was received correctly. If the second node determines that the data packet was received correctly, the data packet is stored in the node and an acknowledgment sequence is transmitted to the first node over the control channel. However, if an error occurs during the transmission of the data packet, the protocol enters an error state by removing all signal information from the data and control channels. Thereafter, a recovery processor, which is associated with each of the plurality of nodes, analyzes the state of the nodes at the time of the error in order to determine which corrective action needed to recover the respective node.

The data link protocol also facilitates the transmission of a data packet from a first node through intermediate nodes to a final node without storing the data packet in the intermediate node. This process is referred to as virtual cut-through. The first node transmits the data packet to the intermediate node when it detects the predetermined sequence of bits on the control channel connected therebetween. If the intermediate node is able to immediately transmit he data packet to its destination node, it does so over a data channel connected between the second node and the final node without storing the packet in the second node. If an error is detected by the second node, the protocol enters the error state and informs the final node to discard the data packet. Recovery processors associated with the various nodes attempt to recovery the respective nodes by retransmitting the data packet by the first node until the packet is received correctly. If an error is detected by the final node, the recovery processors associated with the intermediate and the final nodes attempt recovery by causing a retransmission of the data packet by the intermediate node until the packet is received correctly.

This invention will be described in greater detail by referring to the accompanying drawings and the description of the best mode that follows.

FIG. 1 shows a block diagram of two of a plurality of nodes included in a multiprocessor system.

FIG. 2 is a detail schematic of an input port of a node of FIG. 1.

FIG. 3 is a detail schematic of an output port of a node of FIG. 1.

FIG. 4 shows a format associated with a data packet which is transmitted within the multiprocessor system.

FIG. 5 shows a format of a sequence of fields which are transmitted with each data packet transmitted within the multiprocessor system.

FIGS. 6, 7 and 8 show a state diagram of a data link protocol for transmitting data packets within the multiprocessor system.

FIGS. 9a, 9b, 10a and 10b show flow diagrams of the data link protocol for transmitting data packets within the multiprocessor system.

Referring to FIG. 1, there is shown two nodes A and B of a plurality of nodes which are a part of a multiprocessor system. Node A includes a plurality of input ports 12 (one shown) which are selectively connected through a switch 14 to a plurality of output ports 16 (one shown) by a router circuit 18. One of the plurality of output ports 16 is coupled to one of a plurality of input ports 20 of node B by a data channel and a control channel. The data channel facilitates the transmission of data packets between various nodes in the system while the control channel is primarily used to acknowledge the reception of the data packets. Signals appearing on the data and the control channels are in the form of serial encoded binary data and clock information. A local processor 22 is coupled to an input and an output of the switch 14 in a parallel manner which enables the local processor to feed data packets to and receive data packets from the multiprocessor system, respectively. Node A also includes a recovery processor 24 having read and write access to all state information in the input port 12, the output port 16 and the router circuit 18. The recovery processor 24 is activated by one of the plurality of input ports 12 or one of the plurality of output ports 16 to perform error recovery and is not otherwise involved in error-free transmission of data packets. Moreover, node A is coupled to upstream and downstream nodes by a plurality of data channels and control channels which are associated with the respective input ports 12 and output ports 16.

Node B, which is similar to node A, includes the plurality of input ports 20 (one shown), a plurality of output ports 26 (one shown), a switch 28, a router circuit 30, a local processor 32 and a recovery processor 34. Moreover, node B is coupled to upstream and downstream nodes by a plurality of data channels and control channels which are associated with the respective input ports 20 and output ports 26.

Referring to FIG. 2, there is shown a detailed block diagram of one of the plurality of input ports 12 and 20. The input ports 12 and 20 associated with nodes A and B, respectively, are identical circuits. Therefore, only input port 12 will be described in detail. A data channel 33 and a control channel 35 are coupled to a receiver 36. The receiver 36 accepts serial data from the data channel 33, converts the serial data to data in a parallel format and forwards the converted data to a first-in-first-out (FIFO) buffer 38. The receiver 36 also performs analysis on the incoming data packets and sends control information to (1) an output port associated with an upstream node over the control channel 35 and (2) a FIFO controller 40. The FIFO controller 40 and the FIFO buffer 38 facilitate the storage of incoming data packets and the forwarding of the packets through switch 14 to one of the plurality of output ports 16 selected by the router circuit 18. The selected one of the plurality of output ports 16 is coupled to one of the plurality of input ports 12 as result of a destination address which a part of a header which accompany every data packet.

Referring to FIG. 3, there is shown a block diagram of one of the plurality of output ports 16 and 26. The output ports 16 and 26 associated with nodes A and B, respectively, are identical circuits. Therefore, only the output port 16 will be described in detail. A transmitter 42 is included in each one of the plurality of output ports 16. The transmitter 42 receives data in a parallel format from one of the plurality of input ports 12 coupled thereto through the switch 14. Transmitter 42 then converts the data to a serial format and forwards the converted data to a downstream node over a data channel 44. Transmitter 42 also receives acknowledgment information in a serial form on control channel 46 from the downstream node. The transmitter 42 and the FIFO controller 40 follow a rigorous handshaking procedure during the transmission of a data packet to insure the proper packet transmission. This handshaking procedure is a part of a data link protocol which controls the transmission of data packets within the multiprocessor system.

Referring to FIG. 4, there is shown a format associated with each data packet transmitted within the switching system. Each data packet is preceded by a header which includes the following fields: (1) a destination address which the router circuits 18 and 30 use in coupling the proper output ports 14 and 26 to input ports 12 and 20, respectively, sending the data packet; (2) the length of the data packet in bytes; (3) a variable header which allows for various applications to be performed by the switching system; and (4) the body of the packet which is the data portion of the packet.

Referring to FIG. 5, there is shown a format of a data packet which will be transmitted from a first node to a second or subsequent node. This format includes, in addition to all of the fields shown in FIG. 4, three additional fields to support a sliding-window protocol and error detection schemes. The additional fields are: (1) start of packet which is a sequence of bits indicating the start of a data packet; (2) sequence number is a plurality of bits which redundantly code a logical zero or a logical one in order to prevent duplicate packets from entering the multiprocessor system; and (3) a checksum field which facilitates the detection of errors in the transmitted data packet.

Referring to FIG. 6, there is shown a state diagram of the data link protocol used in transmitting data packets within the multiprocessor system. Referring also to FIGS. 1, 2 and 3, for purposed of illustration, let us assume the FIFO buffer associated with node A has a data packet that must be forwarded to the FIFO buffer associated with node B. The goal of the data link protocol is to successfully transfer the data packet. Moreover, in order to simplify the understanding of the state diagram shown in FIG. 6, the following is a list of symbols and acronyms used throughout the analysis of the protocol.

The states in the state diagram are labeled by Sn where n is an integer and the transitions between states are labeled by integers only. Moreover, the sequence number variable S is used in the state diagram to avoid the need to enumerate conditions where S represents a logical zero and where S represents a logical one. Additionally, there are, as a result, cycles in the state diagram where the logical value of S must toggle. The transitions where S must be toggled are labeled with the letter T.

Referring to FIGS. 1, 2, 3 and 6, the packet switching system begins in state S1 which is a ready/waiting state. T.sub.A is transmitting I on D, receiving I.sub.G and C and is prepared to transmit a data packet from FIFO.sub.A as data packet S. T.sub.A indicates the intended transfer by asserting an Available signal to the router circuit 18. In a similar manner, R.sub.B is receiving I on D, transmitting I.sub.G on C and is ready to accept packet S from T.sub.A. Transmission of packet S begins with transition 1 from state S1 to state S2. The router circuit 18 connects FIFO.sub.A through switch 14 to T.sub.A and asserts a Connected signal to both the FIFOC.sub.A and T.sub.A indicating that a connection exists therebetween. T.sub.A then de-asserts the Available signal which informs the router circuit 18 that OP.sub.A is busy with packet transmission. T.sub.A also places a SOP sequence and the sequence number S on D and asserts a Send signal on the Transmitter status lines. The Send signal instructs the FIFOC.sub.A parallel format, through switch 14. T.sub.A then converts the bytes of the data packet to a serial format, encodes the data packet and transmits the data packet on D. During the forwarding of the data packet by FIFO.sub.A, T.sub.A continuously monitors whether an Underflow signal is asserted by FIFOC.sub.A. The assertion of the Underflow signal indicates that an error in the transmission of the data packet has occurred. When the Underflow signal is asserted by FIFOC.sub.A, the protocol takes transition 41 from state S2 to state S6. T.sub.A then stops the clock on D and takes transition 9 to an error state S7. R.sub.B responds by removing the clock from C.

If the Underflow signal was not asserted, upon receiving SOP on D, R.sub.B decodes the first incoming byte of data, converts it to a parallel format and forwards the converted data to FIFOC.sub.B by asserting the StartOfPacket and Read strobe signals. Subsequent packet bytes are similarly decoded, converted to parallel format and strobed to FIFOC.sub.B. R.sub.B also extracts the sequence number S and the packet length L from the packet header. As the data packet is received, R.sub.B counts down from L, checksums the data packet and verifies that the sequence number S was the one expected. When L equals zero, the last byte of the data packet has been received from T.sub.A and transition 2 is taken. R.sub.B determines whether the checksum for the data packet is correct, strobes the Read line and forwards the last byte of data to FIFOC.sub.B. FIFOC.sub.B has then received the entire data packet without error.

Simultaneously, when FIFOC.sub.A reaches the end of the data packet, it asserts the EndOfPacket signal. T.sub.A, in response to the EndOfPacket signal, transmits the last byte of data, starts a WaitingForAcknowledgement timer, and asserts a Wait signal on the Transmitter status lines. This places FIFOC.sub.A in a wait or suspended state. In transition 3 to state S3, R.sub.B encodes, serializes and transmits acknowledgement A.sub.S to T.sub.A. R.sub.B also toggles the value held in FIFOC.sub.B which represents the sequence number of the next-expected data packet.

Transition 4 is taken from state S3 when FIFO.sub.B has room to accommodate a maximum-length data packet and R.sub.B has placed I.sub.G on C. This restriction exists because once data transmission from IP.sub.A through switch 14 to OP.sub.A to the adjacent node B has commenced, it can not be stopped and then restarted. Thus, the data link protocol requires the FIFOC.sub.B to refuse to accept a new data packet, which potentially is of maximum permissible length, until sufficient room exists in FIFO.sub.B. This refusal is represented by an idle-wait (I.sub.W) pattern on the control channel. Additionally, transition 4 represents that T.sub.A has (1) received the acknowledgement before the WaitingForAcknowledgement timer expired, (2) verified that the acknowledgement received was for the data packet sent (positive acknowledgement) and then (3) asserted a Release signal on the Transmitter status lines. The asserting of the Release signal facilitates the relinquishing of the FIFOC.sub.A to do other work. Additionally, T.sub.A, in response to the appearance of I.sub.G on C, then toggles the sequence number stored in FIFOC.sub.A in preparation for sending the next data packet, S, and asserts the Available signal which informs the router circuit 18 that OP.sub.A is now available to transmit another data packet.

If, as noted above FIFOC.sub.B refuses to accept a new data packet, FIFOC.sub.B asserts a FIFOFull signal, then transition 5 is taken from state S3 to state S4. The FIFOFull signal indicates that FIFO.sub.B has insufficient room therein to accommodate a maximum-length data packet. Subsequently, R.sub.B places I.sub.W on C. T.sub.A, having first seen the correct acknowledgement, A.sub.S, followed by I.sub.W, asserts Release on the Transmitter status lines. This assertion by T.sub.A frees FIFO.sub.A and starts WaitingForI.sub.G timer. Additionally, T.sub.A leaves the Available signal unasserted because it is not ready to send another data packet. Transition 6 is taken from state S4 to state S1 when FIFO.sub.B has room for the maximum-length data packet and FIFOC.sub.B de-asserts the FIFOFull signal. In response to the absence of the FIFOFull signal, R.sub.B places I.sub.G on C and toggles the sequence number stored in FIFOC.sub.B. If T.sub.A detects I.sub.G on C before the WaitingForI.sub.G timer expires, it will toggle the sequence number stored in FIFOC.sub.A, assert the Available signal which informs the router circuit 18 that it is free to send another data packet.

As noted above, when R.sub.B receives the last byte of the data packet S, it checksums the packet in order to determine whether an error has occurred during the transmission thereof. If the checksum indicates that data packet S contained an error, transition 11 is taken from state S1 to state S8 wherein R.sub.B sends a negative acknowledgement, A.sub.S, to T.sub.A on C. Thus, R.sub.B is indicating that data packet S was the last packet it received correctly and which T.sub.A interprets to mean data packet S, the data packet just transmitted, must not have been received correctly by R.sub.B. R.sub.B then sends a Flush signal to FIFOC.sub.B which results in FIFOC.sub.B discarding any remnant of the data packet from FIFO.sub.B. Additionally, if FIFOC.sub.B has already started forwarding the data packet to an output port in node B, it will assert an Underflow signal which enables the output port to terminate the transmission by forcing the data channel it drives into a N state. Transition 12 from state S8 to an error state S7 implies (1) T.sub.A, upon receipt of a negative acknowledgement, always stops the clock on D irrespective of the state of C and (2) R.sub.B, upon transmitting a negative acknowledgement, always stops the clock on C irrespective of the state of D.

Thus, in transition 12, R.sub.B stops the clock on C, signals RP.sub.B that an error has occurred and suspends operation. Similarly, T.sub.A stops the clock on D, informs RP.sub.A of the error, holds the Transmitter status lines in the Wait position and suspends operation. It should be noted that whenever the Transmitter status lines are in a Wait state, FIFOC.sub.A is held in a dormant state, with the data packet T.sub.A just attempted to transmit held in tact in FIFO.sub.A. The retention of the data packet by the FO.sub.A allows RP.sub.A to request FIFO.sub.A to resend the data packet until it is transmitted correctly.

As stated above, during transition 2, T.sub.A started the WaitingForAcknowledgement timer. If because of some serious error at IP.sub.B, the timer expires, transition 13 is taken from state S1 to state S6 wherein T.sub.A stops the clock on D, signals RP.sub.A that an error has occurred and suspends operation. T.sub.A then holds its Transmitter status lines in the Wait state thus suspending operation of FIFOC.sub.A. R.sub.B, if it is able to do so, will then transit to the error state S7 as a result of transition 9.

R.sub.B encodes and transmits to T.sub.A encoded acknowledgement information, I.sub.G and I.sub.W over C. As a result of the coding of I.sub.G and I.sub.W, a control channel reception error at OP.sub.A may cause T.sub.A to believe I.sub.W is appearing on C when I.sub.G is presently on C. If this condition is left unattended, this would lead to channel deadlock with T.sub.A waiting forever for I.sub.G which R.sub.B believes it is transmitting. To avoid this potential deadlock, the data link protocol requires T.sub.A to start the WaitingForI.sub.G timer whenever I.sub.W is detected. Thus, if the WaitingForI.sub.G timer expires prior to the detection of I.sub.G on C by T.sub.A, transition 46 will be taken from state S4 to state S6 wherein T.sub.A stops the clock on D, informs RP.sub.A of the error, holds the Transmitter status lines in the Wait state and suspends operation. R.sub.B will then take transition 9 to the error state S7.

In state S2, T.sub.A is transmitting data packet S to R.sub.B. Before T.sub.A completes the transmission of the last byte of data of the packet S, an acknowledgement, A.sub.X, is received on C which is indicated by transition 7 to state S5. The presence of A.sub.X on C is a result of (1) the length field L erroneously indicated a packet length which is shorter than the actual length of S, hence R.sub.B responded with A.sub.X before the actual end of the packet S has been encountered or (2) the control channel acknowledgement. In either case, the protocol requires T.sub.A to take transition 8 from state S5 to state S6 whenever an acknowledgement is received prior to the completion of transmission of the data packet. T.sub.A stops the clock on D and takes transition 9 to the error state S7 It should be noted that although the transmission of data on D has stopped in state S6, T.sub.A must continue to accept data from FIFO.sub.A until the EndOfPacket signal is asserted. In essence, once a transmitter has asserted the Send signal, the data link protocol does not allow for an abort process between FIFO.sub.A and T.sub.A across the switch.

R.sub.B, which detects the loss of clock information on D, takes transition 9 from state S6 to the error state S7. In transition 9, assuming the packet checksum has failed, R.sub.B signals Flush to FIFOC.sub.B, stops the clock on C, informs RP.sub.B of the error and suspends operation. FIFOC.sub.B, in response to the Flush signal, facilitates the discarding of the data packet from FIFO.sub.B. Additionally, if FIFOC.sub.B has already started forwarding the data packet to an output port in node B, it will assert the Underflow signal which enables the output port to terminate the transmission of the packet by forcing the data channel into the N state.

Prior to the transmission of a data packet, T.sub.A transmits to R.sub.B the SOP, the sequence number and a header. During the transmission of the sequence number, the data channel D corrupts the packet sequence number in such a way that it appears to be its logical complement, S, transition 14 from state S1 to state S9. In state S9, R.sub.B determines that it has already received data packet S, discards the data packet as it is received without forwarding the packet to FIFOC.sub.B. R.sub.B then extracts the length field, L, from the packet only for the purposes of recognizing the end of the packet, transition 15, at which time it sends A.sub.S on C, transition 11 indicating that packet S was received correctly. T.sub.A interprets A.sub.S as a negative acknowledgment and takes transition 12 to the error state as described above.

Additionally, while in state S9, a mid-transmission acknowledgement may be received on C, transition 21 to state S10. Thereafter, T.sub.A takes transition 22 from state S10 to S6 which is similar to transition 8 from state S5 to state S6. R.sub.B then takes the transition 9 from state S6 to the error state S7.

In analyzing the state diagram shown in FIG. 6, the sequence of transitions 14, 15, 16, 17, 18, 19 and 20 may result in a loss of one or two packets. The likelihood of this particular sequence occurring is small in view of the fact that following two specific errors must occur sequentially: (1) the packet sequence number must be logically complemented in transition 14 and (2) acknowledgement A.sub.S must be logically complemented to A.sub.S in transition 16. Briefly, in state S10', T.sub.A incorrectly believes the packet S has been properly received by R.sub.B and informs the FIFOC.sub.A to release the packet. Thus, the packet S is now lost. Toggling the logical meaning of S and S' as transition 17 is taken, T.sub.A then transmits the next packet in transitions 18 and 19. R.sub.B, however, discards the packet as it is received because sequence number of the packet is the logical complement of the sequence number expected. R.sub.B subsequently sends A.sub.S to T.sub.A in transaction 20 which falsely indicates that it has received the packet correctly. T.sub.A interprets A.sub.S to mean the packet just transmitted was received correctly. Hence, two packets are lost. As noted above, the probability of two sequential errors occurring is extremely small.

It is possible for a FIFO buffer to run out of data to present to an output port while in the process of transmitting a packet as a result of a failed checksum or a mid-transmission acknowledgement as noted above. Assume that FIFOC.sub.A runs out of data to present to OP.sub.A because R.sub.A has detected a transmission error and has asserted the Flush signal. FIFOC.sub.A then asserts the Underflow signal. T.sub.A, in response to the Underflow signal, take transition 41 from state S2 or transition 42 from state S9 as stated above. In either case, T.sub.A will stop the clock on D, assert Release on the Transmitter status lines, inform RP.sub.A of the error and suspend operation. R.sub.B will take transition 9 to state S7 in a manner which is similar to that discussed above in response to a the mid-transmission acknowledgement. In order to recover, RP.sub.A must instruct T.sub.A to send a new packet with the same sequence number as the packet which underflowed. RP.sub.A will readily determine this fact when it detects that the Underflow signal is asserted in T.sub.A 's state information. This procedure assures that OP.sub.A and IP.sub.B will not lose synchronism with respect to packet sequence numbers. Retransmission of the failed packet will proceed from the node where the packet was last buffered in its entirety. Referring to state S11, transition sequences 18, 23, 44 and 26, 28, 45 are analogous to transition sequences 1, 41, 9 and 14, 42, 9, respectively.

Referring to FIG. 7, there is shown a state diagram which sets forth the transitions and states associated with error in sequence number, invalid data on the channels, false SOP and acknowledgment errors. An error in a sequence number is defined to be any corruption of the sequence number field other than logical complement which was discussed above as a wrong sequence number. The transition sequence begins in state S1 which is the main ready/waiting state. The sequences described hereafter could also occur while the protocol is in state S11 another ready/waiting state. However, in view of the similarity between state S1 and state S11, only references to state S1 will be made. In transition 66, all of the events described for transition 1 in the normal packet transmission occur up to the extraction of the sequence number S by R.sub.B. At this point, either R.sub.B is able to extract S from the packet but detects an error in the field or R.sub.B is totally unable to decipher what is detected on D. This condition is symbolized by G in the forward channel state in state S34. In either case, R.sub.B stops the clock on C, informs RP.sub.B of the error and suspends operation by taking transition 67 to state S32. Upon detecting loss of clock on C, T.sub.A stops the clock on D but must continue accepting data from FIFOA until the EndOfPacket signal is asserted. T.sub.A thereafter an error has occurred and suspends operation by taking transition 62 to state S7.

Referring again to state S1, while T.sub.A is waiting for an acknowledgement, R.sub.B determines that the packet was received without error and sends a positive acknowledgement to T.sub.A which is corrupted on C (transition 72). Thereafter, I.sub.A will either be able to extract the acknowledgment from C and detect an error in the packet or it will be completely unable to decipher what it saw on C. T.sub.A will immediately stop the clock on D, hold the Transmitter status lines in the Wait state, inform RP.sub.A of the error and suspend operation by taking transition 73 to state S17. R.sub.B then takes transition 32 to state S18 by stopping the clock on C, informing RP.sub.B of the error and suspending operation. It should be noted that following recovery, T.sub.A and R.sub.B will be out of synchronism with respect to S. However, the protocol will resolve this problem after one successful resending of the packet.

Additionally, while T.sub.A is waiting for an acknowledgment in state S1, R.sub.B detects an error in the packet as a result of a checksum failure. R.sub.B then sends a negative acknowledgment to T.sub.A which is corrupted on C (transition 69). Similar action will occur in transition 70 as which occurred in transition 73. Transition 9 is then taken to the error state S7. In this case, synchronism between T.sub.A and R.sub.B is retained and the protocol will recover on the next transmission of the packet.

While in state S1, R.sub.B is ready to receive a new packet, a noise burst strikes D such that what appears to be a valid SOP sequence and sequence number S are present on D. R.sub.B, will begin to receive what it believes to be a valid packet. The checksum should indicate an error in this packet upon which, in transition 61, R.sub.B sends a negative acknowledgment to T.sub.A, stops the clock on C, notifies RP.sub.B and suspends operation. T.sub.A then takes transition 62 to the error state S7. If at this time T.sub.A actually had been sending a real packet on D, it must continue accepting data from FIFOC.sub.A until the EndOfPacket signal is asserted. Recovery will proceed after the assertion of the EndOfPacket signal.

In a similar manner while waiting for an acknowledgment in state S1, noise corrupts the idle pattern on C in such a way that it appears to T.sub.A that R.sub.B is sending an acknowledgment packet (transition 63). R.sub.B will extract the data packet from D. T.sub.A upon detecting an error in the acknowledgement or in time between the packet and what is expected to be I.sub.W or I.sub.G takes transition 64 to state S6 wherein T.sub.A stops the clock on D. R.sub.B responds in taking transition 9 to the error state S7 wherein the clock on C is removed.

Referring to FIG. 8, shown therein is a state diagram which represents the protocol behavior when invalid data is detected on a channel or failure of an analog channel or port occurs. These events may occur at any time and as such are represented by the use of state variables w, x, y and z which are defined as:

w: The data packet the transmitter will send or is sending on the data channel wherein w represents S, S or Q.

x: The state or contents of the data channel wherein x represents I, Q, S, S, N, G or G.sub.S.

y: The packet the receiver expects to receive or is receiving on the data channel wherein y represents S, S or Q.

z: The state or contents of the control channel wherein z represents A.sub.S, A.sub.S, I.sub.G, I.sub.W, Q, N, G, or G.sub.S.

If T.sub.A is unable to determine the binary value of an encoded symbol or value taken from C, it takes transition 74 to state S35 and stops the clock on D. T.sub.A thereafter informs RP.sub.A of an error and suspends operation. However, for example, if T.sub.A is in the process of transmitting a packet S to R.sub.B and an invalid data symbol is detected on C, T.sub.A must continue accepting data from FIFOC.sub.A until the EndOfPacket signal is asserted. Thereafter, T.sub.A informs RP.sub.A of the error, takes transition 75 to state S36 and suspends operation. The recovery process at nodes A and B will be determined by the states of OP.sub.A and IP.sub.A when the error was detected.

Similarly, if R.sub.B is unable to determine the binary value of a data symbol detected on D, it takes transition 76 to state S37 and stops the clock on C. Certain actions may have to be performed before R.sub.B informs RP.sub.B of an error and suspends operation. Using the example immediately above, R.sub.B would have to first assert Flush, hereby ensuring that FIFOC.sub.B discards any traces of the packet received so far before signalling RP.sub.B and halting operation by taking transition 77 to state S36.

Failure of any analog component along D, such as the physical channel itself, will result in complete loss of clock and data information on the channel. The sequence of transitions 74 and 75 represents a failure of the data channel and the subsequent transition to the error state. Similarly, failure of any analog component along C will result in complete loss of signal information on the channel. Sequence of transitions 76 and 77 represents this failure. Transition 78 from state S34 to error state S36 represents the events that take place when both channels D and C fail simultaneously. This type of failure may result if D and C were simultaneously disconnected or severed.

Referring to FIGS. 9a and 9b there is shown a flow diagram of the steps of the protocol involved in transmitting a data packet from node A to node B. Referring also to FIGS. 1, 2 and 3, the protocol starts in a ready/waiting step 100 which is equivalent to state S1 shown in FIG. 6. Step 102 determines whether T.sub.A has asserted the Available signal to the router circuit 18 indicating that it is ready to transmit the data packet. Router circuit 18, in response to the assertion of the Available signal, facilitates the connection of T.sub.A through switch 14 to FIFO.sub.A in step 103. T.sub.A then de-asserts the Available signal, places the SOP sequence and the sequence number S on D and transmits the data packet S to R.sub.B in step 104. Additionally, T.sub.A starts the WaitingForAcknowledgment timer and asserts Wait on the Transmitter status lines. In step 105, a determination is made whether T.sub.A received an acknowledgment sequence from R.sub.B prior to the expiration of the timer. If the timer has expired in step 105, the protocol enters an error state represented by step 106. Alternately , T.sub.A in step 107 determines whether the Underflow signal is asserted by FIFOC.sub.A. In response to the assertion of the Underflow signal, the protocol moves into the error state step 106. In step 108, the protocol determines whether the transmission of the packet is complete. If an acknowledgment is received by T.sub.A prior to the completion of transmission of the packet, the protocol enters the error state by taking steps 109 and 110. Otherwise, R.sub.B determines in step 111 whether the data packet was received correctly. If there was an error in the transmission of the data packet, R.sub.B, as set forth in steps 112, 113 and 106, sends a negative acknowledgment to T.sub.A, flushes FIFO.sub.B, asserts an Underflow signal if FIFO.sub.B is cut through, which facilitates the flushing of any downstream nodes and enters the error state, respectively. However, if the data packet was received correctly, R.sub.B, in step 114, sends a positive acknowledgment to T.sub.A over the control channel C. R.sub.B, in steps 115 and 116, places the I.sub.W sequence on C when FIFOB is full. T.sub.A, in response thereto, asserts the Release signal and starts an I.sub.G timer. If the timer expires prior to the appearance of I.sub.G on C, T.sub.A in step 117 facilitates node A to enter the error state. If the timer in step 117 did not expire and space becomes available in FIFOB, R.sub.B then places I.sub.G on C and returns to the ready/waiting step 100.

Returning to step 115, if there was space in FIFO.sub.B for the entire data packet S, R.sub.B complements the sequence number and the protocol returns to the ready/waiting step 100.

As stated above, when the protocol enters the error state, a port or channel failure has occurred between nodes A and B. Thereafter, the respective recovery processors 24 and 34 can autonomously determine whether recovery from the failure is possible. For purposes of illustration, let us assume that OP.sub.A has entered the error state, RP.sub.A must then examine the state information of the port prior to determining the recovery sequence. RP.sub.A may determine that transitions 1, 2, 11 and 12, as shown in FIG. 6, has occurred and may simply want T.sub.A to re-transmit the data packet to R.sub.B. Simultaneously, RP.sub.B attempts to recover the channel and may place node B in the ready/waiting state S1.

As further stated above, the protocol includes the steps wherein the data packet is transmitted from a first node to a protocol also facilitates the transmission of a data packet from a first node through intermediate node or nodes to a final node without storing the packet in the intermediate node or nodes. This process is called virtual cut-through. Referring to FIGS. 9b, 10a and 10b there is shown a flow diagram of the virtual cut-through process. For purposes of illustration, let us assume that a data packet is to be transmitted from node A through B to node C. Node C is identical to nodes A and B as shown in FIG. 1. Referring to FIGS. 9a and 9b, steps 100 through 107 are also used in the virtual cut-through process. When node B receives the header of the data packet S and determines that the packet is for node C, T.sub.B in step 150 asserts an Available signal to the router circuit 30 and FIFOC.sub.B then connects FIFO.sub.B to T.sub.B. Thereafter, T.sub.B transmits the packet to R.sub.C and starts a WaitingForAcknowledgment timer in steps 152 and 154. The packet S is transmitted to R.sub.C without being stored in FIFO.sub.B. If the timer expires prior to the receipt of an acknowledgment sequence from R.sub.C, T.sub.B facilitates the entering of the error state in steps 156 and 178. However, if an underflow signal is asserted by FIFOC.sub.B in step 160, FIFO.sub.B is flushed in step 162 and FIFOC is flushed in step 167. From step 167, a negative acknowledgment is sent to T.sub.B in step 165 and the error state in step 185 is entered.

Returning to step 160, T.sub.B then determines in step 163 whether transmission of the data packet is complete and whether a premature acknowledgment was received in step 164. If a premature acknowledgment was received, the error state is entered in step 166, otherwise R.sub.C determines whether the data packet was received correctly in step 168. R.sub.C sends a positive acknowledgment to T.sub.B in step 170 if the packet was received correctly. The data packet is stored in FIFO.sub.C in step 172 and the sequence number is complemented in step 174. The protocol thereafter returns to the ready/waiting state.

If in step I72, FIFO.sub.C was full, R.sub.C places I.sub.W on C, T.sub.B asserts the Release signal and starts the I.sub.G timer in step 176. If space becomes available prior to the timer expiring, R.sub.C ready/waiting state as set forth in steps 178, 180 and 182, respectively. Alternately, if the I.sub.G timer has expired prior to the appearance of I.sub.G on C, the protocol enters the error state.

Thus, the data packet was transmitted from node A through node B to its destination node C without storing the packet in the intermediate node B. This process can be used to transmit the data packet through any number of nodes to its destination node. It should be noted that if an Underflow signal is asserted by anyone of the intermediate nodes, the node asserting the signal and all downstream nodes flush their buffers and enter the error state. Re-transmission of the data packet will occur by the node where the data packet was last stored in its entirety.

While the invention has been particularly shown and described with reference to a preferred embodiment thereof, it will be understood by those skilled in the art that various changes in detail maybe made therein without departing from the spirit, scope and teaching of the invention. Accordingly, the data link protocol herein disclosed is to be considered merely as illustrative, and the invention is to be limited only as specified in the claims.