Communication channel

[0002] This application claims priority to S.N. 98402633.6, filed in Europe on Oct. 22, 1998.

[0003] This invention relates to transferring large amounts of data between various nodes of a digital system, and more particularly to providing a communication channel that can have a plurality of synchronized data links provide a higher bandwidth than can be provided by a single data link.

[0004] Within a digital system, streams of data are often transferred from one node in the system to another. This is often done in a word parallel manner using a multi-bit bus, such as a 64 bit data bus or a 32 bit address bus. When a digital system is implemented on different printed circuit (PC) cards, a backplane which has one or more buses is used to interconnect the PC cards. In each case, each word transfer is done in manner so that all of the bits of the word on the parallel signal lines of the data bus remain synchronized with a designated timing signal.

[0005] A digital system may have two or more nodes which are separated by a distance that makes interconnection via a backplane impractical. In this case, data streams are transferred over a communication channel that may be a conductive cable, an optical link, an infared link, or a radio link, for example. Generally, the data stream is transferred in a serial manner over the communication channel so that only a single link is required. Cables may provide multiple signal lines so that the data stream can be transferred in a parallel manner, but the transfer rate is controlled so that all of the bits of the word on the parallel signal lines of the cable remain synchronized with a designated timing signal.

[0006] Any physical media for transferring data inherently causes a delay in the transfer of the data. When parallel data links are employed, each data link may have a slightly different delay characteristic due to physical differences. Thus, a skew is introduced between data transferred on the different links. Induced skew limits the transfer rate at which data can be transferred in a parallel manner on a communication channel.

[0007] An illustrative embodiment of the present invention seeks to provide a method for transferring a single stream of data on a plurality of data links that avoids or minimizes above-mentioned problems.

[0008] Aspects of the invention are specified in the claims. In carrying out principles of the present invention, a method provides for transferring a single stream of ordered data over a plurality of data links each having a transmitter and a receiver, wherein the single stream of data comprises a plurality of words each having a plurality of bits. The method divides the single stream of data into a plurality of sub-streams of data and inserts a frame pulse periodically in each sub-stream of data. Each sub-stream of data is then transmitted over a corresponding data link of the plurality of data links in a parallel manner to form a plurality of received data sub-streams, wherein a first data link has a first delay time that may be different from a second delay time of a second data link, such that a data skew occurs between a first received data sub-stream and a second received data sub-stream. After reception, a byte clock is recovered from each received data sub-stream and the byte clock and frame pulse of each slave received data sub-stream is synchronized to the byte clock and frame pulse of the master received data sub-stream such that the data skew is eliminated. Then, the plurality of received data sub-streams are combined to form a single received stream of ordered data.

[0009] According to another feature of the invention the byte clock of each data slave link if forced to align approximately with the byte clock of the master data link, and a bit tap point on each received data sub-stream is shifted so that byte boundaries of each received data sub-stream align with the byte clock associated with each received data sub-stream.

[0010] According to another feature of the invention, frame synchronization is tested for by determining if the frame pulse of each of the slave received data sub-streams is synchronized with the frame pulse of the master received data sub-stream. If frame synchronization is not present, then a byte tap point in each of the slave received data sub-streams is shifted up to a first number of times until each slave received data sub-stream is frame synchronized with the master received data sub-stream.

[0011] According to another feature of the invention, a byte tap point in the master received data sub-stream is shifted by one position if global frame synchronization is not achieved, and then the slave tap points are again shifted until each slave received data sub-stream is frame synchronized with the master received data sub-stream.

[0012] For a better understanding of the present invention, reference will now be made, by way of example, to the accompanying drawings, in which:

[0013] FIG. 1 is a block diagram of a portion of a digital system illustrating a communication channel for transferring data between two nodes of the system, according to aspects of the present invention;

[0014] FIG. 2 is a block diagram showing more detail of the plurality of data links which interconnect the two nodes of FIG. 1;

[0015] FIG. 3 is an illustration of a typical frame of data which is transferred on the communication channel of FIG. 1;

[0016] FIG. 4 is a more detailed block diagram of a data link transmitter of FIG. 2;

[0017] FIG. 5 is a schematic illustrating interconnections of the plurality of data link transmitters of FIG. 2;

[0018] FIG. 6 is a state diagram illustrating the operation of the data link transmitter of FIG. 4;

[0019] FIG. 7 is a more detailed block diagram of a data link receiver of FIG. 2;

[0020] FIG. 8 is a schematic illustrating interconnections of the plurality of data link receivers of FIG. 2;

[0021] FIG. 9 is a more detailed block diagram of the clock recovery circuit for the receiver of FIG. 7;

[0022] FIG. 10 is a schematic of the bit shifter for the receiver of FIG. 5;

[0023] FIG. 11 is a state diagram which controls the synchronization process of the communication channel of FIG. 2;

[0024] FIG. 12 is a timing diagram illustrating S2PSyncOut signal timing;

[0025] FIG. 13 is a timing diagram illustrating byte clock synchronization of the plurality of data links of FIG. 2, according to an aspect of the present invention;

[0026] FIG. 14 is a timing diagram illustrating bit rotation of the plurality of data links of FIG. 2 to produce byte alignment to a common word clock, according to an aspect of the present invention;

[0027] FIG. 15 is a timing diagram illustrating frame synchronization of the plurality of data links of FIG. 2, according to an aspect of the present invention;

[0028] FIG. 16A is a flow chart illustrating the process of byte aligning the plurality of data links of FIG. 2;

[0029] FIG. 16B is a flow chart illustrating the process of frame synchronization used in each slave link of FIG. 2; and

[0030] FIG. 16C is a flow chart illustrating the process of frame synchronization used in the master link of FIG. 2.

[0031] Corresponding numerals and symbols in the different figures and tables refer to corresponding parts unless otherwise indicated.

[0032] Aspects of the present invention include methods and apparatus for transferring a single stream of ordered data over a communication channel having a plurality of parallel data links which each may have varying amounts of inherent delay. In the following description, specific information is set forth to provide a thorough understanding of the present invention. Well known circuits and devices are included in block diagram form in order not to complicate the description unnecessarily. Moreover, it will be apparent to one skilled in the art that specific details of these blocks are not required in order to practice the present invention.

[0033] FIG. 1 is a block diagram of a portion of a digital system 100 illustrating a communication channel 110 for transferring data between two nodes (120, 130) of the system, according to aspects of the present invention. A reverse channel 111 is provided for transferring data in the opposite direction to channel 110, and uses a separate physical media. Another embodiment may multiplex bi-directional data on the same physical media.

[0034] Node 120 has transmitting circuitry 121, receiving circuitry 122, and processing circuitry 123. Processing circuitry 123 may include a digital signal processor, memory circuits, analog or linear circuits, or any of a wide range of known or novel circuits. Similarly, node 130 has transmitting circuitry 131, receiving circuitry 132, and processing circuitry 133. Physical media 115 connects transmitter 121 to receiver 132, and is twisted pairs of wire in this embodiment. Likewise, physical media 116 connects transmitter 131 to receiver 122.

[0035] Due to a need to transfer a large amount of data across communication channel 110, a plurality of data links are included within channel 110.

[0036] FIG. 2 is a block diagram showing more detail of the plurality of data links which interconnect the two nodes of FIG. 1. Seven data links 110a-110gare illustrated, but communication channels with a greater number or a fewer number of data links can be embodied, according to aspects of the present invention. Circuitry 150a-gdivides a single stream of ordered word data on bus 150 into a plurality of sub-streams of ordered byte data. Data bus 150 is m bits wide for m-bit words, while each of buses 150a-150gare n bits wide and bytes are n-bits. However, in another embodiment, 150a-150gcould be different widths.

[0037] Data links 110a-110gtransmit each data sub-stream serially, but all the data links transmit in a parallel manner so that all of the data sub-streams are received approximately coincidentally. However, each data link 110a-110ghas an inherent transfer delay time, and the transfer delay time of one link is typically different from the transfer delay time of another link due to physical differences in the data links. Therefore, a skew is induced between the various received data sub-streams at receivers 132a-132g. According to aspects of the present invention, receivers 132a-132gcompensate for the skew between data links so that circuitry 160 can combine the plurality of received data streams to form a single received data stream of ordered data which is m bits wide. This single stream of data can be buffered in a single FIFO 170 in response to a common word clock 171 before being sent to processing circuitry 133.

[0038] According to an aspect of the present invention, one of the links is designated as a master link, and all of the other links are designated as slave links. During operation, the slave links are synchronized to the master link by synchronizing circuitry 200 and 202 in each receiver.

[0039] FIG. 3 is an illustration of a typical frame of data which is transferred on each data link of communication channel 110 or 111 of FIG. 1. Each frame includes a frame pulse portion 200 and a data portion 210. The length of the frame is defined by a frame_count_width 220, and is typically 1024 bytes. Each byte comprises a number of bits, which is defined by a datapath_width variable. For a given embodiment of digital system 100, the number of data links 110a-110g, the frame_count_variable and the datapath_width are selected when digital system 100 is designed; however, other embodiments may vary, as will be discussed later with reference to Table 15.

[0040] The framing pulse and pattern formats are given in Table 1. The framing pulse occupies two bytes, and the least significant two bits are masked and used to transfer status information. Framing pulse 200 includes FRAME_SYNC_SEQ 201, status bits FVALID 202 and FERF 203. During synchronization of channel 110/111, stuff characters are inserted in data portion 210 in order to provide sufficient edge density to allow a clock recovery circuit to achieve lock. The stuff characters shown in Table 1 are selected to provide maximum hamming distance between the framing pulses and the stuff characters to eliminate the possibility of false framing. Each stuff character shown in Table 1 is zero filled to occupy two byte positions.

[0041] Two status bits are defined. FVALID 202 indicates that the current frame contains user data. FERF 203 (far end receiver failure) conveys the status of the local receiver's OOF (out of frame indicator) to the downstream transmitter for use in enabling data transfer.

[0042] For example, during synchronization, receiver 132's OOF is active and transmitter 121 is outputting the stuff pattern in the payload. Receiver 132's OOF is transmitted to receiver 122 via bit 0 (FERF) of the header pattern in link 111. This is decoded in receiver 122 and included in the logic which decodes link synchronization in transmitter 121. Hence link 110 does not start transferring valid data until all elements are ready.

[0043] FIG. 4 is a more detailed block diagram of a data link transmitter 400, which is the same as transmitters 121a-121gof FIG. 2. FIG. 5 is a schematic illustrating interconnections of a plurality of data link transmitters 400(i). For multiple data links, each link transmits framing pulses simultaneously. The transmitters are synchronized using a master transmit strobe. The functional blocks are described below.

[0044] Table 2 describes various signals which are connected to transmitter 400 as shown on FIG. 4 and/or FIG. 5. All signals are active high unless explicitly stated otherwise.

[0045] Throughout this document the VHDL attribute syntax has been used to represent bus widths. For example BUS′high represents the integer index of the highest bit in the bus BUS′LOW represents the integer index of the lowest bit in the bus. BUS′range represents the range of bits. The keyword ‘downto’ is used to represent all bits between two indexes.

[0046] For example, an eight bit bus DATA[7:0]:

[0047] DATA′high=7

[0048] DATA′low=0

[0049] DATA′range=7 downto 0

[0050] Table 3 describes signals used by Frame Output circuitry 410 and Multiplexer 430. In response to a timing signal from the TX FSM the frame output circuitry 410 outputs framing bytes to the transmit multiplexer 430. Multiplexer 430 then sends the n-bit wide data sub-stream to the parallel to serial (P2S) converter 450. P2S 450 sends the serialized data sub-stream to physical media interface 460. The output of interface 460 is determined by the type of physical media 115 used in the communication channel.

[0051] FIG. 5 shows an implementation which fully synchronizes the bidirectional link 110/111 before transmitting data. The FERF signal that is conveyed on the master link is the only one used in synchronization. The other FERF inputs may be used to transfer proprietary in-band data.

[0052] During the HUNT, PREBYTESYNC and PRESYNC states (described with reference to FIG. 6) a fixed sequence is output in the frame payload which does not contain false framing patterns (see Table 1). At all other times multiplexer 430 forwards inputs from scrambler 420. All signals are active high unless explicitly stated otherwise.

[0053] It is desirable to provide scrambling on the data to reduce EMC and provide sufficient edge density to ensure a low cost local oscillator may be used to generate the reference clock in an embodiment which uses a clock recovery device. Table 4 describes signals used by scrambler 420. All signals are active high unless explicitly stated otherwise.

[0054] Scrambler 420 randomizes the data using a maximal length shift register (MLSR) with tap points selected to provide pseudo random behavior. The generator polynomial may be set to one of the values listed in Table 5. Alternatively, other polynomials may be used.

[0055] Different scrambling polynomials may be selected according to perceived correlation of the periodicity of the scrambler and the framing pattern. The scrambler may be selected according to the application and may be frame or self synchronizing.

[0056] Correlation and Periodicity

[0057] A pseudo-random sequence generated by an n-bit maximal length shift register is a binary sequence of period r=2ⁿ−1. The output will have a period equal to the least common multiple, LCM, of p (the input period) and r. The LCM is enlarged by selecting a prime number for r.

[0058] X⁷+X⁶+1: This scrambler is defined in ITU-T Rec. G.709 for its performance in clock synchronization.

[0059] X³¹+X²⁸+1: This is the scrambler defined in the ITU-T Rec. I.432 (or in the ETSI ETS 300 299) for the cell based physical layer. It is a distributed sample scrambler having the pseudo random sequence polynomial X³¹+X²⁸+1.

[0060] During the framing sequence frame synchronous scrambler 420 is reset to all 1's. Scrambler 402 may be disabled permanently via the top level interface. When in test mode scrambler 420 ignores the data and outputs the scrambler code, descrambler 720 (see FIG. 7) behaves normally.

[0061] Transmit FSM (Finite State Machine) 440 outputs codes to enable the frame sequence. It also accepts a master framing pulse input to which output frame sequence timing is locked. The transmitter waits for removal of FERF before enabling normal operation of TXEN. In test mode input data is ignored and the frame period is set to 16 bytes. Signals used with Transmitter Finite State Machine 440 are described. in Table 6. All signals are active high unless explicitly stated otherwise.

[0062] Table 7 lists the states and codes of FSM 440. Signal Frame_sync_gate_int is an Internal framing pulse. Signal Frame_sync_disparity is a difference between Frame_sync_gate_int and Frame_sync_gate_ext. In the table, ‘1’ denotes a signal in its active state, ‘0’ inactive. This may not correspond with high and low logic levels within the design.

[0063] FIG. 6 is a state diagram illustrating the operation of the data link transmitter 400.

[0064] FIG. 7 is a more detailed block diagram of data link receiver 700, which is the same as receivers 132a-132 of FIG. 2. Receiver 700 performs the following functions:

[0065] Receives the data sub-stream from physical media 701 in physical media interface circuit 702.

[0066] Recovers the data using an appropriate clock phase in clock recovery circuit 704.

[0067] Converts the serial data into byte data with an appropriate byte clock in serial-to-parallel circuit 706.

[0068] Recovers framing pulses in frame check circuit 710.

[0069] Recovers bit alignment from framing patterns in bit shifter 730.

[0070] Descrambles the encoded data in descrambler 720. Descrambler 720 descrambles the received data sub-stream in a complimentary manner to scrambler 420.

[0071] Recovers byte alignment by accounting for differential skew between links in byte pipe 750 in conjunction with tap point multiplexer 751.

[0072] Test circuit 760 provides methods for determining if receiver 700 is functioning properly.

[0073] FIG. 8 is a schematic illustrating interconnections of a plurality of data link receivers 700(i). Multiple receivers are cascaded to provide wide data-paths and data rates higher than that supported by any one data link technology, according to aspects of the present invention. One data link receiver is designated as the master receiver and the other receivers are designated slaves. This is done when the receivers are instantiated using a CASTE attribute. Alternatively, a CASTE signal can specify the type link.

[0074] Table 8 describes various signals which are connected to receiver 700 as shown on FIG. 7 and/or FIG. 8. All signals are active high unless explicitly stated otherwise.

[0075] FIG. 9 is a more detailed block diagram of the clock recovery circuit 704. Clock generator 900 generates a plurality of clock phase signals 902(i) based on reference clock 903 using known techniques. Each clock phase signal 902(i) differs in phase by approximately ⅛ of a bit period. Clock recovery circuit 901 then compares edge transitions on received data sub-stream 904 to each of clock phase signals 902(i) and selects a clock phase signal that is most in correspondence with the received data sub-stream. In this manner, a bit clock is formed that has a fixed time relation to local reference clock 903.

[0076] S2P circuit 706 then converts the serial received data sub-stream to a sequence of parallel data bytes on data bus 910. S2P 706 also forms byte clock on signal line 911 in response to the bit clock signal. When S2P circuit 706 is instantiated during the integrated circuit design process, the parallel data byte width (n) is selected, as discussed with reference to Table 15.

[0077] According to an aspect of the present invention, signal S2PSYNCOUT is provided by master receiver 700(0) and causes S2P circuit 706 to synchronize byte clock signal 911 with the byte clock signal of master receiver 700(0), as will be explained in more detail later.

[0078] FIG. 10 is a schematic of bit shifter circuitry 1000 that illustrates a byte data path width of five bits. Bit shifter circuitry 1000 provides a shiftable bit-tap point that is operable to align byte boundaries of the received data sub-stream with the byte clock of the receiver. Latches 1001 and 1002 are cascaded so that together they buffer ten bits of sequential data. Multiplexer 1010 is operable to tap any-five bits of data from latches 1001 and 1002 in response to select signal 1021 formed by counter 1020. The tap point is incremented in response to shift_inc_ena signals received from FSM 740. During synchronization, FSM 740 enables shifting of the tap point until a frame-sync-rx is asserted by frame check circuit 710 indicating that a frame pulse has been correctly received. Table 9 describes various signals associated with bit shift circuit 1000. All signals are active high unless explicitly stated otherwise.

[0079] Frame check circuitry 710 produces frame synchronization pulses on signal line frame_sync_rx when a valid frame sequence has been received. These pulses are ungated and may occur outside the normal frame period. Table 10 describes signals associated with frame checker 710. All signals are active high unless explicitly stated otherwise.

[0080] Byte pipeline 750 with byte tap point selected by multiplexer 751 is operable to delay the received data sub-stream a number of bytes equal to the depth of the pipe. The tap point is selected in response to the frame check circuitry so that the received data sub-stream is frame synchronized with the received data sub-stream of the master link. This process will be described in detail later. The depth of the pipeline is specified when the integrated receiver 700 is instantiated during design of an integrated circuit.

[0081] It is this pipeline depth which defines the amount of skew which can be tolerated between the data links, according to an aspect of the present invention. The tap point for this pipeline is set by FSM 740. When appropriate, the master shifts its tap point as well as the slaves. This occurs over a time period sufficient to allow all the slaves to cycles through all possible relative skews on each master tap point. The absolute maximum synchronization lock time is therefore proportional to the number of pipeline stages and the frame length. The pipeline also delays control information frame_sync_out and rdval_out to assure correct synchronization and alignment with the receive data valid signal.

[0082] Table 11 describes signals associated with byte pipe 750. All signals are active high unless explicitly stated otherwise.

[0083] When test mode is enabled, test circuitry 760 checks for set bits in the descrambled data stream. Any set bits found are erroneous when the transmitter is also in test mode. Test status is conveyed to the core via signals tpass and tfin and may also be read by the scan chain, not shown. Table 12 describes signals associated with test circuitry 760. All signals are active high unless explicitly stated otherwise.

[0084] A single test is initiated by asserting the TMODE pin on the transmitter and receiver. This test uses 16 byte frames to provide a faster method of achieving synchronization. Both transmitter and receiver must be placed into test mode as the transmitter must be inhibited from transferring user data. Once the system is synchronized an all zero frame is transferred over the channel and the descrambled results are checked for set bits. Any set bits will result in a test failure, which will be conveyed via TPASS. Resetting the TMODE pin will result in loss of synchronization and therefore a delay before user data transfer may be resumed.

[0085] Testing is done in conjunction with scan chains and may involve, for example, one of the following:

[0086] Integration into an existing scan chain

[0087] Integration into an additional scan chain

[0088] Integration in the boundary scan chain indexed by a particular instruction code within an IEEE1149.1 implementation.

[0089] Communication channel 110/111 supports both clocked scan and multiplexed flip flop test insertion.

[0090] Receive Finite State Machine 740 controls the operation of receiver 700. Table 13 describes signals associated with FSM 740.

[0091] FIG. 11 is a state diagram which controls the synchronization process of receiver 700 The framing patterns are sent at a rate set when transmitter 400 is instantiated, as specified by the frame_count_c variable of Table 15.

[0092] Table 14 provides a detailed description of the state transitions of FSM 740.

[0093] The various states illustrated by FIG. 11 and Table 15 are as follows:

[0094] HUNT (H): During this state the framing sequence is hunted on a bit-by-bit basis. Scrambling is disabled. When a correct framing sequence is identified the state changes to PREBYTESYNC.

[0095] PREBYTESYNC (PB): During this period the state machine shifts the tap point of the byte pipeline until the framing sequences are aligned with the RFPIN (receive framing pulse—driven by the master).

[0096] PRE SYNC (PS): During this period the frame header sequences are checked frame by frame. An incorrect framing sequence results in a return to the HUNT state. δ correct frame sequences complete on all pairs will cause transition to state SYNC. δ is the max_delta_count_c variable specified when receiver 700 is instantiated, as described in Table 15.

[0097] SYNC (S): The scrambler is enabled. α errored frame sequences will cause the state to return to HUNT. α is the MAX_ALPHA_COUNT_C variable specified when receiver 700 is instantiated, as described in Table 15.

[0098] A counter in control circuitry 770 continually monitors progress through a frame. Two strobes are generated which are used throughout the design for timing purposes, as follows:

[0099] Frame_sync_gate is a signal which is active during framing pulses and is used to gate the outputs from the frame checking logic.

[0100] Frame_sync_valid_gate is a signal which is active in the clock cycle immediately following the framing pulse.

[0101] One chain, byte locked chain (blcin/blcout), is used to interconnect a plurality of receivers 700, as illustrated in FIG. 8. Master and slave behavior differs slightly. In the master the chain output reflects its own status, on a slave the chain output represents the chain input ANDed with the its own status.

[0102] The master uses the chain input to drive the multidrop signals (RFPOUT and S2PSYNCOUT) used in the PREBYTESYNCPRESYNC and PRESYNCSYNC state transitions.

[0103] Link synchronization, according to aspects of the present invention, will now be described in detail. During link synchronization the master receiver issues a pulse on signal S2PSyncOut which resynchronizes all slave S2P devices. This is to ensure that a single word clock 1202 may be used to reliably clock all the bytes. The timing of this signal is shown in FIG. 12, illustrating S2PSyncOut signal timing. This resynchronization strobe is active when the master is in the HUNT state, and is responsive to the word clock signal, as indicated at 1200. Word clock 1202 is the inverse of the byte clock of master receiver 700(0). Due to bit synchronization of master receiver 700, data bytes in the master data sub-stream are synchronized with the byte clock of master receiver 700, as indicated at 1201.

[0104] FIG. 13 is a timing diagram illustrating byte clock synchronization of the plurality of data links of FIG. 2, according to an aspect of the present invention. Timing for only three links are illustrated, but it is understood that any number of links can be synchronized in a similar manner. Before link synchronization, each of the links recover a bit clock 1300, 1302, and 1304 as discussed earlier. Each link then forms a byte clock 1301, 1303 and 1305, but the timing of each byte clock signal is not related to the others, as illustrated in FIG. 13. Therefore, disadvantageously, a single clock can not be reliably used to clock all of the received data sub-streams of digital system node 130, as shown at time 1340.

[0105] As discussed above, S2PSYNCOUT signal 1310 is asserted during the HUNT state of the master and sent to the slave data receivers at time 1320. This connection is shown in FIG. 8. S2P 706(i) in each slave receiver 700(i) is operable to reset its own byte clock counter in response to the S2PSYCOUT signal 1310. Thus, each slave receiver realigns its byte clock as shown at 1330 and 1331 to approximately align with the byte clock of the master receiver. Also, each S2P 706(i) places a new data byte on its data_out bus 771(i) in accordance with the realigned byte clocks, as shown at time 1350. Thus, advantageously and according to an aspect of the present invention, a single inverted byte clock 1360 can be used to clock the data sub-streams of all of the receivers, as shown at time 1351. This single byte clock is also referred to as the word clock. While the data transitions at time 1350 have a jitter 1352, reliable clocking at time 1351 is assured.

[0106] However, the data bytes framed by the byte clocks do not necessarily correspond to the underlying data sub-stream. For example, link(1) bytes now begin at bit 3 as shown at 1370, for example. Likewise, link(2) bytes begin a bit 4 at 1371, for example.

[0107] FIG. 14 is a timing diagram illustrating bit rotation of the plurality of data links of FIG. 2 to produce byte alignment to common byte clock 1360, according to an aspect of the present invention. As discussed earlier, the bit tap point of bit shifter circuitry 730 in each receiver is set to align byte boundaries of the received data sub-stream with word clock 1360, as shown at 1400. In FIG. 14, signals 771(i) are the link(i) byte signals on bus 771 of FIG. 7 and signals 772(i) are the rotated link(i) byte signals on bus 772 of FIG. 7. Now, advantageously, link(1) bytes begin with bit 1 as shown at 1410 and link(2) bytes also begin with bit 1, as shown at 1411.

[0108] Thus, according to an aspect of the present invention, each data link now provides a sub-stream of correctly ordered bytes that can be combined to form a single received data stream of ordered data that can be reliably clocked with a single clock into a single processing circuit, such as single FIFO 170 of FIG. 2. However, different inherent delays in each data link may result in byte skews between received data sub-streams.

[0109] FIG. 15 is a timing diagram illustrating frame synchronization of the plurality of data links of FIG. 2 in order to remove unwanted byte skew between received data sub-streams, according to an aspect of the present invention.

[0110] In FIG. 15A, which illustrates only three links for clarity, a master link has a frame pulse 1510 at time 1501, while slave link(1) has a frame pulse 1520 at time 1502, and slave link(2) has a frame pulse 1530 at time 1500. Clearly, three data bytes clocked (one byte from each link) at time 1503, for example, will not be correctly ordered.

[0111] Still referring to FIG. 15A, according to an aspect of the present invention, receiver 700 has a byte pipe 750 which has several tap points, as discussed earlier. Five are illustrated (1511-1514), but other embodiments can have fewer or more, as indicated in Table 15. Advantageously, by selecting tap point 2 on the master, illustrated at 1512a, and tap point 3 on slave 2, illustrated at 1533, all three frame pulses 1510a, 1520aand 1530aare aligned at time 1540. Therefore, three data bytes clocked at time 1503, for example, will be correctly ordered, according to an aspect of the present invention.

[0112] Likewise, refering to FIG. 15B, a skew 1550 of up to four bytes can be corrected with a pipe having five tap points. By selecting tap point 5 illustrated at 1535, and by selecting tap point 4 illustrated at 1514a, all three frame pluses 1510b, 1520b, and 1530bare aligned at time 1560.

[0113] Therefore, as the length of byte pipe 750 is increased in each receiver, the amount of data link skew that can be tolerated is increased, according to an aspect of the present invention.

[0114] FIGS. 16A-16C are flow charts illustrating the process of synchronizing the byte clock and frame pulse of each received data sub-stream to the byte clock and frame pulse of the master received data sub-stream such that data skew is eliminated. FIG. 16A is a flow chart illustrating the process of byte aligning a plurality of data links. In step 1600, a data transmission channel is instantiated using multiple data links. In step 1602,transmission is begun on each link pair with stuff characters in the payload and the framing pulses on all links synchronized. In step 1604, each receiver is synchronized to the data sub-stream using stuff characters to recover a bit clock relative to a receiver local reference clock. In step 1606, a byte clock is formed on each link relative to that link's bit clock. In step 1608, slave link byte clocks are synchronized to a master link byte clock. In step 1610, a test is performed to determine if a frame pulse is being received on each link(i). If not, a bit tap point on link(i) is shifted to rotate data bytes until a frame pulse is recognized. At this point, all links are now synchronized to a common word clock and the byte_locked_chain is asserted at step 1614, according to. an aspect of the present invention.

[0115] FIG. 16B is a flow chart illustrating the process of frame synchronization used in each slave link. After each slave link has successfully byte aligned itself at step 1614, the master asserts signal LBLOUT and a test is performed by each slave(i) to determine if slave(i) is in frame sync with the master at step 1620. If not, at step 1622 the byte tap point of each slave(i) is shifted until frame synchronization is detected. Once each slave(i) is frame synchronized, it asserts its portion of the link_byte_locked chain and waits in step 1624 for the master to assert signal LSOUT indicating all slave(i) are frame synchronized. Once this occurs, each slave data link begins to transmit payload data over the synchronized channel.

[0116] FIG. 16C is a flow chart illustrating the process of frame synchronization used in the master link. After all slave links are byte aligned at step 1614, the master link asserts the LBLOUT signal to start the frame synchronization process at step 1640 and then waits two frames at step 1642. The master then monitors the link_byte_locked chain to determine if all slaves are in frame sync with master at step 1644. If not, then the master shifts its byte tap point one position at step 1646, waits two frames at step 1642 while each of the slave links again attempt to frame synchronize by each performing steps 1620, 1622, and 1624. The master again monitors the link_byte_locked chain to determine if all slaves are in frame sync with master at step 1644. The master repeats steps 1644 and 1646 until all slave links are frame synchronized, or until the master selects the last byte tap point, such as tap point 1514 of FIG. 15A. Once the link_byte_locked chain is asserted, then the master waits a preselected number of additional frames at step 1648 while each data link monitors itself for framing errors, and then the master asserts signal LSOUT at step 1650 and all links begin to transmit payload data over the synchronized channel at step 1652 and step 1626. The preselected number may be five, for example; other embodiments may wait a different number of additional frame at step 1648. In Table 15, the max_delta_count_c parameter specifies the preselected number of additional frames to wait for.

[0117] The number of data links needed in a digital system is determined by the bandwidth requirement of the data stream and the capabilities of the physical media and the transmitter/receiver. Referring back to FIG. 2, other embodiments may have more or fewer data links. To achieve both a specified word width m and a system specified bit rate it may be necessary to adjust the width n of the parallel bytes transmitted on each link 110(i).

[0118] Example: 16 bit interface at 75 MHz on a technology with a maximum individual data link capacity of 400 Mbps.

[0119] Channel bandwidth requirement: 1200 Mbps

[0120] Assume the number of links is i=3. This gives a per pair bandwidth of 400 Mbps which is within the capabilities of the technology. However, since 16/3 is not an integer it is necessary to increase the amount of bits per link to 6, giving an 18 bit wide interface (i.e. two spare bits).

[0121] The required bandwidth including this overhead is then . . .

[0122] 18/16 * 1200 Mbps=1350 Mbps

[0123] Per link bandwidth: 450 Mbps

[0124] This is now not viable with the chosen technology. The only option on this technology node is to increase the link count to four with four bit wide datapaths. There are now no spare bits and a throughput requirement for each link is only 300 Mbps.

[0125] The parameters listed in Table 15 are passed to the computer assisted design (CAD) software in the autogeneration phase of an integrated circuit which will have a communication channel 110. Based on the selected parameters, an embodiment of the present invention is created using design cells from a macro design library.

[0126] Fabrication of digital system 100 involves multiple steps of implanting various amounts of impurities into a semiconductor substrate and diffusing the impurities to selected depths within the substrate to form transistor devices. Masks are formed to control the placement of the impurities. Multiple layers of conductive material and insulative material are deposited and etched to interconnect the various devices. These steps are performed in a clean room environment.

[0127] A significant portion of the cost of producing the data processing device involves testing. While in wafer form, individual devices are biased to an operational state and probe tested for basic operational functionality. The wafer is then separated into individual dice which, may be sold as bare die or packaged. After packaging, finished parts are biased into an operational state and tested for operational functionality.

[0128] An alternative embodiment of the novel aspects of the present invention may include other circuitries which are combined with the circuitries disclosed herein in order to reduce the total gate count of the combined functions. Since those skilled in the art are aware of techniques for gate minimization, the details of such an embodiment will not be described herein.

[0129] An advantage of the present invention is that it is applicable to various physical media, such as wire cables, optical cables, laser optic links, infrared links, radio or microwave links, backplanes, etc.

[0130] Each link can use single ended signaling or differential signaling, for example.

[0131] Another aspect of the present invention is that the single stream of ordered word data can be divided into a plurality of data sub-streams using different techniques on different embodiments. For example, in the described embodiment each word of the data stream was divided into bytes with a bit length equal to the data path width of each data link. In another embodiment, a preselected number of words may be grouped as a single item and then divided into bytes with a bit length equal to the data path width of each data link. In another embodiments, a first portion of a word may be divided into bytes with a bit length equal to the data path width of each data link, and then a second portion of the same word may be divided into bytes, etc. Other schemes for dividing a single stream of data into multiple sub-streams are included within the scope of the present invention.

[0132] As used herein, the terms “applied,” “connected,” and “connection” mean electrically connected, including where additional elements may be in the electrical connection path.

[0133] While the invention has been described with reference to illustrative embodiments, this description is not intended to be construed in a limiting sense. Various other embodiments of the invention will be apparent to persons skilled in the art upon reference to this description. It is therefore contemplated that the appended claims will cover any such modifications of the embodiments as fall within the true scope and spirit of the invention.

[0134] Although the present invention and its advantages have been described in detail, it should be understood that various changes, substitutions and alterations can be made herein without departing from the spirit and scope of the invention as defined by the appended claims.