System and method for synchronizing a selected master circuit with a slave circuit by receiving and forwarding a control signal between the circuits and operating the circuits based on their received control signal

[0001] This application claims the benefit of U.S. Provisional Patent Application Ser. No. 60/290,551 filed on May 11, 2001 in the names of Khalid Azim, Venkat Yadavalli and Keven Hui.

[0002] The present invention relates to operation of a multiple-circuit system. More particularly, the present invention relates to synchronization of a multiple-circuit system and controlling data skew among the multiple circuits.

[0003] FIG. 1 illustrates a data transmitting portion of a serial input/output (I/O) circuit 10. In source synchronous interfaces, a clock signal (TC) is transmitted along with a group of parallel data from a data source 12. This clock signal is used to latch the group of data in a circuit 14 that receives the data. The receiving circuit 14 typically includes a serializer, for example, such as a quad serializer receiving 20-bit parallel data. A quad serializer performs parallel-to-serial conversion and outputs the serial data to four serial data lane links, as shown in FIG. 1. The receiving circuit 14 which references to the clock signal forwarded with a specific group of data is typically implemented in a single circuitry unit or chip.

[0004] As the bus width become wider to achieve faster data rate, more data link connections are required. For example, in channel-based point-to-point connections such as an Infiniband application, a word is encoded and sent out through up to 12 different channels/lanes, and thus it is required to provide a 12-lane high speed serial link (current chip implementations handle up to 4 lanes which provides up to 4×3.125 Gigabits/sec for Ethernet). In order to provide such a multiple data lane connection, a plurality of I/O circuits, typically transceivers, should be ganged together.

[0005] The specification of multi-link connections typically includes requirements for acceptable skew at the serial data outputs across multiple I/O circuits. For example, in the Infinivand application described above, the delay skew across all 12 lanes at the serial outputs has to be 500 picosecond (ps) or less according to the current electrical specification. During the serialization, the I/O circuits are also required to align all incoming parallel data to within the same byte/cycle.

[0006] However, when two or more I/O circuits or chips which are referencing to different forwarded clocks are bundled, the data coming out of the different circuits/chips are not necessarily in sync. Due to the timing skew caused by a number of variations including process, temperatures, voltages, and board traces skew, synchronization of these output data across the multiple circuits/chips becomes a difficult task. Such a multiple-circuit system using different local clocks also puts a significant limitation on chips placement and board routing in order to minimize the skew across all different clocks. Thus, ganging a plurality of I/O circuit/chips without any special circuit techniques will result in large and out-of-spec data skews between different circuits, making the system unusable.

[0007] Accordingly, it would be desirable to provide a scheme for synchronizing a multiple-circuit system such as a system including a plurality of I/O circuitry units, and for controlling data skew across multiple circuits.

[0008] A mechanism for synchronizing a multiple-circuit system, includes (a) selecting a master circuit from a plurality of circuits, the remaining circuits including at least one slave circuit, (b) receiving, at each of the plurality of circuits, input data and a local clock signal associated with the input data, (d) generating at least one control signal at the master circuit using the local clock signal of the master circuit, (e) outputting the control signal from the master circuit, (f) forwarding the control signal to the slave circuit(s), (g) looping back the control signal to the master circuit, (h) processing the input data at the slave circuit(s) using the forwarded control signal, (i) processing the input data at the master circuit using the looped-back control signal, and (j) outputting the processed data from each of the plurality of circuits.

[0020] Embodiments of the present invention are described herein in the context of a method and apparatus for synchronizing a multi-circuit system. Those of ordinary skill in the art will realize that the following detailed description of the present invention is illustrative only and is not intended to be in any way limiting. Other embodiments of the present invention will readily suggest themselves to such skilled persons having the benefit of this disclosure. Reference will now be made in detail to implementations of the present invention as illustrated in the accompanying drawings. The same reference indicators will be used throughout the drawings and the following detailed description to refer to the same or like parts.

[0021] In the interest of clarity, not all of the routine features of the implementations described herein are shown and described. It will, of course, be appreciated that in the development of any such actual implementation, numerous implementation-specific decisions must be made in order to achieve the developer's specific goals, such as compliance with application- and business-related constraints, and that these specific goals will vary from one implementation to another and from one developer to another. Moreover, it will be appreciated that such a development effort might be complex and time-consuming, but would nevertheless be a routine undertaking of engineering for those of ordinary skill in the art having the benefit of this disclosure.

[0022] FIG. 2 schematically illustrates a multiple-circuit system 20 in accordance with an embodiment of the present invention. As shown in FIG. 2, the multiple-circuit system 20 includes a master circuit 22 and at least one slave circuit 24. The master circuit 22 includes a control signal generator 30, a signal processing portion 32, a synchronous input port 34, a control signal output port 36, and a control signal input port 38. Input data (Di) and a local clock signal (TCXi) associated with the input data sent from a data source (not shown) are received at the synchronous input port 34 and input to the signal processing portion 32. The local clock signal is also input to the control signal generator 30. The control signal generator 30 generates at least one control signal using the local clock signal (TCXi).

[0023] Each slave circuit 24 includes a signal processing portion 42, a synchronous input port 44, and a control signal input port 48. Input data (Dm) and a local clock signal (TCXm) associate with the input data sent from a data source (not shown) are received at the synchronous input port 44 and input to the signal processing portion 42.

[0024] The control signal generated in the master circuit 22 is output via the control signal output port 46 onto a control signal bus 26. The control signal bus 26 is coupled from the control signal output port 36 of the master circuit 22 to the control signal input port 48 of each slave circuit 24, and to the control signal input port 38 of the master circuit 22.

[0025] The output control signal is supplied to each slave circuit 24 and received at the control signal input port 48. The signal processing portion 42 of the each slave circuit 24 processes the input data (Dm) in accordance with the control signal. The processed data is output via at least one output port 49 of the slave circuit 24.

[0026] The control signal is also looped back to the control signal input port 38 of the master circuit 22, and the signal processing portion 32 processes the input data (Di) in accordance with the looped-back control signal. The processed data is output via at least one output port 39 to a corresponding data lane. It should be noted that the control signal is not input to the signal processing portion 32 directly from the control signal generator 30, but is output from the master circuit 22 and then distributed thereto via the control signal bus 26. Thus, the master circuit 22 sees the delay in the control signal caused by the process, voltage, and temperature (PVT) variations internal to the master circuit as well as external delay such as traveling the bus.

[0027] Thus, by generating all necessary control signals at the master circuit 22 and forwarding them to the slave circuit(s) 24 and to the master circuit 22 itself, all the circuits of the system are operated by the same control signal including substantially the same delay. This eliminates variations caused by using a different local clock signal in each circuit, and also eliminates most of the delay factors due to variations internal to each circuit, which are random in nature and difficult to control.

[0028] According to one embodiment of the present invention, the master circuit 22 may further include a global control signal input port 31 to receive a global control signal, as shown in FIG. 2. The control signal generator 30 generate at least one control signal based on the global control signal. In addition, the master circuit 22 and the slave circuit(s) 24 may have the identical circuit structure, i.e., the slave circuit(s) 24 may include a portion capable of generating a control signal. In such a case, however, such a portion may be disabled (or not used) so that the slave circuit(s) 24 is operated solely by the control signal supplied from the master circuit 22.

[0029] FIG. 3 schematically illustrates a multiple-circuit system 50 in accordance with an embodiment of the present invention, where the input data includes a group of parallel data, and the signal processing in each circuit involves a parallel-serial conversion. The multiple-circuit system 50 includes at least two circuits 52 (for example, N circuits as shown), one of which will be the master circuit and the remaining is at least one slave circuit. All the circuits 52 in the system may have the same circuit structure, and one circuit is selected from among the plurality of circuits 52 to be designated as the master circuit.

[0030] Each circuit 54 receives input data (Dj) and an associated local clock (TCXj) from the corresponding data source 56 (0≦j≦N). Each circuit includes a serializer 54 to convert the group of parallel data into a group of serial data, and outputs the serial data onto M data lanes via the corresponding serial output ports (1≦M). Thus, the system 50 provides N×M serial data link connections. The multi-circuit system 50 further includes a clock source 58 to supply a common reference clock signal (Refclk) to each data source 56 and each circuit 52.

[0031] Since each circuit 52 receives a separate local clock TCXj, there will be some phase skew between these clocks due to board trace, process, voltage, temperature, and some other variations. If there is no realignment scheme, these parallel data can be latched in different cycle/byte when they are clocked into the serializer 54 by the common clock (Refclk). When such a misalignment happens, the serial data output across the multiple circuits can be off by one whole cycle.

[0032] One embodiment of the present invention provides a scheme to synchronize the serial outputs across multiple circuits/chips within a predetermined time period. The scheme includes two main parts: aligning all parallel input data across a plurality of circuits within one clock cycle; and managing the skew of the serial outputs within a predetermined time period across the multiple circuits. The synchronizing scheme is explained in detail using the following example.

[0033] FIG. 4 schematically illustrates a multiple-circuit system 60 according to a specific embodiment of the present invention, where the serializer is a quad serializer, with a detailed structure of each circuit. A typical application of this embodiment of the present invention may be a data transmission portion of a transceiver, and each circuit may be implemented in a respective core chip. However, it should be noted that the present invention is not limited to a specific application, but applicable to synchronization for any multiple-circuit system or multiple-chip system, including a multi-circuit system 50 shown in FIG. 3.

[0034] Referring to FIG. 4, the multiple-circuit system 60 includes a plurality of circuits, and one of the circuits is selected as the master circuit 62. Any circuit can be the master circuit 62. The remaining at least one circuits is at least one slave circuit 64. Each circuit of the system 60 has a synchronous parallel input port for receiving a group of parallel data and a local clock signal (TCX) associated with the group of parallel data (the data input is not shown in FIG. 4 for simplicity).

[0035] As shown in FIG. 4, the master circuit 62 includes a global control signal input port 70 to receive a global control signal RSTi (for example, Reset/enable signal), a first synchronizer 72 for clocking the global control signal (RSTi) with the local clock signal (TCXi) so as to generate a first control signal (Sync_—RST). The first control signal (Sync_—RST) is output from a first control signal output 74 (as a signal Sync_—REST_—out), and looped back to the master circuit 62 itself and forwarded to each slave circuit 64.

[0036] The master circuit 62 further includes a first control signal input 76 for receiving the first control signal looped back from the first control signal output port 74 (as a signal Sync_—RST_—in), a first enable signal generator 78, a data storage unit 80 (for example, a FIFO storage), a clock input port 82 to receive a common clock signal (RefClk), a second synchronizer 84, a second control signal output 86, a second control signal input 88, and a second enable signal generator 90. The first enable signal generator 78 clocks the first control signal (Sync_—RST_—in) with the local clock signal (TCXi) so as to produce a first enable signal (wrt_—En). The parallel input data is written into the data storage unit 80 in accordance with the first enable signal (wrt_—En).

[0037] In each slave circuit 64, which has the same circuit structure as the master circuit 62, the forwarded first control signal (Sync_—RST_—in) is received at a first control signal input 76′. A first enable signal generator 78′ clocks the first control signal with the local clock signal (TCXn) so as to produce a first enable signal (wrt_—En). The parallel input data is written into the data storage unit 80′ in accordance with the first enable signal (wrt_—En) in the same manner as that in the master circuit 62.

[0038] At the master circuit 62, the first enable signal (wrt_—En) is also supplied to the second synchronizer 84, which clocks the first enable signal (wrt_—En) with the common clock signal (RefClk) so as to generate a second control signal (rdEn). The second control signal is output from the second control signal output 86 (as a signal rdEn_—out), and forwarded to the master circuit 62 itself and to each slave circuit 64.

[0039] At the master circuit 62, the second control signal is received at the second control signal input 88 (as the looped back signal rden_—In), and the second enable signal generator 90 clocks the second control signal with the common clock signal (RefClk) so as to produce a second enable signal (Sync_—rdEn). The input data written to and stored in the data storage unit 80 is read therefrom in accordance with the second enable signal (Sync_—rdEn) and supplied to a serializer 92.

[0040] Similarly, in each slave circuit 64, the forwarded second control signal (rdEn_—In) is received at a second control signal input 76′. A second enable signal generator 90′ clocks the second control signal with the common clock signal (RefClk) received at a clock signal input port 82′ so as to produce a second enable signal (Sync_—rdEn). The input data written to and stored in the data storage unit 80′ is read therefrom in accordance with the second enable signal (Sync_—rdEn) and supplied to a serializer 92′ in the same manner as that in the master circuit 62.

[0041] It should be noted that although each slave circuit 64 has the same structure as the master circuit 62, the portions for generating the control signals (such as the first and second synchronizers) in the slave circuit 64 is disabled and/or not used in operating the circuit. For example, as shown in FIG. 4, the global control signal input port 70′ of the slave circuit may be grounded.

[0042] FIG. 5 illustrates timing and logic waveforms of the synchronization scheme in accordance with an embodiment of the present invention. The wave forms represent, from the top, the local clock (TCX-2) in the master circuit (generated by dividing the input local clock signal TCXi by 2), the common clock signal (Refclk), the global control signal (RSTi) input to the master circuit, the fist control signal generated in the master circuit (Sync_—RST), the first control signal supplied to the master (and slave circuits) (Sync_—RST_—In), a initialization signal for initializing the FIFO (Clear), the first enable signal (Wrt_—En) in each circuits, the second control signal generated in the master circuit (rdEn), the second control signal supplied to each circuit (rdEn_—In), and the second enable signal (Sync_—reEn) in each circuit. The two waveforms at the bottom represent the write operation (Write) and read operation (Read) to the FIFO.

[0043] As shown in FIG. 5, in the master circuit, the first control signal (Sync_—RST) is generated at the first rising edge of the local clock signal (TCX-2) after the global control signal (RSTi) goes low. The first control signal is distributed to each circuit (master, slave) as the input control signal (Sync_—RST_—In) with a delay (Tdlay). Then, the first rising edge of the local clock signal (TCX-2) after the first control signal (Sync_—RST_—In) goes low drives the first enable signal (Wrt_—En) high (a write enable flag on), which triggers the write operation in the next cycle. Similarly, at the first rising edge of the common clock signal (Refclk) after the first enable signal (Wrt_—En) goes high, the second control signal (rden) is generated (goes high). The second control signal is distributed to each circuit as the second input control signal (rden_—In) with a delay (Tdlay). Then, the first rising edge of the common clock signal (Refclk) after the second control signal (rden_—In) goes high drives the second enable signal (Sync rden) high (a read enable pointer on), which triggers the read operation in the next cycle.

[0044] “Tdlay” denotes a delay from the control signals generated in the master circuit to the control signals supplied to each circuit: the delay from the first control signal generated in the master circuit (Sync_—RST) to the first control signal supplied to each circuit (Sync_—RST_—In); and the delay from the second control signal generated in the master circuit (rden) to that supplied to each circuit (rdEn_—In). The delay includes propagation delay of 10 buffers, all the internal cells, packages and board trace flight time. The amount of the delay due to variations internal to the master circuit is substantially identical for each circuit, as discussed above. In addition, the master circuit may be selected so as to minimize the difference in signal path lengths from the master circuit to each slave circuit and to the master circuit itself. Thus, each circuit receives the same control signals at substantially the same timing or with a minimum variation. Since in each circuit the first enable signal is generated using the same first control signal (Sync_—RST_—In), and the second enable signal is generated using the same second control signal (rdEn_—In), instead of generating thees control signals in each circuit, statistical and/or uncontrollable inter-circuit variations due to process, temperature, voltage, etc. are minimized.

[0045] FIG. 6 is a timing diagram illustrating an example of calculating timing budget and margin for the forwarded control signals. Generally, setup and hold margins can be calculated in the following ways.

[0046] For a setup margin,
Setup Margin=Period−2×tsktcx−tsetup−(tinbuf)_min−(tcells+toutbuf+tpkg+tflgt+tpkg)_max,
where Period is the time period for one clock cycle. It should be noted that all clock signals have the same frequency. The above equation is the worst case of setup margin. Master TCX-2 clock is later than slave TCX-2 clock by maximum “tsktcx,” thus master chip is in worst case corner (WNWP, 125° C., Vdd−10%) and slave chips are in best case corner (SNSP, 0° C., Vdd+10%).

[0047] For a hold margin,
Hold Margin=(tcells+toutbuf+tpkg+tflgt+tpkg)_min+(tinbuf)_max−tsktcx−thold
The above equation is the worst case of hold margin. Master TCX-2 clock is later than slave TCX-2 clock by maximum “tskcx” thus master chip is in best case corner (SNSP, 0° C., Vdd+10%) and slave chips are in worst case corner (WNWP, 125° C., Vdd−10%).

[0048] It should be noted that these numbers are used to calculate maximum margins for the control signals only, and they do not represent the timing budget and margin for the maximum lane to lane skew itself.

[0049] Once byte alignment is performed by synchronizing the control signals, as described above, skew at the serial outputs can be controlled to certain range. From the output of the FIFO to the serial output at the pin level, there are several parts that contribute to the total skew. Placement error of the common clock signal (Refclk) or skew of the common clock signal (Refclk) at the serializer's multiplexer level between all the data lanes is one contributor. This skew can depend on process, voltage, and temperature (PVT) variation of insertion delay of all the cells on the clock path and clock generation scheme. Package skew and PVT variation of insertion delay of all the cells on the data path will also contribute to the total data skew.

[0050] FIG. 7 shows a table containing break-down parameters of the contributors described above, in the case where each circuit is implemented as a serializer-deserializer (SerDes) chip with the timing budget of maximum 500 ps. These parameters, however, can be grouped differently or broken down further depending on the clocking and serialization scheme used, and the specific numbers are used for illustrative purpose only, and by no means limit the present invention.

[0051] FIG. 8 is a process flow diagram illustrating a method for synchronizing a multiple-circuit system in accordance with an embodiment of the present invention. The system may be a multiple circuit system 20, or system 50, as described above.

[0052] First, a master circuit is selected from a plurality of circuits, the remaining circuits being at least one slave circuit (100). At least one control signal is generated at the master circuit using the local clock signal of the master circuit (110). The control signal(s) is forwarded to each slave circuit (120), and at the same time looped back to the master circuit (130). The master circuit is operated using the looped-back control signal(s) (140), and the slave circuit(s) are operated using the forwarded control signal(s) (150). By property selecting parameters, operation timings of the circuits (master circuit and at least one slave circuit) are aligned within one clock cycle. The data processed by the operation is output (160), and skew across the circuits is controlled to be within a predetermined time period (170).

[0053] FIG. 9 is a process flow diagram illustrating a method for synchronizing a multiple-circuit system in accordance with an embodiment of the present invention. The system may be a multiple circuit system 60 as described above.

[0054] First, a master circuit is selected from a plurality of circuits, the remaining circuits being at least one slave circuit (200). A global control signal is received at the master circuit (202), and a first control signal is generated by clocking the global control signal with the local clock signal of the master circuit (204). The first control signal is output from the master circuit and distributed to each circuit of the system. That is, the first control signal is looped back to the master circuit (206) and forwarded to the slave circuit(s) (208).

[0055] At the master circuit, the distributed first control signal is clocked with the local clock signal (210) so as to produce a first enable signal (212). A first operation is performed onto the input data in accordance with the first enable signal (214). Similarly, at the slave circuit(s), the forwarded first control signal is clocked with the local clock signal (216) so as to produce a first enable signal (218). A first operation is performed onto the input data in accordance with the first enable signal (220). The first operation may be writing the data into a FIFO.

[0056] A common clock is received at the master circuit (222), and a second control signal is generated using the first enable signal and the common clock signal (224). The second control signal is looped back to the master circuit (226) and also forwarded to the slave circuit(s) (228).

[0057] At the master circuit, the looped-back second control signal is clocked with the common clock signal (230) so as to produce a second enable signal (232). A second operation is performed onto the input data in accordance with the second enable signal (234). Similarly, at the slave circuit(s), the forwarded second control signal is clocked with the common clock signal (236) so as to produce a second enable signal (238). A second operation is performed onto the input data in accordance with the second enable signal (240) in the same manner as in the master circuit. The second operation may be reading the data from the FIFO.

[0058] As described above, the multi-circuits synchronization scheme according to an embodiment of the present invention uses a master circuit and slave circuit(s). One of the circuits can be used as a master circuit which receives the global control signal. The master circuit generates all necessary control signals using its own local clocks, and the control signals are forwarded to all of the slave circuits. By doing this, all slave circuits receive the same control signals referencing to the local clocks of the master circuit thus eliminating multiple clock domains. Furthermore, the control signals are also looped back to the master circuit itself. By receiving its own control signals, the master circuit can also be seen as one of the slave circuits when it comes to calculating the timing budget.

[0059] By using this synchronizing technique of the present invention, the tasks of byte aligning and skew managing across multiple circuits/chips are significantly simplified. As described and shown in the above embodiments, once the bite alignment for input data is accomplished, there is a great flexibility of choosing types of cells, IO buffer, as well as internal cells to manage skew across the circuits. Chips placement and board routing also become less challenging. The synchronization scheme is also independent of process technology used and numbers of chips or lanes need to be synchronized.

[0060] While embodiments and applications of this invention have been shown and described, it would be apparent to those skilled in the art having the benefit of this disclosure that many more modifications than mentioned above are possible without departing from the inventive concepts herein. The invention, therefore, is not to be restricted except in the spirit of the appended claims.