Method of secondary to secondary data transfer with mirroring

[0002] 1. Technical Field

[0003] The present invention relates generally to transferring data in data processing systems and in particular to transferring data from a disk array to a host in a data processing system. Still more particularly, the present invention relates to transferring data from a peripheral interface for a disk array in a data processing system into a main data buffer and out to a host interface with a simple suspend/throttle control scheme.

[0004] 2. Description of the Related Art

[0005] Redundant arrays of inexpensive disks (RAID) such as small SCSI hard disks, have been found to be suitable alternatives to large capacity, single magnetic media. Such arrays appear to the host system as a single media, but provide a very high data transfer rate through a technique called “striping.” The arrays also provide improved reliability, scalability, data availability and power consumption over large magnetic disks.

[0006] Control of the array for READ, WRITE, and other operations may be performed by the host, but is typically effected by a controller. Controllers generally utilize a data buffer between the host and the array, transferring data from the array to the data buffer and from the data buffer to the host.

[0007] Current implementations of buffered controllers support non-cached disk READ operations by performing high-speed data movement between a peripheral interface and a main data buffer and between a main data buffer and a host interface. The total data transfer is generally accomplished in two distinct operations: the inbound transfer places data read from the peripheral interface of the array into the main data buffer, while the outbound transfer removes data from the main data buffer and forwards it to the host interface.

[0008] It would be advantageous to transfer data from the peripheral interface of the array into the data buffer and out to the host interface in a single operation, thereby achieving a higher effective data bandwidth as well as improving the host read data response time.

[0009] Data transfers from the peripheral interface of a disk array to a data buffer are snooped to determine if the starting address of a data transfer matches an entry in a list of starting addresses for requested data. If a match is identified, third party transfer is initiated and the data is simultaneously transferred to the host interface of the host system. The resulting data bandwidth is increased. A throttling/suspension mechanism can temporarily or indefinitely hold up actual data movement into the data buffer to allow for temporary buffering and interface speed matching as data is transferred to the host interface.

[0010] The novel features believed characteristic of the invention are set forth in the appended claims. The invention itself, however, as well as a preferred mode of use, further objectives and advantages thereof, will best be understood by reference to the following detailed description of an illustrative embodiment when read in conjunction with the accompanying drawings, wherein:

[0011] FIG. 1 depicts a data processing system employing a disk array and a controller in accordance with a preferred embodiment of the present invention;

[0012] FIG. 2 is a block diagram illustrating further details of the main data buffer control within the controller depicted in FIG. 1;

[0013] FIG. 3 depicts further details of the peripheral interface control within the controller depicted in FIG. 1;

[0014] FIG. 4 is a block diagram illustrating further details of the host interface control within the controller depicted in FIG. 1;

[0015] FIG. 5 depicts a high level flowchart for a process in the snooping mechanism for determining if a requested data transfer is occurring in accordance with a preferred embodiment of the present invention;

[0016] FIG. 6 is a high level flowchart for a process of throttling/suspending data transfer into the controller main data buffer in accordance with a preferred embodiment of the present invention; and

[0017] FIGS. 7A-7C depict timing diagrams demonstrating the effect of a throttling/suspension mechanism on data transfer into the data buffer.

[0018] With reference now to the figures, and in particular with reference to FIG. 1, a block diagram of a data processing system employing a disk array and a controller in accordance with a preferred embodiment of the present invention is depicted. Controller 100 comprises controller CPU 102 linked to primary PCI bus 104 by PCI main bridge 106. Primary PCI bus 104 also connects main data buffer control 108 to both peripheral interface control 110 and host interface control 112. Main data buffer control 108 is connected to, and controls data transfer to and from, data buffer 114. Main data buffer control 108 is also linked to both peripheral interface control 110 and host interface control 112 by buffer data bus 116 and main control bus 118.

[0019] In operation, peripheral interface control 110 controls the transfer of data from peripheral interface 120 to secondary PCI bus 122, which is connected, for example, by a plurality of SCSI interfaces 124 to the storage media of the disk array. Data read from the storage media is transferred by peripheral interface control 110 into data buffer 114 via buffer data bus 116 and main data buffer control 108. The data may be transferred out of data buffer 114 to host interface control 112 via main data buffer control 108 and buffer data bus 116. Buffer data bus 116 is used to transfer the data itself while main control bus 118 is used to control the transfer and for signaling data request/acknowledge.

[0020] Host interface control 112 controls the transfer of data through host interface 126 to secondary PCI bus 128. Secondary PCI bus 128 is connected to the host system by an appropriate interface such as PCI interface 130, SCSI interface 132, or fiber channel interface 134.

[0021] As described below, host interface control 112 contains a snooping mechanism to detect transfers of requested data from peripheral interface control 110 to main data buffer control 108. Host interface control 112 also contains a throttling/suspension mechanism, which employs throttling control bus 136 between host interface control 112 and main data buffer control 108 to act as a pacing agent for actual data transfer.

[0022] Although depicted separately for clarity in explaining the present invention, those skilled in the art will recognize that buffer data bus 116, main control bus 118, and throttle control bus 136 may employ a single set of conductors and/or pins and that main data buffer control 108, data buffer 114, peripheral interface control 110, and host interface control 112 may be implemented as a single integrated circuit.

[0023] While the exemplary embodiment described herein employs a PCI bus, those skilled in the art will recognize that the principles disclosed are applicable to any data transfer from a source to a destination which employs intermediate buffering, regardless of the bus architecture selected.

[0024] Referring to FIG. 2, further details of the main data buffer control within the controller depicted in FIG. 1 are illustrated in a block diagram. Main data buffer control 108 regulates data movement from buffer data bus 116 to the actual data buffer. Buffer timing and control logic unit 200, connected to main control bus 118 and throttling control bus 136, contains sequential state machine logic controlling both the timing and assertion of signals to the specific buffer devices and the transfer of data from buffer data bus interface unit 202 to data checking and data path unit 204. Buffer timing and control logic unit 200 handles buffer refresh requirements, timing requirements for various cycle control signals, and address/data transfer requirements relative to the buffer devices in the main data buffer. These tasks are common to most memory control logic such as that used for DRAMs and are not unique to the present invention.

[0025] Data checking and data path unit 204 contains registers and steering logic necessary to support movement of data from buffer data bus interface 202 to the data buffer. Control signals for these registers and steering logic are received from buffer timing and control logic unit 200 via signal lines 206. Data is forwarded to the data buffer from data checking and data path unit 204 via data lines 212.

[0026] Buffer data bus interface 202 contains hardware providing an immediate interface of control and data bits from buffer data bus 116. This hardware typically includes at least one level of buffering or temporary storage of data bits for transfer from buffer data bus 116 to data checking and data path unit 204. Controls signals for this hardware are received from buffer timing and control logic unit 200 via signal lines 208, while the actual data transfer to data checking and data path unit 204 flows via data lines 210.

[0027] With the exception of the input for throttle control bus 136 and attendant circuitry for responding appropriately to signals on throttle control bus 136, main data buffer control 108 may be constructed in accordance with known embodiments of disk array controllers.

[0028] With reference now to FIG. 3, further details of the peripheral interface control within the controller depicted in FIG. 1 are depicted. Peripheral interface control 110 controls transfer of data from peripheral interface 120 to buffer data bus 116. Peripheral bus interface unit 300 contains hardware providing an immediate interface of control and data bits from peripheral interface 120. This hardware typically includes at least one level of buffering of data bits for transfer to peripheral FIFO buffer and controls 302. Peripheral bus interface unit 300 receives control signals from peripheral interface control unit 304 via signal lines 306 and forwards data to peripheral FIFO buffer and controls 302 via data lines 308.

[0029] Peripheral interface control unit 304 contains sequential state machine logic controlling data movement between peripheral interface 120 into peripheral bus interface unit 300 as well as data transfer from peripheral bus interface unit 300 to peripheral FIFO buffer and controls 302.

[0030] Peripheral FIFO buffer and controls 302 contains a FIFO (first-in, first-out) buffer and data path steering logic necessary to support efficient movement of input data from peripheral bus interface unit 300 to buffer bus interface unit 310. Control signals for this buffer and steering logic are received from peripheral bus interface unit 300 via signal line 312 and from buffer interface control unit 314 via signal line 316. Data from peripheral FIFO buffer and controls 302 is forwarded to buffer bus interface unit 310 via data lines 318.

[0031] Buffer interface control unit 314 contains sequential state machine logic controlling movement of data from peripheral FIFO buffer and controls 302 into buffer bus interface unit 310 and from buffer bus interface unit 310 to buffer data bus 116. Buffer interface control unit 314 sends control signals to buffer bus interface unit 310 via signal lines 320. Buffer bus interface unit 310 provides an immediate interface for control and data bits being forwarded to buffer data bus 116 and typically includes a minimum of one level of buffering of data bits for transfer from peripheral FIFO buffer and controls 302 to buffer data bus 116.

[0032] The components of peripheral interface control 110 may be constructed in accordance with known embodiments of disk array controllers. The tasks described are common to data transfers from disk arrays and are not unique to the present invention.

[0033] Referring to FIG. 4, further details of the host interface control within the controller depicted in FIG. 1 are illustrated in a block diagram. Host interface control 112 controls data transfer from buffer data bus 116 to host interface 126. Buffer bus interface unit 400 contains hardware providing an immediate interface controlling data movement from buffer data bus 116 to host FIFO buffer and controls 402. This hardware typically includes a minimum of one level of buffering of data bits transferred from buffer data bus 116 to host FIFO buffer and controls 402. Control signals for this hardware are received from buffer interface control unit 404 via signal lines 406, while data is transferred from buffer data bus 116 to host FIFO buffer and controls 402 via data line 408.

[0034] Host FIFO buffer and controls 402 contains a FIFO buffer and data path steering logic necessary to support efficient movement of input data from buffer bus interface unit 400 to host bus interface unit 410. Controls signals for the buffer and steering logic are received from buffer interface control unit 404 via signal line 412 and from host interface control unit 414 via signal line 416. Data is transferred from host FIFO buffer and controls 402 to host bus interface unit 410 via data lines 418.

[0035] Host interface control unit 414 contains sequential state machine logic controlling movement of data from host FIFO buffer and controls 402 to host bus interface unit 410 and from host bus interface unit 410 to host interface 126. Control signals are sent from host interface control unit 414 to host bus interface unit 410 via signal lines 420. Host bus interface unit 410 contains hardware providing an immediate interface of control and data bits to host interface 126. This hardware typically includes at least one level of buffering of data bits transferred from host FIFO buffer and controls 402 to host interface 126.

[0036] With the exception of snooping circuit and queue control logic 422, throttle and suspension logic 424, and modifications to buffer interface control unit 404, as described below, the components of host interface control 112 may be constructed in accordance with known embodiments of disk array controllers. The tasks described are common to data transfers from disk arrays and are not unique to the present invention.

[0037] Buffer interface control unit 404 contains sequential state machine logic which, in conjunction with snooping circuit and queue control logic 422 and throttle and suspension logic 424, controls movement of data from buffer data bus 116 into buffer bus interface unit 400 as well as movement of data from buffer bus interface unit 400 into host FIFO buffer and controls 402. Buffer interface control unit 404 also manages throttle control bus 136.

[0038] Snooping circuit and queue control logic 422 contains hardware enabling host interface control 112 to participate in data transfers occurring on buffer data bus 116. Snooping circuit and queue control logic 422 includes a hardware queueing mechanism containing a list of potential starting addresses for data transfers on buffer data bus 116. This list of starting addresses can be polled sequentially to allow an effective comparison of all actual data transfer starting addresses to all starting address entries in the queue.

[0039] If snooping circuit and queue control logic 422 detects a match between an actual data transfer starting address and a starting address in the queue, it signals buffer interface control unit 404 to initiate transfer of the data on buffer data bus 116 into buffer bus interface unit 400. Thus requested data may be obtained for host interface 126 directly from the peripheral interface control via buffer data bus 116, without first storing the data in the data buffer and subsequently transferring the data via host interface control 112 to host interface 126.

[0040] Snooping circuit and queue control logic 422 also includes a device enabling throttle and suspension logic 424 to initiate monitoring and passive control of the current handshake of data transferred from the peripheral interface control to buffer data bus 116 and received by the main data buffer control. Snooping circuit and queue control logic 422 also contains a mechanism determining when a continuation or suspension of a previously queued data transfer is required. This requires some data transfer length detection or counter logic as well as holding registers for suspended or pending transfers considered to be in progress but not currently transferring on buffer data bus 116.

[0041] Snooping circuit and queue control logic 422 provides an input to buffer interface control unit 404 via signal line 426 and receives an input from buffer interface control unit 404 via signal line 428. Snooping circuit and queue control logic 422 receives addresses from the CPU on primary PCI bus 104.

[0042] Throttle and suspension logic 424 contains logic for determining whether a current data transfer on buffer data bus 116 should be throttled or suspended. Throttle and suspension logic 424 provides an input to buffer interface control unit 404 via signal line 430 for use in managing throttle control bus 136. Throttle and suspension logic 424 receives an input from buffer interface control unit 404 via signal line 432 used in determining if throttling or suspension is required.

[0043] With reference now to FIG. 5, a high level flowchart is depicted for a process in the snooping mechanism for determining if a requested data transfer is occurring in accordance with a preferred embodiment of the present invention. The process is preferably performed by snooping control circuitry included in the host interface control of a disk array controller. The snooping control circuity includes a queue listing possible data transfer starting addresses, and in particular the starting addresses of requested data transfers.

[0044] The process begins at step 500, which depicts the beginning of a data transfer into the main data buffer via the buffer data bus. The process passes next to step 502, which illustrates a comparison of the starting address of the data transfer on the buffer data bus to the list of starting addresses in the queue, and then to step 504, which depicts a determination of whether the starting address of the data transfer matched a starting address in the queue. If not, the process proceeds to step 506, which illustrates the process becoming idle until the beginning of the next data transfer on the buffer data bus. If so, however, the process proceeds instead to step 508, which depicts initiation of third party or “fly-by” data transfer and begin receiving the data as it is transferred from the peripheral interface control to the main data buffer control and data buffer. The process then passes to step 510, which depicts monitoring the data transfer to determine is throttling or suspension is required to allow temporary buffering and interface speed matching as data is transferred to the host interface.

[0045] Referring to FIG. 6, a high level flowchart for a process of throttling/suspending data transfer into the controller main data buffer in accordance with a preferred embodiment of the present invention is illustrated. The throttling/suspension is preferably effected by a signal from the host interface control to the main data buffer control on the throttle control bus is response to a determination by the throttling mechanism in the host interface control that throttling/suspension is required.

[0046] The process begins at step 600, which depicts the beginning of a third party data transfer into the host interface control of data being transferred from the peripheral interface control to the main data buffer. The process then passes to step 602, which illustrates a determination of whether throttling or suspension is necessary due to the host interface buffer filling to a level and/or at a rate which indicates the host system requires additional clock cycles to receive the data already transferred from the peripheral interface control to the main data buffer control. If the host interface buffer is filling, the process proceeds to step 604, which depicts signaling a throttle of the data transfer on the throttle control bus, causing the main data buffer control to assert a signal disabling the data request. The process next passes to step 606, which illustrates a determination of whether the host interface buffer is clearing in response to transfer of data to the host system. If not, the process loops back to step 606 continually until the host interface buffer is sufficiently cleared. If so, however, the process proceeds instead to step 608, described below.

[0047] Referring again to step 602, if throttling or suspension is not required, the process passes to step 608, which depicts a determination of whether the data transfer is complete. If not, the process loops back to step 602 to again determine if throttling or suspension is required. if so, however, the process proceeds to step 610, which illustrates the process becoming idle until the next third party data transfer to the host interface buffer begins.

[0048] With reference now to FIGS. 7A-7C, timing diagrams demonstrating the effect of a throttling/suspension mechanism on data transfer into the data buffer are depicted. The “valid” notation shown for the ADDRESS signal in each timing diagram indicates that a starting address matching an entry in the queue has been detected. Implementation of the START and END signals is a matter of design choice since these signals merely indicate a bounding condition for a given data transfer. The actual points of data transfer are indicated by the numbers in circles. In the depicted example, data is transferred only on the rising clock edge if DATA ACKNOWLEDGE is ON (high). DATA ACKNOWLEDGE can be turned ON for the next rising clock edge if, for the current rising clock edge, DATA REQUEST is ON (high) and DISABLE DATA REQUEST is OFF (low).

[0049] FIG. 7A depicts a timing diagram for an unthrottled, unsuspended data transfer. FIG. 7B depicts a timing diagram for a data transfer which is throttled for two clock cycles and then resumed. FIG. 7C depicts a timing diagram for a data transfer which was suspended (or indefinitely throttled) before any data was actually transferred.

[0050] Controlling data transfers between a host system and a disk array in accordance with the present invention results in efficient data transfer and improved data bandwidth. The present invention also improves the host read data response time.

[0051] The description of the preferred embodiment of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or limit the invention in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiment was chosen and described in order to best explain the principles of the invention and the practical application to enable others of ordinary skill in the art to understand the invention for various embodiments with various modifications as are suited to the particular use contemplated.