Method, apparatus and program product for low latency I/O adapter queuing in a computer system

[0001] The invention relates to a method and apparatus for operating an I/O adapter in a computer system comprising at least one central processing unit (CPU) and a main memory which are connected through a connect network to at least one I/O adapter.

[0002] In data processing and communication systems input and output operations represent a substantial share of the workload to be performed. Such I/O operations are time consuming and interrupt the regular task processing in the central processing units. To relief the CPUs from I/O operations special processors are used, designated as I/O adapters, which are designed to support one or more I/O devices such as disk and tape storages, communication or network devices, printers, scanners, etc. Each I/O adapter is connected through a data channel or bus to the CPU and the main memory of a computer system. The data transfer between these elements have to be organized in a manner which minimizes the interventions in the data processing flow of the CPU and simultaneously provides an efficient execution of I/O commands including read and write accesses to the maim memory.

[0003] U.S. Pat. No. 5,983,292 discloses an I/O system including a processor complex having a CPU and a main memory and being connected by a bus to an I/O adapter which serves a number of I/O devices. The system comprises message queues in the main memory which are pointing to next available memory locations where the CPU may store messages which have to be transferred via the interconnecting bus to the I/O adapter. The I/O adapter have downstream and upstream tracking registers and a message processor for tracking the locations in the main memory from which messages have be transferred to the I/O adapter or to which messages have be transferred from the I/O adapter to the processor complex. By this system the efficiency of the message transfer between the processor complex and the I/O adapter can be improved since the use of message queues reduce the CPU time required to process I/O interrupts. On the other side, the known system requires a plurality of time consuming main memory accesses.

[0004] U.S. Pat. No. 5,548,735 discloses a system for asynchronously processing of I/O store instructions. This system avoids a strict synchronization of a CPU operations with the operations of I/O adapters. A program interface is used which decouples the synchronized instructions of the CPU from the operations of the I/O adapter to allow an asynchronous completion of the instructions by the I/O adapter. The I/O program interface includes an I/O store queue which contains information for identifying the I/O store instructions to be executed.

[0005] In a computer system comprising at least one central processing unit and an input/output adapter the queuing of I/O requests in the I/O adapter is improved. Simultaneously, the requirement of synchronization between the CPU and the I/O adapter is avoided.

[0006] The invention is preferably used a in computer system which comprises at least one CPU having a cache memory. A connection network connects the CPU to a main memory of large capacity containing program instructions and data, and to an I/O adapter. I/O requests are transferred from the CPU through the connection network to the I/O adapter which comprises doorbell apparatus controlled by the CPU to allow the I/O adapter an access to the main memory for reading or writing I/O instructions and data. According to the invention, as defined in the claims, the I/O adapter comprises at least one I/O request mailbox for receiving I/O requests and data from a CPU. The I/O request mailbox is connected to at least one I/O request queue storage for storing a plurality of I/O requests and related data or main memory addresses received by the mailbox. A credit register is provided for storing the filling state of the queue storage. The connection network is designed and operated to transfer the contents of the credit register to the CPU which checks the filling state of the queue, to transfer an I/O request and related data to a queue of a selected request mailbox, and to activate the doorbell apparatus for signalling to the I/O adapter to process the I/O request including an access to the main memory by the I/O adapter if required.

[0007] In accordance with the principles of the invention, each initiator of an I/O request contains in the I/O adapter its own set of resources comprising an I/O request mailbox and associated therewith a credit register and a doorbell register. The initiators of I/O requests may be independent of each other and unsynchronized with each other or with the I/O adapter. The initiators of I/O requests may be one CPU or a number of CPUs or a processing thread or task running on the at least one of said CPUs.

[0008] A method according to the invention, as defined in the claims, comprises the steps of:

• • [0009] a) containing in the I/O adapter at least one I/O request mailbox for receiving I/O requests and data from the CPU;
  • [0010] b) extending the I/O request mailbox by at least one I/O request queue storage for storing a plurality of I/O requests and related data received by the mailbox;
  • [0011] c) connecting a credit register to the I/O request queue storage for registering the filling states of the I/O request queue storage;
  • [0012] d) transferring the contents of the credit register to the CPU for enabling the CPU to check the filling state of the I/O request queue storage;
  • [0013] e) transferring an I/O request and related data to the I/O request mailbox for being stored in the I/O request queue storage connected to the mailbox; and
  • [0014] f) activating the doorbell register for signalling to the I/O adapter to process the I/O request including an access to the main memory by the I/O adapter if required.

[0015] The invention permits that the initiators of I/O requests do not need to synchronize the I/O requests with the operation of the main memory but may perform the enqueueing of I/O requests in the I/O adapter independently of main memory access operations and thus avoids latency due to required or pending ain memory access operations during the enqueueing process.

[0029] FIG. 1 shows a schematic representation of a conventional computer system and its operation to execute I/O requests. This computer system comprises a central processing unit 100, a main memory 120 and I/O adapter 130. The I/O adapter and the central processing unit (CPU) share resources located in the main memory for enqueueing I/O requests to the adapter. FIG. 1 represents only a subsection of an I/O request queue maintained in the main memory. This subsection comprises a storage location 121 of an I/O request and a storage location 123 of the next request pointer address. A storage location 122 contains data to which the I/O request in storage location 121 is related, and a storage location 124 contains a lock pointer tag which is associated with the next request pointer address in location 123. The operation starts with step 101 setting the lock pointer tag in location 124 to secure that the data which are subject of the request are not modified by other main memory accesses. The CPU increments in step 102 the pointer in location 123 and in step 103 stores in this location the incremented pointer as next request pointer. In step 104 an I/O request is generated by the CPU and stored in location 121 addressed by the next request pointer in location 123. In the following step 105 the CPU activates a doorbell latch 131 in the I/O adapter 130 signalling to the I/O adapter that it may perform an access to the main memory for reading or writing I/O instructions and data. The CPU unlocks the next pointer in step 106 so that in step 107 the I/O adapter can fetch the I/O request from the main memory and read the data specified by the request from the main memory. A synchronization of the main memory accesses by the CPU and with main memory accesses by the I/O adapter is performed by synchronization instructions provided by the CPU. This has the result that the CPU has to wait for either main memory access or adapter resource access during the request enqueueing process. These types of latencies can aggregate to a relevant amount for low latency I/O requests or low latency clustering requests.

[0030] The FIG. 2 shows a computer system wherein the principles of the invention are implemented. The computer system comprises a CPU 200 which has a level 1 cache storage 210 with fast access time. The CPU 200 is connected through a connection network 220 to further caches which may be level 2 and 3 caches 225. The connection network 220 may be implemented as conventional data bus or by switches and point-to-point connections. The CPU 200 is also connected through network 220 to a main memory 230 which is usable for storing data and is typically implemented as DRAM. The CPU 200 is further connected through network 220 to an I/O adapter 240 which comprises at least one I/O request mailbox 250 and which is connected to at least one I/O request queue storage 260 to store I/O requests received by the mailbox 250 and the data associated with these I/O requests. The I/O request queue 260 is a local storage or a set of registers contained in the I/O adapter 240 and able to store a plurality of I/O requests and its associated data while waiting for execution.

[0031] The I/O adapter 240 comprises a doorbell register 270 which may be a register or a latch and which, if activated, signals to the I/O adapter 240 to process the I/O request including an access to the main memory by the I/O adapter if required. The I/O adapter 240 further comprises a credit register 275 which stores and indicates the filling state of the I/O request queue 260. A register control logic 280 is provided to maintain the contents of the credit register 275 according to the current filling state of the queue 260. This content of the credit register 275 is read by the CPU 200. The operation of the register control logic 280 is described in more detail with reference to FIG. 4. The execution of the I/O requests is controlled in a conventional manner by a process controller 285 the operation of which is described in with reference to FIG. 5.

[0032] An example of the format of an I/O request, herein also called “I/O command”, is represented in FIG. 3. According to the format shown an I/O request comprises a command identification CMND ID which is used to identify a certain I/O request in the mailbox 250 and in the request queue storage 260 and to identify the response to this request generated by the I/O adapter. The I/O request further comprises a command code I/O CMND specifying the operation to be executed when interpreted by the I/O adapter. Furthermore, the request comprises an initiator address in a field 301 which in the system of FIG. 2 is the address of the CPU 200 but which may also be the address of multiple initiators of the request such as threads performed by CPU 200 or the address of a second CPU contained in the computer system. The field 301 may also contain a network initiator address such as an IP (Internet Protocol) initiator address. The request comprises further in a field 302 a target address of the request command and in a field 303 immediate data which are directly associated with the request. Finally, a field 304 of the request format comprises a list of addresses which are pointers to data associated with the request but are too large to be treated as immediate data. These pointers specify locations in the main memory where the data associated with the request are stored.

[0033] FIG. 4 shows the operation sequence of the control logic 280 to control the components 260, 270 and 275. Initially, the I/O adapter is in a reset status wherein the credit register 275 is set to indicate that the full capacity of the I/O request queue storage 260 is available if only one credit register exists 275, and is set to indicate that a fraction of the capacity of queue storage 260 is available if multiple credit registers exist. The initial status of the control logic 280, as shown at 401, is to wait for a command to activate the doorbell register 270 for signalling to the I/O adapter that the I/O request has to be executed wherein an access to the main memory by the I/O adapter may be performed if this is required. If such command is received, the control logic 280 transfers in step 402 an I/O request received by mailbox 250 to the next free location of the queue storage 260. In step 403 the control logic 280 decrements credit register 275 and sends in step 404 a signal to the request processing controller 285 indicating that work is available, i.e. that an I/O request has to be executed.

[0034] The operation sequence of the controller 285 is shown in FIG. 5. The initial status of the 285 is the wait for “work available” status according to step 501. If the control logic 280 generates a “work available” signal, the controller 285 starts to process the I/O request in step 502. In case that the I/O request to be processed requires a long enqueue operation, step 503 performs an access to the main memory 230 to fetch or store data from or to the main memory 230. After the execution of the I/O request the controller 285 removes in step 504 the processed I/O request from the I/O request queue storage 260, and initiates in step 505 an increment of the credit register 275.

[0035] The method steps of the computer system for a fast enqueueing of I/O requests are described with reference to FIGS. 6 and 7. The CPU 200 starts the operation by step 601 to read the contents of the credit register 275 into a register R1 which is a register in the CPU 200 that is not shown. Alternatively, R1 may be a storage location in the cache 210. Step 602 checks register R1 whether its contents is zero. This is the case when the credit register indicates that the queue 260 is full. I such case a waiting status 603 is entered. If the register R1 contains a non-zero value, the CPU 200 performs a data transfer 214 to bring data which are subject of the next I/O request into a location 211 of cache 210. Subsequently, the CPU 200 creates in step 605 the next I/O request and stores it in location 212 of cache 210. In step 606 the I/O request in cache location 212 and the associated data in cache location 211 are transferred to the mailbox 250 from where they are shifted to the queue store 260 through step 402 (FIG. 4). In step 607 a command is written into the doorbell register 270 signalling to the I/O adapter that the I/O request is to be processed. Usually a transfer of an I/O request from the CPU to the mailbox 250 is followed by a doorbell activating signal. If sequences of I/O requests are created by the CPU 200 or another initiator, a doorbell activating signal may follow after each I/O request or after a predetermined number I/O requests. With the step 607 the operation of register control 280 is initiated. Step 608 decrements register R1 and step 609 checks whether there are more requests to be sent. If this is not the case, the fast enqueueing operation is terminated. Otherwise step 610 checks the content of register R1 and steps 604-610 are repeated except register R1 contains zero in which case step 601 is repeated.

[0036] FIGS. 8 and 9 show the method steps of the computer system for I/O requests having associated data which are too large for being stored in location 211 of cache 210 or are anyhow contained in the main memory. This operation is called “long enqueueing”. The CPU 200 starts the operation by step 701 to read the contents of the credit register 275 into register R1 corresponding to step 601. Step 702 checks register R1 whether its contents is zero and waiting step 703 is entered if the credit register indicates that the queue 260 is full. In case of a non-zero value in register R1, the CPU 200 generates in location 213 of the cache 210 a pointer list to the I/O data which are subject of the next I/O request. This pointer list is generated by using the list of main memory addresses which are included in field 304 of the I/O request as shown in FIG. 4 and which addresses data associated with the request. Step 705 transfers the pointer list from the location 213 of the cache 210 to the mailbox 250. The CPU 200 creates in step 706 the next I/O request and stores it in location 212 of cache 210. In step 707 the I/O request in cache location 212 is transferred to the mailbox 250 from where it is transferred to the queue store 260 through step 402 in FIG. 5A. In step 708 a command is written into the doorbell register 270 for signalling to the I/O adapter that the I/O request has to be executed and that an access to the main memory 230 by the I/O adapter may be performed. With step 708 the operation of register control 280 is activated. Step 709 decrements register R1 and step 710 checks whether there are more requests to be sent. If this is not the case, the long enqueueing operation is terminated. Otherwise step 711 checks the content of register R1 and steps 704-711 are repeated except register R1 contains a zero value in which case step 701 is repeated.

[0037] FIGS. 10 and 11 show an alternative embodiment and the method steps of its operation. This embodiment is part of a computer system that comprises a CPU 900 and a store queue 920 for queuing I/O requests and related data before they are transferred to the I/O adapter 240. Also in this embodiment further CPUs or threads or tasks running in such CPUs may be initiators of the I/O requests as will be described below with reference to FIG. 12. The store queue 920 is preferably a first-in-first-out stack which contains a location 926 for storing a plurality of I/O requests received from the CPU 900 via connection 906, and further contains a location 924 for storing the I/O data related to each of said requests and received from the CPU 900 via connection 904. The store queue stack 920 further comprises a write doorbell location 927 which stores for each of the I/O requests received by stack 920, or for each received sequence of a predetermined number of I/O requests, a doorbell write signal that is provided by the CPU via connection 907. The locations 926, 924 of the store queue stack 920 are connected through connections 936 and 934 to the mailbox 250 of I/O adapter 240. The location 927 of the store queue 920 is connected through a connection 937 to doorbell register 270 in the I/O adapter 240. The implementation of I/O adapter 240 corresponds to the I/O adapter 240 as described above with reference to FIG. 2. Accordingly, the connections 934, 936, 937 may be part of the connection network 220 of FIG. 2.

[0038] The operation of the store queue 920 will be described with reference to FIG. 11. The CPU 900 starts the operation by step 901 to read the contents of the credit register 275 into register R1. This corresponds to step 601 described above. Step 902 checks register R1 whether its contents is zero indicating that I/O request queue 260 is full. If this is the case a waiting step 903 is entered. If register R1 contains a non-zero the contents, the CPU 900 transfers I/O data to location 924 of the store queue stack 920. This takes place through step 904 which corresponds to an activation of connection 904 in FIG. 10. The CPU 900 further transfers through step 906 an I/O request, to which the I/O data in location 924 is related, to location 926 of the store queue stack 920. In the next step the connection 907 is activated to transfer a doorbell signal to location 927 of the store queue stack 920. Simultaneously with the transfers to the store queue 920 by the steps 904, 908 and 907, steps 934, 936 and 937 are activated which trigger asynchronous transfers to the I/O adapter. Steps 934 and 936 start a transfer of I/O data from location 924 and of I/O requests from location 926 to the I/O mailbox 250 of the I/O adapter 240. Step 937 relates to a transfer of the contents of register 927 to the doorbell register 270 of the I/O adapter. These transfer operations are performed asynchronously with the operation of the CPU 900 and further transfer operations according to steps 904, 906 and 907. Actual transfers of the I/O request and the related data to the I/O adapter 240 and their processing is performed in the same manner as described above with reference to FIGS. 2-7. The same applies to transfer of the doorbell signal to the I/O adapter 240 and its processing by the I/O adapter 240. Referring back to FIG. 11, step 908 decrements register R1 and step 909 checks whether there are more requests to be sent. If this is not the case, the operation is terminated. Otherwise step 910 checks again the content of register R1 and steps 904-909 are repeated except the contents of register R1 is zero in which case step 901 is repeated. It can be seen that the transfer operations to the store queue 920 are performed independently of the processing of I/O requests in the I/O adapter and that waiting times of the CPU 900 for pending transfers to the I/O adapter are reduced or avoided.

[0039] The embodiment of FIGS. 10 and 11 may also be used for the handling of I/O requests having associated data which are too large for being stored in location 211 of the store queue stack 920. In such cases the location 924 contains instead of I/O data a list of main memory addresses of the data to which the I/O request is related. The operation of the queue stack 920 in handling these I/O requests corresponds to the operation described with reference to FIG. 11 while the processing of these requests in the I/O adapter 240 corresponds to the long enqueueing operations described with reference to FIGS. 8 and 9.

[0040] FIG. 12 shows a generalized block diagram of a computer system comprising parallel initiators for the enqueueing of I/O requests wherein each initiator have its own set of mailbox, doorbell and credit register. The system of FIG. 12 comprises CPU 1100 and CPU 1200 each working in a multiprogramming mode using in a conventional manner multiple program threads including threads 1101 and 1201. An I/O adapter 810 is connected to both of the CPUs 1100 and 1200 and comprises two I/O request mailboxes 1150 and 1250 each having its own doorbell register 1170 and 1270 and credit register 1175 and 1275, respectively. The I/O adapter 1110 also comprises a queue storage 1160 which is shared by both I/O request mailboxes 1150 and 1250. The threads 1101 and 1201 are initiators for I/O requests which are performed by the components 1150, 1170, 1175 and 1250, 1270, 1275 in the manner described with reference to FIGS. 3-11. The I/O adapter 1110 does the multiplexing and scheduling of the I/O requests received by the mailboxes 1150 and 1250. For this purpose, the common queue storage 1160 may be a first-in-first-out storage. Alternatively, more sophisticated scheduling algorithms can be applied.

[0041] While the invention is disclosed with reference to the described embodiments, modifications or other implementations of the invention are within the scope of the invention as defined in the claims.