Method and Apparatus for Implementing a Stop State for a Processor in a Multiprocessor System

This invention relates generally to computer systems, and more particularly, to multiprocessor systems. Still more specifically, the invention relates to a method for placing a processor in a stopped state in a multiprocessor system.

In order to construct computer systems capable of processing large amounts of data, it is useful to employ multiple individual processors in a single system. One exemplary multiple processor system is the PowerPC symmetric multiple processor ("SMP") system which operates in accordance with the common hardware reference platform ("CHRP") architecture. This system is described in detail in "PowerPC Microprocessor Common Reference Platform: A System Architecture," ISBN 1-55860-394-8, available from IBM. However, combining several individual processors which are simultaneously capable of running in a single system, presents certain difficulties for the system designer. For example, when a multiprocessor system is first powered on, it is conventional that one processor is selected as the "master" to execute the system open firmware for starting, or "booting" a client operating system. Other processors must be put into a halt, or stopped state which does not interfere with the operation of the master processor. If one of the other processors were to come out of the stopped state while the master was booting the system, it is possible that it would begin accessing system memory, e.g., to perform a memory test, or testing I/O adapters, etc. This could cause coherency problems in system memory resulting in a failure of the boot operation.

Also, the CHRP architecture requires that the run time abstraction software ("RTAS") must ensure that a processor in a stopped state will not check stop, or otherwise fail if a machine check or system reset exception, also referred to as a system reset interrupt ("SRI"), occurs. Thus, although all processors in the stopped state will receive the exception, they must perform a null action and remain in the stopped state.

Some of these problems can be alleviated if all the processors used in the system are provided with a hardware implemented stop state. However, many processors, for example, the PowerPC family of processors, do not have a hardware implemented stop state. Moreover, a hardware implemented stop state may not always provide the flexibility required by system designers.

Accordingly, it is an object of the present invention to provide a method for implementing a stopped state for a processor in a multiprocessor system which overcomes these problems. Additional objects and advantages will become apparent in view of the following disclosure.

One aspect of the invention relates to a method useful in a multiprocessor system for operating a processor. In one embodiment, the method comprises storing stopped signature data in a register on the processor, the stopped signature data being representative of whether the processor is in a stopped state; storing start address data in memory which is accessible by the processor; executing an interruptible spin loop with the processor; comparing the stopped signature data with a predetermined stopped signature to determine whether the processor is in a stopped state when an interrupt is received; and reading the start address data from memory to determine whether there is a request to start if the processor is in a stopped state.

Another aspect of the invention relates to an apparatus useful in a multiprocessor system for operating a processor. In one embodiment, the apparatus comprises a means for storing halt signature data in a register on the processor, the halt signature data being representative of whether the processor is in a halt state; a means for storing start address data in memory which is accessible by the processor; a means for executing an interruptible spin loop with the processor; a means for comparing the halt signature data with a predetermined halt signature to determine whether the processor is in a halt state when an interrupt is received; and a means for reading the start address data from memory to determine whether there is a request to start if the processor is in a halt state.

FIG. 1 depicts a machine state register in an exemplary PowerPC processor.

FIG. 2 depicts signature and mailbox registers according to an embodiment of the invention.

FIG. 3 depicts a mailbox location in system memory according to an embodiment of the invention.

FIGS. 4A-4B is a flow chart illustrating the operation of stopped state according to an embodiment of the invention.

FIGS. 5A-5C illustrate exemplary machine code for a PowerPC processor stopped state according to an embodiment of the invention.

The present invention will be described with respect to embodiments adapted to the exemplary PowerPC architecture. However, it is to be understood that this is for purposes of illustration only, and other embodiments adapted to other architectures will be apparent to those of skill in the art in view of the present disclosure.

There are two characteristics of a processor in a stopped state, 1) the processor is halted, i.e., prevented from executing instructions which would potentially interfere with the operation of other processors in the system, and 2) the processor does a null action, i.e., does not come out of the halted state and interfere with the other processors operations, when a machine check or system reset exception occurs, unless it is specifically requested to do so.

A processor can be halted by causing it to execute an infinite loop, also referred to as a spin loop. With respect to the exemplary PowerPC microprocessor family, a PowerPC processor can be placed in an infinite loop by using the branch-to-itself instruction, i.e., b$. The instruction set for the PowerPC processors is described in detail in "PowerPC Microprocessor Family: The Programming Environrments," available from IBM. However, even though the processor is executing a spin loop, it may be popped out of the spin loop upon receipt of an interrupt which causes the instruction execution of the processor to proceed to an interrupt handler. It is therefore important to ensure that before the processor is placed into the stopped state, it has been properly configured to proceed to the correct interrupt handler, and only acknowledge certain selected interrupts.

In the PowerPC processor, operation of the processor is controlled by a machine state register ("MSR"). FIG. 1 shows a copy of the MSR in a 32-bit PowerPC processor. As seen, there are numerous bits which control the operation of the PowerPC processor. The operation of each of these bits is described in detail in the publications described above. However, in order not to obscure the present invention, only certain bit settings will be described which are relevant to the invention.

In one embodiment, before the processor is placed in the infinite loop, the MSR(IP) bit is set high, or logical 1. The IP bit is the exception prefix. The setting of this bit specifies whether an exception vector offset is prepended with F's or O's. If the bit is low, or cleared, then exceptions are vectored to the physical address Ox000n.sub.-- nrnn, in 32-bit implementations. If the bit is high then exceptions are vectored to the physical address 0xFFFn.sub.-- nnnn, where nnnn is the offset of the exception. In this embodiment, the firmware for booting the system, referred to as the boot ROM, is located at the high end of the memory. Thus, all exceptions which interrupt the spin loop of the processor will be vectored into the boot ROM, rather than the kernel code which is stored in system memory. Of course, it will be recognized this is not necessarily true for a running processor which may have interrupts handled by handlers located in the kernel code.

Next, the MSR(EE) bit is cleared. This disables external interrupts from interrupting the stop state of the processor. This prevents the processor from being taken out of the stopped state in the event of an external interrupt. The MSR(IR) and MSR(DR) bits are also cleared to disable instruction and data address translation. This sets the context of the processor to correctly access the memory stored in the boot ROM. Finally, the MSR(ILE) bit is cleared in order to cause the processor to run the exception code in big endian mode. This embodiment of the invention provides two different exception handling paths depending on the endian mode of the system controller. This will be described in greater detail further herein.

In addition to setting up the machine state register, the processor initializes three general purpose registers for use by the exception handler. This is shown in greater detail with respect to FIG. 2. FIG. 2 shows three registers, R29, R30 and R31, which have been initialized according to an embodiment of the present invention. Of course, it will be recognized that these particular registers have been chosen arbitrarily, and other registers on the processor could be used as a matter of design choice. Registers R29 and R30 each contain a portion of a unique signature to indicate that the processor is in the stopped state. This signature is used by the exception handler to determine whether the processor is in the stopped state for purposes of exception handling. Of course, there is a remote possibility that a processor may not be in a stopped state, but the registers, by coincidence, contain data which matches the signature of the stopped state at the time the exception occurs. This could cause the handler to incorrectly determine whether the processor is in a stopped state. To prevent this, in the embodiment shown, each register is 32-bits wide, and thus, the total signature is 64-bits. By using a 64-bit wide signature, it is extremely unlikely that all 64 bits would match with a predetermined signature, and thus, a running processor would not likely be mistaken to be in the stopped state by the exception handler.

A "mailbox" register, in this case register R31, is initialized with the address to a "mailbox" location, e.g., a double word (8-bytes), in system memory. The mailbox location is shown in FIG. 3. The upper word 32 of the mailbox location contains a real address in memory from which a processor which has been requested to begin running may begin instruction execution. It is also used to inform a stopped processor that is has been requested to begin running, i.e., by the fact that a non-zero value is stored in the mailbox. In one embodiment, the double word is initialized to 0. When a running processor uses the double word to start a stopped processor, the stopped processor resets the double word to 0 once it starts running.

The lower word 34 of the double word contains parameters used to initialize the processor when it begins. Various uses may be made of this location as a matter of design choice. In one embodiment, lower word 34 contains the address of a data structure in system memory which contains data used to initialize various registers in the processor. This allows the entire context of the processor to be set up before it begins execution of a certain code sequence. In one specific embodiment, this enables the system to perform processor switching in order to determine whether a processor on the system is properly operating. Specifically, a running processor may be used to execute a given code sequence to derive a result which is stored in system memory. Next, the register state of the running processor is stored in a data structure in memory. Finally, the running processor is placed into the stopped state. A stopped processor is then taken out of the stopped state and its registers are loaded with the context data stored in the memory structure which is specified by the lower word 34. The recently started processor is then caused to execute the same code sequence. The results are then compared, and if they are different, then it is possible that the system contains a faulty processor.

In one embodiment, when a processor is powered on, it executes instructions from the system boot ROM in big endian mode. Later, the endian mode of the processor will be changed to the operating mode of the system memory controller. When a machine check or system reset exception occurs, the stopped processors will be in big endian mode while the system memory controller may be in a little endian mode. Since the system boot ROM is normally in big endian format, this condition will likely lead to an unwanted program exception. To avoid this, the exception handlers for the machine check exception and the system reset exception in the system boot ROM will have two paths. The big endian path runs correctly when the system memory controller is in big endian mode. The little endian path will have little endian code and runs correctly when the system memory controller is in little endian mode. The operation of one embodiment of the invention is described in greater detail with the flow chart shown in FIGS. 4A-4B.

When the processor receives a power on reset, or system reset interrupt, flow proceeds to step 400 where the processor begins executing from 0xFFF00100. The offset 100 is determined as part of the PowerPC architecture. The upper part of the address has been vectored to this location by setting MSR (IP) high, as discussed previously. The handler is set up so that the processor will get different instructions when fetching from address 0xFFF00100 since the system memory controller may not be in the same endian mode as the processor. In other words, the processor is always set up to run in big endian mode upon entering this exception. However, the actual instruction returned from address 0xFFF0100 will depend on the endian mode of the memory controller since it controls the order of the individual bytes returned from a instruction read at 0100.

If the memory controller is in big endian mode then flow proceeds to step 402 where the processor determines whether the system reset interrupt is caused by a power on reset. This is generally done by checking a power on reset flag which is initially active when the system is first powered up. If the SRI was caused by a power on condition, then flow proceeds to step 418 where the power on reset flag is cleared. This prevents the system from taking this branch a second time, unless power has actually been turned off to the system. Afterwards, the processor proceeds to step 420 where it checks whether it is the master processor in a conventional manner. If the processor determines it is the master, then it continues with steps 428 and 430 where it performs such actions as directing the exceptions to a RAM vector, and it completes the normal boot process of the system.

If the processor is not a master, then in step 420 it proceeds to step 422 where it performs the operations described earlier, such as storing the stopped state signature into the assigned registers, storing the address of the restart code into another register, setting up the MSR to direct exceptions to the boot ROM, and finally in step 426 the processor executes the infinite loop to place itself in the stopped state.

If, in step 402, the SRI is not caused by a power on reset, then flow proceeds to step 404 where the processor compares the data stored in the stopped state signature registers with a predetermined stopped state signature from memory in order to determine whether it is actually in a stopped state. If the processor is running, then flow proceeds to step 414 and the processor performs a request to reboot the system in a conventional manner.

If the processor is in a stopped state, then flow proceeds from step 404 to step 406 where the processor next determines whether it is being requested to start running. In one embodiment, the processor determines whether there is a request for it to start running by testing whether there is a non-zero value in the memory location specified by the mailbox register. If so, then the processor proceeds to step 410 where it retrieves the context data to initialize its registers then proceeds to step 412 where it begins executing instructions at the address specified by the mailbox location. On the other hand, if the processor determines there is no request to start in step 406, then it simply proceeds to step 408 where it returns to the infinite loop.

The sequence followed if the memory controller is in little endian mode is similar to that in big endian mode, but there is no need to check whether the SRI is caused by a power on reset condition. Thus, in little endian mode, when a SRI is received, flow proceeds from step 400 to step 432 where the processor determines whether it is in the stopped state. According to an embodiment of the present invention, the little endian path should not be taken if the processor is not in the stopped state. However, if for some reason it is taken, then flow proceeds to step 444 where it returns from the interrupt. Otherwise, flow proceeds to step 434 where it determines whether there is a request to start. If not, then flow proceeds to step 436 and the processor returns to the spin loop. On the other hand, if there is a request to start then flow proceeds to steps 438 and 440 where the processor sets up the registers and begins executing an instruction sequence specified by the address stored in the mailbox location in substantially the same way as that described in the big endian mode.

Of course, those of skill in the art will recognize that the actual implementation of the method described in the flow chart shown in FIGS. 4A-4B is a matter of design choice and will depend greatly on the specific processors used. One very specific embodiment of the invention is shown in FIGS. 5A-5C. FIGS. 5A-5C illustrate a machine code instruction sequence for a PowerPC processor which is useful according to an embodiment of the invention. Interpreting the code is believed within the abilities of those skilled in the art and only certain important features will be pointed out herein in order not to obscure the present invention. Although this exemplary sequence is for a system memory controller supporting true endian system memory, the procedure is easily modified in other embodiments for system memory controllers which only support big endian system memory even if it is running in little endian mode to support the I/O subsystem.

In this case, it is seen that the interrupt handler labeled "big.sub.13 endian.sub.-- sri" is located at offset 100 from the physical base address of the boot ROM. Thus, on a SRI, the processor will begin executing instructions at this location. If the memory controller is in little endian format, then there is an absolute branch to 0xFFF03C00 where the little endian handler is stored. Otherwise, the system executes the code sequence at label "big.sub.-- endian.sub.-- sri". In this case, the processor first checks to determine whether the SRI is caused by a power on. If so, the processor branches to set.sub.-- nhr.sub.-- bit where it sets the NHR bit, i.e., the flag to indicate a POR has occurred, to prevent re-execution of this code unless power is first turned off to the system. Afterwards, the processor branches to code at the label "to.sub.-- por.sub.-- path" which is essentially the power on reset sequence for the processor. This portion of the code is conventional and will depend on the actual system design. Accordingly, it is not shown in detail. If it is not caused by a power on reset, then the processor checks the stopped state signature. In this case, the stopped state signature is stored in the boot ROM as 0x13579BDF87654321, and this value is compared with the data stored in the signature registers R29 and R30. If the processor is not in the stopped state, then flow branches to "do.sub.-- reboot" which is simply conventional reboot code and is not shown herein. Otherwise, the processor branches to the instruction sequence at is.sub.-- stop.sub.-- proc. In this case, the processor reads the data stored in the memory location specified by the mailbox register and either begins execution as described earlier, or executes an rfi instruction to re-enter the spin loop.

While the invention has been described with respect to the above embodiments, it is to be understood that other variations are within the scope and spirit of the present invention. All publications described herein are hereby incorporated by reference as though set forth in full.