Method for performing parallel management operations including and deleting computer systems

[0002] A portion of the disclosure of this patent document contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever.

[0003] 1. Field of the Invention

[0004] The present invention relates to computer systems and networks of computer systems, and, more particularly, to efficient, parallel execution of computer processes over multiple computer systems.

[0005] 2. Description of the Related Art

[0006] Computer systems have attained widespread use in providing computing power to businesses and institutions. Some important fields in which computer systems have become common include manufacturing monitoring and control; electrical, mechanical, and chemical system design and simulation; and network management and service. Productive processing time is of great importance to the users of computer systems in these and other fields because computer system down time can result in significant costs.

[0007] Computer systems are designed to provide a degree of system availability that is necessitated by the applications for which the computer system is intended. High availability (HA) computer systems, as their name implies, are required to be available, or “on,” virtally all of the time. Such computer systems (or networks of such computer systems) should be substantially error free or should be capable of recovery from errors. Also, consumption of resources and down time in such systems due to maintenance tasks should be minimized.

[0008] For example, in a 24×7 manufacturing system, time is extremely critical. Performance of management/maintenance operations as quickly and efficiently as possible is extremely important. In such a manufacturing system, downtime for such operations can be very limited. For example, in one such manufacturing system, scheduled downtime may be as little as one hour per quarter. Therefore, the operations performed during these downtime periods must be optimized to run as quickly and efficiently as possible.

[0009] In an environment where scheduled downtime is extremely limited, it is imperative that operations be performed as quickly and accurately as possible. Also, in a client/server environment where hundreds of systems may be involved, the ability to execute the same command across all systems with minimal effort and in the shortest time possible is extremely important. When a common operation is to be performed across a group of systems, one of two methods is typically used. For example, management operations to be performed across many computer systems in a client/server environment are typically performed either manually on each system, one-by-one, or sequentially using a script. Manual execution on each system involves logging into each system and performing the command manually. This is time consuming and can result in errors and/or omissions. Sequential script execution typically includes bundling the work to be done into a script and then sequentially executing the scripted commands on all systems. If hundreds of systems are involved, this sequential method can also be very time consuming.

[0010] It has been discovered that a method and software application may be used to facilitate dynamic, substantially parallel execution of operations on multiple computer systems over a distributed computing system such as a semiconductor processing system. The application and method thereof defines a set of systems upon which each operation is to be executed and then performs each operation in a substantially parallel fashion. Such an application and method thereof allow system administrators to execute operations across a set of systems substantially in parallel and without writing sequentially executed scripts of the operations. Consequently, system management operations are performed across a multitude of systems in a fast and efficient manner, thereby decreasing system down time.

[0011] In one embodiment, a method of minimizing the time required for executing a function over a network of computer systems in a high availability system is provided. A list of computer systems is provided by a parent process. The list includes a plurality of computer systems upon which a first function is to be performed. The first function is received by the parent process. A plurality of child processes is sequentially created. Each of the child processes corresponds to one of the computer systems on the list. Each of the child processes executes in parallel with at least one other of the child processes. Each of the child processes remotely executes the first function on the computer system corresponding to the respective child process.

[0012] In another embodiment of the invention, a computer program product encoded in computer readable media is provided. The computer program product includes first, second, third and fourth instructions executable on a computer system. The first instructions are for providing a list of computer systems by a parent process. The list includes computer systems upon which a first function is to be performed. The second instructions are for receiving the fist function by the parent process. The third instructions are for sequentially creating child processes. The fourth instructions are for executing each of the child processes in parallel with at least one other of the child processes. Each of the child processes remotely execute the first function on a computer system corresponding to the respective child process.

[0013] The present invention may be better understood, and its numerous objects, features, and advantages made apparent to those skilled in the art by referencing the accompanying drawings. The use of the same reference symbols in different drawings indicates similar or identical items.

[0014] FIG. 1 shows a block diagram of an embodiment of a distributed computing system according to the invention.

[0015] FIG. 2 shows a flow chart of an embodiment of a parallel execution module and/or operation according to the invention.

[0016] The following description is intended to provide a detailed description of an example of the invention and should not be taken to be limiting of the invention itself. Rather, any number of variations may fall within the scope of the invention which is defined in the claims following the description.

[0017] In one embodiment of the invention, a distributed computing system includes a system management server coupled to various compute servers via a network coupling. A system management program initiates a system management process on the system management server to enable a system manager to define a list of computer systems coupled within the distributed computing system and to execute commands in parallel on the listed computer systems. The system management process causes common commands to be executed across multiple computer systems after receiving a command to perform an operation and a list of one or more systems upon which to perform the operation. Using local commands executing on the system management computer system, the system management process allows a user to define a list of computer systems upon which the operation is to be performed. Computer system names may be added to a list interactively or by loading computer system names from file. Once the list of computer systems is defined, the command to be executed is supplied to the system management process. The command is then executed on each of the listed computer systems in a virtually parallel fashion.

[0018] Implementing a system management software application that allows a command to be executed in parallel on multiple computer systems greatly reduces the time required to perform common system management operations. For example, a file copy of a ten kilobyte file to 100 systems can take approximately two minutes using a sequential method. Using the parallel execution method and/or application described herein, such a copy operation can take less than fifteen seconds. Also for example, such a method and/or application therefor to execute a common function such as a factory systems restart can reduce the restart period from approximately 30 minutes down to 15 minutes.

[0019] Referring to FIG. 1, a distributed computing system 100 such as a semiconductor processing system includes various computer systems which are coupled via network coupling 105 to execute various subtasks of system 100. For example, system 100 includes NFS server 110, database server 120, compute servers 130, equipment interface servers 150, and terminals 160. Network 105 may be an Ethernet coupling, a fiber distributed data interface (FDDI) coupling, or any other expedient for coupling the computer systems of system 100. Compute servers 130 include at least one compute server such as compute servers 130-1, 130-2 and 130-n. Equipment interface servers 150 include at least one equipment interface server (EIS) such as EIS 150-1, EIS 150-2 and EIS 150-n. Terminals 160 include at least one terminal such as terminals 160-1, 160-2 and 160-n. System 100 further includes at least one fab tool coupled to the equipment interface servers 150. For example, fab tools 170 include tools 170-1, 170-2 and 170-n coupled to EIS 150-1. Fab tools 170 are coupled to EIS 150-n via any appropriate coupling such as a network, RS-232 standard connection, or any other appropriate expedient(s). System 100 further includes management server 180 for managing the various computer systems and processes of system 100.

[0020] Database server 120 typically has high availability requirements and is accessible by multiple remote users of computer systems coupled within system 100. Computer systems such as database server 120 are often used to facilitate general manufacturing monitoring and control. For example, database server 120 provides information regarding semiconductor manufacturing tools (e.g., fab tools 170) and facilitates control and/or automation of semiconductor manufacturing tools and/or systems by users in a semiconductor manufacturing facility or fab.

[0021] Each of compute servers 130 includes various work sessions. For example, compute server 130-1 includes work sessions 141-1, 141-2. Work sessions 141-1, 141-2 facilitate control of the operation of one or more of fab tools 170. In operation, a terminal user initiates a semiconductor application work session on one of compute servers 130 from one of terminals 160. Thus, for example, work sessions 141-1, 141-2 are initiated from terminal 160-1 and use computer resources of compute server 130-1. Each work session accesses a semiconductor processing database on database server 120 to read and write information regarding the operation of fab tools 170. Each work session provides control information to fab tools 170 via equipment interface servers 150 and returns status information to database server 120. For example, a user of terminal 160-1 initiates work session 141-1 on compute server 130-1 to control fab tool 170-1 and to receive and provide information regarding the operation of fab tool 170-1.

[0022] Management server 180 is used by system managers to manage the various computer systems and processes of system 100. Management server 180 includes memory 185 which includes system management program 181. As described hereinafter, system management program 181 initiates child processes 190-1, 190-2 . . . 190-n to remotely perform a function by remote child processes 191-1, 191-2 . . . 191-n on compute servers 130-1, 130-2 . . . 130-n, respectively. The function is performed on each of compute servers 130 in a substantially parallel fashion in accordance with the invention.

[0023] System 100 provides a semiconductor manufacturing and processing management system for managing information distribution, system control and automation of manufacturing processes in a semiconductor fabrication facility. System 100 operates using the WorkStream factory shop floor application which is available from Consilium, Inc. of Mountain View, Calif. WorkStream is used to drive the manufacturing functions within a semiconductor manufacturing facility or fab. WorkStream facilitates management of the execution of production plans for manufacturing operations by providing control of work instructions to users and by providing control of automated equipment. WorkStream also facilitates recording product genealogies and work conditions.

[0024] WorkStream users initiate WorkStream work sessions on compute servers 130 from terminals 160. Each WorkStream work session accesses database server 120 to read and write information to database 150 regarding the operation of fab tools 170. Each WorkStream work session provides control information to fab tools 170 via equipment interface servers 150 and returns status information to database server 120.

[0025] Up to ten and sometimes more compute servers 130 are coupled to network 105, and up to thirty and sometimes more terminals 160 are coupled to each of the compute servers 130 via network 105. Such a system 100 can support over 1000 work sessions initiated, for example, by users of terminals 160 and executed on compute servers 130 to control and provide information regarding the operation of fab tools 170. Thus, for example, more than 100 WorkStream work sessions may be executing on each of the compute servers 130 from the terminals 160 coupled thereto. In such a computing environment, the ability to quickly and efficiently execute common operations over multiple computer systems becomes especially important due to the high availability nature of system 100.

[0026] FIG. 2 depicts a flow chart of a parallel execution module and/or operation 200 according to the invention. The invention is described hereinafter with reference to an exemplary embodiment including various operations. Such operations are typically performed by software module(s) such as a software application and/or subroutines thereof For example, management server 180 includes system management program 181 which initiates a system management process in active memory of management server 180 for performing substantially parallel execution of management subprocesses.

[0027] During receive command operation 210, a command is received for processing by system management program 181. Exemplary system management program 181 provides a command line driven user interface which prompts the user for various commands for subsequent processing by system management program 181. For example, system management program 181 displays a prompt such as “RCMD>” to the user. At the prompt the user can type in one of several function names such as local function calls or a function call including a remote function to be executed on each of various selected computer systems of system 100.

[0028] After receive command operation 205 and during remote command decision 210, system management program 181 determines whether the received command is a local command or a command for remote execution on multiple computer systems (referred to herein as a remote command). If the received command is a local command, control transitions to execute local command operation 215. If the received command is a remote command, control transitions to select first system operation 220.

[0029] During execute local command operation 215, system management program 181 executes the received command on management server 180. For example, system management program 181 includes various local functions which are executed responsive to the user inputting a corresponding local command at a prompt. Exemplary local commands include system management commands, display management commands, and command execution commands.

[0030] System management commands are those commands that are used to add or delete system names from a remote system list that is maintained by the program.

[0031] The remote system list is a list of computer systems of system 100 upon which a common remote function is to be performed. The “add” command allows a single system to be added to the remote system list. The “delete” command allows a single system to be removed from the remote system list. The “clear” command deletes all systems from a remote system list. The “load” command allows a list of systems to be read in from a file.

[0032] Display management commands allow the user to specify that a window be opened for one or more systems to display the output of the command being executed on the system. For example, the “display” command is used to turn on/off the display for one or more systems. The “logging” command allows output to the display window to be toggled on or off.

[0033] Command execution commands are used to affect the actual command to be run on the remote system list. For example, the “username” command allows the user to specify the username under which the remote command should be executed. The “go” command is where the actual command to be executed is supplied.

[0034] Exemplary system management program 181 also includes various miscellaneous commands such as “help” and “exit.” The miscellaneous commands are used to provide help for each command and to exit the program.

[0035] Once the remote system list has been determined using the above described local, system management commands, the “go” command is used to identify a remote command which is to be run on each computer system on the remote system list. To perform the remote command on all listed systems in a virtually parallel fashion, system management program 181 progresses through the remote system list by setting a variable to the next system name on the list during operations 220, 225, 235, 250. Exemplary system management program 181 then creates a child process for each system included on the remote system list For example, a UNIX fork⬭ system call creates a child process at operations 225, 292. This child process looks at the variable defining the current computer system upon which the command is to execute and remotely executes a function on the current computer system. For example, the child process performs a UNIX rexec⬭ subroutine call which remotely executes the command on the current system. For example, if three systems are defined in the remote system list (e.g., computer systems 130-1, 130-2 and 130-n of FIG. 1), one child process is created for each system resulting in a total of three child processes (e.g., child processes (CPs) 190-1, 191-2 and 190-n of FIG. 1) after the “go” command is entered. The remotely executing function is represented by remote child processes (RCPs) 191-1, 191-2 . . . 191-n. Each child process executes without regard to whether the process created before it has completed and therefore typically executes concurrently with at least one of the other child processes and often with some or all of the other child processes. Each child process uses the rexec⬭ subroutine call to execute the received remote command on the corresponding system.

[0036] Referring again to FIG. 2, during select first system operation 220, system management program 181 selects a computer system of system 100 as the current system. The current system is selected from the remote system list. For example, system management program 181 selects compute server 130-1.

[0037] After select first system operation 220 and during create child process operation 225, system management program 181 creates a child process corresponding to the currently selected system. The child process executes the remote command on the currently selected system. For example, child process 190-1 is created and executes a remote system call to execute the command on compute server 130-1. The operations of an exemplary child process is discussed hereinafter with reference to child process flow 290 which includes operations 292, 294, 296, 298.

[0038] After create child process operation 225 and during last system decision 235, system management program 181 determines whether there is another system on the remote system list. If there is a computer system upon which the remote command has not been executed, control transitions to select next system operation 250. For example, if compute server 130-2 is on the remote system list but has not been selected as the current computer system during an iteration of the loop including operations 225, 235, 250, control transitions to select next system operation 250. If there are no other computer systems upon which the remote command is to be executed, control transitions to receive results operation 240.

[0039] During select next system operation 250, system management program 181 selects another computer system from the remote system list. Continuing the above example, system management program 181 selects compute server 130-2 as the current computer system. After select next system operation 250, control transitions to create child process operation 225 so that a child process may be created for the next computer system which is the newly selected, current system. For example, child process 190-2 is created to execute a function on compute server 130-2.

[0040] Child process flow 290 shows operations of an exemplary child process of the multiple child processes executing on each of the computer systems on the first list. After start child process operation 292, and during execute remote command operation 294, the child process of the current iteration executes an rexec⬭ system call with the received remote command on the corresponding, currently selected system. For example, child process 190-1 executes an rexec⬭ system call to compute server 130-1 which initiates remote child process 191-1. After execute remote command operation 294, control of the child process transitions to report result operation 296. During report result operation 296, the child process reports results of the execution of the remote command on the currently selected system to the parent process and/or system management program 181. For example, a variable is returned with a value indicating the presence or lack of an error in the execution of the child process(es). The child process of the current iteration ends at end child process operation 298.

[0041] During receive results operation 240, system management program 181 receives results from each of the child processes created for each of the computer systems of system 100 upon which the remote command was executed. For example, a variable is returned with a value indicating the presence or lack of an error in the execution of the child process(es). Receive results decision may occur in parallel with or in between operations 225, 235, 250. After receive results operation 240 control of system management program 181 transitions to exit decision 245.

[0042] During exit decision 245, system management program 181 determines whether the system user wishes to exit system management program 181. If an exit command is received at the command line provided by system management program 181, operation of system management program 181 terminates. If an exit command is not received at the command line provided by system management program 181, and another command is received at the command line, control transitions to receive command operation 205.

[0043] The above description is intended to describe at least one embodiment of the invention. The above description is not intended to define the scope of the invention. Rather, the scope of the invention is defined in the claims below. Thus, other embodiments of the invention include various modifications, additions, and/or improvements to the above description.

[0044] In one embodiment, parallel execution operations are executed by source code provided in the Appendix filed herewith and incorporated by reference herein. Such a software application may be written for various UNIX operating systems (e.g., HP-UX, SunOS, Solaris, AIX, etc.) or may be written for any multitasking operating system. For example, in one embodiment, computer systems 110, 120, 130 and 160 operate under control of a UNIX type operating system. For example, compute servers 130 are Hewlett Packard Series 800 computer system running the HP-UX 9.04 type UNIX operating system. Such a computer system may function as a server for client computer systems. Terminals 160 are Xterminals and/or personal computer systems executing software such as Xwindows of PCNFS for managing sessions. It is understood that other computer systems, other UNIX operating systems (e.g., the International Business Machines AIX operating system), and other operating systems (e.g., the Windows NT operating system) may be used in accordance with the invention. Other embodiments may include mainframes, minicomputers, personal computer systems and/or terminals in place of some or all of the computer systems of system 100. Files are accessed by the computer systems of system 100 using the network file server (NFS) protocol. Other appropriate methods of making the files available to computer systems 110, 130, 140 and 160 may be used in accordance with the invention.

[0045] The system management program 181 need not be located on management server 180 as shown in FIG. 1. System management program 181 may execute on any computer system of system 100 which has access to the remote systems upon which the remote commands are to be executed.

[0046] The operations referred to herein may be modules or portions of modules (e.g., software, firmware or hardware modules). For example, although the various modules may include application specific hardware modules, the described embodiment includes software modules and/or includes manually entered user commands. The software modules discussed herein may include script, batch or other executable files, or combinations and/or portions of such files. The software modules may include a computer program or subroutines thereof encoded on computer-readable media.

[0047] Additionally, those skilled in the art will recognize that the boundaries between modules are merely illustrative and alternative embodiments may merge modules or impose an alternative decomposition of functionality of modules. For example, the modules discussed herein may be decomposed into submodules to be executed as multiple computer processes. Moreover, alternative embodiments may combine multiple instances of a particular module or submodule. Furthermore, those skilled in the art will recognize that the operations given in the exemplary embodiment are for illustration only. Operations may be combined or the functionality of the operations may be distributed in additional operations in accordance with the invention.

[0048] Each of the blocks of FIG. 2 may be executed by a module (e.g., a software module) or a portion of a module or a computer system user. Thus, the above described parallel execution method, the operations thereof and modules therefor may be executed on a computer system configured to execute the operations of the method and/or may be executed from a computer-readable storage medium. The method may be embodied in a machine-readable or computer-readable storage medium for configuring a computer system to execute the method. A computer-readable storage medium such as storage 185 of server 180, for example, can include storage devices such as floppy disks, hard drives, optical disks (e.g., CD-ROM, CD-R, etc.), semiconductor-based memory units including FLASH memory, EEPROM, EPROM, ROM or RAM or application specific integrated circuits. Storage 180 and other storage may be permanently, removably or remotely coupled to servers 180 or any other computer system of system 100. Thus, the software modules may be stored within a computer system memory to configure the computer system to perform the functions of the module. Other new and various types of computer-readable storage media may be used to store the software modules discussed herein. Furthermore, the software modules may be received by computer system 180, for example, from computer readable media including any of magnetic storage media, including disk and tape storage media; optical storage media, including compact disk memory and digital video disk storage media, nonvolatile memory storage memory; volatile storage media; and data transmission media including computer network, point-to-point telecommunication, and carrier wave transmission media.

[0049] The invention has been described in the context of system 100. In such a system, each of a plurality of semiconductor manufacturing computer systems provides information, control and/or automation to semiconductor manufacturing tools, systems and/or users in a semiconductor manufacturing facility. Thus, in one embodiment, parallel execution is provided in a computer network which provides information, control and automation of semiconductor manufacturing processes. Other embodiments include other types of distributed computing systems. The distributed computing system need not be a manufacturing control system. Any distributed computing system is especially benefited by the above described parallel execution method and/or module(s).

[0050] Although specific examples of computer systems 110, 120, 130, 140, 150 and 160 are shown in FIG. 1 as included within the exemplary system 100, other embodiments of the invention may include an information processing network having any number of computer systems in other appropriate network configuration. Furthermore, the functionality of the various computer systems may differ from the computer systems of exemplary system 100. System 100 may include additional computer systems of different types or combinations, or the functions of several of the servers may be combined into fewer servers.

[0051] A computer system processes information according to a program and produces resultant output information via the I/O devices. A program is a list of internally stored instructions such as a particular application program and/or an operating system. A computer process typically includes an executing (running) program or portion of a program, current program values and state information, and the resources used by the operating system to manage the execution of the process. A parent process may spawn other, child processes to help perform the overall functionality of the parent process. Because the parent process specifically spawns the child processes to perform a portion of the overall functionality of the parent process, the functions performed by child processes (and grandchild processes, etc.) may sometimes be described as being performed by the parent process.

[0052] A computer system typically includes multiple computer processes executing “concurrently.” Often, a computer system includes a single processing unit which is capable of supporting many active processes alternately. Although multiple processes may appear to be executing concurrently, at any given point in time only one process is actually executed by the single processing unit. By rapidly changing the process executing, a computer system gives the appearance of concurrent process execution. The ability of a computer system to multiplex the computer system's resources among multiple processes in various stages of execution is called multitasking. Systems with multiple processing units, which by definition can support true concurrent processing, are called multiprocessing systems. Active processes are often referred to as executing concurrently when such processes are executed in a multitasking and/or a multiprocessing environment.

[0053] Computer systems are often coupled to other computer systems to form a network to facilitate the exchange and handling of information. A computer network typically includes a group of computer systems and associated hardware devices (e.g., physical connections) and software applications (e.g., communications protocol programs and data). A computer network may include multiple computer processes executing on several computer systems as a distributed multiprocessing platform in which the several computer systems are accessible by each of the computer processes. Additionally, multiple computer processes executing on the same computer system often communicate with each other in a similar fashion as multiple computer processes executing on different computer systems. Consequently, it often makes sense to discuss interaction between computer processes instead of, or in addition to, interaction between computer systems.

[0054] Also, the variable identifier “n” is used in several instances in FIG. 1 to more simply designate the final element (e.g., compute server 130-n, terminal 160-n, etc.) of a series of related or similar elements (e.g., compute servers 130, terminals 160, etc.). The repeated use of such variable identifiers is not meant to imply a correlation between the sizes of such series of elements. The use of such variable identifiers does not require that each series of elements has the same number of elements as another series delimited by the same variable identifier. Rather, in each instance of use, the variable identified by “n” may hold the same or a different value than other instances of the same variable identifier. For example, compute server 130-n may be the tenth compute server in a series of compute servers, whereas terminal 160-n may be the fiftieth terminal in a series of terminals.

[0055] Also, with regard to the above described computer processes, system 100 may include the active computer processes 181, 190-1, 190-2 . . . 190-n, 191-1, 191-2 . . . 191-n executing during the above described operations. System 100 may include modules/programs for executing the computer processes stored in memory of computer systems of system 100 when system 100 is not filly operational.

[0056] The above description is not intended to present an exhaustive list of embodiments of the invention. Although an attempt has been made to outline some exemplary embodiments and exemplary variations thereto, other embodiments and/or variations are within the scope of the invention as defined in the claims below.