Expert System for Computer System Resource Management

A portion of the disclosure of this patent document contains material which is subject to copyright protection. The owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever.

This invention relates generally to the process of controlling a large-scale software system and, more particularly, to a methodology for tuning a large-scale system to effectively utilize system resources such as a central processing unit and its associated database files and database servers.

Effectuating efficient processing in the day-to-day operation of a large-scale software system has conventionally required staff personnel with expertise to interpret manifestations of system performance, and has consumed a significant amount of staff time and financial resources to identify actual and potential problem areas.

System performance is generally gauged by two very different measures, known respectively as system response time and system throughput. The former measure relates to the speed of responding to a single system command, whereas the latter means the efficiency of processing large amounts of data. Balancing these measures is the domain of the system expert, and overall efficiency depends on carefully arranging both the hardware and software that does the processing as well as the information that is stored in the system's databases. The procedure of allocating resources so that system processing is shared by the resources on a balanced basis is called "tuning."

Tuning of this type, namely, allocation of resources by a human system expert, is still carried out despite the advent of computerized, autonomous resource managers designed to optimize performance. These computerized managers are typically part of the operating system software and are designed to service general work loads, that as a rule are not backlogged. Representative of such computerized managers of the non-backlogged type is the subject matter of the disclosure of Watanabe et al. U.S. Pat. No. 4,890,227. For the non-back-logged case, it is possible to optimize both response time and throughput. For the backlogged case, improvements in response time come at the expense of throughput, with the converse also being true. However, these managers operate under the handicap that the current allocation of files to disks is outside the scope of their optimization. Also, of necessity, these managers lack application-specific knowledge. This limitation precludes them from including in their optimizations other, non-performance related concerns. A prime example of one such concern is allocating files to disks to minimize the impact on the system as a whole in the event that a particular disk fails. This concern is referred to here as developing a damage limitation policy and its implementation has a direct bearing on system availability. Finally, these managers do not attempt to regulate the flow rate of transactions into the system. Consequently, their optimization is a local, not a global optimization.

To place the response time and system throughput measures and their ramifications in a practical setting, reference is made to an illustrative example of a large-scale software system, designated the FACS system, which finds widespread use in the telecommunications environment. The FACS system assigns and inventories telephone outside plant equipment (e.g. cable pairs) and central office facilities (e.g. cable appearances on a main distribution frame). Currently, the FACS system is embodied in approximately 1 million lines of source code. It runs on a mainframe or host computer composed of a CPU complex with 2-4 processors, an I/O system containing 6-8 dual disk controllers, and 60-80 600 million byte disks. Because of the complexity and size of the FACS system as well as its sophisticated execution environment, operating the FACS system with acceptable response time while maintaining high system throughput is an on-going challenge which requires tuning skills of an expert to achieve a high level of system performance.

Formulating a thorough diagnosis and just one possible remedy for performance problems in such a large system typically takes expert analysts several days. Starting with performance symptoms, analysts manually tuning a FACS system first deduce which of several kinds of data they need to analyze the problems. Then they apply formulas and guidelines based on their own experience to arrive at a basic understanding of the problem areas--for instance, occasionally transactions stack up in queues leading to inefficient use of central-processing resources. Next, the analysts cull the data, searching for specific explanations for the degradation of performance. The final step, identifying solutions, again calls for using so much knowledge and data that short cuts based on past experience are a practical necessity. Of course, once changes are made, another cycle of analysis must be undertaken to verify that problems are corrected. Because the analysis is so time consuming and difficult, performance issues are often addressed only after system performance has degraded.

When systems of this size go awry, there are typically many symptoms to analyze. It is difficult to isolate those that are truly performance affecting from those that merely appear to affect performance. To cull the important symptoms, and then synthesize assessments of the current state of the system requires an understanding of how a symptom (such as a large number of concurrently active processes) affects the users' perception of the responsiveness of the system as a whole. Developing this view requires deep analysis, facilitated by the mathematics of queueing theory. The analysis techniques themselves are difficult to understand and to properly interpret the results obtained from them requires insight into the dynamics of the underlying system.

These deficiencies as well as other limitations and shortcoming of these and other techniques are obviated, in accordance with the present invention, by an expert system implemented by computer software that utilizes measured data and deploys a search technique which references a knowledge base, including a set of rules derived from domain experts, to tune the large-scale system.

In the preferred embodiment of the present invention, the host system that is to be monitored for tuning is linked to a workstation which implements expert system software. The host system is sized to insure concurrent transaction backlog during peak hour, and includes disks, disk control units, and files on the disks controlled by the disk control units; the host also has measurable central processing unit (CPU) and input/output (I/O) service times. A set of operational principles relating to the operation of the host system is defined by capturing operational and descriptive information supplied by a host system expert who acts as a mentor for the expert system. The operational principles are transformed into an executable knowledge base and then stored in a memory of the workstation. The CPU and I/O service times are measured on the host, as controlled by the expert system. The I/O service times are stored in the memory, and the CPU service times are processed by the expert system before storage in the memory. The stored CPU and I/O service times are then processed to generate system state data. In one aspect of the present invention, sequencing through the set of operational principles with the system state data is effected to obtain CPU-response time information with is used to modify the concurrency of the host system as determined from the CPU-response time information. In another aspect of the present invention, the expert system is loaded with the current I/O configuration arrangement of the host system, and sequencing through the set of operational principles with both the I/O service times and the I/O configuration information is effected. The result is I/O modification information which is used to reallocate the current allocation of files to disks. When tuning is recommended, program code in a form executable by the host is automatically generated by workstation software. The code may be uploaded to the host for execution to thereby implement the recommended changes.

Accordingly, the subject matter of the present invention is composed of a methodology that enhances the performance and availability of a dedicated, application specific host or main frame computer, sized to insure a transaction back-log during peak hour, where the number of concurrent transactions, the allocation of files to disks and across disk control units, are the parameters to be optimized. The subject matter of this invention not only explicitly addresses the response time-throughput trade off, but also explicitly considers a file allocation policy during system optimization, tempered by damage limitation considerations.

The ART system (ART is a trademark of the Inference Corporation) and the Lisp language are the implementation vehicles for the heuristic aspects of the file allocation optimization in this invention. A key component of this functionality involves the use of an Assumption-based Truth Maintenance System to determine file relocation solutions to I/O subsystem tuning problems. The process involved is a heuristic search through plausible file relocation solutions, considering application and system level factors. The ART system and Lisp give no guidance or structure for performing or controlling this search. The structure and control of the search are part of this invention. The role of the ART system is to provide generic truth maintenance utilities to ease the implementation task.

In performing such a search using the ART system, it is not simply the case that the ART system is missing the knowledge (data) itself which is then supplied. With the present invention, the search algorithm is devised independently of the ART system or the knowledge for which the ART system is designed to work.

The organization and operation of this invention will be understood from a consideration of the detailed description of the illustrative embodiment, which follows, when taken in conjunction with the accompanying drawing.

FIG. 1 depicts a logical view of a portion of the I/O system of the large-scale system to be tuned;

FIG. 2 depicts the large-scale system to be tuned in block diagram form as well as the workstation which implements the expert system (ES) analysis;

FIG. 3 is illustrative of the CPU Performance Analysis Presentation output display from the expert system;

FIGS. 4A and 4B depict the I/O System Diagram output display from the expert system.

FIG. 5 presents the repair recommendation for the particular FACS system under evaluation; and

FIG. 6 illustrates frames with I/O subsystem data.

To place in perspective the detailed description of the present invention, it is instructive to gain a basic understanding of the manual technique for tuning a large-scale software system. Accordingly, the first part of this detailed description discusses the conventional methodology as it is applied to the FACS system. This approach also has the additional advantage of introducing notation and terminology which will aid in elucidating the various aspects of the present invention.

The next part of the description then focuses on a discussion of the functionality of the expert system (ES) as well as the processing effected by the software modules comprising ES. After this, the actual code for ES is presented.

A human analyst begins by first identifying which area of the tuning spectrum should receive intensive analysis. The analyst does this by determining the magnitude and source of the queueing delays present in the response time of the average FACS transaction. The source of delay may be, for example, the CPU complex or the I/O system. Once quantified, the analyst them judges where to concentrate. For illustrative purposes to introduce the notions of tuning, it is supposed that the I/O system was selected. With reference to FIG. 1, a "logical" view (in contrast to a block diagram or schematic view) of one subsystem of the FACS system, namely, FACS I/O system 100, is depicted. Elements 101,102, . . . are dual disk controllers (typically there are 6-8 disk controllers), with controller 101 being composed of single controllers 1011 (A1) and 1012 (A2), respectively, and so forth. Elements 111-114 (D11,D12,D13,D14), 121-124 (D21,D22,D23,D24), . . . are disk packs (typically there are 60-80 600 million byte packs); disk 111 is shown as being comprised of files 1111-1114 (D11F1,D11F2,D11F3,D11F4), respectively. Files typically are of specific types such as a library or database. Disks 111,121, . . . are controlled by controller 1011, disks 112,122 . . . are controlled by controller 1012, and so forth.

The types of problems that can occur in system 100 include: Disk D11 is "hot," that is, is heavily used because of the types of files stored by the disk pack; file D11F1, which is a database, and file D11F4, which is a library, are on the same disk pack which has the potential for causing an access time problem; or the average "utilization" of controller 101 exceeds a prescribed threshold.

To identify "hot" control units or disks, experienced performance analysts check a high-level report, referred to as the Controller and Disk Activity Report, by comparing the measured control unit utilization and disk utilization with their respective threshold values. To further pinpoint the possible causes of equipment being hot, expert analysts consult a report called the I/O Trace Summary Report which details I/O statistics at file level by device. Performance analysts manually calculate the utilization of each file on the hot equipment using the statistics reported on the I/O Trace Summary Report and single out the ones that have the highest utilization because their removal will most significantly relieve the workload on the hot equipment. In addition, performance analysts must also verify if file placement guidelines are carefully followed across the I/O system. The characteristics of the FACS system and the capability of computing hardware prohibit multiple, heavily used files to be placed on the same pack to avoid contention, and require certain system and application files to be duplexed, that is, the creation of an identical version of a given file, for the purpose of recovery. In the given I/O system, performance analysts examine the I/O Trace Summary Report looking for offending files located on a hot disk, say D11. This pack contains D11F1, D11F2, D11F3, D11F4, and other files. Assisted by their familiarity with their system's file numbering schemes (e.g., which number stands for which file) and their knowledge of the file naming convention, and a detailed File Placement Report, performance analysts identify that a data base file (D11F1) and a library file (D11F4) coexist on the same disk. Since both are heavily used files and placing them together on the same pack may cause the disk to be overly utilized, performance analysts compare the file utilization of both files and relocate the most heavily used one, e.g. the library file, to a less busy pack. Other than moving files around, performance analysts may take different action (e.g. duplex the offending file to reduce traffic) depending on the nature of the problem and the type (e.g., database or library file) of offending file.

Manually finding a best offload location for a file move is a complex task. Ideally, analysts must first identify all available offload equipment having the same physical characteristics but which are not as busy as the current equipment by going through the File Placement Report and the Controller and Disk Activity Report. They then need to manually adjust the performance statistics on the currently located pack and each possible offload pack to reflect the shifted load, based on the statistics reported on the Controller and Disk Activity Report and the I/O Trace Summary Report. To verify that no new performance problems are introduced, they must apply the file placement guidelines used originally to detect problems to the reorganized I/O system. Finally, performance analysts select the one with the lowest utilization and fewest potential problems. Because of the complexity of the task, performance analysts do not always follow the procedure described above. There is a tendency to stop the searching process when they find the first location that fits the problem file. To verify if existing problems are resolved and no new problems are created, performance analysts must collect and analyze another data sample.

With reference to FIG. 2, pictorial diagram 200 depicts an arrangement of the computerized expert system in accordance with the present invention, namely, ES 210 operating in conjunction with modeling system 220, shown as analyzing and providing corrective performance adjustment information, via feedback paths 211 and 212, to improve the responsiveness of FACS production host 250. Typically host 250 is configured, for purposes of discussing an illustrative embodiment of the present invention, to include: arrival queue 251; host concurrency control 252; CPU queue 253; CPU 254; I/O complex 255; and measurement complex 256. ES 210 is implemented by computer software resident within workstation 205. Typically, modeling system 220 is also implemented with software resident within workstation 205. The adjustment information is of the type that provides: (a) parameter information for setting the maximum permissible concurrently active transactions (lead 211); and (b) executable code that effectuates file moves on production host 250 (lead 212). The adjustment information may also be observed by a responsible user of system 250, via display panels 240 and 245; this user is responsible for insuring that production host 250 provides acceptable service to its community of end users. The responsible user oversees the interaction between ES 210 and production host 250 and can monitor changes to production host 250 as guided by the adjustment information. As alluded to, the adjustment information is specific to two areas. The first is the response time-throughput trade off, and the second is file placement. In each case, the implementation of the adjustment information results in changes to production host 250. The effect of implementing the response time-throughput adjustment information is to modify the number of transactions permitted to process concurrently. The effect of implementing the file placement adjustment information is to move or modify the configuration of files to eliminate over utilized disks and control units or minimize the impact on production host 250 in the event that a disk or control unit fails. When a file is moved or modified for this reason, it enhances the availability of production host 250. File moves or configuration changes triggered by this reason are referred to as damage limitation modifications.

(Reference is made to the Glossary for the complete definitions of terms and symbols contained in the remainder of this specification). To gain additional insight into the nature of the response time-throughput adjustment information identified in panel 240 of FIG. 2, an understanding of the nature of the response time-throughput time trade off, the subject of the adjustment information, must be elucidated. There exists a direct relationship between the number of transactions, T.sub.hreads, concurrently active in host 250 the rate at which they are served, .lambda.. Although this relationship can be developed for any system, it is particularly important for systems designed to operate at high processor utilization, a condition present in back-logged systems, i.e. system where transactions are waiting to begin processing, as indicated by the state of arrival queue 251 on production host 250.

For any finite system, as the number of transactions concurrently admitted to service increases, the service rate asymptotically approaches an upper bound, .lambda..sub.max. This bound is governed by the physical characteristics of the host system, i.e., the number and speed of the system's physical devices, CPUs, disks and control units, and the host system resource cost, the CPU and disk service time, necessary to satisfy each transaction's demand for service. When a host system operates in the neighborhood of the bound .lambda..sub.max, admitting additional transactions only increases the wait-time experienced by each transaction. This effect manifests itself as an elongated transaction existence time, T. Balancing the admission of additional transactions which results in higher contention, induced wait-time against the increase in throughput obtained from the higher concurrency is known as the response time-throughput trade off. In alternate terms, the trade off consists of striking a balance between quick response for an individual transaction and the rate at which large numbers of transactions are processed by production host 250.

There are many ways to depict this trade off. The method chosen for illustrative purposes of the present invention is to display the rate of service, i.e., the transaction throughput, .lambda..sub.M, as a function of the amount of CPU queueing, B.sub.CPU, while explicitly stating the number of transactions concurrently active in the system. This relationship is portrayed by the curve in display panel 240 of FIG. 2. Note the asymptotic behavior--as additional transactions are admitted the throughput approaches .lambda..sub.max.

Once this relationship is developed, the final step in producing adjustment information is to quantify as well as to make known to the responsible user the implications of choosing one realization of the response time-throughput time trade off over another, as well as supplying for the contemplated realization, the

(1) maximum allowable concurrency that should be set in host 250 to effect the realization, T.sub.hreads.sbsb.M

(2) the expected processing rate, .lambda..sub.M

(3) expected CPU queuing time, B.sub.CPU

(4) the expected existence time, T.sub.M, and

(5) the expected CPU utilization, .rho.CPU.sub.M.

Elucidating these implications is accomplished by a set of operational principles, which also provide the operating point information stated above. The elucidation is in the form of statements that quantify changes from the current CPU queuing time and system throughput. The elucidation plus the vector (T.sub.hreads.sbsb., .lambda..sub.M, B.sub.CPU, T.sub.M, .rho.CPU.sub.M) at the contemplated realization makes up the "Response Time-Throughput Adjustment Information", namely, panel 240 in FIG. 2. The complete set of principles that generate this adjustment information is stated in Appendix A.

Since the observer of the adjustment information is the responsible user, the form of the adjustment information, as distinguished from its content, is also important. The illustrative embodiment of ES 210 offers this adjustment information using both a graphical and textual presentation, as exemplified in FIG. 3. The default display 310 of FIG. 3 is the pictorial interpretation of the busy hour model, as traced by curve 311. The X-axis shows the amount of time each transaction waits for service by the CPU complex while the Y axis shows the throughput, given that the user is willing to tolerate the selected wait time which is an input parameter. The current wait time in the FACS system is shown by vertical straight line 312. A responsible user may investigate the tradeoff between the wait time and system throughput by moving arrow 313 toward the origin (reducing the wait time) or away from the origin (increasing the wait time.) Table 314 reports the actual measurements (average processing rate, CPU queueing, total existence time, concurrency, and CPU utilization) and the forecasted results based on the proposed wait time specified by the user. This computation module of ES 210 also allows the user to explore other hours available on menu 315. Display 310 is redrawn each time the responsible user chooses a different hour and table 314 reflects the statistics of the selected hour.

It is recalled that the objective of producing the file placement adjustment information is to move or modify the configuration of files to eliminate over-utilized disks and control units, or minimize the impact on production host 250 in the event that a disk or control unit fails.

The rationale for the decision to move or modify the configuration of files is that high device utilizations cause unnecessarily long I/O existence times, resulting in unacceptable performance. The central issue here is "What constitutes unacceptable performance"? ES 210 bases decisions on non-transitory, abnormally high device utilizations. Non-transitory is important. In practice, just as a human analyst does not react to floating hot spots, busy components caused by high, but short lived traffic to a particular file, neither does ES 210. Transitory I/O traffic can be induced by variations in the transaction workload. To avoid recommending moves one day only to discover that the move was unnecessary later, ES 210 will not generate code to move files for a-priori designated files whose traffic is known to be transitory. The responsible user identifies this type of file to ES 210.

To quantify high device utilizations, analyses determined the correlation between a hardware component's utilization and the queueing delays experienced by an average transaction. These analyses, specific for the FACS application, culminated in utilization thresholds, that if consistently exceeded, lead to the conclusion that a disk or control unit has unacceptable performance. These thresholds are supplied as defaults in ES 210, and can be modified by the responsible user.

Similarly, system availability improvements can be achieved if steps are taken to minimize the impact on production processing, if a hardware component fails, i.e. if a damage limitation policy is implemented. In the illustrative embodiment, ES 210 implements a FACS-specific damage limitation policy by consolidating files containing key application data bases on the same disk, or when appropriate, duplexing files. Duplexing files means that two live copies of a file can exist, with either copy capable of servicing an application. In the event that a disk containing one of the two copies fails, processing continues without interruption. If a file is duplexed both copies should not be placed on the same disk, or if one is conservative, on the same string.

ES 210 produces file placement adjustment information by implementing the following reasoning process. First, ES 210 detects high utilization control units and disks by comparing measured utilizations with the thresholds identified above. Next, ES 210 determines which files are causing the problem by identifying high traffic files. ES 210 will also examine how the files are allocated across packs and disk strings looking for files that violate the damage limitation policy. The hardware performance of individual disks is checked against the manufacturer's performance specification. Finally, file placements, known to have caused problems in the past, perhaps on other FACS systems, are identified. This phase of the analysis process culminates in a list of problem files.

Given the problem list, ES 210 then develops the file placement adjustment information, including file moves, duplexing, or main memory caching recommendations. This adjustment information is composed of not only moving high traffic files to less heavily utilized disks, but also preferentially taking advantage of operating system file duplexing and main memory caching capabilities to reduce the traffic to files identified as problems.

This is the most demanding portion of the process. In the case of file moves, there are potentially many candidate locations. Using a search technique, ES 210 finds the "best" new location for each offending file, as measured by minimum post-move disk and controller utilization, and smallest spare slot, large enough to accommodate the offending file, usually on the same disk. Most importantly, ES 210 does this without creating new problems. The search culminates in a list of new locations for the problem files. The current and new locations are used by ES 210 to generate code, which when executed on production host 250, effectuates the file moves.

In the illustrative embodiment of the invention shown in FIG. 2, the "File Placement Adjustment Information" of panel 245 includes the following types of recommendations, which may or may not all be present depending upon the condition of the I/O subsystem.

(1) file moves, expressed in the form of executable code, plus a pictorial representation of the effect of implementing the moves,

(2) file duplexing recommendations,

(3) main memory file caching recommendations.

Appendix B lists the operational principles that produce the file placement adjustment information.

As was the case for the response time-throughput adjustment information, in order to keep the responsible user fully informed of the curative measures ES 210 deems necessary, the illustrative embodiment presents its adjustment information in an easily understood format. The human-readable interface presents a compact, coherent, graphical representation of the I/O complex, depicting the connectivity of its control units, and disks, and the location of problem files, and animates the file moves, showing their migration from the current, problematic locations to the new locations specified in the file placement adjustment information. Other curative measures contained in the placement adjustment information, i.e. duplexing and caching recommendations, are presented in textual form. FIG. 4 is an example of this I/O display, and panel 504 of FIG. 5 is an example of the high level code which effectuates file moves when executed. Appendix C shows actual source code deployed to generate FACS-compatible code. In particular, regarding FIG. 4, so as to visualize I/O system performance problems, a compact graphical display of the I/O system may be called into view on workstation 205; the result of this activity is shown as overlay panel 401 in FIG. 4. Background panel 402 is used to summarize detailed diagnoses in textual format. Overlay panel 401 shows in graphical form the large complex comprising the disk control units (e.g., unit 411), the disks controlled (e.g., disk 420), and the files (e.g., files 421,422) stored by the disks; a partial view of overlay panel 401 was depicted in FIG. 1.

An operational principle is a succinct statement of a possible contingency, which, if it occurs, triggers a conclusion that obviates the contingency. A knowledge base is a set of related operational principles, whose contingencies and conclusions encompass the outcomes expected from some process under observation. Here the process being observed is the operation of a main frame computer, the contingencies are the various symptoms indicative of problematic performance or availability that might occur, where the symptoms are embedded in measurement data. The conclusions are specific actions designed to obviate problematic performance or improve system availability. Thus, a knowledge base, in its totality, is a representation of how a human expert reasons about a problem domain.

In constructing the operational principles that make up a knowledge base, it is first necessary to understand the problem domain. In this embodiment, the problem domain consists of both the response time-throughput trade off, and the factors that affect the performance and availability of an I/O complex supporting the host application. Understanding the problem domain includes knowing how the I/O components should be connected, the expected performance of each type of disk and control unit, and how files should be allocated across the I/O complex. Given this understanding, a human expert augments it with first conceptualizing contingencies that, should they occur, are performance or availability affecting, then postulates and confirms conclusions that when implemented, obviate the contingencies. Note that this conceptualization is the essence of defining the operational principles that make up a knowledge base and is independent of the computer language, or expert system knowledge encoding facility used to eventually represent the knowledge.

Part of the conceptualization process is recognizing which contingencies can be detected when observing a system and which cannot. This is a measurement issue. The robustness of the measurement instrumentation on production host 250 shown in FIG. 2 constrains the operational principle set to those principles whose contingencies will be triggered by particular values in the data. For if an event is not measurable, it is by definition not observable, and hence cannot be included in the knowledge base. In the illustrative embodiment, the complete set of data used by ES 210 is provided in Appendix D.

The transformation of operational principles into an executable knowledge base occurs through the creation of two primary types of structures in the knowledge base: frames and forward chaining rules implemented in the ART system language. These structures are broadly defined as follows:

frame--A collection of information about a stereotypical object, act, or event, and the relationship of the object to other objects. For example, with reference to FIG. 6, stereotypical disk pack 630 in FACS has expected performance information as well as descriptive attributes, and actual FACS disk pack object 640, which is connected through the instance-of(HAS INSTANCE) relationship, has actual performance information plus information inherited from stereotypical disk 630. Actual disk pack object 640 controls files 650 and 660. Similarly, stereotypical control unit object 610 has expected performance information which is inherited by actual FACS control unit object 620.

forward chaining rule--A segment of computer code which designates that

IF a particular situation is described by the program's data

THEN the program should take a particular action

Forward chaining rules are executed by a general interpreter, which for the illustrative embodiment is provided by the ART system language, that matches rules with data, and executes rules whenever their IF conditions are satisfied. This data-driven type of computer program is contrasted with traditional procedural computer programs, where ordering of the program instructions determine how the program is executed. Appendix G gives specific examples, in ART code, of forward chaining rules.

Frames give an internal representation of the external view of the FACS system. For the I/O subsystem, frames represent physical objects (disk controllers, disks) and computer software objects (files). Hand-crafted, stereotypical, frames contain expected performance (I/O existence time) of specific models of disk controllers and disks, as shown in FIG. 6. This information is inherited by instances of those objects which represent actual instances in the FAGS computer configuration, defined by the data obtained via the Reduction and Formatting software 260 identified in FIG. 2. These instances of disk controllers, disks, and files now have actual I/O existence time data, which can be compared to expected data values. Similarly, the system state space, which provides data on tradeoffs between response time and throughput as the number of customers vary, is represented as frames.

Heuristics (rules of thumb), implemented as forward chaining rules, constitute the second type of structure in the executable knowledge base. For I/O subsystem processing, these heuristics compare expected to actual service times along with relative comparisons of actual service times to pinpoint bottlenecks in the I/O subsystem. For CPU processing, rules interpret the state space relative to the amount of queueing (wait time) a user is willing to tolerate.

Lisp language computer programs provide a support structure which makes possible specialized processing, such as that needed to obtain I/O modification information as described in the Section to follow.

FIG. 2 shows information labeled "RAW DATA" on lead 259, flowing between Measurement Complex 256 and Reduction, Formatting and Modeling Software 260. This data is made up of detailed information contained in the System Log File, the I/O Trace file, the SIP File and the Master File Directory, and is produced by host supplied software. The purpose of Reduction, Formatting and Modeling Software 260 is to condense and transform this detail into a form readily used by ES 210 to support the operational principles that generate the response time-throughput and file placement adjustment information.

In FIG. 2, two flows are shown emanating from Reduction and Formatting Software 260, namely information of leads 261 and 262, with both terminating in ES 210 and under control of control information provided over lead 263 from ES 210. The right hand flow over lead 261 is direct and provides the information necessary to drive the File Placement operational principles. This flow consists of the Control Unit, Disk, and File I/O service times, an inventory of the files that support the FACS application, with an accounting of the disk space these files require, the available free disk space, and an accounting of the free locations, and the connectivity of control units and disk drives, also known as I/O configuration information.

The left hand flow over lead 262 initially consists of CPU and disk service times. It first traverses Modeling System 220, where it is transformed into System State Data appearing on lead 221 before arriving at ES 210. The System State Data supports the response time-throughput operational principles, and a portion of this data eventually becomes part of the response time-throughput adjustment information.

For the complete specification of each of the individual data items that make up the generic information categories emanating from Reduction, Formatting, and Modeling Software 260, reference is made to Appendix D. Also, see this same Appendix for a list of the individual software components that make up Reduction, Formatting and Modeling Software 260.

The state space is a compact representation of key parameters of the system derived under different conditions of transaction concurrencies. More formally, the system state space consists of an array whose rows consist of the five elements, (T.sub.hreads.sbsb.M, .lambda..sub.M, B.sub.CPU, T.sub.M, .rho.CPU.sub.M). Each row corresponds to a particular number of transactions concurrently active in the system, i.e. The first row corresponds to one transaction, the second row, to two transactions, etc.

Note that a major portion of the response time-throughput adjustment information described earlier is a row of the system state space array, developed under the guidance supplied by the response time-throughput operational principles.

The state space array may be generated by any one of several techniques, a discrete event simulation, or modeling the system as a product form network then solving the model using a convolution or mean value analysis algorithm. All of these techniques require the service times as their inputs, and use an algorithm of choice to transform service times into the system state data.

In the current embodiment, the model is solved using Buzen's (see, for example, Allen, A. O., "Probability, Statistics, and Queueing Theory", Academic Press, 1978) recursive solution, a convolution algorithm, for a single work load class, non-priority, central server model that includes the CPU complex and all active disk devices. The model is self-calibrating in that iterative solutions attempt to match the measured transaction existence time at the measured transaction volume, and transaction concurrency, by adjusting the service time of internal software locks whose occurrences are measurable, but whose durations are not. After achieving a satisfactory match, the final model is solved to produce the system state array for all feasible transaction concurrencies. Appendix E is a listing of the code which implements the model solution above.

Forward chaining rules reason over the data described above to identify problem-causing files in the I/O subsystem. There are two categories of problem-causing files:

(1) Hot (highly utilized) files which impact the utilization of a control unit or disk, and therefore are labeled primary move candidates.

(2) Poorly placed files, which because of their placement relative to the locations of certain other files have the potential for causing performance or recovery problems. This type of file has secondary move priority.

The structure of the domain modeled above, such as control of disks by a certain control unit, or storage of file fragments by a certain disk, is used by the diagnosis heuristics to identify hot files. An example of this type of heuristic is:

IF the transfer rate of a hot file explains a hot control unit

THEN designate this hot file as a primary move candidate

The analysis heuristics make use of performance evaluation formulas, such as this formula for the heuristic above: ##EQU1## That is, the contribution of a hot file to the utilization of a dual control unit is equal to the transfer rate of that hot file times the I/O rate in requests per second, divided by two.

A heuristic which may identify other problem files (secondary move candidates) is:

IF a database spans multiple disks, and the database fits on one disk

THEN consolidate the database on one disk

Various heuristics also identify, for example, sub-par performance of I/O subsystem hardware. Heuristics and formulas thus identify files which should be moved to help balance the load across the I/O subsystem, and other heuristics identify and advise the end-user of isolated problems.

After finishing its I/O problem diagnosis stage, ES 210 attempts to repair the I/O subsystem problems identified. ES 210 arrives at its recommendations by simulating sequences of file moves, and choosing the sequence that solves the most problems (without creating any major new ones). These sequences of simulated moves are represented internally in ES 210 as paths in a search tree, where the root of the tree represents the initial (actual) system state, and each node represents a hypothetical system state resulting from an experimental file relocation. An Assumptive Truth Maintenance System (ATMS), provided by the ART system Viewpoint mechanism, handles the details of maintaining the distinction between ES 210's real and hypothetical system states. The search strategy devised as part of the inventive aspects of the present invention, described in the following paragraph, makes use of this Viewpoint mechanism to simplify the implementation of the search for I/O problem repairs.

For each file that it tries to relocate, ES 210 ascertains the set of disks that have enough spare space and utilization (including controller utilization) to accommodate that file. Heuristics set forth in Appendix B aid in the selection of potential new locations for a problem file. These heuristics, which represent system attributes and various FACS application characteristics that can affect performance, include the following examples:

IF reference files are being relocated

THEN place no more than two reference files on a disk

and

IF a disk has more than one space large enough for the file

THEN use a best-fit algorithm to choose one.

Now that it has a set of candidate new locations, ES 210 expands its search tree by creating a new node for each of the hypothetical new locations found for the file under consideration. ES 210 can then temporarily believe as fact the hypothetical assumptions (i.e. The file movements) that formed the new node, and reason independently about the system state embodied in each of these nodes. With the relocated file occupying the previously free space, and the disk and control unit utilizations adjusted to reflect the shifted load, ES 210 calls upon the same rules that it used for problem diagnosis in the original system state. This time, however, ES 210 does not report the diagnoses to the end user; rather, it uses them to judge the merit of the file move being considered. If any new problems are severe enough, the search tree node representing the experimental file placement is removed. This immediate pruning helps to control the size of the search tree.

Some of the problems that may be detected at a file's new location are not absolute constraints against moving the file there. ES 210 again uses heuristics to gauge the severity of the new problems and the original reasons for moving the file.

After it has considered new locations for all of the problem files, ES 210 examines the search tree and chooses the path that relieved hot spots most successfully while satisfying the greatest number of file placement heuristics. ES 210 extracts the set of file moves that compose this path and passes these recommended moves to host concurrency control 252 of FIG. 2; also, the responsible human is notified of these moves via display panel 240.

Many search techniques could be considered for this task, but those not used would fail. Alternatives to ATMS for belief revision in this repair task would not support the required reasoning to find the best solution. Brute force generate and test of consistent assignments would quickly lead to a combinatorial explosion of nodes in the search tree. Chronological backtracking would explore futile branches which were previously discovered to involve contradictory file assignments, and would re-expand various nodes of the search tree, again leading to efficiency problems. Dependency-directed backtracking (see, for example, Stallman, R., and Sussman, G., "Forward Reasoning and Dependency-Directed Backtracking in a System for Computer-Aided Circuit Analysis," Artificial Intelligence 9, pp. 135-196, 1977) as used in truth maintenance systems (see, for example, Doyle, J., "A Truth Maintenance System," Artificial Intelligence, Vol. 12, 1979) could find a solution to the file assignment problem, but would not easily find the best solution to this problem because search would stop at nodes where contradictions occur, and comparison of alternative file assignments would be almost impossible since only one alternative would be available in the database at any given time (see, for example, Williams, C., "Managing Search in a Knowledge-based System," Inference Corporation Technical Report). Therefore, the system is likely not to find the most desirable file assignments. Thus, ATMS, which allows the exploration and comparison of hypothetical file assignments, is used to handle belief revision and the search strategy entailed in the file assignment problem in ES 210.

Appendix F lists the source code to obtain the response time-throughput adjustment information for the CPU system.

Appendix G lists the source code for the rules to find the optimum placement for the files in the I/O subsystem according to what is known about those files and the disks and controllers they reside on.

Appendix H lists the source code for the frame definitions and rules the knowledge base uses to analyze the I/O subsystem, and the rules by which this analysis is accomplished. After these rules fire, ES 210 knows what problems exist in the I/O subsystem.

It is to be understood that the above-described embodiment is simply illustrative of the application of the principles in accordance with the present invention. Other embodiments may be readily devised by those skilled in the art which may embody the principles in spirit and scope. Thus, it is to be further understood that the methodology described herein is not limited to the specific forms shown by way of illustration, but may assume other embodiments limited only by the scope of the appended claims. ##SPC1##