System and Method for Cross-Triggering a Software Logic Analyzer and a Hardware Analyzer

The present invention relates generally to computer systems and more particularly to a system and method for logical analysis of software and hardware integration problems using a software logic analyzer with cross-triggering capabilities.

Computer Operating System

Broadly, a computer operating system controls the activities of a computer. An operating system can be viewed as a very large computer software program that controls resource allocation scheduling, data management and input/output control. One aspect of an operating system is that it provides localized support for software/hardware integration so that individual applications need not be concerned with controlling hardware. One such operating system is HP-UX, an operating system which is manufactured by and made available from Hewlett-Packard Company, California, USA.

The structure of HP-UX can be broken down into two levels. At the core of HP-UX is the kernel. The kernel controls the hardware, schedules tasks and manages data storage. The kernel is "on" (i.e., running) at all times the computer system is on. The kernel is typically shielded from the end-user.

The next level of the HP-UX operating system is known as the user level. Located within this user level is a shell which acts as an interface between the end-user and the kernel. Typically, the shell interprets commands entered by an end-user and translates these commands into a machine language which the kernel understands. The primary task of the shell is to launch other user programs (i.e., call and execute programs from memory). Tools and applications are also located in this second level of the HP-UX operating system. Examples of applications include word processors, data management programs, computer graphic packages and financial spreadsheets.

One of the features of HP-UX is its ability to interact with hardware and hardware interface systems. Typical computer systems comprise a processing unit, memory units, a system bus, and a number of input/output (I/O) interface units. The I/O interfaces control data communications between the processing unit and the outside world, including the user and various peripheral devices.

The kernel of HP-UX contains kernel device drivers which, when configured, permit communication with and control of hardware devices, I/O cards located in the backplane of the computer, and other device drivers (e.g., network layers). Included among these device drivers are drivers for an HP-IB disk controller, magnetic tape drives, mass storage devices, and SCSI direct access storage devices. Typically, the structure of a device driver corresponds to the structure of the I/O hardware the driver is controlling. For example, a small computer system interface (SCSI) disk and tape device connected to a SCSI interface card installed on an EISA computer bus requires the following drivers and/or modules: general I/O services; EISA I/O services; general SCSI services; EISA SCSI interface card driver; SCSI disk driver; and a SCSI tape driver.

Analyzing Hardware Integration Problems

The complexity of the HP-UX operating system, particularly that portion which supports software/hardware integration, increases the probability of integration errors. This is compounded when the inherently peculiar problems associated with controlling the behavior of hardware are considered. One of the primary tasks of the kernel integration engineer is to analyze computer system operation to determine whether the operating system is performing as desired, and if an error is encountered, conduct a debugging process in order to correlate source code execution with the particular error. In the case of device drivers, the process of discovering and diagnosing problems is difficult because drivers are designed to "hide" the problems (e.g., auto-retry functionality). When a problem occurs in a driver, an important first step is to determine the software control flow and to correlate hardware behavior with software behavior, or hardware state to software state.

The primary tools used by kernel integration engineers in debugging integration problems are hardware logic analyzers, software debuggers and rudimentary event logging. Hardware logic analyzers provide the capability to monitor logical activity at the external terminals of a digital system (i.e., I/O busses). The logic analyzer is an oscilloscope-type instrument having dozens of channels, large memory and the ability to disassemble/decode bus states and phases. Typically, a logic analyzer is electrically coupled to a particular hardware interface and, while stepping through a process, the analyzer captures data. One problem with present hardware logic analyzers is that they capture data at a very low level of detail. Thus, it is often difficult to collect a sufficient amount of this low- level-detailed analyzer data to be correlated with source code execution to pinpoint root cause analysis information. This is particularly true for complex input/output protocols such as SCSI. Another disadvantage of the hardware logic analyzer is the high cost of the analyzer.

Software debuggers are programs which aid the kernel integration engineer by providing breakpoints in code execution and dump routines. Typically, software debuggers disassemble, decode and/or interpret code. They also allow variables to be examined so that the values of these variables and the like may be checked against expected values. While debuggers provide a good correlation between analysis data and source code, they are extremely invasive, resulting in decreased real-time execution timing and speed. Thus, hardware integration problems often will not manifest themselves while the code is being debugged.

A third tool available to kernel integration engineers is simple event logging. This process involves adding explicit event log points to the source code in order to store useful software state and path flow information. During code execution, event data is accumulated in an event log which holds data for the last N events. One problem with this scheme is that the appropriate size of the event log is difficult to determine. Typically, the log is a circular buffer; hence as the N+1 event is logged, the first event is pushed out. Increasing the size of the event log consumes memory. Decreasing the event log may result in critical data being pushed out of the log. Another problem with simple event logging is determining the number of log points. Selecting too many log points is invasive and will slow down the software. This also leads to the possibility of pushing important data out of the log. Selecting too few log points may result in missing critical dam. Multi-tasking systems with re-entrant drivers introduce another problem: events resulting from control of different hardware in multi-tasking systems with re-entrant drivers will often push out critical data previously logged.

The present invention provides a system and method for analyzing software problems by providing cross-triggering capabilities in a software logic analyzer. The present method incorporates hardware logic analyzer concepts such as event limiting, event triggering and cross-triggering via a user program interface into an event log process. In addition to limiting and triggering, the event log process utilizes symbolic decoding to ensure important log information is captured with minimal memory space.

A portion of the present method resides in the kernel of the operating system. This kernel log machine collects predetermined events identified by a kernel engineer. Event limiting and event triggering are additional capabilities provided to more efficiently use the event log buffer by storing only relevant events. A cross-triggering capability is also provided to facilitate simultaneous use of multiple analysis tools for debugging particularly difficult problems. A cross-trigger port is provided into and out from the kernel log machine.

The cross-trigger port permits the software logic analyzer to be connected to other analysis tools, such as hardware logic analyzers, oscilloscopes and other software logic analyzers. The cross-trigger port is bi-directional and can be created by using an I/O signal path from one of the I/O interfaces of the system comprising the kernel. The cross-triggering capability is non-invasive due to the synchronous nature of event logging: trigger input ports can be simply polled when events are logged and trigger output ports can be simply written to when the kernel log machine changes states.

Another portion of the present method resides outside the kernel of the operating system. This portion includes a log interface and log decoder. The log interface permits the kernel engineer to provide event limits and triggers. The log decoder converts the event log to a more readable format.

A preferred embodiment of the present method involves the small computer system interface (SCSI) subsystem. SCSI subsystem logging provides a valuable tool to analyze hardware integration problems. Explicit log points are added to the source code during system development. Critical log information (e.g., including time, device, source file, source line, functional sub-part, parameter name, parameter value and comments) is encoded into 16 bytes. The log decoder capability is provided to transform the log information into an easily readable format.

FIGS. 1a-b, shows a block diagram of a computer memory system incorporating the present invention.

FIG. 2 shows an overview of the software of the present invention.

FIG. 3 shows a simplified flow chart of the overall flow of the present method for software logic analysis.

FIG. 4 shows a more detailed flow chart of the overall flow of the present method.

FIG. 5 shows a block diagram of the software logic analyzer.

FIG. 6 shows a block diagram of a general purpose digital computer system incorporating the present invention.

FIG. 7 shows a flow diagram of the kernel log machine according to the present invention.

FIG. 8 is a continuation of FIG. 7.

The present invention is a system and method for analyzing software problems by providing enhanced capabilities in a software logic analyzer. By incorporating hardware logic analyzer techniques in a software event logging capability, dramatic improvement in event logging can be achieved. Specifically, event limiting and triggering techniques are employed to provide a more effective and efficient logging function. A cross-triggering capability is also provided via a cross trigger port into and out from a kernel log machine. These added capabilities provide a kernel engineer with the ability to focus on the debugging process and achieve a closer correlation of hardware and software behavior.

The present invention provides a means to analyze a specific software process, known as the code of interest. A kernel engineer, or other end-user, can set predetermined trigger and limit conditions via a user-interface located within a user process of an operating system. These predetermined conditions act as a filter for the explicit log points placed within the code of interest prior to initializing the logging function. A capture and decoding facility is provided to transform the event log into a more readable decoded report.

FIG. 1a shows a block diagram a general purpose digital computer system 10 incorporating the present invention. A processing unit 20 is connected to system bus 15. The system bus 15 facilitates communications between the processing unit 20 and a data storage disk 30 and an input/output (I/O) interface device 40. A memory 100, also connected to the system bus 15, stores the software of the present invention as well as all data collected and generated by the present invention. The input/output interface device 40 controls data communications between the bus 15 and various devices (e.g., device 1, device 2, and device N). For illustrative purposes, these devices may be a display mechanism 42, an analysis tool 44 and a keyboard 46. Any number of devices may be attached to the system 10 via the I/O interface 40, and these devices may be considered a integral part of the system 10, or a mechanism that can be attached to the system 10 from the outside, such as a printer or an analysis tool.

FIG. 1b shows a block diagram of the computer memory system incorporating the present invention. The memory unit 100 comprises a kernel 110, a shell 120, a task 130 and a log decoder 140. The kernel 110 and shell 120 form part of a multi-tasking operating system. In a preferred embodiment this operating system is HP-UX available from Hewlett-Packard Company, California, USA. The kernel 110 further comprises a kernel log machine 115 and a kernel log machine interface 117. The memory unit 100 can communicate with the remaining computer system using a computer bus (not shown). In a preferred embodiment the memory unit 100 is a random-access memory having at least 8 Megabytes (MB) of space.

FIG. 2 shows an overview of the software of the present invention. The software in a computer system 200 comprises a kernel 210, a shell 220 and an applications process 230. The kernel 210 controls hardware interfaces and devices, schedules tasks and oversees data storage. To accomplish this, the kernel 210 further comprises a kernel log machine 215, a kernel log machine interface 217 and device drivers 219. The system software 200 performs other functions which are not part of the present invention.

The shell 220, also known as a command parser, provides an interface between an end-user and the kernel 210. Basically, the shell 220 is one of the many processes which runs on the system 200. The shell 220 is unique in that it collects command arguments from the end-user and launches other processes to execute those commands. An event log interface 225 is one of the processes which can be launched by the shell 220. The applications processes 230 comprise various end-user tools and applications such as a data management program, computer graphics package and financial spreadsheets. These are represented as tasks 1 through N (items 232, 234, 236) and can be launched by the shell 220. A log decoder 240 also resides within the applications processes 230.

FIG. 3 shows a simplified flow chart of the overall flow of the event log interface for a commonly used analysis mode using the present method for software logic analysis. Due to the large number of commands available to the end-user (see Appendix A), a simplified flow chart is used to describe a typical overall method for software logic analysis. An end-user first configures 310 the log controller to capture events based upon predetermined conditions. Once the event log configuration is in place, the software logic analysis may begin 320. A wait 330 is inserted to allow the analyzer to continue processing until a predetermined condition is met, at which time the analyzer stops and the event log is dumped 340. Other modes can be used in conjunction with the present method without departing from the scope and spirit of the present invention. For example, a reconfigure step could be inserted after the wait 330. Multiple reconfigure-run-wait routines could also be utilized in this position.

FIG. 4 shows a more detailed flow chart of the sample measurement presented in FIG. 3. Block 412 permits the end-user to write a trigger condition. A trigger condition can be some predetermined condition that, upon occurrence, halts the kernel log machine in the present method immediately or after some number of additional events. For example, a trigger condition can be set to halt logging events when a particular routine in the code of interest is called.

Block 414 permits the end-user to write a limit condition. Limit conditions act as filters to ensure that only log points which satisfy a predetermined condition are logged. For example, a limit condition can be set so that the kernel log machine only captures each instance of opening or closing of a device driver. Other trigger conditions and limit conditions are available and are listed in Appendix A. While it is often desirable to set trigger and limit conditions, the present method does not require that these conditions be written; event logging may occur without triggers or limits.

Once the trigger and limit conditions are set, block 422 commences analyzer measurements. This is simply the process of logging the predetermined log points selected by the end-user based upon the trigger and limit conditions (i.e., the kernel log machine is started). A wait state is inserted via block 432 which permits the analyzer to continue collecting the log points until a trigger condition is met or until the end-user halts the kernel log machine. After the kernel log machine stops, blocks 442, 444 and 446 read, decode and print the event log respectively.

FIG. 5 shows a block diagram of the software logic analyzer. The software logic analyzer 500 essentially incorporates hardware logic analyzer concepts into an event logging process. A program 505 contains a plurality of procedures (e.g., PROC 1 through PROC 4); program 505 is the code of interest. Inserted throughout the program 505 are explicit log points (e.g., LOG PT. 1 through LOG PT. 4). These log points may be placed anywhere the end-user deems necessary. Log points may be placed before or after a procedure, as well as within a procedure. A log point is a data value; generally, log points are simply values of process variables. An example of a log point is the tape head position of a tape driver. In a preferred embodiment, these log points are inserted during code development by a source code developer. This method is advantageous since the developer may preselect important analysis data and decoding information. In another preferred embodiment, the log points are automatically inserted by a preprocessing or compilation tool.

A kernel log machine interface 510 provides a capability to an end-user to control and decode the event log via a kernel log machine 530. Besides providing run/stop control 522 of the kernel log machine, the end-user may select a specified event trigger 520 which, upon occurrence of that condition, will disable event logging. An event limit 524 may be introduced which will cause only those events which satisfy the limit condition to be logged. Additionally, the kernel log machine interface 510 provides the end-user with the capability to decode the event log and format the printing. The kernel log machine 530 uses the trigger and limit conditions written by the end-user to collect the log points, or raw event data 526, for the event log. The operation of the kernel log machine is discussed more fully below with respect to FIGS. 7 and 8.

A log decoding data extraction process 540, in conjunction with log decoding data 528, will decode the event log into a readable format and output the decoded event log 550 to the display screen or to a memory location. A preferred embodiment of the log decoding data is listed in Appendix B.

A trigger port 560 is provided into the kernel log machine 530. The trigger port 560 permits the software logic analyzer 500 to more effectively participate in analysis configurations involving multiple tools. The trigger port 560 is bidirectional; that is, the trigger port 560 is both an input port and an output port. Connected to the trigger port 560 is an external analyzer 565. The external analyzer 565 may be a hardware logic analyzer, oscilloscope or any other test and measurement device. In a preferred embodiment, the external analyzer 565 is an HP B3740A software analyzer manufactured and made available by Hewlett-Packard Company, California, USA. In another preferred embodiment, the external analyzer 565 is an HP 546xxB series oscilloscope manufactured and made available by Hewlett-Packard Company, California, USA. Additionally, the external analyzer 565 may be another software logic analyzer located within a different digital computer system or a different I/O subsystem of the same digital computer system (item 10 in FIG. 1a).

In one preferred embodiment, the trigger port 560 can be implemented via any one of a plurality of I/O signal lines within the I/O interface (item 40 in FIG. 1a). For example, a SCSI software logic analyzer may use signal lines in a parallel or serial interface on the same system to provide a hardware trigger port 560. Additionally, a dedicated special-purpose trigger interface may be used. When the trigger port 560 is being used as a trigger input port, the external analyzer 565 may arm (i.e., start) or trigger (i.e., stop) the analyzer 500. When the trigger port 560 is being used as a trigger output port, the analyzer 500 can send a message to the external analyzer 565, the message being indicative of when the measurement is complete (i.e., the trigger condition has been met). Additionally, when the trigger port 560 is configured as an output port, the external analyzer 565 can download data from the I/O interface. In another preferred embodiment, the trigger port 560 can be implemented in either shared or global memory. In this embodiment, the external analyzer 565 is another software logic analyzer which may be located in another subsystem within the computer system (item 10 in FIG. 1a) or in a separate system.

FIG. 6 shows a preferred embodiment block diagram of the general purpose digital computer system of FIG. 1a. A small computer system interface (SCSI) interface 630 is connected to the system bus 15. The SCSI interface 630 connects a variety of SCSI devices (items 632, 634 and 636) to the system bus 605. One such device that can be connected to the system 10 via the SCSI interface 630 is a hardware logic analyzer; another such device is an oscilloscope. The SCSI interface 630 uses a byte-wide or a multi-byte bi-directional parallel protocol with handshaking. In a preferred embodiment, program 505 (see FIG. 5) controls the SCSI interface 630.

FIGS. 7-8 show a flow diagram of the kernel log machine according to the present invention. Essentially, FIGS. 7-8 run in parallel to the wait process described with respect to block 330 in FIG. 3 and block 432 in FIG. 4. Block 710 initiates a call to the kernel log machine (item 530 in FIG. 5). When a log point is encountered in the code of interest, control is passed to decisional block 712 to determine whether the kernel log machine is running. If it is not running, control is returned 714 to the main program (i.e., the code of interest) to continue code execution. If the kernel log machine is running, control is passed to decisional block 716. Decisional block 716 determines whether the kernel log machine has set the STATE to `ARMED` for the external trigger input (item 560 in FIG. 5). If the STATE is ARMED for the external analyzer, control is passed to decisional block 718 which determines whether the external trigger has been met. If the STATE is not ARMED, then control passes to decisional block 722.

Decisional block 722 determines whether an internal trigger condition is present and, if a condition is present (see item 412 in FIG. 4 and items 510, 520 in FIG. 5), control is passed to decisional block 724 to determine if the trigger condition has been satisfied. If no internal trigger condition has been written, or if the trigger condition has not yet been satisfied, control is passed through item A to enter the method in FIG. 8.

If the trigger condition has been satisfied, block 726 sets the state of the kernel log machine to "TRIGGERED." Control is then passed to decisional block 728 which determines whether the external trigger output has been enabled. If the output has been enabled, block 730 sets the trigger output signal before passing control to block 732. The trigger output signal is set to indicate a trigger condition depending on how the external logic analyzer is configured. Thus, the exact value to which the trigger output signal is set depends upon the trigger configuration. For example, the signal could be set to a high or low logic level. If the external trigger output is not enabled, control is passed to block 732 which saves the data in the event log before going to decisional block 838 (item B, FIG. 8).

Returning to decisional block 718, if the external trigger condition has been met, block 720 resets the STATE equal to `NOT ARMED` before passing control to block 732 which saves the data in the event log. If the external trigger condition has not been met, control is passed to decisional block 830 (item A, FIG. 8).

Referring now to FIG. 8, decisional block 830 determines whether a limit condition is present (see item 414 in FIG. 4 and items 510, 524 in FIG. 5). If there is a limit condition present, then decisional block 832 determines whether the limit condition has been satisfied and, if the limit hasn't yet been satisfied, transfers control to block 850 to return to the code of interest. If the limit condition is satisfied, or if no limit condition was written, block 834 saves the data in the event log.

After the data is stored in the event log, decisional block 836 determines whether the kernel log machine is in a TRIGGERED state. A non-TRIGGERED state lets the kernel log machine to return 850 control to the code of interest. A TRIGGERED state brings the method to item B.

Decisional block 838 determines whether post-trigger events have been stored. Post-trigger events are a predetermined number of events to log after the trigger condition is met. This permits the end-user to examine events surrounding the trigger condition (i.e., just prior to and immediately following). If post-trigger events have not been or do not need to be stored, the method returns control to the code of interest (item 850). If the post-trigger events have been stored, block 840 resets the kernel log machine state to "STOP" before returning to the code of interest.

While the present invention has been illustrated and described in connection with the preferred embodiment, it is not to be limited to the particular structure shown. It should be understood by those skilled in the art that various changes and modifications may be made within the purview of the appended claims without departing from the spirit and scope of the invention in its broader aspects. ##SPC1##