Test executive system with progress window

[0002] 1. Field of the Invention

[0003] The present invention relates to electronic systems for performing automated tests of complex electronic, electromechanical and mechanical equipment and products. More particularly, the present invention relates to an electronic test system with a progress window that enables the user to see in a glance the progress and results of an entire test.

[0004] 2. Statement of the Problem

[0005] Complex electronic, electromechanical and mechanical products and equipment are generally tested using automated test systems. Such tests can include validation tests which run through the various operations that the device under test (DUT) is capable of and records whether each operation was performed properly; environmental tests which expose the DUT to various combinations of temperature, pressure, and humidity, and records the results; production tests, etc. Generally, both the DUT and the systems providing the environmental and other constraints on the DUT are controlled electronically. In the last decade or so, computerized programs which are capable of controlling a variety of automated tests, referred to in the art as “test executive” programs, have been developed.

[0006] Test executive programs in the prior art include internal test executive programs developed by Agilent Technologies and TESTSTAND software developed by National Instruments Corporation, which is described as a ready-to-run test executive for organizing, controlling, and executing automated prototype, validation, or production test systems. The prior art Agilent Technologies programs did not use a graphical user interface (GUI), therefore limiting the ability of the program to display large amounts of data in a simple fashion. The TESTSTAND software, while using a GUI, requires the user to scroll through multiple windows to determine the overall progress of a test.

[0007] Tests usually are defined by a set of rules or specifications to which the DUT is compared. The rules or specifications generally comprise various inputs defined by electrical and mechanical parameters applied to the DUT, such as voltage, current, specified manipulations of controls and device parts, as well as environmental parameters under which the test is conducted, such as temperature, humidity, pressure, and the time period over which a parameter is applied. Each test will include many combinations of the parameters applied to each element of the DUT, and often will be repeated many times. Each combination of parameters will define a measurement that results in one or more datapoints, which are recorded and compared to numerical or Boolean limits defining the specifications. Thus, as equipment and products become more complex, electronic test programs have become very long and complex, often requiring several days, or even a week or more to run a complete test.

[0008] In prior art test systems, the test results are displayed as text on a computer display screen as the test progresses, with the current test conditions and results, or a small portion of the conditions and results just prior to the current time, visible. If the user desires to see results from earlier portions of the test, the user scrolls through a display to extract information on the status of the test. Because of the length and complexity of the test, it is not physically possible to display the entire report of a test on the screen at one time. At any given point in the scrolling, the user has only a partial view of the results, making it difficult for the user to draw conclusions on the overall progress of the test without time consuming and tedious recording and summarizing.

[0009] In actual practice, the user almost always will permit the test to run unattended for hours, overnight, or for days, while other work is attended to. Because of the length and complexity of the tests, when the user returns to check on the test, it takes considerable time and effort to review the results to determine the progress of the test. In such cases, it often happens that the time necessary to do a thorough analysis of the results is not available as the test is being run. As a result, much test time is wasted when test results are reviewed after a run is completed, and it is found that certain elements of the test were improperly set up, or the DUT was faulty in some respect that was not recognized during the test.

[0010] It would be highly desirable to have a test executive system in which, at any given point in a lengthy test, the overall progress and status of the test could be quickly determined.

[0011] The present invention solves the above and other problems in prior art by allowing the user to see the progress of the entire procedure on a small graphical display called a progress window. The progress window enables the user to evaluate an entire test in a single glance while the test is still running. This allows the user to save time by immediately seeing if the number of failed or false results is unacceptable. The user does not have to search through the whole listing of results to determine if the test is running properly. Tests that are not progressing properly according to a set of rules, or specifications, predetermined by the user are quickly identified.

[0012] The electronic test output is displayed through an easy to use graphical user interface (GUI). It is preferably divided into windows containing either icons for controlling the test procedure or windows for displaying the results as text, tables, graphical points or graphical colored bars, and other GUI elements. The user can control the test procedure by accessing the programs represented by the icons and windows with a pointing device such as a mouse, a keyboard, a trackball, a touch pad, a roller ball and a joystick.

[0013] The portion of the display containing the progress window is preferably a small portion of the graphical user interface (GUI) where the results are displayed as single graphical elements such as graphical points, graphical bars, graphical sectors or single alphanumeric characters.

[0014] The graphical elements are grouped in a manner to display the progress of the test. Preferably, the graphical elements are grouped on this window along a first axis, and the distance of these graphical elements extends along the first axis representing the total length of the whole test procedure.

[0015] Preferably, each graphical element is color coded to facilitate rapid distinguishing of the test progress. Preferably, the color coding indicates that the corresponding result is a “pass”, i.e., within specification, a “fail”, i.e., outside specification, or “marginal”, i.e., just barely within specification.

[0016] Preferably, each graphical element is a bar extending in a direction perpendicular to the first axis. Preferably, each bar is color coded to indicate if the corresponding result is a “pass”, i.e., within specification, a “fail”, i.e., outside specification, or “marginal”, i.e., just barely within specification. Monitoring the progress window allows the user to check the time left to accomplish the test procedure but, more importantly, it allows the user to check the status of the test; that is, how many of the results are pass, fail or marginal.

[0017] The pass/fail criterion is determined by comparing the results to a specification. Preferably, the specification is displayed on the progress window by at least one specification limit. Most preferably, there are two specification limits, which are preferably an upper specification limit and a lower specification limit. Preferably, each specification limit is a line. Preferably, the upper specification limit is a line above and parallel to the first axis. Preferably, the lower specification limit is a line below and parallel to the second axis.

[0018] Preferably, the length of each bar is normalized so it is proportional to the ratio of the numerical value of the datapoint and the numerical value of the corresponding specification. A mathematical criterion is applied to determine if the result is within the specification limits, outside the specification limits, or barely inside the specification limits, which criterion determines if the test is pass, fail or marginal. As indicated above, preferably, a color is assigned to the vertical graphical bar, representing the value of the datapoint, depending on its pass, fail or marginal status. This color code is a visual tool helping the user to determine instantly how the test is progressing. Preferably, the lines representing the upper and lower specification limits are placed at a unit (1) distance above and below the first axis, respectively. Thus, if the data point is within specification, its length is such that it is within the area defined by the specification limit lines, and if the data point is outside specification, its length extends beyond the corresponding specification limit line, and if the datapoint is marginal, its length is equal to or close to the corresponding specification limit line.

[0019] The present invention not only provides an electronic test program that displays the overall output in a form that can be observed in a glance, but also displays the output in a form which permits the user to quickly check individual results. Other features, objects and advantages of the invention will become apparent from the following description when read in conjunction with the accompanying drawings.

[0020] FIG. 1 is a block diagram showing the principal hardware components of the preferred embodiment of the present invention connected to a device under test;

[0021] FIG. 2 is a block diagram showing the hierarchical structure of the preferred embodiment of a test program according to the invention;

[0022] FIG. 3 shows a view of the graphical user interface of the electronic test system;

[0023] FIG. 4 shows a progress window according to the invention;

[0024] FIG. 5 shows an alternative embodiment of a progress window according to the invention; and

[0025] FIG. 6 shows a flow chart illustrating the determination of the pass, fail, or marginal status for each graphical element for a single-sided specification.

[0026] The present invention relates to an electronic test system with a progress window that enables the user to observe in one glance the progress of the test. Referring to FIG. 1, in the preferred embodiment, the test system preferably comprises a computer 100. Computer system 100 includes a memory 101, a microprocessor 102, an input device 104, and an output device 106. Memory 101 communicates with microprocessor 102 via electrical line 110. Input device 104 communicates with microprocessor 102 through electrical line 112. Microprocessor 102 outputs the data through output device 106 via electrical line 114.

[0027] In another embodiment, the test system can take the form of software stored in memory 101 and run or executed by processor 102. The user interacts with the test system through input device 104 such as, but not limited to, a keyboard, a mouse, a track ball, a touch pad, and a joystick. The input device 104 allows moving a cursor or a pointer on the output device 106 (a display system such as a Cathode Ray Tube monitor or a Liquid Crystal Display). The results of the test are displayed on the output device 106. The test is controlled by processor 102, which communicates the instructions of the test program to the product under test or device under test (DUT) 108 via electrical line 116. Processor 102 controls the test equipment 117 via electrical line 118. The test results are processed by processor 102 and stored in memory 101 to be visualized by display 106. The display information contains both information as to the configuration and position of the cursor and other visual information of interest to the user such as the results of the test and how the test is progressing.

[0028] The invention may be implemented in a variety of actual electronic devices that follow the general form outlined on FIG. 1. For example, the test system may be embodied in a computer system or it may also be embodied in other hardware as a logic circuit, embodied as an electronic testing device such as, but not limited to, an electronic analyzer.

[0029] To better understand the workings of the invention, it is helpful to describe the hierarchical structure of the preferred test program of the present invention, and the order in which the test is performed. Referring to FIG. 2, there is shown a block diagram 200 illustrating the hierarchical, i.e., multi-level, characteristic of the test program. The first level 201 corresponds to the product model, a file the test developer creates to test a family of specific device model numbers. It contains test procedures and inputs.

[0030] The next level 202 corresponds to the procedure itself. A procedure is an ordered list, sequence, or script of tests to be run. Several procedures may exist, which is represented in FIG. 2 by a deck of cards 202, each representing a different procedure. Each procedure includes a plurality of tests, i.e., test 1, test 2, test 3 . . . test N, as shown at 203. Each test includes a plurality of measurements. This is illustrated in FIG. 2 for test 2 shown at 205. As shown, test 205 includes measurements 207, i.e., measurement 1, measurement 2 . . . measurement N. Each measurement includes one or more datapoints, represented as a deck of cards 210, 211, 212, associated with each measurement. A procedure is defined by writing a program or a code to build a structure of software objects. In one embodiment, the software objects are Component Object Model (COM) objects. COM is a language independent component architecture, not a programming language. It is meant to be a general purpose, object-oriented means to encapsulate commonly used functions and services. See Newton's Telecom Dictionary by Harry Newton, Publishers Group West, page 197.

[0031] A test 205 is a group of measurements 207 in a procedure 202 that share the same test algorithm or the same test software code. Some examples of tests include amplitude accuracy test, test of harmonic distortion, etc. The test program repeatedly calls a test for each measurement and datapoint.

[0032] A measurement, such as measurement 206, is a configuration or a set up for a test. Each measurement, in measurements 207, within a test 205 can have different setups or configuration parameters. Tests are parameter driven and the parameters are inputs at the measurement level. Measurements are elements such as range in volts, frequency in kilohertz or harmonic (an integer number). The test procedure 202 views the measurements 207 as data to be passed from the procedure to a test. A measurement is also a phase of test execution. During the measurement phase of test execution, the measurement is started but data is not collected. This allows for multiple devices under test (DUTs) to be configured and triggered together.

[0033] A datapoint such as 210, 211, 212, is a subset of a measurement, such as 206, containing additional parameters that select a result when one measurement generates multiple results. Some examples of multiple datapoints for a measurement are the minimum and maximum of a spectrum analyzer sweep or each channel of a device.

[0034] For each datapoint, such as 210, in measurement 206, a value result is extracted. The results obtained are compared to specifications. Specifications are numerical limits, string match, or Boolean pass/fail. There are three sets of limits: marginal limits, line limits and customer limits. Each limit has an upper and a lower value.

[0035] All the inputs and outputs of the electronic test system (FIG. 1) are handled preferably through a graphical user interface (GUI). FIG. 3 shows the graphical user interface 300 as displayed on the output device 106. The tape recorder buttons 301 on the top left are used to control the test. From left to right: abort 302, restart test 303, restart measurement 304, pause 305, run 306, skip measurement 307, skip test 308.

[0036] The bottom of the screen 309 shows a window describing the hierarchy of tests, measurements, and datapoints. Icons, such as 313, indicate pass, fail, marginal and not yet tested. A “smiling” 310, “surprised” 311, and “sad” 312 icon correspond respectively to pass, marginal and fail. The pass/fail icon 313 on top of the window is for the entire procedure. It summarizes the status of all the tests with the fail having priority. That is, if there is one fail test the whole procedure is fail. An algorithm that promotes the least optimal result is used to compute the final status of the procedure.

[0037] On the right side of the screen 300, a window 314 shows a series of rows 315 and columns 316 of alphanumeric test information. Window 314 displays the time the test is run, such as at 317, as well as the status of the test, such as at 318. Window 314 also displays the test that is being carried out, such as amplitude accuracy at 319, the type of measurement, such as (range=5 Vp, Frequency=1 kHz) at 320, the datapoint or channel under test, such as (Ch=1, Ch=2, Ch=3) at 321, the value or result of the measurement, such as (0.1235 dB) at 322, the specification, such as (±0.2) at 323, and the frequency, such as (1 kHz) at 324. The test can be sorted by clicking on the icon view 325 on top of the window 314. This function allows the test results to be filtered and displayed depending on their status.

[0038] The bottom right window 330 represents the preferred embodiment of a progress window according to the invention. FIG. 4 shows an enlarged view of this window. Referring to FIG. 4, axis 400 represents the length of the entire procedure. Datapoints corresponding to the test results are preferably plotted as datapoint units 410 along a progress axis 400. Each datapoint unit 410 preferably is a single graphical element, such as 402, 403, 404, etc. Preferably, each graphical element comprises a bar. The length of each bar in the direction perpendicular to the progress axis 400 is normalized to “1”. That is, the vertical length of the bar is proportional to the ratio of results numerical value to specification numerical value. The progress window 330 also preferably includes specification limit elements 401A and 401B. Preferably, the specification limit elements 401A and 401B comprise dotted horizontal lines 401A and 401B, with 401A indicating the upper limit and 401B indicating the lower limit. That is, the line 401A above the axis 400 corresponds to the upper specification limit and the line 401B below the axis 400 corresponds to the lower specification limit. Since the length of the bars are normalized, the specification lines 401A and 401B are preferably placed at a unit (1) distance above and below axis 400. Thus, if the bar representing the result is within specification, that is, its length is such that it is within the area defined by the specification limit lines, such as 404, the result is pass. If the bar representing the result is outside specification, that is, its length extends beyond the corresponding specification limit line, such as 409, the result is fail. If the bar representing the result is barely inside the specification limits, that is, its length is equal to or close to the corresponding specification limit line, such as 403, the result is marginal.

[0039] Preferably, each graphical element is also color coded to indicate the pass 404, fail 409 or marginal 403 status. For example, the color red is chosen for fail, the color green for pass and the color yellow for marginal. Preferably, the graphical elements are colored bars 402, 403, etc. Thus, each datapoint results in a colored vertical bar, such as 402, 403, etc. In FIG. 4, the graphical elements 410 are lined for color. As shown at 440, a hatched lining indicates a yellow color; as shown at 441, a diagonal lining indicates a green color; and, as shown at 442, a vertical lining indicates a red color. Though the bars are preferably color coded to indicate the status of the test measurements, the color code is not essential since the user can distinguish the status of each measurement by looking at the length of each bar relative to the specification lines. However, adding a color to the bar facilitates the determination of the status of each test result.

[0040] An alternative embodiment of the progress window 500, according to the invention, is shown in FIG. 5. Referring to FIG. 5, the progress window is comprised of graphical elements, such as 504. The graphical elements are grouped in a circle 501 using radial coordinates. The radius, such as 505, for each element, such as 504, corresponds to the numerical value of the result, and the angle, such as 503, conveys the time at which the result was measured. Each datapoint results in a sector, such as 504, wherein the radius of that sector is proportional to the ratio of results numerical value to specification numerical value. The specification limit 520 is preferably plotted as a dotted circle inside circle 501 defining the progress window. The sectors, corresponding to the datapoints or results, are preferably color coded or design coded to show the pass, fail and marginal status of each result. Again, the graphical elements, such as 505, are lined for color and a reference chart 530 is provided in the figure to explain the lining. The color or the design feature is only intended as a visual tool to help the user distinguish the status of each test result since the user can determine the status of each result by observing if the sector is inside, barely inside or outside specification circle 520. However, each sector, corresponding to the value result, is preferably assigned a color or a design according to the length of its radius by applying a mathematical criterion. For example, if the radius of the sector corresponding to the datapoint is greater than the radius of the circle corresponding to the specification, the sector is assigned a design or color so as to display that the result is “fail”. If the radius of the sector corresponding to the datapoint is less than the radius of the circle corresponding to the specification, the sector is assigned a design or color so as to display that the result is “pass”. If the radius of the sector corresponding to the datapoint is equal or very close to the radius of the circle corresponding to the specification, the sector is assigned a design or color so as to display that the result is “marginal”.

[0041] The progress window 500 not only allows the user to distinguish between possible outcomes of test results; it also permits the user to check the time left to accomplish the test procedure by observing how much of the area inside the circle 501 is covered by the sectors corresponding to the results. The window 500 also includes a start line 512 and progress lines 513, 514, and 515 which divide the window into quadrants to assist in quickly showing how much of the test is completed.

[0042] Referring to FIG. 6, there is shown a flow chart 600 illustrating the determination of the status of a result and the normalization process. The inputs used to determine the status include the value (V) 601 for the result, the upper specification limit (U) 602, the lower specification limit (L) 603, the upper marginal limit (UM) 604, and the lower marginal limit (LM) 605. The value (V), which is determined by the test measurement, and the limits U, L, UM, and LM, which are entered by the user, are stored in memory 101 (FIG. 1). In step 608, the test system 100 determines if V is less than or equal to U. If it is not, then the system assigns a fail status to the result at step 609. If it is, the system goes to step 610 and determines if V is greater than or equal to L. If it is not, the system assigns a fail status to the result at step 611. If it is, the system proceeds to step 612, where it determines if V is less than or equal to UM. If it is not, the system assigns a marginal status to the result in step 613. If it is, the system proceeds to step 614, where the system determines if V is greater than or equal to LM. If it is not, in step 615, the system assigns a marginal status to the result. If it is, the system proceeds to step 617 and assigns a pass status to the result.

[0043] The ability for the user to distinguish easily in one glance at the progress bar if the test is pass, fail or marginal is a feature of the invention. All the information on the test results is available on one window. The user may take immediate action as to whether to let the test progress or terminate the test and modify some of the parameters.

[0044] For instance, if a series of fails are observed, that is, if too many red bars are seen on the progress bar such as in situation 430 in FIG. 4, the user may take action and stop the test. If there are only a few marginal results with most of the results being pass, that is, green bars are in the majority, such as in situation 420 on FIG. 4, the user may decide to let the test continue.

[0045] There has been described what at present is considered to be the preferred embodiment of the invention. It will be understood that the invention can be embodied in other specific forms without departing from its spirit or essential characteristics. For example, while the invention has been described in terms of an electronic test program, other systems may be implemented based on hardware instead of software. The program may, for instance, sit on a programmed processor or other platform. In addition, more than one progress window may be added to the graphical user interface to display the test results of two or more devices under test. The progress window may be displayed in other forms where the graphical elements may take a different form, such as a single alphanumeric element, or may be grouped in a different manner. The present embodiment is, therefore, to be considered as illustrative and not restrictive. The scope of the invention is indicated by the appended claims.