Method and Apparatus for Enhancing Computer-User Selection of Computer-Displayed Objects Through Dynamic Selection Area and Constant Visual Feedback

Fitts, P. M. 1954. "The Information Capacity of the Human Motor System in Controlling Amplitude of Movement," Journal of Experimental Psychology, vol. 47, pp. 381-391.

Greenstein, J. S. and L. Y. Arnaut. 1988. "Input Devices," Handbook of Human Computer Interaction. M. Helander, ed. New York: Elsevier.

Welford, A. T. 1968. Fundamentals of Skill. London: Methuen.

1. Field of the Invention

This invention relates to computers having a visual display and a user input pointing device such as a mouse, trackball, touchscreen, joy stick, etc. More specifically, the invention pertains to a method and apparatus for enhancing a computer-user's selection of objects displayed on his computer-screen.

2. Description of the Related Art

Current user-interface designs typically require a computer-user to place a cursor directly upon an object to be selected. Once the user locates the object he desires to select, the user must move his cursor the entire length from its instant location to the object to be selected. Once the cursor and object overlap, the operator provides an input, such as through a mouse, trackball, joy stick, etc., indicating that the overlapped object is to be selected. Visual feedback is presented only after the user performs an action to select the object. That is, it is only after the user tries to select an object does he know whether he has hit or missed the object he desires to select.

Thus, in a typical case, a user must move his cursor to an object to be selected, estimate that his cursor appropriately overlaps the limited area directly over an object to be selected, and then must make a selection action. Visual confirmation of the success of the selection action is received by the operator only after the selection action has taken place.

The United States Navy has been and is currently, investigating the use of nonconventional control-display configurations for use in future console designs. A primary user task at these consoles is the selection (called track "hooking") of displayed symbols/objects. The symbols represent real-world objects.

Other user tasks include the selection of various computer functions as well as making data entry into a computer. The Navy is also investigating the use of various computer input devices, and on-screen, direct manipulation methods, such as filling in pop-up menus to complete console tasks. These methods contrast with current use of programmable (variable label) function keys and cursor arrow keys.

The Navy envisions the need for a substantial increase in cursor placement tasks, especially in quick-response, weapon system control environments. User response speed and selection accuracy of symbol objects and other menu objects must be optimized in these environments.

In the past, researchers have studied cursor selection tasks for several input devices utilizing various control/display ratios, see Greenstein et al. referred to above. Yet design aspects related to visual feedback (object highlighting), as well as distance and size enhancements have not been well studied.

In general, symbol/object selection performance can be described by Welford's variation of Fitts's Law, see the Fitts and Welford references cited above.

Fitts's Law states that, within certain limits, distance moved to an object to be selected and object size affect selection performance time. It is known that cursor positioning time is affected by design factors such as control/display ratio, cursor velocity, and quality of visual feedback. One can imply from Fitts's Law that user-interface aids which decrease cursor travel and enhance the size of selected objects would improve object selection speed.

Yet for practical reasons, such as symbol clutter and overlap, computer-displayed objects must remain as small as possible, within limits prescribed by minimum visual requirements. For some displays, the distance between displayed objects will vary according to real-world perspective as well as user selection of display range-scale. Computer displayed objects may be tightly grouped and therefore be overlapping, or they may be spread about the entire computer display screen. The challenge proposed is to effectively decrease distance between objects and increase object size, without visually altering the size of the displayed objects (display symbology).

Object selection aids here disclosed were developed and applied to both symbol/object selection as well as pop-up menu object fill-in tasks. The selection aids of the invention reduce required cursor travel distance from the cursor to an object by increasing the effective selection area for an object. The object selection area is changed without altering the displayed size of either the symbology or cursor. Further, the selection areas surrounding objects may change dynamically with changes in the objects displayed. This permits selection optimization regardless of whether the arrangement of objects is pre-known.

The invention is accomplished through the constant computation of a "selectable" cursor target, defined as the object closest to the current cursor position on the computer-display. The invention identifies the "selectable" object by providing visual highlighting to the computer-user before a selection action is made.

A computer determination of which object lies closest to a computer-user controlled cursor is made for all displayed objects in real-time as a cursor is moved. If a cursor is equidistant from two or more objects, one of these objects is made "selectable" according to a decision making routine.

Contrary to prior art schemes, the "selection area" of an object is not limited to that directly over the object itself and is not limited to a predefined region. In the invention, the selection area for computer-displayed objects, such as symbols or pop-up menu items, is made as large as possible depending upon the arrangement of the objects on a screen. This selection area is computer determined and is defined as the area surrounding the object that is closest to that object as contrasted to being closest to another object. The selection area may dynamically change with changes in displayed object arrangements and does not depend upon any previous knowledge of object location.

Once an operator has placed his cursor in the area closest to an object he will at once be notified of this as the object will become visually identified as being "selectable." A selection confirmation action by the computer-user, such as the pressing of a mouse button, would select the "selectable" object. This action would be made known to the computer-user as the object will become visually identified as being "selected".

It is an object of the invention to enhance computer-user selection of computer-displayed objects.

Another object of the invention is to improve the speed and accuracy of computer-user selection of computer-displayed objects.

Another object of the invention is to effectively decrease the distance between computer-displayed objects without visually altering the displayed symbology of the objects.

A further object of the invention is to effectively increase computer-displayed object size and hence object selection area without visually altering the displayed symbology of the object.

Still a further object of the invention is to effectively increase computer-displayed object size and hence object selection area dynamically so that selection areas will change according to changes in the computer-displayed objects.

Yet another object of the invention is to improve visual feedback to a computer-user tasked with selecting computer-displayed objects.

Yet a further object of this invention is to indicate to a computer-user whether an object will be selected if he makes a selection decision.

Still another object of this invention is to indicate to a computer-user whether in fact a computer-displayed object has been selected.

Other objects, advantages and new features of this invention will become apparent from the following detailed description of the invention when considered in conjunction with the accompanied drawings.

FIG. 1 illustrates an exemplary technique that may be used to calculate the distance between two objects from a cursor. FIG. 1 also shows a "selectable" object visual indicator.

FIGS. 2 and 3 illustrate the enlarged selection area made possible through use of the invention.

FIG. 4 illustrates a visual indicator that may be used to show a "selected" object.

In FIG. 5 a representative pop-up menu is shown in which cross-hatching illustrates some of the enlarged object selection areas of this menu.

FIGS. 6A and 6B are a flow chart charting the operations of one implementation of the invention.

As previously discussed, the selection aid of the invention reduces required cursor travel distance to a cursor target by increasing the effective cursor target selection area when possible. This is done without enlarging the displayed size of either the symbology or the cursor. The selection aid of the invention is accomplished through the constant computation of a "selectable" object, which is the computer-displayed object lying closest to the cursor position on a computer-display. The invention identifies t he "selectable" object by providing a visual indication to the user before a selection action is made. Determination of a "selectable" symbol or menu item will now be discussed.

As shown i n FIG. 1, a computer is used to determine the x and y coordinates of objects with respect to cursor center location. These distances can be computed in conventional ways such as by the Pythagorean Theorem. Of course, for those objects that are horizontally or vertically aligned with the cursor, only the x or y coordinate, respectively, need be determined.

In the invention the distance to all displayed symbol-objects is computed in real-time as a cursor is moved. If the cursor is equidistant or overlapping two or more symbols, a decision making routine is used to highlight one of the objects. For example, the last closest symbol-object could be chosen for highlighting or whichever of the overlapping objects was displayed first could be chosen.

Referring once again to FIG. 1, it can be seen that the edge of object 2 is closer to the cursor. The dashed circle around this object indicates that it would be selected if a selection action is performed by a computer-user. This continuous visual feedback could be provided by a variety of means such as a color change, inverse video, border color change, etc.

In FIG. 2 objects 1 and 2 are shown with demarcation line 10 tracing a border equidistant between the objects.

Referring now to FIG. 3, the selection areas closest to objects 1 and 2 are shown separated by border 10. Cursor placement anywhere within the region closest to these objects will permit the corresponding object to be selected. According to the invention, the computer-user will be informed which object is closest to his cursor by a visual indicator marking the closest object as being "selectable". It should be noted that the selection areas shown in FIG. 3 will constantly change whenever the objects displayed change. That is if objects are added, subtracted or moved, the selection areas will be recomputed in real-time accordingly to adjust to the change in object display. In this regard the selection areas can be considered "dynamic" and have no need to rely upon previous knowledge of object location.

Referring now to FIG. 4, once a computer-user finds an object he desires to select, he moves the cursor anywhere within the selectable area of that object, in this case area 12, and then makes a confirmation action indicating that the "selectable" object is desired to be selected. The invention will then inform the user that such a selection has taken place.

Such selection action will be indicated to the user by the "selectable" visual indicator being changed to a "selected" visual indicator. In FIG. 4 a solid circle indicates that object 2 is currently selected. Of course other visual indications such as a color change, inverse video, border color change, etc., may be used to indicate the "selected" status of object 2.

Referring now to FIG. 5, a representative pop-up menu 14 is illustrated. The selection areas for computer-displayed objects of pop-up menu 14, such as menu item 1, menu item 2, menu item 3, etc. will be made as large as possible depending upon the location of these selection "items" with respect to each other. As shown in FIG. 5, the selection areas for two objects will differ depending upon their locations in relation to other displayed objects.

"Check Box 1" has a large selection area because it is at the end (top) of a list, while "Menu Item 2" has a smaller selection area, being in the middle of an object group. Highlighting of the "selectable" menu object can be done by placing a green square around the object, which turns to inverse video if the object is selected. Of course other schemes may be used. In FIG. 5 a solid border is placed around "Menu Item 2" to indicate that this object has been selected.

Referring now to the flow chart of FIGS. 6A and 6B, a representative decision-making scheme such as may be utilized in the invention is shown. Referring now to flow chart 16 of FIGS. 6A and 6B, an exemplary program begins by displaying a desired window area on a computer-screen (step 18). In step 20 objects are drawn on the computer-screen and the object positions are read. In step 22 the position of a user-controlled cursor is obtained. In step 24 the program computes the distance to all the displayed objects from the cursor. After calculating which object is closest to the cursor, a "selectable" object tag is placed around the closest object. This is shown as step 26.

The program now enters a loop that constantly monitors user inputs. The program then performs certain actions, depending upon what type of "event" is generated. In the enfolding description, event is meant to indicate a particular scenario that may take place.

The monitoring loop of the program is illustrated as step 28. In step 28, the machine (computer) is polled for the next occurring "event".

For the example program illustrated in FIGS. 6A and 6B, three different type events can take place.

A "null event" is an event generated when no other event has occurred. In step 30 a decision block is used to query whether this event has taken place. On this event, the computer reads the cursor position, step 32, and calculates which object is closest to the cursor, step 34. If the newest closest object is the same as the old closest object, step 36, the program goes back to the beginning of the event loop and asks for the next event, step 28. Otherwise, a query is made as to whether the closest object is the same as the currently "selected" tagged object, step 38. If the answer to this query is no, the "selectable" object tag is erased from around the old closest object, step 40, and in step 42 the "selectable" object tag is displayed around the new closest object and the "selected" object tag is displayed around the currently selected object if one exists. The program then goes back to the beginning of the event loop 28.

An "update event" is generated when something was previously obscuring the displayed window but has now moved, leaving a previously obscured portion of the window viewable. If the answer to this query, step 44, is positive, the computer simply redraws the area of the window that is now viewable including the objects, the "selectable" object tag and the "selected" object tag, shown as step 46.

A "selection event" is generated when the switch on an input device (mouse, trackball, touch panel, etc.) is actuated. This is shown as step 48. If there is such an event, the next query is whether there is a currently "selected" object, step 50. If there is not a currently "selected" object, the computer draws a "selected" object tag around the closest object. This is shown as step 52. The loop starting with step 28 begins once again. If there is a currently "selected" object, it must then be asked whether the currently selected object is the same as the closest object, step 54. If this is not the case, the program will erase the "selected" object tag from the previously "selected" object and draw the "selected" object tag around the closest object tag, this being shown as step 56. The program then enters the loop starting with step 28, in which the computer is polled for the next "event." If, in step 54, there is a currently "selected" object and the selected object is the same as the "selectable" (closest) object, the program will erase the "selected" object tag from the previously "selected" object and draw the "selectable" object tag around the closest object. The program will once again enter the loop starting with step 28.

Attached to this specification as an appendix is an object code listing of a computer program suitable for implementing the above described program functions. This "C" language program references commercially available software libraries known as "CChore" and "CCollaborator" copyrighted to the Symantec Corporation. It is to be understood, that this "C" language program is given by way of example and that one skilled in the art will realize that other programs may be utilized within the scope of the present invention to accomplish the tasks described herein.

Studies have been conducted to measure the effect of using the selection aid of the invention. These studies focused on the use of the invention with various computer-user input device configurations. Speed and accuracy of user performance for symbology selection and pop-up menu selection tasks were measured.

The mean selection times for input devices such as touchscreen, trademark "UNMOUSE", trackball and mouse, when used with the program of the invention, suggested high selection speed and low error rate. The use of the invention improved performance with all the input devices used. Pop-up menu selection accuracy was significantly increased for all input devices studied.

In the past symbol hooking on OJ-194 Navy trackball consoles resulted in relatively poor hooking performance. A personal computer trackball utilizing the invention demonstrated a substantial improvement in hooking speed without an increase in hooking errors. In addition to these benefits, the easier hooking implied that console users would not have to change their display range-scale as often to increase symbol spacing for hooking. The user would therefore not have to sacrifice seeing the "big picture", that is a view of a larger map range-scale, to select symbols which are closely positioned. Thus, existing Navy consoles can be significantly improved by incorporating the invention.

For all input devices tested, the invention improved speed of symbol selection for both relatively low and relatively high symbol densities and improved symbol selection accuracy under low-density symbol conditions. Low density in this case consisted of a test pattern of six 7-cm diameter symbols arranged in an elliptical pattern confined to a quarter of a 19-inch RADIUS color cathode ray tube. The high-density track pattern consisted of four separate approximately 25-cm symbol clusters containing two, three, and five adjacent symbols in close proximity.

A significant reduction of menu-item selection errors was also found to be shown with use of the invention. This is especially true with touchscreen and touchtablet technologies.

The increase in hooking performance speed and reduction (or no increase) in hooking errors using the method of the invention is considered to have important implications not only for Navy consoles, but for other cursor-pointing tasks.

Despite the continuous calculation of cursor to object distances, a particular implementation of the invention with an Apple Macintosh IIci exhibited no visible graphic updating delays, even when 200 tracks were placed on a screen during software debugging. For this implementation, the number of processor instructions per symbol was kept small (approximately 10.), and computer performance was not diminished even when performing calculations for many objects using a 25 MHz 68030 processor.

Other applications of the invention could include selection of "desktop" items such as folders and file icons, which can be placed at various display locations in many current interface styles. The program of the invention may also benefit selection of small objects such as window "close boxes" or small screen applications such as notebook computers. It is envisioned that handicap users with marginal pointing skills may also benefit from using the invention.

Considering the task hours which are spent pointing and clicking with current user-interface styles, the invention could produce substantial productivity benefits for a variety of users and applications.

Obviously, many modifications and variations of the invention are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims, the invention may be practiced otherwise than as has been specifically described. ##SPC1##