Intelligent Programmable Touchscreen System

The present invention relates to computer systems used to define a target program in response to a user's touching a screen of the system, and more particularly relates to means to make full use of the intelligence of a touchscreen system.

A touchscreen is disclosed in a co-pending application Ser. No. 533,384 by Thomas Earl Kilbourn and Scott Robert McClelland. As mentioned in that application, the touchscreen provides an effective source of information about the movement of an operator's touch action on a display surface such as a CRT. Other devices have been developed that allow the user to select a position on the display and once selected, to initiate whatever operation is associated with that position. These devices include such apparatus as "mice", graphics stylus/pads and light-pens. These devices actually consist of two mechanisms, one to provide position information and one to provide activation information to the target program. In these systems, it is up to the target program to translate the position information into the corresponding operation directive and to cause that operation to be performed when activation is required by the user. The target program is thus involved in the selection/activation process even if the operation to be performed is external to the target program. These systems usually require the use of a cursor for position feedback to the user so that the user can see what position is currently being selected. Thus, there is no intelligence inherent in these systems. They simply provide position and activation information to the target program.

In other systems, mechanical function keys are provided on the physical keyboard arranged in a manner such that they correspond in relative position to labels that are displayed along the bottom of the display screen. By performing a visual correlation between the position of a label on the display and the position of a function key on the keyboard, the user is able to select the appropriate function key to be pressed in order to perform the operation desired. Because the function labels are separate from the function key, it is possible to make an erroneous selection.

In accordance with a preferred embodiment of the present invention, there is provided an intelligent programmable touchscreen system. The main effect of providing this environment is that the target program can define a "soft keyboard" on the display that is totally independent of a physical keyboard and hence is not constrained by the limitations of the traditional physical keyboard. The "keys", in the present environment, are implemented using touch fields. The keys can be of any size, shape and position on the display within the physical constrains of the dimensions and character resolution of the display. Furthermore, the target program can define keys that are of different forms, using different user feedback modes and having different operation types. The operation type of a particular set of keys can change dynamically with the state of the target program. And the whole keyboard can change from one keyboard form to another dynamically with the state of the target program. Thus, the input/output functionality constraints imposed upon the target program by the traditional keyboard/display environment can be greatly reduced by the use of the touchscreen firmware environment of the present invention. This allows the target program to be a more powerful tool for the user than is otherwise possible. Hence, the present invention provides us with an intelligent programmable touchscreen system.

FIG. 1 is a block diagram of a preferred embodiment of the present invention describing touchscreen control and data paths of the system.

FIG. 2 is a block diagram of the mapping mechanism from touch position to control and data operations of the system.

FIG. 3 is a block diagram of the touchscreen system used for defining, selecting and activating a system operation.

FIG. 4 is a block diagram illustrating normal use of the conventional keyboard to issue commands.

FIG. 5 is a block diagram illustrating use of a touchscreen to issue commands.

In FIG. 1, an intelligent programmable touchscreen system is shown as a configuration of hardware, firmware and software. In the embodiment of the invention, the touchscreen system includes touchscreen hardware 101, video hardware 102, keyboard hardware 103, and data communication hardware 104. The touchscreen system also includes touchscreen firmware 111, video firmware 112, keyboard firmware 113, keycode processor firmware 114, data communication firmware 115, control sequence parser firmware 116, and other firmware 117. The system further includes an operating system 123 such as MS/DOS available from Microsoft Inc., of Bellevue, Wash., operating under AIOS (Alphanumeric input output system) 121 and BIOS (Basic input output system) 122 to control local application, user, or system software programs 124.

Hereunder the intelligent programmable touchscreen system of this invention will be explained more precisely in order of the touchscreen firmware 111, the AIOS touchscreen interface 121, touchscreen control sequences, and touchscreen report sequences.

The touchscreen firmware module 111 consists of a set of firmware routines which define a simple to use yet powerful virtual machine for touchscreen operations. The virtual machine accepts data input from the the touchscreen hardware 101 and control input from the target program 105, 117 and 124, and outputs data to the target program 105, 117 and 124, and control operations to other parts of the system firmware 112, 113, and 114. The target program is usually a software application program 124 running under the MS/DOS operating system 123. However, the target program can also be another firmware module 117 (for example, configuration firmware), system software 124 (for example, Personal Application Manager PAM), software 124 written by the user, or a program 105 running on a remote computer, communicating by means of a data communication port 115 and 104.

The following description is an overview description of the functionality of the touchscreen firmware 111. A detailed description of the control input and data output formats for the touchscreen firmware 111 and the control operations initiated by the touchscreen firmware is provided in the AIOS touchscreen interface description and the touchscreen control and report sequence description.

The touchscreen hardware 101 detects an initial touch on the display surface, a change of touch position on the display surface and the disengagement of touch with the display surface, called "release", and sends this information to the touchscreen firmware 111. In the first two cases position information is sent to the touchscreen firmware 111. In the third case, information is transmitted indicating that a release has occured. In all cases, the touchscreen firmware 111 checks an internal coordinate map to determine what operation should be performed for the particular position on the screen as shown in FIG. 2. Depending on the current state of the touchscreen firmware 111 and the mapped operation for the position reported by the touchscreen hardware 101, some operation may be initiated.

The touchscreen firmware 111 is called by the target program in order to establish certain environmental parameters as shown in FIG. 1. The overall result is to define what function is to be performed when a particular position cell on the display is touched by the user.

This mode causes the touch and/or release report data from the touchscreen hardware to be translated into a screen relative coordinate number pair corresponding to the row number and column number of the position touched by the user. The row and column data report is sent to the target program only if there is no field defined for the touched position (see sec. 6 below) or if the field reporting mode is disabled (see sec. 2 below). Field reporting always takes precedence over row and column reporting for a given screen position cell (FIG. 2).

This mode causes the touch and/or release report data from the touchscreen hardware 101 to be translated into the particular operation defined for the touch field that occupies the position on the screen that has been touched by the user. See the discussion of touch fields in sec. 6 below.

This is controlled on an individual label by label basis. When enabled, touching a particular softkey label will cause the same action to be performed as would occur if the user operated a corresponding function key (i.e., an f-key) on the physical keyboard.

When on, all functions as defined by the above described mode parameters will be operational. When off, all operations performed by the touchscreen firmware 111 will be deactivated. The touchscreen hardware 101 will still send touch data to the firmware 111 but the data will be ignored.

The particular details of the reset state depend upon the mode parameters that have been defined per the above discussion. However, the reset state can be defined to be that state which exists after all operational modes have been defined but before any touch data has been received from the touchscreen hardware 101.

This function allows the target program 120 to define rectangular areas of the screen so that the areas behave in a particular manner when touched and/or released. There are several different types of touch fields that can be defined. The different field types as described below have basic behavior modes that differ somewhat from one type to another but there is an underlying functionality that is common to them all.

(a) The rectangular area of the screen that is enclosed by the bounds of a particular field responds to touch as a whole entity. Touching one point within the area produces exactly the same effect as touching a different point within the area. If the field has been defined such that the display enhancement changes when touched and/or released, then the whole area will change its enhancement regardless of which point within the area has been touched.

(b) The field is de-bounced. After the initial touch has been made in the field, further movement within the field is ignored until either a movement is made to a position outside of the bounds of the field or the touchscreen is released.

(c) The field causes a particular operation to be initiated when it is touched and/or released by the user. There is a one to one mapping from an individual field to the operation that it initiates.

(d) Visual and audio user feedback mechanisms (e.g., display brightness enhancement or an audible click) are utilized to assist the user in determining when his action to touch and/or release a field has been successful.

This function causes either an individual touch field or all existent touch fields to be deleted from the touchscreen environment.

Touchscreen data may be sent to the target program by the touchscreen firmware 111 as a result of a touchscreen action performed by the user as shown in FIG. 1. With the exception of ASCII fields, the data generated is formatted in a manner so that the target program can distinguish it from other types of data (e.g., keyboard data). The touchscreen firmware 111 is capable of generating touchscreen data in both an internal binary form for communication with target programs running internally to the system and also in a control sequence form for communication with a remote target program. The data generated by ASCII fields is undistinguishable from data generated by operating the physical keyboard for either local or remote target programs.

There are several types of control output from the touchscreen firmware 111 as shown in FIG. 1.

Touching a softkey label causes the corresponding softkey function to be performed. This is considered to be a control function because it simulates operation of the equivalent function key on the physical keyboard. The operation of the particular softkey may be such that it sends data to the target program 120.

Touch fields can be defined such that they behave exactly like a particular control key on the physical keyboard (e.g., cursor up, backspace and carriage return).

All operations initiated by the touchscreen firmware 111 as a result of a user touch and/or release action causes an audible simulated keyboard click to occur. In addition, touch fields can be defined such that they cause an audible beep to occur when touched and so that the display enhancement of the touch field changes when touched and/or released.

Touch fields can also be defined such that the cursor is positioned at the upper left corner of the field when touched and/or released.

AIOS environment description with respect to touchscreen and touchscreen function definitions. This interface description applies only to local target programs.

Four types of touch fields are available as follows:

1. ASCII Field:

In this mode, a response string of ASCII characters is obtained by consecutive input operations. The first input obtains the first ASCII byte, and the second input obtains the second ASCII byte, etc. The response string is generated when the field is activated (touched and/or released) and is indistinguishable from the typing of the same string from the keyboard. Auto repeat, equivalent to depressing and holding a key on the physical keyboard, is also performed.

2. Keycode Field:

The two data words of the response string are treated as a keycode and a qualifier and is processed by the regular keyboard routines. The final result of the touch will depend on the state and mode of keyboard processing. The touch simulates a key press on the physical keyboard and release simulates a key release. Auto-repeat is also performed.

3. Toggle Field:

The touch field is defined as a toggle switch. Each touch of the area will toggle the field on or off. Whenever the field is touched, a report is generated for the application program. The report consists of three data bytes. The data can be obtained by three consdecutive keyboard input operations. The qualifier word of each data byte returned to the application contains the touchscreen identification. The three data bytes in hexidecimal notation of each report are:

01=toggle on field report opcode

d1=response string first byte

d2=response string second byte

02=toggle off field report opcode

d1=response string first byte

d2=response string second byte

4. Normal field:

The touch field is defined so as to generate a report on touch and/or release. Each report consists of three data bytes. The data is again obtained by three consecutive keyboard input operations. The qualifier word of each data byte returned to the application program will have the touchscreen identification, and auto-repeat is also performed. The three data bytes of each report are:

05=field touched report opcode

d1=response string first byte

d2=response string second byte

06=field released report opcode

d1=response string first byte

d2=response string second byte

Touch fields can overlap. If they do, then the most recent definition for a screen position cell takes precedence.

The response bytes of row and column reporting, generated when a touch occurs, are obtained byte by byte using the keyboard input function of the operating system. Three data bytes are returned for each report. The qualifier word returned with each byte of data will have the touchscreen identification. The data bytes for touch report of row and column are:

03=row column touch report opcode

d1=touched row number in binary

d2=touched column number in binary

The data bytes for release report of row and column are:

04=row column release report opcode

d1=touched row number in binary

d2=touched column number in binary

The following set of functions allow control of the touchscreen by a local target program.

This function enables or disables one of the eight softkey label touch fields. When touched, the field will produce the same response as if the corresponding function key key is pressed. The default condition is that all softkey touch fields are enabled.

Deletes the touch field at the row and column position. Nothing happens if there is no touch field there. The row and column are screen relative coordinates.

Sets all fields to the off or deactivated state.

This function controls the types of the touch screen reports and the reporting mode (touch/release) for row and column reports.

0=Disable all reports. Softkey fields are still enabled.

1=Enable row and column reporting only. All touches will report a row and column position. Touch fields are disabled.

2=Enable touch field reporting only. Row and column reporting is disabled.

3=Enable both row and column and field reporting. Row and column reporting will occur for areas not defined as touch fields.

4=Toggle touchscreen on/off. When off, all touchscreen activity is disabled, including softkey fields. When on, all touchscreen activity is enabled as defined previously.

10..14=Same as 0..4 but causes escape sequence reports to be generated. These codes are to be used only by the system parser.

The following set of escape sequences allow control of the touchscreen from a remote target program running on a remote computer. All sequences described have an exact correspondence to operations supported by the AIOS touchscreen interface 121. A description of the functionality of those operations is explained in the next section IV.

The default condition of the touchscreen is that all reports are disabled. Consequently, it is possible to define fields and have them not respond to touch. The "Set modes" sequence as explained in section E must be used to enable the touchscreen before any reporting can proceed.

If the r and c parameters are omitted then all fields will be deleted.

The following sequences are returned to the remote application as the result of the user touching and/or releasing the screen. For ASCII fields, the report string is returned as though it were typed from the keyboard. For NORMAL and TOGGLE fields, the following sequences are returned:

As explained above, the touchscreen firmware in combination with the touchscreen hardware creates an intelligent programmable device. There are two basic modes in which the touchscreen system will be used by target programs. Firstly, the target program can define a touch field such that, when touched by the user, it will initiate some system control operation (e.g., tab, enter remote mode, clear display, etc.). When selected by the user, the control operation will occur without the target program ever being, or needing to be informed of it as shown in FIG. 3. Secondly, the target program can define touch fields that operate in a manner similar to traditional programmable function keys. When touched, the field will return a character string to the target program. This string, which is defined by the target program, can be a command for the target program. The string is identical to the equivalent command entered via the keyboard as shown in FIGS. 4 and 5. In FIG. 4 the user must perform several sequential actions successfully in order to issue the command via the keyboard. In FIG. 5 the user must perform only one action successfully in order to have the touchscreen device issue the command. In addition, the string can be several commands concatenated together or can be data for the program. Alternatively, the report string can be used as a field identification code. In this later case, the target program must keep track of the "meaning" of each field that is defined and when the target program receives a particular field identification code, the corresponding operation is initiated by the target program. This method requires more intelligence on the part of the target program than prior methods, but still requires less intelligence than previous menu selection systems. The mapping from touchscreen position to field selection and the associated user feedback functions are still performed automatically for the target program. These methods of operation are different from prior art programmable function keys in that the size, shape and position of the field (key labels) on the screen are defined by the target program. Also, selection and activation is done by touching the screen as opposed to operating a key switch on the physical keyboard.

In the second case of touch activated softkeys, the user is able to select the desired operation by touching the function label on the display screen. Because the function label and the function key are not separate, the possibility of erroneous selection is very low.