Apparatus for manipulating an object displayed on a display device by using a touch screen

[0001] This is a divisional application of prior application Ser. No. 08/437,563 filed May 9, 1995, which is a continuation of application Ser. No. 07/953,101, filed Sep. 29, 1992.

[0002] 1. Field of the Invention

[0003] The present invention relates to an apparatus for manipulating an object displayed on a display device by using a touch screen.

[0004] As use of computer systems for data processing has become widespread in recent years, more and more users are being required to handle same, i.e., to input data and converse with data processors such as work stations and personal computers. A vast variety of application programs are available for recent data processors and even a complicated application can be processed by using such application programs in combination. However, it is a great problem that such data processors are very difficult to handle, especially to manipulate an object displayed on a display device, for those who have less than a substantial knowledge of computers.

[0005] Therefore, an apparatus for manipulating an object displayed on a display device, which is easily handled even for a person who has no special knowledge of computers, is in great demand.

[0006] 2. Description of the Related Art

[0007] FIG. 1 illustrates a computer system with a conventional user interface.

[0008] A computer system with a conventional user interface consists mainly of a central processing unit (CPU) 4, a main memory 5, a keyboard/mouse 2, a frame memory 6 and a hard disk interface 71, which are interconnected via a system bus, and also a hard disk 7 and a display unit 3, which are connected to system bus via the hard disk interface and the frame memory 6, respectively. The main memory 5 stores a system control program and application programs which handle graphics processing, and provides a work area for use by the programs. The CPU 4 performs display operations under control of the programs. The hard disk 7 stores a data file for graphics to be displayed on the display unit 3. The frame memory 6 stores a frame of picture (or object) data to be displayed on the display unit 3.

[0009] To manipulate an object displayed on a display unit 3 in the above system, an operator is required to input a command, for manipulating the object by using a keyboard/mouse 2, or to select an icon (a symbolic representation of a computer function), displayed on a display unit 3 by using the keyboard/mouse 2, in order to command a desired function. However, it is troublesome and annoying to use a keyboard/mouse and icons and a person with less than a substantial knowledge of computers tends to be reluctant even to touch a keyboard/mouse.

[0010] Therefore, it is a great problem that such data processors are very difficult to handle for those who have less than a substantial knowledge of computers.

[0011] It is an object of the present invention to provide an apparatus which can easily manipulate an object displayed on a display unit.

[0012] It is another object of the present invention to provide a user interface with which a user can easily manipulate an object displayed on a display unit.

[0013] To achieve the above and other objects, the present invention provides a touch screen, a plurality of data files, display information storage means and display control means.

[0014] In the apparatus for manipulating an object displayed on a display device, the touch screen, which is a transparent panel and is mounted on the display surface, or screen, of a display device and is sensitive to the touch, e.g., of a finger of a human operator, outputs touch screen information representing the motion of the body. The plurality of data files store object data for displaying the object in different states. The display information storage means stores object information including at least an object type which specifies the shape and physical properties of the object, display position information which specifies a position where the object is displayed on the display device, file information which specifies the size and location of a part of the object data stored in one of said plurality of data files, and a file name which specifies one of said plurality of data files. The display control means recognizes a manipulation to be conducted on the object, based on the touch screen information from the touch screen and on the object information included in the display information storage means, and displays the object on the display device in accordance with the aforesaid recognition.

[0027] Throughout the above-mentioned drawings, identical reference numerals are used to designate the same or similar component parts.

[0028] FIG. 2 is a block diagram of the configuration of a touch screen-equipped workstation, in which the present invention is implemented.

[0029] In addition to the conventional system shown in FIG. 1, the system of FIG. 2, for implementing the present invention, comprises an input-output (abbreviated to I/O) port 8, a touch screen controller 15 and a touch screen unit 1 with a touch screen 11. The touch screen controller 15, connected to the input-output port 8 through an RS-232C interface, controls the touch screen unit 1. The touch screen unit 1, which is sensitive, at a position (defined by X-Y coordinates) at which the touch screen 11 is touched and particularly to the pressure applied thereon when so touched, acts as a user interface that allows a user to send signals to a computer by touching that area thereon with an element, such as the finger, . . . etc.

[0030] FIG. 3 is a schematic diagram illustrating the principle of the present invention.

[0031] For easy understanding of the principle, the input-output port 8, touch screen controller 15 and touch screen unit 1 shown in FIG. 2 are represented in FIG. 3, in the composite, by the touch screen unit 1 and the frame memory 6 and display unit 3 and, as before-noted, the touch screen 11 actually is a transparent panel which is mounted on the display surface, or screen, of the display unit 3 but is shown separately therefrom in FIG. 3 of FIG. 2 are represented in FIG. 3, in the composite, by the display unit 3. The hard disk interface 71 and hard disk 7 of FIG. 2 are represented in FIG. 3, in the composite, by the hard disk 7. A system controller 50, a touch discriminator 51, a display controller 52, which are programs stored in the memory 5 and executed by the CPU 4 (or may be constructed by hardware), and a display information table 1T, stored in the main memory 5, control the display operations performed by the present invention.

[0032] FIG. 4(a) shows a display information table; FIG. 4(b) shows touch screen information.

[0033] The display information table 1T, which is provided in the main memory 5 and corresponds to objects, includes object type information, display position information, file information, normal-display file name and special-state file name. The object type defines the type, including the shape, properties, circumstances, etc., of the object. The display position information defines the size of the object (width “W” and height “H”), and the position (top-left coordinates X, Y) and the angle at which the object is displayed on the display unit 1. The file information, which is used for an object which is so large in size that it requires scrolling to view the whole object, defines the size (width W, height H) of the whole object relative to the display screen size, and also the position (top-left coordinates X, Y) of the object portion being displayed on the display device, relative to the whole object whose data is stored in the normal display file. The normal-display file name specifies a display data file where object data for displaying a normal state of the object is stored. The special-state file name specifies a display data file where object data for displaying a special data (e.g., turn-over indication of display color, used for selectively displaying an intermediate process step, or stage, in manipulating the object) of the object is stored. The special state can be displayed selectively for each manipulation.

[0034] Touch-screen information 2I, which is sent from the touch screen unit 1, includes a touch position (X-Y coordinates) where the touch screen 11 is touched and the pressure applied thereon. The touch-screen information 2I may include two sets of X-Y coordinates shown in FIG. 4(b) depending on the type of touch, one set for a thumb and another for an index finger when the two fingers pick the object on its opposite sides, for example. (1) The touch discriminator 51, based on the touch screen information 2I from the touch screen unit 1, discriminates the type of touch which the operator's finger has on the touch screen 11, that is, a touch type including, e.g., a “continuous touch start” and “continuous touch end” as explained later. The touch discriminator 51 sends, to the system controller 50, the result of the touch discrimination performed thereby as a touch report 3R, which includes touch type and, according to the touch-screen information 2I, one or two sets of touch coordinates.

[0035] Based on the touch report 3R from the touch discriminator 51 and the display information table 1T, the system controller 50 determines the type of manipulation which was conducted by the operator and, according to the determination, updates the display information table 1T. Then, the system controller 50 sends, to the display controller 52, a display update request 4Q including “display update data” which reflects the updated contents of the display information table 1T including display position information, filed information and normal display file name and special state file name.

[0036] On receipt of the display update request 4Q from the system controller 50, the display controller 52 reads a display data file (including object data), specified by the file name, from the hard disk 7 and stores the data into the main memory 5. The display controller 52 then updates the object data in accordance with the display update data from the system controller 50 and loads the thus-updated object data into the frame memory 6 (FIG. 2) thereby to display the object on the display unit 3 (FIG. 2), as manipulated by the operator on the touch screen unit 1.

[0037] Thus, the system of the present invention determines a manipulation to be conducted on the displayed object, based on the touch screen information 2I which results from an operator's touching the touch screen 11 and the display information table 1T (see, FIG. 7(b)) which defines the object's shape, physical properties, display position, etc. The system then displays the object, according to the manipulation as thus determined and as intended by the operator, on the display unit 3.

(2) Pick Manipulation (See FIGS. 5 and 6(a) to 6(c)).

[0038] A pick manipulation is conducted in such a way that an object is picked up at a position on the display surface of the display unit 3 and placed at another position thereon.

[0039] FIG. 5 is a flowchart illustrating a pick manipulation.

[0040] FIGS. 6(a) and 6(b) are related and interconnected diagrams illustrating, and FIG. 6(c) is a table explaining, a pick manipulation.

[0041] A pick manipulation is carried out according to the following steps (S1-S8) in FIG. 5:

[0042] (S1) The system controller 50 receives a touch report 3R including two sets of touch coordinates, from the touch discriminator 51.

[0043] (S2) The system controller 50 (FIG. 3) checks whether the object-finger relation is a pick manipulation, as shown in FIG. 6(a), based on the touch report 3R and contents of the display information table 1T shown in FIG. 6(c). When the relation is not a pick manipulation, the system controller 50 checks the touch report 3R for other possible manipulations.

[0044] (S3) When the relation is a pick manipulation, the system controller 50 sends a display update request 4Q including “display update data”, commanding that the special-state file (turn-over indication) be displayed at the position specified by the display information table 1T.

[0045] (S4) The system controller 50 receives a touch report 3R.

[0046] (S5) The system controller 50 determines whether the touch report 3R includes a “continuous touch end”, which occurs when the finger-object relation is as shown in FIG. 6(b). When a “continuous touch end” is reported, the operation goes to step (S8).

[0047] (S6) Otherwise, the system controller 50 updates the display position information “coordinates (X, Y)” of the display information table 1T so that the object is positioned between the two fingers of the operator.

[0048] (S7) The system controller 50 sends display update request 4Q to the display controller 52, commanding that the special-state file be displayed according to the display information table 1T, as updated, and returns to step (S4).

[0049] (S8) When “continuous touch end” is reported by a touch report 3R, the system controller 50 sends a display update request 4Q to the display controller 52, commanding that the normal-display file be displayed at the position specified in the display information table 1T.

[0050] The following manipulations are carried out in the same way as descried in the above flowchart of the pick manipulation.

(3) Scroll Manipulation (See FIGS. 7(a) and 7(b)).

[0051] A scroll manipulation is conducted in such a way that an object, extending outside of the display surface of the display unit 3, is moved into and out of the display surface.

[0052] FIG. 7(a) is a diagram illustrating, and FIG. 7(b) is a table explaining, a scroll manipulation.

[0053] On determining that the operator's finger moves while touching the touch screen 11, based on the touch screen information 2I from the touch screen unit 1, the discriminator 51 sends, to the system controller 50, a touch report 3R as previously discussed including, e.g., “continuous touch start” for the “touch type” and also, e.g., “coordinates (800, 800)” for the touch position. As another touch screen information 2I comes in, the discriminator 51 sends a touch report 3R including, e.g., “continuous touch in progress” and “coordinates (780, 800)” (i.e., the movement to the left by the finger as shown in FIG. 7(a), and thus from X=800 to X=780 while at a fixed Y=800). When the touch screen information 2I is not sent for more than 100 milliseconds, for example, the discriminator 51 sends a touch report 3R including “continuous touch end” and, e.g., “coordinates (700, 800)” (i.e., the final X,Y coordinate as of the “touch end”) to the system controller 50.

[0054] When a “continuous touch start” is reported and the “object type” is defined as “out-screen” in the display information table 1T as shown in FIG. 7(b), the system controller 50 recognizes the manipulation as a scroll and the object as a large one extending beyond the display screen. Then, the system controller 50 determines the speed at which the finger has moved from right to left, for example, based on a change in the X-coordinate in terms of data, between a touch report 3R and the following one.

[0055] Depending on whether the finger has moved at a speed of more (high-speed) or less (normal-speed) than 20 dots, for example, the object display position on the display screen is scrolled initially at corresponding intervals of 100 or 500 milliseconds, respectively. Then, the interval, at which the display update request 4Q is sent to the display controller 52, is increased by a factor of 1.5 at each touch report 3R and, when the interval reaches 2 seconds, the scrolling is stopped.

[0056] Practically, the display position area of the screen is so controlled that it starts scrolling at the appropriate speed, as above-mentioned, after the operator's finger has moved a distance of 4 dots or more. That is, on recognizing that the finger has moved by that distance, the system controller 50 updates the file information “display position X” of the display information table 1T so that the object is displayed, shifted to the left by 10 dots, for example. The, it sends, to the display controller 52, a display update request including display position information, file information and normal display file name from the display information table 1T, as updated.

[0057] The display controller 52 reads from the hard disk 7 a display file specified by the normal display file name and loads it in the main memory 5. The display controller 52 then transfers only the part of the display file, as specified by the file information “display position X” of the display information table 1T, from the main memory 5 to the appropriate location of the frame memory 6.

[0058] In the same way, the system controller 50 sends a display update request 4Q to the display controller 52 every time it receives a touch report 3R.

[0059] When another “continuous touch” is reported before the scroll currently in progress comes to a stop, a new scroll can start from this point and at the first speed described above.

(4) Scroll-Stop Manipulation (See FIGS. 7(a) and 7(b)).

[0060] FIG. 7(a) is a diagram illustrating, and FIG. 7(b) is a table explaining, a scroll manipulation.

[0061] When a touch position given by a touch report 3R is the same as or up to approximately 5 dots apart from the position of the scrolling currently in progress, the system controller 50 doubles the frequency with which display update request 4Q are sent to the display controller 52, in order to put an end to the scrolling.

(5) Push Manipulation (See FIGS. 8(a) to 8(c)).

[0062] A push manipulation is conducted in such a way that an object is pushed on the display surface of the display unit 3.

[0063] FIG. 8(a) is a diagram illustrating, and FIG. 8(c) is a table explaining, a push manipulation.

[0064] The system controller 50 determines the type of a manipulation, based on the touch report 3R and contents of the display information table 1T shown in FIG. 8(c). When the manipulation is a push manipulation as shown in FIG. 8(a), the system controller 50 sends, to the display controller 52, a display update request 4Q including display position information, file information and normal display file name so that the object is displayed close to the finger position reported by the touch report 3R. The above display operation is repeated until a “continuous touch end” is reported by a touch report 3R.

(6) Push-While-Rotate Manipulation (See FIGS. 8(b) and 8(c)).

[0065] A push-while-rotate manipulation is conducted in such a way that an object is pushed at a position off its center (or the center of gravity shown as X in FIG. 8(b)) and it moves rotating on the display surface of the display unit 3.

[0066] The system controller 50 determines the type of a manipulation, based on the touch report 3R and contents of the display information table 1T shown in FIG. 8(c). When the manipulation is a push-while-rotate manipulation as shown in FIG. 8(b), the system controller 50 sends, to the display controller 52, display update requests 4Q with the angle of rotation increasing by 2 degrees, i.e., while increasing the angle in the display information table 1T shown in FIG. 8(c).

[0067] The display controller 52 reads the display file from the hard disk and loads the data in the main memory 5, rotates the object by the specified angle and with the left-top coordinates (X, Y) as a rotational center, as specified by the display update request 4Q, and transfers the data, with the object rotated, from the main memory 5 to the frame memory 6.

(7) Flip Manipulation (See FIGS. 9(a) to 9(c)).

[0068] A flip manipulation is conducted in such a way that an operator's finger flips an object or touches (i.e., impacts) the object from a remote position at a high speed on the touch screen 11 with a corresponding result on the display surface of the display unit 3.

[0069] FIG. 9(a) is a diagram illustrating, and FIG. 9(c) is a table explaining, a flip manipulation.

[0070] When a touch report 3R is input from the touch discriminator 51, the system controller 50 discriminates the type of the manipulation, based on the touch report 3R and the contents of the display information table 1T shown in FIG. 9(c). When the manipulation is a flip manipulation as shown in FIG. 9(a), the system controller 50 obtains the finger speed based on the touch report 3R and also obtain the object speed (i.e., the respective intervals at which display update requests 4Q are sent to the display controller 52), in the same way as described in item (3). The system controller 50 sends display update requests 4Q to the display controller 52, while updating the display position information left-top coordinates (X, Y) of the display information table 1T so that the object moves in the direction the finger moves. The system controller 50 stops moving the object when the above-mentioned interval reaches 2 seconds.

(8) Flip-Under-Gravity Manipulation (See FIGS. 9(b) and 9(c)).

[0071] A flip-under-gravity manipulation is conducted in such a way that an object, which is subjected to gravity, is flipped by a finger on the touch screen 11 and with a corresponding result on the display surface of the display unit 3.

[0072] FIG. 9(a) is a diagram illustrating, and FIG. 9(c) is a table explaining, a flip manipulation.

[0073] When the finger manipulation is a flip as in the above item (8) and the display information table 1T defines the object type as “gravity” meaning that the object is subjected to gravity, for example, the object moves while receiving a combination of the forces of inertia and gravity, as shown in FIG. 9(b). Therefore, the system controller 50 sends display update requests 4Q to the display controller 52, while updating the display position information left-top coordinates (X, Y) by adding a value to the Y-coordinate of the display information table 1T. The value is represented by 2 to the Nth power (N: the number of display update requests 4Q which are sent). In this case, too, the system controller 50 stops moving the object when the above-mentioned interval reaches 2 seconds.

(9) Roll Manipulation (See FIGS. 10(a) and 10(b)).

[0074] A roll manipulation is conducted in such a way that a rollable object is rolled by the operator's finger on the touch screen 11 and with a corresponding result on the display surface of the display unit 3.

[0075] FIG. 10(a) is a diagram illustrating, and FIG. 10(b) is a table explaining, a roll manipulation.

[0076] When a touch report 3R is input from the touch discriminator 51 and the display information table 1T defines the object type as “rollable”, meaning that the object is constructed such that it rolls when pushed by a finger, like a globe or a cylinder and as shown in FIG. 10(a), the system controller 50 sends display update requests 4Q to the display controller 52, while updating the display position information left-top coordinates (X, Y) of the display information table 1T so that the object moves a distance 10 percent behind the distance moves on the display screen and in the direction the finger moves. The system controller 50 sends display update request 4Q to the display controller 52 until “continuous touch end” is reported from the touch discriminator 51.

(10) Distort-Restore Manipulation (See FIGS. 11(a) and 11(b)).

[0077] A distort-restore manipulation is conducted in such a way that an elastic object is pressed by a finger on the display surface of the display unit 3.

[0078] FIG. 11(a) is a diagram illustrating, and FIG. 11(b) is a table explaining, a distort-restore manipulation.

[0079] When a touch report 3R is input from the touch discriminator 51 and the display information table 1T defines the object type as “elastic”, meaning that the object can be distorted and restored according to a pressure applied thereon by a finger and as shown in FIG. 11(a), the system controller 50 calculates an amount of distortion of the object based on the pressure reported by the touch report 3R. It stores in the display information table 1T, a special-state file name specifying a corresponding one of the special-state files (for displaying a distorted state of the object in a turn-over indication) corresponding to the amount of distortion calculated. Then, the system controller 50 sends a display update request 4Q to the display controller 52, commanding that the special-state file be displayed at the current display position. The above operation is repeated as necessary and when a “continuous touch end” is reported by a touch report 3R, the system controller 50 sends a display update request 4Q (with a normal display file name specified) to the display controller 52, commanding that a normal display file (normal indication) be displayed at the current display position. A plurality of special-state files are provided in the hard disk 7, corresponding to the amount of distortion of the object which results from the pressure applied on the touch screen 11.

[0080] As is apparent from the above description, the present invention regards a display screen as a visual space. It defines conditions and physical properties of an object (e.g., weight, hardness, frictional resistance, center of gravity) in the display information table 1T. It also receives touch screen information 2I, indicating a finger-touched position and pressure, as input from a touch screen unit 1. Based on the touch screen information 2I and the display information table 1T, the present invention determines a manipulation to be conducted on the displayed object, e.g, scrolling, picking, pushing, rolling, distorting the object on the display surface of the display unit 3. Thus, the present invention allows a user to manipulate an object displayed on a display device quite easily, even when the user has little knowledge of computers.