Touchless computer input device to control display cursor mark position by using stereovision input from two video cameras

[0001] The present invention relates to computer peripheral input devices for allowing operator to control display cursor mark position and perform operations known as click events.

[0002] Modern computer operating systems require a peripheral input device capable of interacting with a graphical user interface, moving cursor marker along the screen workspace, clicking on displayed buttons or dragging graphical objects. Commonly known devices of such kind are computer mice for desktop systems or touchpads for portable computers.

[0003] While pressing physical keyboard buttons is a simple and obvious operation, using a computer mouse for graphical input requires some experience and takes additional operator's attention. This happens because the number of reasons.

[0004] First, the mouse is usually located and moved along the horizontal computer desk surface, while the computer display screen that displays the manipulated objects and provides visual feedback of the user's actions normally stands vertically in the front of the operator and away from the mouse. This spatial displacement, navigation in the orthogonal planes and a different scale between the display's screen and the mouse navigation area are the first type of inconvenience in operating the mouse.

[0005] The second problem is also a result of the displacement between the mouse and the display screen. Each time before starting cursor manipulation, operator has to locate the mouse or the touchpad sensor, position the cursor marker at the desired place on the screen and only then perform the required graphical input operation.

[0006] Touchscreen systems are free from those disadvantages but may cause screen contamination and wearing. They also may be not very accurate without using special stylus.

[0007] These problems can be overcome by introducing peripheral graphical input device that has reduced displacement between display cursor marker and operator's hand manipulated in the region near and parallel to the screen surface.

[0008] In the present invention the stereovision input from two video cameras is used to measure the position of the operator's finger and to move the cursor marker to the place on the screen where the finger is located. The same video cameras are used to measure the finger's movement direction and angular rotation speed to produce events interpreted by operating system as mouse left and right button click events.

[0009] Having the same type of functionality, this device can provide the same software interface as conventional mouse device does and hence may be installed on modern computers without expensive operating system modifications.

[0010] FIG. 1 is a front view of the computer display with mounted video cameras (1a), (1b) on the top of it and the cursor marker (3) displayed on the screen (2).

[0011] FIG. 2 is the left side view of the said display and the region (4) in the front of it.

[0012] FIG. 3 is a top view of the said display and the region (4) in the front of it.

[0013] FIG. 4 is a schematic perspective view of the said display and the pointing subject (5) located in the said region (4).

[0014] FIG. 5 is a schematic illustration of the pointing subject projected coordinates (6) definition.

[0015] FIG. 6 is an illustration of the pointing subject translation velocity vector (8), vector orthogonal to the screen surface (7) and an angle () between them definitions.

[0016] FIG. 7 is an illustration of the pointing subject angular rotation speed (9) definition.

[0017] The device described in the present invention consists of the following logical parts:

[0018] (A) Obtaining 2D coordinates of the pointing subject (5)further referenced as a pointerin the 2D display screen surface space.

[0019] (B) Means for sensing and triggering mouse left and right buttons clicking events.

[0020] (C) Means to communicate that data to the operating system of the computer controlling the display screen (2).

[0021] Part (A) is implemented as the following:

[0022] There are several video cameras mounted in the space adjacent to the display screen having their optical axis parallel or about to the screen plane. That cameras are taking images of the scenes in the front of the display and further sending said images, further also referenced as frames, to the processing unit. In some situations, when said video cameras are mounted close to the screen itself, there could be a distortion or focusing problems for images of the pointer when it is too close to one of the cameras. In that case having three or more video cameras will solve this problem because there always will be at least two cameras far enough from the pointer to take appropriate images. Having more then two cameras also improves an accuracy of the measured coordinates of the pointer.

[0023] The second step for the part (A) is to recognize the pointer in the each frame obtained from the video cameras. It can be done by a number of ways. The most obvious is to find the difference between several frames taken by the same camera at the different time and consider the difference to be an image of the moving pointer. The second way is to store the image of the pointer, say as a part of the adjustment or installation process and then locate this image in the frames coming from cameras in the real time. As usual, the combination of methods will give the best result: accumulating and adjusting of the image of the pointer taken from the real-time frames would be more precise and flexible.

[0024] The third step of the part (A) is to calculate 2D projected coordinates (6) of the pointer. First, the linear (1-dimensional) coordinates of the pointer should be obtained for each frame. It can be done by, for example, taking the coordinate of the pixel of the pointer image nearest to the screen surface or by more precise methods as calculating the middle of the pointer, its direction, etc. The second is to convert obtained set of two or morethe number is equal to the number of used video cameras1D coordinates from each camera into 2D coordinates in the screen space, the method of doing so is commonly referenced as a stereo geometry method. The positions and angles of video cameras should be knownby for example, calculating them at the calibration process. This step completes the description of the part (A) of the invention.

[0025] Part (B) of the invention provides means for implementing in the device an events trigger that can be interpreted by the operating system as standard mouse clicking events.

[0026] In order of the invented device to be complete emulation of the standard mouse, there should be means for the user to trigger two types of events: a mouse left button click event and a mouse right button click event. The left button click event is usually associated with pressing GUI buttons and the right mouse button click event is associated with properties request. Hence, it would be natural to consider pointer approaching the screen user action as the left mouse button push event and pointer moving away from the screen as the button release event. Further, there is another type of event ready to be recognized by the set of video cameras. It is the rotation of the pointer, which can be interpreted as the required right mouse button click event.

[0027] Part (C) of the invention provides a cost effective way to install this device into the existing environment of computers and operating systems. Most of cursor manipulators like mice or touch pads communicate the user's actions by means of interface, known as a mouse driver interface. Because the invented device provides at least the minimal set of functionality required by mouse driver interface, it can be implemented as a part of it.