Method and apparatus for user interaction using pattern image

[0001] The present invention claims priority of Korean Patent Application No. 10-2010-0133143, filed on Dec. 23, 2010, which is incorporated herein by reference.

[0002] The present invention relates to a method and apparatus for a user interaction using a structured light, and more particularly, a method and apparatus for a user interaction using an image of a pattern that is generated using laser and diffraction gratings, which are capable of recognizing a user input through an interaction with the user in a computer system or the like equipped with a projector and a camera.

[0003] As technique is developed in recent years, projectors and cameras are reduced in size and equipped in mobile devices; therefore, they become more useful.

[0004] Further, a small projector or camera is produced in a wearable form to be put around a neck or shoulder, and being developed in a prototype for provision of various services. Also, a system of a portable type is being developed.

[0005] A conventional wearable projection computer, a conventional stand-type projection computer or the like is an example of a projection computer system which is equipped with an output device such as a projector, an input device such as a camera, and the like.

[0006] For interaction between such projection computer and a user, it is very important to sense and recognize a user input on a projected image.

[0007] For efficient interaction in a mobile environment, an effective method, in which an image is projected on a palm or table and a finger of a bare-hand is used for interaction, is applied. In order to perform such method in a mobile-embedded system, a low computational recognition technique is required.

[0008] According to the prior art, for increasing a recognition performance of interaction through a user's posture recognition, an interaction recognition improving unit, such as a color marker or the like, was physically worn around user's hand or finger, which caused inconvenience in the aspect that the user should hold the interaction recognition improving unit all the time.

[0009] To overcome such inconvenience, a bare-hand interaction technology is being developed, but a natural image captured by a camera is image-processed to recognize a user input. In this case, in order to identify the user input on the natural image, a high-level computation is required and thus an interaction processing takes a long time. Also, an embedded system or the like is unable to provide a fast response time. Especially, an operation of touching a surface, such as a palm or a table, using a bare finger, is a very difficult technique requiring a large computational amount.

[0010] Further, for providing services with better qualities by using a projector, appropriately adjusted brightness, color, focus and the like of an output image should be provided and no distortion should generates. To this end, the brightness, color, focus and the like of the projector needs to be adjusted, and an output screen should accurately match on a specific space. Thus, this related art also requires the high-level computation for such adjustment and matching, thereby spending a long-term of time for the adjustment and matching.

[0011] In view of the above, the present invention provides a method and apparatus for a user interaction, which are capable of reducing an interaction processing time by virtue of a low computation with respect to a captured image with allowing for a user interaction with a bare hand, and quickly executing an interaction between an output image and the user in a low computational form by quickly executing correction and matching of the output image in a dynamic manner.

[0012] In accordance with an aspect of the present invention, there is provided a user interaction apparatus using a pattern image. The apparatus includes: an image output unit for outputting a general image on a display region; a laser radiating unit for radiating laser beams for a pattern image; a pattern image forming unit for forming the pattern image by using the laser beams; a pattern image output unit for outputting the pattern image by overlapping the same on the display region of the general image; an image acquiring unit for capturing the pattern image output on the display region; and an image recognizing unit for recognizing a user input by performing an image processing on the pattern image captured by the image acquiring unit.

[0013] In accordance with another aspect of the present invention, there is provided a user interaction method using a pattern image. The method includes: outputting a pattern image formed by using laser beams by overlapping the same on a display region of a general image; capturing the pattern image output on the display region; and recognizing a user input by performing an image processing on the captured pattern image.

[0020] Embodiments of the present invention are described herein, including the best mode known to the inventors for carrying out the invention. Variations of those preferred embodiments may become apparent to those of ordinary skill in the art upon reading the foregoing description. The inventors expect skilled artisans to employ such variations as appropriate, and the inventors intend for the invention to be practiced otherwise than as specifically described herein. Accordingly, this invention includes all modifications and equivalents of the subject matter recited in the claims appended hereto as permitted by applicable law. Moreover, any combination of the above-described elements in all possible variations thereof is encompassed by the invention unless otherwise indicated herein or otherwise clearly contradicted by context.

[0021] In the following description of the present invention, if the detailed description of the already known structure and operation may confuse the subject matter of the present invention, the detailed description thereof will be omitted. The following terms are terminologies defined by considering functions in the embodiments of the present invention and may be changed operators intend for the invention and practice. Hence, the terms should be defined throughout the description of the present invention.

[0022] Combinations of respective blocks of block diagrams attached herein and respective steps of a sequence diagram attached herein may be carried out by computer program instructions. Since the computer program instructions may be loaded in processors of a general purpose computer, a special purpose computer, or other programmable data processing apparatus, the instructions, carried out by the processor of the computer or other programmable data processing apparatus, create devices for performing functions described in the respective blocks of the block diagrams or in the respective steps of the sequence diagram. Since the computer program instructions, in order to implement functions in specific manner, may be stored in a memory useable or readable by a computer aiming for a computer or other programmable data processing apparatus, the instruction stored in the memory useable or readable by a computer may produce manufacturing items including an instruction device for performing functions described in the respective blocks of the block diagrams and in the respective steps of the sequence diagram. Since the computer program instructions may be loaded in a computer or other programmable data processing apparatus, instructions, a series of processing steps of which is executed in a computer or other programmable data processing apparatus to create processes executed by a computer so as to operate a computer or other programmable data processing apparatus, may provide steps for executing functions described in the respective blocks of the block diagrams and the respective steps of the sequence diagram.

[0023] Moreover, the respective blocks or the respective steps may indicate modules, segments, or some of codes including at least one executable instruction for executing a specific logical function(s). In several alternative embodiments, it is noticed that functions described in the blocks or the steps may run out of order. For example, two successive blocks and steps may be substantially executed simultaneously or often in reverse order according to corresponding functions.

[0024] Hereinafter, an embodiment of the present invention will be described in detail with the accompanying drawings which form a part hereof.

[0025] FIG. 1 is a block diagram illustrating the configuration of a user interaction apparatus using a pattern image which has been generated by laser and diffraction gratings in accordance with the embodiment of the present invention.

[0026] As illustrated in FIG. 1, the user interaction apparatus 100 may include an image output unit 110, a laser radiating unit 120, a pattern image forming unit 130, a pattern image output unit 140, an image acquiring unit 150, an image recognizing unit 160, an image correcting unit 170 and the like.

[0027] The image output unit 110 outputs a general RGB image on a display region. For example, the image output unit 110 may be implemented as a projector, which enlarges an input image and projects the enlarged image onto a display region, such as a palm, a table, a screen or the like.

[0028] The laser radiating unit 120 radiates laser beams for a pattern image. For example, the laser radiating unit 120 may be implemented as infrared rays generator for radiating infrared laser beams.

[0029] The pattern image forming unit 130 forms a pattern image by using the laser beams. For example, the pattern image forming unit 130 includes at least one of the diffraction gratings for forming a specific pattern image by diffracting laser beams, in which the pattern image may be formed in a grid, lines, dots or the like; however, it is not limited to those exemplified patterns.

[0030] The pattern image output unit 140 outputs the pattern image by overlapping the same on the display region of the general image. For example, the pattern image output unit 140 may insert a pattern image frame between general image frames at preset regular intervals.

[0031] The image acquiring unit 150 captures the pattern image output on the display region. For example, the image acquiring unit 150 is implemented as an infrared camera for capturing an infrared pattern image. When the pattern image output unit 140 outputs the pattern image frame for the preset period of time at the preset regular intervals, the image acquiring unit 150 captures the pattern image frame at the preset regular intervals.

[0032] The image recognizing unit 160 recognizes a user input by performing an image processing on the pattern image captured by the image acquiring unit 150, and additionally recognizes at least one or all of a focus state, a distortion state and the like.

[0033] When the image recognizing unit 160 recognizes the focus status or the distortion status of the pattern image captured by the image acquiring unit 150, the image correcting unit 170 corrects the focus status or the distortion status of an output image of the image output unit 110 based on the recognized state.

[0034] FIGS. 2A and 2B are views illustrating an inserted state and a captured state of a pattern image in accordance with an embodiment of the present invention.

[0035] Referring to FIG. 2A, a pattern image frame is inserted between general image frames at the preset regular intervals. In this case, the image acquiring unit 150 captures the pattern image frame at the preset regular intervals. Especially, the pattern image frame should be invisible to user's eyes for the quality of an output screen, and thus, there is a limitation on the number of pattern image frames which are able to be inserted per unit time. When it is intended to insert many pattern image frames in order to enhance the quality of recognizing a user's input, a high-speed pattern image insertion and a high-speed screen capture are required. Meanwhile, referring to FIG. 2B, in case of using an infrared pattern image, when an image is captured by an infrared camera, only a pattern image may be acquired. Thus, an image processing can be further facilitated and the pattern image may be invisible to the user, which may reduce limitation on the number of pattern image frames being inserted per unit time.

[0036] FIG. 3 is an exemplary view illustrating a projector/camera-integrated projection computer and a pattern image in accordance with the embodiment of the present invention.

[0037] Referring to FIG. 3, as the pattern image, a grid image 201 is used. When a user touches a surface, such as a palm, a table, a screen or the like with a finger 203, the finger 203 and the surface may be positioned on a nearly same plane. Accordingly, the image recognizing unit 160 may recognize that the change of, e.g., distortion, brightness, thickness or the like in the pattern image hardly occurs. When the user does not touch the surface with the finger 203, the image recognizing unit 160 may recognize that the change in the pattern image is caused due to a distance difference between the finger 203 and the surface. Herein, the change in the pattern image may be used to detect and trace a movement of a hand and a fingertip.

[0038] In FIG. 3, it is illustrated that a projection computer 300 includes a projector 310 as an example of the image output unit 110, an optical lens 340 belonging to the pattern image output unit 140, and a camera as an example of the image acquiring unit 150. Here, in the present embodiment of the present invention, the pattern image output unit 140 includes the optical lens 340, but the optical lens 340 may be omitted. That is, when the pattern image output unit 140 installed inside outputs a pattern image with a desired size, there is no need to separately enlarge or output the pattern image, so the optical lens 340 may be omitted.

[0039] FIGS. 4A and 4B illustrates changed shapes of a finger touch operation and a finger non-touch operation, as an example of a pattern image frame captured by a user interaction apparatus using a pattern image in accordance with the embodiment of the present invention.

[0040] In the drawing, FIG. 4A shows an image captured by the image acquiring unit 150 when the finger touch operation is performed, and FIG. 4B shows an image captured by the image acquiring unit 150 when no finger touch operation is performed. When the user touches a surface, such as a palm or a table, with a finger, the finger and the surface may be almost flush with each other. Accordingly, when the image recognizing unit 160 extracts a pattern image, the change (e.g., distortion, brightness, thickness or the like) in the pattern image may be rarely caused at the fingertip. Also, without the user's touch on the surface with the finger, when the image recognizing unit 160 extracts the pattern image, the change in the pattern image may be caused due to the distance difference between the finger and the surface.

[0041] FIG. 5 is a flowchart illustrating a user interaction method using a pattern image in accordance with the embodiment of the present invention.

[0042] In the following, a description of the user interaction method by the user interaction apparatus will be given in an order of time in accordance with an embodiment of the present invention, with reference to FIGS. 1 and 5.

[0043] First, after a system initialization is performed in step S410, the image output unit 110 outputs a general RGB image on a display region in step S421.

[0044] When outputting such general image, the laser radiating unit 120 radiates laser beams for a pattern image; the pattern image forming unit 130 forms the pattern image by using the laser beams; and the pattern image output unit 140 outputs the pattern image by overlapping the same on the display region of the general image in step S423.

[0045] The image acquiring unit 150 acquires an image by capturing the general image and the pattern image. For example, when the pattern image output unit 140 outputs a pattern image frame at the preset regular intervals, the image acquiring unit 150 captures the pattern image frame at the same intervals in step S430.

[0046] Next, in step S440, the image recognizing unit 160 recognizes a user input by performing an image processing on the pattern image captured by the image acquiring unit 150, and additionally recognizes at least one or all of a focus state and a distortion state of the pattern image to provide the recognition results to the image correcting unit 170. By recognizing the user input, the image recognizing unit 160 may detect and trace the movement of a user's hand or fingertip. Then, the image correcting unit 170 corrects a focus status or a distortion status of an output image of the image output unit 110 based on the focus status, or the distortion status of the pattern image recognized by the image recognizing unit 160 in step S450.

[0047] In accordance with the embodiment of the present invention, a user interaction even with a bare-hand can be enabled and a low computation with respect to a captured image is required, whereby a short time is taken for an interaction processing, thus resulting in a fast response time.

[0048] Also, the use of the pattern image can result in exhibition of high recognition with respect to skin color and ambient light.

[0049] The foregoing present invention can be applicable not only to a projector system such as a projection computer or the like but also to a mobile device having an image output unit and an image acquiring unit. In particular, the present invention can be much useful in a mobile device or a wearable system with severe changes in environments such as ambient light intensity, lighting, fluctuation or the like, a small embedded system or the like requiring a low computation technique.

[0050] While the invention has been shown and described with respect to the particular embodiments, it will be understood by those skilled in the art that various changes and modification may be made without departing from the scope of the invention as defined in the following claims.