Image rectification method and related device for a video device

[0001] This application claims the benefit of U.S. Provisional Application No. 61/020,151, filed on Jan. 10, 2008 and entitled “Real-Time Free Viewpoint Rendering System for Face-to-face Video Conference”, the contents of which are incorporated herein by reference.

[0002] 1. Field of the Invention

[0003] The present invention relates to an image rectification method and related device for a video device, and more particularly, to an image rectification method and related device for rectifying a received image according to a position of the received image shown in the video device.

[0004] 2. Description of the Prior Art

[0005] Real-time text messages and VoIP phone are popular internet applications in modern life. With the advancement of internet technology, people are dissatisfied with pure text or voice communication, and therefore videoconferencing applications arise. People can make videoconference calls with real-time images, which brings many more interactive effects, to communicate with others. Communications are much more precise and interesting via videoconferencing applications.

[0006] In a common video device, such as a desktop computer or a laptop computer, a camera is usually set above a screen of the computer. A computer user, hereinafter call transmitter, views a facial image of another computer user, hereinafter called a receiver, shown in the screen, and the transmitter uses the camera to capture real-time images of the transmitter and transmits images to the receiver. Please refer to FIG. 1, which illustrates a relationship between video devices and the video device users. As shown in FIG. 1, a transmitter using a video device A makes a videoconference call to a receiver using a video device B. Note that there is a distance between a camera of the video device A and the facial image of the receiver shown in a screen of the video device A. Further, line of sight of the transmitter is toward the image of the receiver, not toward the camera, such that an angular deviation θ is formed between the line of sight of the transmitter and the direction from the transmitter's eyes to the camera.

[0007] In the viewpoint of the receiver, the receiver may feel that the transmitter is not looking at the receiver but is looking down with the angular deviation θ. Hence, the experience of the receiver using the videoconference application is affected. The receiver can receive an eye-to-eye image only when the transmitter is looking at the camera, whereas the transmitter may not know whether the receiver is looking at the transmitter because the transmitter is looking at the camera, not looking at the image of the receiver. In other words, in the prior art, both the transmitter and the receiver feel that the opposite is not looking at itself.

[0008] In order to overcome the above problem, another conventional video device assumes that a distance between the transmitter and the camera is fixed, and performs some image synthesis procedures on the transmitted image to rectify line of sight of the transmitter before transmitting the image. Please refer to FIG. 1 again. The video device A can obtain the angular deviation θ formed by a distance D1 from transmitter's eyes to the camera and a distance D2 from the transmitter's eye to a facial image of the receiver. The video device A performs an image rectification procedure on the captured face image of the transmitter, and transmits the rectified image of the transmitter to the video device B. Therefore, the receiver feels that the transmitter is looking at the receiver and therefore communications with natural eye contact are achieved.

[0009] The two above conventional video devices do not consider how the receiver uses the video device. Please refer to FIG. 1 again. The video device B is running other tasks such as document editing or programming in other windows at the same time as performing the videoconference call, and the window of the videoconference call is not in the middle of the screen. In this situation, even if the video device A already performed the image rectification process on the face image of the transmitter, the receiver still feels that line of sight of the transmitter is in direction V1, and not in direction V2. In other words, the receiver still feels that the transmitter is not looking at the receiver.

[0010] From the above, the conventional video device only takes the position of the transmitter's eyes and the position of the camera into consideration when rectifying the image of the transmitter and does not consider how the receiver uses the video device. When the videoconference window is not in the middle of the screen, the receiver can not experience natural eye interaction.

[0011] It is therefore a primary objective of the claimed invention to provide an image rectification method and related device for a video device.

[0012] The present invention discloses an image rectification method for a video device. The image rectification method comprises receiving an image that is a facial image of a transmitter from the transmitter, obtaining a first angular deviation with respect to line of sight of the transmitter according to the image, obtaining a second horizontal angular deviation and a second vertical angular deviation with respect to line of sight of a receiver using the video device, and performing an image synthesis procedure on the image according to the first angular deviation, the second horizontal angular deviation and the second vertical angular deviation, for generating an eye-to-eye image sent to the receiver.

[0013] The present invention further discloses an image rectification device for a video device. The image rectification device comprises a receiving unit, an angle measurement unit and an image synthesis unit. The receiving unit is utilized for receiving an image that is a facial image of a transmitter from the transmitter. The angle measurement unit is coupled to the receiving unit, and is utilized for obtaining a first angular deviation with respect to line of sight of the transmitter according to the image and obtaining a second horizontal angular deviation and a second vertical angular deviation with respect to line of sight of a receiver using the video device. The image synthesis unit is coupled to the receiving unit and the angle measurement unit, and is utilized for performing an image synthesis procedure on the image according to the first angular deviation, the second horizontal angular deviation and the second vertical angular deviation, for generating an eye-to-eye image sent to the receiver.

[0014] These and other objectives of the present invention will no doubt become obvious to those of ordinary skill in the art after reading the following detailed description of the preferred embodiment that is illustrated in the various figures and drawings.

[0020] Please refer to FIG. 2, which is a flowchart of a process 20 according to an embodiment of the present invention. The process 20 is utilized in a video device, such as a laptop computer supporting videoconferencing functions, for rectifying a received image such that a user of the video device can see an eye-to-eye image during a videoconference call. The process 20 comprises the following steps:

[0021] Step 200: Start.

[0022] Step 202: Receive an image that is a facial image of a transmitter from the transmitter.

[0023] Step 204: Obtain a first angular deviation with respect to line of sight of the transmitter according to the image.

[0024] Step 206: Obtain a second horizontal angular deviation and a second vertical angular deviation with respect to line of sight of a receiver using the video device.

[0025] Step 208: Obtain an inclined angle between a screen of the video device and a plumb line.

[0026] Step 210: Perform an image synthesis procedure on the image according to the first angular deviation, the second horizontal angular deviation, the second vertical angular deviation and the inclined angle of the video device for generating an eye-to-eye image sent to the receiver.

[0027] Step 212: End.

[0028] In the process 20, the receiver is the user of the video device where the process 20 is used, who is performing a videoconference call with the transmitter. Actually, number of receivers and transmitters are not limited in a videoconference call, and the process 20 can be applied in every video device participating in the videoconference call.

[0029] Please both refer to FIG. 1 and FIG. 2 to see a relationship between the video devices and the corresponding users. In FIG. 1, the receiver uses the video device B and the transmitter uses the video device A. The embodiment of the present invention supposes that the transmitter does not perform any image rectification procedure on the transmitted image. After receiving the image of the transmitter, in Step 204, the video device B obtains the first angular deviation θ, as shown in FIG. 1, according to the image. The first angular deviation θ is formed between line of sight of the transmitter toward the image of the receiver shown in a conference window on a screen of the video device A and the direction from the transmitter's eyes to a camera of the video device A. The video device B measures the depth information of the image of the transmitter according to a depth-based scheme, such as a Motion Estimation algorithm, and thereby obtains the first angular deviation.

[0030] Please note that, as shown in FIG. 1, when the receiver performs several tasks on the video device B, such as document editing, programming and videoconferencing, at the same time, the videoconference window may not be in the middle of the screen of the video device B. Therefore, the receiver feels that the transmitter is not looking at the receiver, even if the video device B already rectified the image of the transmitter according to the first angular deviation. Please refer to FIG. 3, which is a schematic diagram of a relationship between a received facial image and a camera of a video device and associated angular deviations. In Step 206, the video device B further obtains the second horizontal angular deviation d_y and the second vertical angular deviation d_z, as shown in FIG. 3, according to a position of the camera of the video device B, a position of the receiver's eyes and a position of the facial image of the transmitter. S_w and S_L represent resolution of the screen of the video device B; P is a vertical distance between the receiver's eyes and the screen, W and L are width and length of the screen, r_y and r_z are a horizontal component and a vertical component represented as a distance in pixels between the camera of the video device B and the facial image of the transmitter. The second horizontal angular deviation d_y and the second vertical angular deviation d_z are derived by the following equations:
d_y=tan⁻¹(r_y/s_w*W/P)  (1)
d_z=tan⁻¹(r_z/s_L*L/P)  (2)

[0031] In the embodiment of the present invention, the position of the image of the transmitter is determined by a cursor controlled by the receiver. When the receiver moves the cursor to the position of the transmitter's eyes, the video device B obtains the most precise d_y and d_z and the image of the transmitter is therefore rectified precisely. Please note that, measurement of the second horizontal angular deviation d_y and the second vertical angular deviation d_z illustrated previously is based on an assumption that the screen of the video device B is parallel with a plumb line.

[0032] Please refer to FIG. 4, which is a schematic diagram of an inclined angle of a screen of a video device. As shown in FIG. 4, there is an inclined angle ψ between the screen of the video device and a plumb line shown by a dotted line. If the inclined angle ψ is formed between the screen of the video device B and the plumb line, the receiver may feel that the transmitter is not looking at the receiver but is looking upward, even if the video device B already rectified the image of the transmitter according to the first angular deviation, the second horizontal angular deviation and the second vertical angular deviation. In Step 208, the video device B further obtains the inclined angle ψ. Note that, in the process 20, Step 204 to Step 208 are used for obtaining angles and actually no specific order is required for obtaining these angles. Next, in Step 210, the video device B performs the image synthesis procedure on the image of the transmitter according to the first angular deviation, the second horizontal angular deviation, the second vertical angular deviation and the inclined angle, for generating a preferred image of the transmitter that is also an eye-to-eye image sent to the receiver, so that the receiver feels that the transmitter is looking at the receiver during the videoconference call. Precisely, the video device B performs the image synthesis procedure by an appropriate algorithm, such as a Depth Image Based Rendering (DIBR) method, to generate the eye-to-eye image.

[0033] Please note that, the process 20 is one embodiment of the present invention, and those skilled in the art can make alterations and modifications accordingly. For example, the first angular deviation can be measured according to algorithms other than the Motion Estimation algorithm; or, the video device B can perform the image synthesis procedure according to algorithms other than the DIBR method. In another embodiment of the present invention, it may be assumed that the screen of the video device B is parallel with the plumb line, so that Step 208 of obtaining the inclined angle is ignored.

[0034] Please refer to FIG. 5 for a hardware implementation of the process 20. FIG. 5 is a schematic diagram of an image rectification device 50 according to an embodiment of the present invention. The image rectification device 50 is utilized in the video device B and comprises a receiving unit 500, an angle measurement unit 502 and an image synthesis unit 504. The receiving unit 500 is utilized for receiving the image of the transmitter IMG. The angle measurement unit 502 is coupled to the receiving unit 500 and is utilized for obtaining the first angular deviation θ, the second horizontal angular deviation d_y, the second vertical angular deviation d_z and the inclined angle ψ. The image synthesis unit 504 is coupled to the receiving unit 500 and the angle measurement unit 502, and is utilized for performing the image synthesis procedure on the image of the transmitter IMG according to the first angular deviation θ, the second horizontal angular deviation d_y, the second vertical angular deviation d_z and the inclined angle ψ, for generating the eye-to eye image sent to the receiver. Detail of the above angles and associated algorithms for obtaining the first angular deviation and performing the image synthesis procedure are described previously and not repeated here. In another embodiment of the present invention, the inclined angle ψ can be ignored so that the angle measurement unit 502 is not required to measure the inclined angle ψ.

[0035] In conclusion, the embodiment of the present invention performs the image synthesis procedure on the image of the transmitter according to the first angular deviation, the second horizontal angular deviation, the second vertical angular deviation and the inclined angle, for generating the eye-to eye image sent to the receiver. Therefore, the embodiment of the present invention provides the image of the transmitter with natural eye contact, so that the receiver feels that the transmitter is looking at the receiver during the videoconference call and the user experience is improved.

[0036] Those skilled in the art will readily observe that numerous modifications and alterations of the device and method may be made while retaining the teachings of the invention. Accordingly, the above disclosure should be construed as limited only by the metes and bounds of the appended claims.