Head tracking and color video acquisition via near infrared luminance keying

[0002] The present invention generally relates to object acquisition and tracking techniques and, more particularly, to real-time head tracking techniques performed in a mutually-immersive environment.

[0003] Telepresence systems allow a local user to view a remote location (i.e. a conference room) as if they were present at the remote location. Mutually-Immersive telepresence system environments allow the local user to interact with individuals present at the remote location. In a mutually-immersive environment, the local user sits inside a display area, with a projection surface surrounding the local user outside of the display area; thereby, surrounding (or immersing) the local user. Cameras are positioned about the display area to collect images of the local user. In application, live color images of the local user are acquired by the cameras and subsequently transmitted to the remote location, concurrent with projection of live video from the remote location on the projection surfaces surrounding the local user. The local user is able to move about inside the display area; thus algorithms are needed to track the head position of the local user.

[0004] Conventional head tracking methods include generating a representation of a user's head based on the detection of the user's eyes or other facial features. An example of such a method would be to use the retro-reflectivity property of the human eye, when illuminated by light, to detect and track head position. A drawback associated with such an approach is that the head of the person being tracked must always be facing a camera. If the user turns away from the camera, eye reflectivity can no longer be detected. Thus, head position tracking cannot be accurately maintained.

[0005] Another conventional head tracking method calls for the local user to wear tracking hardware, such as, used for motion capture in computer graphics, and to transform the position information obtained from the tracking hardware into a bounding box image based on the lens focal length of the tracking hardware and the particular system geometry. A drawback associated with this approach is that the local user would have to be burdened with wearing the oftentimes cumbersome tracking hardware. Also, the tracking hardware degrades final image quality, as the tracking hardware would be visible in any resulting image.

[0006] A drawback of specific relevance to mutually-immersive environments, as described above, is that the projected views that surround the local user often contain images of other people from the remote location. Consequently, differentiating the local user's head from those projected from the remote location becomes difficult. A known approach used to distinguish the head of the local user from the projected heads is to subtract the projected video images from still images acquired from the local cameras; this is commonly referred to as difference keying. The synchronization between the projected video images and the acquired still images can be tricky, however, due to delays caused by various system components. In addition, difference keying is computationally expensive since the video images are large (on average 720×480) and must be warped and manipulated (e.g. subtracted) in real time.

[0007] The aforementioned and related drawbacks associated with conventional head tracking methods are substantially reduced or eliminated by the head tracking technique of the present invention. The present invention is directed to using luminance keying as a head tracking technique for use in conjunction with a mutually immersive telepresence environment. The head of the local user is tracked in real time by uniformly illuminating a rear projection screen that surrounds a display cube with light having a wavelength in the near-infrared spectrum. A near-infrared image of the head of the local user is acquired by a near-infrared camera equipped with visible-cut near-infrared pass filters that discern the difference between the illuminated rear projection screen, representing the background, and any foreground illumination. A color image of the head of the local user, and any color images on the rear projection are acquired by a color camera. A bounding box is then provided around the head of the local user in the near-infrared image. This bounding box image is then translated to the view space of the color camera. The translated image is then used to crop the color image, which is then transmitted to a remote location.

[0008] In application, the local user is placed within a display cube. Each side of the display cube is covered with a projection screen. Thus, the projection screen is always positioned substantially behind the local user. A plurality of near-infrared illuminators are positioned behind the projection screen. The near-infrared illuminators provide near-infrared light uniformly against the projection screen. A projector, which is adapted not to emit light in the near-infrared spectrum, is positioned about the near-infrared illuminators behind the projection screen. The projector provides a video image of the remote location on the projection screen. A camera unit, including a stacked color camera and a near-infrared camera, is positioned at the corners of the display cube. In an alternate embodiment, the camera unit is located in front of the local user. The near-infrared camera detects any luminance differences between an object located within the display cube relative to the luminance value of the projection screen. According to the present invention, such object is considered to be the head of the local user. This is referred to as luminance keying.

[0009] The color camera detects the color image of the remote location projected onto the projection screen by the projector and the local user located in front of the screen. The images detected by the near-infrared and color cameras are then transferred to a processor. The processor performs a bounding box process on the pixels that represent the local user's head in the near-infrared image. The processor then translates the bounding box to the view space of the color camera, then crops the color images based on the translated bounding box. This cropped, color version of the local user's head in front of the projected image is then transmitted to the remote location.

[0010] An advantage of the present invention is that it provides the ability to distinguish a locally present object from projected images of objects at remote locations.

[0011] Another advantage of the present invention is that distinguishing between local objects and remote objects is performed in real time.

[0012] A feature of the present invention is that it is economical and straightforward to implement.

[0013] The aforementioned and related advantages and features of the present invention will become apparent upon review of the following detailed description of the invention, taken in conjunction with the following drawings, where like numerals represent like elements, in which:

[0014] FIG. 1 is a graphic representation of a projected view provided by a conventional immersive telepresence system and a local user;

[0015] FIG. 2 is a schematic top view of the display cube used in the mutually-immersive telepresence system of the present invention;

[0016] FIG. 3 is a schematic view of the camera unit used in the display cube illustrated in FIG. 2;

[0017] FIG. 4 is a schematic representation of a rear image of the user's head provided by the camera unit in accordance with the tracking method of the present invention;

[0018] FIGS. 5-5A are flowcharts illustrating the head tracking method of the present invention used in conjunction with the display cube illustrated in FIG. 2;

[0019] FIG. 6 is a schematic diagram of the user's head illustrated in FIG. 4 after being cropped according to the head tracking method of the present invention; and

[0020] FIG. 7 is a schematic diagram of the resulting image provided by the head tracking method of the present invention.

[0021] The luminance keying head tracking technique of the present invention will now be described with reference to FIGS. 1-7. As generally discussed above, in mutually-immersive telepresence environments, the local user is placed within a display area, with a projection of the remote location being provided on the projection surface that surrounds the local user. Concurrent with the display of the remote location within the display area, an image of the local user is transmitted to the remote location. This concurrent transfer of images provides the effect of the local user being physically present at the remote location. A drawback associated with such telepresence environments is that the color camera of the system cannot easily differentiate between the local user and images of other people from the remote location projected onto the projection surface, as illustrated in FIG. 1. FIG. 1 is a graphic representation of a projected view acquired by the color camera of a conventional telepresence system, with the local user 40 being surrounded by a plurality of projected images 11-14. As shown in FIG. 1, the face of the local user and one of the remote users 12 are facing away from the corresponding cameras. The present invention overcomes such image differentiating problems by providing a robust, real-time head detection and tracking technique for use in mutually-immersive telepresence environments.

[0022] FIG. 2 is a schematic top view of the display cube used to provide the mutually-immersive environment and adapted to track the position of the head 40 of the local user according to the present invention. The display cube 10 comprises a display room having a rear projection screen 22 provided on each side of the display room; thereby, surrounding the local user. A plurality of near-infrared illuminators 24 is positioned behind rear projection screen 22 along each side of the display cube 10. The near-infrared illuminators 24 are operative to uniformly illuminate the rear projection screen 22 with light having a wavelength in the near-infrared (700 Å-1000 Å) spectrum. In an exemplary embodiment of the present invention, the near-infrared illuminators 24 are implemented using incandescent light bulbs, covered with visible-cut near-infrared pass filters. Other illumination mechanisms, such as a bank of light emitting diodes (LEDs) having a peak output wavelength of about 880 nm and a bandwidth of about 80 nm can also be used.

[0023] A rear projector 26 is also positioned behind the rear projection screen 22 along each side of the display cube 10. Each rear projector 26 is generally positioned between the near-infrared illuminators 24 and is used to project a video image of the remote location on the rear projection screen 22. The output beam lines from each rear projector 26 are shown as lines 23. As the user is positioned in the central portion of the display cube 10, the video image provided by the rear projector 26 (encompassed within the beam lines 23) surrounds the local user. In this fashion, the local user is provided with a sensation of actually being at, or within, the remote location. Each rear projector 26 includes a series of color (red, green, blue) filters (not shown) that filter out substantially all of the near-infrared light energy generated by the lamp within the projector. As such, only negligible amounts of near-infrared light are provided by each rear projector 26. Thus, the only significant sources of near-infrared light within the display cube 10 are the near-infrared illuminators 24. This becomes important when detecting head position as discussed in greater detail below.

[0024] Positioned at each corner of the display cube 10 is a camera unit 28. The camera units 28 are used to acquire different positional views of the user's head. Each camera unit 28 is connected (line 31) to a corresponding processing device, such as computer 30, used to process the signals provided by the camera units 28. The computers 30 are interconnected via line 34 to form a network. Each of the camera units 28 includes a color camera 19 and a monochrome near-infrared camera 20 (FIG. 3). As illustrated in greater detail in FIG. 3., the color camera 19 includes a pin hole lens 21, provided on an extension 25. The pin hole lens has an opening 19p. Likewise, the near-infrared camera 20 includes a pin hole lens, provided on an extension 27. The pin hole lens of the near-infrared camera 20 has an opening 20p. In an exemplary embodiment, the color camera 19 is a conventional charge-coupled device (CCD) camera equipped first with an R-pass only filter; next with a B-pass only filter; then with a G-pass only filter that are interleaved in two dimensions over the CCD pixel array. The near-infrared camera 20 is a monochrome CCD camera equipped with a visible-cut near-infrared pass filter which only allows passage of light having a wavelength in the near-infrared spectrum.

[0025] As shown in FIG. 3, in an exemplary embodiment, the color camera 19 is stacked on top (i.e., in a vertical orientation) of the near-infrared camera 20 in order to reduce the paralax error between the two cameras. In an alternate embodiment, the camera position can be switched. Also, in another alternate embodiment, the color camera 19 and the near-infrared camera 20 can be positioned side-by-side (i.e., in a horizontal orientation). Other arrangements using a common lens and a beam splitter may also be used. What is required is that the two cameras be placed in close proximity to one another in order to reduce the paralax error between the two cameras.

[0026] In operation, the near-infrared illuminators 24 uniformly illuminate the rear projection screen 22 with light having a wavelength in the near-infrared spectrum. The near-infrared light provides the background that is used by the near-infrared camera 20 in detecting the luminance difference between the head of the local user 40 and the rear projection screen 22. Any luminance difference detected by the near-infrared camera 20 is considered to be in the foreground. This is referred to as luminance keying. As will be discussed in greater detail below, based on this foreground information, the user's head can be detected and accurately tracked. An important advantage provided by the luminance keying method of the present invention over conventional head tracking techniques, particularly retro-reflectivity techniques, is that the local user is not required to always be looking into a camera, or comparable mechanisms, to provide head position information. All that is necessary is a difference in luminance within the interior of the display cube 10, with respect to the rear projection screen 22. Thus, as illustrated in FIG. 4, the local user's head 40 can be accurately detected by the near-infrared camera 20 even when the local user is looking away from the camera unit.

[0027] The processing steps performed to define and track the head 40 of the local user will now be discussed with reference to FIGS. 5-5A. At the outset, it should be understood that the processing steps discussed below could be performed by any suitable processing device, such as computer 30 (FIG. 2). The processing steps can also be performed by a dedicated device, such as an application specific integrated circuit (ASIC) or the like. In an exemplary embodiment of the present invention, the processing devices are computers, each equipped with a video capture card that provides for image processing on a frame-by-frame basis. As discussed above, the rear projection screen 22 is uniformly illuminated with near-infrared light by the near-infrared illuminators 24. The video image of the remote location, if any, is then projected on to the rear projection screen 22 by the rear projector 26. The local user is positioned generally within the center of the display cube 10.

[0028] The first step of the tracking process (step 502) begins with detecting luminance differences within the display cube 10. This is done through the use of the near-infrared pass, visible-cut cameras 20. As discussed above, each of the rear projectors 26 emit negligible amounts of light in the near-infrared spectrum. Thus, the color images provided by the rear projectors 26 have no impact on the luminance value of the rear projection screen 22 or what the near-infrared cameras 20 detect. As each of the near-infrared cameras 20 are equipped with visible-cut near-infrared pass filters, they are only capable of detecting light in the near-infrared spectrum. Thus, the changing color images on the rear projection screen 22 are not detected by the near-infrared cameras 20. An object within the display cube 10, for example the head 40 of the local user, having a different luminance value from the rear projection screen 22 is assumed to be in the foreground of the image. The video signals representing the background and foreground portions of the image are transmitted to the corresponding computer 30 for processing. The foreground and background images may also be displayed on a suitable monitor. Video signals displayed on a video monitor sometimes contain artifacts along the outer boundaries of the image. Such artifacts are ignored by the method of the present invention. This is referred to as overscan.

[0029] Now that the foreground has been distinguished from the background, the user's head is defined. The foreground image is scanned (step 504) from top to bottom in order to determine the top of the local user's head. The foreground image is scanned in a series of parallel lines (i.e. scan lines) until a predetermined number, h, of adjacent pixels within a scan line, having a luminance value within foreground tolerance are detected (step 506). In an exemplary embodiment, h equals 10. This detected region is assumed to be the top of the local user's head. By requiring a significant number of adjacent pixels to have similar luminance values, the detection of false signals due to video noise or capture glitches are avoided.

[0030] At this point, a measurement of the width of the local user's head is performed (step 508). The measurement begins by moving a distance equal to a percentage of the total number of scan lines down from the top of the originally detected (captured) foreground image. The percentage used is a user-definable parameter that controls how far down the image to move when measuring the width of the local user's head. In an exemplary embodiment of the present invention, a value of 10% is used. For example, if 480 scan lines were used to capture the foreground image, the method of the present invention causes the computer 30 to move forty eight scan lines down from the top of the image to measure the width of the image, which is assumed to be the head of the local user. The left edge of the foreground image is acquired by detecting the leftmost foreground pixel value of the scan line. The right edge of the foreground image is acquired by detecting the rightmost foreground pixel value of the scan line. The leftmost pixel value is subtracted from the rightmost pixel value to determine the width of the foreground image, which corresponds to the width of the local user's head at its widest point.

[0031] Based on the calculated width of the foreground image, the position of the top of the foreground image and the aspect ratio of a typical human head, a bounding box of the user's head is computed in step 510. By computing the bounding box based on the aforementioned values, the local user's head can be captured independently of the width of the shoulders of the local user. Upon completion of the detection process described above, the user's head is bounded within a box in step 512. After performing some post-process filtering of the pixels within the bounding box that represent the local user's head, a classic close portrait format showing the local user's face, neck and upper chest is provided on the monitor as illustrated in FIG. 6. The process then continues to step 514 (FIG. 5A).

[0032] Tracking head movements of the local user within the display cube can also be accomplished by the present invention. Generally, there are two possible head movements: (1) slow, long-term movements; or (2) higher-frequency movements. An example of a high-frequency movement would be head nodding. According to the present invention, head movements, are tracked by averaging the corners of the bounding box, generated by the process presented in FIG. 5, with previous bounding box corner points using an exponentially weighted average. This is calculated by taking the vertices of a new bounding box, multiplying them by x=1/n, then multiplying the old average by (1−x) and adding the two values. In an exemplary embodiment, n=20 and x=0.05. Since at each new frame the old average is multiplied by (1−x) and added to the old frame value, new data contributes to the averages as (1−x)^f/n, where f is the number of frames that have passed since the data arrived. The net result is that when there is a large change in head position, the bounding box moves slowly but steadily until it nears the new position and then it settles down more slowly.

[0033] Through experimentation, the inventors have discovered that by using a value for n that is slightly larger than the frame rate (e.g. n=20 when the frame rate is 15 fps) works the best. Because the local user's head is not tightly surrounded by the border of the bounding box, there is room for a user to nod their head without falling outside of the calculated bounding box.

[0034] After computing the bounding box, the resulting detected image of the viewer can be transmitted to the remote location. As illustrated in FIG. 5A, after the bounding box has been applied in step 512, the portrait of the local user is translated from the view of the near-infrared camera 20 to the view of the color camera 19 in step 514. This is accomplished by using standard graphics techniques. Next the translated bounding box image is used to crop the color image in step 516. In other words, all pixel data located outside of the bounding box is not used in generating the cropped color image. The cropped color image is illustrated in FIG. 6, which shows the local user 40 in the foreground facing the projected persons 11, 12 present at the remote location. At step 518, this cropped color image is transmitted to the remote location via line 36 (FIG. 2). Thus, the remote location is provided with a cropped color version of the head of the local user. In this fashion, the individuals located at the remote location can see and, therefore, directly interact with the local user as if the local user were physically present at the remote location.

[0035] Depending on the size and aspect ratio of the display at the remote location, the bounded image of the user can be expanded to fill the display at the remote location. The detecting and tracking method of the present invention can also be expanded to detect, track and transmit hand gestures. This is accomplished by first performing head tracking, then modifying both the horizontal and vertical dimensions of the image. For example, to show hand movements near the local user's head (such as waving to another person), the horizontal dimension must be compressed relative to the vertical dimension.

[0036] The above detailed description of the present invention has been presented for the purpose of illustration and description. Although the present invention has been described with respect to a specific embodiment, various changes and modifications may be suggested to persons of ordinary skill in the art, and it is intended that the present invention encompass such changes and modifications as fall within the scope of the claims appended hereto.