Method and apparatus for visualizing 2D product images integrated in a real-world environment

[0001] The present invention relates generally to retail shopping systems, and more particularly, to methods and apparatus for assisting shoppers in making purchase decisions by visualizing products from 2D images embedded in their own physical environment.

[0002] Augmented reality research explores the application of computer-generated imagery in live-video streams as a way to expand the real-world.

[0003] Portable devices, e.g. mobile phones, with the necessary capabilities for executing augmented reality applications have recently become ubiquitous.

[0004] The portable devices incorporate a digital camera, a color display and a programmable unit capable of rendering 2D and 3D graphics onto the display.

[0005] The processing power of the portable devices allows for basic tracking of features from the camera's image stream.

[0006] The portable devices are often equipped with additional sensors like compass and accelerometers.

[0007] The connectivity of the portable devices allows downloading data, like 2D images and product descriptions from the Internet at almost all times.

[0008] A software application, which uses a hand-held augmented reality device and augmented reality techniques to reconstruct a 2D image of the user's environment augmented with two-dimensional images of consumer products which appear to be part of the physical scene.

[0016] Referring now to the invention in more detail, in FIG. 1 is shown a hand-held augmented reality device 100 that would be used to run the application, which corresponds to this invention. The hand-held augmented reality device comprises a camera 102, fixed or pivoting; a display 101 which in at least one configuration of the camera points in roughly the opposite direction of the camera; a processing unit associated to the camera and the display; an optional networking unit; and an input device capable of taking user input either via a touch screen/pad, keyboard or any other mechanism. The hand-held augmented reality device might also comprise additional sensors like a compass, an accelerometer, gps, gyroscope or any other sensor that could be used to compute the devices position, orientation, velocity and/or acceleration.

[0017] FIG. 3 shows a product image 202 used by the present invention. In this case the product selected is a sofa. The main purpose of the present invention is to synthesize an augmented image by embedding a product image 202 within an environment image 200. Once the product image 202 has been embedded in the environment image 200, it becomes an embedded product image. The embedded product image is a transformation of the product image that causes it to blend with the environment image and create the illusion of being part of the environment.

[0018] The environment image 200 is a still or a video image captured by the camera 102 of the hand-held augmented reality device 100.

[0019] Within its preferred embodiments, this invention includes a dynamic and a static embodiment.

[0020] FIG. 2 shows a sequence of screens that would appear to the user as he/she interacts with the device in a dynamic embodiment of the invention.

[0021] FIG. 2A shows the device 100 displaying on screen: the environment image 200 from the device's camera 102; a three-dimensional environment model 201 rendered by the processing unit associated with the display; and an instruction icon 203 indicating to the user to rotate or move the device in some direction. In this case the instruction is to move the device towards the left hand side of the user.

[0022] The three-dimensional environment model 201 may or may not be rendered over the environment image 200 either in a wireframe mode or in a semi-transparent mode in order for the environment image 200 to show through. This way the user can visualize both the environment image and the three-dimensional environment model overlaid. In the dynamic embodiment of the present invention, the three-dimensional environment model doesn't need to be rendered as the user can go by the instructions alone. However, in the static embodiment of the invention, the visual representation of the three-dimensional environment model is what the user uses to determine the correct camera position and orientation.

[0023] The three-dimensional environment model 201 shown in FIG. 2A comprises a floor and set of walls that resemble as much as practical the environment image in the life image 200. In this case two existing windows in the environment image were represented in the three-dimensional environment model 201.

[0024] The three-dimensional environment model 201 also comprises a virtual camera, which is used to project the three-dimensional environment model onto a 2D image that can be rendered on the device's display.

[0025] The three-dimensional environment model 201 also comprises a product billboard 203 with the product image 202 projected as a texture.

[0026] The product billboard 203 comprises a three-dimensional plane in the three-dimensional environment model 201.

[0027] The product billboard 203, also comprises a normal projection of the product image 202. The transparent sections of the product image 202 will make the product billboard's plane surface invisible making it appear like the object is part of the three-dimensional environment model.

[0028] The instruction icon 203 in FIG. 2A represents one of 12 possible instructions to the user. The 12 possible instructions are a positive and a negative direction for each of the 6 degrees of freedom of the device, namely, X rotation, Y rotation, Z rotation, X translation, Y translation and Z translation.

[0029] The said instructions are computed by a camera awareness engine, which is aware of the device's position and/or orientation in respect to the environment image and uses such information to direct the user towards a position and orientation that matches that of the virtual camera in respect to the three-dimensional environment model.

[0030] FIG. 2B shows the device 100 displaying on screen a situation resulting from the user responding to the instruction in the instruction icon 203 of FIG. 2A. It can be seen that the three-dimensional environment model 201 didn't change in respect to FIG. 2A but the environment image panned to the right in respect to FIG. 2A, as the user moved left.

[0031] FIG. 2C shows the device 100 displaying on screen a situation resulting from the user responding to the instruction on the instruction icon 203 of FIG. 2B. In this case the user found a position, which matches that of the virtual camera causing the three-dimensional environment model and the real scene to be perfectly aligned. In this condition, the product billboard 203 appears to be in the environment image.

[0032] FIG. 4 shows a flow chart of a static embodiment of the present invention. In a static embodiment the system is not aware of the camera's position and/or orientation in respect to the environment image and therefore it can't instruct the user to move or rotate in the desired direction.

[0033] In a static embodiment the user is fully responsible of finding the position and orientation that would match the virtual camera.

[0034] In a static embodiment the user relies on intuition and understanding of perspective to find a position and orientation that would make the three-dimensional environment model and environment image align on screen in the way they do on FIG. 2C.

[0035] User edits three-dimensional environment model 404 on FIG. 4 represents an interaction mode in which the user can edit certain properties of the three-dimensional environment model in order to make it represent as close a possible the environment image. In said mode, the user also has the opportunity to determine the desired position and orientation of the object by moving the product billboard 203 in 3D space.

[0036] In FIG. 4 is shown the event of a user capturing a still snapshot 408 of the composition which will have removed the three-dimensional environment model except for the product billboard 203. This snapshot can be saved to persistent memory or sent and shared via a network connection or tether connection of the device to a personal computer. The snapshot is used by the user's enjoyment and to evaluate how the consumer product would look if purchased and placed on a particular location of their environment image.

[0037] FIG. 5 shows a flow chart of a dynamic embodiment of the present invention. In a dynamic embodiment of the invention, the system comprises a camera awareness module, which deduces the device's camera position and/or orientation in respect to the environment. Using said information the system is able to instruct the user to move or rotate in a particular way.

[0038] In a dynamic embodiment of the present invention, there is a feedback loop between the user moving and rotating the camera 406, the camera awareness module computing the new camera position and/or orientation 502, and the instructions given by the system to the user 503.

[0039] FIG. 6 is a flow chart showing the different components of the present invention.

[0040] In an embodiment of the present invention, a catalog 602 which can be remote and accessible by the hand-held augmented reality device via a network or can be local to the device's memory comprises sets of product images. A set of product images containing at least one product image 202 per product. The catalog 602 may also comprises of image data sets. Each of the image data sets comprises at least one product image set and its corresponding camera parameters 604. Optionally, the said image data set may comprise as well an anchor point 304 and meta-data 603 including real world product dimensions, common product configurations and any other available data pertaining the product.

[0041] Camera parameters 603 constitute a camera model, which describes the projection of a three-dimensional scene onto a two-dimensional image as seen by a real-world camera. There are multiple camera models used in the computer vision field, [Tsai87] being an example of a widely used one which comprises internal camera parameters:

[0042] f—Focal length of camera,

[0043] k—Radial lens distortion coefficient,

[0044] C_ox, C_y—Co-ordinates of centre of radial lens distortion,

[0045] S_x—Scale factor to account for any uncertainty due to imperfections in hardware timing for scanning and digitization,

[0046] And external camera parameters:

[0047] R_x, R_ye, R_z—Rotation angles for the transformation between the world and camera co-ordinates,

[0048] T_x, T_y, T_z—Translation components for the transformation between the world and camera co-ordinates.

[0049] In the context of the present invention camera parameters 604 associated with the product image 603 are provided by the photographer of the product image or are extracted from the product image via a camera calibration process.

[0050] Still in reference to FIG. 6, the network and processing unit 606 refers to the software components which are assumed to execute on a typical programmable processing unit with it's associated memories, buses, network specific hardware and graphics processing unit.

[0051] In more detail, still referring to invention in FIG. 6, a data interface module 607 is responsible for accessing the data in the catalog 602 and making it available to the other software modules. The data interface 607 may comprise networking specific code to access a remote catalog and/or local file management code to access locally available data.

[0052] The data interface may implement a data caching mechanism in order to make recurring access to local or remote data more efficient.

[0053] Still referring to the invention in FIG. 6, the image-rendering unit 609 is associated with the display 614 and is, responsible for generating graphical primitive instructions that translate into visual elements on the hand-held augmented reality device's display screen. In more detail, the image rendering unit 609 takes the environment image from the camera, a three-dimensional environment model from the environment model generation unit 608 and an instruction from the camera awareness module 610 and generates one augmented image like the ones seen on FIG. 2 to be presented to the user.

[0054] Still referring to the invention in FIG. 6, the environment model generation unit 608 is responsible for generating a three-dimensional environment model 201, which represents a set of features from the environment image 200. The features represented by the three-dimensional environment model 201 are used, together with a representation of their counterparts in the environment image 200, by the user 601 and/or by the camera awareness module 610 to determine the device's camera discrepancy with the virtual camera.

[0055] In more detail, referring still to the environment model generation module 608, the virtual camera used to render the three-dimensional environment model is modeled after the object camera parameters 604. Initially, the product billboard 203 which is part of the three-dimensional environment model, is positioned and orientated in respect to the virtual camera such that when projected through the virtual camera it produces an image identical to the object image.

[0056] The rest of the three-dimensional environment model features, for instance, floor plane, wall planes, windows, etc, are positioned in respect to the product billboard and virtual camera in order to create the illusion, from the point of view of the virtual camera, of the object being in a plausible configuration within the three-dimensional environment model.

[0057] The object meta-data 603 is used by the environment model generation 608 to create an initial three-dimensional environment model, which has the object in a plausible configuration. For example, the object meta-data might specify that the object is commonly laying on the ground with its back face against a wall, which, for example, would be the case for a sofa. Said information about the product's configuration together with the product's real-world dimensions and the product's anchor point 304 is enough to generate a three-dimensional scene with a floor and a wall in which the object lays in a plausible configuration.

[0058] Still referring to the environment model generation unit 608 in FIG. 6, the user might be able to modify the three-dimensional environment model by repositioning its three-dimensional features.

[0059] Also, the user might be able to edit the scene by moving and scaling the object in respect to the device's screen. Moving the object in respect to the devices screen can be achieved via a rotation of the virtual camera, which doesn't affect the relative position of the camera in respect to the object.

[0060] Scaling the object in screen space can be achieved by changing the focal length of the virtual camera.

[0061] The object's relative scale in the three-dimensional environment model is well known from the object dimensions in the object meta-data 603.

[0062] The scale factor of the three-dimensional environment model in respect to the environment image needs to be roughly 1.0, for the object to appear of the correct scale in the final composition. There are multiple methods that can be used to get the user to provide a scale reference in the environment image. For example, a user can provide a distance between two walls or the width of a window in the three-dimensional environment model based on a measurement made on the environment image. Alternatively, a user can be instructed to position the camera at a certain distance from the wall against which the object will be positioned. Said distance can be computed based on the known focal length of the device's camera.

[0063] Referring to the camera awareness module 610 in FIG. 6, said module uses inputs from the camera and other sensors to deduce the position and/or orientation of the camera in respect to the environment image. Examples of types of inputs that could be used by different embodiments of the camera awareness module are: compass, accelerometer and the camera images themselves.

[0064] Using an accelerometer the camera awareness module 610 can detect gravity and deduce the vertical pitch of the camera.

[0065] Using a compass the camera awareness module 610 can deduce the horizontal orientation of the camera. Using computer vision algorithms like camera calibration and feature tracking from the video, the camera awareness module 610 can deduce, the internal camera parameters.

[0066] The camera awareness module can use some or all the available cues to make an estimation of the camera's position and/or orientation in respect to the environment image. The estimation generated by the camera awareness module is compared against the virtual camera in order to generate an instruction 203 for the user.

[0067] Referring to the camera awareness initialization module 611 in FIG. 6, the said module mostly implements an interaction mode where input is obtained from the user in order to initialize the camera awareness module 610.

[0068] Some of the cues used by the camera awareness module might require a reference value in order to be usable. For example, the compass will provide with an absolute orientation, however, the orientation of the environment image is unknown and therefore an absolute orientation alone is not sufficient to deduce the camera orientation in respect to the environment image. In the mentioned example, a user would point the camera in a direction perpendicular to the “main” wall in the environment image and press a button to inform the camera awareness initialization module 611 of the absolute orientation of said wall.

[0069] In the case of the accelerometer sensing gravity, there is no need for an initialization step because gravity is constant in all familiar frames of reference.

[0070] In the case of computer vision algorithms applied to the camera's video, there are many different algorithms and techniques that can be used. In some of such techniques, a set of features in the three-dimensional environment model need to be matched with their counterparts in the environment image. After such initialization step, typically a feature-tracking algorithm keeps the correspondence persistent as the camera moves and turns. A camera calibration algorithm uses the correspondence information together with the 3d and 2d coordinates of the tracked features to estimate the camera parameters. Other camera calibration algorithms might not require an initialization phase by using a well know object as a marker, which is placed on the real-worlds scene and detected by the camera awareness module in the camera video images.

[0071] FIG. 7 shows a computer that may have a processing unit, data interface, image manipulation module, camera, user interface, display, and sensors connected by a bus.