US 6,697,062 B1Grant

Reflection space image based rendering

Issue Date:2004-02-24

•32 Claims

•10 Drawing Sheets

Abstract

The present invention provides a method, system, and computer program product for reflection space image based rendering of an object at an interactive frame rate. A set of source radiance environment maps associated with a set of source viewing vectors are warped to create a destination radiance environment map associated with a destination viewing vector in a current frame. Blending and weighting operations can also be applied in creating the final destination radiance environment map. An object is then rendered with texture environment mapped from the destination radiance environment map.

Metadata

Assignee

Microsoft Corporation

Inventors

Brian Keith Cabral
Marc Olano
Philip Nemec

Application Information

Application Number:US 09/369,359

Filing Date:1999-08-06

Priority Date:1999-08-06

Art Unit:7

Classifications

IPC:

G06T 1700

Field of Search:

345418345419345582345583345586345426345423345429345581345646345644345645345420345424

Patent Drawings (10 sheets)

Description

BACKGROUND OF THE INVENTION

[0002] 1. Field of the Invention

[0003] The present invention relates to the field of computer graphics.

[0004] 2. Related Art

[0005] Offline rendering algorithms have to a great extent conquered physically accurate photo-realism and complex synthetic shading. A result of over twenty years of research, these techniques all solve the lighting or rendering equation in some manner. See, the techniques in Blinn, J. F. and Newell, M. E., Comm. ACM19:542-546 (1976); Cook, R. L., et al., “The Reyes image rendering architecture,” in Computer Graphics(SIGGRAPH'87 Proc.), vol. 21, Stone, M. C., ed., (July 1987), pp. 95-102; Debevec, P., “Rendering synthetic objects into real scenes: Bridging traditional and image-based graphics with global illumination and high dynamic range photography,” in SIGGRAPH98 Conf. Proc., Cohen, M., ed., Annual Conference Series, ACM SIGGRAPH, Addison Wesley (July 1998), pp. 189-198; He, X. D., et al., “A comprehensive physical model for light reflection,” in Computer Graphics(SIGGRAPH'91 Proc.), vol. 25, Sederberg, T. W., ed. (July 1991), pp. 175-186; Jensen, H. W. and Christensen, P. H., “Efficient simulation of light transport in scenes with participating media using photon maps,” in SIGGRAPH98 Conf. Proc., Cohen, M., ed., Annual Conference Series, ACM SIGGRAPH, Addison Wesley (July 1998), pp. 311-320; Miller, G. S. and Hoffinan, C. R., “Illumination and reflection maps: Simulated objects in simulated and real environments, in SIGGRAPH'84 Advanced Computer Graphics Animation seminar notes(July 1994); Poulin, P. and Fournier, A., “A model for anisotropic reflection, ” in Computer Graphics(SIGGRAPH'90 Proc.), vol. 24, Baskett, F., ed., (August 1990), pp. 273-284; Veach, E. and Guibas, L. J., “Metropolis light transport,” in SIGGRAPH97 Conf. Proc., Whitter, T., ed., Annual Conference Series, ACM SIGGRAPH, Addison Wesley (August 1997), pp. 65-76, and the rendering equation in Kajiya, J. T., “The rendering equation,” Computer Graphics(SIGGRAPH '86 Proc.), vol. 20, Evans, D. C. and Athay, R. J., eds., (August 1986), pp. 143-150.

[0006] A trade-off between speed and quality exists in off-line rendering and interactive rendering. The outstanding rendering challenge now becomes how to increase the performance of sophisticated shading algorithms without losing the advancements made in quality. This implies that many orders of magnitude in performance improvements must be found. Traditionally, this has been accomplished by vastly simplifying the approximations used in the shading and lighting equations—resulting in a significant loss in complexity and quality.

[0007] Environment mapping is one method used to improve the realism of interactive rendering. As originally described by Newell and Blinn (Blinn, J. F. and Newell, M. E., Comm. ACM19:542-546 (1976)), a simple environment map is used to quickly find reflections of distant objects from a perfectly mirrored surface. Other researchers refined this notion by generalizing the BRDF used, though some of these refinements lost the interactivity of simple environment mapping. See, Cabral, B., et al., “Bidirectional reflection functions from surface bump maps,” in Computer Graphics(SIGGRAPH'87 Proc.), vol. 21, Stone, M. C., ed., (July 1987), pp. 273-281; Greene, N., “Applications of world projections,” in Proc. Graphics Interface'86, Green, M., ed. (May 1986), pp. 108-114; Miller, G. S. and Hoffman, C. R., “Illumination and reflection maps:

[0008] Simulated objects in simulated and real environments, in SIGGRAPH'84 Advanced Computer Graphics Animation seminar notes(July 1994); Poulin, P. and Fournier, A., “A model for anisotropic reflection, ” in Computer Graphics(SIGGRAPH'90 Proc.), vol.24, Baskett, F., ed., (August 1990), pp. 273-284.

[0009] Another method used to bridge the gap between realism and interactivity is image based rendering (McMillan, L. and Bishop, G., “Plenoptic modeling: An image-based rendering system,” in SIGGRAPH95 Conf. Proc., Cook, R., ed., Annual Conference Series, ACM SIGGRAPH, Addison Wesley (August 1995), pp. 39-46). Image based rendering (IBR) avoids solving the lighting equation during interactive rendering by warping existing photographs or images. These images can be thought of as radiance maps (Gershbein, R., et al., “Textures and radiosity: Controlling emission and reflection with texture maps,” in Proc. SIGGRAPH'94 (Orlando, Fla., Jul. 24-29, 1994), Glassner, A., ed., Computer Graphics Proceedings, Annual Conference Series, ACM SIGGRAPH, ACM Press (July 1994), pp. 51-58), and generalized to light fields (Levoy, M. and Hanrahan, P., “Light field rendering,” in SIGGRAPH96 Conf. Proc., Rushmeier, H., ed., Annual Conference Series, ACM SIGGRAPH, ACM Press (August 1996), pp. 31-42) and lumigraphs (Gortler, S. J., et al., “The lumigraph,” in SIGGRAPH96 Conf. Proc., Rushmeier, H., ed., Annual Conference Series, ACM SIGGRAPH, ACM Press (August 1996), pp. 43-54). This works well for predefined scenes or images, but not for dynamically changing synthetic objects.

[0010] Recently, Debevec (Debevec, P., “Rendering synthetic objects into real scenes: Bridging traditional and image-based graphics with global illumination and high dynamic range photography,” in SIGGRAPH98 Conf. Proc., Cohen, M., ed., Annual Conference Series, ACM SIGGRAPH, Addison Wesley (July 1998), pp. 189-198) combined captured environment maps and synthetic objects to produce compelling renderings with both synthetic objects and image based environments. His techniques do not work at interactive rates since he computes the lighting equation integration as he renders using RADIANCE (Ward, G. J., “The RADIANCE lighting simulation and rendering system,” Proc. SIGGRAPH'94 (Orlando, Fla., Jul. 24-29, 1994), Glassner, A., ed., Computer Graphics Proceedings, Annual Conference Series, ACM SIGGRAPH, ACM Press (July 1994), pp. 459-572).

[0011] What is needed is an interactive photo-realistic rendering algorithm or system.

SUMMARY OF THE INVENTION

[0012] The present invention provides a method, system, and computer program product for reflection space image based rendering of an object at an interactive frame rate. A set of source radiance environment maps associated with a set of source viewing vectors are warped to create a destination radiance environment map associated with a destination viewing vector in a current frame. Blending and weighting operations can also be applied in creating the final destination radiance environment map. An object is then rendered with texture environment mapping from the destination radiance environment map.

[0013] According to one embodiment, at set-up, geometry information for a global set of source viewing vectors and a global set of source radiance environment maps are loaded. A viewpoint tracking loop is then carried out for a current frame. In the viewpoint tracking loop, a destination viewing vector that points along a view direction of the current frame is determined. A subset of source viewing vectors is then determined from the previously loaded global set of source viewing vectors. In one preferred example, the subset of source viewing vectors are a number of viewing vectors (e.g., three source viewing vectors) closest to the destination viewing vector. A set of corresponding weights for the subset of source viewing vectors is also determined.

[0014] A warping and blending loop is then performed iteratively on each source radiance environment map corresponding to the subset of source viewing vectors. In one embodiment, each source radiance environment map is warped and blended per texel according to a warping function to a create a destination radiance environment map.

[0015] In another preferred embodiment, a fast approximation is made which avoids warping per texel over the entire source radiance environment map. A tessellated mesh with a set of warped texture coordinates is generated. For example, in the case where the source radiance environment maps are sphere maps, the tessellated mesh is simply a tessellated disk. The set of warped texture coordinates at each vertex of the tessellated mesh represent a respective source radiance environment map texel which has been warped according to a warping function evaluated at the destination viewing vector. The mesh is then rendered (also referred to as drawn) using the respective source radiance environment map as a texture map to obtain a texture image. These texture images are also blended with weights corresponding to the respective source viewing vectors. The blended texture images are accumulated in a buffer (such as a frame buffer) to obtain a final destination radiance environment map. Control returns to the viewpoint tracking loop.

[0016] The final destination radiance environment map can be further normalized by weight if the weights were not previously normalized. The final destination radiance environment map is then loaded as environment map. An object or scene is then rendered with texture environment mapped from the final destination radiance environment map.

[0017] The present invention can be implemented in software, firmware, hardware, or in any combination thereof. The present invention further can be implemented in any graphics architecture or system that supports texture mapping and environment mapping. In one example, not intended to limit the present invention, a routine implementing the present invention is executed on a host processor using a graphics API or library, such as, OpenGL®.

[0018] Thus, the present invention goes beyond the work of Debevec by pre-integrating the lighting equation to allow interactive rendering—in part by constraining the geometry of the global radiance environment maps. This introduces a view-dependence in the generated maps, which is overcome with a new form of IBR in reflection space. The result is interactive rendering of synthetic objects using a wide range of surface reflectances (including but not limited to those described by BRDFs) in arbitrary lighting environments.

[0019] There are at least two contributions made by the present invention. The first is the application of IBR techniques in reflection space. This is called reflection space IBR—even though the invention does not necessarily operate on normal images, nor is the result a rendered image. Second is our hybrid rendering algorithm. The algorithm of the present invention and the nature of the source radiance environment maps allows new surface reflectances (e.g., BRDFs) and new geometry not found in the original source images to be used—thus extending the class of renderings possible with interactive IBR algorithms.

[0020] In sum, the present invention is an intermediate step toward interactive photo-realistic rendering—an algorithm that uses IBR techniques to approximate the integral lighting equation. Such an interactive algorithm is in demand now and will change the nature and work-flow of artists, designers and scientists.

[0021] Further features and advantages of the present invention, as well as the structure and operation of various embodiments of the present invention, are described in detail below with reference to the accompanying drawings.

BRIEF DESCRIPTION OF THE FIGURES

[0022] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate the present invention and, together with the description, further serve to explain the principles of the invention and to enable a person skilled in the pertinent art to make and use the invention.

[0023] FIGS. 1A and 1B are a flowchart of a routine for reflection space image based rendering according to one embodiment of the present invention.

[0024] FIG. 2 is a diagram of a radiance environment sphere map.

[0025] FIG. 3 is a diagram of an example viewing vector v that lies within a spherical patch.

[0026] FIG. 4 shows an image of an automobile in an environment generated according to one automobile example of the present invention.

[0027] FIG. 5 shows example images of source radiance environment maps, corresponding destination radiance environment maps after warping and blending with appropriate spherical barycentric weighting, and a final generated radiance environment map according to an example of the present invention.

[0028] FIG. 6 shows all of the textures used to render an example image of a full model car.

[0029] FIG. 7 is a diagram of an example architecture that can support the routine of FIG. 1.

[0030] FIG. 8 is a block diagram of an example graphics system that can support the routine of FIG. 1.

[0031] FIG. 9 is an example graphical user-interface computer system that can support the routine of FIG. 1.

[0032] The present invention is described with reference to the accompanying drawings. In the drawings, like reference numbers indicate identical or functionally similar elements. Additionally, the left-most digit(s) of a reference number identifies the drawing in which the reference number first appears.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0033] 0. Overview and Terminology

[0034] 1. Routine for Reflection-Spaced Image-Based Imaging

[0035] 1.1 Set-up

[0036] 1.2 Viewpoint Tracking Loop

[0037] 1.3 Warping and Blending Loop

[0038] 2. Radiance Environment Maps

[0039] 2.1 Obtaining Radiance Environment Maps

[0040] 3. Reflection Space IBR

[0041] 3.1 Sampling View Directions

[0042] 3.2 Map Warping

[0043] 3.3 Specific Warps

[0044] 3.4 Spherical Barycentric Interpolation

[0045] 4. Rendering Algorithm

[0046] 5. Examples

[0047] 6. Example Implementations

[0048] 7. Example GUI Computer System

[0049] 8. Conclusion

[0050] 0. Overview and Terminology

[0051] High quality, physically accurate rendering at interactive rates has widespread application, but is a daunting task. The present invention bridges the gap between high quality offline and interactive rendering by using existing environment mapping hardware in combination with a new Image Based Rendering (IBR) algorithm. One aspect of the present invention lies in performing IBR in reflection space. The present invention can be applied to ordinary environment maps, but for more physically accurate rendering, reflection space IBR is applied to radiance environment maps. In one example, a radiance environment map pre-integrates a Bidirectional Reflection Distribution Function (BRDF) with a lighting environment. A BRDF can be, but is not limited to, an arbitrary four dimension (4D) function that describes how a physical surface (or non-physical surface) reflects light. Using the reflection-space IBR algorithm on radiance environment maps allows interactive rendering of arbitrary objects with a large class of complex BRDFs in arbitrary lighting environments. The ultimate simplicity of the final algorithm suggests that it will be widely and immediately valuable given the ready availability of hardware-assisted environment mapping.

[0052] The term “radiance environment map” is used broadly herein to refer to any type of environment map including viewpoint dependent and viewpoint independent maps. A “radiance environment map” can include but is not limited to, any conventional environment map (both pre-integrated and non-pre-integrated). In one example, a radiance environment map can include but is not limited to one which pre-integrates a Bidirectional Reflection Distribution Function (BRDF) with a lighting environrment.

[0053] 1. Routinefor Reflection-Spaced Image-Based Imaging

[0054] 1.1 Set-up

[0055] FIGS. 1A and 1B are a flowchart of a routine for reflection-space, image-based rendering (IBR) according to one embodiment of the present invention. Routine 100 includes steps 110 and 120, and a viewpoint tracking loop 125. Steps 110 and 120 are carried out as part of a setup stage. The viewpoint tracking loop 125 is carried out for each frame being rendered at a destination viewpoint. Viewpoint tracking loop 125 includes steps 130 to 190, including a warping and blending loop 155. Warping and blending loop 155 includes a warping and blending operation (step 160) which is carried out for each source radiance environment map in a subset to create a destination radiance environment map at the destination viewpoint of the current frame. FIG. 1B shows the steps involved in the warping and blending loop of step 160 in further detail (steps 162-169) according to one example of the present invention.

[0056] At setup, geometry information of source viewing vectors is loaded (step 110). A global set of N source radiance environment maps is also loaded (step 120). Each of these steps is similar to conventional radiance environment mapping except that multiple source viewing vectors and multiple source radiance environment maps are used. For example, in the case of spherical source radiance environment maps are used, spherical geometry information is loaded in step 110. For example, geodesic locations of each of the N source viewing vectors is loaded. Similarly, in step 120, multiple source radiance environment maps representing spherical environment maps taken at different viewpoints are loaded. In general, each of the source radiance environment maps can be taken from any different viewpoint. In one preferred example, twelve source radiance environment maps are used representing twelve approximately equidistant viewpoints in a three-dimensional space around a sphere.

[0057] The present invention can be used with any type of environment map including, but not limited to, radiance environment maps of a spherical or cubic type. In one preferred example, preintegrated radiance environment maps are used of either a spherical or cubic type. An example radiance environment sphere map is shown in FIG. 2. Prior to loading, of course, the multiple source radiance environment maps must be generated off-line. The multiple source radiance environment maps can be generated through photography or computer generated using ray tracing algorithms or other techniques. The generation of the source radiance environment maps is described in further detail below in section 2.

[0058] 1.2 Viewpoint Tracking Loop

[0059] First, destination viewing vector coordinates are computed (step 130). These coordinates define the destination viewing vector of the current frame in three-dimensional space. In general, any three-dimensional coordinate system or coordinate space can be used. An example destination viewing vector v_din three-dimensional space is illustrated in FIG. 3, as described further below in section 3.4.

[0060] Next, a subset of the set of source viewing vectors is determined (step 140). This subset has a number of source viewing vectors M, where M is less than N. In one preferred example, the determination involves finding a number of source viewing vectors (e.g., three) in the set of N source viewing vectors which are nearest to the computed destination viewing vector. In this way, the subset of source viewing vectors corresponds to a subset of the previously-loaded source radiance environment sphere maps which were generated for viewpoints closest to the destination viewpoint of the current frame being rendered.

[0061] In step 150, weights are then computed for the subset of viewing vectors determined in step 140. In the case of a sphere geometry example, each of the weights correspond to respective areas of a spherical patch. For example, as illustrated in FIG. 3 and described in further detail in section 3.4 below, if three viewing vectors v₀, v₁, v₂are determined to be nearest to a particular point of view given by a destination to viewing vector V_D, then a spherical patch is defined by the three viewing vectors v₀, v₁, v₂. This spherical patch is a spherical triangle formed by the viewing vectors v₀, v₁, v₂. Destination viewing vector V_Ddefines the line of sight of the viewer and forms three spherical triangles within the largest spherical patch. As shown in FIG. 3, areas a₀, a₁, and a₂represent three weights for the sphere maps associated with the viewing vectors v₀, v₁, and v₂, respectively. In this way, the contribution of texture from a source environment map at a source viewing vector can be weighted so that its contribution to a final destination source radiance environment map is proportional to how close the source viewing vector is to the destination viewing vector.

[0062] 1.3 Warping and Blending Loop

[0063] In loop 155, each source radiance environment map corresponding to the subset M of source viewing vectors determined in step 140 is warped and blended to create a destination radiance environment map at the destination viewpoint of the current frame (step 160). In general, each source radiance environment map is warped to create the destination radiance environment map. Blending is used to further improve the quality of the final destination radiance environment map. Blending, of course, could be omitted in certain applications. Also, any type of blending can be used, including, but not limited to the examples of blending described herein.

[0064] Specific warping functions for mapping texels from each source radiance environment map to a destination radiance environment map are described in further details in Section 3.2 and 3.3 below. Warping can be expensive if performed per texel over an entire source radiance environment map. Even the case where three source radiance environment maps are warped to create a single, final destination radiance environment map can be expensive and impractical for an interactive application if warping is performed per texel. FIG. 1B shows an example routine that uses a fast, accurate approximation by rendering a final destination radiance environment map as a tessellated mesh. In an example of a destination sphere map, the mesh is a tessellated disk. Texture coordinates at each mesh vertex are then chosen according to a warping function, and the source radiance environment map or image is used as a texture while rendering a tessellated disk into the frame buffer.

[0065] An example of this process is shown in the loop 162 in FIG. 1B (steps 164-169). Loop 162 is repeated for each viewpoint v; in subset M. For example, in the case of three viewing vectors in subset M, loop 162 is repeated three times (i=0 to M−1).

[0066] A tessellated mesh is generated with a respective set of warped texture coordinates (step 164). In the case of a sphere map, the tessellated mesh corresponds to a tessellated disk. The set of warped texture coordinates corresponds to the respective source radiance environment map warped according to a warping function. In the present invention, a warping function warps a radiance environment map for one view direction to another view direction. Any warping function can be used depending on a particular application and desired effect.

[0067] Two examples of warp functions are further described in Section 3.3 below. One warp function is best used where a central reflection is a surface normal as for diffuse objects. The second warp function is for BRDFs where the central reflection direction generally aligns with the mirror reflection direction as is the case in more shiny objects.

[0068] An identity warping function can also be used. An identity warping function is one where essentially no warping only blending occurs. This can be used for example in a case of a mirror warp for a cube map (in other words, in an identity P_dwould equal p_sin the warp described below).

[0069] Next, the mesh is rendered or drawn (step 166). The respective warp texture coordinates are used to map texture from the respective source radiance environment map to obtain a textured image. Any conventional texture mapping technique can be used including, but not limited to, mip-mapping. The textured image is further blended at a respective weight and accumulated to obtain a weighted destination radiance environment map in a frame buffer or other accumulation buffer (step 168). In step 169, a check is made to determine if i=M−1, that is, whether each source radiance environment map has been warped and blended as described with respect to steps 164-168 . Once i=M−1 and all source radiance environment maps in subset M have been warped and blended, then control returns to step 170.

[0070] In step 170, the weighted destination radiance environment map is further normalized according to the M weights. Note that step 170 is optional. Step 170 can also be omitted if the weights are first normalized in step 150. The normalized weighted destination radiance environment map is then loaded into memory as an environment map (step 180). An object or scene is then rendered using the normalized destination radiance environment map (step 190). Any conventional environmental mapping technique can be used to map texture from the loaded normalized weighted destination environment map to the object during rendering in step 190 as would be apparent to a person skilled in the art given this description.

[0071] After step 190, viewpoint tracking loop 125 repeats for each frame at interactive rates. Interactive rates of approximately 10 frames per second and higher are achieved. In one example, on a Silicon Graphics® Onyx2™ InfiniteReality2™, the interactive viewing program ran at a sustained frame rate of 20 Hz. Such rates are not intended to limit the present invention, however, and in general, faster and slower interactive rates are possible depending on a particular application and hardware support that is provided. Routine 100 is further described with respect to example radiance environment maps and specific warps in Sections 2 through 3 below. An example of routine 100 written in pseudo-code is presented in section 4. Examples of routine 100 applied to rendering an automobile in an environment are further shown below in Section 5. Examples implementations of routine 100 and software, firmware and/or hardware and different types of graphics architectures are also described in Section 6 below.

[0072] 2. Radiance Environment Maps

[0073] A traditional environment map records the incident radiance, L_i, from each direction. The two most common representations are the sphere map and the cube map. A sphere map is a view-dependent representation, equivalent to an orthographic view of a reflective sphere. Note that it is not necessary to render an orthographic view when using a sphere map, for example the sphere mapping in OpenGL® 1.2 includes the appropriate correction for perspective views. A cube map is a view-independent representation, created by projecting the environment onto the faces of a cube. These examples are illustrative and not intended to limit the present invention. Other representations of an environment in a reflection space are also possible as would be apparent to a person skilled in the art given this description. For example, a parabolic map can be used. See View-independent environment maps ;Wolfgang Heidrich, and Hans-Peter Seidel; Proceedings of the 1998 EUROGRAPHICS/SIGGRAPH workshop on Graphics hardware, 1998, Page 39.

[0074] Instead of recording incoming radiance, a radiance environment map records the total reflected radiance, L_r, for each possible surface orientation. It is defined by the classic lighting equation (Immel, D. S., et al., “A radiosity method for non-diffuse environments,” Computer Graphics(SIGGRAPH'86 Proc.), vol. 20, Evans, D. C. and Athay, R. J., eds., (August 1986), pp. 133-142):

L_{r} (θ_{r}, φ_{r}) = \int_{H} f_{r} (θ_{i}, φ_{i}; θ_{r}, φ_{r}) L_{i} (θ_{i}, φ_{i}) \cos θ_{i} ⅆ ω_{i}

[0075] Thus, L_ris the environment (L_i) modulated by the BRDF (f_r). H is the hemisphere above the surface, which varies with surface orientation (see FIG. 2). Each texel in a radiance environment map captures the pre-integrated value for L_rfor one possible orientation of H. Since the radiance environment map is indexed by orientation, we can choose any of the storage representations used for traditional environment mapping.

[0076] FIG. 2 shows an example sphere map representation. FIG. 2 is a diagram of a radiance environment sphere map. A radiance environment sphere map, L_r;s, is defined on the image plane shown on the left. Each point (s, t) in L_r;sis associated with a normal, n, and a reflection vector, r. The normal specifies the hemisphere, H(n) over which L_i^H(n)is defined.

[0077] The pre-integrated radiance environment map introduces a couple of restrictions for rendering. Since all surface points that have a common normal use the same reflected radiance, only the lighting contribution from a distant environment can be captured, not reflection of local objects or inter-reflection of a single surface. Also, in one preferred example isotopic BRDFs are used (with only a single reflected radiance stored in the map per surface normal it is difficult to record BRDF variation with rotation around the normal).

[0078] An approximation is also introduced in the true lighting equation. A radiation environment map is computed with a single view direction, so it is technically incorrect to use it with a perspective view, where the view direction changes from pixel to pixel. While graphics hardware corrects for perspective-induced changes in mirror reflection direction, the correction is not always appropriate for the radiance environment map. Regardless, in the present invention perspective views can be rendered anyway. The usually minor resulting errors, if any, are easily accepted as part of the price for interactivity.

[0079] 2.1 Obtaining Radiance Environment Maps

[0080] Prior to step 110, a set of radiance environments must first be obtained or generated. In one preferred example, pre-integrated radiance environment maps are first obtained or generated in a pre-processing stage. In general, any conventional method for obtaining a set of radiance environment maps for different viewpoints can be used.

[0081] One method to obtain radiance environment maps is to take photographs in the desired environment of a physical sphere whose surface BRDF matches that of the target object. The photographs should be taken with a narrow field view lens to approximate an orthographic projection and to minimize the reflection of the camera. The resulting images are the radiance environment maps, with the integration done by nature. The algorithm of the present invention uses several radiance environment maps, so the inventors use several such images along with the camera orientation for each.

[0082] A second method is to compute lighting integral numerically given a desired BRDF and lighting environment. The lighting environment should be known with high dynamic range for good integration results. Such environments can be captured through photographs by the methods of Debevec (Debevec, P. E. and Malik, J., “Recovering high dynamic range radiance maps from photographs,” in SIGGRAPH97 Conf. Proc., Cohen, M., ed., Annual Conference Series, ACM SIGGRAPH, Addison Wesley (August 1997), pp. 369-378), or rendered with a package like RADIANCE (Ward, G. J., “The RADIANCE lighting simulation and rendering system,” Proc. SIGGRAPH'94 (Orlando, Fla., Jul. 24-29, 1994), Glassner, A., ed., Computer Graphics Proceedings, Annual Conference Series, ACM SIGGRAPH, ACM Press (July 1994), pp. 459-572). Six photographs or images have been used to represent L_i, arranged as a cube environment map in the present invention. See also Voorhies, D. and Foran, J., “Reflection vector shading hardware,” Proc. SIGGRAPH'94 (Orlando, Fla., Jul. 24-29, 1994), Glassner, A., ed., Computer Graphics Proceedings, Annual Conference Series, ACM SIGGRAPH, ACM Press (July 1994), pp. 163-166. Since the BRDF and the environment map, L_i, are decoupled, the lighting environment can be reused to compute L_rfor many different surface types. Results using maps computed in this way are shown in FIG. 4.

[0083] 3. Reflection Space IBR

[0084] With conventional IBR, the light field is sampled by a discrete set of images. For the present invention's algorithm, these samples are a set of radiance environment maps taken from different viewpoints. These maps must be warped to match a new point of view, then blended together.

[0085] In addition to matching the viewpoint, the warping correlates features on the different maps. For traditional IBR, the image correlation may require only an affine or projective warp (Seitz, S. M. and Dyer, C. R., “View morphine: Synthesizing 3D metamorphoses using image transforms,” in SIGGRAPH96 Conf. Proc., Rushmeier, H., ed., Annual Conference Series, ACM SIGGRAPH, Addison Wesley (August 1996), pp. 21-30). For general light fields, it can require gathering light rays in a variety of discontinuous patterns (Levoy, M. and Hanrahan, P., “Light field rendering,” in SIGGRAPH96 Conf. Proc., Rushmeier, H., ed., Annual Conference Series, ACM SIGGRAPH, ACM Press (August 1996), pp. 31-42).

[0086] Since each point in a radiance environment map is an integration of the environment and BRDF, the warp that best correlates features in the environment can vary from BRDF to BRDF. By choosing a warp that models the BRDF well, one can significantly reduce the number of radiance environment maps required for good reconstruction. If the BRDF is a perfect mirror and the warp models it as a perfect mirror, one need only sample well enough to catch the highest frequencies in the environment. If the warp does not match the BRDF, one must sample well enough for the product of the highest frequencies in the BRDF and environment. This is because the lighting integral is essentially a convolution of the BRDF with the environment.

[0087] According to a further feature, for BRDFs that are principally reflective, a warp is used that matches the reflection directions of the different maps. So a point on the source image warps to the point on the destination image that reflects in the same direction. Primarily diffuse BRDFs suggest a warp that matches surface normal directions. One can find a well-matched warp for any BRDF that is radially symmetric about a principal direction and does not change shape across the surface. With such a BRDF, the same area of the environment will be integrated for corresponding points from different views.

[0088] Lambertian diffuse reflection and Phong specular reflection both satisfy this restriction, but most more realistic BRDFs do not. Fortunately, since the radiance environment maps sample a smooth, continuous function, the present invention can effectively handle a much wider class of BRDFs that are close to the symmetric ideal without requiring a large number of sample maps. For example, the inventors have used a numerically computed BRDF with Fresnel effects and diffuse, specular and backscatter components. For the BRDF, the inventors use a warp that matches mirror reflections. It works because the specular lobe is the only high-frequency component of the BRDF and its shape does not vary too much from texel to texel. The Fresnel effects are naturally handled by the method, and the other components do not require a large number of sample maps because they are low frequency.

[0089] Once the sample maps have been warped to the new viewpoint, they must be combined with some reconstruction filter. Oppenheim and Schafer (Oppenheim, A. V. and Schafer, R. W., Digital Signal Processing, Prentice-Hall, Englewood Cliffs, N.J. (1975)) describe many sampling and reconstruction choices. For simplicity and efficiency, linear interpolation is used between neighboring images. The linear interpolation uses a spherical form of barycentric weights, presented in section 3.4. Thus, for any given viewpoint, the three nearby radiance environment maps are warped and blended to create an approximation to the new map (see FIG. 5).

[0090] 3.1 Sampling View Directions

[0091] Certain environment map representations (e.g., cube maps) are viewpoint independent, while others (e.g., sphere maps) depend on the viewpoint. In contrast, each radiance environment map, whether it is stored in cube map, sphere map or another form, is correct for only a single viewpoint. This is because the radiance environment map captures Fresnel reflection and other view-dependent effects.

[0092] As alluded to above, the view-dependence does not limit the use of each map to only one view. This limitation is overcome by precomputing a set of maps—denoted L_rj, jε{0 . . . N−1}—at various viewpoints. The unit view vectors can be thought of as points lying on a sphere. The inventors use reflection-space IBR to reconstruct the map for rendering from the L_rjmaps, but in one example, still require reasonable coverage of the sphere of possible view directions to avoid aliasing artifacts. In one case, the inventors have used one L_rfor each viewpoint defined at the vertices of an icosahedron. The number of samples has been sufficient for the environments and BRDF which have been employed and is desirable because its symmetry means that each viewpoints is handled in an unbiased manner.

[0093] 3.2 Map Warping

[0094] Each warp is between a source map L_r;s(from the precomputed set L_rj) and a destination map, L_r;d(for the current rendering viewpoint). Points in these maps will be called p_sand P_drespectively.

[0095] For each map point, p, there is a vector r along the central reflection direction of the BRDF. For Phong specular or perfect mirror reflections, r is the geometric reflection vector. For diffuse surfaces, r is the surface normal. To assist in the warp, define an invertible function

[0096] g:p→r

[0097] g(p) depends on both the BRDF and the map representation. It is most easily defined in terms of a local coordinate system for each map, so the inventors also define a transformation per map to convert the local coordinate system to a common global space

[0098] T:r→{overscore (r)}

[0099] The composition of these functions defines the full warp from p_sto P_d:

[0100] p_d=g_d⁻¹·T_d⁻¹·T_s·g_s(p_s)

[0101] This takes a point in L_r;s(defined by s and t texture coordinates for a sphere map representation). It is converted first to a 3D reflection vector in the local coordinate system associated with L_r;s. This 3D vector is transformed to the global space, then to a vector in the local coordinate system associated with L_r;d. Finally, the resulting vector is converted to a point in L_r;d(once again given by two texture coordinates if one uses the sphere map representation).

[0102] 3.3 Specific Warps

[0103] Two specific warp functions are derived. For clarity, only two warping functions are described in detail. However, other warping functions can be used as would be apparent to a person skilled in the art given this description. Both use a sphere-map representation for L_r;sand L_r;d. The first is for BRDFs where the central reflection direction is the surface normal. The second is for BRDFs where the central reflection direction aligns with the mirror reflection direction.

[0104] For both warps, the local coordinate system associated with each map is aligned with the camera used to create the map. The x-axis points right, the y-axis points up and the z-axis points from the origin toward the camera. Thus transformations T_sand T_dare defined as 3×3 matrices with columns equal to three axes expressed in global coordinates.

g_{norma l} (s, t) = (\begin{matrix} 2 s - 1 \\ 2 t - 1 \\ \sqrt{1 - {(2 s - 1)}^{2} + {(t2 - 1)}^{2}} \end{matrix})

g_{normal}^{- 1} (x, y; z) = (\frac{x}{2} + 0.5, \frac{y}{2} + 0.5)

[0105] We base the mirror reflection warp, g_mirror, on the x, y and z produced by g_normal:

g_{mirror} (s, t) = (\begin{matrix} 2 xz \\ 2 yz \\ 2 z^{2} - 1 \end{matrix})

g_{mirror}^{- 1} (x, y, z) = g_{normal}^{- 1} (\frac{(x, y, z + 1)}{\sqrt{x^{2} + y^{2} + {(z + 1)}^{2}}})

[0106] In one example, since graphics hardware is available to do mirror reflections with a sphere map, the final stage of both warps is modified to use g_mirror⁻¹. The following functional composition chains define the two warps:

[0107] p_d=g_mirror⁻¹·T_d⁻¹·T_s·g_normal(p_s)

[0108] p_d=g_mirror⁻¹·T_d⁻¹·T_s·g_mirror(p_s)

[0109] Performing three of these warps per texel in the target map for every rendered frame is expensive and impractical for an interactive application. A fast, accurate approximation is possible by rendering the destination sphere map as a tessellated disk. Texture coordinates at each mesh vertex are chosen according to the warping function, and the source image is used as a texture while rendering the disk into the frame buffer. To account for the non-linearity of the warp functions, the mesh is finer toward the edges of the disk and coarser near the center. The 3D coordinate system associated with the destination map changes as the view moves, but the same disk tessellation can always be used. The reflection vectors, r_d, also remain constant for each mesh vertex and can be precomputed.

[0110] The piecewise linear approximation to the warp is accurate for most of the sphere map area. In a sphere map representation, the mirror warp has a singularity at the limit of extreme grazing reflections around the edge of the map—the reflection direction exactly opposite the view vector. The warp equation from L_r;sto L_r;dcan fail at this singularity.

[0111] One can locate P_dfor the singularity by warping the problem reflection direction (the negated source map view vector) into the destination map. Near this point in the destination map, the source map will become pinched and unreliable instead of warping cleanly. Thus, the inventors use a simple distance from p_din the destination map to weight our confidence in the warped source image. This weight is used to fade the contribution of each source near its respective singularity.

[0112] The textured disk method accelerates the warping operation in two ways. First, s and t are not explicitly calculated for all the points on the sphere map, only at the vertices of the mesh. Second, a major bottleneck in performing the warp is accessing memory associated with the source and destination maps. The inventors leverage the rendering and texturing hardware to solve this memory bottleneck.

[0113] 3.4 Spherical Barycentric Interpolation

[0114] Once the warps have taken place the warped images must be blended. One example interpolation scheme according to the present invention is a spherical variant of classic affine barycentric interpolation, as defined in Farin (Farin, G., Curves and Surfaces for Computer Aided Geometric Design, Academic Press, (1990)). Classic barycentric interpolation uses the ratio of the areas of triangles. The inventors instead use the ratio of spherical triangles.

[0115] FIG. 3 is a diagram of an example viewing vector v that lies within a spherical patch. Illustrated is the spherical patch defined by source viewing vectors v₀, v₁and v₂associated with a particular point of the view given by destination viewing vector v_d. By definition, v_ddefines in the line at sight of the viewer and in general forms three spherical triangles within the larger spherical triangle patch. Areas a₀, a₁and a₂represent the three weights for the sphere maps associated with vertices v₀, v₁and v₂respectively, where α₁, β₁and γ₁are the dihedral angles used to compute a₁.

[0116] Any given view vector, v_d, will in general lie within a spherical patch as illustrated in FIG. 3. Each vertex, v_i, of this spherical triangle is the view vector for one of the source images that have been warped to match v_d. v_dis used to form three interior spherical triangles. The weight for the source image at vertex i is a ratio of the areas of the spherical triangle opposite v_iand the overall spherical triangle. The area of an interior spherical triangle, a_i, on a unit sphere is given by the spherical excess formula (Bronshtein, I. and Semendyayev, K., Handbook of Mathematics, Van Nostrand Reinhold Company, (1985)):

[0117] a_i=α_i+β_i+γ_i−π; iε0,1,2

[0118] The dihedral angles α_i, β_i, and γ_iare defined as:

[0119] α_i=cos⁻¹((v_d{circle around (X)}v_(i−1)ø3)·(v_i+1)ø3{circle around (X)}v_(i−1)ø3))

[0120] β_i=cos⁻¹((v_(i+1)ø3{circle around (X)}v_d)·(v_i+1)ø3{circle around (X)}v_(i−1)ø3))

[0121] γ_i=cos⁻¹((v_(i+1)ø3{circle around (X)}v_d)·(v_i−1)ø3{circle around (X)}v_d))

[0122] Where {circle around (X)} is the normalized cross product and ø is an index wrapping operator, defined as

a Ø b = {\begin{matrix} b - 1 & if a < 0 \\ 0 & if a \geq b \\ a & otherwise \end{matrix}

[0123] 4. Rendering Algorithm

[0124] The present invention leads to a straightforward interactive rendering algorithm. Pseudo-code for the algorithm is given here. It leverages texture mapping graphics hardware in two ways: once to perform the warping and blending between the sample images; and again using the generated sphere map in the final rendering.

[0125] main ( )

[0126] //Set up radiance maps and sphere geometry

[0127] G_v←LoadGeodesicLocations( ); νε{(θ₀, φ₀) . . . (θ_N−1, φ_N−1)}

[0128] Lr,j←LoadSphereMaps (G, L_i, f_r);jε{0 . . . N−1}

[0129] //Viewpoint tracking loop

[0130] loop for each frame

[0131] (x_d, y_d, v_d)←ComputeViewCoordinateSystem( )

[0132] (v₀, v₁, v₂)←FindSubtendedTriangle(G, v_d)

[0133] (a₀, a₁, a₂)←ComputeWeights( (v₀, v₁, v₂), v_d)

[0134] glClearAccum(0, 0, 0, 0)

[0135] //Warp and blending loop

[0136] loop for each of the three vertices, i

[0137] mesh←ComputeMesh(v_i, v_d)

[0138] DrawMesh(mesh, L_r,i)

[0139] glAccum(GL_ACCUM, a_i)

[0140] end vertex loop

[0141] glAccum(GL_RETURN, 1.0/(a0+a1+a2))

[0142] LoadNewSphereMap( )

[0143] RenderObject ( )

[0144] end frame loop

[0145] The interactive rendering program outlined above reads in a set of sphere maps at a prescribed set of geodesic locations along with the associated triangle faces. This decouples the interactive program from any specific choice of sphere map sampling view directions.

[0146] 5. Examples

[0147] The inventors have validated this technique with several examples. FIG. 4 shows an image of an automobile in an environment generated according to one automobile example of the present invention. FIG. 5 shows example images of source radiance environment maps, corresponding destination radiance environment maps after warping and blending with appropriate spherical barycentric weighting, and a final generated radiance environment map according to an example of the present invention. The outer images are source radiance environment maps for a test environment and a mirror BRDF. The next layer of images show each map warped to the new viewpoint with appropriate spherical barycentric weighting. The center image is the final generated radiance environment map. FIG. 6 shows all of the textures that would be used to render a full model of a car, with separate radiance environment maps generated for the paint, chrome trim, and windows.

[0148] FIG. 4 shows an automobile rendered according to the present invention. The car shows an isotropic BRDF modeling automobile paint, computed using techniques similar to that found in Gondek (Gondek, J. S., et al., “Wavelength dependent reflectance functions,” in Proc. SIGGRAPH'94 (Orlando, Fla., Jul. 24-29, 1994), Glassner, A., ed., Computer Graphics Proceedings, Annual Conference Series, ACM SIGGRAPH, ACM Press (July 1994), pp. 213-220). In one example, the inventors modeled the clear polyurethane coat over a paint substrate containing paint particles. An offline ray tracer customized to model the car's paint and polyurethane surfaces was used to directly solve the lighting integral for each point in twelve pre-computed radiance environment maps. Thus the under coat of the paint and the clear coat are both modeled with high fidelity. Each sphere map took several minutes to create.

[0149] Each of the articles and references cited herein are incorporated in their entirety herein by reference.

[0150] 6. Example Implementations

[0151] Routine 100 as described above can be implemented in software, firmware, hardware, or in any combination thereof. Several example environments that can support an implementation of routine 100 are further described with respect to FIGS. 7 to 9. These examples are illustrative and not intended to limit the present invention.

[0152] FIG. 7 is a diagram that shows different layers of an architecture 700 related to graphics processing in which routine 100 can be implemented.

[0153] Architecture 700 includes one or more layers 710-740. As would be apparent to a person skilled in the art given this description, routine 100 can be implemented in any one of layers 710-740, or in any combination of layers 710-740. Layer 710 represents an application program. Layer 720 represents a 3D graphics tool kit. For example, an object-oriented 3D graphics toolkit, such as, OPEN INVENTOR licensed by Silicon Graphics, Inc. See, e.g., Strauss et al., An Object-Oriented 3D Graphics Tool Kit, Computer Graphics, 26(2) July 1992, pp. 341-349 (incorporated in its entirety herein by reference).

[0154] Layer 730 represents a graphics application programming interface (graphics API) and/or other system support (such as, operating system and windowing system). OpenGL® is an example graphics API. See, e.g., Woo et al., OpenGL Programming Guide, Second Ed., version 1.1., Addison-Wesley: MA, 1997 (incorporated in its entirety herein by reference). In one implementation the present invention can be implemented in OpenGL®. OpenGL®, of course, otherwise does not need the present invention to operate.

[0155] Layer 740 represents hardware, including graphics hardware. Graphics hardware 740 can be any graphics pipeline or system including, but not limited to, a computer graphics processor (single chip or multiple chips), high-end to low-end graphics workstations, gaming platforms, systems and consoles, network architectures (e.g., client/server, local, intermediate or wide area networks), and virtual machine (e.g., a Java-created application). For example, a computer graphics system can include, but is not limited to, Indigo², Indy, Onyx, Onyx2, Infinite Reality, Infinite Reality 2, or O₂graphics workstations manufactured by Silicon Graphics, Inc.

[0156] FIG. 8 is a diagram of an example computer graphics system 800 implementing routine 100 in further detail. Computer graphics system 800 includes one or more host processors 810 coupled to memory 815 and graphics subsystem 805. Graphics subsystem 805 includes a geometry engine 820, rasterizer 830, texture mapper 840, an environment mapper 850, and texture memory 855. Graphics subsystem 805 can write to a frame buffer 860. Pixel data in frame buffer 860 is ultimately displayed on a screen display 880.

[0157] Routine 100 can be implemented as a program running on host processor 810 as shown in FIG. 8. Routine 100 then provides control for graphics system 805 as described above with respect to FIG. 1 and further indicated schematically by control lines 812. Apart from the program executing routine 100 and control lines 812, each of the components in system 800 (including graphics subsystem 805, host processor 810, memory 815, frame buffer 860, and display 880) can be a conventional or commercially available off the shelf (COTS) graphics component. Processing functionality for implementing routine 100 can also be transferred all or in part from host processor 810 to a separate logic unit (not shown) or to the graphics subsystem 805 as would be apparent to a person skilled in the art given this description.

[0158] 7. Example GUI Computer System

[0159] FIG. 9 further illustrates an example graphical user-interface (GUI) computer system that can implement routine 100 according to one software embodiment of the present invention. FIG. 9 is a block diagram illustrating an example environment in which the present invention can operate. The environment is a computer system 900 that includes one or more processors, such as processor 904. Computer system 900 can include any type of computer graphics computer, virtual machine, gaming platform, processor (single bus, multiple bus, or bus-less processor(s)), workstation, and network architecture. In one preferred implementation, an OpenGL® machine can be used including, but not limited to, lndigo2™, lndy®, Onyx®, lnfiniteReality®, InfiniteReality2™, or O2® graphics workstations manufactured by Silicon Graphics, Inc.

[0160] The processor 904 is connected to a communications infrastructure 902 (e.g., a bus, cross-bar switch, or network). Various software embodiments are described in terms of this example computer system. This description is illustrative and not intended to limit the present invention. After reading this description, it will be apparent to a person skilled in the relevant art how to implement the invention using other computer systems and/or computer architectures.

[0161] Computer system 900 includes a graphics subsystem 903. Graphics subsystem 903 can be implemented as one or more processor chips. The graphics subsystem 903 can be included as part of processor 904 as shown in FIG. 9 or as a separate graphics engine or processor. Graphics data is output from the graphics subsystem 903 to the bus 902. Display interface 905 forwards graphics data from the bus 902 for display on the display unit 906.

[0162] Computer system 900 also includes a main memory 908, preferably random access memory (RAM), and can also include a secondary memory 910. The secondary memory 910 can include, for example, a hard disk drive 912 and/or a removable storage drive 914, representing a floppy disk drive, a magnetic tape drive, an optical disk drive, etc. The removable storage drive 914 reads from and/or writes to a removable storage unit 918 in a well known manner. Removable storage unit 918 represents a floppy disk, magnetic tape, optical disk, etc., which is read by and written to by removable storage drive 914. As will be appreciated, the removable storage unit 918 includes a computer usable storage medium having stored therein computer software and/or data.

[0163] In alternative embodiments, secondary memory 910 may include other similar means for allowing computer programs or other instructions to be loaded into computer system 900. Such means can include, for example, a removable storage unit 922 and an interface 920. Examples can include a program cartridge and cartridge interface (such as that found in video game devices), a removable memory chip (such as an EEPROM, or PROM) and associated socket, and other removable storage units 922 and interfaces 920 which allow software and data to be transferred from the removable storage unit 922 to computer system 900.

[0164] Computer system 900 can also include a communications interface 924. Communications interface 924 allows software and data to be transferred between computer system 900 and external devices via communications path 926. Examples of communications interface 924 can include a modem, a network interface (such as Ethernet card), a communications port, etc. Software and data transferred via communications interface 924 are in the form of signals which can be electronic, electromagnetic, optical or other signals capable of being received by communications interface 924, via communications path 926. Note that communications interface 924 provides a means by which computer system 900 can interface to a network such as the Internet.

[0165] Graphical user interface module 930 transfers user inputs from peripheral devices 932 to bus 902. These peripheral devices 932 can be a mouse, keyboard, touch screen, microphone, joystick, stylus, light pen, voice recognition unit, or any other type of peripheral unit.

[0166] The present invention is preferably implemented using software running (that is, executing) in an environment similar to that described above with respect to FIG. 9. In this document, the term “computer program product” is used to generally refer to removable storage unit 918 or a hard disk installed in hard disk drive 912. These computer program products are means for providing software to computer system 900.

[0167] Computer programs (also called computer control logic) are stored in main memory and/or secondary memory 910. Computer programs can also be received via communications interface 924. Such computer programs, when executed, enable the computer system 900 to perform the features of the present invention as discussed herein. In particular, the computer programs, when executed, enable the processor 904 to perform the features of the present invention. Accordingly, such computer programs represent controllers of the computer system 900.

[0168] In an embodiment where the invention is implemented using software, the software may be stored in a computer program product and loaded into computer system 900 using removable storage drive 914, hard drive 912, or communications interface 924. Alternatively, the computer program product may be downloaded to computer system 900 over communications path 926. The control logic (software), when executed by the processor 904, causes the processor 904 to perform the functions of the invention as described herein.

[0169] In another embodiment, the invention is implemented primarily in firmware and/or hardware using, for example, hardware components such as application specific integrated circuits (ASICs). Implementation of a hardware state machine so as to perform the functions described herein will be apparent to persons skilled in the relevant art(s).

[0170] Description in these terms is provided for convenience only. It is not intended that the invention be limited to application in this example environment. In fact, after reading the following description, it will become apparent to a person skilled in the relevant art how to implement the invention in alternative environments.

[0171] 8. Conclusion

[0172] Interactive photo-realistic rendering is difficult to achieve because solving the lighting integral is computationally expensive. In one example, the inventors pre-compute this integral into several view-dependent radiance environment maps, each of which would allow realistic rendering of arbitrary geometry from one fixed viewpoint. The inventors further dynamically create new maps for new viewpoints by combining the precomputed maps using an application of IBR techniques in reflection space. The resulting algorithm is readily implementable on most texture mapping capable graphics hardware. The technique allows high rendering quality at interactive rates and from arbitrary viewpoints.

[0173] Traditional IBR could not have achieved these results; it is limited to rendering geometry contained in the source images. Traditional rendering, even using radiance environment maps, could also not have achieved these results; it could not provide the viewpoint independence without a fast way to create new intermediate maps. As recognized by the inventors, by applying IBR to an intermediate image representation in traditional rendering, it becomes possible to produce new algorithms that combine the strengths of both.

[0174] While various embodiments of the present invention have been described above, it should be understood that they have been presented by way of example only, and not limitation. It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined in the appended claims. Thus, the breadth and scope of the present invention should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.

Claims

What is claimed is:

1. A method for reflection space image based rendering of an object at an interactive frame rate, comprising the step of:

loading a sphere geometry for a global set of source viewing vectors; and

loading a global set of source radiance environment maps;

for a frame,

computing a destination viewing vector that points along a viewpoint of a current frame;

determining a number of source viewing vectors in said global set of source viewing vectors which are nearest to the destination viewing vector computed in said computing step:

warping each source radiance environment map in a set of source radiance environment maps associated with the number of source viewing vectors nearest the destination viewing vectors to create a destination radiance environment map associated with a destination viewing vector, each source radiance environment map including values representative of pre-integrated reflected radiance for surface orientations of the object; whereby the object can be rendered with texture environment mapping from the destination radiance environment map without performing integration of a lighting equation; and

blending each warped source radiance environment map at a respective weight, and accumulating the blended warped source radiance environment maps in a buffer to create the destination radiance environment may.

2. The method of claim 1, wherein said determining step comprises determining at least three source viewing vectors in said global set of source viewing vectors which are nearest to the destination viewing vector computed in said computing step.

3. The method of claim 1, further comprising the step of computing a set of weights for the set of source viewing vectors determined in said determining step from said global set of source viewing vectors.

4. The method of claim 3, further comprising the step of normalizing the set of weights computed in said computing step.

5. The method of claim 3, further comprising the step of normalizing the destination radiance environment image based on a normalization of the set of weights computed in said computing step.

6. The method of claim 5, further comprising the steps of:

loading the normalized destination environment map image into a texture memory; and

rendering the object, wherein said rendering step includes environment mapping the normalized destination environment map onto the object.

7. The method of claim 1, wherein said warping step further comprises the step of:

generating meshes with warped sets of texture coordinates, each mesh having a respective set of warped texture coordinates corresponding to a respective warped source radiance environment map.

8. A system for reflection space image based rendering of an object at an interactive frame rate, comprising:

means for loading a sphere geometry for a global set of source viewing vectors;

means for loading a global set of source radiance environment maps;

for a frame,

means for computing a destination viewing vector that points along a viewpoint of a current frame;

means for determining a set of source viewing vectors, the set of viewing vectors representing different viewpoints associated with a corresponding set of source radiance environment maps and determining a number of source viewing vectors which are nearest the destination viewing vector; and

means for warping each source radiance environment map in the set of source radiance environment maps associated with the number of source viewing vectors nearest the destination viewing vector to create a destination radiance environment map associated with a destination viewing vector, each source radiance environment map including values representative of pre-integrated reflected radiance for surface orientations of the object; whereby the object can be rendered with texture environment mapping from the destination radiance environment map and without performing integration of a lighting equation; and

mean for blending each warned source radiance environment map at a respective weigh,t and accumulating the blended warped source radiance environment maps in a buffer to create the destination radiance environment map.

9. A computer program product comprising a computer useable medium having computer program logic recorded thereon for enabling a processor in a computer system to perform reflection space image based rendering of an object at an interactive frame rate, said computer program logic comprising:

means for loading a sphere geometry for a global set of source viewing vectors;

means for loading a global set of source radiance environment maps;

for a frame,

means for computing a destination viewing vector that points along a viewpoint of a current frame;

means enabling the processor to determine a set of source viewing vectors, the set of viewing vectors representing different viewpoints associated with a corresponding set of source radiance environment maps and to determine a number of source viewing vectors which are nearest the destination viewing vector;

means enabling the processor to warp each source radiance environment map in the set of source radiance environment maps associated with the set of source viewing vectors to create a destination radiance environment map associated with a destination viewing vector, each source radiance environment map including values representative of pre-integrated reflected radiance for surface orientations of the object; whereby the object can be rendered with texture environment mapping from the destination radiance environment map and without performing integration of a lighting equation; and

means for blending each warped source radiance environment map at a respective weight, and accumulating the blended warped source radiance environment maps in a buffer to create the destination radiance environment map.

10. A method for rendering an object having lighting reflections, comprising the steps of:

determining a set of source radiance environment maps representing different viewpoints of the object, each source radiance environment map including information indicative of a light reflective surface property of the object;

determining a destination viewing vector for rendering the object from a destination viewpoint;

determining a set of source viewing vectors comprising:

determining a global set of source viewing vectors; and

determining a subset of the global set of source viewing vectors which are nearest to the destination viewing vector, wherein the set of source viewing vectors are based on the determined destination viewing vector and the set of source viewing vectors associated with the set of source radiance environment maps;

warping each source radiance environment map of the set of source radiance environment maps based on the determined set of source viewing vectors to create a destination radiance environment map of the object, the destination radiance environment map associated with viewing the object from the destination viewpoint, the warping performed without integration of a lighting equation; and

11. The method as recited in claim 10, wherein the step of warping comprises mirror reflection warping each source radiance environment map.

12. The method as recited in claim 10, wherein the step of warping comprises surface normal reflection warping each source radiance environment map.

13. The method as recited in claim 10, further comprising the step of:

providing the destination radiance environment map to a graphic hardware system to render the object with texture environment mapping of the destination radiance environment map.

14. The method as recited in claim 10, further comprising the step of:

rendering the object with texture environment mapping of the destination radiance environment map.

15. The method as recited in claim 10, further comprising the step of:

pre-integrating a bidirectional reflection distribution function with a lighting environment to create the set of source radiance environment maps.

16. The method as recited in claim 10, further comprising the step of loading a sphere geometry for the set of source viewing vectors.

17. The method as recited in claim 10, wherein the subset of the global set of source viewing vectors comprises at least three source viewing vectors which are nearest to the destination viewing vector.

18. The method of claim 10, wherein the warping step further comprises the step of:

generating meshes with warped sets of texture coordinates, each mesh having a respective set of warped texture coordinates corresponding to a respective warped source radiance environment map.

19. A computer-readable medium having computer-readable instructions stored thereon for rendering an object having lighting reflections, the computer-readable instructions when stored on a processor causing the processor to perform the steps of:

determining a destination viewing vector for rendering the object from a destination viewpoint;

determining a set of source viewing vectors comprising:

determining a global set of source viewing vectors; and

blending each warped source radiance environment map at a respective weight and accumulating the blended warned source radiance environment maps in a buffer to create the destination radiance environment map.

20. The computer-readable medium as recited in claim 19, wherein the step of warping comprises mirror reflection warping each source radiance environment map.

21. The computer-readable medium as recited in claim 19, wherein the step of warping comprises surface normal reflection warping each source radiance environment map.

22. The computer-readable medium as recited in claim 19, further causing the processor to perform the step of:

providing the destination radiance environment map to a graphic hardware system to render the object with texture environment mapping of the destination radiance environment map.

23. The computer-readable medium as recited in claim 19, further causing the processor to perform the step of:

rendering the object with texture environment mapping of the destination radiance environment map.

24. The computer-readable medium as recited in claim 19, further causing the processor to perform the step of:

pre-integrating a bidirectional reflection distribution function with a lighting environment to create the set of source radiance environment maps.

25. A system for rendering an object having lighting reflections, comprising:

a processor that determines a set of source radiance environment maps representing different viewpoints of the object, each source radiance environment map including information indicative of a light reflective surface property of the object, determines a destination viewing vector for rendering the object from a destination viewpoint,

determines a global set of source viewing vectors;

determines a subset of the global set of source viewing vectors which are nearest to the destination viewing vector, wherein the set of source viewing vectors are based on the determined destination viewing vector, the set of source viewing vectors associated with the set of source radiance environment maps, and warps each source radiance environment map of the set of source radiance environment maps based on the determined set of source viewing vectors to create a destination radiance environment map of the object, the destination radiance environment map associated with viewing the object from the destination viewpoint, the warping performed without integration of a lighting equation; and

blends each warned source radiance environment map at a respective weight and accumulates the blended warped source radiance environment maps in a buffer to create the destination radiance environment map.

26. The system as recited in claim 25, wherein the processor mirror reflection warps each source radiance environment map.

27. The system as recited in claim 25, wherein the processor surface normal reflection warps each source radiance environment map.

28. The system as recited in claim 25, wherein the processor further provides the destination radiance environment map to a graphic hardware system to render the object with texture environment mapping of the destination radiance environment map.

29. The system as recited in claim 25, wherein the processor further renders the object with texture environment mapping of the destination radiance environment map.

30. The system as recited in claim 25, wherein the processor further pre-integrates a bidirectional reflection distribution function with a lighting environment to create the set of source radiance environment maps.

31. The system as recited in claim 25, further comprising a graphic hardware system that receives the destination radiance environment map created by the processor and renders the object with texture environment mapping of the destination radiance environment map.

32. The system as recited in claim 31, wherein the graphic hardware system comprises:

a geometry engine that receives the destination radiance environment map and performs geometric computations on the destination radiance environment map;

a rasterizer that receives the computed destination radiance environment map and rasterizes the computed destination radiance environment map;

a texture mapper; and

an environment mapper, the texture mapper and environment mapper mapping texture and environment to the rasterized destination radiance environment map for rendering the object.

Patent Citations (20)

Patent	Date	Inventor	Cited By
US5175808(A)	1992-12-01	Sayre	Examiner
US5204944(A)	1993-04-01	Wolberg et al.	Examiner
US5396583(A)	1995-03-01	Chen et al.	Examiner
US5704024(A)	1997-12-01	Voorhies et al.	Examiner
US5710876(A)	1998-01-01	Peercy et al.	Examiner
US5748192(A)	1998-05-01	Lindholm	Examiner
US5748194(A)	1998-05-01	Chen	Examiner
US5923334(A)	1999-07-01	Luken	Examiner
US5949430(A)	1999-09-01	Robertson et al.	Examiner
US5963213(A)	1999-10-01	Guedalia et al.	Examiner
US5995110(A)	1999-11-01	Litwinowicz	Examiner
US6011558(A)	2000-01-01	Hsieh et al.	Examiner
US6044181(A)	2000-03-01	Szeliski et al.	Examiner
US6097394(A)	2000-08-01	Levoy et al.	Examiner
US6097854(A)	2000-08-01	Szeliski et al.	Examiner
US6215496(B1)	2001-04-01	Szeliski et al.	Examiner
US6236405(B1)	2001-05-01	Schilling et al.	Examiner
US6268846(B1)	2001-07-01	Georgiev	Examiner
US6271847(B1)	2001-08-01	Shum et al.	Examiner
US6384824(B1)	2002-05-01	Morgan et al.	Examiner

Non-Patent Literature (21)

Bastos, Rui et al., “Increased Photorealism for Interactive Architectural Walkthroughts,” computer graphics proceedings, annual conference series, Apr. 1999, pp. 183-190.Applicant
Blinn, James F. and Newell, Martin E., “Texture and Reflection in Computer Generated Images,” Communications of the ACM,vol. 19, No. 10, Oct. 1976, pp. 542-547.Applicant
Cabral, Brian et al., “Bidirectional Reflection Functions from Surface Bump Maps,” Computer Graphics, vol. 21, No. 4, Jul. 1987, pp. 273-281.Applicant
Cook, Robert L. et al., “The Reyes Image Rendering Architecture,” Computer Graphics, vol. 21, No. 4, Jul. 1987, pp. 95-102.Applicant
Debevec, Paul E. and Malik, Jitendra, “Recovering High Dynamic Range Radiance Maps from Photographs,” Computer Graphics Proceedigns. Annual Conference Series, 1997, pp. 369-378.Applicant
Debevec, Paul, “Rendering Synthetic Objects into Real Scenes: Bridging Traditional and Image-based Graphics with Global Illumination and High Dynamic Range Photography,” Computer Graphics Proceedings, Annual Conference Series, 1998, pp. 189-198.Applicant
Gershbein, Reid et al., “Textures and Radiosity: Controlling Emission and Reflection with Texture Maps,” Computer Graphics Proceedings, Annual Conference Series, 1994, pp. 51-58.Applicant
Gondek, Jay S. et al., “Wavelength Dependent Relfectance Functions,” Computer Graphics Proceedings, Annual Conference Series, 1994, pp. 213-220.Applicant
Gortler, Steven J. et al., “The Lumigraph,” Computer Graphics Proceedings. Annual Conference Series, 1996, pp. 43-54.Applicant
He, Xiao D. et al., “A Comprehensive Physical Model for Light Reflection,” Computer Graphics, vol. 25, No. 4, Jul. 1991, pp. 175-186.Applicant
Immel, David S. et al., “A Radiosity Method For Non-Diffuse Environments,” Computer Graphics, vol. 20, No. 4, Aug. 1986, pp. 133-142.Applicant
Jensen, Henrik Wann and Christensen, Per H., “Efficient Simulation of Light Tranpsort in Scenes with Participating Media using Photon Maps,” Computer Graphics Proceedings, Annual Conference Series, 1998, pp. 311-320.Applicant
Kajiya, James T., “The Rendering Equation,” ACM SIGGRAPH'86, vol. 20, No. 4, 1986, pp. 143-150.Applicant
Levoy, Marc and Hanrahan, Part, “Light Field Rendering,” Computer Graphics Proceedings, Annual Conference Series, 1996, pp. 31-42.Applicant
McMillan, Leonard and Bishop, Gary, “Plenoptic Modeling: An Image-Based Rendering System,” Computer Graphics Proceedings, Annual Conference Series, 1995, pp. 39-46.Applicant
Miller, Gene S. and Hoffman, C. Robert, “Illumination and Reflection Maps: Simulated Objects in Simulated and Real Environments,” Course Notes for Advances Computer Graphics Animation SIGGRAPH 84, Jul. 23, 1984, 13 pages.Applicant
Poulin, Pierre and Fournier, Alain, “A Model for Anisotropic Reflection,” Computer Graphics, vol. 24, No. 4, Aug. 1990, pp. 273-282.Applicant
Seitz, Steven M. and Dyer, Chalres R., “View Morphing,” Computer Graphics Proceedings, Annual Conference Series, 1996, pp. 21-30.Applicant
Veach, Eric and Guibas, Leonidas J., “Metropolis Light Transport,” Computer Graphics Proceedings, Annual Conference Series, 1997, pp. 65-76.Applicant
Voorhies, Douglas and Foran, Jim, “Reflection Vector Shading Hardware,” Computer Graphics Proceedings, Annual Conference Series, 1994, pp. 163-166.Applicant
Ward, Gregory J., “The Radiance Lighting Simulation and Rendering System,” Computer Graphics Proceedings, Annual Conference Series, 1994, pp. 459-472.Applicant

Abstract