Synthesizing an image from a virtual perspective using pixels from a physical imager array weighted based on depth error sensitivity

[0001] This application claims the benefit of U.S. Provisional Application No. 62/692,619 filed on Jun. 29, 2018, which is incorporated by reference herein.

[0002] This application relates generally to image synthesis, and more specifically to a pixel-weighting technique for synthesizing an image corresponding to a virtual perspective.

[0003] High-performance display technology and graphics hardware enable immersive three-dimensional environments to be presented in real-time with high levels of detail. Head mounted displays (HMDs) have critical performance requirements of low latency and high frame rate to maintain immersion and prevent user discomfort resulting from discrepancies between user motion and corresponding changes of the visual field. Currently these immersive environments are primarily limited to the context of video games and simulations where the environment is rendered from a game engine with assets and textures created by artists during development. However, these environments fall short of photorealistic appearance and this virtual world paradigm does not allow for mediated interactions with the real world around the user. In applications where users interact with their physical environment, streaming of video data (often from a single imager) is used. However, the perspective and motion of the user is tied directly to that of the physical imager. Furthermore, merely overlaying information on the video stream lacks the immersion and engagement provided by a synthesized viewpoint that accurately recreates the real world, while seamlessly integrating additional information sources.

[0004] Image based or light field rendering represents a variety of techniques for synthesizing novel perspective views from sets of overlapping images of a scene. These sets of images represent a measurement of the local light field, which is described as a vector function representing the amount of light traveling in every direction through every point in the measured space. An ideal light field imager would contain pixels that record the amount of light incident on that point from all directions. Several techniques exist to densely sample the light field by capturing images of a scene from many, similar viewpoints. These conventional techniques include use of specialized imaging hardware utilizing arrays of micro or macro lenses in front of a single image sensor, a single imager undergoing smooth motion, or a large array of regularly spaced imagers. The dense sampling of the light field contains continuous, highly correlated information that is leveraged for processing. In the densely sampled case, a virtual view can be directly generated from pixel data with explicit ray correspondences between the virtual perspective and physical imagers. However, the required large volume of data poses a challenge for real-time capture, transmission, and rendering. Sparsely sampling the light field with a distributed array of imagers provides an alternative approach with data rates that are tractable by commercial hardware for real-time processing. The primary challenge under sparse capture becomes interpolation of the limited, discrete camera views. Simple schemes, such as closest ray selection, result in significant artifacts that degrade the fidelity of the rendered scene.

[0012] The figures and the following description relate to preferred embodiments by way of illustration only. It should be noted that from the following discussion, alternative embodiments of the structures and methods disclosed herein will be readily recognized as viable alternatives that may be employed without departing from the principles of what is claimed.

[0013] Reference will now be made in detail to several embodiments, examples of which are illustrated in the accompanying figures. It is noted that wherever practicable similar or like reference numbers may be used in the figures and may indicate similar or like functionality. The figures depict embodiments of the disclosed system (or method) for purposes of illustration only. One skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles described herein.

[0014] A method is described for image interpolation based on pixel weighting that results in high fidelity rendered output and is suitable for real-time performance on consumer hardware. Various aspects of the described embodiments provide systems and methods for assigning weights to physical imager pixels from sparse imager views for interpolation into an arbitrary virtual perspective. One aspect identifies a series of virtual pixel locations that correspond with each physical imager pixel in the presence of geometry measurement error. For example, in a particular embodiment, a method obtains a plurality of images of a scene from respective physical images and determines an initial predicted surface geometry of the scene. Based on the predicted surface geometry, a plurality of candidate pixels from different physical images are determined that are predicted to correspond to a same world point in the scene as a virtual pixel to be computed. For each of the plurality of candidate pixels, respective sensitivities to an error in the initial predicted surface geometry are detected. Pixel weights are determined for each of the plurality of candidate pixels based on the respective sensitivities. A value for the virtual pixel may be determined based on a weighted combination of the plurality of candidate pixels using the pixel weights.

[0015] Another aspect of the described embodiments includes a function between the measurement error and virtual pixel disparity that is used to determine the respective sensitivities. An estimate of the measurement error may be found using the measurement of the corresponding world point. The world point location may be refined by maximizing the correlation between corresponding imager pixel values. A sensitivity derivative is calculated for the described function for the determined error value and corresponding world point based on the refined measurement. The sensitivity derivative may be used for calculation of an imager pixel weight using a weighting function as described above.

[0016] More specifically, in one embodiment, a virtual pixel may be computed based on the pose of a virtual viewpoint, and the scene geometry. For a given virtual pixel of the virtual viewpoint, the corresponding observed world point is determined given a camera model of the virtual perspective by defining where a ray from the center of projection through the pixel intersects scene geometry. For each physical imager, a ray is defined between its center of projection and the determined world point. This ray identifies the physical imager pixel to use for interpolation. A pixel weighting technique assigns weights that depend on uncertainty in the projected virtual pixel location due to error in the measurement of the world point location. Given a model of error in the world point location, the location of the surface is shifted by some amount along the surface normal at the point. New world points from the viewpoint of each imager are found by back-projecting along the imager rays to the intersection with the new error surface. New virtual rays are then defined between these error surface points and the virtual perspective center of projection. These virtual rays define new virtual pixel locations and the difference between these and the original virtual pixel are taken. The sensitivity of the function with respect to distance along the projecting virtual ray that relates virtual pixel disparity to geometry measurement error is evaluated at the refined depth estimate. These sensitivity values are used to weight the contribution of the corresponding imager pixels such that lower sensitivity results in higher weighting. Finally, the weighted sum of imager pixel values results in the value of the corresponding virtual pixel. This depth-aware and surface-aware interpolation method results in a synthetic view that has high accuracy and visual fidelity to the captured scene. Furthermore, the described method is capable of real-time rendering, at high frame rates (e.g., over 90 frames per second) on commercial hardware, with sparsely sampled light field data.

[0017] In an embodiment, the image synthesis technique can be applied in a mediated-reality system for surgical application. In this application, the mediated-reality system tracks the surgeon's head position and generates real-time images of the surgical site from a virtual camera perspective corresponding to the surgeon's head position to mimic the natural viewpoint of the surgeon. The mediated-reality system may overlay metadata or pre-operative images together with real-time images captured by the mediated-reality system to enhance the surgeon's ability to perform a procedure with high efficiency and precision.

[0018] FIG. 1 illustrates an example system 100 for synthesizing an image corresponding to a virtual perspective. The system 100 comprises an imager array 110, an image processing device 120, a display device 130, and an input controller 140. The imager array 110 comprises a plurality of imagers 115 (e.g., cameras) that each capture images of a scene 150. The imagers 115 may be a physically arranged such that their physical locations and orientations relative to each other is fixed and known or may be automatically derived. For example, the imagers 115 may be structurally secured by a mounting structure that mounts the imagers 115 at predefined fixed locations and orientations relative to the scene 150. The imagers 115 may be positioned such that neighboring imagers 115 in the imager array 110 share overlapping views of the scene 150. The imagers 115 in the imager array 110 may furthermore be synchronized to capture images substantially simultaneously (e.g., within a threshold temporal error). The imager array 110 may also optionally include a projector 124 that projects a structured light pattern onto the scene 150. In an embodiment, the imagers 115 comprise light-field cameras that capture light field information of the scene 150, which may optionally be aided by the structured light pattern. Here, the imagers 115 capture both the intensity of light and the directions of light rays reflected from the scene 150. The light field images captured by the imagers 115 may therefore encode depth information representing a surface geometry of the scene 150.

[0019] The imagers 115 can comprise CCD or CMOS image sensors and associated optics. Optics may include a variety of configurations including lensed or bare individual image sensors in combination with larger macro lenses, micro-lens arrays, prisms, and negative lenses. In the case where multiple views of the scene 150 are incident on an individual image sensor, these sub-images may be treated as independent imagers 115 in the algorithms described below.

[0020] The image processing device 120 receives images captured by the imagers 115 of the imager array 110 and processes the images to synthesize an output image corresponding to a virtual imager perspective. Here, the output image corresponds to an approximation of an image of the scene 150 that would be captured by an imager 115 placed at an arbitrary position and orientation corresponding to the virtual imager perspective. The image processing device 120 outputs the output image corresponding to the virtual imager perspective to the display device 130 where it is displayed. The location and orientation associated with the virtual imager perspective may be controlled by an input controller 140. Furthermore, the image processing device 120 may synthesize output images corresponding to a virtual imager perspective in substantially real-time as the input controller 140 updates the location and orientation associated with the virtual imager perspective. For example, the image processing device 120 may generate output images at least as fast as a frame rate of the imager array 110 to generate an output video comprising a sequence of synthesized output images that track the virtual imager perspective in substantially real-time in response to the input controller 140.

[0021] As part of the image synthesis process, the image processing device 120 may detect the structured light projected onto the scene 150 by the projector 118 to estimate depth information of the scene and generate a depth map representing an estimated surface geometry of the scene 150. Alternatively, the projector 118 may be omitted and the image processing device 110 may derive the depth information solely from the images captured by the imagers 115 using techniques such as light field correspondence or stereo block matching. In other alternative embodiments, dedicated depth detection hardware may be utilized such as a depth camera or LiDAR detector to estimate the surface geometry of the scene 150.

[0022] In an embodiment, the image processing device 120 may perform a calibration process to detect the positions and orientation of each of the imagers 115 in three-dimensional space with respect to a shared origin and/or the amount of overlap in their respective fields of view. In one embodiment, the image processing device 120 may perform calibration by processing captured images from each of the imagers 115 of a fiducial marker placed in the scene 150 and performing an optimization over the imager parameters and distortion coefficients to minimize reprojection error for key points.

[0023] In an embodiment, the image processing device 120 calculates a depth frame for each imager 115 based on the scene geometry and imager pose information. For any given imager pixel, a ray is defined between this pixel and the imager center of projection. The value of the depth frame at a pixel location corresponds with the closest point at which this ray intersects the scene geometry, defining the distance between the imager center of projection and this world point. In the pinhole model, the color and lighting of this point corresponds directly to the pixel value in the image. Therefore, each pixel contains associated color and depth information.

[0024] In an embodiment, the image processing device 120 may furthermore processing the captured images to perform object tracking of an object within the vicinity of the scene 150. Object tracking may be performed using image processing techniques or may utilize signals from dedicated tracking hardware that may be incorporated into the imager array 110 and/or the object being tracked. In a surgical application, for example, a tracked object may comprise a surgical instrument or a hand or arm of a physician or assistant. In an embodiment, the image processing device 120 may recognize the tracked object as being separate from the surgical site of the scene 150 and may apply a visual effect to distinguish the tracked object such as, for example, highlighting the object, labeling the object, or applying a transparency to the object.

[0025] The image processing device 120 may comprise a processor and a non-transitory computer-readable storage medium that stores instructions that when executed by the processor, carry out the functions attributed to the image processing device 120 as described herein.

[0026] The display device 130 may comprise, for example, a head-mounted display device or other display device for displaying the synthesized output images received from the image processing device 120. In an embodiment, the input controller 140 and the display device 130 are integrated into a head-mounted display device and the input controller 140 comprises a motion sensor that detects position and orientation of the head-mounted display device. The virtual imager perspective can then be derived to correspond to the position and orientation of the head-mounted display device such that the virtual perspective corresponds to a perspective that would be seen by a viewer wearing the head-mounted display device. Thus, in this embodiment, the head-mounted display device can provide a real-time rendering of the scene 150 as it would be seen by an observer without the head-mounted display. Alternatively, the input controller 140 may comprise a user-controlled control device (e.g., a mouse, pointing device, handheld controller, gesture recognition controller, etc.) that enables a viewer to manually control the virtual imager perspective displayed by the display device 130.

[0027] FIGS. 2-4 are diagrams illustrating techniques applied by the image processor device 120 to generate an output image corresponding to a virtual perspective representing a view of a virtual imager of the scene 150 from an arbitrary pose. FIG. 2 shows a typical relation between a subset of imagers 115 (e.g., imagers C₁ and C₂) and a desired virtual perspective (C_v). For simplicity in this example, two imagers 115 are shown, but the number is not limited to two, and in practice many more are used. For a given virtual pixel (p_v) of the synthesized output image (where p_v refers to a location (e.g., x-y location) of the pixel within the image), a corresponding world point (w) is calculated using the virtual imager pose and the scene geometry. The world point w represents a point in the scene 150 corresponding to the virtual pixel p_v based on the predicted virtual imager pose and the predicted scene geometry. A ray (r_v) is defined from w to the virtual perspective (C_v) and rays (r₁ and r₂) are defined from the same world point w to the imagers C₁ and C₂. These rays r₁, r₂ identify corresponding imager pixels p₁ and p₂, which are candidate pixels whose values x_p1, x_p2 may be interpolated to calculate the value x_pv of p_v. For example, the value of p_v may be calculated as a weighted combination of the values x_p1, x_p2 of p₁ and p₂. The pixel weights may be calculated as described in further detail below.

[0028] In an embodiment, the pixel weights may be computed based on characteristics of the local region around the world point w. The goal is to determine how errors in the prediction of the scene geometry affect the actual world point w seen by an imager 115. In FIG. 2, two levels of measurement error of the scene depth around the world point w are shown (δ₀, δ₁), but any number of levels may be considered bounded by the depth of the scene 150 or desired computational complexity. In an embodiment, a ratio of the spatial resolution of the geometry capture to that of the imagers 115 may be used to determine the number of error levels and the step size is set to the effective spatial resolution of the imagers 115. For each point in the local region of w a normal vector is calculated relative to the predicted scene geometry. This may be done by fitting a local plane to each point. Alternatively, globally consistent normal values can be determined by propagation as part of scene geometry acquisition. Another embodiment explicitly represents the scene geometry as a function, where the normal at any point is given by the gradient. In an embodiment, dot product between the point normal and the camera ray is checked and the point is considered in the pixel weighting calculation only when the dot product is above a certain threshold to avoid contributions from points that correspond to edges or back faces. The points in the local region around w are then translated by the step size along their normal vector to form respective candidate error surfaces 202 (e.g., error surfaces E0 and E1), which may be fit with a Nth order polynomial, where the order is bounded by the desired accuracy.

[0029] Given an error surface E and a camera ray r for a given imager 115, the candidate points the imager 115 would see given this error level are found as the intersection of the ray r and the fit error surface E. For example, for camera r₁ for imager C₁ the error point for error surface E0 is w₁^δ0 and the error point for error surface E1 is w₁^δ1. New error rays between the virtual perspective C_v and these points are r₁^δ0 and r₁^δ1 respectively for error surface E0 and E1. These error rays r₁^δ0 and r₁^δ1 intersect virtual pixel locations p_v^δ0 and p_v^δ1. Thus, the surface estimation error results in virtual pixels p_v that correspond to different pixels p of the actual imagers 115 than otherwise expected for an accurate surface estimation. The difference between each of these virtual pixel locations p_v^δ0 and p_v^δ1 due to the error surface and the original virtual pixel p_v are calculated as follows:
e₀=p_v1^δ0−p_v
e₁=p_v1^δ1−p_v
where e_i is a distance error in the virtual pixel position corresponding to a physical imager pixel resulting from a particular depth estimation error. The relation between virtual pixel error e and level of depth measurement error δ (e.g., [(δ₀, e₀), e₁)] is fit with a Nth order polynomial function such that e=ƒ (δ). Thus, the virtual pixel error function ƒ maps a particular depth measurement error to the resulting virtual pixel error. The sensitivity of the virtual pixel error function with respect to error is found by evaluating the derivative. The pixel weight is a function of this sensitivity value. In one embodiment, the sensitivity derivative is evaluated at an error level that maximizes relevant pixel correlation, δ_correlation:

[0030] $s=\frac{\mathrm{df}}{d{\delta }_{{\delta }_{\mathrm{correlation}}}}$
where s is the sensitivity. A process for determining δ_{δcorrelation} is described in further detail below with respect to FIG. 4.

[0031] The imager pixel weight is calculated as a piece-wise function of the sensitivity s. For example, in an embodiment, the imager pixel weight is computed as follows:

[0032] $c=\{\begin{array}{cc}g\left(s\right),& s<\mathrm{uncertainty}\mathrm{threshold}\\ k,& s>\mathrm{uncertainty}\mathrm{threshold}\\ 0,& \mathrm{obstructed}\mathrm{or}\mathrm{occluded}\end{array}$
where the uncertainty threshold may be a predefined value and where k is a predefined constant. In an embodiment, the weights c associated with each imager 115 are normalized (e.g., by applying a normalizing factor) such that all the weights c add up to 1 for a given p_v. In one embodiment, g(s) is chosen to be a radial basis function such that the weighted sum is dominated by the imager pixel with lowest sensitivity. The constant region k above the uncertainty threshold ensures that even in cases where the virtual view is at a large angular displacement from any physical imagers, a view is still rendered. In an embodiment, these high uncertainty regions are presented on the display device 130 with lower brightness or other visual demarcation to provide a visual cue to the high uncertainty. As discussed above, imagers 115 that are obstructed or occluded from the scene point do not contribute to the final pixel value x_pv and thus receive a zero weight. In the provided example of FIG. 2 with two imagers C₁, C₂, the pixel value x_pv virtual pixel p_v corresponding to a world point w may be computed as:
x_pv=n(c₁d₁x_p1+c₂d₂px_p2)
where x_pv is the virtual pixel value, n is the normalizing factor, c_n is the nth imager pixel weight, d_n is the nth imager pixel fade, and x_pn is the imager pixel value. The pixel fade d_n is calculated as:
d=ƒ(pixel_x,pixel_y,image_width,image_height)
where pixel_x represents the x-coordinate of the pixel p, pixel_y represents the y-coordinate of the pixel p, image_width represents an overall width of the imager image, and image_height represents the overall height of an imager image. Thus, d is smaller the closer the pixel is to the edge of the image. This fade term reduces discrete boundaries from appearing in the interpolated view due to variation in the imager properties such as white balance and vignetting.

[0033] A result of the above described technique is that the pixel weights are weighted based on the sensitivity to depth estimation error of a particular virtual pixel's correspondence to an imager pixel. In FIG. 2, for example, even though r₁ has a smaller angular displacement from r_v than r₂, p₁ has a lower weight than p₂ because it has a higher error sensitivity due to the relation between the surface geometry, imager locations, and virtual perspective.

[0034] FIG. 3 illustrates examples of scenarios in which the weights of the imager pixels p₁ and p₂ may be set to zero, making no contribution to the pixel value x_pv of pixel p₁. In FIG. 3, the view of the world point w by imager C₁ is obstructed by the scene geometry. Furthermore, the view of the world point w by imager C₂ is occluded by an object 302 from outside the scene 150. For example, the object 302 may correspond to a surgical tool positioned between the imager array 110 and a surgical site corresponding to the scene 150. In one embodiment, the image processing device 120 detects obstruction by comparing the z coordinate of the world point w in the imager's perspective based on the scene geometry to the value of the depth frame for the expected corresponding pixel p₁ captured by the imager C₁. If a mismatch is detected, the image processing device 120 may detect that world point w is occluded from the view of the imager C₁ and set a weight associated with the contribution of the pixel value x_p1 pixel p₁ to the pixel value pixel value x_pv of virtual pixel p_v to zero.

[0035] In another embodiment the image processing device 120 may determine a radiometric correspondence between imager pixels that are expected, based on the scene geometry and imager poses, to share a common world point w, such as imager pixels p₁ and p₃. For example, one or more outlier pixels may be detected that has a below threshold correspondence in its pixel value with a majority of other pixels mapping to the same world point w. The one or more outlier pixels may be detected as corresponding as representing an occluding object between the imager and the world point w.

[0036] In other embodiments that utilize separate depth capture hardware, hand tracking, or tool tracking, occlusion may be detected by checking if the imager rays (e.g., r₂ or r₃) intersect a position of a tracked asset or a portion of the depth map that is discontinuous from the majority of the scene 150.

[0037] FIG. 4 illustrates an example of a technique for determining the depth estimation error level δ_{δcorrelation}, that maximizes pixel correspondence as used in the pixel weighting calculation described above. Here, a series of world points are identified at the intersection of each surface and r_v, shown in FIG. 4 as w₀, w₁, w₂. Rays between those points and the imagers identify a series of pixels, one per error level. For imager C₁ the rays r₁^w0, r₁^w1, r₁^w2 correspond to pixels p₁^w0, p₁^w1, p₁^w2 respectively. δ_correlation is the surface that results in the maximum imager pixel correlation, such as between the pixel values of p₁^w0 and p₂^w0. The level that maximizes pixel correlation is a refined estimate of the true surface position. In one embodiment, the effective spatial resolution of the imagers is higher than that of the depth capture modality. In that case, the refined depth value can update the estimated value as well as upsample the resolution of the scene geometry.

[0038] FIG. 5 is a flowchart illustrating an example embodiment of a process performed by the image processing device 120 to generate a synthesized image corresponding to a virtual perspective. The image processing device 120 obtains 502 image from each of the imagers 115. The image processing device 120 furthermore obtains 504 the scene geometry representing depth estimates of the scene. The image processing device 120 may furthermore obtain other information such as the poses of each of the physical imagers 115 (e.g., positions and orientations) in the image array 110. In an embodiment, the image processing device 120 is calibrated such that these quantities are obtained in a unified coordinate system. The image processing device 120 then identifies a plurality of candidate pixels that may contribute to the virtual pixel value. For example, a world point may be identified for a virtual pixel to be computed (based on the virtual perspective and the scene geometry). The candidate pixels may be identified based on the poses of the physical imagers 115 and the scene geometry as pixels corresponding to the same world point. In an embodiment, rays from the world point to each of the physical imagers 115 may be identified to determine one candidate pixel per physical imager 115. Sensitivities of each of the candidate pixels to errors in the estimated scene geometry are determined 510. Here, a low sensitivity generally indicates that an error in the depth estimate around the world point will have a relatively small effect on the virtual pixel location corresponding to the world point represented by the candidate pixel. A high sensitivity generally indicates that an error in the depth estimate around the world point will have a significant effect on the virtual pixel location corresponding to the world point represented by the candidate pixel.

[0039] Pixel weights are then determined 510 for each candidate pixel based on their respective sensitivities. In an embodiment, the weights are calculated using a piece-wise continuous function. For example, if the sensitivity is below a given uncertainty threshold, the weight is calculated from a radial basis function, whereas if it is below the threshold the weight is a small constant value. A normalization factor is then calculated such that the weights sum to one.

[0040] In an embodiment, determining 510 the pixel weights may further include identifying candidate pixels that are obstructed or occluded from view of the world point. Candidate pixels that are obstructed or occluded are assigned a weight of zero such that they do not contribute to the virtual pixel value.

[0041] The value of the virtual pixel is then determined 512 using the pixel weights. For example, the virtual pixel value may be calculated using the physical imager pixel values, weights, fades, and the normalization factor.

[0042] The process of FIG. 5 generally iterates over every virtual pixel to be rendered, thereby determining a pixel value at each iteration until pixel values for the entire output image are determined.

[0043] FIG. 6 illustrates an embodiment of a process for detecting 508 the sensitivities of each candidate pixel. A set of rays are generated 602 for a series of error surfaces corresponding to the virtual perspective and the physical imagers 115. For example, error surfaces may first be determined at different fixed distances from the estimated surface geometry. The intersection points between these error surfaces and the imager rays are determined. New virtual rays are defined between these intersection points and the virtual perspective and new imager rays are defined between the intersection points and the imagers 115. In an embodiment, surface normals for a region of relevant world points about the original world point are determined from the scene geometry and used to calculate the multi-dimensional error surfaces. This calculation can be performed using local information or in a globally consistent manner using information from the entire scene.

[0044] The surface geometry estimate is then refined 604 based on the rays for the error surfaces. For example, the intersection points between the error surfaces and the virtual ray identify a series of points corresponding to a series of physical imager pixels. For each intersection point, the correspondence between relevant physical imager pixels is computed. The intersection point that maximizes correspondence between imager pixels is a refined estimate of the scene geometry at the world point. The refined depth value may be used to refine and upsample the scene geometry used in the pixel weighting process of FIG. 5.

[0045] Differences are then computed 606 between the original virtual pixel and error virtual pixels pointed to by rays associated with the error surfaces. For example, intersection points between the error surfaces and imagers are used to define rays to the virtual perspective, which identify a series of virtual pixels. The difference between these virtual pixel positions and the original virtual pixel position is calculated for each imager 115. Each error surface is characterized by its displacement from the original world point. A pixel error function is determined 608 relating the virtual pixel positional error to the displacement for each imager 115.

[0046] The sensitivity of the pixel error function is evaluated 610 for each imager 115. In one embodiment, this sensitivity is calculated as the derivative of the pixel error function with respect to displacement at the value of displacement given by the difference between the refined scene geometry and original geometry measurement.

[0047] FIG. 7 illustrates an embodiment of a process for determining 512 the value of the virtual pixel using the pixel weights. In this embodiment, a subset of candidate pixels from the imagers 115 may be selected to contribute to the pixel value of a given virtual pixel, that does not necessarily include all of the candidate pixels. Here, the image processing device 120 obtains 702 the candidate pixel values and respective weights corresponding to a virtual pixel for rendering. The image processing device 120 also obtains 704 a quality threshold for the virtual pixel. The quality threshold may comprise a static predefined value or a value that is dynamically updated. Furthermore, the quality threshold may be dependent on the pixel location of the virtual pixel as described in further detail below. The imager pixel value for the candidate pixels are then summed 706 in order of decreasing weight until the sum of the pixel weights reaches the quality threshold. For example, if the quality threshold is set to 70%, candidate pixel values are added until a pixel weight sum of 0.7 is reached and the weighted sum is outputted 708. Using fewer than all of the candidate pixels may improve rendering performance by reducing the number of imager pixels interpolated per virtual pixel, by reducing image memory access. Furthermore, fewer imager pixels being utilized can decrease image download time via region-of-interest selection as described below.

[0048] In an embodiment, the quality level thresholds are defined separately for different virtual pixels. Here, a weight map can be defined for a virtual perspective, such that every virtual pixel has a corresponding quality value. This map can be selected in many ways including uniform, spatially varying, or dependent on the user's level of engagement or focus (e.g., by setting higher thresholds for areas where the viewer is focused). A uniform level can be appropriate for scaling the computational requirements of rendering for different hardware platforms. Spatially varying, such as decreasing from the center of the display towards the edges, can focus attention of the user or reduce peripheral motion during translation of the perspective. In one embodiment, the user's focus is captured with an eye tracker. An example of weighting based on user engagement is that as a user interacts with a visual menu the quality of the rendered scene is decreased. In another case, weight maps are dynamically generated based on the center of focus of the eye, the foveal position, and the falloff in visual acuity about this region. In this case, pixels corresponding with the user's foveal view are given 100% quality, while other pixels are assigned a weight based on user's visual acuity at these points. Visual acuity maps from the medical literature may be used or such a map can be uniquely determined for the user during a calibration routine.

[0049] In an embodiment, the pixel weighting algorithm may balance tradeoffs between image quality and effective frame rate. Particularly, the pixel weighting algorithm can determine selected pixels to download to the display device 130 for display, without necessarily downloading all of the pixels, thereby significantly decreasing download time and increasing application frame rates. In this embodiment, instead of directly using the pixel weights for interpolation as described above, the pixel weights are instead stored in a weight frame. The weight frame is the same resolution as the captured images and each pixel value consists of a list of calculated weights. The weight frame is initialized to zero such that image pixels that are not referenced in the weighting calculation unit have zero weight. For illustrative purposes, consider a simple scheme where only the maximum weight is stored for an individual pixel. In this case, the weight frame has a single value per pixel, which is the maximum weight assigned to that pixel for all virtual pixels considered. Other techniques may alternatively be used to calculate the weight frame data structure. In one embodiment, only pixels with non-zero value are downloaded from the image processing device 120.

[0050] In another embodiment, the imager 115 does not support streaming of individual pixels, but regions or macro-blocks of the image may be individually downloaded. In this case an algorithm, such as sliding window average, can be used to assign weights to image macro-blocks and only blocks with a combined weight above a threshold will be downloaded from the image processing device 120 for display by the display device 130. In other embodiments, the image processing device 120 determines on a frame-by-frame basis whether or not to send the entire image frame to the display device 130. Here, a calculation based on the entire weight frame can be used for this determination. Furthermore, some combination of the above-described schemes may be used given different thresholds for pixels, macro-blocks, or whole imagers based on the performance of the hardware and desired rendered output quality.

[0051] In another embodiment, the above described technique using the weight frame can be combined with the quality threshold technique described above. Here, the image processing device 120 can directly set the weight frame values of pixels that are not used for rendering at a given quality level to zero. Alternatively, various threshold values can be calculated for a given quality level such that pixels, macro-blocks, or entire images below their respective threshold are not downloaded.

[0052] There are several approaches for temporally implementing the above-described. The weighting frame may be calculated at the start of an application loop, then used for selective image download, and finally perspective rendered. In this case, the depth frame structure may allow weights to be queried based on their virtual pixel correspondence so that weights are not re-calculated during rendering. Predictive approaches may use weight frames from a previous application loop directly or calculate a weight frame from a series of previous frames and a model of the user or perspective motion. The weight frame can be based on virtual pixels for a single virtual perspective or many virtual perspectives that may have different size, resolution, and quality requirements.

[0053] The described embodiments can be applied to a variety of imager layouts with imager poses, rays, and scene surface geometry in three-dimensional space. The presented algorithms are furthermore applicable to any number of imagers. For example, the described embodiments enable rendering of multiple pairs of stereoscopic virtual perspectives in real-time on commercially available hardware using between 10 and 20 high-resolution cameras running at 60 frames per second. In alternative embodiments, different numbers of cameras, different frame rates, or customized imagers may be used. Additionally, the methods can apply to calculating weights based on an approximation of the probabilistic motion of light ray paths through the scene with imager pixels treated as discrete point or using a more general model that accounts for projection of each pixel's solid angle into the scene in order to encompass a distribution of world points. The described embodiments encompass the use of world point distributions, generalizing the algorithms to act on probability distributions.

[0054] Further still, the described embodiments may apply to a homogeneous collection of imagers with similar resolution and optical properties or to heterogeneous collections of imagers with different resolutions and/or optical properties. One such heterogeneous embodiment utilizes imagers with increasing focal length towards the center of the imager array. In this case, refinement of the depth estimate occurs in similar imager groupings from lowest effective spatial resolution to highest, where the lower resolution estimate is used to bound the range of error surfaces for the next imager grouping. Furthermore, sub-pixel sampling methods and super-resolution imaging techniques may be used to interpolate pixels from heterogeneous imagers.

[0055] An example application of the described embodiments is rendering a pair of virtual perspectives with stereoscopic proportion on a display device to present an immersive, three-dimensional environment. This stereoscopic perspective can be delivered to the user through a variety of technologies including, but not limited to, a three-dimensional capable monitor or head mounted display. While the described invention can be used to present fully real environments, the captured images can also be transformed in many ways, seamlessly enhanced with additional information streams, and transmitted to a variety of users or mediums. For example, views may be rendered in a divided way to enable a user to simultaneously view multiple different virtual perspectives of the scene 150. The physical relation between the user and the space can also be modified so that the virtual perspective does not necessarily correspond to the viewer's head position. For example, the interpupillary distance between the stereoscopic views can be decreased such that the user shrinks in relation to the scene. The effect of this is to provide a high-quality, magnified view of a portion of the scene to the user. Enhancement of the scene information may present the user with sharper detail, larger depth of field, enhanced wavelength sensitivity, or broader spectrum visibility than is possible with their own vision. Additional enhancements include measurement, labeling, or segmentation. Segmentation can be used to highlight or provide information on salient portions of the scene or detected objects. Such physical objects can have an aligned digital representation that changes their appearance in order to provide additional information or customize the user experience. Furthermore, fully digital assets can also be added such as virtual displays, drawings or annotations, and models. These models can be full synthesized such as by a team of artists. Alternatively, they can represent real world objects or data such as a CAD model or clinical data from magnetic resonance imaging. In this way, the distinction between real and virtual becomes nonspecific and the system is most accurately described as a medium for integrating and transmitting information to the user. Furthermore, capture and presentation of the information becomes decoupled.

[0056] The described embodiments can also enable a remote user to view a virtual perspective of the scene from any distance (e.g., by transmitting the rendered images to a remote display device 130). Furthermore, any number or combination of co-located and remote users can interact with the scene simultaneously. The scene data may also be saved to a medium, allowing for viewing a snapshot or video of the scene at a later time.

[0057] A particular example application of the described embodiments is for performance of surgery. In this case, a co-located user wearing a head-mounted display device 130 operates on a portion of the body (corresponding to the scene 150) captured by the imager array 110. The captured scene 115 may furthermore include the surgeon's hands, tracked tools, surgical site, and peripheral body portions. The ergonomics for the surgeon are improved by decoupling of scene capture and presentation. Instead of looking down at the patient or through an optical microscope, the surgeon can gaze forward naturally while viewing the surgical site with the best possible perspective. The surgeon can seamlessly magnify the scene as needed without switching between or adjust optical magnification hardware that limits their field of view and depth of focus. Pre-operative volumetric data such as computed tomography or magnetic resonance imaging may furthermore be aligned and overlaid on the visual representation of the surgical site allowing the surgeon to see under the surface of the tissue to structures that are not yet directly visible by the imager array. Furthermore, pre-operative surgical plans including tool trajectories, hardware placement locations, and insertion depths may be overlaid and displayed to the surgeon via the display device 130.

[0058] The surgeon's hands or tools may furthermore be tracked using object tracking techniques to display measurements of position or to provide closed-loop feedback in the form of deviations from the planned trajectory. The hands may be displayed as fully real, colored, or stylized based on surgeon preference. Furthermore, due to the multitude of imager views, the hands may be shown with opacity or fully removed from the rendered view to enhance the visibility of the surgical site. Similarly, the tools can be stylized with only the relevant portions, such as knife edge, being shown. Other relevant data to the surgeon, such as the patient vitals, is presented in their view as a heads-up display. The surgeon's environment can seamlessly integrate the natural appearance of their surroundings, the enhanced perspective on the surgical site, as well as information overlays. The salience of these data streams can be modified by user preference, relevance during the time course of the procedure, or measured user engagement.

[0059] In further embodiments, the surgical application can include one or more remote viewers that view the representation of the scene via remote display device 130. For example, other members of the surgical team that are directly involved in the surgery, passive observers, and active observers may all view renderings of the scene 150. Each individual remote user may take the perspective of surgeon or their own unique perspective, with any level of enhanced information that is relevant to them directly. Passive users can be medical students in training that are physically present in the operating room. Currently these students may get a single camera feed of the surgical site on a monitor present in the room. The described system provides an immersive perspective for them to view what the surgeon is doing. Other passive observers may be students or doctors interested in a specific procedure in other hospitals around the world. Active observers can be remote experts who provide expertise to the surgeon during specific portions of the surgery regardless of their physical location.

[0060] The foregoing description of the embodiments of the invention has been presented for the purpose of illustration; it is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Persons skilled in the relevant art can appreciate that many modifications and variations are possible in light of the above disclosure.

[0061] Some portions of this description describe the embodiments of the invention in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are commonly used by those skilled in the data processing arts to convey the substance of their work effectively to others skilled in the art. These operations, while described functionally, computationally, or logically, are understood to be implemented by computer programs or equivalent electrical circuits, microcode, or the like. Furthermore, it has also proven convenient at times, to refer to these arrangements of operations as modules or engines, without loss of generality. The described operations and their associated modules may be embodied in software, firmware, hardware, or any combinations thereof.

[0062] Any of the steps, operations, or processes described herein may be performed or implemented with one or more hardware or software modules, alone or in combination with other devices. In one embodiment, a software module is implemented with a computer program product comprising a non-transitory computer-readable medium storing instructions including computer program code, which can be executed by a computer processor for performing any or all of the steps, operations, or processes described. Furthermore, any computing systems referred to in the specification may include a single processor or may be architectures employing multiple processor designs for increased computing capability.

[0063] Upon reading this disclosure, those of skill in the art will appreciate still additional alternative structural and functional designs for the disclosed embodiments as disclosed from the principles herein. Thus, while particular embodiments and applications have been illustrated and described, it is to be understood that the disclosed embodiments are not limited to the precise construction and components disclosed herein. Various modifications, changes and variations, which will be apparent to those skilled in the art, may be made in the arrangement, operation and details of the method and system disclosed herein without departing from the scope of the described embodiments.