US 12,586,294 B2Grant
Systems and methods for generating stabilized images of a real environment in artificial reality
Issue Date:2026-03-24
•20 Claims
•15 Drawing Sheets
Abstract
A method includes detecting an object of interest in a real environment and depth information of the object; determining one or more anchor locations in a three-dimensional space that correspond to a position of the object in the three-dimensional space; and generating a virtual surface anchored in the three-dimensional space. The method may further determine a pose of a camera when an image is captured and determine a region in the image that corresponds to the virtual surface. The method may further determine a first viewpoint of a first eye of the user; render a first output image based on (1) the first viewpoint relative to the virtual surface and (2) the image region corresponding to the virtual surface; and display the first output image on a first display of the computing device, the first display being configured to be viewed by the first eye of the user.
Metadata
Assignee
- Meta Platforms Technologies, LLC
Inventors
- Alessia Marra
- Gioacchino Noris
- Panya Inversin
Application Information
Application Number:US 18/767,583
Filing Date:2024-07-09
Priority Date:2020-12-30
Art Unit:2621
Classifications
IPC:
G06F3/01G02B27/01G06F3/0346G06T7/70G06T15/20H04N13/344G06F3/04815G06T19/00G06V10/25H04N5/272
Field of Search:
G02B 27/07G02B 2027/0134G02B 2027/0138G02B 2027/0187G06F 3/011G06F 3/012G06F 3/013G06F 3/017G06F 3/0346G06F 3/04815G06F 2200/00G06T 7/70G06T 15/20G06T 2207/30244G06V 10/25G06V 20/20G06V 40/18G09G 2340/10G09G 2340/12H04N 5/272H04N 13/344
Patent Drawings (15 sheets)
Description
Priority
[0001] This application is a continuation of U.S. patent application Ser. No. 17/972,269, filed on Oct. 24, 2022, entitled “Systems And Methods For Generating Stabilized Images Of A Real Environment In Artificial Reality,” which is a continuation of U.S. patent application Ser. No. 17/138,414, entitled “Systems And Methods For Generating Stabilized Images Of A Real Environment In Artificial Reality,” each of which is herein fully incorporated by reference in its respective entirety.
Technical Field
[0002] This disclosure generally relates to computer graphics and 3D reconstruction techniques.
Background
[0003] Artificial reality is a form of reality that has been adjusted in some manner before presentation to a user, which may include, e.g., a virtual reality (VR), an augmented reality (AR), a mixed reality (MR), a hybrid reality, or some combination and/or derivatives thereof. Artificial reality content may include completely generated content or generated content combined with captured content (e.g., real-world photographs). The artificial reality content may include video, audio, haptic feedback, or some combination thereof, and any of which may be presented in a single channel or in multiple channels (such as stereo video that produces a three-dimensional effect to the viewer). Artificial reality may be associated with applications, products, accessories, services, or some combination thereof, that are, e.g., used to create content in an artificial reality and/or used in (e.g., perform activities in) an artificial reality. The artificial reality system that provides the artificial reality content may be implemented on various platforms, including a head-mounted display (HMD 104) connected to a host computer system, a standalone HMD 104, a mobile device or computing system, or any other hardware platform capable of providing artificial reality content to one or more viewers.
Summary of Particular Embodiments
[0004] “Passthrough” is a feature that allows a user to see his physical environment while wearing an artificial reality device. Information about the user's physical environment is visually “passed through” to the user by having the device display information captured by the headset's external-facing cameras. The present invention improves the stability and resource requirements associated with rendering and displaying passthrough images by defining a particular region or virtual surface of the virtual environment, for example a plane or planar surface, for displaying passthrough images (a “passthrough area”). When the virtual surface corresponds to, for example, a particular physical surface in the real environment that the user frequently interacts with (e.g., the user's desk, a tabletop, the user's hands, or a canvas on a wall), there is no need to continuously calculate the depth of the passthrough area as the user moves throughout a three-dimensional environment, because the particular physical surface in the real environment remains at a fixed pose.
[0005] In particular embodiments the computing system requests the user to perform a one-time setup to specify anchor locations and determine a pose of the virtual surface in the real environment, through for example, one or more controller or hand movements. Once defined, the computing system may receive images and identify one or more regions of the image that correspond to the virtual surface. The computing system may display the passthrough images on a device display associated with a user based on one or more criteria, for example the distance between the user and the virtual surface. Additionally, the computing system may also display passthrough images for the surrounding walls, floor, etc. that are nearby or adjacent to the virtual surface. In particular embodiments, the computing system may apply one or more visual effects to enhance the display of the one or more passthrough images to the user.
[0006] The embodiments disclosed herein are only examples, and the scope of this disclosure is not limited to them. Particular embodiments may include all, some, or none of the components, elements, features, functions, operations, or steps of the embodiments disclosed herein. Embodiments according to the invention are in particular disclosed in the attached claims directed to a method, a storage medium, a system and a computer program product, wherein any feature mentioned in one claim category, e.g. method, can be claimed in another claim category, e.g. system, as well. The dependencies or references back in the attached claims are chosen for formal reasons only. However any subject matter resulting from a deliberate reference back to any previous claims (in particular multiple dependencies) can be claimed as well, so that any combination of claims and the features thereof are disclosed and can be claimed regardless of the dependencies chosen in the attached claims. The subject-matter which can be claimed comprises not only the combinations of features as set out in the attached claims but also any other combination of features in the claims, wherein each feature mentioned in the claims can be combined with any other feature or combination of other features in the claims. Furthermore, any of the embodiments and features described or depicted herein can be claimed in a separate claim and/or in any combination with any embodiment or feature described or depicted herein or with any of the features of the attached claims.
Brief Description of the Drawings
[0007] FIG. 1A illustrates an example artificial reality system worn by a user, in accordance with particular embodiments.
[0008] FIG. 1B illustrates an example of a passthrough feature, in accordance with particular embodiments.
[0009] FIGS. 2A and 2B illustrate top-down visualizations of a 3D mesh being deformed to represent the contours of an observed environment.
[0010] FIG. 3 provides an illustration of 3D-passthrough rendering based on the 3D mesh.
[0011] FIG. 4 illustrates one or more cameras associated with the artificial reality device of a user capturing the physical environment of the user.
[0012] FIG. 5 illustrates a first-person perspective of a user defining a virtual surface for displaying passthrough images in a three-dimensional environment.
[0013] FIG. 6A illustrates rendering one or more output passthrough images on the virtual surface.
[0014] FIG. 6B illustrates rendering one or more passthrough images on the virtual surface from the perspective of a user's eyes.
[0015] FIG. 7 illustrates one or more passthrough images displayed on a device display associated with a user.
[0016] FIGS. 8A-8C illustrate selectively displaying passthrough images based on a location or orientation of an artificial reality device.
[0017] FIG. 9 illustrates an example method for displaying stabilized images of a real environment in artificial reality.
[0018] FIG. 10 illustrates an example network environment associated with a social-networking system.
[0019] FIG. 11 illustrates an example computer system.
Description of Example Embodiments
[0020] FIG. 1A illustrates an example of an artificial reality system 100 worn by a user 102. In particular embodiments, the artificial reality system 100 may comprise a head-mounted device (“HMD 104”) 104, a controller 106, and a computing system 108. The HMD 104 may be worn over the user's eyes and provide visual content to the user 102 through internal displays (not shown). The HMD 104 may have two separate internal displays, one for each eye of the user 102. As illustrated in FIG. 1A , the HMD 104 may completely cover the user's field of view. By being the exclusive provider of visual information to the user 102, the HMD 104 achieves the goal of providing an immersive artificial-reality experience. One consequence of this, however, is that the user 102 would not be able to see the physical (or real) environment surrounding him, as his vision is shielded by the HMD 104. As such, the passthrough feature described herein is needed to provide the user with real-time visual information about his physical environments.
[0021] Referring again to FIG. 1A , the HMD 104 may have external-facing cameras, such as the two forward-facing cameras 105A and 105B shown in FIG. 1A . While only two forward-facing cameras 105A-B are shown, the HMD 104 may have any number of cameras facing any direction (e.g., an upward-facing camera to capture the ceiling or room lighting, a downward-facing camera to capture a portion of the user's face and/or body, a backward-facing camera to capture a portion of what's behind the user, and/or an internal camera for capturing the user's eye gaze for eye-tracking purposes). The external-facing cameras 105A and 105B are configured to capture the physical environment around the user and may do so continuously to generate a sequence of frames (e.g., as a video).
[0022] In particular embodiments, the pose (e.g., position and orientation) of the HMD 104 within the environment may be needed. For example, in order to render an appropriate display for the user 102 while he is moving about in a virtual environment, the system 100 would need to determine his position and orientation at any moment. Based on the pose of the HMD 104, the system 100 may further determine the viewpoint of either of the cameras 105A and 105B or either of the user's eyes. In particular embodiments, the HMD 104 may be equipped with inertial-measurement units (“IMU”). The data generated by the IMU, along with the stereo imagery captured by the external-facing cameras 105A-B, allow the system 100 to compute the pose of the HMD 104 using, for example, SLAM (simultaneous localization and mapping) or other suitable techniques.
[0023] In particular embodiments, the artificial reality system 100 may further have one or more controllers 106 that enable the user 102 to provide inputs. The controller 106 may communicate with the HMD 104 or a separate computing unit 108 via a wireless or wired connection. The controller 106 may have any number of buttons or other mechanical input mechanisms. In addition, the controller 106 may have an IMU so that the position of the controller 106 may be tracked. The controller 106 may further be tracked based on predetermined patterns on the controller. For example, the controller 106 may have several infrared LEDs or other known observable features that collectively form a predetermined pattern. Using a sensor or camera, the system 100 may be able to capture an image of the predetermined pattern on the controller. Based on the observed orientation of those patterns, the system may compute the controller's position and orientation relative to the sensor or camera.
[0024] The artificial reality system 100 may further include a computer unit 108. The computer unit may be a stand-alone unit that is physically separate from the HMD 104 or it may be integrated with the HMD 104. In embodiments where the computer 108 is a separate unit, it may be communicatively coupled to the HMD 104 via a wireless or wired link. The computer 108 may be a high-performance device, such as a desktop or laptop, or a resource-limited device, such as a mobile phone. A high-performance device may have a dedicated GPU and a high-capacity or constant power source. A resource-limited device, on the other hand, may not have a GPU and may have limited battery capacity. As such, the algorithms that could be practically used by an artificial reality system 100 depends on the capabilities of its computer unit 108.
[0025] “Passthrough” is a feature that allows a user to see his physical environment while wearing HMD 104. Information about the user's physical environment is visually “passed through” to the user by having the HMD 104 display information captured by the headset's external-facing cameras 105A and 105B. FIG. 1B illustrates an example of the passthrough feature. A user 102 may be wearing an HMD 104, immersed within a three-dimensional space, for example and not by way of limitation, a virtual reality environment. A physical table 150 is in the physical environment surrounding the user 102. However, due to the HMD 104 blocking the vision of the user 102, the user 102 is unable to directly see the table 150. To help the user perceive his physical environment while wearing the HMD 104, the passthrough feature captures information about the physical environment using, for example, the aforementioned external-facing cameras 105A-B. Methods for rendering and displaying passthrough images are described further in U.S. application Ser. No. 16/746,128, filed Jul. 24, 2020, entitled “Systems, Methods, and Media for Generating Visualization of Physical Environment in Artificial Reality,” hereby incorporated by reference in its entirety.
[0026] Simply displaying the captured images would not work as intended, however. Since the locations of the cameras 105A-B do not coincide with the locations of the user's eyes, simply displaying the captured images would not provide the user 102 with an accurate view of the physical environment since the cameras 105A-B cannot physically be located at the exact same viewpoint as the user's eyes. As such, the passthrough feature described herein uses a re-projection technique that generates a 3D representation of the physical environment and then renders images based on the 3D representation from the viewpoints of the user's eyes. The captured information may then be re-projected to the user 102 based on his viewpoints. In particular embodiments where the HMD 104 has a right display 160A for the user's right eye and a left display 160B for the user's left eye, the system 100 may individually render (1) a re-projected view 150A of the physical environment for the right display 160A based on a viewpoint of the user's right eye and (2) a re-projected view 150B of the physical environment for the left display 160B based on a viewpoint of the user's left eye.
[0027] In particular embodiments, the 3D representation may be generated based on depth measurements of physical objects observed by the cameras 105A-B. Depth may be measured in a variety of ways. In particular embodiments, depth may be computed based on stereo images. For example, the two forward-facing cameras 105A-B may share an overlapping field of view and be configured to capture images simultaneously. As a result, the same physical object may be captured by both cameras 105A-B at the same time. For example, a particular feature of an object may appear at one pixel pA in the image captured by camera 105A, and the same feature may appear at another pixel pB in the image captured by camera 105B. As long as the depth measurement system knows that the two pixels correspond to the same feature, it could use triangulation techniques to compute the depth of the observed feature. For example, based on the camera 105A's position within a 3D space and the pixel location of pixel pA relative to the camera 105A's field of view, a line could be projected from the camera 105A and through the pixel pA. A similar line could be projected from the other camera 105B and through the pixel pB. Since both pixels are supposed to correspond to the same physical feature, the two lines should intersect. The two intersecting lines and an imaginary line drawn between the two cameras 105A and 105B form a triangle, which could be used to compute the distance of the observed feature from either camera 105A or 105B or a point in space where the observed feature is located.
[0028] In addition, since the images have no depth, simply displaying the images would not provide the user with proper parallax effects if he were to shift away from where the images were taken. Incorrect parallax, coupled with user motion, could lead to motion sickness. Thus, to generate correct parallax, particular embodiments of the passthrough feature extracts information about the environment from the captured images (e.g., depth information), use the information to generate a 3D model (a geometric scene representation) of the physical environment, and reconstruct a scene of the modeled environment from the user's current viewpoint.
[0029] Photon-to-visuals latency is another issue addressed by the passthrough feature. The delay between a photon hitting the camera and it appearing on the screen (as part of the captured image) determines the visual comfort of interacting in a dynamic world. Particular embodiments of the passthrough feature overcomes this issue by updating the 3D model representation of the environment based on images captured at a sufficiently high rate (e.g., at 30 Hz, 60 Hz, etc.) and rendering images based on the latest known head pose of the user.
[0030] The passthrough image can be unstable (e.g., it warps visibly) and resource-intensive to render and display because the 3D representation must be continuously updated as the user moves throughout the physical environment. Further, the depth estimations of objects are often inaccurate. Over time, this leads to inaccuracies and inconsistencies in the 3D representation from the viewpoints of the user's eyes, especially with regard to the estimated depth of one or more objects in the physical environment. Thus, while usable for basic functions, these passthrough techniques make it difficult for a user to perform tasks or interact with objects in the physical environment (e.g., typing on a real keyboard, picking up a mobile phone from a table) while in the virtual environment. Further, the inaccuracies in the 3D representation may result in an aesthetically inconsistent experience due to warping of the displayed images.
[0031] FIGS. 2A and 2B illustrate top-down visualizations of a 3D mesh being deformed to represent the contours of an observed environment. For clarity, the figures are drawn in 2D, but it should be understood that the 3D mesh is a 3D construct. FIG. 2A illustrates an embodiment of the 3D mesh 200A being initialized to be equal-distance (e.g., 1, 2, 2.5, or 3 meters) from a viewer 210 (represented by a camera). In the particular example being shown, the radius of the 3D mesh 200A is 2 meters. Since the 3D mesh 200A is equal-distance away from the viewer 210, it forms a hemisphere around the user. For clarity, FIG. 2A illustrates a portion of a cross-section of that hemisphere, resulting in the half-circle shown. FIG. 2A further illustrates points (e.g., 220) in the point cloud that are deemed reliable. These points 220 represent observed features in the environment and may be generated using the embodiments described elsewhere herein.
[0032] The 3D mesh 200A may be deformed according to the points 220 in order to model the contour of the environment. In particular embodiments, the 3D mesh 200A may be deformed based on the viewer's 210 position and the points 220 in the point cloud. To determine which portion of the 3D mesh 200A corresponds to each point in the point cloud 220, the computing device may cast a conceptual ray from the viewer's 210 position towards that point. Each ray would intersect with a primitive (e.g., a triangle or other polygon) of the 3D mesh. For example, FIG. 2A shows a ray 230 being cast from the viewer 210 towards point 220A. The ray 230 intersects the 3D mesh 200A at a particular location 240. As a result, mesh location 240 is deformed based on the depth value associated with the point 220A. For example, if the point 220 is 2.2 meters away from the viewer 210, the depth value associated with the mesh location 240 may be updated to become 2.2 meters from its initial value of 2 meters. FIG. 2B illustrates the deformed 3D mesh 200B that may result from the deformation process. At this point, the deformed mesh 200B represents the contour of the physical environment observed by the viewer 210.
[0033] FIG. 3 provides an illustration of 3D-passthrough rendering based on the 3D mesh. In particular embodiments, the rendering system may determine the user's 102 current viewing position relative to the environment. In particular embodiments, the system may compute the pose of the HMD 104 using SLAM or other suitable techniques. Based on the known mechanical structure of the HMD 104, the system could then estimate the viewpoints of the user's eyes 300A and 300B using offsets from the pose of the HMD 104. In particular embodiments the computing system may determine a viewpoint that corresponds to a first eye of the user (e.g., a first viewpoint that corresponds to a right eye of the user). In particular embodiments the computing system may determine a viewpoint that corresponds to a second eye of the user (e.g., a second viewpoint that corresponds to a left eye of the user). One or more viewpoints may be determined by utilizing any of the sensor data or image data received by the computing system. A viewpoint of a particular eye may comprise a location, an orientation, and a field of view from the particular eye. In particular embodiments the location and orientation comprising the viewpoint may be represented using one or more coordinate systems, for example and not by way of limitation, via an absolute global coordinate system (e.g., x, y, z), or via a localized coordinate system relative to one or more components of artificial reality system, for example and not by way of limitation, the HMD 104.
[0034] The system may then render a passthrough image for each of the user's eyes 300A-B. For example, to render a passthrough image for the user's right eye 300A, the system may cast a ray 320 from the estimated viewpoint of the right eye 300A through each pixel of a virtual screen space 310A to see which portion of the mesh 300B the rays would intersect. This ray casting process may be referred to as a visibility test, as the objective is to determine what is visible from the selected viewpoint 300A. In the particular example shown, the ray 320 projected through a particular pixel 322 intersects with a particular point 321 on the mesh. This indicates that the point of intersection 321 is to be displayed by the pixel 322. Once the point of intersection 321 is found, the rendering system may sample a corresponding point in a texture image that is mapped to the point of intersection 321. In particular embodiments, the image captured by the cameras 105A-B of the HMD 104 may be used to generate a texture for the mesh 300B. Doing so allows the rendered image to appear more like the actual physical object. In a similar manner, the rendering system may render a passthrough image for the user's left eye 300B. In the example shown, a ray 330 may be cast from the left-eye viewpoint 300B through pixel 332 of the left screen space 310B. The ray 330 intersects the mesh 300B at location 331. The rendering system may then sample a texture image at a texture location corresponding to the location 331 on the mesh 300B and compute the appropriate color to be displayed by pixel 332. Since the passthrough images are re-rendered from the user's viewpoints 300A-B, the images would appear natural and provide proper parallax effect.
[0035] The present invention remedies the stability and resource requirements associated with updating the 3D mesh by defining a particular region or virtual surface, for example a plane or planar surface (hereinafter “virtual surface”) in the virtual environment for displaying passthrough images (a “passthrough area”). When the virtual surface corresponds to, for example, a particular physical surface in the real environment that the user frequently interacts with (e.g., the user's desk, a tabletop, the user's hands, or a canvas on a wall), the computer system does not need to continuously calculate the depth of the passthrough area as the user moves throughout a three-dimensional environment, because the particular physical surface in the real environment remains at a fixed pose (e.g., location and orientation). Thus, in one embodiment the virtual surface may be located at a fixed pose (e.g., location and orientation) in the virtual environment that corresponds to or is associated with the pose of the particular physical surface. In this manner the pose of the virtual surface may be anchored with respect to the physical environment (e.g., the virtual surface is fixed to a physical tabletop). In particular embodiments the computing system requests the user to perform a one-time setup to specify anchor locations and determine a pose of the virtual surface (e.g., the user could designate a coordinate or the dimensions of the desk in the real environment using a controller).
[0036] In particular embodiments the surface geometry of the virtual surface may static and remain temporally fixed in the virtual environment, such that the user-defined virtual surface simply coincides with the location of a physical surface. Because the surface geometry is defined independently from the actual depth measurements on the corresponding physical surface (in contrast to the methods described in FIGS. 2 and 3 ), the virtual surface remains the same regardless of changes to the physical surface. For example, if the user relocates one or more objects located on the particular physical object or surface that corresponds to the virtual surface (e.g., the user picks up their phone off the tabletop), the surface geometry may remain constant. The computing system projects the passthrough images of random objects on the physical surface onto the virtual surface, regardless of the physical depth of the random objects.
[0037] By defining and fixing these parameters, the computing system can generate a corresponding virtual surface that is fixed in the virtual environment and provide temporally stable passthrough images for the user. With the virtual surface anchored within the virtual environment, the computing system can render and display passthrough images without having to continuously track and update the depth of the virtual surface as the user moves throughout the physical environment. Further, because the defined surface is virtual, the real surface can contain any number of real objects (which can also be moved on the surface) without the usual difficulty of updating and reconstructing a 3D representation or 3D mesh of the passthrough area and accompanying objects. This significantly improves the passthrough experience.
[0038] In particular embodiments the computing system may capture one or more images of the real environment (also referred to as the physical environment) and physical surroundings using a camera worn by the user. For example, the computing system may have a camera capture one or more images from a particular current perspective of the user. In particular embodiments the camera worn by the user is associated with an artificial reality device, for example a head-mounted device worn by the user that blocks the user from seeing the real environment directly. In one embodiment for generating passthrough images, a computing system may access one or more images of an environment captured by cameras 105A and 105B of a device worn by a user. The one or more images may include, for example, a particular physical surface that the user frequently interacts with (e.g., the user's desk, a tabletop, the user's hands, or a canvas on a wall) that was utilized to determine the virtual surface. The system may generate in some embodiments, based on the one or more images, depth measurements of one or more objects in the environment. In some embodiments, the system may generate a mesh covering a field of view of the user and then update the mesh based on the depth measurements to represent a contour of the objects in the environment.
[0039] In particular embodiments the computing system may determine a pose of each of the one or more cameras 105A and 105B when the image was captured. The camera pose may be determined by utilizing any of the sensor data or image data received by the computing system, for example, image or sensor data received by the HMD 104 of the user. In particular embodiments the camera pose may be determined based on a known spatial relationship between the pose of the one or more cameras and the pose of one or more components of the artificial reality system, for example the HMD 104. Based on, for example, the pose of the HMD 104, the computing system may further determine the pose of either of the cameras 105A and 105B or either a pose of one or more of the user's eyes. The camera pose may comprise a location and an orientation of the camera. In particular embodiments the location and orientation comprising the camera pose may be represented using one or more coordinate systems, for example and not by way of limitation, via an absolute global coordinate system (e.g., x, y, z), or via a localized coordinate system relative to one or more components of artificial reality system, for example and not by way of limitation, the HMD 104 or the one or more controllers 106.
[0040] FIG. 4 illustrates one or more cameras associated with the artificial reality device of a user capturing the physical environment of the user. The physical environment 400 may contain one or more physical surfaces 405 that could be utilized to define a virtual surface according to the methods described herein, for example a tabletop or a painting as depicted in FIG. 4 . The one or more physical surfaces 405 may further include one or more objects the user wishes to interact with, for example a keyboard, cell phone, or documents as depicted in FIG. 4 . As the user moves through the physical environment, the one or more cameras 105A-B may capture one or more images of the physical environment. The computing system may display these one or more images to assist the user in defining one or more anchor locations or virtual surfaces.
[0041] FIG. 5 illustrates a first-person perspective of a user defining a virtual surface for displaying passthrough images in a three-dimensional environment. In particular embodiments the computing system may determine one or more poses or anchor locations in order to generate a virtual surface. For example, the computing system may display one or more images 500 to the user of an artificial reality system, where the images 500 represent the physical environment while the user is wearing HMD 104. By viewing their physical environment, the user or computing system can identify and determine one or more anchor locations 510 (e.g., one or more coordinates, lines, or perimeters) to define a virtual surface 505 with which to render and display passthrough images. In particular embodiments, the anchor locations and corresponding virtual surface 505 may correspond to a location or orientation of a particular physical surface in the real environment. In particular embodiments, the virtual surface may be determined by a defined dimension, area, contour, or orientation (e.g., the user may define a virtual surface that is a rectangle with dimensions 2 feet by 3 feet).
[0042] In particular embodiments, the computing system may determine and designate the one or more coordinates of the virtual surface 505 based on the computing system tracking one or more motions performed by the user. As depicted in FIG. 5 , the user defines the virtual surface 505 by using one or more motions to define or trace one or more anchor locations 510 (e.g., a perimeter of the virtual surface). These one or more motions may be performed, for example and not by way of limitation, by hand motions or eye motions that are tracked by the computing system. The one or more motions may be further associated with an interaction with one or more controllers 106, for example, movements with the controllers or input received through one or more buttons or mechanical input mechanisms. The motions for designating one or more anchor locations 510 may comprise, for example and not by way of limitation, placing a hand or controller at a particular coordinate or pointing at or selecting a particular coordinate in the three-dimensional environment. The motions for determining one or more dimensions may comprise, for example and not by way of limitation, using a hand or controller to trace a length width or height, or tracing a perimeter of an area (e.g., tracing, click-and-drag, or drag-select motions). In particular embodiments the computing system may render and display one or more icons, images, or representations overlaid over the passthrough images 500 so the user can visualize the virtual surface 505 and one or more anchor locations 510 as they are defined.
[0043] In particular embodiments the one or more anchor locations 510 or dimensions may be determined using computing vision techniques. Based on, for example and not by way of limitation, an image captured from a viewpoint in the environment, a computer system may recognize objects (e.g., an art canvas, a desk, etc.) or portions of objects (e.g., the corner of a table) in the image and compute depth information to determine the locations of the one or more objects or portions of objects in the three-dimensional environment. With the accurate locations of the objects in the three-dimensional environment, the system may perform a variety of functions, such as determining one or more poses of objects (e.g., a location and orientation in the three-dimensional space) or anchor locations 510 (e.g., one or more coordinates, lines, or perimeters) within a three-dimensional space in order to determine a virtual surface for rendering and displaying passthrough images.
[0044] In particular embodiments the one or more anchor locations may represent a location in the real environment that remains at a fixed (i.e., “anchored”) position and orientation over time. One or more anchor locations may comprise one or more coordinates, a dimension (e.g., a length), an area, or a volume. The one or more anchor locations may be represented by one or more coordinate systems in the three-dimensional space, for example and not by way of limitation, via an absolute global coordinate system (e.g., x, y, z) or via a localized coordinate system relative to one or more objects in the three-dimensional space.
[0045] In particular embodiments the one or more anchor locations correspond to a location of one or more objects, regions, or surfaces in the real environment (e.g., one or more anchor locations may correspond to the corner of a desk, a length of a wall, a masking canvas area, etc.). For example and not by way of limitation, the computing system may determine a coordinate that corresponds to the corner of a table, or an area of the surface of a desk. In particular embodiments the computing system may use the one or more coordinates or dimensions to determine the one or more anchor locations for determining the virtual surface.
[0046] In particular embodiments the computing system may generate a virtual surface 505 or volume in the three-dimensional space. This virtual surface 505 may be generated based on the one or more of the anchor locations determined according to the methods described herein. In particular embodiments the position, orientation, shape, contour, or dimensions of the virtual surface 505 may remain at a fixed (i.e., “anchored”) position and orientation over time. These fixed properties may correspond to one or more of the defined anchor locations, or one or more objects in the physical environment (e.g., a virtual surface with defined 3 feet by 2 feet rectangular dimensions may correspond to a rectangular table top in the physical environment with the same dimensions). The generated virtual surface 505, which remains fixed in the three-dimensional space by the computing system, allows for temporal stability when projecting the passthrough images, as the defined location and corresponding depth of the 3D surface relative to the user and HMD 104 need not be constantly estimated and updated.
[0047] In particular embodiments, the computing system may utilize the virtual surface to render one or more output passthrough images for the virtual surface. FIG. 6A illustrates rendering one or more output passthrough images on the virtual surface. In particular embodiments the computing system may receive one or more images captured by the cameras 105A-B to render one or more passthrough images from the perspective of the camera 610 on the virtual surface. The computing system may receive one or more images to determine one or more regions in the one or more images that, as viewed from the pose of the cameras 105A or 105B, corresponds to the virtual surface when viewed from the perspective of the user of the HMD 104. In particular embodiments this determination may be based on, for example, the known spatial relationships between, for example, the pose of the one or more cameras on the HMD 104, the pose of the user 102 or the HMD 104, or the pose of the one or more anchor locations or virtual surface. For example, if the user turns their head such that the virtual surface 505 is now on the peripheral of one or more images captured by the cameras 105A or 105B, the computing system may use the pose of the HMD 104 or the one or more cameras 105A or 105B relative to the pose of the anchor locations or virtual surface 505 to ensure the passthrough images remain properly projected onto the corresponding regions of virtual surface. This may require the computing system to determine a camera or HMD 104 pose associated with an image, and use the spatial properties of this pose (e.g., the location, orientation, and field-of-view) to determine a region of the image contains at least a portion of the virtual surface. As another example, in particular embodiments this determination of a region of the images that correspond to the virtual surface can be based on one or more computer vision techniques (e.g., object recognition). In particular embodiments the computing system may further render one or more pixels to the virtual surface 505 to map the geometry to the image information to, for example, generate texture for the virtual surface.
[0048] However, in order to provide the correct perspective, the computing system must render the output passthrough images from the perspective of the user's eyes 500A-B. FIG. 6B illustrates rendering one or more passthrough images on the virtual surface from the perspective of a user's eyes. If the computing system simply displayed the one or more rendered images 610 from the perspective of the cameras 500A-B, the projected images would not provide the user 102 with an accurate view of the physical environment. Thus, the computing system may render one or more passthrough images from the perspective of the user's eyes 615. In particular embodiments the computing system may render a first output passthrough image from the perspective of the user's eyes based on (1) a first viewpoint or perspective from a first eye of the user relative to the virtual surface and (2) the image region corresponding to the virtual surface. In particular embodiments the computing system may render a second output image based on (1) a second viewpoint or perspective from a second eye relative to the virtual surface and (2) the image region corresponding to the virtual surface. When the system renders a one or more output passthrough images from the perspective of the user's eyes 615 for display, the system may perform visibility tests from the perspectives of the user's eyes. For example, the system may cast one or more rays 620 into the 3D space from a viewpoint that corresponds to each eye of the user. In this manner, the rendered images 615 that are displayed to the user would be computed from the perspective of the user's eyes, rather than from the perspective of the external-facing cameras 105A-B.
[0049] In particular embodiments the computing system may display the passthrough images on a device display associated with a user, for example an HMD 104 display. FIG. 7 illustrates one or more passthrough images displayed on a device display associated with a user. In particular embodiments a first output image (e.g., a left eye image) may be displayed on a first display (e.g., a left eye display) of the device configured to be viewed by the first eye (e.g., the left eye) of the user, and a second output image (e.g., a right eye image) may be displayed on a second display (e.g., a right eye display) of the device configured to be viewed by the second eye (e.g., the right eye) of the user. In particular embodiments the computing system may display passthrough images outside the virtual surface. For example, if the virtual surface corresponds to a table in the real environment, the computing system may display the passthrough images on the virtual surface, as well as passthrough images for the surrounding walls, floor, etc. Passthrough images outside the virtual surface may be rendered and displayed according to the methods described in U.S. application Ser. No. 16/746,128, filed Jul. 24, 2020, entitled “Systems, Methods, and Media for Generating Visualization of Physical Environment in Artificial Reality,” hereby incorporated by reference in its entirety.
[0050] In particular embodiments the display of the passthrough images may be based on a location or orientation of the HMD 104 or user wearing the HMD 104. FIGS. 8A-8C illustrate selectively displaying passthrough images based on a location or orientation of the HMD 104. The computing system may determine a location or orientation of the HMD 104, user 102, or cameras 105A and 105B and, based on the distance of the HMD 104, user 102, or cameras 105A and 105B from the virtual surface 505, determine whether to display the passthrough images on a device display associated with the user.
[0051] As depicted in FIG. 8A , in particular embodiments the passthrough images may be displayed based on a horizontal distance of the HMD 104, user 102, or cameras 105A and 105B from the virtual surface 505, for example a minimum horizontal threshold distance. For example, if virtual surface 505 represents a table in the real environment, user 102 may only be interested in viewing the passthrough images of the table when they are within a certain horizontal proximity from the table (e.g., when the user moves away from their work station). In these embodiments the computing system may determine a horizontal distance of the HMD 104, user 102, or cameras 105A and 105B from the virtual surface 505, based on for example, sensor data or image data received by the computing system, and selectively display the one or more passthrough images based on this horizontal distance from virtual surface 505. Although FIG. 8A depicts only displaying passthrough images when the user 102 exceeds a minimum horizontal threshold distance from virtual surface 505, in particular embodiments the computing system may only display passthrough images when the user 102 is within a minimum horizontal threshold distance from virtual surface 505.
[0052] As depicted in FIG. 8B , in particular embodiments the passthrough images may be displayed based on a vertical distance of the HMD 104, user 102, or cameras 105A and 105B from the virtual surface 505, for example a minimum vertical threshold distance. For example, if virtual surface 505 represents a table in the real environment, user 102 may only be interested in viewing the passthrough images of the table when they exceed a certain vertical proximity from the table (e.g., when the user stands up from sitting at their workstation). In these embodiments the computing system may determine a vertical distance of the HMD 104, user 102, or cameras 105A and 105B from the virtual surface 505, based on for example, sensor data or image data received by the computing system, and selectively display the one or more passthrough images based on this vertical distance from virtual surface 505. Although FIG. 8B depicts only displaying passthrough images when the user 102 exceeds a minimum vertical threshold distance from virtual surface 505, in particular embodiments the computing system may only display passthrough images when the user 102 is within a minimum vertical threshold distance from virtual surface 505.
[0053] As depicted in FIG. 8C , in particular embodiments the passthrough images may be displayed based on an orientation of the HMD 104, user 102, or cameras 105A and 105B relative to the virtual surface 505, for example a minimum threshold orientation. For example, if virtual surface 505 represents a table in the real environment, user 102 may only be interested in viewing the passthrough images of the table when they are looking downwards towards the table, or when the table is within the field of view of the user (e.g., when the user looks down at an object on the table). As another example, when the user tilts his head down to look at his own body (e.g., to reach into his pocket), the portion of his field-of-view that intersects a defined volume around his body would display the corresponding passthrough images based on the current view or field of view of the user. In these embodiments the computing system may determine an orientation of the HMD 104, user 102, or cameras 105A and 105B relative to the virtual surface 505 (e.g., whether the user is looking at the virtual surface, whether the camera field of view includes the virtual surface or real object, etc.), based on for example, sensor data or image data received by the computing system, and selectively display the one or more passthrough images based on this orientation of the HMD 104, user 102, or cameras 105A and 105B relative to the virtual surface 505. Although FIG. 8C depicts only displaying passthrough images when the orientation of the view of user 102 is within a minimum threshold orientation relative to the virtual surface 505, in particular embodiments the computing system may only display passthrough images when the orientation of the view of user 102 exceeds the minimum threshold orientation relative to the virtual surface 505.
[0054] In particular embodiments, the computing system may apply one or more visual effects to enhance the display of the one or more passthrough images to the user. For example, the computing system may utilize the methods described herein to determine one or more regions of the image that correspond to the virtual surface, and only display the regions of the image to the user that correspond to the virtual surface (e.g., if the virtual surface represents a table in the real environment, the computing system may omit from display the regions of the image that correspond to, for example, the floor immediately adjacent to the table). As another example, in particular embodiments the computing system may use shading or other visual techniques to create a gradual gradient between the passthrough images and the virtual environment. As yet another example, in particular embodiments the display of the passthrough images may gradually fade in or fade out based on the proximity of the user, HMD 104, or cameras 105A and 105B from the virtual surface as illustrated in FIGS. 8A-8C and outlined above (e.g., rather than abruptly display the passthrough images when the user exceeds the minimum horizontal threshold distance, the computing system may gradually fade the passthrough images in or out as the user approaches the minimum horizontal threshold distance).
[0055] FIG. 9 illustrates an example method 900 for displaying stabilized images of a real environment in artificial reality. The method may begin at step 910, where a computing system tracks motions performed by a hand of a user. The one or more motions may be further associated with an interaction with one or more controllers 106, for example, movements with the controllers or input received through one or more buttons or mechanical input mechanisms.
[0056] At step 920, the computing system determines, based on the tracked motions, one or more anchor locations in a three-dimensional space. An anchor location may represent a location in the three-dimensional space that remains at a fixed (i.e., “anchored”) position and orientation over time. One or more anchor locations may comprise one or more coordinates, a dimension (e.g., a length), an area, or a volume. In particular embodiments the one or more anchor locations correspond to a location of one or more objects in the real environment (e.g., an anchor location may correspond to the corner of a desk, a length of a wall, a masking canvas area, etc.).
[0057] At step 930, the computing system generates, based on the one or more anchor locations, a virtual surface anchored in the three-dimensional space. In particular embodiments the virtual surface may remain at a fixed (i.e., “anchored”) position and orientation over time. This fixed position and orientation may correspond to the position and orientation of the one or more anchor locations, or physical objects in the real environment. The generated virtual surface, which remains fixed in the three-dimensional space by the computing system, allows for temporal stability when projecting the passthrough images, as the defined location and corresponding depth of the 3D surface relative to the user and HMD 104 need not be constantly estimated and updated.
[0058] At step 940, the computing system captures an image of a real environment using a camera worn by the user. The camera may be associated with a device worn by the user (e.g., an HMD 104) that blocks the user from seeing the environment directly. The images may be stereo image pairs captured simultaneously by two cameras with a shared field of view.
[0059] At step 950, the computing system determines a pose of the camera when the image is captured. The camera pose may be determined by utilizing any of the sensor data or image data received by the computing system. In particular embodiments the camera pose may be determined based on a known spatial relationship between the pose of the camera and the pose of one or more components of the artificial reality system, for example the HMD 104.
[0060] At step 960, the computing system determines a region in the image that, as viewed from the pose of the camera, corresponds to the virtual surface. In particular embodiments this determination may be based on, for example, the known spatial relationships between, for example, the pose of the one or more cameras on the HMD 104, the pose of the user or HMD 104, and the pose of the one or more anchor locations or virtual surface. As another example, in particular embodiments this determination can be based on one or more computer vision techniques (e.g., object recognition).
[0061] At step 970, the computing system determines a first viewpoint of a first eye of the user (e.g., the right eye). In embodiments where the HMD 104 has one display per eye, the system may further determine a second viewpoint of a second eye of the user (e.g., the left eye). At step 980, the computing system renders a first output image based on the first viewpoint relative to the virtual surface and the image region corresponding to the virtual surface. If a second output image is desired for the user's other eye, the system may further render the second output image based on the second viewpoint relative to the virtual surface and the image region corresponding to the virtual surface.
[0062] At step 990, the computing system displays the first output image on a first display of the device and/or the second output image on a second display of the device. The first display is configured to be viewed by the first eye of the user and the second display is configured to be viewed by the second eye of the user.
[0063] Particular embodiments may repeat one or more steps of the method of FIG. 9 , where appropriate. Although this disclosure describes and illustrates particular steps of the method of FIG. 9 as occurring in a particular order, this disclosure contemplates any suitable steps of the method of FIG. 9 occurring in any suitable order. Moreover, although this disclosure describes and illustrates an example method for displaying stabilized images of a real environment in artificial reality including the particular steps of the method of FIG. 9 , this disclosure contemplates any suitable method for displaying stabilized images of a real environment in artificial reality including any suitable steps, which may include all, some, or none of the steps of the method of FIG. 9 , where appropriate. Furthermore, although this disclosure describes and illustrates particular components, devices, or systems carrying out particular steps of the method of FIG. 9 , this disclosure contemplates any suitable combination of any suitable components, devices, or systems carrying out any suitable steps of the method of FIG. 9 .
[0064] FIG. 10 illustrates an example network environment 1000 associated with a social-networking system. Network environment 1000 includes a client system 1030, a social-networking system 1060, and a third-party system 1070 connected to each other by a network 1010. Although FIG. 10 illustrates a particular arrangement of client system 1030, social-networking system 1060, third-party system 1070, and network 1010, this disclosure contemplates any suitable arrangement of client system 1030, social-networking system 1060, third-party system 1070, and network 1010. As an example and not by way of limitation, two or more of client system 1030, social-networking system 1060, and third-party system 1070 may be connected to each other directly, bypassing network 1010. As another example, two or more of client system 1030, social-networking system 1060, and third-party system 1070 may be physically or logically co-located with each other in whole or in part. Moreover, although FIG. 10 illustrates a particular number of client systems 1030, social-networking systems 1060, third-party systems 1070, and networks 1010, this disclosure contemplates any suitable number of client systems 1030, social-networking systems 1060, third-party systems 1070, and networks 1010. As an example and not by way of limitation, network environment 1000 may include multiple client system 1030, social-networking systems 1060, third-party systems 1070, and networks 1010.
[0065] This disclosure contemplates any suitable network 1010. As an example and not by way of limitation, one or more portions of network 1010 may include an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), a portion of the Internet, a portion of the Public Switched Telephone Network (PSTN), a cellular telephone network, or a combination of two or more of these. Network 1010 may include one or more networks 1010.
[0066] Links 1050 may connect client system 1030, social-networking system 1060, and third-party system 1070 to communication network 1010 or to each other. This disclosure contemplates any suitable links 1050. In particular embodiments, one or more links 1050 include one or more wireline (such as for example Digital Subscriber Line (DSL) or Data Over Cable Service Interface Specification (DOCSIS)), wireless (such as for example Wi-Fi or Worldwide Interoperability for Microwave Access (WiMAX)), or optical (such as for example Synchronous Optical Network (SONET) or Synchronous Digital Hierarchy (SDH)) links. In particular embodiments, one or more links 1050 each include an ad hoc network, an intranet, an extranet, a VPN, a LAN, a WLAN, a WAN, a WWAN, a MAN, a portion of the Internet, a portion of the PSTN, a cellular technology-based network, a satellite communications technology-based network, another link 1050, or a combination of two or more such links 1050. Links 1050 need not necessarily be the same throughout network environment 1000. One or more first links 1050 may differ in one or more respects from one or more second links 1050.
[0067] In particular embodiments, client system 1030 may be an electronic device including hardware, software, or embedded logic components or a combination of two or more such components and capable of carrying out the appropriate functionalities implemented or supported by client system 1030. As an example and not by way of limitation, a client system 1030 may include a computer system such as a desktop computer, notebook or laptop computer, netbook, a tablet computer, e-book reader, GPS device, camera, personal digital assistant (PDA), handheld electronic device, cellular telephone, smartphone, augmented/virtual reality device, other suitable electronic device, or any suitable combination thereof. This disclosure contemplates any suitable client systems 1030. A client system 1030 may enable a network user at client system 1030 to access network 1010. A client system 1030 may enable its user to communicate with other users at other client systems 1030.
[0068] In particular embodiments, client system 1030 may include a web browser 1032, and may have one or more add-ons, plug-ins, or other extensions. A user at client system 1030 may enter a Uniform Resource Locator (URL) or other address directing the web browser 1032 to a particular server (such as server 1062, or a server associated with a third-party system 1070), and the web browser 1032 may generate a Hyper Text Transfer Protocol (HTTP) request and communicate the HTTP request to server. The server may accept the HTTP request and communicate to client system 1030 one or more Hyper Text Markup Language (HTML) files responsive to the HTTP request. Client system 1030 may render a webpage based on the HTML files from the server for presentation to the user. This disclosure contemplates any suitable webpage files. As an example and not by way of limitation, webpages may render from HTML files, Extensible Hyper Text Markup Language (XHTML) files, or Extensible Markup Language (XML) files, according to particular needs. Such pages may also execute scripts, combinations of markup language and scripts, and the like. Herein, reference to a webpage encompasses one or more corresponding webpage files (which a browser may use to render the webpage) and vice versa, where appropriate.
[0069] In particular embodiments, social-networking system 1060 may be a network-addressable computing system that can host an online social network. Social-networking system 1060 may generate, store, receive, and send social-networking data, such as, for example, user-profile data, concept-profile data, social-graph information, or other suitable data related to the online social network. Social-networking system 1060 may be accessed by the other components of network environment 1000 either directly or via network 1010. As an example and not by way of limitation, client system 1030 may access social-networking system 1060 using a web browser 1032, or a native application associated with social-networking system 1060 (e.g., a mobile social-networking application, a messaging application, another suitable application, or any combination thereof) either directly or via network 1010. In particular embodiments, social-networking system 1060 may include one or more servers 1062. Each server 1062 may be a unitary server or a distributed server spanning multiple computers or multiple datacenters. Servers 1062 may be of various types, such as, for example and without limitation, web server, news server, mail server, message server, advertising server, file server, application server, exchange server, database server, proxy server, another server suitable for performing functions or processes described herein, or any combination thereof. In particular embodiments, each server 1062 may include hardware, software, or embedded logic components or a combination of two or more such components for carrying out the appropriate functionalities implemented or supported by server 1062. In particular embodiments, social-networking system 1060 may include one or more data stores 1064. Data stores 1064 may be used to store various types of information. In particular embodiments, the information stored in data stores 1064 may be organized according to specific data structures. In particular embodiments, each data store 1064 may be a relational, columnar, correlation, or other suitable database. Although this disclosure describes or illustrates particular types of databases, this disclosure contemplates any suitable types of databases. Particular embodiments may provide interfaces that enable a client system 1030, a social-networking system 1060, or a third-party system 1070 to manage, retrieve, modify, add, or delete, the information stored in data store 1064.
[0070] In particular embodiments, social-networking system 1060 may store one or more social graphs in one or more data stores 1064. In particular embodiments, a social graph may include multiple nodes—which may include multiple user nodes (each corresponding to a particular user) or multiple concept nodes (each corresponding to a particular concept)—and multiple edges connecting the nodes. Social-networking system 1060 may provide users of the online social network the ability to communicate and interact with other users. In particular embodiments, users may join the online social network via social-networking system 1060 and then add connections (e.g., relationships) to a number of other users of social-networking system 1060 to whom they want to be connected. Herein, the term “friend” may refer to any other user of social-networking system 1060 with whom a user has formed a connection, association, or relationship via social-networking system 1060.
[0071] In particular embodiments, social-networking system 1060 may provide users with the ability to take actions on various types of items or objects, supported by social-networking system 1060. As an example and not by way of limitation, the items and objects may include groups or social networks to which users of social-networking system 1060 may belong, events or calendar entries in which a user might be interested, computer-based applications that a user may use, transactions that allow users to buy or sell items via the service, interactions with advertisements that a user may perform, or other suitable items or objects. A user may interact with anything that is capable of being represented in social-networking system 1060 or by an external system of third-party system 1070, which is separate from social-networking system 1060 and coupled to social-networking system 1060 via a network 1010.
[0072] In particular embodiments, social-networking system 1060 may be capable of linking a variety of entities. As an example and not by way of limitation, social-networking system 1060 may enable users to interact with each other as well as receive content from third-party systems 1070 or other entities, or to allow users to interact with these entities through an application programming interfaces (API) or other communication channels.
[0073] In particular embodiments, a third-party system 1070 may include one or more types of servers, one or more data stores, one or more interfaces, including but not limited to APIs, one or more web services, one or more content sources, one or more networks, or any other suitable components, e.g., that servers may communicate with. A third-party system 1070 may be operated by a different entity from an entity operating social-networking system 1060. In particular embodiments, however, social-networking system 1060 and third-party systems 1070 may operate in conjunction with each other to provide social-networking services to users of social-networking system 1060 or third-party systems 1070. In this sense, social-networking system 1060 may provide a platform, or backbone, which other systems, such as third-party systems 1070, may use to provide social-networking services and functionality to users across the Internet.
[0074] In particular embodiments, a third-party system 1070 may include a third-party content object provider. A third-party content object provider may include one or more sources of content objects, which may be communicated to a client system 1030. As an example and not by way of limitation, content objects may include information regarding things or activities of interest to the user, such as, for example, movie show times, movie reviews, restaurant reviews, restaurant menus, product information and reviews, or other suitable information. As another example and not by way of limitation, content objects may include incentive content objects, such as coupons, discount tickets, gift certificates, or other suitable incentive objects.
[0075] In particular embodiments, social-networking system 1060 also includes user-generated content objects, which may enhance a user's interactions with social-networking system 1060. User-generated content may include anything a user can add, upload, send, or “post” to social-networking system 1060. As an example and not by way of limitation, a user communicates posts to social-networking system 1060 from a client system 1030. Posts may include data such as status updates or other textual data, location information, photos, videos, links, music or other similar data or media. Content may also be added to social-networking system 1060 by a third-party through a “communication channel,” such as a newsfeed or stream.
[0076] In particular embodiments, social-networking system 1060 may include a variety of servers, sub-systems, programs, modules, logs, and data stores. In particular embodiments, social-networking system 1060 may include one or more of the following: a web server, action logger, API-request server, relevance-and-ranking engine, content-object classifier, notification controller, action log, third-party-content-object-exposure log, inference module, authorization/privacy server, search module, advertisement-targeting module, user-interface module, user-profile store, connection store, third-party content store, or location store. Social-networking system 1060 may also include suitable components such as network interfaces, security mechanisms, load balancers, failover servers, management-and-network-operations consoles, other suitable components, or any suitable combination thereof. In particular embodiments, social-networking system 1060 may include one or more user-profile stores for storing user profiles. A user profile may include, for example, biographic information, demographic information, behavioral information, social information, or other types of descriptive information, such as work experience, educational history, hobbies or preferences, interests, affinities, or location. Interest information may include interests related to one or more categories. Categories may be general or specific. As an example and not by way of limitation, if a user “likes” an article about a brand of shoes the category may be the brand, or the general category of “shoes” or “clothing.” A connection store may be used for storing connection information about users. The connection information may indicate users who have similar or common work experience, group memberships, hobbies, educational history, or are in any way related or share common attributes. The connection information may also include user-defined connections between different users and content (both internal and external). A web server may be used for linking social-networking system 1060 to one or more client systems 1030 or one or more third-party system 1070 via network 1010. The web server may include a mail server or other messaging functionality for receiving and routing messages between social-networking system 1060 and one or more client systems 1030. An API-request server may allow a third-party system 1070 to access information from social-networking system 1060 by calling one or more APIs. An action logger may be used to receive communications from a web server about a user's actions on or off social-networking system 1060. In conjunction with the action log, a third-party-content-object log may be maintained of user exposures to third-party-content objects. A notification controller may provide information regarding content objects to a client system 1030. Information may be pushed to a client system 1030 as notifications, or information may be pulled from client system 1030 responsive to a request received from client system 1030. Authorization servers may be used to enforce one or more privacy settings of the users of social-networking system 1060. A privacy setting of a user determines how particular information associated with a user can be shared. The authorization server may allow users to opt in to or opt out of having their actions logged by social-networking system 1060 or shared with other systems (e.g., third-party system 1070), such as, for example, by setting appropriate privacy settings. Third-party-content-object stores may be used to store content objects received from third parties, such as a third-party system 1070. Location stores may be used for storing location information received from client systems 1030 associated with users. Advertisement-pricing modules may combine social information, the current time, location information, or other suitable information to provide relevant advertisements, in the form of notifications, to a user.
[0077] FIG. 11 illustrates an example computer system 1100. In particular embodiments, one or more computer systems 1100 perform one or more steps of one or more methods described or illustrated herein. In particular embodiments, one or more computer systems 1100 provide functionality described or illustrated herein. In particular embodiments, software running on one or more computer systems 1100 performs one or more steps of one or more methods described or illustrated herein or provides functionality described or illustrated herein. Particular embodiments include one or more portions of one or more computer systems 1100. Herein, reference to a computer system may encompass a computing device, and vice versa, where appropriate. Moreover, reference to a computer system may encompass one or more computer systems, where appropriate.
[0078] This disclosure contemplates any suitable number of computer systems 1100. This disclosure contemplates computer system 1100 taking any suitable physical form. As example and not by way of limitation, computer system 1100 may be an embedded computer system, a system-on-chip (SOC), a single-board computer system (SBC) (such as, for example, a computer-on-module (COM) or system-on-module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive kiosk, a mainframe, a mesh of computer systems, a mobile telephone, a personal digital assistant (PDA), a server, a tablet computer system, an augmented/virtual reality device, or a combination of two or more of these. Where appropriate, computer system 1100 may include one or more computer systems 1100; be unitary or distributed; span multiple locations; span multiple machines; span multiple data centers; or reside in a cloud, which may include one or more cloud components in one or more networks. Where appropriate, one or more computer systems 1100 may perform without substantial spatial or temporal limitation one or more steps of one or more methods described or illustrated herein. As an example and not by way of limitation, one or more computer systems 1100 may perform in real time or in batch mode one or more steps of one or more methods described or illustrated herein. One or more computer systems 1100 may perform at different times or at different locations one or more steps of one or more methods described or illustrated herein, where appropriate.
[0079] In particular embodiments, computer system 1100 includes a processor 1102, memory 1104, storage 1106, an input/output (I/O) interface 1108, a communication interface 1110, and a bus 1112. Although this disclosure describes and illustrates a particular computer system having a particular number of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable number of any suitable components in any suitable arrangement.
[0080] In particular embodiments, processor 1102 includes hardware for executing instructions, such as those making up a computer program. As an example and not by way of limitation, to execute instructions, processor 1102 may retrieve (or fetch) the instructions from an internal register, an internal cache, memory 1104, or storage 1106; decode and execute them; and then write one or more results to an internal register, an internal cache, memory 1104, or storage 1106. In particular embodiments, processor 1102 may include one or more internal caches for data, instructions, or addresses. This disclosure contemplates processor 1102 including any suitable number of any suitable internal caches, where appropriate. As an example and not by way of limitation, processor 1102 may include one or more instruction caches, one or more data caches, and one or more translation lookaside buffers (TLBs). Instructions in the instruction caches may be copies of instructions in memory 1104 or storage 1106, and the instruction caches may speed up retrieval of those instructions by processor 1102. Data in the data caches may be copies of data in memory 1104 or storage 1106 for instructions executing at processor 1102 to operate on; the results of previous instructions executed at processor 1102 for access by subsequent instructions executing at processor 1102 or for writing to memory 1104 or storage 1106; or other suitable data. The data caches may speed up read or write operations by processor 1102. The TLBs may speed up virtual-address translation for processor 1102. In particular embodiments, processor 1102 may include one or more internal registers for data, instructions, or addresses. This disclosure contemplates processor 1102 including any suitable number of any suitable internal registers, where appropriate. Where appropriate, processor 1102 may include one or more arithmetic logic units (ALUs); be a multi-core processor; or include one or more processors 1102. Although this disclosure describes and illustrates a particular processor, this disclosure contemplates any suitable processor.
[0081] In particular embodiments, memory 1104 includes main memory for storing instructions for processor 1102 to execute or data for processor 1102 to operate on. As an example and not by way of limitation, computer system 1100 may load instructions from storage 1106 or another source (such as, for example, another computer system 1100) to memory 1104. Processor 1102 may then load the instructions from memory 1104 to an internal register or internal cache. To execute the instructions, processor 1102 may retrieve the instructions from the internal register or internal cache and decode them. During or after execution of the instructions, processor 1102 may write one or more results (which may be intermediate or final results) to the internal register or internal cache. Processor 1102 may then write one or more of those results to memory 1104. In particular embodiments, processor 1102 executes only instructions in one or more internal registers or internal caches or in memory 1104 (as opposed to storage 1106 or elsewhere) and operates only on data in one or more internal registers or internal caches or in memory 1104 (as opposed to storage 1106 or elsewhere). One or more memory buses (which may each include an address bus and a data bus) may couple processor 1102 to memory 1104. Bus 1112 may include one or more memory buses, as described below. In particular embodiments, one or more memory management units (MMUs) reside between processor 1102 and memory 1104 and facilitate accesses to memory 1104 requested by processor 1102. In particular embodiments, memory 1104 includes random access memory (RAM). This RAM may be volatile memory, where appropriate. Where appropriate, this RAM may be dynamic RAM (DRAM) or static RAM (SRAM). Moreover, where appropriate, this RAM may be single-ported or multi-ported RAM. This disclosure contemplates any suitable RAM. Memory 1104 may include one or more memories 1104, where appropriate. Although this disclosure describes and illustrates particular memory, this disclosure contemplates any suitable memory.
[0082] In particular embodiments, storage 1106 includes mass storage for data or instructions. As an example and not by way of limitation, storage 1106 may include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disc, a magneto-optical disc, magnetic tape, or a Universal Serial Bus (USB) drive or a combination of two or more of these. Storage 1106 may include removable or non-removable (or fixed) media, where appropriate. Storage 1106 may be internal or external to computer system 1100, where appropriate. In particular embodiments, storage 1106 is non-volatile, solid-state memory. In particular embodiments, storage 1106 includes read-only memory (ROM). Where appropriate, this ROM may be mask-programmed ROM, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), electrically alterable ROM (EAROM), or flash memory or a combination of two or more of these. This disclosure contemplates mass storage 1106 taking any suitable physical form. Storage 1106 may include one or more storage control units facilitating communication between processor 1102 and storage 1106, where appropriate. Where appropriate, storage 1106 may include one or more storages 1106. Although this disclosure describes and illustrates particular storage, this disclosure contemplates any suitable storage.
[0083] In particular embodiments, I/O interface 1108 includes hardware, software, or both, providing one or more interfaces for communication between computer system 1100 and one or more I/O devices. Computer system 1100 may include one or more of these I/O devices, where appropriate. One or more of these I/O devices may enable communication between a person and computer system 1100. As an example and not by way of limitation, an I/O device may include a keyboard, keypad, microphone, monitor, mouse, printer, scanner, speaker, still camera, stylus, tablet, touch screen, trackball, video camera, another suitable I/O device or a combination of two or more of these. An I/O device may include one or more sensors. This disclosure contemplates any suitable I/O devices and any suitable I/O interfaces 1108 for them. Where appropriate, I/O interface 1108 may include one or more device or software drivers enabling processor 1102 to drive one or more of these I/O devices. I/O interface 1108 may include one or more I/O interfaces 1108, where appropriate. Although this disclosure describes and illustrates a particular I/O interface, this disclosure contemplates any suitable I/O interface.
[0084] In particular embodiments, communication interface 1110 includes hardware, software, or both providing one or more interfaces for communication (such as, for example, packet-based communication) between computer system 1100 and one or more other computer systems 1100 or one or more networks. As an example and not by way of limitation, communication interface 1110 may include a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wire-based network or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network, such as a WI-FI network. This disclosure contemplates any suitable network and any suitable communication interface 1110 for it. As an example and not by way of limitation, computer system 1100 may communicate with an ad hoc network, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or one or more portions of the Internet or a combination of two or more of these. One or more portions of one or more of these networks may be wired or wireless. As an example, computer system 1100 may communicate with a wireless PAN (WPAN) (such as, for example, a BLUETOOTH WPAN), a WI-FI network, a WI-MAX network, a cellular telephone network (such as, for example, a Global System for Mobile Communications (GSM) network), or other suitable wireless network or a combination of two or more of these. Computer system 1100 may include any suitable communication interface 1110 for any of these networks, where appropriate. Communication interface 1110 may include one or more communication interfaces 1110, where appropriate. Although this disclosure describes and illustrates a particular communication interface, this disclosure contemplates any suitable communication interface.
[0085] In particular embodiments, bus 1112 includes hardware, software, or both coupling components of computer system 1100 to each other. As an example and not by way of limitation, bus 1112 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a front-side bus (FSB), a HYPERTRANSPORT (HT) interconnect, an Industry Standard Architecture (ISA) bus, an INFINIBAND interconnect, a low-pin-count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCIe) bus, a serial advanced technology attachment (SATA) bus, a Video Electronics Standards Association local (VLB) bus, or another suitable bus or a combination of two or more of these. Bus 1112 may include one or more buses 1112, where appropriate. Although this disclosure describes and illustrates a particular bus, this disclosure contemplates any suitable bus or interconnect.
[0086] Herein, a computer-readable non-transitory storage medium or media may include one or more semiconductor-based or other integrated circuits (ICs) (such, as for example, field-programmable gate arrays (FPGAs) or application-specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard drives (HHDs), optical discs, optical disc drives (ODDs), magneto-optical discs, magneto-optical drives, floppy diskettes, floppy disk drives (FDDs), magnetic tapes, solid-state drives (SSDs), RAM-drives, SECURE DIGITAL cards or drives, any other suitable computer-readable non-transitory storage media, or any suitable combination of two or more of these, where appropriate. A computer-readable non-transitory storage medium may be volatile, non-volatile, or a combination of volatile and non-volatile, where appropriate.
[0087] Herein, “or” is inclusive and not exclusive, unless expressly indicated otherwise or indicated otherwise by context. Therefore, herein, “A or B” means “A, B, or both,” unless expressly indicated otherwise or indicated otherwise by context. Moreover, “and” is both joint and several, unless expressly indicated otherwise or indicated otherwise by context. Therefore, herein, “A and B” means “A and B, jointly or severally,” unless expressly indicated otherwise or indicated otherwise by context.
[0088] The scope of this disclosure encompasses all changes, substitutions, variations, alterations, and modifications to the example embodiments described or illustrated herein that a person having ordinary skill in the art would comprehend. The scope of this disclosure is not limited to the example embodiments described or illustrated herein. Moreover, although this disclosure describes and illustrates respective embodiments herein as including particular components, elements, feature, functions, operations, or steps, any of these embodiments may include any combination or permutation of any of the components, elements, features, functions, operations, or steps described or illustrated anywhere herein that a person having ordinary skill in the art would comprehend. Furthermore, reference in the appended claims to an apparatus or system or a component of an apparatus or system being adapted to, arranged to, capable of, configured to, enabled to, operable to, or operative to perform a particular function encompasses that apparatus, system, component, whether or not it or that particular function is activated, turned on, or unlocked, as long as that apparatus, system, or component is so adapted, arranged, capable, configured, enabled, operable, or operative. Additionally, although this disclosure describes or illustrates particular embodiments as providing particular advantages, particular embodiments may provide none, some, or all of these advantages.
Claims
What is claimed is:
1. A method comprising, by a computing device in communication with a mixed-reality device worn by a user:
detecting an object of interest in a real environment;
determining one or more anchor locations in a mixed-reality environment presented via the mixed-reality device that correspond to a position of the object of interest in the real environment;
determining (i) a pose of a camera when an image is captured and (ii) a viewpoint of the user of the mixed-reality device;
determining a region in the image that, as viewed from the pose of the camera, corresponds to at least one of the one or more anchor locations;
causing display of a first output image based on (1) the viewpoint and (2) the region in the image corresponding to at least one of the one or more anchor locations.
2. The method of claim 1, further comprising determining a horizontal distance of the user from the one or more anchor locations; wherein causing display of the first output image is based on the horizontal distance of the user from the one or more anchor locations exceeding a minimum threshold distance.
3. The method of claim 1, further comprising determining a vertical distance of the user from the one or more anchor locations; wherein causing display of the first output image is based on the vertical distance of the user from the one or more anchor locations exceeding a minimum threshold distance.
4. The method of claim 1, further comprising determining an orientation of the mixed-reality device relative to the one or more anchor locations; wherein causing display the first output image is based on the orientation of the mixed-reality device.
5. The method of claim 1, further comprising:
causing display of a second output image based on (1) another viewpoint and (2) the region in the image corresponding to at least one of the one or more anchor locations.
6. The method of claim 1, wherein the one or more anchor locations are further determined based on one or more tracked motions of a hand of the user.
7. The method of claim 6, wherein the one or more tracked motions include a first coordinate that remains at a fixed position in the real environment, wherein the one or more anchor locations are based on the first coordinate.
8. The method of claim 1, wherein the object of interest in the real environment is detected using a computer vision technique.
9. The method of claim 1, wherein a camera worn by the user is associated with the mixed-reality device worn by the user that blocks the user from seeing the real environment directly.
10. A non-transitory computer-readable medium including executable instructions that, when executed by a computing device that is in communication with mixed-reality device worn by a user, cause the computing device to:
detect an object of interest in a real environment;
determine one or more anchor locations in a mixed-reality environment presented via the mixed-reality device that correspond to a position of the object of interest in the real environment;
determine (i) a pose of a camera when an image is captured and (ii) a viewpoint of the user of the mixed-reality device;
determine a region in the image that, as viewed from the pose of the camera, corresponds to at least one of the one or more anchor locations;
cause display of a first output image based on (1) viewpoint and (2) the region in the image corresponding to at least one of the one or more anchor locations.
11. The non-transitory computer-readable medium of claim 10, wherein the executable instructions further cause the computing device to:
determine a horizontal distance of the user from the one or more anchor locations; wherein causing display of the first output image is based on the horizontal distance of the user from the one or more anchor locations exceeding a minimum threshold distance.
12. The non-transitory computer-readable medium of claim 10, wherein the executable instructions further cause the computing device to:
determine a vertical distance of the user from the one or more anchor locations; wherein causing display of the first output image is based on the vertical distance of the user from the one or more anchor locations exceeding a minimum threshold distance.
13. The non-transitory computer-readable medium of claim 10, wherein the executable instructions further cause the computing device to:
determine an orientation of the mixed-reality device relative to the one or more anchor locations; wherein causing display the first output image is based on the orientation of the mixed-reality device.
14. The non-transitory computer-readable medium of claim 10, wherein the executable instructions further cause the computing device to:
cause display of a second output image based on (1) another viewpoint and (2) the region in the image corresponding to at least one of the one or more anchor locations.
15. The non-transitory computer-readable medium of claim 10, wherein the one or more anchor locations are further determined based on one or more tracked motions of a hand of the user.
16. A system comprising:
a mixed-reality device worn by a user;
one or more processors; and
one or more computer-readable non-transitory storage media coupled to the one or more of processors and including executable instructions that, when executed by the one or more of processors, to cause the system to:
detect an object of interest in a real environment;
determine one or more anchor locations in a mixed-reality environment presented via the mixed-reality device that correspond to a position of the object of interest in the real environment;
determine (i) a pose of a camera when an image is captured and (ii) a viewpoint of the user of the mixed-reality device;
determine a region in the image that, as viewed from the pose of the camera, corresponds to at least one of the one or more anchor locations;
cause display of a first output image based on (1) viewpoint and (2) the region in the image corresponding to at least one of the one or more anchor locations.
17. The system of claim 16, wherein the executable instructions further cause the system to:
determine a horizontal distance of the user from the one or more anchor locations; wherein causing display of the first output image is based on the horizontal distance of the user from the one or more anchor locations exceeding a minimum threshold distance.
18. The system of claim 16, wherein the executable instructions further cause the system to:
determine a vertical distance of the user from the one or more anchor locations; wherein causing display of the first output image is based on the vertical distance of the user from the one or more anchor locations exceeding a minimum threshold distance.
19. The system of claim 16, wherein the executable instructions further cause the system to:
determine an orientation of the mixed-reality device relative to the one or more anchor locations; wherein causing display the first output image is based on the orientation of the mixed-reality device.
20. The system of claim 16, wherein the executable instructions further cause the system to:
cause display of a second output image based on (1) another viewpoint and (2) the region in the image corresponding to at least one of the one or more anchor locations.
Patent Citations (131)
| Patent | Date | Inventor | Cited By |
|---|---|---|---|
| US8724894(B1) | 2014-05-01 | Jensen et al. | Applicant |
| US9002099(B2) | 2015-04-01 | Litvak et al. | Applicant |
| US9344707(B2) | 2016-05-01 | Lee et al. | Applicant |
| US9392264(B1) | 2016-07-01 | Chang | Applicant |
| US9836839(B2) | 2017-12-01 | Champlin et al. | Applicant |
| US10001645(B2) | 2018-06-01 | Norden | Applicant |
| US10124257(B2) | 2018-11-01 | Stafford | Applicant |
| US10261594(B2) | 2019-04-01 | Marcolina et al. | Applicant |
| US10776997(B2) | 2020-09-01 | Schmalstieg et al. | Applicant |
| US10803663(B2) | 2020-10-01 | Wang | Examiner |
| US10817749(B2) | 2020-10-01 | Biswas et al. | Applicant |
| US10861223(B2) | 2020-12-01 | Noris et al. | Applicant |
| US10950034(B1) | 2021-03-01 | Garcia et al. | Applicant |
| US10987579(B1) | 2021-04-01 | Borovikov et al. | Applicant |
| US11037359(B1) | 2021-06-01 | Bleyer et al. | Applicant |
| US11107280(B1) | 2021-08-01 | Clohset et al. | Applicant |
| US11113891(B2) | 2021-09-01 | Noris et al. | Applicant |
| US11200745(B2) | 2021-12-01 | Johnson et al. | Applicant |
| US11210860(B2) | 2021-12-01 | Noris | Applicant |
| US11392211(B2) | 2022-07-01 | Holz | Applicant |
| US11410387(B1) | 2022-08-01 | Alderman et al. | Applicant |
| US11436790(B2) | 2022-09-01 | Noris et al. | Applicant |
| US11451758(B1) | 2022-09-01 | Chaurasia et al. | Applicant |
| US11461962(B1) | 2022-10-01 | Parra Pozo et al. | Applicant |
| US11481960(B2) | 2022-10-01 | Marra | Examiner |
| US11501471(B2) | 2022-11-01 | Imao | Applicant |
| US11501488(B2) | 2022-11-01 | Garcia et al. | Applicant |
| US12033270(B2) | 2024-07-01 | Marra | Examiner |
| US2003/0034974(A1) | 2003-02-01 | Welch et al. | Applicant |
| US2008/0084472(A1) | 2008-04-01 | Trudeau et al. | Applicant |
| US2008/0088529(A1) | 2008-04-01 | Tang | Applicant |
| US2009/0259967(A1) | 2009-10-01 | Davidson et al. | Applicant |
| US2010/0283781(A1) | 2010-11-01 | Kriveshko et al. | Applicant |
| US2011/0298827(A1) | 2011-12-01 | Perez | Applicant |
| US2011/0302293(A1) | 2011-12-01 | Buban | Applicant |
| US2012/0092328(A1) | 2012-04-01 | Flaks et al. | Applicant |
| US2012/0264510(A1) | 2012-10-01 | Wigdor et al. | Applicant |
| US2012/0309516(A1) | 2012-12-01 | Langridge et al. | Applicant |
| US2013/0005467(A1) | 2013-01-01 | Kim | Applicant |
| US2013/0083063(A1) | 2013-04-01 | Geisner et al. | Applicant |
| US2013/0093788(A1) | 2013-04-01 | Liu et al. | Applicant |
| US2013/0201188(A1) | 2013-08-01 | Choi et al. | Applicant |
| US2013/0336550(A1) | 2013-12-01 | Kapur et al. | Applicant |
| US2014/0035901(A1) | 2014-02-01 | Chen et al. | Applicant |
| US2014/0147014(A1) | 2014-05-01 | Mallet et al. | Applicant |
| US2014/0204002(A1) | 2014-07-01 | Bennet et al. | Applicant |
| US2014/0306951(A1) | 2014-10-01 | Toiyama | Applicant |
| US2014/0336461(A1) | 2014-11-01 | Reiter et al. | Applicant |
| US2014/0341464(A1) | 2014-11-01 | Fan et al. | Applicant |
| US2015/0235610(A1) | 2015-08-01 | Miller et al. | Applicant |
| US2015/0249839(A1) | 2015-09-01 | Shimizu et al. | Applicant |
| US2016/0040892(A1) | 2016-02-01 | Wiseman et al. | Applicant |
| US2016/0093108(A1) | 2016-03-01 | Mao et al. | Applicant |
| US2016/0162039(A1) | 2016-06-01 | Eilat et al. | Applicant |
| US2016/0320863(A1) | 2016-11-01 | Shimoda | Applicant |
| US2017/0052507(A1) | 2017-02-01 | Poulos et al. | Applicant |
| US2017/0052595(A1) | 2017-02-01 | Poulos et al. | Applicant |
| US2017/0140552(A1) | 2017-05-01 | Woo et al. | Applicant |
| US2017/0186165(A1) | 2017-06-01 | Taylor et al. | Applicant |
| US2017/0213320(A1) | 2017-07-01 | Yücer et al. | Applicant |
| US2017/0243324(A1) | 2017-08-01 | Mierle et al. | Applicant |
| US2017/0287215(A1) | 2017-10-01 | Lalonde et al. | Applicant |
| US2017/0294030(A1) | 2017-10-01 | Coglitore | Applicant |
| US2017/0309071(A1) | 2017-10-01 | Benko et al. | Applicant |
| US2017/0365100(A1) | 2017-12-01 | Walton | Applicant |
| US2017/0372510(A1) | 2017-12-01 | Ye et al. | Applicant |
| US2018/0067316(A1) | 2018-03-01 | Lee et al. | Applicant |
| US2018/0068488(A1) | 2018-03-01 | Hart et al. | Applicant |
| US2018/0088323(A1) | 2018-03-01 | Bao et al. | Applicant |
| US2018/0173404(A1) | 2018-06-01 | Smith | Applicant |
| US2018/0176483(A1) | 2018-06-01 | Knorr et al. | Applicant |
| US2018/0211398(A1) | 2018-07-01 | Schmidt | Applicant |
| US2018/0232056(A1) | 2018-08-01 | Nigam et al. | Applicant |
| US2018/0286053(A1) | 2018-10-01 | Labbe et al. | Applicant |
| US2018/0288354(A1) | 2018-10-01 | Anderson et al. | Applicant |
| US2018/0299922(A1) | 2018-10-01 | Park | Applicant |
| US2018/0315329(A1) | 2018-11-01 | D'Amato et al. | Applicant |
| US2018/0364801(A1) | 2018-12-01 | Kim et al. | Applicant |
| US2019/0087011(A1) | 2019-03-01 | Kim et al. | Applicant |
| US2019/0096119(A1) | 2019-03-01 | Petkov et al. | Applicant |
| US2019/0101758(A1) | 2019-04-01 | Zhu et al. | Applicant |
| US2019/0197765(A1) | 2019-06-01 | Molyneaux et al. | Applicant |
| US2019/0213789(A1) | 2019-07-01 | Uyyala et al. | Applicant |
| US2019/0220002(A1) | 2019-07-01 | Huang | Examiner |
| US2019/0243448(A1) | 2019-08-01 | Miller et al. | Applicant |
| US2019/0325661(A1) | 2019-10-01 | Baumbach et al. | Applicant |
| US2019/0349648(A1) | 2019-11-01 | Watanabe et al. | Applicant |
| US2019/0362505(A1) | 2019-11-01 | Kobayashi | Applicant |
| US2019/0385372(A1) | 2019-12-01 | Cartwright | Examiner |
| US2019/0391724(A1) | 2019-12-01 | Holz et al. | Applicant |
| US2020/0020166(A1) | 2020-01-01 | Menard et al. | Applicant |
| US2020/0026922(A1) | 2020-01-01 | Pekelny | Examiner |
| US2020/0143584(A1) | 2020-05-01 | Noris et al. | Applicant |
| US2020/0225761(A1) | 2020-07-01 | Sinha et al. | Applicant |
| US2020/0238909(A1) | 2020-07-01 | Yamamoto et al. | Applicant |
| US2020/0280678(A1) | 2020-09-01 | Hariyani et al. | Applicant |
| US2020/0296354(A1) | 2020-09-01 | Bickerstaff et al. | Applicant |
| US2020/0311397(A1) | 2020-10-01 | Sawhney et al. | Applicant |
| US2020/0334877(A1) | 2020-10-01 | Pokorny et al. | Applicant |
| US2020/0334908(A1) | 2020-10-01 | Wilson et al. | Applicant |
| US2020/0342671(A1) | 2020-10-01 | Ishihara | Applicant |
| US2020/0357108(A1) | 2020-11-01 | Zhou et al. | Applicant |
| US2020/0401804(A1) | 2020-12-01 | Grundhoefer et al. | Applicant |
| US2021/0042950(A1) | 2021-02-01 | Wantland et al. | Applicant |
| US2021/0142497(A1) | 2021-05-01 | Pugh et al. | Applicant |
| US2021/0392316(A1) | 2021-12-01 | Bleyer et al. | Applicant |
| US2022/0121292(A1) | 2022-04-01 | Lin | Applicant |
| US2022/0206298(A1) | 2022-06-01 | Goodman | Applicant |
| US2022/0253144(A1) | 2022-08-01 | Hu et al. | Applicant |
| US2022/0262065(A1) | 2022-08-01 | Godbole | Applicant |
| US2023/0037750(A1) | 2023-02-01 | Marra et al. | Applicant |
| CN107533230(A) | 2018-01-01 | Applicant | |
| CN107850936(A) | 2018-03-01 | Applicant | |
| CN110383343(A) | 2019-10-01 | Applicant | |
| EP2660643(A2) | 2013-11-01 | Applicant | |
| EP3163407(A1) | 2017-05-01 | Applicant | |
| GB2376397(A) | 2002-12-01 | Applicant | |
| JP2009123018(A) | 2009-06-01 | Applicant | |
| JP2009140296(A) | 2009-06-01 | Applicant | |
| JP2013050895(A) | 2013-03-01 | Applicant | |
| JP2014026590(A) | 2014-02-01 | Applicant | |
| JP2015118332(A) | 2015-06-01 | Applicant | |
| JP2018109835(A) | 2018-07-01 | Applicant | |
| JP2019516180(A) | 2019-06-01 | Applicant | |
| JP2019527881(A) | 2019-10-01 | Applicant | |
| WO2012135554(A1) | 2012-10-01 | Applicant | |
| WO2015111283(A1) | 2015-07-01 | Applicant | |
| WO2016014873(A1) | 2016-01-01 | Applicant | |
| WO2017079657(A1) | 2017-05-01 | Applicant | |
| WO2019031015(A1) | 2019-02-01 | Applicant | |
| WO2022197825(A1) | 2022-09-01 | Applicant |
Non-Patent Literature (71)
- Ballan; et al., “Unstructured Video-Based Rendering: Interactive Exploration of Casually Captured Videos,” ACM Transactions on Graphics (Proc. SIGGRAPH), Jul. 2010, vol. 29 (4), Article 87, 10 pages.Applicant
- Chaurasia; et al., Silhouette-Aware Warping for Image-Based Rendering, Computer Graphic Forum (Proc. EGSR), Jul. 2011, vol. 30 (4), 11 Pages.Applicant
- Chaurasia; et al.. “Depth Synthesis and Local Warps for Plausible Image-based Navigation,” ACM Transactions on Graphics, Jun. 2013, vol. 32 (3), Article 30, 12 pages.Applicant
- Chen; et al., “View Interpolation for Image Synthesis,” In ACM SIGGRAPH Conference Proceedings, Sep. 1, 1993, pp. 279-288.Applicant
- Chen; “QuickTime VR—An Image-Based Approach to Virtual Environment Navigation,” Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, SIGGRAPH, Sep. 15, 1995, pp. 29-38.Applicant
- Clarke; et al., “A System for a Hand Gesture-Manipulated Virtual Reality Environment,” Sep. 26-28, 2016, pp. 1-10, XP058299882.Applicant
- Corrected Notice of Allowance mailed Mar. 19, 2024 for U.S. Appl. No. 17/972,269, filed Oct. 24, 2022, 2 pages.Applicant
- Fanello; et al., “Low Compute and Fully Parallel Computer Vision with HashMatch,” In the IEEE International Conference on Computer Vision (ICCV), Oct. 2017, 11 pages.Applicant
- Gu; et al., “Application of Motion Vector in Live 3D Object Reconstruction,” Patterns 2011: The Third International Conferences on Pervasive Patterns and Applications, Sep. 25, 2011, pp. 41-46.Applicant
- Hausweisner; et al., “Coherent Image-Based Rendering of Real-World Objects,” Proceedings of the Symposium on Interactive 3D Graphics and games, Association for computing machinery USA, 13D, Feb. 18-20, 2011, pp. 183-190, XP055677618.Applicant
- Hedman; et al., “Casual 3D Photography,” ACM Transactions on Graphics, Nov. 2017, vol. 36 (6), Article 234, pp. 1-15.Applicant
- Hedman; et al., “Instant 3D Photography,” ACM Transactions on Graphics, Aug. 2018, vol. 37 (4), Article 101, pp. 1-12.Applicant
- Hirschmuller; et aL, “Memory Efficient Semi-Global Matching. ISPRS Annals of the Photogrammetry,” Remote Sensing and Spatial Information Sciences, XXII ISPRS Congress, Melbourne, Australia, Aug. 25, 2012-Sep. 12, 2012, vol. 1-3, pp. 371-376.Applicant
- Hirschmuller; et al., “Stereo Processing by Semiglobal Matching and Mutual Information,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Feb. 2008, vol. 30 (2), pp. 328-341.Applicant
- Holynski; et al., “Fast Depth Densification for Occlusion-aware Augmented Reality,” ACM Transactions on Graphics, Nov. 2018, vol. 37 (6), Article 194, pp. 1-11.Applicant
- Hornung; et al., “Interactive Pixel-Accurate Free Viewpoint Rendering from Images with Silhouette Aware Sampling,” Computer Graphics, Dec. 2009, pp. 1-13.Applicant
- International Search Report and Written Opinion for International Application No. PCT/US2020/012238, mailed Mar. 27, 2020, 12 Pages.Applicant
- International Search Report and Written Opinion for International Application No. PCT/US2020/012829, mailed Jun. 26, 2020, 17 Pages.Applicant
- International Search Report and Written Opinion for International Application No. PCT/US2020/067020, mailed Apr. 19, 2021,9 Pages.Applicant
- International Search Report and Written Opinion for International Application No. PCT/US2020/067055, mailed Apr. 21, 2021, 13 Pages.Applicant
- International Search Report and Written Opinion for International Application No. PCT/US2021/065655, mailed Mar. 29, 2022, 13 pages.Applicant
- Kanade; et al., “A Stereo Machine for Video-Rate Dense Depth Mapping and its New Applications,” Proceedings of 15th Computer Vision and Pattern Recognition Conference (CVPR), San Francisco, Jun. 18-20, 1996, 07 pages.Applicant
- Kang; et al., “Extracting View-Dependent Depth Maps from a Collection of Images,” International Journal of Computer Vision, Jul. 2004, vol. 58 (2), pp. 139-163.Applicant
- Khattak; et al., “A Real-Time Reconstructed 3D Environment Augmented with Virtual Objects Rendered with Correct Occlusion,” IEEE Games Media Entertainment, Oct. 22-24, 2014, 8 Pages.Applicant
- Levin; et al., “Colorization using Optimization,” The Hebrew University of Jemsalem, ACM, Inc, Aug. 1, 2004, pp. 689-694.Applicant
- Lipski; et al., “Virtual Video Camera: Image-Based Viewpoint Navigation Through Space and Time,” Computer Graphics, Jan. 1, 2009, 11 pages.Applicant
- Martino; et al., “An Analysis and Implementation of Multigrid Poisson Solvers with Verified Linear Complexity,” Image Processing on Line, 8, Jul. 2018, pp. 192-218.Applicant
- Matzen; et al., “Low-Cost 360 Stereo Photography and Video Capture,” ACM Transactions on Graphics, Jul. 2017, vol. 36 (4), Article 148, pp. 1-13.Applicant
- Mcmillan; et al., “Plenoptic Modeling: An Image-Based Rendering System,” Proceedings of SIGGRAPH 95, Los Angeles, California, Aug. 6-11, 1995, pp. 1-8.Applicant
- Mueller; et al., “Real-Time Hand Tracking under Occlusion from an Egocentric RGB-D Sensor,” IEEE International Conference on Computer Vision, Oct. 2017, 10 pages.Applicant
- Nanni; et al. “Combination of Depth and Texture Descriptors for Gesture Recognition,” Advances in Machine Learning Research, Jun. 2014, 35 pages.Applicant
- Notice of Allowance mailed Mar. 6, 2024 for U.S. Appl. No. 17/972,269, filed Oct. 24, 2022, 8 pages.Applicant
- Notice of Allowance mailed Apr. 17, 2024 for U.S. Appl. No. 17/965,500, filed Oct. 13, 2022, 9 pages.Applicant
- Nover; et al., “ESPRESSo: Efficient Slanted PatchMatch for Real-time Spacetime Stereo,” Google, Sep. 5, 2018, pp. 1-9.Applicant
- Office Action mailed Jan. 9, 2024 for Japanese Patent Application No. 2021-524065, filed on Jan. 3, 2020, 8 pages.Applicant
- Office Action mailed Sep. 10, 2024 for Japanese Patent Application No. 2022-524107, filed on Dec. 24, 2020, 10 pages.Applicant
- Office Action mailed Aug. 13, 2024 for Japanese Patent Application No. 2022-523612, filed on Dec. 25, 2020, 9 pages.Applicant
- Office Action mailed Aug. 13, 2024 for Japanese Patent Application No. 2022-534366, filed on Jan. 8, 2021, 12 pages.Applicant
- Office Action mailed Jun. 27, 2024 for Chinese Application No. 202080008174.6, filed Jan. 3, 2020, 11 pages.Applicant
- Perez; “Poisson Image Editing,” Microsoft Research UK, Jul. 1, 2003, pp. 313-318.Applicant
- Reinert; et al., “Proxy-guided Image-based Rendering for Mobile Devices,” Pacific Graphics, Oct. 27, 2016, vol. 35, No. 7, pp. 353-362.Applicant
- Shum; et al., “Image-Based Rendering,” Springer Science Business Media, LLC, Jan. 2007, 425 Pages.Applicant
- Sinha; et al., “Piecewise Planar Stereo for Image-Based Rendering,” Microsoft, Sep. 29, 2009, pp. 1-8.Applicant
- Stich; “View and Time Interpolation in Image Space,” Pacific Graphics, Oct. 2008, vol. 27 (7), pp. 1-7.Applicant
- Szeliski; “Computer Vision: Algorithms and Applications,” Facebook, Aug. 5, 2010, pp. 1-481.Applicant
- Szeliski; et al., “Locally Adapted Hierarchical Basis Preconditioning,” Microsoft Research, May 2006, 38 pages.Applicant
- Valentin; et al., “Depth from Motion for Smartphone AR,” ACM Trans, Graph, Nov. 2018, vol. 37 (6), Article 193, pp. 1-19.Applicant
- Van Waveren; “The Asynchronous Time Warp for Virtual Reality on Consumer Hardware,” Proceedings of the 22nd ACM Conference on Virtual Reality Software and technology, VRST, ACM Press, New York, USA, Nov. 2-4, 2016, pp. 37-46, XP058306942.Applicant
- Vangorp; et al., “Perception of Perspective Distortions in Image-Based Rendering,” ACM Transactions on Graphics, Jul. 4, 2013, vol. 32 (4), pp. 1-35.Applicant
- Vangorp; et al., “Perception of Visual Artifacts in Image-Based Rendering of Facades,” Computer Graphics Forum, Wiley, Proceedings of the Eurographics Symposium on Rendering, Jul. 8, 2011, vol. 30 (4), pp. 1-11.Applicant
- Wang; et al., “Superpixel-Based Hand Gesture Recognition with Kinect Depth Camera,” IEEE Transactions on Multimedia, Nov. 24, 2014, vol. 17, No. 1, pp. 29-39.Applicant
- Xiao; et aL, “Neuralpassthrough: Learned Real-Time View Synthesis for VR,” Special Interest Group on Computer Graphics and Interactive Techniques Conference Proceedings (SIGGRAPH), Aug. 7-11, 2022, 9 pages, Retrieved from the Internet: URL: https://dl.acm.org/doi/pdf/10.1145/3528233.3530701.Applicant
- Zitnick; et al., “High-Quality Video View Interpolation Using a Layered Representation,” ACM Transactions on Graphics (Proc. SIGGRAPH), Aug. 1, 2004, vol. 23, No. 3, pp. 600-608.Applicant
- Zitnick; et al., “Stereo for Image-Based Rendering using Image Over-Segmentation,” Kluwer Academic Publishers, Jul. 11, 2006, pp. 1-32.Applicant
- Guizilini V., et al., “Learning Optical Flow, Depth, and Scene Flow Without Real-world Labels,” arXiv:2203.15089v2, Jun. 10, 2022, 8 Pages.Applicant
- International Preliminary Report on Patentability for International Application No. PCT/US2023/034854, mailed Apr. 24, 2025, 14 pages.Applicant
- International Preliminary Report on Patentability for International Application No. PCT/US2023/034891, mailed Apr. 24, 2025, 10 pages.Applicant
- International Search Report and Written Opinion for International Application No. PCT/US2023/034854, mailed Jul. 18, 2024, 18 pages.Applicant
- International Search Report and Written Opinion for International Application No. PCT/US2023/034891, mailed Jan. 30, 2024, 13 pages.Applicant
- Office Action mailed Apr. 10, 2025 for Chinese Application No. 202080008174.6, filed Jan. 3, 2020, 1 page.Applicant
- Office Action mailed Mar. 17, 2025 for European Patent Application No. 20845498.3, filed on Dec. 25, 2020, 8 pages.Applicant
- Office Action mailed Sep. 17, 2025 for Chinese Application No. 202080083121.0, filed Dec. 25, 2020, 7 pages.Applicant
- Office Action mailed Apr. 24, 2025 for Korean Application No. 10-2022-7027957, filed Dec. 24, 2020, 5 pages.Applicant
- Office Action mailed Apr. 24, 2025 for Korean Application No. 10-2022-7028616, filed Dec. 25, 2020, 10 pages.Applicant
- Office Action mailed Jun. 27, 2025 for Chinese Application No. 202080083121.0, filed Dec. 25, 2020, 6 pages.Applicant
- Office Action mailed Jun. 27, 2025 for Chinese Application No. 202080086666.7, filed Dec. 24, 2020, 8 pages.Applicant
- Xianfeng L., et al., “Farmland Weed Recognition Method Based on Machine Vision,” Beijing University of Technology, Nov. 2011, 7 Pages.Applicant
- Office Action mailed Jan. 14, 2025 for Chinese Application No. 202080083121.0, filed Dec. 25, 2020, 9 pages.Applicant
- Office Action mailed Jan. 16, 2025 for Chinese Application No. 202080086666.7, filed Dec. 24, 2020, 8 pages.Applicant
- Office Action mailed Jan. 30, 2025 for European Patent Application No. 21851763.9, filed on Dec. 30, 2021, 11 pages.Applicant
- Office Action mailed Nov. 12, 2025 for Korean Application No. 10-2021-7017038, filed Jan. 3, 2020, 7 pages.Applicant