Visualizing space systems modeling using augmented reality

[0001] The present application claims priority to and incorporates by reference for all purposes provisional application 60/423,322 filed Nov. 1, 2002.

[0002] The present invention relates to augmented reality systems. More specifically, the present invention relates to augmented reality for space-based phenomena.

[0003] In the present system, space phenomena includes satellite orbits, satellite coverage area, satellite trajectory (both historical and projected), satellite positioning, and the like. In the present application, the term satellite is used broadly to encompass any item of interest that travels in space, e.g. orbiting satellites, the ISS, natural or man-made space debris, ICBMs, interceptors, laser communications beams, outbound satellites (e.g. Galileo, Pioneer, etc.), or the like.

[0004] In the past, users typically used traditional display technologies and a variety of indirect pointing and/or selection tools to interact with satellite data. Current “three-dimensional” (3D) visualization of space systems modeling is accomplished by presenting graphical representations of models on a two-dimensional (2D) display. Traditional displays included CRTs, LCDs, and the like for displaying the two-dimensional representations of space, satellites, and the like, and, conventional pointing devices included mice, joysticks, trackballs, “space orbs”, roller wheels, and the like. These devices allowed users to interact with items in interest in the view presented to the user.

[0005] The space phenomena data is fundamentally three-dimensional (3D) in nature, and thus not suited for two-dimensional display and manipulation. Accordingly, current satellite visualization systems do not provide a natural or intuitive means to view and comprehend images. Further, current systems do not provide the user with intuitive methods to interact with such images.

[0006] Another drawback is that, collaboration of different systems using current satellite visualizations systems is difficult. For example, presently, users sit in front of their own terminals, and may discuss issues with other users who are all glued to their own terminals. Accordingly, it is very difficult for users to interact with each other, and to refer to the same data, as though they were in the same room.

[0007] In light of the above, what is needed are systems that provide the capability described above with fewer drawbacks.

[0008] The present invention relates to augmented reality systems. More specifically, the present invention relates to augmented reality for space-based phenomena.

[0009] An augmented reality (AR) combines video and computer graphics to produce an image integrated with reality giving the impression that models are physically present in the room with observers. This revolutionary approach for augmented reality visualization has the potential to decrease systems development and analysis time by a factor of two. This would be accomplished by increasing human comprehension and memory at each stage of analysis and design using the spatial cues present in an augmented reality display. Recent DARPA-funded studies using functional Magnetic Resonance Imaging (fMRI) at Carnegie Mellon University and University of Virginia have proved that memory can be improved by 56% using spatial (3D) cues when compared with a conventional computer display. Furthermore, when testing subjects for memory using the MRI, it was shown that the spatial centers of their brains were engaged, in addition to their memory centers, while those that received information on a conventional computer display only used their memory centers. A different study showed that comprehension of complex 3D diagrams could be increased by a factor of 2.2 using the type of kinetic depth cues present in our augmented reality system. In this study, subjects that were presented views of 3D diagrams on a flat computer screen had much more errors in understanding a 3D diagram of given complexity than did the experimental group, which could interactively (real-time) rotate, move, and view the 3D diagram.

[0010] What is proposed is that satellite modeling information can be presented using an augmented reality system for a much more cognitive, effective visualization. The benefit of this improved visualization will result in faster conceptual engineering design, enhanced mission analysis, and aid scientific data interpretation as applied to both Satellite System Design and Analysis, Missile Defense Systems, and the like.

[0011] In embodiments of the present invention, each user is typically fitted with a video camera on a pair of heads-up glasses. The video camera image is used by the system to capture images of a real-world stage. The real-world stage is fitted with special pre-defined visual markers. Using image recognition software, the orientation of the user with respect to the real-world stage can be determined. Next, using a modeling software, the system specifies and constructs three-dimensional images of a view of a virtual world desired. The three-dimensional images are then projected to the user who wears the heads-up glasses. To the user, the net effect is similar to a hologram floating on top of the real-world stage. Additional data may be projected to the user via the heads-up glasses, including satellite data, or other status data, such as time, orientation, longitude and latitude lines, and the like. In other embodiments, the images provide the user with a two-dimensional image.

[0012] In additional embodiments, users can manipulate markers or paddles that include pre-defined visual markers. These paddles are recognized by the system, and thus the position of the paddles within the real-world stage is also determined. This real-world stage position is then translated into the virtual world, allowing the user to interact with the virtual world. This interaction may include selecting satellites, selecting orbits, selecting geographic areas, zooming-in and out of the three dimensional world, and the like.

[0013] Advantages to embodiments of the present invention include that it allows more than one user to interact with the three-dimensional image at the same time. Accordingly, multiple users can collaborate with each other while viewing from different positions of the three-dimensional world. Further, users can see and interact with each other.

[0014] Another advantage is that it allows users to directly select and manipulate objects in the virtual world as though such objects were tangible objects in the real-world. This is especially superior to prior art indirect manipulation methods using mice, space balls, etc.

[0015] Yet another advantage include is that it provides a large virtual display for the user to visualize the space data. In contrast, prior display-based systems were limited to screen size.

[0016] According to one aspect of the invention, a visualization system for a computer system is described. One apparatus includes a positioning portion configured to determine a position of a viewer with respect to a virtual geographic location, and a modeling portion configured to specify the virtual geographic location in response to a three-dimensional model of the virtual geographic location. One system also includes a model specification portion configured to specify a representation of satellite status data in response to the position of the viewer with respect to the virtual geographic location and in response to the satellite status data, and a three-dimensional output portion configured to provide at least two images of the virtual geographic location and the representation of the satellite status data to the viewer in response to the position of the viewer with respect to the virtual geographic location.

[0017] According to another aspect of the invention, a method for a computer system is disclosed. One technique includes determining a position of a viewer with respect to a virtual geographic location, and determining a model of the virtual geographic location in response to a three-dimensional model of the virtual geographic location. Various methods also include determining a representation of satellite status data in response to the position of the viewer with respect to the virtual geographic location and in response to the satellite status data, and displaying to the viewer a three-dimensional representation of the virtual geographic location and the representation of the satellite status data in response to the position of the viewer with respect to the virtual geographic location.

[0018] According to yet another aspect of the invention, a visualization method for a computer system is disclosed. One method includes displaying to a viewer a three-dimensional representation of a virtual location, a representation of the satellite status data as overlays on a physical location. The representation of the virtual location is determined in response to a model of the virtual location, and in response to a position of the viewer with respect to the model of the virtual location. Further, the representation of the satellite data is determined in response to satellite status data, and in response to a position of the viewer with respect to the model of the virtual location.

[0019] According to one embodiment of the invention, a visualization system for a computer system comprises a positioning portion configured to determine a position of a viewer with respect to a real world and a position of the viewer with respect to a virtual world. The positioning portion is also configured to allow the viewer to interact with the virtual world. The system further comprises a modeling portion configured to specify the virtual world in response to a model of the virtual world and a model specification portion configured to specify a representation of satellite status data in response to the position of the viewer with respect to the virtual world and in response to satellite status data. The system also comprises an output portion configured to provide an image of the virtual world super-imposed on an image of the real world. The image of the virtual world includes the representation of the satellite status data to the viewer in response to the position of the viewer with respect to the virtual world.

[0020] According to one aspect of the invention, a method for visualization of augmented reality comprises: determining a position of a viewer with respect to a real world and a position of the viewer with respect to a virtual world; determining a model of the virtual world; determining a representation of satellite status data in response to the position of the viewer with respect to the virtual world and in response to satellite status data; and displaying to the viewer a representation of the virtual world super-imposed on a representation of the real world. The representation of the virtual world includes the representation of the satellite status data in response to the position of the viewer with respect to the virtual world.

[0021] According to another aspect of the invention, a visualization method for a computer system comprises: displaying to a viewer a representation of a real world overlaid with a representation of a virtual world. The representation of the virtual world includes satellite status data. In addition, the representation of the virtual world is determined in response to a model of the virtual world, and in response to a position of the viewer with respect to the virtual world. The representation of the satellite status data is determined in response to satellite status data, and in response to a position of the viewer with respect to the virtual world. Furthermore, the viewer is allowed to interact with the virtual world.

[0022] According to one embodiment of the invention, a visualization system for a computer system comprises a positioning portion configured to determine a position of a viewer with respect a real world and a position of the viewer with respect to a virtual world. The positioning portion is also configured to allow the viewer to interact with the virtual world. The system further comprises a modeling portion configured to specify the virtual world in response to a model of the virtual world and a model specification portion configured to specify a representation of object status data in response to the position of the viewer with respect to the virtual world and in response to the object status data. The system also comprises an output portion configured to provide an image of the virtual world super-imposed on an image of the real world. The image of the virtual world includes the representation of the object status data to the viewer in response to the position of the viewer with respect to the virtual world.

[0029] FIG. 1 is a block diagram of typical computer rendering system 100 according to an embodiment of the present invention.

[0030] In the present embodiment, computer system 100 typically includes a monitor 110, computer 120, a keyboard 130, a user input device 140, a network interface 150, and the like.

[0031] In the present embodiment, user input device 140 is typically embodied as a computer mouse, a trackball, a track pad, wireless remote, and the like. User input device 140 typically allows a user to select objects, icons, text and the like that appear on the monitor 110.

[0032] Embodiments of network interface 150 typically include an Ethernet card, a modem (telephone, satellite, cable, ISDN), (asynchronous) digital subscriber line (DSL) unit, and the like. Network interface 150 are typically coupled to a computer network as shown. In other embodiments, network interface 150 may be physically integrated on the motherboard of computer 120, may be a software program, such as soft DSL, or the like.

[0033] Computer 120 typically includes familiar computer components such as a processor 160, and memory storage devices, such as a random access memory (RAM) 170, disk drives 180, and system bus 190 interconnecting the above components.

[0034] In one embodiment, computer 120 is a PC compatible computer having one or more microprocessors such as Pentium4™ microprocessor from Intel Corporation. Further, in the present embodiment, computer 120 typically includes a Windows™-based operating system.

[0035] RAM 170 and disk drive 180 are examples of tangible media for storage of data, audio/video files, computer programs, applet interpreters or compilers, virtual machines, embodiments of the herein described invention including driver-level software, three-dimensional data of the earth, satellite position data, satellite orbit data, projected trajectories, ground assets, other satellite data, application software such as ARToolKit, Satellite ToolKit (STK), WorldUp, custom software, and the like. Other types of tangible media include floppy disks, removable hard disks, optical storage media such as CD-ROMS and bar codes, semiconductor memories such as flash memories, read-only-memories (ROMS), battery-backed volatile memories, networked storage devices, and the like.

[0036] In the present embodiment, computer system 100 may also include software that enables communications over a network such as the HTTP, TCP/IP, RTP/RTSP protocols, and the like. In alternative embodiments of the present invention, other communications software and transfer protocols may also be used, for example IPX, UDP or the like.

[0037] Also shown in FIG. 1 are an image acquisition sensor 200, and a heads-up projection system 210. In the present embodiment, image acquisition sensor 200 is a video camera mounted upon heads-up projection system 210. The video camera may be monochromatic or color. As will be described further below, image acquisition sensor 200 is used to capture an image of a “real-world” stage including orientation identifiers or markers. The captured image is processed by processors 160 to determine the position of user 220 with respect to the real-world stage, and with respect to the virtual world. In one embodiment, image acquisition sensor 200 is fixed to head-up projection system 210.

[0038] In this embodiment, heads-up projection system 210 is a pair of virtual reality glasses to which processors 160 feed different images to the right lens and the left lens. As is known, user 220 peers through the glasses and receives the different images. User 220 interprets the images as a three-dimensional image of a virtual world. In one embodiment, a Sony Glasstron heads-up display can be used, and in other embodiments, other heads-up-displays can be used.

[0039] As is illustrated, in the present embodiment, user 220 also sees the real-world with the virtual world super-imposed thereon. In embodiments of the present invention, a specialized graphics card may be used to drive heads-up projection system 210. In other embodiments, processors 160 feed identical images to the right and left lenses. Accordingly, the viewer sees a two dimensional image.

[0040] FIG. 1 is representative of computer rendering systems capable of embodying the present invention. It will be readily apparent to one of ordinary skill in the art that many other hardware and software configurations are suitable for use with the present invention. For example, the use of other micro processors are contemplated, such as Xeon™ or Itanium™ microprocessors; Opteron™ or AthlonXP™ microprocessors from Advanced Micro Devices, Inc; PowerPC G3™, G4 microprocessors from Motorola, Inc.; and the like. Further, other types of operating systems are contemplated, such as Windows® operating system such as WindowsXP®, WindowsNT®, or the like from Microsoft Corporation, Solaris from Sun Microsystems, LINUX, UNIX, MAC OS from Apple Computer Corporation, and the like.

[0041] In one embodiment of the present invention, more than one heads-up projection system 210 and image acquisition sensor 200 may be provided per system 100 to allow different users to collaborate. In another embodiment, more than one system 100 may provided and linked by network, to allow different users to collaborate.

[0042] FIG. 2 illustrates a logical block diagram according to an embodiment of the present invention. As discussed above, application software may be provided to system 100. This software may include commercial, off-the-shelf software such as Satellite ToolKit (STK) 300 from Analytical Graphics, Inc. This software is a common tool used for space systems modeling that follows the traditional visualization of 3D graphics on a 2D flat screen display. It presents updates to a scene that generally includes data such as earth rotation, updated locations of any number of objects in the atmosphere and exo-atmosphere around it, and the like. As the scenario advances according to “simulation time” the software updates the objects in the scene according to time. In other embodiments of the present invention, other software may be used for modeling space systems, and the like.

[0043] Other software includes image recognition software 310, such as software based upon ARToolKit, an open source vision tracking library for augmented reality applications. ARToolKit was initially developed at the University of Washington HIT lab. In the present embodiment ARToolKit is used to write image recognition programs that process video images from image acquisition sensor 200, to locate pre-defined markers in the real-world. This data is used to determine the position and/or orientation of user 220 with respect to the markers. The position of user 220 with respect to the virtual world is used as input to the image rendering software. In the present embodiment, the markers may be static, upon a stage, or may be placed upon paddles and moved by user 220. In other embodiments of the present invention, additional image recognition software 310 may be used for the marker tracking and orientation determinations.

[0044] Additional software includes virtual world builder software 320, such as WorldUp, a product used for 3D-scene composition. WorldUp is a product developed by EAI Sense8 to define and develop 3D objects, scenes, and animation. In the present embodiment, WorldUp is used to create VRML models. Using WorldUp, a 3D scene can be defined and developed to include models for the earth, orbiting satellites, and the like. In this embodiment, the geometry and positions of each object modeled are defined relative to the scene origin, using data derived from image recognition software 310. The constructed image is saved as a VRML file to be loaded and viewed using the image rendering software (e.g. TView, described below).

[0045] In other embodiments of the present invention, other software packages may be used for three dimensional scene composition. Further, in other embodiments, other modeling languages may be used than VRML.

[0046] Custom software used in the present embodiment includes TView 330 and VIS 340 software, internally developed by the assignee of the present invention. In the present embodiment, VIS software 340 is a performance, analysis simulation, and modeling tool that provides remote sensing and space science capability. VIS software 340 may be used instead of SDK 300 to provide updates to the simulation model according to simulation time. Tview 330 is a software package that combines video and computer generated imagery to present an augmented view of reality to user 220. In this embodiment, it renders and generates images of the VRML models to user 220.

[0047] In other embodiments of the present invention, other software packages may be used to provide the space science and remote sensing capability, as well as to create images from the VRML data, and the like.

[0048] In the present embodiment, specifying the updates to the virtual world for user 220 is performed by either STK 300 or VIS 340. Both applications provide similar satellite simulation and modeling capabilities and simple methods of data extraction. Both packages provide run time-driven scenario simulations in which object models, such as satellites, etc., in the virtual world are updated according to the simulation time. At each time increment, the positions of the objects are extracted and are provided to TView 330 for display to user 220. Illustrated in FIG. 2 are message passing components 360. In one embodiment of the present invention, satellite simulation and other “live” modeling update data is provided by STK 300 or VIS 340 to TView 330 via message passing client/server components 360. The updated data may include updated satellite position, earth rotation, object orientation, and the like.

[0049] FIG. 3 illustrates a more detailed block diagram of a portion of an embodiment. More specifically, the operation of STK 300 and/or VIS 340 and TView 330 is shown in greater detail.

[0050] In this embodiment, STK 300 and/or VIS 340 run according to a simulation or “virtual world” time. For example, the simulation time may be in the past, or in the future, may run in real-time, slower than real-time, faster than real-time, be static, or the like. In either case, according to the virtual world time 400, STK and/or VIS 340, update the simulation model parameters 410, and pass the updated model parameters 420 via message client 430 to TView 330.

[0051] In response, TView 330, receives the updated model parameters 440, determines the model attributes in the virtual world 450, and updates the images of the virtual world 460.

[0052] In embodiments of the present invention, messages are pre-defined for several object manipulations including some relevant to satellite system movements such as translation, rotation, and scaling, and the like. APIs can be used to implement communications from TView 330 back to STK 300 or VIS 340.

[0053] FIGS. 4A-4B illustrate a flow diagram according to an embodiment of the present invention.

[0054] Initially a model of the virtual reality world is specified and positioned within the real-world, step 500. As an example, in this step a designer may specify that the earth should appear as a one meter diameter sphere resting upon a table (real-world stage), or the like.

[0055] Next, an image of user 220 point of view is captured, step 510. In this example, user 220 may have put-on their integrated heads-up projection system 210 and image acquisition sensor 200 on their head. Next, an image from image acquisition sensor 200 captures one or more frames of video data.

[0056] In the present embodiment, the video data is processed by image recognition software 310 to determine the orientation of user 220 with respect to the real-world stage, step 520. As discussed above, one embodiment of the image recognition software may be based upon ARToolkit and upon one or more pre-defined “markers” positioned in the real-world stage. Continuing the example above, one or more markers may be placed upon the table.

[0057] The positioning data of user 220 to the stage is then input into the rendering software, step 530.

[0058] Next, the virtual world modeling software 320 generates a VRML model of the virtual world, step 540. This VRML model is then provided to the rendering software 330, step 550. Further, in response to the position of user 220, the model specification software updates the virtual model of the world, including updated satellite data, etc., step 560. For example, the software may specify updates to one or more orbital paths, satellite historical and projected trajectory, satellite coverage, the position and/or orientation of the viewer in the virtual world, and the like. The software may also specify updates to satellite status data, and the like. This data is passed to the rendering software 330, using the messaging client/server 360 described above, step 570.

[0059] In the present embodiment, based upon the VRML model and the model specification data, the rendering software 330 generates two separate output images, one for the right eye and one for the left eye of user 220, step 580. The image pair are then output to heads-up display 210, step 590. In response, user 220 sees the real-world stage overlaid with the images generated above, step 600.

[0060] In the present embodiment, the pair of images generated is provided to projection system 210, so user 220 sees a three-dimensional virtual world. In the present embodiment, the viewing points are determined in response to distance between the eyes of user 220, the distance of user 220 to the virtual world, the scale of the image being viewed, and the like. For example, for a view of the earth as a 1 meter high sphere, at 1 meter away from user 220, and the left eye is approximately 2 inches apart from the right eye in the real-world. In this example, the viewing point difference between the two eyes is calculated to be approximately 2500 kilometers. Accordingly, the viewing point difference is input into the rendering software to generate the right eye and the left eye images. In some embodiments, a standardized distance between the eyes of user 220 may be used, and in other embodiments, a customized distance may be used.

[0061] In other embodiments, rendering software 330 may generate a single image that is provided to both to the right eye and to the left eye.

[0062] In the present embodiment, as the user moves around the real-world stage, the process repeats for the updated position of user 220. For example, as the user moves away from a table, an image of the earth may shrink; as the user moves clock-wise with respect to an image of the earth, the user will move “west” in the virtual world; and the like.

[0063] In some embodiments, markers may be placed upon “paddles” in the real-world. As these paddles are moved around the real-world, image recognition software 310 can track the position of the marker in the virtual world. In this way, users 220 can interact with the virtual world. For example, the user may select a satellite in orbit to view its status; the user may target a portion of the earth; the user may direct a satellite to move to a different position; and the like. Many other types of interaction are contemplated in other embodiments of the present invention.

[0064] In the present embodiment, the term “augmented reality” (AR) is used to describe the system, because real-world images are augmented with computer generated images. An example of the advantages of embodiments of the present invention are illustrated in FIG. 2. In FIG. 2, an output image 365 is illustrated. As can be seen, part of the virtual-world can be seen, including a portion of a map 370. Additionally, the real-world can be seen, including a human hand 370 gesturing to a position on map 370 and a chair 390. As can be seen, users can collaborate together in the virtual world in a way much more intuitive than was previously possible.

[0065] In Space Systems applications involving joint-forces war fighting missions, AR visualization could provide the basis of an advanced human-systems interface for achieving and maintaining battlespace awareness and tactical dominance. For example, a detailed 3D geo-spatial visualization of battlespace terrain with location of friendly and enemy forces and their GPS-related assets could provide future GPS satellite operators the information superiority required to provide our theater war fighters with optimal GPS signal coverage and power in the successful execution of time-critical missions.

[0066] Embodiments of the present invention can be extended from satellite system applications into missile defense command and control and training, as illustrated in FIG. 5. For example, software implementing Recognition-Primed Decision models can be incorporated that suggests appropriate solutions for problems based on recognition of the class of problem. Hence, easier recognition of the problem leads to a faster result with the appropriate solution. Military commanders in charge of missile defense would benefit from augmented reality visualization during threat situations. Faster recognition of the situation at hand would allow faster responsive action decisions and faster interceptions of threat vehicles. Training for missile defense command would also be more effective using AR visualization. FIG. 5 depicts STK visualization of a missile defense threat scenario. This type of visualization could also be presented in Augmented Reality to provide improved user comprehension and user collaboration.

[0067] In applications of Missile Defense Systems, AR visualization of tracking algorithms would greatly enhance cluster-tracking abilities. Threat tracking has evolved from reading and interpreting track tables of numbers to 2D paper plotting to 3D flat screen displays. Each evolution has resulted in great strides in the ability to identify and track the real threat. AR visualization would make the next great stride. It would enable faster anomaly identification leading to discrimination algorithm refinement.

[0068] Other embodiments can be applied to Satellite System Design and Analysis, illustrated in FIG. 6. In such embodiments, augmented reality visualization would reduce development time, and improve constellation and mission analysis capabilities. AR could be used for visualization of satellites, constellations, communications links, and satellite performance projections on the earth's surface, as well as other possible applications. Visualization of this information in AR would improve cognition of design issues and help evaluate analysis data.

[0069] In the foregoing specification, the invention has been described with reference to specific exemplary embodiments thereof. In other embodiments of the present invention, different types of satellite data may be represented in augmented reality including current, past, and future positions of the satellite, orientations of the satellite, ground coverage of the satellite, trajectories of the satellite, satellite sensor orientation and position, orientation vectors to other satellites, or objects, missile threat clouds, coverage analysis, when satellites are in a view of a region, satellite revisit time, communication links and networks, beam strength of space and land based laser devices, and the like.

[0070] Many changes or modifications are readily envisioned. In light of the above disclosure, one of ordinary skill in the art would recognize that any number of combinations of visualization techniques, modeling techniques, display techniques, and the like may be used. For example, other sources of three-dimensional modeling can be used, other image recognition software and techniques can be used, or the like.

[0071] Further embodiments can be envisioned to one of ordinary skill in the art after reading this disclosure. In other embodiments, combinations or sub-combinations of the above disclosed invention can be advantageously made. The block diagrams of the architecture and flow charts are grouped for ease of understanding. However it should be understood that combinations of blocks, additions of new blocks, re-arrangement of blocks, and the like are contemplated in alternative embodiments of the present invention.

[0072] The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense. It will, however, be evident that various modifications and changes may be made thereunto without departing from the broader spirit and scope of the invention as set forth in the claims.