Background of the Invention
The present invention relates to client server computer systems. In particular, the invention relates to a novel approach to sending data, particularly graphics data from a server to a client system.
As computer systems have gained both acceptance and popularity, ease of use has become an issue. In many respects, so-called "user friendly" interfaces have made a difference in their successful deployment. Users now expect that computers will not be merely text based, but that they will include graphical interfaces, as well as graphical applications.
Heretofore, a limitation which has existed in the successful implementation of graphical systems has been that they have been extremely hardware intensive. In particular, in order to advance the art in computer graphics, the standard approach has been to upgrade the hardware--to gain better resolution, to gain faster screen displays, and to otherwise gain faster and better access to the available data.
Up to a point, this "brute force" approach has actually been needed. For example, a high resolution monitor is required in order to display high resolution graphics, and a fast processor is required in order to be able to update images quickly on the screen. However, a problem which remains is that the brute force approach, itself, has limitations. In particular, it is very expensive to continuously upgrade hardware, and unless all aspects of the hardware can be upgraded, there will be limitations on the overall system benefits simply because system performance will always be limited to the weakest link in the system.
These limitations are particularly true in client-server systems, i.e., those systems in which a server containing large amounts of data, is called upon by a client system to provide the data so that the client can render and image. In such systems, the previous approach of sending all of the available data to the client, and letting the client process the data and create a screen image, is simply too inefficient. This inefficiency is further amplified where the server has large amounts of data available for use in generating three-dimensional ("3-D") images on the client system.
Summary of the Invention
In accordance with the preferred embodiment of the invention, the problem of transferring large amounts of graphical data, particularly data associated with 3-D graphical images, from a server to a client has been solved.
The invention is an improved multi-level client server, large model traverser of the type which includes a server system and a client system which are connected together by a data path. In accordance with the invention, the present, improved system includes pruning means within the server system for pruning objects which will not be rendered by the client. The pruning means eliminates those objects which will not be rendered by the client from the data which the server will transmit to the client.
The client contains culling means for eliminating the data associated with objects which will not be rendered by the client. Accordingly, object data which is not needed by the client is pruned by the server system prior to sending data over the data path to the client system, and graphics data which will not be part of the scene is culled by the client system prior to being rendered.
An improved method of rendering graphics scenes on a client system served by a server through a data path is also described. The improvement comprises the steps of storing the data in objects on the server. The object data is stored in a hierarchical tree structure in which objects contained by other objects are arranged in such a manner that they can be pruned in accordance with rules associated with where they are in the hierarchy.
In the data structure of the preferred embodiment of the invention, information is stored as to the position of each data object, its bounding geometry, and its opaque radius. Using this information, various objects whose data has been sent to the client can be culled without rendering by determining whether they would be within the view cone of the observer, and by whether they would be obscured by objects in front of them. In addition, based upon their distance from the observer and upon their size, a level of detail based culling operation which would render small, distant objects at lower detail, or invisible, could be provided. The invention also employs steps which determine whether any part to the bounding box which includes all of an object's data crosses the view frustum of the observer.
Brief Description of the Drawing
In the Drawing:
FIG. 1 is a schematic diagram of a typical client-server system of the type referred to herein;
FIG. 2 is an illustration of a hierarchical data organization of the type used in the present invention;
FIG. 3 is an illustration of the steps taken by the server in response to a client's expressed need to display a scene from a particular location looking in a particular direction;
FIG. 4 is an illustration of the steps which the server takes in conducting its pruning operations;
FIG. 5 is an illustration of the tree-based data structure used by the present invention;
FIGS. 6 and 7 are illustrations of the viewing geometry used in the View Cone Elimination step;
FIG. 8 is an illustration showing how an object can be within (or outside of) the Field of View of another object;
FIG. 9 is an illustration showing how an object can be obscured by a closer object;
FIG. 10 is a flow chart showing the culling steps performed by the client; and
FIG. 11 is an illustration showing how a volume bounding an object is used to determine whether the object will be visible to an observer based upon whether or not any part of the bounding volume is within the frustum which is visible to the observer.
Detailed Description of the Preferred Embodiment of the Invention
In order to understand the present invention, it is helpful to first review the systems of the prior art. With reference, therefore, to FIG. 1, a typical client server system 10 is illustrated. In the system 10, a server 12, which stores data, is able to send that data to a client 14 which has need for the data over a data path 16. Various problems with this approach result from the amount of data which is stored on the server 12, as compared to the amount of data which may actually be needed by the client 14 in order for the client 14 to be able to provide a desired graphics display to a user.
The problems have generally been solved before by the client 14 requesting data from the server 12, the server 12 transferring data over the data path 16 to the client 14, and then having the client 14 determine which of the data actually was to be displayed by the graphics display of the client 14. As the resolution of graphics displays got better, and as servers were able to store more data, the result was that each time a client 14 needed to produce ("render") an image, the data had to be transferred from the server 12 over the data path 16 to the client 14 which had to select the data to be displayed and then display it. With each advance in resolution, further advances were required to transfer and handle the data. This "brute force" approach continued to place additional requirements on data access rates, data transfer rates, and display hardware and software.
Referring to FIG. 2, in accordance with the present invention, a new approach has been developed in which data is represented and stored on the server 12 in a multi-level hierarchy. In particular, data is referred to as objects 18, which are made up of graphics 20, which are, in turn, made up of primitives 22 (such as lines, triangles, and polygons). Accordingly, in accordance with the present invention, the multiple levels of data representation can be used initially by the server 12 to determine if the data relating to a particular object 18 is even needed by the client 14. If a particular object 18 is not needed to satisfy a particular request of the client 14, then it is not necessary for the server 12 to transfer either the graphics 20 or the primitives 22 associated with that data object 18 to the client 14.
If it is possible for the server 12, to make an initial determination that the client 14 would not have a particular object 18 "in view", then no data transfer (with respect to that object 18) is even necessary. Thus, as illustrated in FIG. 3, the server 12 is responsible for performing two steps, namely removing ("pruning") unnecessary objects 24, and transferring to the client 14 that data which is associated with possibly needed objects 26. The first step of the present invention, therefore, is to eliminate ("prune") objects which the server 12 can determine are not needed by the client 14. This step is carried out by a process involving four steps, as illustrated in FIG. 4.
As shown in FIG. 4, the step of pruning unnecessary objects is accomplished by the steps of Object Hierarchy Elimination 28, View Cone Elimination 30, Front-to-Back Object Sorting 32, and the trivial rejection of obscured objects, i.e., the step of Obscured Object Rejection 34. These steps, which all occur at the server 12, will be explained in greater detail hereinafter. However, as will be understood by those skilled in the art, such pruning of object data significantly reduces the amount of data which must be transferred to the client 14, thereby significantly reducing the time required by the client 14 to render a scene. Notwithstanding these benefits, there is no need for higher speed hardware by the client 14. Thus, the "brute force" approach of the prior art is eliminated.
The step of Object Hierarchy Elimination 28 (of FIG. 4) can now be explained as follows. In accordance with the invention, at a high level of data abstraction, an Object is considered to be a container for other Objects. In particular, an Object may be contained by one other Object (its "Parent"), and it may contain other Objects (its "Children"). Each Object can also have "Siblings" (corresponding to the other Children of that Object's Parent). Accordingly, using this abstraction, an Object can be considered to have the following data structure:
In accordance with the foregoing data structure, an Object is considered to have a pointer to its Parent (*Parent), and it may have a pointer to one of its Children (*Children). As each Object in the preferred embodiment of the data structure has pointers forming a linked list, to its Siblings (*Prev and *Next), it is always possible for an Object to find all of its Children, even though it has only a single pointer to one of its Children.
The data structure also indicates that each Object knows where it is in 3-D space (Pos3!), and that it knows both the bounding sphere containing all of its graphics data (Radius) and the radius which it has and which obscures all Objects which are behind the Object (its opaque radius, called "OpaqueRad").
With reference now to FIG. 5, a tree structure implementing the data structure of the present invention is illustrated. By way of example, a graphical object, namely a ball 36, is contained in a cup 38, which is contained in a cabinet 40, which is contained in a room 42, which contained is in a house 44.
The cup 38 has a plate 48 and a bowl 50 as its siblings. Similarly, the cabinet 40 (containing the cup 38) has a table 52 and a chair 54 as its siblings; the room 42 (containing the cabinet 40) has a room 56 as its sibling; and the house 44 (containing the room 42) has houses 58, 60 as its siblings. As illustrated, the houses 44, 58, 60, are all contained in the world 46 (the world 46 being the Object which contains all of the other Objects). While a number of objects are illustrated in FIG. 5, many more could be part of the data on the server 12, as indicated symbolically by the various " . . . more" indicia.
In accordance with the preferred embodiment of the present invention, whenever a client 14 indicates to a server 12 that the client 14 wants to render a scene, the server 12 must provide data to the client 14 which the client 14 needs to render the scene based upon the location of an observer who is looking in a particular direction from a particular point. Accordingly, the present invention assumes that there is an observer 48 who is in the location of the object (the ball 36) which is being viewed.
A textual representation of the tree hierarchy illustrated in FIG. 5 is:
In accordance with the present invention, an observer, such as Observer 48, can see only certain Objects. Consequently, those Objects which the Observer 48 cannot see can be pruned from the data structure (as illustrated at reference 24, FIG. 3). Objects which cannot be pruned, i.e., those which may be visible, are all of its siblings "across", i.e., the ball 36; the children of its siblings "down", of which there are none in FIG. 5; its parent, the cup 38, and its parent's siblings, the plate 48 and the bowl 50; its grandparent, the cabinet 40, and its grandparent's siblings, the table 52 and the chair 54; etc., all the way up to the root of the tree.
In pseudocode:
The foregoing algorithm works well for pruning several kinds of scenes, including standard city-house-room scenes and scenes involving space, i.e., the solar system-planet-moons-satellite system. Using the foregoing algorithm one can prune all unseen objects based simply upon their hierarchy in the tree structure. Accordingly, the foregoing procedure is referred to herein as Object Hierarchy Elimination (step 28 in FIG. 4), and it constitutes the first pruning step performed by the server 12.
Following Object Hierarchy Elimination, the next step in the present invention is called View Cone Elimination (step 30 in FIG. 4). View Cone Elimination, which is the second level of pruning performed by the server 12, involves the trivial rejection of objects which are outside of the view cone of the Observer 48.
As used herein, the term "view cone" is defined by the position of the observer, a view axis, and a field of view ("FOV"). Thus, the definition of the FOV data structure is
As shown above, an object being considered for trivial rejection has a Position (in 3-D space), and it has a Radius (inside which all of its graphics fit) which are defined as part of its data:
Referring to FIG. 6, an illustration of the viewing geometry is shown. In the illustration, the vector ObjDir is computed as the vector difference between the Object's Position, ObjPos, and the Observer's Position, Pos, so:
If the length of the vector ObjDir is less than the radius of the Object, i.e., Radius, then trivial rejection is not possible, since the Observer's location, Pos, is inside the Object, as shown in FIG. 7. Otherwise, trivial rejection may be possible as explained below.
In FIG. 6, the angle .alpha. (corresponding to the angular half-size of object, as viewed by the observer) and the angle .beta. (corresponding to the angular separation of object from the observer's viewing axis) may be computed as follows:
where, ObjDist is the length of the original ObjPos-Pos vector; ObjDir is the ObjPos-Pos vector normalized (to a length of 1); and VDOT is a function which computes the vector dot product of the vectors contained within the parenthesis.
Given .alpha., .beta., and FOV, we can then trivially reject an object if:
As shown in FIG. 8, the Object 62 is outside of the field of view (FOV) of the observer, so it can be trivially rejected based upon the View Cone Elimination step (reference 30 in FIG. 4).
Following View Cone Elimination, Front-to-Back Object Sorting (reference 32 in FIG. 4) is conducted. In order to accomplish Front-to-Back Object Sorting, all objects which have not been rejected are placed into a list which is sorted based upon its distance from the Observer. The sorted list is traversed from front-to-back (closest to furthest from the Observer), with the remaining objects being sent to the Client 14 for rendering, unless they fail the Obscured Object Rejection test described below.
The Obscured Object Rejection test is conducted as follows. In addition to position and radius information, each object's data includes an "opaque radius" number. An object's opaque radius is the radius inside of which the object is completely opaque. As the sorted list is traversed, rendered objects are placed into another data structure designed for fast searches. An example of one such data structure is called a "quadtree" in the view plane. Each new object is compared against the objects which have already been selected to be rendered. The object is then retained for rendering unless an object which has already been rendered (or selected for rendering) obscures it (i.e., It is fully behind a "closer" object which fully obscures it.).
Obscuring is based upon the "opaque radius" of the objects, thus with reference to FIG. 9, the Opaque Field of Object 64 is calculated as follows:
Thus, Obj is completely obscured by Other if, and only if:
With reference to FIG. 9, Object 65 is completely behind, and obscured by, Object 64, so Object 65 can be culled. The traversal of the sorted list can be expressed in pseudocode as follows:
Note that lines a, b, c, and d can be precomputed in AddObjToQuadtree. Additionally, FIRST.sub.-- POSSIBLE.sub.-- OBSCURER (Vec) uses the quadtree to find the first in the quadrant that Vec crosses.
NEXT.sub.-- POSSIBLE.sub.-- OBSCURER returns the next one in that quadrant. Finally, a heuristic can be used to reduce the number of objects in the quadtree. Thus, only objects having at least a certain opaque field or larger need to be stored.
In accordance with the present invention, no further processing occurs at the server 12. That means that this is where the data relating to the remaining objects whose rendering could not be eliminated by the foregoing pruning procedures is sent over the data path 16 to the Client 14, where the data will either be culled or rendered, as illustrated in FIG. 10. In other words, this is where the network transport layer occurs, i.e., where data is actually passed to the client 14 over the data path 16 (See FIG. 1).
In the preferred embodiment of the invention, Objects are each comprised of several graphics which are identified by unique 24-bit IDs, which are sent to the Client 14, along with a 4.times.3 transform matrix per graphic. In accordance with the invention, the client 14 takes this matrix and graphic ID and renders the requested ID based on further culling checks.
The first culling check, which is conducted by the client 14 is called Level-of-Detail Culling. In order to accomplish Level-of-Detail Culling, the client 14 associates each graphic with a 3-D bounding volume. That 3-D bounding volume is transformed by the matrix supplied to the client 14 by the server 12, and its computed on-screen size is used to compute a Level-Of-Detail ("LOD") bit. For example, four LOD bits, called LOD.sub.-- TINY, LOD.sub.-- SMALL, LOD.sub.-- MEDIUM, and LOD.sub.-- LARGE, respectively, are preferably computed. After the appropriate bit is computed, if a particular graphic has that bit set, then it is rendered. Otherwise, it is not rendered. Since Graphic IDs may indicate a group of associated Graphic IDs, large numbers of primitives can be culled using only one check.
Level-of-Detail Culling can be controlled in two ways in accordance with the preferred embodiment of the present invention. First, the designer of objects and graphics can tag the graphics with flags indicating whether the graphic is to be displayed when it is tiny, small, medium, or large on screen. The rendering client 14 will then select the appropriate graphic given the pixel area of the bounding box on-screen.
The second way of controlling Level-of-Detail Culling is more automatic in that the graphic designer designs the graphics at a high resolution, and the client 14 automatically creates the other three versions of the graphic based upon reductions in resolution. For example, a sphere may be designed to be tesselated into a rather high number of rows and columns for the high-resolution graphic (e.g., 64 by 32). The client 14 will then automatically create a 32 by 16 graphic, a 16 by 8 graphic, and an 8 by 4 graphic version of the sphere to use as the medium, small, and tiny versions, respectively. This works with all the graphics types supported by the client 14, including (but not limited to), spheres, torii (doughnuts), cones, cylinders, rings and disks, convex polygons, triangle meshes, quadrilateral meshes, altitude meshes, surfaces of revolution, and extrusions.
The exact on-screen pixel areas which cause the various levels-of-detail to be selected are turnable, and they are subject to experimentation. The manner in which the level of detail is selected and rendered is illustrated by the following pseudocode:
Referring to FIG. 11, the next step, also conducted by the client 14, is called Bounding. In Bounding, there is a volume based trivial reject of graphics. In order to accomplish Bounding, the transformed bounding volume used in Level-of-Detail Culling is reused here. As illustrated in FIG. 11, the view frustum 66 is bound by six planes, a bottom 68, a left side 70, a right side 72, a front 74, and a back 76, and a top side 78.
The sides of the transformed view volume 80 are defined by eight vertices 82, 84, 86, 88, 90, 92, 94, 96, as shown. If each of the vertices 82, 84, 86, 88, 90, 92, 94, 96 of the transformed view volume 80 lie outside any of the six bounding planes 68, 70, 72, 74, 76, 78 of the view frustum 66, then the graphic is culled. In determining whether any of the vertices 82, 84, 86, 88, 90, 92, 94, 96, lies within the view frustum 66, each of the planes 68, 70, 72, 74, 76, 78 of the view frustum 66 is considered in turn. Again, the grouping construct can eliminate large numbers of primitives with a single check.
In considering the transformed view volume 80, shown in FIG. 11, it can readily be determined that all eight of its vertices lie to the right of the right side 72 of the view frustum 66. Therefore, the graphic defined within the view volume 80 is not visible, so it is culled.
At this point, the remaining (unpruned and unculled) graphics are converted to primitives, and they are rendered by the client 14. In the preferred embodiment of the invention, the graphics are preferably rendered through a graphics API, such as Hewlett-Packard's Starbase or OpenGL. The graphics API performs the final two levels of pruning/culling--namely, backface culling and view clipping. As these later two levels are part of the graphics API, those skilled in the art will recognize how they operate.
As will be recognized by those skilled in the art, the order of the Level-of Detail Culling and Bounding steps may be swapped without affecting greatly the performance of the present inventive method.
As will be recognized by those skilled in the art, alternative methods for View-Cone Elimination and Obscured Object Rejection can be used. Thus, while a method for using spheres to do View Cone Elimination and Obscured Object Rejection was described, any convex polyhedron could also be used without departing from the current invention. Thus, by way of example, using a bounding box.
As illustrated, View Cone Elimination becomes View Frustum Elimination. Again, just like for graphics (on the client side), the transformed vertices of the bounding box are used to determine whether they all lie on the outside of one of the six planes of the view frustum. This is easily done with a simple plane test, as is well known to those skilled in the computer graphics art.
In order to perform Obscured Object Rejection, in pseudocode, one can perform the following procedure:
As described above, the opaque area and the bounding volume can be an arbitrary convex polyhedron. In the general case this is accomplished by substituting the number of vertices for "8" in the inner loop, and substituting a RayPolyhedronlntersect call for RayBoxintersect. Of course, with sufficiently complex polyhedra, more time then needed could be used by the server 12 doing obscuration testing. Accordingly, in practice, a bounding box is efficient and sufficient.