System and method for multiple node display

[0001] This application claims the benefit of provisional application Ser. No. 60/474,313, filed May 30, 2003, the disclosure of which is hereby incorporated by reference in its entirely.

[0002] The present invention relates to zooming user interfaces (ZUI) for computers.

[0003] Most present day graphical computer user interfaces are designed using visual components of a fixed spacial scale. The visual content can be manipulated by zooming in or out or otherwise navigating through it. However, the precision with which coordinates of various objects can be represented is extremely limited by the number of bits, usually between 16 and 64, designated to represent such coordinates. Because of their limited representational size, there is limited precision.

[0004] In the context of the zooming user interface, the user is easily able to zoom in, causing the area which previously covered only a single pixel to fill the entire display. Conversely, the user may zoom out, causing the contents of the entire display to shrink to the size of a single pixel. Since each zoom in or out may multiply or divide the xy coordinates by numerous orders of magnitude, just a few such zooms completely exhaust the precision available with a 64 bit floating point number, for example. Thereafter, round-off causes noticeable degradation of image quality.

[0005] It is an object of the present invention to provide a ZUI in which a larger range of zooms is possible.

[0006] It is a further object of the invention to provide a ZUI in which the precision in which coordinates are represented is related to the required precision needed at a particular zoom level of detail. It is a further object of the present invention to allow a pannable and zoomable two-dimensional space of a finite physical size, but of an arbitrarily high complexity or resolution, to be embedded into a well-defined area of a larger pannable and zoomable two-dimensional space.

[0007] A further objective of the present invention is to allow zooming out after a deep zoom-in to behave like the “back” button of a web browser, letting the user retrace his or her steps through a visual navigation.

[0008] A further objective of the present invention is to allow zooming in immediately after zooming out to behave analogously to the “forward” button of a web browser, letting the user precisely undo the effects of an arbitrarily long zoom-out.

[0009] A further objective of the present invention is to allow a node, a visual object as defined more precisely below, to have a very large number of child nodes (for example, up to 10^28).

[0010] A further objective of the present invention is to allow a node to generate its own children programmatically on the fly, enabling content to be defined, created or modified dynamically during navigation.

[0011] A further objective of the present invention is to enable near-immediate viewing of arbitrarily complex visual content, even if this content is ultimately represented using a very large amount of data, and even if the data are stored at a remote location and shared over a low-bandwidth network.

[0012] A further objective of the present invention is to allow the user to zoom arbitrarily far in on visual content while maintaining interactive frame rates.

[0013] A further objective of the present invention is to allow the user to zoom arbitrarily far out to get an overview of complex visual content, in the process both preserving the overall appearance of the content and maintaining interactive frame rates.

[0014] These and other broader objectives of the present invention will become apparent to those skilled in the art from a review of the specification that follows.

[0015] The above and other objects of the present invention are accomplished by displaying visual content as plural “nodes.” Each node preferably has its own coordinate system and rendering method, but may be contained within a parent node, and may be represented in the coordinate system and rendering method of the parent node. As a user navigates the visual content, by for example, zooming in or out, a node is only “launched” when the zooming results in an appropriate level of detail. The launching of the node causes the node to be represented in its own coordinate system and/or rendering method, rather than in the coordinate system and/or rendering method of a different node.

[0016] Prior to the node being launched, the node is either represented in the coordinate system of the parent node, or not represented at all. By launching nodes only when they are required, the precision of a coordinate system is a function of the zoom level of detail of what is being displayed. This allows a variable level of precision, up to and including the maximum permissible by the memory of the computer in which the system operates.

[0024] We assume a user interface metaphor in which the display is a camera, through which the user can view part of a two-dimensional surface, or 2D universe. For convenience, although it is not necessary to do so, we ascribe physical dimensions to this universe, so that it may be, for example, one meter square. The invention is equally applicable to N-dimensional representations.

[0025] The exemplary universe in turn contains 2D objects, or nodes, which have a visual representation, and may also be dynamic or interactive (i.e. video clips, applications, editable text documents, CAD drawings, or still images). For a node to be visible it must be associated with a rendering method, which is able to draw it in whole or in part on some area of the display. Each node is also endowed with a local coordinate system of finite precision. For illustrative purposes, we assume a node is rectangular and represented by a local coordinate system.

[0026] These two parameters, the rendering method and coordinate system, specify how to display the node, and the positions of items in the node. Each node may have 0 or more child nodes, which are addressed by reference. The node need not, and generally does not, contain all the information of each child node, but instead only an address providing information necessary to obtain the child node. As a user navigates, for example, zooms in and out, the nodes are displayed on the screen, as shown, for example in FIG. 1.

[0027] Generally, a “node” is the basic unit of functionality in the present invention. Most nodes manifest visually on the user's display during navigation, and some nodes may also be animated and/or respond to user input. Nodes are hierarchical, in that a node may contain child nodes. The containing node is then called a parent node. When a parent node contains a child node, the child's visual manifestation is also contained within the parent's visual manifestation. Each node has a logical coordinate system, such that the entire extent of the node is contained within an exemplary rectangle defined in this logical coordinate system; e.g. a node may define a logical coordinate system such that it is contained in the rectangle (0,0)–(100,100).

[0028] Each node may have the following data defining its properties:

• • [0029] the node's logical coordinate system, including its logical size (100×100 in the above example);
  • [0030] the identities, positions and sizes of any child nodes, specified in the (parent) node's logical coordinates;
  • [0031] optionally, any necessary user data;
  • [0032] executable code defining these operations or “methods”:
  • [0033] initialization of the node's data based on “construction arguments”
  • [0034] rendering all or a portion of the node's visual appearance (the output of this method is a rendered tile);
  • [0035] optionally, responding to user input, such as keyboard or mouse events.

[0036] The executable code defines a “node class”, and may be shared among many “node instances”. Node instances differ in their data content. Hence a node class might define the logic needed to render a JPEG image. The “construction arguments” given to the initialization code would then include the URL of the JPEG image to display. A node displaying a particular image would be an instance of the JPEG node class. Plural instances of a node may be viewable in the same visual content, similar to the way a software application may be instantiated numerous times simultaneously.

[0037] Note that in a complex visual document or application, it is usually possible to divide the necessary functionality into nodes in many different ways. For example, a scripted web-page-like document containing multiple images, pull-down menus and buttons could be implemented as a single node with complex rendering and user input methods. Alternatively, it could be implemented as a parent node which only defines the overall layout of the page, with every constituent image and button a child node. This has the obvious advantage of reusing or “factoring” the functionality more effectively: the buttons may all have the same behavior, and hence all be instances of the same node class; the images may all be in the same format and so also be instances of a common node class, etc. This also simplifies rearranging the layout—the parent node can easily move or resize the child nodes.

[0038] In accordance with the present invention, visual content may be displayed in a manner that depends upon the state of navigation input by a user. For example, FIG. 1 shows a node 105 which may be the image of a portion of the city. Node 105 may contain child nodes 101–103. Node 101 may be an image of a building in the city, node 102 could be an image of a playground, and node 103 might be a sports arena. At the level of zoom shown, nodes 101–103 are relatively small, so they can be represented as a small darkened area with no detail in node 105, located at the correct location in the coordinate system of node 105. Only the coordinate system and the rendering method of node 105 is needed.

[0039] Consider the case where the user now zooms in so that a different level of detail (LOD) such as that shown in FIG. 2 results. In the LOD of FIG. 2, nodes 101 and 102 are no longer visible on the screen, due to the fact that the visual content is displayed as much larger. Additionally, it is noted that the because the size at which sports arena node 103 is displayed is now much larger, the details of the sports arena, such as the individual seats, the field, etc, now must be displayed.

[0040] In furtherance of the foregoing, sports arena node 103 would now be displayed not as a darkened area with no detail in the coordinate system of node 105, but rather, it would be “launched” to be displayed using its own coordinate system and rendering method. When displayed using its own coordinate system and rendering method, the details such as seating, the filed of play, etc. would be individually shown. Other functions discussed above, and associated with the node 103, would also begin executing at the point when node 103 is launched. The particular navigation condition that causes the launching of node 103, or any node for that matter, is a function of design choice and is not critical to the present invention.

[0041] The precision with which the node 103 will be displayed is the combined precision of the coordinate system utilized by node 105, as well as that of node 103. Thus, for example, if the coordinate system of each of said nodes utilizes 8 bits, then the combined precision will be 16 bits because the coordinate system of node 103 is only utilized to specify the position of items in node 103, but the overall location of node 103 within node 105 is specified within the coordinate system of node 105. Note that this nesting may continue repeatedly if sports arena 103 itself contains additional nodes within it. For example, one such node 201 may in fact be a particular concession stand within the sports arena. It is represented without much detail in the coordinate system and rendering method of node 103. As a user continues zooming in on sports arena 103, at some point node 201 will launch. If it is displayed using 8 bits of precision, those 8 bits will specify where within the node 201 coordinate system particular items are to be displayed. Yet, the location of node 201 within node 103 will be maintained to 8 bits of precision within the coordinate system of node 103, the location of which will in turn be maintained within the coordinate system of node 105 using 8 bits. Hence, items within node 201 will ultimately be displayed using 24 bits of precision.

[0042] By nesting nodes within nodes, the precision at which visual content may ultimately be displayed is limited only by the memory capacity of the computer. The ultimate precision with which visual content in a node is displayed after that node is launched is effectively the combined precision of all parent nodes and the precision of the node that has launched. Hence, depending upon the level of nesting, the precision may increase as needed limited only by the storage capacity of the computer, which is almost always much more than sufficient. Additionally, the increased precision is only utilized when necessary, because if the image is at an LOD that does not require launching, then in accordance with the above description, it will only be displayed with the precision of the node within which it is contained if that node has been launched. Thus, for nodes nested within other nodes, as one moves from the outermost node inward, one may traverse nodes that have launched until finally reaching a node that has not launched yet. Any such unlaunched node, and nodes further within it, will be displayed only with the precision of the last traversed node that has launched.

[0043] This results in an “accordion” type precision, wherein the precision at which visual content is displayed expands and contracts as necessary and as dictated by the navigational input of the user, maximizing the efficiency of system resources by using them only when necessary for higher precision.

[0044] It is also noted, that when a node launches the display of that node changes from being based upon the coordinates and rendering method of the parent node to the coordinates and rendering method of the child node. That change is optimally made gradual through the use of blending, as described, for example, in copending U.S. patent application Ser. No. 10/790,253. However, other methodologies of gradually changing from the display of the information in the coordinate system and rendering method the parent node to the child node are possible. The system could be programmed, for example, that over a particular range, the blending from parent to child occurs. Then, as the user traverses through that range during a zoom, the changeover occurs, unless the navigation is ceased during that range, in which case the blending may continue until fully displayed in the appropriate coordinate system.

[0045] An additional issue solved by the present invention relates to a system for maintaining the spatial interrelationship among all nodes during display. More particularly, during dynamic navigation such as zooming and panning, many different coordinate systems are being used to display potentially different nodes. Some nodes, as explained above, are being displayed merely as an image in the coordinate system of other nodes, and some are being displayed in their own coordinate systems. Indeed, the entire visual display may be populated with nodes displayed at different positions in different coordinate systems, and the coordinate systems and precisions used for the various nodes may vary during navigation as nodes are launched. Hence, it is important to ensure that the nodes are properly located with respect to each other, because each node is only knowledgeable of its own coordinate system. The present invention provides a technique for propagating relative location information among all of the nodes and for updating that information when needed so that each node will “know” the proper position in the overall view at which it should render itself.

[0046] The foregoing may be accomplished with the addition of a field to the node structure and an additional address stack data structure. The expanded node definition includes a field which we term the “view” field, and which is used by the node to locate itself relative to the entire display. The view field represents, in the coordinates of that node, the visible area of the node—that is, the image of the display rectangle in the node's coordinates. This rectangle may only partially overlap the node's area, as when the node is partially off-screen. Clearly the view field cannot always be kept updated for every node, as we cannot necessarily traverse the entire directed graph of nodes in real time as navigation occurs.

[0047] The stack structure is defined thus:

• • [0048] Stack<Address>viewStack;
  • [0049] where this stack is a global variable of the client (the computer connected to the display). For exemplary purposes we assume that navigation begins with an overview of a universe of content, defined by a root node; then this root node is pushed onto the viewStack, and the root node's view field might be initialized to be the entire area of the root node, i.e.
  • [0050] rootNode.view=rootNode.coordSystem;
  • [0051] Push(viewStack, rootNode);

[0052] Schematically, the viewStack will specify the addresses of a sequence of nodes “pierced” by a point relative to the display, which we will take in our exemplary implementation to be the center of the display. This sequence must begin with the root node, but may be infinite, and therefore must be truncated. In an exemplary embodiment, the sequence is truncated when the nodes “pierced” become smaller than some minimum size, defined as minimumArea. The current view is then represented by the view fields of all of the nodes in the viewStack, each of which specify the current view in terms of the node's local coordinate system. If a user has zoomed very deeply into a universe, then the detailed location of the display will be given most precisely by the view field of the last node in the stack. The last element's view field does not, however, specify the user's viewpoint relative to the entire universe, but only relative to its local coordinates. The view field of the root node, on the other hand, does specify where in the universe the user is looking. Nodes closer to the “fine end” of the viewStack thus specify the view position with increasing precision, but relative to progressively smaller areas in the universe. This is shown conceptually in FIG. 3, where it can be seen that of the three nodes 301, 302, and 303 that have been launched, node 303 provides the most accurate indication of where the user is looking, since its coordinate system is the “finest”, but node 301 provides information, albeit not as fine, on a much larger area of the visual content.

[0053] The problem then reduces to the following: the views (i.e. view fields) of all visible nodes must be kept synchronized as the user navigates through the universe, panning and zooming. Failure to keep them synchronized would result in the appearance of nodes moving on the display independently of each other, rather than behaving as a cohesive and physically consistent 2D surface.

[0054] Changing the view during any navigation operation proceeds as follows. Because the last node in the viewStack has the most precise representation of the view, the first step is to alter the view field of this last node; this altered view is taken to be the correct new view, and any other visible nodes must follow along. The second step is to propagate the new view “upward” toward the root node, which entails making progressively smaller and smaller changes to the view fields of nodes earlier in the stack. If the user is deeply zoomed, then at some point in the upward propagation the alteration to the view may be so small that it ceases to be accurately representable; upward propagation stops at this node. At each stage of the upward propagation, the change is also propagated downward to other visible nodes. Hence, first, the last node's parent's view is modified; then, in the downward propagation, the last node's “siblings”. The next upward propagation modified the grandparent's view, and the second downward propagation modifies first uncles, then first cousins. The downward propagation is halted, as before, when the areas of “cousin nodes” become smaller than minimumArea, or when a node falls entirely offscreen.

[0055] The foregoing technique involves translating the layout of the various nodes into a tree, which conceptually is illustrated in FIGS. 4 and 5. As can be seen from FIGS. 4 and 5, there is a corresponding tree for a particular displayed set of nodes, and the tree structure may be used to propagate the view information as previously described. Each of the nodes 401 through 408 corresponds to a point on the tree with a similar number shown in FIG. 5. Shown conceptually in FIG. 3 is the information that would be stored with respect to a node, such as the pointer and the information indicative of the size and location of another node, for example. FIG. 6 is a block diagram corresponding to a portion of the tree diagram of FIG. 5, in which exemplary values are employed to illustrate the data that may be used to define the properties of node 403. In this exemplary case, node 403 includes a pointer to child node 406, information about the logical size (100×100) of child node 406, and data specifying the location of child node 406 in the coordinates of node 403: 10, 20.

[0056] A panning operation may move the last node far enough away that it no longer belongs in the viewStack. Alternatively, zooming in might enlarge a child to the extent that a lengthening of the viewStack is required, or zooming out might bring the last node's area below a minimum area requiring a truncation of the viewStack. In all of these cases the identity of the last node changes. These situations are detected during the downward propagation, which may alter the viewStack accordingly, potentially leaving it longer or shorter.

[0057] One simple case of the foregoing is that during zooming, a node gets launched so that now it needs to be placed in the view stack. Another example is that by zooming out, a previously visible node becomes so small that it must be removed from the viewStack.

[0058] An extension of the idea is to avoid truncating the viewStack immediately in response to a long outward zoom. Truncating the viewStack is only necessary if the user then pans. Although a long outward zoom will cause the view fields of deeply zoomed nodes to grow very large (and therefore numerically inaccurate), a field

• • [0059] Point2D viewCenter;

[0060] can be added to the Node structure, representing the central point of the view rectangle; zooming without panning therefore does not alter the viewCenter field of any node. This construction allows zooming far outward to be followed immediately by zooming back in. Because the viewStack has been left intact, the user can then return to precisely the starting view. This behavior is analogous to the “back” and “forward” buttons of a web browser: “back” is analogous to zooming out, and “forward” is analogous to zooming back in. In a web browser, if a user uses “back” to return to a previous web page, but then follows an alternate link, it is at this point that “forward” ceases to work. Following an alternate link is thus analogous to panning after zooming out.

[0061] The foregoing provides that visual content may be displayed and navigated in a variety of fashions with substantially infinite precision, limited only by the capacity of the computer system on which the application is running. The visual content displayed at any given time is then displayed as an assembly of nodes, wherein only the nodes needed for the particular view have been launched, and all other nodes are displayed without launching as part of another node or not at all. It is understood that various other embodiments will be apparent to those of ordinary skill in the art, and that the invention is not limited to the embodiments described herein.