Method and system for automatically creating and displaying a customizable three-dimensional graphical user interface (3D GUI) for a computer system

[0001] This application claims the benefit of U.S. Provisional Patent Application, serial No. 60/290,291, filed May 11, 2001.

[0002] This application is filed with a computer program listing on CD-ROM, which is hereby incorporated by reference. The contents of the CD-ROM include:

File Name	File Size	Date	Description
3DNA_Desktop	21 MB	3/5/02	3DNA ATI Branded
ATI_050012.exe			3DNA Desktop
3DNA_Hub	45 MB	4/15/02	3DNA ATI Branded
ATI_010005.exe			Hub World
3DNA	145 MB	5/7/02	3DNA
Desktop_Demo.avi			Demonstration
			Movie

[0003] 1. Field of the Invention

[0004] The present invention relates generally to the application of 3D to Graphical User Interfaces (3D GUI). It takes advantage of a number of technologies to improve the GUI including:

[0005] automatically scanning the user's computer system and using the information to create a customized 3D representation of all the information; procedural generation of 3D geometry, textures, and 3D scenes; dynamic reconfiguration of 3D content; improved organization and viewing of information; a 3D net architecture; and enhanced data visualizations. The technology makes it simpler for the average computer user to create and use 3D environments with which they can interface to their computers and their information, as well as other people and information over the Internet.

[0006] The immediate application of this GUI invention is for personal computers (PCs), but there is additional application to game consoles, set-top computing platforms, mobile devices, virtual reality (VR) devices, and other computing platforms.

[0007] 2. Background of the Invention

[0008] Over 80% of personal computers (PCs) are already capable of supporting 3D computer graphics and nearly all of the over 120 million new desktop computers shipped in 2000 were equipped with 3D graphics chipsets. In addition, there are over 100 million game consoles (such as the Nintendo, Gamecube and Xbox) capable of generating and displaying 3D scenes on a standard television set.

[0009] PC and video games represent the most common application of 3D technology. However, computer aided design (CAD), medical imaging, animation, and video production (using commercial software applications such as Discreet's 3ds and Alias/Wavefront's Maya) represent alternative common applications of 3D technology.

[0010] Perhaps the best illustrations of a 3D GUI for controlling a computer come from fictional examples in Hollywood movies such as Jurassic Park, Disclosure, Lawnmower Man, and Johnny Mnemonic. However, to date, there have been no successful commercial examples of using 3D for the graphical user interface (3D GUI) that the user uses to interact with their computer. Since the introduction of the Macintosh 2D desktop in 1984, there have been few attempts or patents that improve upon this paradigm or take advantage of the pervasive 3D hardware technology now primarily used to play games. The major companies such as Microsoft, Sony, Xerox, IBM, and SGI have contributed to the patent literature with niche applications, but none of these solve the basic technical problems that limit the ability to use 3D for the GUI. There have also been a number of small startup firms claiming to have created a 3D desktop (including: Clockwise Technologies, EiDoxis, and Pentad Resources) but these are simplistic examples of a non-commercially-viable 3D GUI and suffer from the following technical problems that are overcome by the present invention:

[0011] 1) 3D scenes are huge, thus 3D files cannot easily be transferred over the Internet. The only solution is to use low-resolution images that look poor.

[0012] 2) 3D scenes are static, thus a 3D desktop created for one user cannot be used for another since each user will have different applications, files, folders, shortcuts, images and song lists.

[0013] 3) 3D scenes are complex and difficult to create, thus the average person cannot create or modify one without lots of training and an expensive software package. Since the average user will want their 3D desktop to be personalized to their tastes, this represents a large barrier to the pervasive use of a 3D desktop.

[0014] Even though 3D hardware is relatively pervasive, these technical limitations have prevented the pervasive use of a 3D GUI that runs on these hardware systems. Examples of previous attempts to create a 3D graphical user interface and related technologies are described in the following U.S. patents, the specifications of which are hereby incorporated by reference.

Patent	Issued	Inventor(s)	Applicant(s)	Title
5,528,735	June 1996	Strasnick,	Silicon	Method and apparatus for displaying
		Tesler	Graphics Inc.	data within a three-dimensional
				information landscape
5,786,820	July 1998	Robertson	Xerox Corp.	Method and apparatus for increasing
				the displayed detail of a tree structure
5,880,733	March 1999	Horvitz,	Microsoft	Display system and method for
		Sonntag,	Corp.	displaying windows of an operating
		Markley		system to provide a three-dimensional
				workspace for a computer system
5,956,038	September 1999	Rekimoto	Sony Corp.	Three-dimensional virtual reality
				space sharing method and system, an
				information recording medium and
				method, an information transmission
				medium and method, an information
				processing method, a client terminal,
				and a shared server terminal
6,085,256	July 2000	Kitano,	Sony Corp.	Cyber space system for providing a
		Honda,		virtual reality space formed of three
		Takeuchi		dimensional pictures from a server to
				a user via a service provider
6,111,581	August 2000	Berry,	IBM Corp.	Method and system for classifying
		Isensee,		user objects in a three-dimensional
		Roberts		(3D) environment on a display in a
				computer system
6,121,971	September 2000	Berry,	IBM Corp.	Method and system for providing
		Isensee,		visual hierarchy of task groups and
		Roberts,		related viewpoints of a tree
		Bardon		dimensional environment in a display
				of a computer system
6,175,842	January 2001	Kirk,	AT&T Corp	System and method for providing
		Selfridge		dynamic three-dimensional multi-user
				virtual spaces in synchrony with
				hypertext browsing
6,230,116	May 2001	Ronen,	Clockwise	Apparatus and method for interacting
		Amihai	Technologies	with a simulated 3D interface to an
				operating system operative to control
				computer resources

[0015] The present invention allows computer users to create and use a customizable 3D GUI for complete access to their computer's operating system, applications, and files as well as the Internet and on-line web portal functions.

[0016] The software application automatically creates a 3D environment customized to the users preferences, automatically creates the interface to the user's computer with content representative icons, allows simple customization by the end user, and provides tools for the user to interact with the computer. The invention enhances the user experience with a 3D GUI that seamlessly integrates personal desktop, web portal, and data visualization functions in an interactive and compelling 3D environment.

[0017] FIG. 1 illustrates a prior art 2D GUI using a desktop metaphor, as exemplified by the Macintosh operating system from Apple Computer, Inc. and the Windows operating system from Microsoft Corp.

[0018] FIG. 2 illustrates how the use of a 3D GUI can increase the effective display area on a computer monitor.

[0019] FIG. 3 illustrates how the 3D Desktop can be made to look like an office where software applications and operating system functions are represented as realistic 3D icons. Web pages, documents, and other 2D windows and icons can be deployed in the 3D space to improve access to large quantities of information.

[0020] FIG. 4 illustrates how the user can view dozens of browser windows in a single glance, dynamically reconfigure content according to specifications, and seamlessly re-enter the 2D Windows metaphor as desired by simply clicking on any icon.

[0021] FIG. 5 illustrates the concept of procedural generation of textures, scenes, and GUI interfaces.

[0022] A typical configuration for a personal computer system includes a processing unit, a display, a keyboard, and a mouse. The processing unit executes a computer program called the operating system (O/S) which allows the operation of the computer system to be controlled either directly by the user or by other computer programs called applications. A user operates the computer system by entering commands using the keyboard and mouse; in response, the processing unit executes the commands and presents feedback to the user via the display. The portion of the operating system that accepts the user commands and presents feedback to the user is called the user interface.

[0023] Various schemes for implementing the user interface are generally categorized by the manner in which the user interacts with the system. For example, in a typical command line user interface such as DOS (by Microsoft), the user inputs text from a keyboard; in response, the computer system returns text messages to the display. However, in a Graphical User Interface (GUI) such as Windows (by Microsoft) the user can interact with the computer system by manipulating graphical objects on the display screen using the keyboard and/or the mouse and/or other peripheral devices.

[0024] The typical 2D GUI is normally described as a desktop metaphor. The desktop is the background (also called wallpaper if an image of some sort is used as the background) and superimposed onto the desktop are a number of icons and/or rectangular graphical objects called windows. Users can interact with the computer either by working within the window application (e.g. using a word processing application or drawing application inside the window) or by launching new applications by clicking or double-clicking on icons. Another interface to the computer is achieved through the task bar typically located to the bottom of the screen in Microsoft Windows or the top of the screen in a Macintosh operating system.

[0025] The 2D GUI that computer users have become accustomed to since the introduction of the Macintosh in 1984 and later, Microsoft's Windows, is known as the desktop metaphor and is depicted in FIG. 1. This example of a 2D Desktop shows shortcut icons on the left, a couple of application windows, and a Task Bar at the bottom. This metaphor has an inherent limitation. The typical monitor simply does not have the screen area to adequately display, access, and organize the many different applications, files, and sources of data in our wired lives. This is why users often require many overlapping windows, and important functions may be buried under layers of menus and multiple mouse clicks.

[0026] The 3D Desktop or 3D GUI invention significantly increases the real estate of the monitor and allows improved productivity, access to information, user customization, and a superior look and feel. A simple illustration of how 3D can increase the effective display area on a computer monitor is shown in FIG. 2. Multiple 2D Desktops are viewable in a 3D scene, illustrating the ability for a 3D Desktop to increase screen real estate. As can be seen, the user can simply take a virtual step back from their normal 2D desktop and see a number of additional 2D desktops that reside in the 3D space. They can step forward to limit their actions to the standard desktop, or step back to increase the amount of real estate.

[0027] In its simplest form, one of the goals of the present 3D Desktop invention is to achieve the intuitive organization and ease of access that we enjoy in the real world by simulating a complete work environment in a 3D virtual space. For example, as shown in FIG. 3, the 3D Desktop can actually look like an office where software applications and operating system functions are represented as realistic 3D icons (instead of the normal icons used in the standard 2D GUI). Web pages, documents, and other 2D windows and icons can be deployed in the 3D space to improve access to large quantities of information. This example of a 3D Desktop shows representational access to Operating System features such as files accessed via the filing cabinet icons, the calculator program accessed via the calculator on the desk, shortcuts accessed via traditional shortcut icons shown in the 3D space, and Outlook email management software accessed via the icon of a letter and pen. As can also be seen, the user can still click on the traditional 2D Desktop shortcut icons on the left or can click on the Task Bar at the bottom of the screen. An example of seamless integration of live Internet content can be seen by the message Adding a New Dimension to Digital Life streamed off of the Internet in a stock ticker style display.

[0028] Users continue to interact with the computer by clicking on icons or windows, but these icons and windows are no longer constrained to the static real estate of the 2D desktop and can be deployed in a 3D space instead. Navigation throughout the 3D space takes place with the mouse and keyboard in much the same way that most PC games are played (e.g. the arrow key moves you forward and back, left and right . . . ).

[0029] The 3D Desktop application replaces the static wallpaper with a custom 3D environment that includes customized 2D and 3D icons throughout the 3D environment to create a 3D GUI. The user can continue to click on the standard shortcuts or the standard task bar, but now has the alternative to interact with their computer by navigating in the 3D GUI and clicking on icons in the 3D GUI.

[0030] As can be seen, the 3DNA 3D GUI solves the three major technical problems articulated in the Background section of this patent: the 3D space is composed of procedurally generated graphic textures to reduce file size and increase resolution (note the shadows and reflections as an indication of the quality of the graphics); the user can customize every image, every shortcut, and website as well as the colors of the walls (additional authoring tool upgrades allow the ability to change the geometry or create Art Deco, Gothic and other looks and feel), the entire 3D scene can be procedurally generated, and the icons deployed about the space can be automatically arranged. The 3DNA program will scan the computer's hard drive, determine what applications and documents are available, and then procedurally generate a personalized 3D desktop with appropriate icons. In one of the preferred embodiments, there are a variety of different rooms and/or bays, specially designed to group related applications and data together as well as provide sophisticated improvements in productivity and content organization. As can be seen in FIG. 4, the user can view dozens of browser windows in a single glance, dynamically reconfigure content according to specifications, and seamlessly re-enter the 2D Windows metaphor as desired by simply clicking on any icon. Many websites can be seen at a single glance in the 3D Desktop's browser bay. Each What You See Is What You Get or WYSIWYG Icon is a screen capture of the actual website. Clicking on any of the WYSIWYG Icons launches the corresponding web site. (In this example, the traditional 2D Desktop shortcut icons on the left have been turned off in the program.)

[0031] Seamless integration of desktop and online functions means that the weather website information can be represented by the scene outside, today's news can be streamed onto the desktop in a stock ticker, and surfing the Web is as easy as stepping from one room on your 3D desktop into another room accessed over the Internet. The user can access 3D stores, 3D chat rooms with avatars, and explore immersive 3D worlds accessed over the Internet.

[0032] The concept of procedural generation of textures, scenes, and GUI interfaces can be understood with reference to FIG. 5. As can be seen, a traditional image of a 3D block of marble is composed of the 6 polygons that create the block and bit-mapped textures that are painted on the surface. Each bit-mapped texture can be 100 to 500k in size and will become aliased (blocky) when viewed up close. However, by using procedural generation of the textures, the user can simply change 2 values using the slider control and generate all forms of marble without ever being aliased. Thus, rather than sending several hundred bytes of data, the user can transmit simply 2 numbers (plus the cube command) to send the image of a 3D block of marble. The textures can be painted onto any surface such as a cube, sphere, or walls, floors . . . Using this same analogy, the present invention can procedurally generate, not only textures (e.g. marble, wood, sky, carpets, roads . . . ), but also the underlying geometry (rooms, columns, walls, beaches, terrain, trees . . . ), as well as the entire 3D GUI (the scene, icons, content, and links to the users applications).

[0033] As can be seen, the 3DNA invention includes the following features:

[0034] A customizable 3D virtual scene (e.g. a room, building, beach, or world).

[0035] The ability to automatically create the scene (e.g. if there are more files on the user's hard drive, the room will be larger)

[0036] The ability to dynamically alter the scene (e.g. the user can alter the color of the walls, the style of the architecture, or move objects).

[0037] The ability to seamlessly integrate content from the Internet (e.g. the weather outside can be generated based on the actual weather, a stock ticker can be integrated into the scene based on actual stock info . . . )

[0038] The ability to create high quality images due to the use of procedural generation of graphics and scenes.

[0039] The ability to navigate in the scene and interact with the objects in it.

[0040] The ability to dynamically alter the navigation characteristics (e.g. to change your walking speed, fly, teleportation locations . . . )

[0041] The ability for multiple users to interact with the scene and objects at once.

[0042] The ability to interface with the computer, operating system, Internet, files, or underlying data through the icons in 3D space.

[0043] The ability to automatically scan the user's hard drive and create an icon for each program, shortcut, file, favorite website . . .

[0044] The ability for the user to change the icon (e.g. to use a different 3D model of a calculator, change the image used to depict the underlying application or file . . . )

[0045] The ability to automatically create a content-representative icon (a What You See Is What You Get or WYSIWYG Icon) of the underlying file (e.g. the word processing document actually looks like the document, the website icon looks like the actual website, the image icon looks like the actual image . . . )

[0046] The ability for multiple users to interact with the scene and objects at once.