Script-based user interface implementation defining components using a text markup language

[0002] The invention relates to systems and methods for implementing user interfaces in television-based systems that also provide Web browsing features.

[0003] In areas relating to user input and output, personal computer hardware has become very standardized. As an example, most personal computers utilize keyboards having identical sets of keys that generate identical electrical signals. Computer mice are another example of this trend. Even where hardware is slightly different, as with video display monitors, most differences are handled transparently by driver software. As a result, developers of application programs can focus their efforts on higher levels of functionality rather than having to deal with the details of hardware interaction.

[0004] With the widespread use of graphical user interfaces such as the Windows is family of operating systems (sold by Microsoft Corporation), the details of user interaction have also become somewhat standardized. The Windows operating system allows developers to utilize pre-written software components to implement user controls and makes possible a unified “look and feel” across a wide range of applications.

[0005] These degrees of standardization have not yet been attained in other technological products that require complex user interaction, such as interactive television products and products that attempt to combine Internet Web browsing with traditional and enhanced television services. In these products, different manufacturers implement different types of input hardware (keyboards, mice, etc.) and wish to design and experiment with different methods of user interaction. Indeed, there is often an attempt by each manufacturer to implement a unique “look and feel,” to differentiate a product from competing products.

[0006] This presents a problem for a company such as Microsoft Corporation that wants to design and sell a single software product such as an operating system and Web browser for use with products manufactured by different companies. The software product must adapt to different hardware for each manufacturer's product, and must also be customizable to implement each manufacturer's unique graphical user interface.

[0007] Normally, customizing a software product for each manufacturer is extremely inefficient from both production and maintenance standpoints. Although the manufacturers could conceivably be allowed to modify a software product's source code to implement a desired interface, this too involves maintenance problems. Furthermore, many such manufacturers are not equipped to perform computer programming at this level.

[0008] The invention utilizes an Internet Web browser architecture that allows the visual components of a user interface to be implemented through a combination of documents or source files written in conventional Internet hypertext formats. In particular, the user interface is implemented using individual HTML (hypertext markup language) components in conjunction with a global script written in a popular scripting language such as JavaScript.

[0009] The global script file is written so that it responds to user input and other events with conventional and extended JavaScript event handlers such as “onKeyDown”, “onClick”, etc. In response to user input, the global script file opens and displays appropriate HTML elements and responds to user selection of such elements. These elements can communicate with each other and with the global script using JavaScript methods and properties. The script file and HTML elements also interact with device hardware in response to defined events.

[0010] The invention utilizes a supervisory application program that runs under an operating system such as Windows CE. The application program implements an extended document object model for use by the global script and by scripts embedded in HTML control elements. The global script runs perpetually, under the supervision of the supervisory application program, as does one or more instances of a conventional Web browser.

[0011] The extended document object model includes methods relating to the creation and maintenance of user interface components, referred to above as HTML elements and also referred to herein as dialogs. These methods accept designations of local HTML files representing user controls. Such HTML files exist and are specified apart from informational content obtained as a result of browsing.

[0012] A particular HTML control can be opened either as a modal dialog or as a modeless dialog. The extended document object model methods allow the dialogs to be opened, closed, resized, moved, and closed. The dialogs act as conventional HTML components, potentially containing event handler scripts that respond to user input events and to other events, including custom events defined by OEMs in conjunction with their own hardware. The HTML controls have access to the extended document object model and to properties and methods implemented within the global script. In addition, the global script has access to methods and properties defined and exposed by script code within the HTML controls. Accordingly, the global script and the various HTML components can communicate and interact with each other to produce a full and flexible user interface.

[0013] Using this architecture, a software manufacturer can sell a single product such as a Web browser to many different manufacturers. As shipped, the product includes a default global script file and associated HTML controls, for implementing a default graphical user interface in conjunction with default hardware. However, the default global script file is easily modified or replaced by customers using well-known Web content creation tools rather than more complex high-level compiled languages such as C or C++. Thus, it is very easy for a customer to create an entirely new user interface, and to adapt it to new hardware, without disturbing more critical underlying aspects of the software product.

[0014] FIG. 1 illustrates a television-based system that incorporates Web browsing features.

[0015] FIG. 2 is a block diagram of a system box.

[0016] FIG. 3 shows software components.

[0017] FIG. 4 shows a browser window and its components.

[0018] FIG. 5 shows an expanded user dialog.

[0019] The World Wide Web (WWW or Web) of the Internet is a body of multimedia content referred to as “hypermedia.” “Hypermedia” is a metaphor for information in which text, images, sounds, and actions are linked together in a complex, non-sequential web of associations that permit a user to “browse” or “navigate” through related topics, regardless of the presented order of the topics. The term “hypermedia” arises from “hypertext,” a term which was originally coined to describe text-based documents including links to other such documents to form a non-sequential web of associated ideas.

[0020] Hypermedia content is commonly organized as individual “documents” or “pages” with embedded control information. A document or page normally corresponds to a single file that is retrievable either locally or through a network. The embedded control information includes formatting specifications, indicating how a document is to be rendered. In addition, such control information can include links or “hyperlinks”: symbols or instructions indicating where to find other related Web documents (files) on the Internet. A hyperlink from one hypermedia topic to another is normally established by the author of a hypermedia document, although some applications allow users to insert hyperlinks to desired topics.

[0021] An application program referred to as a Web browser is used to retrieve and render hypermedia content from the WWW. A Web browser allows a user to browse or navigate through the Web and to thereby find items of specific interest to the user. The terms “browse” and “navigate” indicate a process of following a sequence of hyperlinks through a plurality of different documents or other Web resources.

[0022] In response to activation or selection of a particular hyperlink, a Web browser loads and renders the file or other multimedia content specified by the hyperlink. In accordance with conventional practice, such content typically contains hyperlinks to additional media resources (thus creating the “web” of inter-linked documents or resources). A user can select any such hyperlink and in response the viewing software will load the media resources identified by that hyperlink.

[0023] A hyperlink is typically rendered by a Web browser as a graphical icon or as highlighted keywords. A user “activates” or “follows” a hyperlink by clicking on or otherwise selecting the icon or highlighted keywords. Activating a link causes the Web browser to retrieve and render the document or resource that is targeted by the hyperlink. The target of the hyperlink is specified in the document containing the hyperlink.

[0024] In concept, the target of a hyperlink can be virtually any type of object—including executable programs, text or multimedia documents, sound clips, audio segments, still images, computers, directories, and other hyperlinks. In WWW documents, hyperlink targets are most often files that can reside on any computers connected to the Internet. However, a hyperlink target can also be a particular location within a document, including the document that is currently being rendered.

[0025] Hypermedia content utilized by the WWW is commonly written using what is referred to as a “text markup language.” “SGML” (Standard Generalized Markup Language) is one such language, defined formally as “a language for document representation that formalizes markup and frees it of system and processing dependencies.” SGML is a language for describing the structure of documents and for describing a tagging scheme to delineate that structure within text.

[0026] For creating hypermedia content, WWW documents utilize a specialization of SGML called “HTML” (Hypertext Markup Language). An HTML textual document can be thought of as plain text that contains formatting instructions in the form of HTML markup codes or “tags.” Tags tell Web browsers how to render and print documents, and are also used to specify hyperlinks.

[0027] The primary purpose of HTML is to lay out and format text and images. However, HTML documents can also include executable scripts. A script is a sequence of instructions that potentially include logical decisions, looping, and conditional looping. Scripts are often interpreted at the time of execution rather than being compiled.

[0028] JavaScript is an example of a currently popular scripting language. JavaScript allows a designer to add interactivity to HTML, to allow for user interaction and feedback, multimedia and animation, and to link HTML to other technologies such as Java, ActiveX, and plug-ins.

[0029] To allow interactivity, JavaScript uses so-called “event handlers.” With an event handler, a JavaScript can capture events that occur on a hypermedia document, such as the clicking of a form button or the mouse moving over a link. The event handler can then execute code in response to these actions. As an example, a JavaScript can contain an “onClick” event handler. If the event handler is associated with a button or some other discrete HTML component, clicking on that component will cause execution of the “onClick” event handler.

[0030] JavaScript uses what is referred to as a “document object model.” This is a hierarchy of objects and their properties and methods. Through such properties and methods, a script can access and specify aspects of the Web browser itself, such as status lines, window positions and characteristics, date and time values, and a host of other features. A JavaScript can also expose its own properties and methods for use by other JavaScript components.

[0031] For additional details regarding HTML and JavaScript, refer to R. Stout, The World Wide Web Complete Reference, (Berkeley: Osborne McGraw-Hill, 1996) and N. Heinle, Designing with JavaScript: Creating Dynamic Web Pages, (Sebastopol: Songline Studios, 1997), which are hereby incorporated by reference. The following discussion assumes some familiarity with these topics.

[0032] The invention is described in the environment of a television-based computer with Internet connectivity that has come to be referred to generically as a “Web television” or an “Internet television.” Devices such as this can be constructed in a variety of configurations, such as standalone TVs, as system or set-top boxes that are to be used in conjunction with TVs, or as more traditional computers with television viewing capabilities. These devices combine the network browsing functions of a computer with the television viewing capabilities of traditional televisions. In addition, so-called Internet televisions provide enhanced functionality like interactive programming guides and the linking of hyperlinked Web content to television content.

[0033] FIG. 1 shows one example of television-based computer or browser system 50 within which the invention can be implemented. In this case, the system comprises a conventional television receiver 52 and a system or set-top box 54. Television receiver or display 52 is connected to receive a video/audio signal from system box 54. A keyboard 68 is connected to the keyboard input port of system box 54, preferably by way of a wireless infrared connection. The system box has a connection to the Internet and receives a television signal from a terrestrial source, from a satellite, or from a cable television system.

[0034] FIG. 2 shows the most pertinent components of system box 54. System box 54 is essentially a computer, having a data processor or microprocessor 56, primary memory 58 including volatile and non-volatile memory, optional secondary memory 59 such as a hard disk, floppy disk, or other removable media, various I/O components 60, and other components as are common in personal computers. The pertinent I/O components include a keyboard input port, a video/audio output port, a network interface, and a television signal input port. The network interface connects to a private or public network such as the Internet. The network interface might comprise a telephone-line modem, an Ethernet interface, or some analogous interface. System box 54 also has a television tuner 70 for receiving and demodulating traditional television signals.

[0035] The microprocessor of system box 54 is programmed by means of programs and instructions stored at different times in the different computer-readable storage media of the system box. Programs are typically distributed, for example, on floppy disks or CD-ROMs. From there, they are installed or loaded into the secondary memory of a computer. At execution, they are loaded at least partially into the computer's primary electronic memory. The invention described herein includes these various types of computer-readable storage media when such media contain instructions or programs for implementing the described steps in conjunction with a microprocessor or other data processor. The invention also includes the system or computer itself when programmed according to the methods and techniques described below.

[0036] In the specific embodiment described, programs are stored in and executed directly from non-volatile ROM.

[0037] As shown in FIG. 2, the system includes an operating system 72, one or more application programs 74, and program data 76. For purposes of illustration, programs, program components, and data are shown in FIG. 2 as discrete blocks within system box 54, although it is recognized that such programs and components reside at various times in different storage components of the computer-based system.

[0038] The operating system is preferably a multitasking operating system such as the Windows CE operating system sold by Microsoft Corporation. The operating system employs a graphical user interface in a windowing environment.

[0039] In the described embodiment of the invention, the operating system is not the component that is most visible to a user. Rather, a supervisory computer application program supervises and manages most of the apparent functionality of system 50, such as network browsing, video or television viewing capabilities, and user interface functions. The supervisory application program is configured to startup automatically whenever system 50 is turned on and used.

[0040] FIG. 3 shows such a supervisory application, generally referenced by numeral 100. The application is written in a conventional compiled programming language using well-known Windows programming techniques. Application program 100 runs in conjunction with operating system 72, also shown in FIG. 3. Supervisory application 100 communicates with hardware devices such as the network interface television tuner, keyboard, and display components through conventional Windows operating system calls. In addition, the operating system provides other services in accordance as are commonly provided by modem operating systems.

[0041] Applications 74 include a browser program 104 that is actually one or more instances of the Microsoft Web Browser ActiveX control, available as part of Microsoft's INET Software Developer's Kit. This ActiveX control is a fully functional network hyperlink browser, with capabilities such as those found in Microsoft's Internet Explorer program. It allows a user to browse or navigate through the WWW of the Internet or any similar resources. In this case, however, browser instances are configured to execute without their normal user interface components. User interface components 105 are rendered as HTML content separately from the browsable content, as will be described in more detail below, in a separate browser instance.

[0042] Thus, supervisory application 100 runs on top of windows CE and hosts one or more instances of a browser program that displays browsable content. As will be described in more detail below, the supervisory application also extends the standard object model used by JavaScript and provides event brokering services.

[0043] Particular Web resources can be specified by direct user entry, by selecting a hyperlink, by other application programs through standard program interfaces, or by scripts contained in Web content itself In most cases, a newly-specified resource will replace the previously rendered content. However, it is also possible for new content to be opened in a new window, partially overlying the original window.

[0044] Specified resources can also include ActiveX components or controls. An ActiveX control is an executable program in the form of a DLL (dynamically linked library). ActiveX controls can be downloaded as Web content or obtained locally. ActiveX components optionally expose methods and properties that can be used by scripts associated with Web content. Television or video content is preferably viewed in an ActiveX control. Such a control uses its own browser instance, and runs as a window on top of any other content.

[0045] FIG. 4 shows an example of how a system might be configured to allow Web viewing and access while also allowing the user to simultaneously watch live television. The visual configuration includes three primary components: a user interface component 150, an Internet content window 152, and live television window 154. All three components are implemented as browser instances.

[0046] The Internet component 152 renders Internet Web content. Within this component, the user can move a focus from link to link, scroll the rendered content, and navigate to new destinations, all in accordance with conventional browsing technology.

[0047] The television component 154 displays a live video stream. In this example, this component is displayed in a dedicated window, which is overlaid with the Internet content component. This component does not receive focus or expose any visual components other than the video stream. The television component utilizes an ActiveX control to communicate with TV-tuner hardware which might vary between systems. This control exposes a set of methods and properties to allow scripts to access TV specific functionality and information.

[0048] In FIG. 4, user interface component 150 is shown in a default, minimized state. In this state, the user interface component gives only informational feedback. In this example it displays a variable message in a description area 155, a pair of scroll indicators 156, and a connection status area 158.

[0049] In order to access specific functionality of system 50, the user can activate and expand the displayed user interface component 150 by pressing an assigned key on a remote control or keyboard. The result is shown in FIG. 5: an expanded toolbar 160 containing a plurality of menu items graphically presented as buttons 162. A user can select any of these buttons to perform the indicated functions.

[0050] The user interface components shown in FIGS. 4 and 5 are only examples of the many different types of user interface components that might be implemented in conjunction with a television-based Web device such as described herein. As will be apparent to many, the possibilities are endless and the need for flexibility is great.

[0051] To provide such flexibility in implementing the user interface portion of the device, all functions and components relating to the user interface are implemented using technology that is widely used in conjunction with Internet content. Specifically, user interface controls and dialogs are implemented as HTML files or resources. Furthermore, the overall logic and sequence of the user interface is established by a global script file written in a scripting language such as JavaScript. The script file is specified at run-time, and can be replaced or re-specified during execution to implement different modes of operation. Such a script file is shown in FIG. 3, referenced by numeral 106.

[0052] Script 106 is a JavaScript program designed and configured to implement the user interface of system 50. It contains JavaScript event handlers for execution in response to different events such as user input, browser activities, and a variety of other events that are defined both by supervisory application 100 and by custom ActiveX controls loaded by browsed documents and by the global script file itself. Application program 100 invokes the event handlers in response to defined events. In turn, event handlers and other code within global script 106 can invoke functions or methods provided by application program 100. These functions are invoked through an extended document object model implemented by application program 100.

[0053] Controls and dialogs themselves are implemented as text and graphics documents written in a text markup language such as HTML. In order to present options to a user, global script 106 generates dialog boxes and controls by specifying local HTML graphical elements (in the form of HTML files and their URLs) to supervisory application 100. Supervisory application 100 passes such the specifications to a browser instance, which displays the specified HTML graphical elements in conjunction with whatever material is already being displayed as a result of user browsing.

[0054] The supervisory application program 100 implements an extended document object model for utilization by the global script 106 and by script code within the HTML graphical elements that are specified as user interface components. The extended document object model includes methods relating to the creation and maintenance of user interface graphical elements, which are also referred to as dialogs. These methods accept designations of local HTML files representing user controls.

[0055] A particular HTML component or control can be opened either as a modal dialog or as a modeless dialog. The extended document object model methods allow the dialogs to be opened, resized, moved, and closed. When open, the dialogs act as conventional HTML components, potentially containing event handler scripts that respond to user input and other events. The HTML controls themselves have access to the extended document object model and to properties and methods implemented within the global script. In addition, the global script has access to methods and properties defined by script code within the HTML controls. Accordingly, the global script and the various HTML components can communicate and interact with each other to produce a full and flexible user interface.

[0056] In order to implement the functionality described above, application program 100 supports an extended document object model that is accessible through JavaScript scripts such as the global script and scripts embedded in HTML controls.

[0057] Generally, the following syntax is used to access UI-related methods and properties in this extended model:

[0058] Window.external.app.method( )

[0059] Window.external.app.property( )

[0060] Window.external.dialog.method( )

[0061] Window.external.dialog.property( ).

[0062] The following tables list the pertinent methods and properties related to user interface components:

[0063] The primary methods that the global script uses to generate user interface components are the ShowModalDialog( ), ShowModelessDialog( ), CloseModalDialog( ), and CloseModelessDialog( ) methods, referred to herein as “open” and “close” interface methods.

[0064] Scripts within specified HTML components can access methods and properties of the global script by using the following syntax, where method( ) and property are replaced by the names of the methods and properties exposed by the global script:

[0065] Window.external.script.method( ) Window.external.script.property.

[0066] The standard object model is also extended by defining additional events, corresponding to potential event handlers implemented by global script 106. The following table shows events that may be generated by application program 100 for potential handling by global script 106:

[0067] In addition, keyboard events are defined for certain default keyboard configurations. As an example, a “MenuKey” event might be defined for a dedicated keyboard key labeled “Menu”. Hardware manufacturers can register additional keyboard key events by registering keycodes in the system registry.

[0068] In systems having unique hardware that is not described by the default configuration, an ActiveX control can be prepared and loaded as a non-visible part of a persistent dialog. Such an ActiveX control exposes methods and properties that allow the global script to access features of the unique hardware.

[0069] The following is an example of JavaScript code for creating a new dialog and then working with its methods. This code could be implemented as part of an event handler in the global script, for instance to create a new user interface component in response to a user pressing a specific key.

[0070] Toolbar=Window.external.UI.showModelessDialog(“Toolbar.htm”, 0, 0, 0, 0);

[0071] Window.external.Dialog.Document(Toolbar).all.TitleText.innerText=“WebTV”;

[0072] Window.external.Dialog.Window(Toolbar).scrollTo(0, 0);

[0073] Window.external.Dialog.Position(Toolbar, PosX, PosY, Width, Height);

[0074] Window.external.Dialog.SetFocus(Toolbar);

[0075] Window.external.UI.closeModelessDialog(Toolbar);

[0076] The following is an example of an HTML resource that implements a TVTuner component:

[0077] CONTENT.HTML <OBJECT width=224px height=176px ID=TVTuner style=“position:absolute; LEFT: 60px; TOP: 244px; WIDTH: 224px; HEIGHT: 176px” CLASSID=“clsid:32F4F485-7F80-11 D0-B835-00A0C911837F”></OBJECT>

[0078] When rendered, this resource displays an ActiveX TVTuner control as specified by the class identifier (CLASSID). Assuming that this HTML text is in a file called “tuner.htm”, the global script could open the control with the following JavaScript statement:

[0079] TVWindow=Window.external.UI.showModelessDialog(“tuner.htm”, PosX, PosY, Width, Height);

[0080] A statement such as this could be part of an event handler that opens a TV viewing window in response to a user pressing a specific key. The resulting TVTuner control exposes methods and properties that are accessible from scripts. As an example, the following JavaScript statement sets a variable named “channel” to the value of the current channel number:

[0081] channel=browser.document.all.TVTuner.CurrentChannel( );

[0082] The following JavaScript statement sets the channel to the value contained in variable “channel”:

[0083] browser.document.all.TVTuner.CurrentChannel=(channel);

[0084] Suppose that the extended document object model includes a definition for an event named “OnChannelUp” that takes place when a user presses a key labeled “Channel Up”. The following JavaScript code is an event handler for this event that changes the channel to the next channel:

[0085] Function OnChannelUp( )

[0086] {channel=browser.document.all.TVTuner.CurrentChannel( );

[0087] browser.document.all.TVTuner.CurrentChannel=(channel+1);}

[0088] In accordance with the invention, the global script contains numerous event handlers such as this to both control hardware and to display appropriate graphic elements. In addition, the global script contains whatever logic is necessary to control the interaction of displayed controls with underlying hardware.

[0089] The extended document object model of application program 100 does not provide explicit methods for communicating with OEM-specific hardware or software. When functionality of this kind is required, the global script should be configured to open a specific ActiveX control that is aware of the hardware or software and that exposes the needed methods and/or interfaces for dealing with the hardware or software. This ActiveX control can be used as bridge (generating events if necessary to the global script code) between the OEM hardware of software and the global script.

[0090] The described architecture provides a very flexible way of implementing a user interface, which allows individual hardware manufacturers to customize the user interface in an infinite number of ways. The architecture is also advantageous to the software manufacturer, since a single software product can be marketed to various hardware manufacturers. Customizations are implemented using Web-based techniques that are well within the capabilities of most companies.

[0091] Although the invention has been described in language specific to structural features and/or methodological steps, it is to be understood that the invention defined in the appended claims is not necessarily limited to the specific features or steps described. Rather, the specific features and steps are disclosed as preferred forms of implementing the claimed invention.