Methods and apparatus for low overhead enhancement of web page and markup language presentations

[0001] This application claims the benefit of priority of U.S. Provisional Patent Application Ser. No. 60/368,950, filed Mar. 29, 2002, entitled “Methods and Apparatus for Low Overhead Enhancement of Web Page and Markup Language Presentations,” the teachings of which are incorporated herein by reference.

[0002] The invention pertains to markup language presentation and, particularly, to the enhancement of web page presentations by web browsers. The invention has application (by way of non-limiting example) in facilitating use of web pages, as well as in training aids and accessibility aids, whether delivered over networks or operating on stand-alone computers.

[0003] Though the computer was invented over forty years ago, use of the ubiquitous device remains a challenge for most people. Bound paper manuals gave way to on-line help as the computer went mainstream in the early 1980's. Software publishers have since struggled to make context-sensitive help screens that are comprehensive yet simple, thereby meeting the demands of novices and seasoned users alike. Both vendor and client have provided help-desk attendants which provided sources of assistance but at the increased cost, and often inefficient utilization, of staffing resources.

[0004] Responding to the needs of sight-impaired users presents a host of additional problems. Basic program screens and help displays may need to be magnified, visually revamped or played in audio. The size of the quality assurance task may, consequently, double.

[0005] These problems are further compounded by the decreased use of stand-alone applications and the corresponding increased reliance on remote applications that “execute” via users' web browsers. Solutions to many of the aforementioned problems applicable to local applications often prove inapplicable to remotely executed ones.

[0006] An object of this invention is to provide improved methods and apparatus for enhancement of web page and other markup language presentations, e.g., by browsers.

[0007] A more particular object is to provide such methods and apparatus as facilitate web page use and as can be applied as software training aids and accessibility aids for Internet- and other computer network-based applications, as well as for stand-alone local software applications.

[0008] A further object is to provide such methods and apparatus as can be readily implemented at low cost, and without consumption of undue processor or memory resources.

[0009] The foregoing are among the objects attained by the invention which provides, in one aspect, a method of enhancing presentation of a web page on a browser of the type that generates a display or other presentation on the basis of HTML (or other markup language) elements in the page. The method includes encoding, within the web page page, identification (referred to below as a “tag”) of at least one selected markup language element the display of which is to be enhanced. The method further includes encoding, within the page, functionality (referred to below as a “player”) that enhances presentation of the selected markup language element on a basis of the identification. In a method according to this aspect of the invention, the encoding of the identification of the element to be enhanced is separate from the encoding of the functionality that performs the enhancement.

[0010] According to related aspects of the invention, the step of encoding the identification (or “tag”) includes encoding an enhancement itself and/or a reference thereto. Encoding the enhancement can include directly encoding help text, graphics or other information to be presented to by the browser in conjunction with the markup language element of interest, optionally, with formatting indicating how the enhancement is to be presented.

[0011] Encoding a reference to an enhancement, according to further related aspects of the invention, can include encoding a unique (or non-unique) I.D. for the selected markup language element or the web page on which it resides. The I.D. is associated, for example, with any of help text, graphics and other information that is maintained, for example, in a store (e.g., an XML database) accessible by the player or otherwise by the browser.

[0012] Further aspects of the invention provide methods as described above in which the player is encoded in the web page as a script, applet or a control. The script can be, for example, a Visual Basic or JavaScript script. The applet can be a Java applet. The control can be, for example, an ActiveX control. Regardless, a player according to related aspects of the invention is encoded with functionality (e.g., by way of script or programming instructions) to present the help text, graphics or other information associated with the selected markup element or referenced thereby.

[0013] Further aspects of the invention provide methods as described above in which the step of encoding the identification of the markup element includes encoding a non-conventional attribute in an otherwise conventional markup language element. A conventional markup element is an element specified, for example, in a standard such as that promulgated by the World Wide Web (W³) consortium or other standards association. A non-conventional attribute is, for example, one that does not interfere with browser processing of a conventional element in which it (the non-conventional attribute) is included and, preferably, which is parsed by the browser for inclusion in the internal representation of the page (i.e., the document object model or DOM), yet, is not otherwise interpreted by the browser.

[0014] Yet still further aspects of the invention provide methods of enhancing presentation of a web page through inclusion of a player as described above. Further related aspects provide methods of operating such a player.

[0015] Further aspects of the invention provide methods for enhanced web page presentation in which a web page described above is presented by a web browser executing on a digital data processor. The player, executing as an applet, a control or a separate executable module, is automatically or selectively actuated and, thereafter, monitors elements identified, e.g., by way of the aforementioned encoded identifications. When a user takes action with respect to such an element (sometimes referred to below as a “user interface object”), the player modifies the web page presentation in accord with the enhancement associated with the element. According to related aspects of the invention, the player can modify the web page presentation under a variey of other circumstances, e.g., highlighting a user interface object that the user is to next act upon, highlighting all elements on a page which the user is to act upon, generating a pop-up display indicating a sequence of elements to act upon, and so forth.

[0016] In related aspects of the invention, enhancement can include emphasizing (by color, typefont, graphics or otherwise) the user interface object. In further related aspects, the player modifies the presentation to include audio or audio-visual components, all as specified directly or indirectly by the encoded identification. In still other related aspects of the invention, the player enhances the presentation by executing a script or effecting a state machine, e.g., providing wizard-like operation.

[0017] These and other aspects of the invention are evident in the drawings and in the text that follows.

[0022] FIG. 1 depicts a digital data processing system, according to one practice of the invention, comprising a server 12 and a client computer 18 coupled via an intranet, extranet, Internet or other network or communications medium that supports HTTP or other communications consistent with the teachings below and otherwise in the conventional manner known in the art. A web browser 16 operates within the client 18 (or is otherwise in conjunction therewith) and is in communication coupling with server 12, also in the conventional manner known in the art.

[0023] The server and client computers may be of the conventional type known in the art that serve and access HTML pages, respectively. The server 12 may, for example, be a mainframe computer, workstation or other digital data processing apparatus suitable for serving web pages. In addition to disk drives (not shown) that store the pages themselves, the server may have an associated database 24 that contains text, graphics, scripts or other information for enhancing web page presentation, as discussed below. The client computer 18 may, for example, be a personal computer or other digital data processing apparatus providing a platform suitable for executing a browser that accesses for display or other presentation, e.g., audio, tactile or otherwise (collectively, “display” or “presentation”) the web pages. The server 12, client 18 and web browser 16 operate in the conventional manner known in the art as modified in accord with the teachings below to enhance web page display. Browser 16 represents a conventional web browser of the type commercially available, e.g., under tradenames Internet Explorer™, Netscape Navigator, Mozilla, and so forth. More generally, the term browser refers to software or other functionality suitable for presenting markup language documents.

[0024] Though the illustrated embodiment utilizes a networked client-server configuration, those skilled in the art will appreciate that the invention can be practiced in other digital data processing configurations. Non-limiting examples include a stand-alone computer, e.g., with a browser that accesses web pages stored on hard disk, DVD, CD, hand-held device or other medium. Moreover, though the illustrated embodiment is intended to facilitate web page presentation with respect to HTML web pages, those skilled in the art will appreciate that the teachings herein can be applied to displays or other presentations of other markup languages, as well, e.g., DHTML, XML, to name a few.

[0025] The illustrated embodiment enhances web page presentation through use of (i) web pages that include non-conventional, HTML-like attributes or “tags” that are placed in conjunction with conventional HTML elements whose presentation is to be enhanced, and (ii) a script, program, other software or functionality (collectively, “player”) that enhances the browser presentation of the conventional HTML elements based on the tags. The conventional HTML elements may correspond to any aspect of the web page presentation that is to be enhanced though, in typical embodiments, these are elements that cause the browser to present information to the user and/or accept user input. In addition to elements specified by the World Wide Web (W³) consortium and other standards bodies, the conventional HTML elements include applet, object, ShockWave-Flash™, QuickTime™, RealAudio™, SVG and other known extensions.

[0026] The tags may take the general form of conventional HTML attributes, though with labels not used under currently-known HTML standards. The tags may be substantially self-contained insofar as they contain text, graphics, or other information from which alone the player 20 can enhance the web page presentation. Such is the case, for example, of tags that contain simple help text, perhaps, along with directives specifying how that text is to be displayed. Upon invocation (e.g., in response to a user help request), the player can present that text in connection with the conventional HTML element with which it is used. Alternatively, or in addition, the tags may provide (indirect) reference to text, graphics, or other information with which the presentation is to be enhanced. Thus, for example, a tag may provide an I.D. (associated with an HTML element or a web page) or other identifying information which can be used, for example, by the player to look up and/or request from the server 12 (or other source) information from which the presentation can be enhanced.

[0027] The term “tag” as used herein refers to attributes (whether conventional or non-conventional) in conventional HTML elements, which attributes are used by the player as direct or indirect identifiers of enhancements that are to be made in the web page presentation. In some embodiments, the tags may comprise entire HTML elements and not merely attributes. In any event, however, “tag” refers to attributes (or elements) used by the player for web page presentation enhancement in accord with the teachings herein and should not be confused with the more conventional use of the word “tag” merely as a synonym for HTML element.

[0028] Preferred tags are those which do not affect browser presentation of the web pages' conventional HTML elements. Still more preferably, such tags are of a form that are (i) parsed by the browser for inclusion in a document object model (“DOM”) in conjunction with the conventional HTML elements with which those tags are placed, yet, (ii) are not otherwise interpreted by the browser.

[0029] Instead of, or in addition to, using tags to identify HTML elements of the web page that are to be enhanced, some embodiments of the invention include functions, tables, stores, lists, controls or other invocable or addressable mechanisms to identify such HTML elements. For example, according to one preferred practice, web pages that are to be enhanced in accord with the invention include a JavaScript function that, when invoked, returns a list of identifiers for the HTML elements to be enhanced. Such a function, which can obviate the use of tags, may include a “hard-coded” list of identifiers or may effect a database or table look-up to identify such identifiers. Provided with the list, the player can then search the HTML document or DOM to find the HTML elements to be enhanced.

[0030] Tags or their substitutes (as discussed in the immediately preceding paragraph) can be encoded within a web page in a variety of ways. One is through hand-coding. Another is through use of functionality which monitors exemplary uses of the web page and allows an operator to identify HTML elements to be enhanced. Such functionality can record the operator's input in response to such elements, recording text, graphics, or other information to be displayed to an end-user in response to his or her help or other requests. A preferred general example of such a methodology (albeit, not one applied in the context of web page presentation enhancement) is that utilized in the developer module of the product OnDemand Personal Navigator™, commercially available from the assignee hereof. In conjunction with either the hand-coding or recorder-based techniques, a grammar-based methodology can be used to identify UIs for enhancement. Such a methodology is disclosed in commonly owned U.S. patent application Ser. No. 09/925,613, filed Aug. 9, 2001 titled “Methods and Apparatus for Grammer-Based Recognition of User-Interface Objects in HTML Application,” incorporated herein by reference.

[0031] With further reference to FIG. 1, there is depicted, by way of non-limiting example, an embodiment of the invention used to enhance a conventional form-based web page to include wizard-like text messages, dialog boxes and so forth to guide a user through the form.

[0032] At the outset, the server 12 “serves” or otherwise uploads HTML data streams or web pages over a network or other remote communications device. See, step A. This can be an intranet, extranet, Internet or other network that supports HTML applications, as noted above. The HTML streams or web pages can represent fill-in forms (e.g., a mortgage application as will be described herein), a simple display (e.g., a map or instruction sheet), a combination of the foregoing (e.g., a catalog), or a program, such as a server-based word processor, database system, or any other “application” capable of operating in a client-server and other web-based environment.

[0033] As noted above, a web page downloaded by the server contains conventional HTML elements, as well as tags of the type described above. In addition, the web page can include JavaScript, VisualBasic, activeX, XSLT or other controls or other functionality embodying the above noted player 20—though, in other embodiments the player 20 is implemented as a stand-alone executable (e.g., that runs “along side” the browser) or browser-resident module of the type commonly referred to as a plug-in.

[0034] In preferred embodiments, the player is a “zero-footprint” application (e.g., a small application that does not require pre-installation) that includes relatively little native code (or script) but, rather, that relies on information encoded within the tags, or in databases or other stores to which the tags refer, in order to enhance the web page presentation. While a generic player can potentially be used with all web pages (or at least all web pages that abide by the same tagging scheme), the player can also be customized for use with specific web pages or portions thereof. One such player implementation is presented in the Appendix attached hereto.

[0035] Turning now to the client browser 16, per convention it receives the HTML from the server 12 and parses it to generate the document object model, or DOM 22. See, step B. This is an internal representation of the web page, which the browser 16 uses to generate a presentation on client computer 18. It includes representations of the conventional HTML elements contained in the web page, as well as tags embedded therein. As suggested above, the browser typically parses such non-conventional tags and places them in the DOM, along with the representation of the conventional HTML elements, in which they are contained. However, the browser does not otherwise process those tags.

[0036] The player does, however. In the illustrated embodiment, it accesses the DOM to identify the tags and enhances the presentation based thereon. This can be accomplished via generation of further HTML or via direct modification of the DOM.

[0037] Regardless, enhancements effected by the player can include highlighting UI objects on the client computer 18 console, filling-in fields, selecting radio buttons, setting sliders, and the like—all based on text, graphics and/or formatting information directly encoded within the tags or referenced thereby. For sight-impaired users, the enhancements can also include, by way of non-limiting example, magnifying or enhancing the contrast of UI objects, generating synthesized speech, and so forth—again, based on information encoded within the tags or referenced thereby.

[0038] Instead of, or in addition to, text, graphics and formatting information, the tags can include directly encoded or indirectly referenced scripts. These can be JavaScripts, VisualBasic scripts, and so forth, that are directly executable within the virtual machine environment established by the browser or, alternatively, they can be pseudo code, state tables or other intermediate forms that are interpreted by the player itself. Regardless of their form, these “scripts” can manipulate UI field contents, e.g., causing the player to blank out certain fields, highlight others in sequence, substitute a fill-in field with a combo-box or, more generally, subsitute an original UI with a different UI of similar purpose, thereby, aiding the users's understanding of the web page. Other examples are scripts that effect wizard-like operation, effectively, automating the web page operation for the user—e.g., filling in some UIs based on others' values, copying values from some UIs to others in exact or augmented form, skipping unnecessary UIs, presenting pop-ups, tables and dialog boxes and so forth to ease user understanding or use of the page.

[0039] Still other examples are scripts that establish a live link, preferably, via a browser window, between the user and an automated or live “help desk” attendant. In establishing the link, the script can cause the player to transmit relevant web page state information, as well as client computer information, web page HTML code, and screen captures to the attendant. The script can also facilitate direct implementation of changes suggested by the attendant. Thus, for example, if the attendant suggests that a UI presented by the browser be filled in with a specific value, the script can cause the player 20 to fill the suggested value in on the user's behalf.

[0040] In other embodiments, the scripts can cause the player to generate or confirm values for “cookies” maintained on the client computer 18. Use of the scripts and player in this regard, can reduce the need for information transfer between the client and server, e.g., insofar as the player acts as an agent for the server in establishing potentially proprietary or otherwise sensitive cookie values.

[0041] Whether effected by the player from directly or indirectly encoded text, graphics, scripts or other information, the enhancements appear as an integral part of web page from the user perspective. Nonetheless, they may be developed and implemented separately from the original web page development effort. This can speed implementation of the web page, since the persons involved with it can work independently of those who author tags, scripts and so forth for use by the player in enhancing web page presentation.

[0042] In step C, the browser 16 displays or otherwise presents the web page, e.g., text, pictures, video, hyperlinks and other output, while accepting text, user selections and other input, all in the conventional manner. The client user can activate the player 20, e.g., by selecting an option in the client display, clicking an icon, or other method. See, step D. Alternatively, the player 20 can be activated automatically upon download of the web page.

[0043] Referring to step E, when the player 20 is activated it accesses the DOM 22 to determine which conventional HTML element the user has currently selected or is near (e.g., as in the case of a pointer that is hovering over a button or other UI object), if any. The player also traverses the DOM to determine whether there are any tags associated with that HTML element.

[0044] If the tag for the identified element contains a text, graphic or other information for use in enhancing the presentation, the player uses that information accordingly. This can be by direct modification of the DOM though, preferably, it is by generation of further HTML for display by the browser in the conventional manner. If the tag includes an I.D. (unique or otherwise) of the HTML element (or of associated enhancement text, graphics, scripts, etc.), the player 20 looks to an auxiliary store for the enhancement information. In the illustrated embodiment, this is by way of a request to XML or other database maintained on server 12 or by another digital data processor (including client computer 18, itself). See, step F. In other embodiments, it is by reference to a function, table or other invocable or addressable mechanism maintained in the web page or elsewise.

[0045] Referring to step G, in the illustrated embodiment, the request by the browser 16 to the server 12 includes a UI identifier and/or HTML page identifier. The server 12 uses this to fetch or obtain text, graphics, scripts or other presentation enhancement information from data base 24. See, step G. The enhancement is communicated back to the player 20 (or, alternatively, directly to the browser 16). See, step H. The player 20, in turn, generates HTML reflecting the enhancement or modifies the DOM 22 accordingly. See, step 1. The web page presentation is thereby enhanced. See, step J. In the event that the enhancements downloaded from the server, e.g., in step H, include “scripts” as noted above, the player 20 executes those in order to provide further web page enhancement which ends, e.g., at script termination or upon user request. In some embodiments of the invention, the web page may include JavaScript, VisualBasic, or other scripts or controls that interact with an API of player 20 to cause it to start, stop or perform a specific action. Conversely, the player 20 may take action based on the web pages'specific state.

[0046] When the user has completed the web page or portion thereof, information collected in UI fields is communicated to the server 12 in the normal course, e.g., via HTTP directives.

[0047] The teaching above can be further understood via a simplified example of the illustrated embodiment depicted in FIG. 2 and FIGS. 3A-3C. Of course, as can be appreciated by one with ordinary skill in the art, a typical web page can be more complex that shown in the example.

[0048] Referring now to FIG. 2, an example of the illustrated embodiment using a state-table script 50 is presented. In this example, the script is designed to aid a client user in entering information into a simplified mortgage loan application. Therefore, it should be recognized by one skilled in the art that this example is but one of many possible implementations. In fact, the methods presented herein are suitable for many common user programs including, by way of non-limiting example, word processors, database systems, business enterprise systems, static web site pages, and any others requiring client user actions.

[0049] A user initiates download of an HTML web page in any of a number of common ways. The browser 16 creates the DOM 22 as described above. Here, that DOM is identified as element 52 representing the first HTML page of the loan application, is a directed graph indicating that there are four UIs: name 54, current address 56, social security number 58 and type of loan desired 60. Each of these UIs has an embedded tag, for example, name 54 has a tag with the value of “1” 62; address has a tag with the value of “2” 64, and so on. In this example the tags have only an integer value, although the tags can be more complex, e.g., containing embedded scripts, displayable attributes and the like.

[0050] The client user, in this example, activates the player 20 at this point, although the client user could pick any other point throughout the execution of the web pages. Here, the player identifies the current UIs being displayed via the DOM and sends a request to the server for enhancement information for those UIs. The server accesses the database and returns a state table 50 and instructions for displaying “pop-up” text explaining the UIs for the client user.

[0051] Each of the four states shown 62 through 68, has a set of “rules.” For example, in state-162, the rules 70, here denoted using “pseudo code” for ease of understanding, instruct the player to load displayable prompts (e.g., text for use in pop-up text boxes) and to get information from client user input UIs. The rules can contain JavaScript or other browser executable instructions, and can also contain information such as attributes, points to scripts, and the like as described above. Contained within the rules are the tag identifiers (e.g., element 62) relating to the UIs.

[0052] The rules 70 are processed by the player, resulting in a client display as indicated in FIG. 3A. These rules 70 are used to modify the presentation as shown in FIG. 3A. As can be seen in the illustration, two pop-up text boxes 84a and 84b are displayed corresponding to UIs name 54 (FIG. 2) and type 60. Further, both UIs are displayed in an enhanced mode 82a, 82b.

[0053] Continuing the example, here the client user is applying for a refinance mortgage, hence, “REFIN” is entered in the corresponding UI 82b in accordance with the displayed pop-up text 84b. This information is returned to the server but also is returned to the state machine script 50 via the player.

[0054] At this point, the server accepts the input, and downloads the HTML for a second web document and the browser in turn creates the corresponding DOM 76. This document requests the client user to input additional information concerning the property location and attendant financial information. The web page, at this point, has made no differentiation based on the type of loan application requested by the client user, but has rather, downloaded a generic screen corresponding to the application HTML.

[0055] The state table, however, transitions from state-162 to state-264 based on the value “REFIN” in the UI “Type.” In state-264, a corresponding set of rules 78 is also present. As before, the rules require for the loading and displaying of certain prompts. Additionally, because the type of mortgage is a refinance, the property address is the same as the current address, hence the rules automate this entry by automatically filling in the value for property address as indicated within the rules 78.

[0056] Referring to FIG. 3B, the prompts 86a, 86b are displayed informing the client user that the property address and price are not required for a refinance. Further, according to the rules 78 for state-264, “prop address” and “price” are blocked, meaning that any entry in those UI will not be accepted. The rules have automated this process by filling the property address with the value in “current address” automatically.

[0057] When the client user has completed the entries for this second document, the information is returned to the server which uploads a third HTML document where in turn, the browser creates a corresponding DOM 100. Here, the state table transitions to state-468 regardless of the type of loan application because the information is applicable to either a “new” or “refinance” loan type. Again, the server returns further information concerning state-468, the browser updates the DOM 100, and the display is updated (FIG. 3C) according to the DOM 100 and the rules for state-468.

[0058] Methods and system according to the invention have a variety of uses. In addition to enhancing presentation of static web pages, business and other applications programs that are presented via web pages, and so forth, those methods and systems can be used to improve web page accessibility, e.g., to the blind or otherwise impaired. They can also be used for purposes of web page testing, for example, by automically exercising the web page (e.g., using the player and the embedded tags) and/or by guiding human testers through the web page to perform test sequnces. Moreover, they can be used to present the web pages to end users in alternate languages. Thus, for example, the player can be used to present the text on a web page in an alternate language and/or it can be used to present prompts that instruct the user in the alternate language. It will also be appreciated that methods and systems according to the invention provide enhanced security, since they can provide for web page enhancement solely from “inside,” e.g., without the use of executables external to the page and/or browser itself.

[0059] Of course, this example has been simplified to allow one skilled in the art to recognize the principles and methods behind the illustrated embodiment. Those skilled in the art will appreciate that various modifications can be made to these embodiments without departing from the scope of the present invention. These principles and methods can be more complex, encompassing extended automation scripts, video, audio and the like. In other embodiments, these methods can be used for database management and retrieval, front-end processors for complex data entry, and pre-processing information for multiple server applications, to name only a few. These methods and techniques can be used with a variety of markup languages in addition to HTML.