Apparatus and method for retrieving data from a network site

[0002] This invention generally relates to data transmission networks and, more particularly, to retrieving data from a first network site via a second network site.

[0003] The World Wide Web is a collection of servers connected to the Internet that utilize the Hypertext Transfer Protocol (“HTTP”). HTTP is a known application protocol that provides users with access to documents (e.g., web pages) written in a standard mark-up page description language known as Hypertext Markup Language (“HTML”). HTTP is used to transmit HTML web pages between a remote computer (e.g., a server) and a local computer in a form that is understandable to browser software (e.g., Netscape Navigator™, available from Netscape Communications Corporation of Mountain View, Calif.) executing on the local computer.

[0004] Among a number of basic document formatting functions, HTML enables software developers to specify graphical pointers (commonly referred to as “hyperlinks”) on displayed web pages (“base web pages”) that point to other web pages (“remote web pages”) typically resident on remote servers. Once the remote web page is displayed, a user of a local computer system may freely review its contents and perform any functions that it provides. One such function, for example, may be obtaining specified data (“data”) from the remote site. After the data is retrieved, it may be displayed by the local computer system in a selected format specified by the remote web page. Problems may arise, however, when utilizing such web page function. Primarily, access to the data through the remote web page interface may be cumbersome and thus, not intuitive to the user. Accordingly, the user may not be able to retrieve the desired data from the remote site. Similarly, even if the data is retrieved from the remote site, its display in the selected format also may be cumbersome and thus, not in a form that is easily understood by the user.

[0005] The art has responded to these and similar problems by enabling a base web site to automatically extract data from a remote web page, and then display the retrieved data in a format specified by the base web site. Accordingly, the base web site, and not the user, accesses the remote page to retrieve the data. A typical process that may be used for retrieving and displaying such data may begin when a user requests the data while accessing a base web page. In response, the base web site directs a data request to the remote site requesting the data. After retrieving the request, the remote site typically generates a response web page having the data. The response web page then is directed to the base web page for processing.

[0006] Instead of displaying the response web page which, undesirably, is in a form specified by the remote site, the base site executes a specially designed scanning procedure that scans the response web page for the data. Once the data is retrieved from the response by the scanning procedure, it may be displayed, via the base web page, in a format that is designed specially by the base web page.

[0007] As noted above, the scanning procedure is specially designed to retrieve the data from the remote web page. Such scanning procedure is implemented by writing an application program that utilizes either conventional procedural or object oriented programming techniques. To be effective, such program must be preconfigured with the location of the data to be retrieved within the remote web page. Accordingly, a new scanning application program must be written each time the format of a response web page is modified. Developing such new scanning application programs are very time consuming, however, thus adding to the overall cost of developing and maintaining the base web site.

[0008] It therefore would be desirable to have a method and apparatus that enables a base web page to efficiently retrieve information from a remotely linked web site without requiring that a scanning program be developed.

[0009] In accordance with one aspect of the invention, an apparatus for retrieving data from a first network site for use by a second network site utilizes a template that specifies the location of the data within a response solicited from the first network site. The template is a mark-up document having a similar format to the response and thus, it is not an application program. A marker is included within the template to determine the location of the data within the response. A matching mechanism, which may be used with any template, is utilized to compare the template with the response to determine the exact location of the data within the response. The data may be retrieved when its location within the response is ascertained. Once retrieved, the data may be used by the second network site as specified by the second network site. Accordingly, the location of the data is ascertained within the response with a template and not with a scanning application program.

[0010] In accordance with another aspect of the invention, a method of retrieving data from a first network site for use by a second network site first directs a request for the data to the first network site. Receipt of the request by the first network site generates a response having a predetermined location for the data. The method then receives a template having a marker for identifying the predetermined location of the data within the response. The marker in the template then is matched with the data in the response. Through this matching operation, a variable within the marker is assigned the same value as the data. Finally, the variable is provided to the second network site for any use.

[0011] In accordance with yet another aspect of the invention, the step of matching for the above noted method first determines the location of the predetermined location in both the template and the response. The value of the data in the predetermined location then is assigned to the variable.

[0012] The foregoing and other objects and advantages of the invention will be appreciated more fully from the following further description thereof with reference to the accompanying drawings wherein:

[0013] FIG. 1 schematically shows a commonly used network arrangement in which a local computer system may communicate with various network sites via the Internet.

[0014] FIG. 2 is a flow chart showing the more significant steps of a process for retrieving data requested by a user of a local computer system, and then displaying the requested data on the local computer system.

[0015] FIGS. 3A and 3B show a preferred process that may be used by a matching mechanism for locating variable information within an HTML response.

[0016] FIG. 1 schematically shows a commonly used network arrangement in which a local computer system 100 may communicate with various network sites via the Internet 101. When utilized with a preferred embodiment of the invention, the network sites include a base World Wide Web site (“base web site 102”) for direct access by a user of the local computer system 100, a remote World Wide Web site (“remote web site 104”) that, as discussed in detail below, is accessed by the base web site 102, and a plurality of other World Wide Web sites as shown by the ellipses. By way of example, the base web site 102 may be operated by a software distributor, and the remote web site 104 may be operated by a shipping company that ships software for the software distributor. It should be noted that the base web site 102 and the remote web site 104 may be on the same hardware device (e.g., a network server), or on different hardware devices that communicate through the Internet 101.

[0017] In accordance with a preferred embodiment of the invention, in response to a local computer system user's request for data, the base web site 102 is constructed to retrieve the requested data from the remote network site, and then display the retrieved data, on a display device at the local computer system 100, in a preselected format specified by the base web site 102. The user therefore does not directly access the remote web site 104. A generalized flow chart showing the more significant steps of a process for retrieving and displaying the requested data is shown in FIG. 2.

[0018] The process shown in FIG. 2 starts at step 200 in which an HTML form (not shown) is displayed on the display device at the local computer system 100. The HTML form, which is generated and formatted by the base web site 102 in accordance with conventional processes, includes at least one field for entering information (“form-input”) that is required for retrieving the data from the remote web site 104. Once the form-input is entered into the form, the user may select a “submit” button to transmit a message having the form-input information to the base web site 102 for processing. Continuing with the above example, the data to be retrieved may be the delivery date of a software program to the user, and the form-input information required to retrieve the delivery date may be the user's name.

[0019] The process then continues to step 202 in which the base web site 102 first extracts the form-input from the message and then generates an HTTP “POST” request. The POST request includes the form-input information which, when received by the remote web site 104, solicits an HTML response from the remote web site 104 having the requested data. The form-inputs thus are used by the remote web site 104 to retrieve the requested data from a storage device such as, for example, a database in a non-volatile storage medium.

[0020] In an alternative embodiment, the form-input information may be processed at the base web site 102 into a specified form to be used by the remote web site 104. For example, the shipping web site may require a customer number instead of a customer name. The software developer site therefore must use the form-input customer name to retrieve the customer number from a local database, and then direct that customer number to the shipping web site via the POST request. In a similar manner, the form-input information (or information derived from the form-input information) may be added to multiple POST requests that each are directed to different remote web sites. It is expected that such POST requests would solicit responses having different formats.

[0021] The process then proceeds to step 204 in which the HTML response is received by the base web site 102 for processing. At step 206, a matching mechanism matches the HTML response against a template to locate the requested data within the HTML response. The details of the matching mechanism and its interaction with the template of step 206 are the subject of FIGS. 3A and 3B, both of which are discussed in detail below.

[0022] The template used in step 206 merely is an edited version of the HTML response document utilizing Meta-HTML (MHTML) markings. Meta-HTML is described in detail in the “MAWL 2.0 Tutorial”, the copyright of which is owned by Lucent Technologies and is available on the World Wide Web at “http://www.bell-labs.com/project/MAWL/tutorial.html.” The disclosure of this document is incorporated herein, in its entirety, by reference. MHTML also is described in detail in “Mawl 2.0 Quick Language Reference,” the copyright of which is owned by Lucent Technologies and is available on the World Wide Web at “http://www.bell-labs.com/project/MAWL/quickref.html#Mhtml.” The disclosure of this document also is incorporated herein, in its entirety, by reference. Both references are referred to herein as “MHTML references.”

[0023] The MHTML markings act as markers to identify the location of the data within the His response document. MHTML markings used in a preferred embodiment of the invention include “MVAR,” “MITER,” and “/MITER.” When creating the template (i.e., while editing the HTML code of the response document), the MHTML markings are inserted in the HTML code of the response in place of the variable parts of the response. Such insertion positions the MHTML markings relative to the HTML code in the response (which subsequently becomes the template) so that the location of the data may be ascertained by the matching mechanism (discussed below). For example, when a template is being constructed for the software distributor web site, the MVAR variable will be placed in the location of the shipping date if the shipping date part of the shipping web site response is the variable part of that response. Accordingly, when using that template to determine the shipping date from a shipping web site response, the matching mechanism (discussed below) must determine the relative locations of the MVAR variable and the HTML code in the template, and then apply that relationship to the response to locate the shipping date in the response.

[0024] In general terms, the MVAR marking is used to match text to a variable. A generalized form of an MVAR marking is as shown below:

[0025] <MVAR NAME=x DELIM=string>

[0026] As will be more apparent upon review of FIGS. 3A and 3B and the MHTML references, the NAME attribute of the MVAR marking indicates that a character string at that location in the response under examination must be assigned to a variable named “x.” The DELIM attribute identifies a string of individual characters that each may be used as a flag to identify the end of the character string assigned to the variable “x.” Reference is made to the MHTML references which describe the function of the MVAR marking in greater detail.

[0027] The MITER and /MITER markings (both of which are short for “MHTML iteration”) are used in the template to retrieve a type of information that may be repeated within a response. For example, an unknown number of shipping dates may be used for multiple programs ordered from the software developer by the user. When used in conjunction with the MVAR marking, variable information is retrieved from the response locations corresponding to the area between the MITER and /MITER markings in the template. A generalized form of the MITER marking is as shown below:

[0028] <MITER NAME=NAMELIST CURSOR=I DELIM=string>

[0029] As will be more apparent upon review of FIGS. 3A and 3B and the MHTML references, the NAME attribute of the MITER marking identifies the name of an array to store the repeated information. The name of the array in this example is “NAMELIST.” The CURSOR attribute indicates that the character “I” will be used to represent entries in the array for data between this MITER marking and a subsequently positioned /MlTER marking. As is known by those skilled in the art, the cursor attribute enables MITER markings to be nested within other MITER markings. The DELIM marking identifies a string of characters that each may be positioned directly after the last character in the last character string of interest in the response, thus effectively marking the end of the series of character strings. As is known by those skilled in the art, MITER markings and /MITER markings cooperate to retrieve data between such markings by utilizing MVAR markings and other nested MITER and /MITER markings. Reference is made to the MHTML references which describe the function of the MITER and /MITER markings in greater detail.

[0030] Returning to the flow chart shown in FIG. 2, after step 206, it then is ascertained if there is a match between the template and the response (step 208). This is ascertained with the matching mechanism shown in FIGS. 3A and 3B. A match indicates that the relative locations of the data and the HTML code in the response are the same as when the template was written. If the template and response are not a match, however, then either the relative locations of the data and HTML code in the response have been modified, or the template is defective. The markings in the template are ineffective if the relative locations of the HTML code and the data in the response have been modified. Accordingly, the template must be rewritten to conform with the new relative locations of data and the HTML code in the response.

[0031] If there is not a match at step 208, then the process proceeds to step 210 in which an error message may be displayed on a display device at the base web site 102 indicating that the data may not be retrieved through the base web page. Conversely, if there is a match at step 208, then the data is displayed at the local computer system 100 in a format specified by the base web site 102.

[0032] FIGS. 3A and 3B show a preferred process that may be used by the matching mechanism (steps 206) for locating the variable information within the response. The process examines each character in the template with a template pointer, and each character in the HTML response with a response pointer. The process begins at step 300 in which the first character in the template is examined. It then is determined if that character is either an MVAR marking (step 302), a MITER marking (step 314), or HTML code (step 324). The process thus proceeds based upon the type of character under examination.

[0033] If the character is an MVAR marking, the process moves to step 304 in which the template pointer is incremented to the DELIM marking to determine the delimiting character (i.e., the first character immediately after the variable under examination). At step 306, the character corresponding to the DELIM character is located in the response, and the response pointer is moved to point to such character. The response pointer thus skips all of the text from the character corresponding to the beginning of the MVAR marking, to the character identified by the DELIM character. It then is determined at step 308 if the NAME attribute is defined in the MVAR marking. As is known by those skilled in the art, the NAME attribute is not defined in an MVAR marking if it is not necessary to retrieve the variable information corresponding to that MVAR marking. Accordingly, if the NAME attribute is not defined, then the process loops back to step 309, thereby moving the pointers to examine the next corresponding characters in the response and template.

[0034] If the NAME attribute is defined, then the process moves to step 310 in which the value of the character or character string that was skipped in the response is assigned to the variable in the MVAR marking. The value of the variable is stored in memory at step 312. The process then loops back to step 309 in which the pointers are moved to examine the next corresponding characters in the response and template.

[0035] If at step 302 the character was not an MVAR marking, then it is determined if the character is a MITER marking. As noted above, a MITER marking indicates that a list of character strings may be retrieved from the response. If a MITER marking is detected, the process moves to step 316 in which the characters in the response corresponding to the area between the MITER and /MITER markings in the template are matched in accordance with the matching process. Specifically, the characters and markings between the MITER and /MITER markings are treated as a sub-template, and the characters in the corresponding response locations similarly are treated as a sub-response. Accordingly, the entire matching process shown in FIGS. 3A and 3B is iterated within step 316 for the sub-template and sub-response. Once the characters between the MITER and /MITER markings are matched via the iterated process of step 316, the process loops to step to step 309.

[0036] If at step 314 it is determined that the character is not a MITER marking, then the process continues to off-page connector “A” to step 324 (FIG. 3B) to determine if the character under examination in the template is HTML code. If such character is not HTML code, then the end of the response has been reached and the variables of interest presumably have all been retrieved.

[0037] In such case, the process proceeds to step 330 in which a “match” variable (used by step 208 in the process shown in FIG. 2) is set to “true.”

[0038] Conversely, if at step 324 it is determined that the character under examination in the template is HTML code, then the process continues to step 328 in which it is determined if the character under examination in the response is the same HTML code as that examined in the template. If such character is not HTML code, then the template format is different than the response format and thus, information may not be retrieved for display on the local computer system 100 because the formats are incompatible. Accordingly, if the character is not HTML code, then the process continues to step 330 in which the “match ” variable (used by step 208 in the process shown in FIG. 2) is set to “false.” When the match variable is set to false, a new template must be made to retrieve the data.

[0039] Conversely, if in step 328 it is determined that the character under examination in the response is the same HTML code, then the process loops to step 309, via off page connector “B”, in which the template pointer is moved to the next MVAR or MITER marking in the template and the response pointer also is moved to such corresponding location. The process thus continues by examining the next character in the template and the response.

[0040] When implemented in the base web site, the process shown in FIG. 2 preferably is an application program that utilizes the templates for retrieving and displaying the data. Such application program may be a procedure that is called by a common gateway interface script for retrieving and displaying the data.

[0041] It should be apparent that a template must be constructed for each response from a remote web site that is used with the base web site 102. As noted above, the template is a document written in a mark-up language. Constructing a new template for each response from one or more web sites therefore is much simpler and less time consuming than constructing a new application program for each of such responses. Accordingly, the cost of constructing and maintaining the base web site 102 incorporating the invention is significantly less than if the base web site 102 utilized a prior art data retrieval application program.

[0042] In one embodiment of the invention, an application program incorporating the invention may be utilized at the local computer system 100 to directly access the remote site 104. Accordingly, such application program may include a graphical user interface for entering the form-input. Once the form input is entered, the program may execute the process shown in FIG. 2 to display the data in a format specified by the application program. Moreover, instead of merely displaying the data, the application program may utilize the data for any desired purpose. For example, the data may be processed to produce new data, added to a paper printout, or passed as input to another application program.

[0043] In an alternative embodiment, the invention may be implemented as a computer program product for use with a computer system. Such implementation may include a series of computer instructions fixed either on a tangible medium, such as a computer readable media (e.g., a diskette, CD-ROM, ROM, or fixed disk) or transmittable to a computer system, via a modem or other interface device, such as a communications adapter connected to a network over a medium. Medium may be either a tangible medium (e.g., optical or analog communications lines) or a medium implemented with wireless techniques (e.g., microwave, infrared or other transmission techniques). The series of computer instructions embodies all or part of the functionality previously described herein with respect to the system. Those skilled in the art should appreciate that such computer instructions can be written in a number of programming languages for use with many computer architectures or operating systems. Furthermore, such instructions may be stored in any memory device, such as semiconductor, magnetic, optical or other memory devices, and may be transmitted using any communications technology, such as optical, infrared, microwave, or other transmission technologies. It is expected that such a computer program product may be distributed as a removable media with accompanying printed or electronic documentation (e.g., shrink wrapped software), preloaded with a computer system (e.g., on system ROM or fixed disk), or distributed from a server or electronic bulletin board over the network (e.g., the Internet or World Wide Web).

[0044] Although various exemplary embodiments of the invention have been disclosed, it should be apparent to those skilled in the art that various changes and modifications can be made which will achieve some of the advantages of the invention without departing from the true scope of the invention. These and other obvious modifications are intended to be covered by the appended claims.