System and method for automated construction of URL, cookie, and database query mapping

[0002] Embodiments of the present invention claim priority from U.S. Provisional Patent Application Serial No. 60/285,972 entitled “Automated Construction of URL, Cookie, and Database Query Mapping,” filed Apr. 24, 2001, and are related to U.S. Utility Patent now U.S. Pat. No. 6,591,266 entitled “System and Method for Intelligent Caching and Refresh of Dynamically Generated and Static Web Contents,” filed Aug. 14, 2000. The content of these applications is incorporated by reference herein.

[0003] 1. Field of the Invention

[0004] The present invention relates, generally, to content delivery systems, and, in preferred embodiments, to the automated construction of URL, cookie, and database query mappings in systems and methods for intelligent caching and refreshing of dynamically generated and static web content.

[0005] 2. Description of the Related Art

[0006] The need to account for users' quality perceptions in designing Web servers for e-commerce systems has been well recognized, for the brand name of an e-commerce site is often associated with the type of experience users receive. Response time is a key point of differentiation among e-commerce Web sites. Snafus and slow-downs at major Web sites during special events or peak times demonstrate the difficulty of scaling up e-commerce sites. Such slow response times and down times can be devastating for e-commerce sites as indicated in a recent study on the relationship between Web page download time and user abandonment rate. The study shows that only 2% of users will leave a Web site (i.e. abandonment rate) if the download time is less than 7 seconds. However, the abandonment rate jumps to 30% if the download time is around 8 seconds. The abandonment rate goes up to 70% when the download time is around 12 seconds. This study clearly establishes the importance of fast response times to an e-commerce Web site to retain its customers.

[0007] In technical terms, to ensure the fast delivery of fresh dynamic content and engineer highly scalable e-commerce Web sites for special events or peak times puts heavy pressure on IT staffs due to the complexity of current e-commerce applications. For many e-commerce applications, Web pages are created dynamically based on the current state of a business, such as product prices, inventory, and other information stored in database systems. This characteristic requires e-commerce Web sites to deploy cache servers, Web servers, application servers, and database systems at the backend. The roles played by these servers are illustrated in FIG. 1 and summarized as follows:

[0008] 1 . A database management system (DBMS) 10 or other external data sources 26 to store, maintain, and retrieve all necessary data and information to model a business.

[0009] 2. An application server (AS) 12 that incorporates all the necessary rules and business logic to interpret the data and information stored in the database. AS 12 receives user requests 14 for HTML pages and cookies 40, and depending upon the nature of a request, may need to access the DBMS 10 or external data source 26 via queries 28 or external data requests 30 and retrieve database results 32 or file/network access results 34 to generate the dynamic components of the HTML page 22.

[0010] 3. A Web server (WS) 16 which receives user requests 18 and cookies 36 from end users 20 and delivers the dynamically generated Web pages 24 back to the end users 20.

[0011] 4. Cache servers (edge caches or frontend caches) (not shown in FIG. 1) to accelerate content delivery.

[0012] One possible solution to scale up database-driven e-commerce sites is to deploy network-wide caches so that a large fraction of requests can be served remotely rather than being served from the origin Web site. This solution has several advantages, including improved content delivery times and reduced traffic at the Web sites. Many content delivery network (CDN) vendors provide Web acceleration services, and studies have shown that CDN can have a significant performance impact. However, for many e-commerce applications, HTML pages are created dynamically based on the current state of a business, such as product prices and inventory, rather than static information. As a result, the time to live (TTL) for these dynamic pages can not be estimated in advance, and content delivery by most CDNs is typically limited to the handling of fairly static pages and streaming media rather than the full spectrum of dynamic content.

[0013] Because the application servers, databases, Web servers, and caches are independent components, there is no efficient mechanism to have database content changes reflected in the cached Web pages. To ensure the freshness of dynamic content in the caches, integration of the caches, Web servers, application servers, and back-end database systems is required. Ideally, when updates in the database are observed, the pages which are impacted by such changes should be identified and such pages in the cache should be invalidated or refreshed accordingly. However, the information required for such integration includes the knowledge of what database queries and/or other external/internal data source access were the result of a dynamic Web page request. In other words, a mapping between URL requests and queries is required, and this knowledge is missing in conventional dynamic content caching solutions. Thus, there is a need for the automated construction of URL, cookie, and database query mappings to enable the efficient invalidation or refreshing of cached web content.

[0014] Note that knowledge about dynamic content is distributed across three or more different servers, including the Web server, the application server, and the database management server. Consequently, it is not straightforward to create a mapping between the data and the corresponding Web pages automatically. Some approaches, for example, assume that such mappings are provided by system designers. In other systems, programmers must re-engineer the application server programs to use a set of specific APIs (Application Program Interfaces) to generate such mappings.

[0015] More recently, other systems and methods for the construction of URL, cookie, and database query mappings have been proposed. For example, in U.S. Utility Patent U.S. Pat. No. 6,591,266 entitled “System and Method for Intelligent Caching and Refresh of Dynamically Generated and Static Web Contents” (“the '208 application”), the embodiment of FIG. 3 in the '208 application does not require modification of application server program, but it does over-invalidate cached pages because it is not 100% accurate. The embodiments of FIGS. 5, 6, and 7 in the '208 application require modification of application server programs or database application programs to pass additional parameters, but they are 100% accurate. The values of the URL string and cookie must be explicitly passed to the JDBC (Java Database Connectivity) (i.e. the database connection API).

[0016] Therefore, it is an advantage of embodiments of the present invention to provide a system and method for the construction of URL, cookie, and database query mappings in which the users do not need to manually specify such mappings, as in other systems.

[0017] It is a further advantage of embodiments of the present invention to provide a system and method for the construction of URL, cookie, and database query mappings in which the existing application server programs do not need to be changed. Company Web sites may be reluctant to have their mission-critical application server programs modified by content delivery service providers. In addition, the source code may not be provided by the application server vendor.

[0018] It is a further advantage of embodiments of the present invention to provide a system and method for the construction of URL, cookie, and database query mappings which do not result in the over- or under-invalidation or refreshing of cached Web content.

[0019] It is a further advantage of embodiments of the present invention to provide a system and method for the construction of URL, cookie, and database query mappings which are automated, and wherein the mapping is “plug and play” compatible in the software of the Web site architecture, without requiring re-booting, recompiling, etc.

[0020] It is a further advantage of embodiments of the present invention to provide a system and method for the construction of URL, cookie, and database query mappings which can be selectively applied to one or more servlets in an application server, or enabled/disabled on the fly, without otherwise disrupting the application server.

[0021] It is a further advantage of embodiments of the present invention to provide a system and method for the construction of URL, cookie, and database query mappings in systems in which the application server employs a multiple-threaded or multi-tasking operating system.

[0022] These and other advantages are accomplished according to a method employed within a content delivery system comprising a Web server and application server, wherein the Web server is coupled for receiving a user information request destined for an original servlet in the Web server. The user information request may comprise a URL request. The servlets are capable of calling one or more database applications, which includes databases and the databases of suitable DBMSs, or other programs or servlets. The method comprises redirecting the user information request to a wrapper servlet, which includes statements for extracting user information request identification information from the user information request, and assigning a job identification system variable containing the user information request identification information to the redirected user information request. The user information request identification information may include URL string and cookie information. The redirected user information request is then forwarded to the original servlet in the form of an HttpServletRequest.

[0023] The HttpServletRequest is then communicated to the application server, which issues at least one query destined for accessing a database application such as a database, an original database connectivity API in the DBMS, or other programs or servers. The at least one query includes the job identification system variable. The query is then redirected to a wrapper database connectivity API, which includes statements for recovering the user information request identification information from the at least one query and constructing a user information request identification information and query mapping. At any of the above-described steps, timestamp information may be captured for Web site profiling purposes.

[0024] These and other objects, features, and advantages of embodiments of the invention will be apparent to those skilled in the art from the following detailed description of embodiments of the invention, when read with the drawings and appended claims.

[0025] FIG. 1 is a block diagram illustrating an example of a conventional architecture and data flow of a database driven e-commerce Web site.

[0026] FIG. 2 is a block diagram illustrating an intelligent Web page caching and refreshing mechanism that automatically constructs a URL, cookie, and database query mapping according to an embodiment of the present invention.

[0027] In the following description of preferred embodiments, reference is made to the accompanying drawings which form a part hereof, and in which is shown by way of illustration specific embodiments in which the invention may be practiced. It is to be understood that other embodiments may be utilized and structural changes may be made without departing from the scope of the preferred embodiments of the present invention.

[0028] Response time is one key point of differentiation among electronic commerce (e-commerce) Web sites. For many e-commerce sites, Web pages are created dynamically based on the current state of a business stored in database systems. Snafus and slow-downs during special events or peak times create challenges for engineering high performance database-driven e-commerce Web sites. One way to achieve this goal is through dynamic content caching. Caching has been viewed as an effective way to scale up Web sites.

[0029] To ensure the freshness of dynamic content in the caches, integration of the caches, Web servers, application servers, and back-end database systems is required. Ideally, when updates in the database are observed, the pages which are impacted by such changes should be identified and such pages in the cache should be invalidated or refreshed accordingly. However, the information required for such integration includes the knowledge of what database queries and/or other external/internal data source access were the result of a dynamic Web page request. In other words, a mapping between URL requests and queries is needed.

[0030] Thus, embodiments of the present invention provide for the automated construction of URL, cookie, and database query mappings to enable the efficient invalidation or refreshing of cached web content. It should be understood that although embodiments of the present invention are described herein primarily using Java-related terminology for purposes of illustration and explanation, the scope of the present invention is not so limited, and is applicable to other virtual machine and interpretive dynamic languages and environments, such as application servers based on CGI (Common Gateway Interface), which use scripts other than Java.

[0031] A example system environment for embodiments of the present invention is illustrated in FIG. 2. As indicated in FIG. 2, if an end user 42, through interaction 56 with a client browser 54, makes a request 46 for a Web page that is not stored in cache 44, or is invalid in cache 44, the request 46 will be passed on to the Web server 48. Note that cookies 62 may also be passed to the cache 44 and the Web server 48 along with the request 46.

[0032] After the Web server 48 receives the request 46, it may send the request to the application server 52 along with some optional parameters. When the Web server 48 calls the application server 52, two classes (objects) may be used for communication between the Web server 48 and the application server 52. These objects may include an HttpServletRequest 58 for the communication of certain attributes (e.g., the URL request string and cookie information 62), and HttpServletResponse 60 for communicating the result returned by the application server 52. The cookie information 62 is essentially an additional key that may be necessary to identify the end user 42 and the specific Web page that should be delivered to the end user 42. The attribute values for the URL string and the cookie in HttpServletRequest 58 may then be extracted by the application server 52.

[0033] For example, Java statements to extract cookie information 62 may include the following:

[0034] URL information and related necessary parameters can be extracted in a similar fashion. These parameters are needed for the application server 52 to perform necessary computations and prepare the result page to put into HttpServletResponse 60 in the form of a string or a handle. During the performance of computations and result page preparation, the application server 52 may issue one or more queries 68 to one or more databases, the databases of suitable management systems 70, or other programs to retrieve the information necessary to prepare the result page. When the Web server 48 receives the HttpServletResponse 60 from the application server 52, the Web server 48 may parse the object, extract the result page 88, and send the page 88 to the end user 42. Note that although FIG. 2 illustrates the Web server 48, application server 52, and database management system 70 as separate blocks, it should be understood that these functions may be located in separate servers or the same server.

[0035] Embodiments of the present invention automate the construction of the URL/Query Mapping by first redirecting HttpServletRequests 58 from the Web server 48 to a wrapper (dummy) servlet 64 instead of the intended (original) servlet 66. In one embodiment, the redirection may be specified in a Web server vendor property file 50. In the example of FIG. 2, this vendor property file 50 is located in the Web server 48, although it should be understood that the vendor property file 50 may also be located in the application server 52. For example, the following statements in the Web server vendor property file 50 can redirect the HttpServletRequests 58 to CachePortalServlet (an example wrapper servlet) instead of the original servlet HouseServ (an example application related to housing) in the application server 52:

[0036] # weblogic.httpd.register.HouseServ=HouseServ

[0037] weblogic.httpd.register.HouseServ=CachePortalServlet

[0038] weblogic.httpd.initArgs.HouseServ=actualServlet=HouseServCP

[0039] weblogic.httpd.register.HouseServCP=HouseServ

[0040] The first line is the original statement in the vendor property file 50, which gets commented out in the present invention, and three additional statements are added. The original statement relates a particular URL request with a particular original servlet (HouseServ in this example).

[0041] The second line will redirect the URL requests for HouseServ to CachePortalServlet. After the wrapper servlet (CachePortalServlet in this example) receives the request, CachePortalServlet will extract the URL string, cookie information, and other related information and parameters, if applicable. The URL string, cookie information, and other related information and parameters can be used to identify each individual request to the Web site as well as the page sent to the users or stored in the caches. CachePortalServlet will then use the URL string, cookie information, and other parameters to assign a “job name” or “job identification” to the request. The “job identification” is not a parameter, but a system variable that is automatically passed whenever the servlet calls other programs or processes.

[0042] The third line is for syntax purposes, the declaration of a servlet to be used later.

[0043] The fourth line will forward the URL request to the original servlet HouseServ. The original servlet HouseServ will then perform its housing functions.

[0044] In another embodiment, the redirection may be implemented without modification of any application server property files by moving the original servlet to another location or renaming it, and then assigning the wrapper servlet the name of the original servlet. Continuing the previous example for purposes of illustration only, the original servlet HouseServ can be renamed HouseServ′ and the wrapper servlet CachePortalServlet can be renamed HouseServ. After the wrapper servlet receives the request from the Web server, it performs the URL and cookie extraction and job name assignment tasks described above, then forwards the request to HouseServ′.

[0045] In either of the above-described embodiments, after the request is forwarded to the original servlet 66 (HouseServ or HouseServ′ in the examples given above) along with the “job identification” system variable, the application server 52 may issue queries 68 to the JDBC (the database connectivity API) 76. It should be understood that the application server 52 is not limited to sending queries 68 to a single or multiple DBMSs 70. The application server 52 may also issue external calls, API calls, or http requests 72 to other file systems, DBMSs, or servers 74. Generally, it should be understood that although most of the description herein is given with reference to a DBMS 70, in embodiments of the present invention accesses 72 may be made to external data sources represented by blocks labeled “File system+network” 74.

[0046] Although the application server 52 may issue queries 68 or requests 72 to an intended (original) JDBC 84 within a DBMS 70 or external information source 74, in embodiments of the present invention the requests or database queries may be redirected through a common database connectivity API 76 such as, but not limited to, JDBC (Java Database Connectivity), to a wrapper (dummy) JDBC 78 (if JDBC is used to connect the databases). Note that the “job identification” system variable is passed implicitly from one servlet to another. Thus, when a database query 68 is issued through the database connectivity API 76 and redirected to the wrapper JDBC 78, the API will output the query statement and its “job identification.” The wrapper JDBC 78 can then recover the URL and cookie information from the “job identification” system variable, and capture a timestamp associated with the operation. The wrapper JDBC 78 also knows the query (SQL statement), because the SQL statement is one of the JDBC parameters. Therefore, a mapping 80 between URL, cookie, and database statements are automatically constructed. The mapping 80 may then be output to a file or database within, or accessible by, a content delivery server 82. The wrapper JDBC 78 will then forward the query to the original JDBC 84 so that the appropriate data can be retrieved from the databases. Note that although in the example of FIG. 2, the database connectivity API 76, wrapper JDBC 78, and original JDBC 84 are located in the DBMS 70, it should be understood that the these files and programs may also be located in the application server 52 or in the application programs of other machines.

[0047] The data retrieved from the databases may then be forwarded back to the database connectivity API 76, and then returned to the application server 52 in the form of results 86. The application server 52 may then return the results to the Web server 48 within the HttpServletResponse 60. The Web server 48 may then compile the requested Web page, and communicate the page content 88 back to the cache 44 and client browser 54.

[0048] When a data change is detected by the content delivery system, the mapping information 80 may be used to identify which Web pages stored in cache 44 may be out of date, and therefore will have to be invalidated or refreshed. The content delivery server 82 keeps track of where Web pages have been stored, and thus when it becomes necessary to refresh or invalidate a Web page, the content delivery server 82 will send an invalidate or refresh request or message 90 to the appropriate location.

[0049] Although the example embodiment described above involved a single original servlet HouseServ sending an HttpServletRequest to the application server 52, in other example embodiments, several original servlets may call each other before an HttpServletRequest is sent to the application server 52. For example, HouseServ may call AccountServ, which may then call PersonnelServ, which may then send an HttpServletRequest to the application server 52. Thus, in alternative embodiments of the present invention, the redirection scheme may be applied to every original servlet, database connectivity API, and program in the Web site. By redirecting the requests to wrapper Servlets instead of the original Servlets and attaching timestamp information, the timing and sequence of calls to the various original servlets can be monitored and determined. This information may be useful in analyzing or profiling Web site performance.

[0050] In addition, when the database connectivity API 76 receives the database query results, it may cache the query results 86 in the application server 52, and record a time stamp when the database finishes the query processing. The timestamp may be useful in analyzing or profiling Web site performance, and identifying which query results 86 should be cached in the application server 52. Subsequent HttpServletRequests 58 to the application server 52 will result in the retrieval of those cached query results 86, without a need to issue queries 68 and requests 72 to the DBMS 76 or the filesystem+network 74. Similar to the cache refreshing or invalidation scheme described above, when a data change is detected by the content delivery system, the mapping information 80 may be used to identify which results stored in the application server 52 may be out of date, and therefore will have to be invalidated or refreshed. The content delivery server 82 keeps track of where results have been stored, and thus when it becomes necessary to refresh or invalidate a stored result, the content delivery server 82 will send an invalidate or refresh request or message 90 to the appropriate location in the application server 52.

[0051] Therefore, embodiments of the present invention provide a system and method for the construction of automated and accurate URL, cookie, and database query mappings in which the users do not need to manually specify such mapping, and the existing application server programs do not need to be changed. Embodiments of the present invention also provide a system and method for the construction of URL, cookie, and database query mappings which are “plug and play” compatible in the software of the Web site architecture, and that can be used in systems in which the application server employs a multiple-threaded or multi-tasking operating system.