Method, system and program for providing indexed web page contents to a search engine database

[0002] 1. Technical Field

[0003] The present invention relates to an improved method, system and program for indexing web page contents, and in particular to an improved method, system and program for providing indexed web page contents to a search engine database. Still more particularly, the present invention relates to a method, system and program for indexing web page contents of each web page requested from the Internet by a user and providing the indexed web page contents to a search engine database.

[0004] 2. Description of the Related Art

[0005] In the prior art, it has been well known that computer systems can be utilized to manage indices of records of databases. Many techniques are known to parse, index, and search databases. However, managing extremely large databases presents special problems.

[0006] In recent years, a unique distributed database has emerged in the form of the World-Wide-Web(Web). The database records of the Web are in the form of web pages accessible via the Internet. Here, tens of millions of pages are accessible by anyone having a communications link to the Internet.

[0007] The pages are dispersed over millions of different computer systems all over the world. Users of the Internet constantly desire to locate specific pages containing information of interest. However, a current problem with the Web is the lack of ability to search and browse (collectively these activities are referred to as “navigate”) the information in the Web. Searching can be described as looking for the resources that contain particular information of interest, such as a specific set of keywords, while browsing is a less focused “looking around.”

[0008] Currently, it is impossible to efficiently navigate all of the Web. The amount of information available through web pages and other data available through the Internet grows by vast amounts each day. In an effort to provide a directory to data on the Internet, many search engines have been created whereby a user can search web pages by a keyword, phrase, topic, etc. However, each search engine typically only accesses a directory of web pages that have been previously “crawled” or indexed for that search engine or manually created. The indexing data is typically stored in a database which can be searched in various ways to provide users with locations of web pages which may be relevant to a user's particular interest.

[0009] With the amount of information on the Internet growing exponentially, there is little chance that the vast majority of the information will be effectively indexed with the techniques utilized today by the various search engine sites that rely on centralized computers accessing and indexing the distributed content of the Internet in a limited manner. This, in turn, leads to the current statistics which estimate that only 15-20% of the information available on the Internet is readily accessible via current search engine indexing methods.

[0010] However, users access a multitude of web pages that have not been indexed by any search engine. Therefore, it would be desirable to retrieve index data from each web page that a user accesses over the Internet in order to update search engine databases with pages that have not yet been indexed. Further, it would be desirable to free bandwidth typically utilized to crawl for non-indexed pages and shift to utilizing data retrieved from user accesses. In particular, indexing pages retrieved from user accesses would be both more efficient and potentially allow a derivation of the value of particular web pages based on the number of times indexes of a particular page are returned to a search engine from different users. Effectively, by receiving index data created during user accesses, the creation of a usefulness value for web pages within search engines may be determined.

[0011] In view of the foregoing, it is therefore an object of the present invention to provide an improved method, system and program for indexing web page contents.

[0012] It is another object of the present invention to provide an improved method, system and program for providing web page contents to a search engine database.

[0013] It is yet another object of the present invention to provide a method, system and program for indexing web page contents of each web page requested from the Internet by a user and providing the indexed web page contents to a search engine database.

[0014] In accordance with the method, system and program of the present invention, in response to each user request for a web page, user access to the web page is provided from a temporary copy of the web page which is stored on a device which accesses the web page and which is accessible to the user. Indexing data is then automatically recorded at that device from the temporarily stored copy of the accessed web page, wherein the indexing data corresponds to contents of the accessed web page. The indexing data is thereafter transmitted from the device to a remote data storage device which provides a search engine database. According to one object of the invention, the indexed data is incorporated into the search engine database, such that previously unknown indexed web page contents are provided to a search engine database in response to a user access of that web page. According to another object of the invention, a statistical count of a number of times that indexing data for a particular web page is provided to a search engine database is maintained.

[0015] All objects, features, and advantages of the present invention will become apparent in the following detailed written description.

[0016] The invention itself, as well as a preferred mode of use, further objects, and advantages thereof, will best be understood by references to the following detailed description of an illustrative embodiment when read in conjunction with the accompanying drawings, wherein:

[0017] FIG. 1 illustrates a block diagram of a typical computer system that may utilize a preferred embodiment of the present invention;

[0018] FIG. 2 depicts a block diagram of a general network environment which may be utilized in the implementation of the present invention; and

[0019] FIG. 3 illustrates a high level logic flowchart of a process for accessing web pages in response to a user request in accordance with the present invention;

[0020] FIG. 4 depicts a high level logic flowchart of a process for acquiring indexed web page contents and providing the indexed web page contents to a search engine database in accordance with the present invention.

[0021] The present invention may be executed in a variety of systems, including a variety of computers under a number of different operating systems. The computer may be, for example, a personal computer, a network computer, a server, a midrange computer or a mainframe computer. In addition, the computer may be a stand-alone system or part of a network such as a local-area network (LAN) or a wide-area network (WAN). For the purposes of illustration, a preferred embodiment of the present invention, as described below, is implemented on a personal computer, such as the Aptiva series manufactured by International Business Machines Corporation.

[0022] Referring now to the drawings and in particular to FIG. 1, there is depicted a block diagram of a typical computer system that may utilize a preferred embodiment of the present invention. As shown, a central processing unit (CPU) 12, a cache 30, a read-only memory (ROM) 13, and a Random-Access Memory (RAM) 14 are connected to a system bus 11 of a computer system 10. CPU 12, Cache 30, ROM 13, and RAM 14 are also coupled to a PCI local bus 20 of computer system 10 through a PCI host bridge 16. PCI host bridge 16 provides a low latency path through which processor 12 may directly access PCI devices mapped anywhere within bus memory and/or I/O address spaces. PCI host bridge 16 also provides a high bandwidth path for allowing PCI devices to directly access RAM 14.

[0023] Also attaching to PCI local bus 20 are communications adapter 15, small computer system interface (SCSI) 18, and expansion bus bridge 29. Communications adapter 15 is for connecting computer system 10 to a network 17. SCSI18 is utilized to control high-speed SCSI disk drive 19. Expansion bus bridge 29, such as a PCI-to-ISA bus bridge, may be utilized for coupling ISA bus 25 to PCI local bus 20. As shown, a keyboard 26 and a mouse 28 are attached to ISA bus 25 for performing certain basic I/O functions. In addition, an audio adapter 23 is attached to PCI local bus 20 for controlling audio output through speaker 24. A graphics adapter 21 is also attached to PCI local bus 20 for controlling visual output through display monitor 22. In alternate embodiments of the present invention, additional peripheral components may be added.

[0024] Computer system 10 also preferably includes an interface such as a graphical user interface (GUI) and an operating system (OS) that reside within machine readable media to direct the operation of computer system 10. Any suitable machine-readable media may retain the GUI and OS, such as RAM 14, ROM 13, SCSI disk drive 19, and other disk and/or tape drive(e.g. magnetic diskette, magnetic tape, CD-ROM, optical disk, or other suitable storage media). Any suitable GUI and OS may direct CPU 12. For example, the AIX operating system is one of IBM's operating systems which may be implemented.

[0025] Further, computer system 10 preferably includes at least one software application(e.g. program product) that resides within machine readable media, for example program 27 within RAM 14. The software application may be accessible by an operating system, or may be incorporated into an operating system control program. Preferably, the software application contains instructions that when executed on CPU 12 carry out the operations depicted in the flow chart of FIGS. 3 and 4 and others described herein. In particular, a software application of the present invention will receive a user request for a particular web page address, implement searching of network 17 for the particular web page address, and retrieve data stored for the web page address if available whereby a temporary copy of the data is stored in cache 30. Typical software applications which perform these functions are provided by browsers, proxy servers, and other distributed devices which provide access to the Internet and typically store a temporary copy of any web pages accessed. In particular, a software application of the present invention also retrieves index data for the contents of each temporarily stored web page which is preferably retrieved in response to a user request. The retrieved index data is preferably compressed and transmitted to a location, preferably a data storage device for a search engine, whereby the index data may be translated to a format usable by the search engine database, as will be further described herein.

[0026] Referring now to FIG. 2, there is depicted a block diagram of a general network environment which may be utilized in the implementation of the present invention. Network environment 40 includes multiple servers 42a-42gwhich are connected through a network 48, such as the Internet. In general, servers 42a-42grespond to commands from clients. For example, a file server may contain stored web pages. When a client submits a request for a page, the server transfers a copy of the file to the client.

[0027] In particular, each of servers 42a-42gmay also function as clients within network environment 40. For example, a personal computer 44, connected to network 48 through server 42gmay send a command to server 42gfor a particular web page. Server 42gis a server to personal computer 44, which is a client. However, server 42gthen searches through network 48 for the server containing the requested web page and sends a command to the server, for example server 42a, for the web page. Server 42ais then a server for server 42g, which is a client. Server 42gwill temporarily store a copy of the web page retrieved from server 42a. In addition, personal computer 44 will typically store a temporary copy of the retrieved web page, as received from server 42g.

[0028] In the implementation of the present invention, for example, server 42gmay include an application which retrieves index data from the web page contents of each web page temporarily stored in cache and/or other volatile memory of server 42g. The retrieved index data is then transmitted to a data storage system, such as search engine data storage system 46 which is connected to network 48 by server 42f. In another implementation of the present invention, for example, personal computer 44 may include a browser or other application which also retrieves index data from the web page contents of each web page temporarily stored in cache and/or other volatile memory of personal computer 44. The retrieved data is then transmitted to search engine data storage system 46 or other designate data storage systems. In addition, in the implementation of the present invention, other distributed devices which access network 48, for example each servers 42a-42g, may include a software application which also retrieves index data from the web page contents of each web page temporarily stored in cache and/or other volatile memory of each of servers 42a-42g.

[0029] Indexing a full text document is performable by a variety of methods which are well known in the art. Various internet search engines and other software products use differing methods for indexing data. Some of the more common indexing methods are inverted file schemes, such as B+ trees, hashed indexes, signature files and simple keyord-out-of-context mechanisms. Preferably, in the present embodiment, web pages are indexed in such a manner that the indexed data is efficient and easily translatable by the various internet search engines into their proprietary format for subsequent use. For example, indexed web page data may be efficiently compressed by a simple inverted file mechanism utilizing a standard compression technique such as LZ or Huffman for transfer, since many times an indexed file's size can exceed the original document size. Upon receipt of the compressed file by a search engine, the compressed indexed data would be decompressed and translated into the format required to be integrated with the rest of the search engines data

[0030] Indexing data which is provided to data storage system 46 not only provides the search engine data storage system with indexing data for each page which has been accessed by a user, but also the frequency at which particular pages are accessed. Thereby, the indexed data received at data storage system 46 may also be utilized in a statistical manner. By charting the number times the index data for a particular web page is accessed at data storage system 46, a usefulness value which indicates the relative use of a web page may be determined.

[0031] With reference now to FIG. 3, there is illustrated a high level logic flowchart of a process for accessing web pages in response to a user request. As depicted, the process starts at block 64 and thereafter proceeds to block 66. Block 66 depicts a determination as to whether a web address is requested from a user. If a web address is not requested, the process iterates at block 66. If a web address is requested from a user, the process passes to block 68. Block 68 illustrates searching the Internet for the server containing the requested web address, as is well known in the art. Thereafter, block 70 depicts a determination as to whether the web address is located. If the web address is not located, the process passes to block 72. Block 72 illustrates returning an error message for display and thereafter the process ends. If the web address is located, the process passes to block 74. Block 74 depicts retrieving web page data for the requested web address from the server. Thereafter, block 76 illustrates storing a temporary copy of the retrieved web page data. Next, block 78 depicts providing access to the temporary copy of the retrieved data and thereafter the process ends.

[0032] Referring now to FIG. 4, there is depicted a high level logic flowchart of a process for acquiring indexed web page contents and providing indexed web page contents to a search engine database. As depicted, the process starts at block 50 and thereafter proceeds to block 52. Block 52 depicts a determination as to whether or not there are any high priority requests. In particular, requests such as accessing a new file, transmitting a temporary file and other such activities may be considered high priority requests. If there are high priority requests, the high priority requests are performed, as illustrated at block 54. If there are not any high priority requests the process passes to block 56. Block 56 depicts a determination as to whether or not any temporarily stored web pages are not yet indexed. If there are not any temporarily stored web pages that are not yet indexed, the process passes to block 52. If there are temporarily stored web pages that are not indexed, the process passes to block 58. Block 58 illustrates retrieving indexing data from textual contents/scripting of each temporarily stored web page. Next, block 59 depicts compressing the retrieved indexing data into a transmittable file format. Thereafter, block 60 illustrates transmitting the compressed indexing data file to a remote data storage system, such as a search engine data storage system, wherein the indexing data can be incorporated into a search engine database and/or utilized for statistical data.

[0033] As has been described, the present invention provides a method and system for providing indexed web page contents to a search engine database. In particular, indexed web page contents are retrieved from temporarily stored copies of web pages that have been requested by a user. However, while the invention is described in terms of retrieving web pages from a network such as the Internet, the invention is also applicable for other data available through the Internet. In addition, the invention is not limited to the network known as the Internet, but may be utilized for other networks. For example, the invention is applicable to retrieving indexing data for data files within a network such as an intranet, wherein in response to a user accessing files within an intranet, indexing data for the contents of the files is retrieved from temporarily stored copies of the accessed files and transmitted to a data storage system which incorporates the data into a intranet search engine. Further, it is to be understood that any distributed system which stores a temporary copy of data accessed by a user from a network can be utilized to retrieve indexing data from the temporarily stored copy and transmit the indexing data to a data storage system whereby the indexing data can be incorporated into a searchable database.

[0034] It is important to note that, although the present invention has been described in the context of a fully functional computer system, those skilled in the art will appreciate the mechanisms of the present invention are capable of being distributed as a program product in a variety of forms, and that the present invention applies equally regardless of the particular type of signal-bearing media utilized to actually carry out the distribution. Examples of signal bearing media include, but are not limited to, recordable-type media such as floppy disks or CD ROMs and transmission-type media such as analogue or digital communication links.

[0035] While the invention has been particularly shown and described with reference to a preferred embodiment, it will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the invention.