Method of surveilling internet communication

[0001] This application is a Continuation of International Patent Application No. PCT/IL01/00471 with an international filing date of 23 May 2001, the entire contents of the application which is hereby incorporated in its entirety.

[0002] The present invention is directed to a method and system for enabling surveillance and monitoring of networks communications by analysis of data traversing therethrough.

[0003] A huge amount of traffic is flowing through today's computer networks, not all of which is benign. Thus, an owner or supervisor of a given network may be most interested lo be able to track or “listen in” in real time in order to effectively monitor an/or secure the network. Such monitoring or surveillance can be achieved by connecting a probe to the network in order to monitor data traveling between two of more nodes (e.g., user workstations) on the network.

[0004] In a system where communication between two nodes is in a form of discrete packets, the network probe can “read” a packet of data in order to gather information, such as regarding the sources and the destination addresses of the packet, or the protocol of the packet In addition, statistical and related information can be computed such as the average or total amount of traffic of a certain protocol type during a given period of time, or the total number of packets being sent to or from a node. This information may be reported to a system administrator in real-time, or may be stored for later analysis.

[0005] Various attempts have already been made in this direction. For example, Clear View Network Window, a software program available from Clear Communications Corporation, of Lincolnshire, Ill., U.S.A, allegedly offers predictive/proactive maintenance, intelligent root-cause analysis, and proof-of-quality reports. However, the output is designed for network fault management, which is not the same as “tapping” into a communication between nodes in the network. Thus, the Clearview system does not allow monitoring of data transferred between two nodes In the network With regard to contents or characteristics.

[0006] Livermore National Laboratory, Livermore, Calif., U.S.A, developed a group of computer programs to protect the computers of the U.S. Department of Energy by “sniffing” data packets that travel across a local area network, The United States Navy used one of these programs, known as the “iWatch” program, in order to wiretap on communications of a suspected computer hacker who had been breaking into computer systems at the U.S. Department of Defense and NASA. The iWatch program uses a network probe to read all packets that travel over a network and then “stores” this information in a common database. A simple computer program can then be written to read through the stored data, and to display only predefined “interesting” pieces of information.

[0007] Whenever an interesting piece of information is found, the stored data is rescanned and a specific number of characters located at both sides of the “interesting” piece are reported. These interesting characters are then reviewed in order to determine the content of the message and used as, a guide to future monitoring activity,

[0008] This system is restricted to history analyze of user activities and does not enable complete “tapping” of all user activities and full simulation of the users surfing activity.

[0009] Three major problems are encountered in the way of achieving continuous and reliable tracking:

• • [0010] (a) Individual browsers do not report all the activities performed to a web server. For example, when a browser loads web pages from its browser cache space or from a proxy server, it does not send requests to any “remote” web server through the cyberspace autostrade;
  • [0011] (b) Application programs designed to perform certain features by web browser of one manufacturer are usually not compatible with those manufactured by another vendor because browser interface mechanisms are different and proprietary to each one of them; and
  • [0012] (c) Individual browsers send their requests to web servers in a non-systematic order in other words, with regard to a given web server, a preceding request has no relation to a subsequent request. In processing of requests, a web site has no control over the sequences of the requests.

[0013] In an attempt to overcome these problems, U.S. Pat. No. 5,951,643 refers to a mechanism for dependably organizing and managing information for web synchronization and tracking among multiple consumer browsers.

[0014] However, this solution is limited to tracking activities of identified users, who agreed to be “tapped” and willingly cooperated and be connected to the host with designated application.

[0015] It is thus the prime object of the invention to provide a monitoring and surveillance method and system enabling network communication suppliers to tap any user connected to the network.

[0016] It is a further object of the invention to provide a tapping methodology enabling network communication suppliers to watch in real time all user activities while communicating a network.

[0017] It is a still further object of the invention to enable web-site owner to monitor and tap users contacting their web site

[0018] Thus provided according to the present invention is a method of tracking a network communication line by network probe terminal (“terminal agent”) simulating a browser (“original browser”) activity of a given terminal comprising the steps of accessing the network communication line, tracing TCP/IP data packets routed through the communication line, selecting TCP/IP data packets relating to a given IP address; (“identified data packets”), selecting from the identified data packets current requests for new connections (“original requests”), selecting from the identified data packets current web-page components indicating new addresses (“new navigation components”), dividing the new navigation components into two categories, embedded objects or frames (“false new components”), hyperlinks (“true new components”), dividing tie original requests into original requests matching true the new components, or original requests failing to match any new connection components and belonging to HTTP or POST type as “primary requests”, original requests matching the false components as “secondary requests”, selecting from identified data packets, HTML data files relating to primary requests; (“respective primary responses”), generating “virtual” secondary requests according to the respective secondary responses, selecting from identified data packets responses relating to secondary virtual requests, (“respective seconday responses”) and simulating web page presentation on the terminal agent according to the respective secondary responses.

[0026] Referring to FIG. 1, let us assume that terminals 01, 02 . . . are connected to the same communication line, where the communication line is used as internal network (“Intranet”), or external network such as the Internet. According to the present invention it is proposed to connect a designated network probe (hereinafter called “the Terminal Agent”) to the data is communication line. Alternatively, the terminals 01, 02 etc., and the terminal agent may be connected to different data communication lines, or located at different local networks.

[0027] The general scheme of the terminal Agent operation is illustrated in FIG. 2.

[0028] The Terminal Agent is exposed to all data frames passing through the communication line. The data frames may contain information transferred between the terminals or external data transmission to external sources such as internet servers.

[0029] Let us further assume that the “Owner” of the data communication line, such as ISP or network of a private organization, is interested in monitoring in real time, the actual communication activities of a given terminal when surfing the internet.

[0030] The operation of the Terminal Agent is to first analyze the data frames for tracing TCP/IP data packets. As illustrated in FIG. 3, the data analysis is processed according to the different protocol hierarchy (see RFC 0793 of the internet protocol), namely, first to analyze the local network protocol, filtering external data transmission (“gateway level”) then identifying internet protocol (IP) data frames, and finally detecting TCP(“Transition Control Protocol”) data packets of the “host level”.

[0031] Upon analyzing the IP HEADER of the data packets, the IP addresses of the requesting terminal and of the message destination are identified. The owner of the communication line can easily relate the IP address to the users terminals. Therefore it is possible to filter out all other irrelevant data packets and restrict further processing to data transmission of one selected terminal (hereinafter called “the identified data packets”).

[0032] The identified data packets are further analyzed according to the RFC 079 specification enabling; full management and control of data communication ports.

[0033] According to known routines of managing TCP data communication ports, as processed by conventional browsers, e.g. the Internet Explorer, the terminal which operates the browser is the original source of all data transmission. For example let us assume that the terminal placed a request for YAHOO0! home page, which request is delivered through the network to YAHOO! server. In response, the server sends an. HTML data file containing all information of yahoo home web page components. Accordingly the browser sends new requests for receiving all components of the web-pate by opening new communication “virtual” ports, where each port is used for transmitting different components of the same web-page. An “outsider” terminal, exposed to all data requests and respective responses is unable to differentiate between initial “primary” requests, e.g. requesting the complete YAHOO! home page and “secondary” requests for receiving the components thereof. For simulating the activity of the original browse probe terminal it is essential to identify the primary requests as such.

[0034] FIG. 4 Illustrates the process for differentiating the primary requests from the secondary requests. Primary requests are originated from different operations such as entering a new URL by the user, choosing a hyperlink, etc. Therefore, in order tat detect same one must analyze the previous information transmitted to the same IP address. All new navigation components (addressing the browser to new location) of the web page received by the terminal are sorted according to their type, all embedded objects, frames, etc., are marked as “false” components, while hyperlinks are marked as “true” components. All data is stored in the incoming buffer responses database for later use.

[0035] When identifying a request for a new connection according to TCP analysis, the request is examined according to the respective navigation components (RNC) in the incoming respond buffer. If the RNC is marked as “false” the request is ignored; if the RNC is marked as “true” the request is classified as primary; otherwise, if there is no RNC relating the said request, the connection type should be identified, If the connection is of an HTML type, or “post” type, it is classified as a primary request,

[0036] In order to view and monitor the activities of a terminal, all “original” browser activities must be reconstructed. For that purpose it is suggested to use a “virtual” browser. This virtual browser possesses all the capabilities of a “real” browser to download in real time web pages from the Internet. However its connection with the Internet is virtual in the sense that no actual date exchange with the Internet servers is preformed, but only simulating the activities of the original “real” browser.

[0037] The first function of the virtual browser is illustrated in FIG. 5. The browser is receiving all primary requests of the “real” browser. These primary requests and the respective primary responses from the Internet are analyzed and processed according to the conventional browser operation. However the outcome of secondary virtual requests (in conventional browser used to complete the process of downloading web page components) are not transferred directly as usual through the Internet to the appropriate server but stored in a the virtual “secondary” requests buffer database

[0038] Although the virtual browser connection is not “real”, all TCP protocol management of opening and controlling ports connection is processed by the terminal agent as if the connections are “real” ones.

[0039] The final process of simulating and presenting the web pages in the virtual browser is further illustrated in FIG. 6. All original secondary responses coming through the communication line are analyzed and recorded in tie incoming responses buffer database. The virtual requests are compared to the respective secondary responses stored in the incoming responses buffer database, by the order of their arrival. If the respective secondary responses already exists in the buffer, these responses are transferred to the virtual browser, and processed (according to conventional browser operation) to present the visual picture, of the respective web page components. As a result, the terminal agent is simulating in real time the exact process of downloading Internet web pages as it has been performed by the original terminal.

[0040] In case the respective responses do not appear in the incoming responses buffer database, activity of an original local cache is deduced. If the original local cache was not used with respect to said virtual request, it is suspend in the buffer database until the original secondary respective responses arrive. Otherwise, if the real local cache was used relating to this respond, the local cache of the virtual browser Is examined, and if respective secondary responses exist in the local cache, then the respective respond is transferred to the virtual browser and processed as described above, In case the respective responses do not exist in the virtual cache, either of the following alternatives may be applied. According to one, “passive” version of the terminal agents, no further action is taken to find the “missing” respond, and an “error” message will appear at the agent terminal instead of the web page component which appeared in the real terminal. According to this version, the simulation of the real terminal is not complete but the tapping activity is undetectable. According to another, “active” version, the terminal agent addresses the web page server to request the “missing” respond. Although this version enables the terminal agent to present more exact picture of the real terminal activities, it is traceable for more experienced terminal users, who are able to detect the tapping activity.

[0041] According to a further mode of implementation of the of the present invention, it is proposed to tap not only to related web page data packets, but to trace also related messages data packets e.g. e-mail or chats. To enables such tapping, the same method and principals as described above are applied at request for receiving and sending messages through the network other than requests for web pages. The process of analyzing such requests and the respective responses is more streamlined since there is no need to check the cache memory activity, as by definition such information is always new.

[0042] Finally, it should be appreciated that the above-described embodiments are directed to Internet communication environment. However, the invention in its broad aspect is equally applicable to computerized network communication in general, such as satellite, cellular and others.

[0043] While the above description contains many specificities, these should not be construed as limitations on the scope of the invention, but rather as exemplification of the preferred embodiments. Those skilled in the art will envision other possible variations that are within its scope, Accordingly, the scope of the invention should be determined not by the embodiments illustrated, but by the appended claims and their legal equivalents.