System and method for capturing browser sessions and user actions

[0001] The present invention relates to a system and a method for capturing user actions, browser sessions, and in particular, for such a system and method in which user input device activity and user interface responses are captured.

[0002] The Internet, as well as local or otherwise restricted networks such as intranets, are increasingly being used to provide interactive experiences for computer users. For example, rather than providing merely static Web pages to such users, interactive experiences such as chat or real-time text messaging facilities, video-conferencing and/or audio-conferencing functions and other interactions with the user, are being provided through Web sites or Web applications. These increasingly sophisticated functions enable the user to interact with an operator at the Web site and/or other users in a realistic manner.

[0003] The desire to decrease costs of technical and/or other types of customer support, as well as to increase the availability of these services, has led to the sharp growth of Trade Floor and Call Center activities through such Web sites/applications and/or other types of network functions. However, these human agents must be monitored with regard to their interactions with the users, in order to provide adequate QA (Quality Assurance) for these agents and services.

[0004] This change toward supplying customer support services through Web sites on the Internet and/or other types of networks dictates the need for quality assurance technology and tools, which enable Web site operators and customer support (contact) center managers to understand the needs of users as customers or clients, and to perform quality assurance on their Web sites and network services.

[0005] On the other hand, the users of such Web sites are also performing an increased number of actions on the Web such as buying, selling, and entering information into different forms. Currently, the only mechanism for archiving these actions is to print hard copies of each Web page onto paper, or save individual Web pages without regard to the sequence of actions which produced these Web pages. Furthermore, for such real-time communications as chat functions, currently available Web browser technology does not provide any suitable mechanism for saving data related to the actual chat. Thus, neither the users of such Web sites (end-users) nor the operators of these sites currently have useful tools for collecting information about interactive sessions between the user and the Web site.

[0006] The background art does not teach or suggest a solution to the problem of collecting information about an interactive session over a network. The background art also does not teach or suggest a solution to the problem of archiving Web site sessions. In addition, the background art does not teach or suggest a solution to the problem of quality assurance/quality control for interactions between a user and an agent through a Web site and/or other type of network interactions.

[0007] The present invention overcomes these problems of the background art by providing a system and method for the collection of data with regard to the actions of the user during an interactive session over a network. The present invention preferably features a client which is operated by the computational device of the user, and which more preferably collects information both about user input device actions and the reactions which are displayed through the user interface. The user interface is preferably a GUI (graphical user interface), but optionally and more preferably is a Web browser interface. Alternatively, the user interface is any type of interface which becomes altered upon receiving user input through a user input device.

[0008] The client preferably collects these different types of data and then more preferably sends the data to be stored in a central database. Optionally and preferably, the data is organized according to a particular structure before being stored in the database. Also optionally, such organization of the data is performed by the client, such that the data is organized in a message which is sent to the storage computational device operating the database. Alternatively, the data is sent to the storage computational device before it is organized into the final structure, such that the storage computational device and/or a third entity performs the organization into the storage structure.

[0009] According to an optional embodiment of the present invention, the client is outside an organizational network, such that the collected data must pass through a firewall. Preferably, the collected data is sent to a proxy server before entering the organizational network. More preferably, such a proxy server is located in the DMZ (demilitarized zone) between an outer and an inner firewall, such that the client cannot directly access a computational device inside the organizational network. Instead, the proxy server at least partially processes the collected data, which may then optionally be retrieved by the storage computational device from within the inner firewall for storage in the database.

[0010] The present invention is suitable for interacting with substantially any type of user input device. Examples of such user input devices include, but are not limited to, a pointing device such a mouse for example; a keyboard and/or keypad; handwriting or stylus-based input; or any type of mechanism which can be at least partially controlled by the user.

[0011] Hereinafter, the term keypad refers to any keyboard in which at least one key features a plurality of alphanumeric characters and/or symbol portions, which may also be termed a reduced keypad. One non-limiting example of such a keypad is typically found on cellular telephones, for entering numeric and/or alphanumeric characters, and/or symbol portions for languages with an ideographic writing system.

[0012] Examples of suitable applications for implementation with the present invention include, but are not limited to, the analysis of the actions of users on a Web site for Web site operators, for example for Quality Assurance; and archiving interactive sessions with a Web site by the users for their own purposes.

[0013] Hereinafter, the term computational device refers to any type of computer hardware system and/or to any type of software operating system, or cellular telephones or any type of hand-held device such as a PDA (personal data assistant), as well as to any type of device having a data processor and/or any type of microprocessor, or any type of device which is capable of performing any function of a computer.

[0014] For the present invention, a software application or program could be written in substantially any suitable programming language, which could easily be selected by one of ordinary skill in the art The programming language chosen should be compatible with the computational device according to which the software application is executed. Examples of suitable programming languages include, but are not limited to, Delphi, C, C, C and Java.

[0015] Hereinafter, the term Web browser refers to any software program which can display text, graphics, or both, from Web pages on World Wide Web sites. Hereinafter, the term Web page refers to any document written in a mark-up language including, but not limited to, HTML (hypertext mark-up language) or VRML (virtual reality modeling language), dynamic HTML, XML (extended mark-up language) or related computer languages thereof, as well as to any collection of such documents reachable through one specific Internet address or at one specific World Wide Web site, or any document obtainable through a particular URL (Uniform Resource Locator). Hereinafter, the term Web site refers to at least one Web page, and preferably a plurality of Web pages, virtually connected to form a coherent group. Hereinafter, the term Web server refers to a computer or other electronic device which is capable of serving files and data according to the HTTP protocol to a Web browser.

[0016] Hereinafter, the term applet refers to a self-contained software module written in an applet language such as Java or constructed as an Active control. Hereinafter, the term client refers to any type of software program and/or code and/or other instructions, which are operated and/or performed by the computational device of the user.

[0017] Hereinafter, the term network refers to a connection between any two or more computers which permits the transmission of data.

[0018] Hereinafter, the phrase display a Web page includes all actions necessary to render at least a portion of the information on the Web page available to the computer user. As such, the phrase includes, but is not limited to, the static visual display of static graphical information, the audible production of audio information, the animated visual display of animation and the visual display of video stream data.

[0019] Hereinafter, the term user refers to the person who operates the Web browser or other GUI interface and interacts with the system of the present invention by operating a computational device.

[0020] The invention is herein described, by way of example only, with reference to the accompanying drawings, wherein:

[0021] FIG. 1 is a schematic block diagram of an exemplary system according to the present invention;

[0022] FIGS. 2A and 2B are flowcharts of exemplary methods for capturing data according to the present invention;

[0023] FIG. 3 is a flowchart of an exemplary method for playing back data according to the present invention; and

[0024] FIG. 4 is a schematic block diagram of a second exemplary system according to the present invention.

[0025] The present invention is of a system and method for the collection of data with regard to the actions of the user during an interactive session over a network. The present invention preferably features a client which is operated by the computational device of the user, and which more preferably collects information both about user input device actions and the reactions which are displayed through the user interface. The user interface is preferably a GUI (graphical user interface), but optionally and more preferably is a Web browser interface. Alternatively, the user interface is any type of interface which becomes altered upon receiving user input through a user input device.

[0026] The client preferably collects these different types of data and then more preferably sends the data to be stored in a central database. Optionally and preferably, the data is organized according to a particular structure before being stored in the database. Also optionally, such organization of the data is performed by the client, such that the data is organized in a message which is sent to the storage computational device operating the database. Alternatively, the data is sent to the storage computational device before it is organized into the final structure, such that the storage computational device and/or a third entity performs the organization into the storage structure.

[0027] According to an optional embodiment of the present invention, the client is outside an organizational network,(Intranet or VPN (virtual private network)), such that the collected data must pass through at least one firewall. Preferably, the collected data is sent to a proxy server before entering the organizational network. More preferably, such a proxy server is located in the DMZ (demilitarized zone) between an outer and an inner side of the DMZ, such that the client cannot directly access a computational device inside the organizational network. Instead, the proxy server at least partially processes the collected data, which may then optionally be retrieved by the storage computational device from within the inner firewall for storage in the database.

[0028] The present invention is suitable for interacting with substantially any type of user input device. Examples of such user input devices include, but are not limited to, a pointing device such a mouse for example; a keyboard and/or keypad; handwriting or stylus-based input; or any type of mechanism which can be at least partially controlled by the user.

[0029] In addition, the present invention is useful for the collection of data about substantially any type of user interface function. Examples of such user interface functions include but are not limited to any type of GUI window activity; activity with GUI gadgets such as buttons, sliders or any function provided through a GUI window; the display of any image and/or text, including but not limited to Web pages and/or any component thereof; information provided through an audible interface such as a synthesized voice; information provided through the display of video data; and any type of information which is provided through, or otherwise detectable by, the operating system of the user computational device.

[0030] The principles and operation of the method according to the present invention may be better understood with reference to the drawings and the accompanying description. It should be noted that the present invention is described with regard to Web browsers for the purposes of clarity only and without any intention of being limiting.

[0031] Referring now to the drawings, FIG. 1 shows an illustrative system 10 for collecting data about user actions. A user interacts with a Web browser 12 being operated by a user computational device 14. These interactions may optionally include, but are not limited to, user input device actions, displaying Web pages and any type of GUI (graphical user interface) activities. As such, these interactions preferably include both user actions, such as entering information through a keyboard and/or clicking on or otherwise selecting a GUI gadget through a pointing device; and reactions to these user actions, such as displaying a Web page for example. These interactions are preferably collected by a client 16 which is operated by user computational device 14.

[0032] Client 16 is optionally implemented as an applet, such as a Java applet or ActiveX control (Web browser 12 forms the container); or alternatively as a software application. The ActiveX control and the software application may both optionally use the Web browser COM interface or perform hooking on the OS, in order to capture user action and sniffing on the network layer in order to capture Web browser sessions.

[0033] User computational device 14 is preferably connected to a network 18. Network 18 may be the Internet, for example, as shown with regard to the exemplary embodiment of system 10 in FIG. 1. User computational device 14 is preferably able to communicate, directly or indirectly, with a storage computational device 20 for transmitting the collected data, collected by client 16. Storage computational device 20 is preferably able to operate a database 22 for storing the collected data.

[0034] According to an exemplary but preferred embodiment of the present invention, storage computational device 20 organizes the data according to a particular structure before the data is stored in database 22. Alternatively, such organization may optionally be performed by client 16 before transmission to storage computational device 20. In any case, such organization preferably includes decomposing the captured data into a plurality of components. Most preferably, such organization includes entering specific types of data into particular fields, such that each type of event has related data for entry to these fields.

[0035] Client 16 is optionally and more preferably able to detect substantially any type of user input device activity including, but not limited to, pointing device activities such as clicking on or otherwise selecting a GUI gadget or any other portion of the GUI, for example with a mouse. Other types of activities for which data may be collected include detecting keystrokes from a keyboard and/or keypad, as well as information entered as handwriting or stylus-based input. Voice input may also optionally be collected, as well as input for any type of mechanism which can be at least partially controlled by the user.

[0036] In addition, the present invention is useful for the collection of data about substantially any type of user interface function. Examples of such user interface functions include but are not limited to any type of GUI window activity; activity with GUI gadgets such as buttons, sliders or any function provided through a GUI window; the display of any image and/or text, including but not limited to Web pages and/or any component thereof; information provided through an audible audio interface such as a synthesized voice; information provided through the display of video data; and any type of information which is provided through, or otherwise detectable by, the operating system of user computational device 14. Examples of GUI window activities include but are not limited to scrolling through the window, and data about the active window and its properties such as size, location and so forth.

[0037] Client 16 is preferably able to filter the collected data, more preferably in order to separate the collected data into components for transmission to storage computational device 20. In addition, client 16 preferably also collects and transmits timing information to storage computational device 20, such as a timestamp for example, in order for storage computational device 20 to be able to synchronize the different types of events and associated data.

[0038] More preferably, storage computational device 20 captures each action of the user and each reaction of the user interface as a separate event, while the associated data is stored as parameters for that event. Therefore, storage computational device 20 most preferably does not capture a screenshot as a bitmap or other type of image. Instead, the meta-data is preferably captured and organized. More preferably, only the meta-data is captured. For example, a Web page could optionally be captured as HTML code and one or more images, such that the Web page could be reconstructed, but is more preferably not captured as a single complete image.

[0039] More preferably, storage computational device 20 is able to organize and display the stored data in order to be able to reconstruct the interactive session with the user. Most preferably, storage computational device 20 is able to provide this data organized such that the interactive session is replayed. Optionally and most preferably, the interactive session can be replayed to any Web browser, as the reconstructed data is most preferably provided in a format which is suitable for being replayed by any Web browser. For this preferred embodiment, the data is stored as a series of events, for which the data stored in the field(s) forms the parameters for each event, such that the Web browser can optionally be used to read the data event by event, for replaying the interactive session.

[0040] A separate display client (not shown) may optionally be provided for displaying the events and hence replaying the session. Alternatively, client 16 may also optionally be used to replay the session. Such a display client could optionally be implemented as an ActiveX control, which preferably connects to storage computational device 20 and pulls data from it. Such an ActiveX control would optionally separate the HTML data from the events, push the HTML pages to a Web browser component of the display client, and then simulate the user events synchronously with the HTML pages. Playback of Web-based sessions could optionally be done by using a user name and password to connect to storage computational device 20.

[0041] According to an optional but preferred embodiment of the present invention, the user input device actions and the user interface reactions are captured through the operating system of user computational device 14. An example of a method for capturing such data is given below with regard to the Windows operating system (Microsoft Inc, USA), it being understood that this is being given as an illustrative example only and without any intention of being limiting. The API calls and WIN32 functions that are being used are native operating system functions that are supported by all Windows operating system environments.

[0042] For each running Microsoft Internet Explorer instance as an example of Web browser 12, a Hook function is set in order to capture the keyboard strokes and mouse movements and clicks. The captured data is placed into a binary file in a structure that contains the details about the activity and the time stamp of the operation. Such capture actions preferably occur only after the Web page is loaded completely by Web browser 12.

[0043] The Hook function for keyboards is optionally and preferably implemented by calling the Win32 function SetWindowsHookEx with the WH_GETMESSAGE parameter set. The Hook function for the mouse is optionally and preferably implemented by calling the Win32 function SetWindowsHookEx with the WH_MOUSE parameter set.

[0044] The keyboard callback procedure preferably receives all browser keyboard actions (and other window messages), which then waits for the next window message, which is WM_SYSKEYDOWN; WM_KEYDOWN; WM_SYSKEYUP; or WM_KEYUP. Each such message contains data about which key was pressed and which window received the character. By detecting and capturing those messages, the character is known, as well as where the user typed. The differences between SYSKEY and KEY is that SYSKEY commands contain information about system keys like CTRL, SHIFT and so forth..

[0045] For the mouse callback procedure, preferably all browser mouse actions are sent to this callback which waits for the next window messages, which is preferably one of the following messages: WM_MOUSEMOVE; WM_LBUTTONDOWN; WM_LBUTTONUP; WM_MOUSEWHEEL; WM_RBUTTONDOWN; WM_RBUTTONUP; WM_MBUTTONDOWN; WM_MBUTTONUP; or WM_LBUTTONDBLCLK.

[0046] The captured data is then preferably stored according to a particular organizational structure as previously described. For example, the mouse event structure preferably includes the following fields: _timeb now; DWORD dwFlags;DWORD dx; DWORD dy; DWORD dwData; and LPARAM dwExtralnfo. The keyboard event structure preferably includes the following fields: timeb now; BYTE bVk; BYTE bScan; DWORD dwFlags; and LPARAM dwExtralnfo. These are standared Win32 structures that hold data about the mouse event. This structure is described in Microsoft Win32 documentation.

[0047] For the purpose of replaying the session and/or otherwise playing back the stored data, preferably for each Internet explorer running instance (as an example of Web browser 12), after the Web page is loaded, preferably a hook file related to this Web page may optionally be played. The replayed data is preferably obtained from the file as recorded by the capture component and containing the structure described above. The time stamps that were entered at the recording stage are preferably used to synchronize the playing operation. Each stored event holds information about the corresponding action which occurred. For example, a mouse event describes the location of the mouse on the screen and whether an action was performed, such as whether the mouse left button was pressed. If a time stamp is placed on each event at the capturing stage, these events can be simulated (played back) according to the delta of time which passed between one event to the other, as determined according to the timestamps.

[0048] In order to control the mouse and keyboard functions of the computational device being used to play back or display the stored data, the following API functions are preferably used: mouse_eventoperates the mouse as if an end user is activating it; keybd_eventoperates the keyboard as if an end user is activating it.

[0049] Turning back to FIG. 1, according to an optional but preferred embodiment of the present invention, storage computational device 20 and database 22 are preferably separated from user computational device 14 and related functions. More preferably, such a separation is performed with at least one firewall 24. According to the preferred embodiment shown, two such firewalls 24 are used, a first outer firewall 24 to separate between an organizational entity 26 and network 18; and a second inner firewall 24 to separate a DMZ 28 from the remainder of organizational entity 26. DMZ 28 preferably contains a log server 30 for receiving the data from user computational device 14 and for optionally organizing such data, and a Web server 32. In this preferred implementation, storage computational device 20 would pull the received data from log server 30.

[0050] FIG. 2 shows a flowchart of an exemplary method for collecting data according to the present invention. This method is divided into two parts for clarity of explanation only: FIG. 2A shows the capturing part of the method, while FIG. 2B shows the archiving part of the method.

[0051] As shown in FIG. 2A, preferably in stage 1, the user connects with the Web browser to a requested Web site according to the URL of the Web site. The Web site is optionally and more preferably served by the Web server of the organizational entity of FIG. 1, although this is not necessary. In stage 2, preferably a server of some type, whether operated through the Web server or otherwise provided, determines whether the client of FIG. 1 is installed on the user computational device. If not, then the client is downloaded. Otherwise, in stage 3, the client preferably connects to a log server on port 80. The log server may optionally be the storage computational device of FIG. 1.

[0052] In stage 4, the client determines whether a session has been established between the user computational device and the Web server which is being monitored. If such a session has not been established and/or is disrupted, then preferably the client disconnects from the log server. Otherwise, in stage 5, the user actions and GUI reactions (such as displayed Web pages for example) are preferably captured; messages are built; and a time stamp is added. In stage 6, the captured data is sent by the client the log server. Stages 5 and 6 are optionally repeated at least once, and more preferably are repeated until the session is ended.

[0053] FIG. 2B shows an exemplary archiving method according to the present invention. The log server listens on port 80 for incoming data in stage 1, preferably in the form of messages as previously described. Also, as previously described, the log server may optionally be the storage computational device of FIG. 1. However, if the log server is different from the storage computational device of FIG. 1, in stage 2, more preferably the log server connects to the storage computational device.

[0054] In stage 3, the log server determines whether new data has arrived. If new data has not arrived, the log server preferably waits. Otherwise, the log server preferably determines whether a request to start recording a session has been received in stage 4. If such a request has been received, then in stage 5, the log server preferably inserts data into catch, such that the log server keeps the received data in a temporary storage buffer, or catch.

[0055] In stage 6, the storage computational device preferably determines whether data can be pulled from the log server. If such data is available in the catch of the log server, then the storage computational device preferably retrieves and stores the data in the database, in stage 7. This process is optionally repeated, preferably as often as necessary to retrieve any received data.

[0056] FIG. 3 shows an exemplary method for playing back data according to the present invention. As shown, in stage 1, the user preferably uses a Web browser to connect to the address (URL) of the log server, which is operating in this example as the playback server. In stage 2, the log server determines whether the playback client is installed on the user computational device, for playing back the captured data. If not, and if the user is authorized, then the playback client is preferably downloaded.

[0057] In stage 3, once the playback client is available, then the user selects a Web-based session to play back. In stage 4, the playback client connects to the log server, for example through port 80. In stage 5, the log server optionally notifies the storage computational device, if the two servers are separate components, and then waits to receive data.

[0058] In any case, the log server sends the data to the playback client in stage 6. In stage 7, the playback client preferably checks the data status. If there is no data, then the playback client preferably disconnects from the log server, and more preferably also determines whether a session is still present. Otherwise, in stage 8, the playback client preferably pushes Web pages to the Web browser of the user computational device and replays the data for simulating the previously performed actions of the user for replaying the session.

[0059] The methods of FIGS. 2 and 3 may optionally be performed in the presence of absence of a DMZ and/or one or more firewalls, with similar functioning, apart from the requirement for two separate servers for receiving and storing the data in the case of an implementation with the DMZ as shown in FIG. 1.

[0060] According to optional embodiments of the present invention, both the capture and storage functions may optionally be performed by the same computational device. For this implementation, the capture client, the storage server and the play back functions may all optionally be implemented as separate processes which are operated by the same computational device. This implementation is shown for example with regard to FIG. 4, a system 34 features a Web server 36 for serving Web pages through the Internet 38 or other network. These Web pages are downloaded to a user computational device 40, which features a Web browser 42 for displaying these Web pages.

[0061] User computational device 40 preferably also features a capture component 44 for capturing the previously described user actions and user interface reactions; and a playback component 46 for playing back the captured data, also as previously described. Rather than sending the captured data to a remote server, capture component 44 preferably stores the captured data in a database 48. User computational device 40 can therefore preferably be used to both capture and playback the data.

[0062] Optionally and more preferably, Web browser 42 preferably contains both capture component 44 and playback component 46, to enable any client to playback the relevant sessions recorded by that client.

[0063] While the invention has been described with respect to a limited number of embodiments, it will be appreciated that many variations, modifications and other applications of the invention may be made.