System and method for a computer based cooperative work system

[0001] 1. Field of the Invention

[0002] The present invention generally relates to computer supported collaborated work and, more particularly, to intelligently collaborating with computers in a network with one or more agents.

[0003] 2. Background Description

[0004] With ubiquitous connectivity on the horizon, collaborative computing promises to become one of this new century's core applications. People will be more and more involved in Computer Supported Cooperative Work (CSCW) because of the pressure from companies to improve their product-development and decision making process and because of the convenience brought by the information super-highway.

[0005] There are four modes conceptualized by CSCW researchers on how people work; synchronous mode, distributed synchronous mode, asynchronous mode, and distributed asynchronous mode. Synchronous mode refers to the situation where activities occur at the same time and in the same place. Distributed synchronous mode refers to the situation where activities occur at the same time but at different places. Asynchronous mode refers to the situation where activities occur at different times in the same place. Distributed asynchronous mode refers to the situation where activities occur at different times and places.

[0006] Many computer systems support simultaneous interaction by more than one user. However, most of them support multiuser interaction in a way that prohibits cooperation; that is, they give each user the illusion that the user is the only one using the system. To support and encourage cooperation, cooperative applications must allow users to be aware of the activities of others. The purpose of a cooperative multiuser interface is to establish and maintain a common context, allowing the activities or events associated with one user to be reflected on other users' screens. For example, Lotus® Sametime is a family of real-time collaboration products which provides instant awareness, communication, and document sharing capabilities, bringing the flexibility and efficiency of real-time communication to the business world.

[0007] With awareness of coworkers, partners, or customers online, users can communicate in a variety of ways. However, a direct reflection of all the activities on other users' screen is not approachable. The first reason is that it wastes communication bandwidth, especially when users are far apart and the amount of data to be transmitted, such as video data, is huge. The second reason is that many users may not like the situation that his or her activities are broadcasted to all the other members of the team. The third reason is that each user is concentrating on his or her own work and does not have the energy and motivation to monitor every movement of other users.

[0008] Thus, it is critical for CSCW interface to analyze activities of a given user, detect that important events have occurred, and only reflect necessary events to other agents.

[0009] Event perception will be even more important to CSCW in the pervasive computing world, where the dominance of the traditional PC as the primary computing resource is replaced by a large collection of devices with embedded computing. These intelligent, interconnected devices will be seamlessly embedded within our offices, constantly sensing and reacting to the environment. The information provided by these pervasive devices within an office environment will be very important in CSCW applications.

[0010] Autonomous agents are expected to be of great value to a CSCW system and a certain amount of future research on CSCW will be centered on multi-agent aspect of groupware. A multi-agent approach to CSCW can capture the dynamics of a team work and even re-shape its form and characteristics. The automation brought by CSCW agents will dramatically reduce certain types of frictional costs during team work. Furthermore, the intelligence of a multi-agent CSCW system will be able to keep the privacy of its user and the security of each user's local work.

[0011] Collaborative computing systems, sensing devices of various kinds that provide input to computer systems, and knowledge base or expert systems are generally known in the prior art. Some examples include the following:

[0012] U.S. Pat. No. 5,996,002 to Katsurabayashi et al. discloses a collaborative work support system that is performed on plural computers, each of which is assigned to an operator, and supports collaborative work in which the plural computers display common data and each operator operates the displayed common data through his or her own computer.

[0013] U.S. Pat. No. 5,948,057 to Berger et al. discloses a method for computer-supported matching of a number of data copies of a stored data file stored in at least one computer, in the reintegration of a number of data copies that were changed during decoupled work phases by uses of a shared work environment, ad thus exhibit inconsistencies. The reintegration is conducted so that the number of matchings is reduced on the basis of protocol data files.

[0014] U.S. Pat. No. 5,781,732 to Adams discloses a shared document framework for use by an application program that provides collaborative access to a shared document by means of a caucus service associated with the shared document. The caucus service receives messages from caucus members and broadcast transmits them to all caucus members in global order.

[0015] U.S. Pat. No. 5,708,853 to Sanemitsu discloses an integrated circuit (IC) card having a camera, a microphone and a modem for transmitting electrical signals from the camera and microphone to a telephone or communication line, or transmits signals received from the communication line to a terminal, such as a personal computer (PC).

[0016] U.S. Pat. No. 5,396,265 to Ulrich et al. discloses a tactile computer input device which simulates an object being designed. The input device is used with a computer aided design (CAD) system and allows a user to manually manipulate the input device as if it were the object under design.

[0017] U.S. Pat. No. 5,068,645 to Drumm discloses a device for controlling a cursor on a data terminal display screen. The device is in the form of a headset and includes an orientation sensor which provides an electrical signal related to the orientation of the device without it being adjacent to any fixed surface.

[0018] U.S. Pat. No. 5,418,889 to Ito discloses a knowledge base generating system that includes a knowledge base having a first knowledge base containing sets of causal relation knowledge described cause and effect relations of events taking place within a target machine, and having a second knowledge base containing sets of membership knowledge describing a structure of members of the target machine, each event of the cause and effect relations having data to identify one of the members, so that the first and second knowledge bases have mutually retrievable data.

[0019] U.S. Pat. No. 5,353,384 to Yoshida discloses an expert system which includes a first knowledge base for storing detailed knowledge, a second knowledge base for storing compiled knowledge, an inference engine for solving a problem using the second knowledge base, and an analysis engine for extracting knowledge having high utilization from the first knowledge base and storing the extracted knowledge in the second knowledge base.

[0020] U.S. Pat. No. 5,295,067 to Cho et al. discloses a system for order planning that translates an order configuration into a list of bills of materials list. The system operates based on a first logical specification of relationships between models, optional features or device codes and required material components.

[0021] It is therefore an object of the present invention to provide a new event perception algorithm that can perceive events in the user's workspace.

[0022] It is another object of this invention to provide an architecture of agent mediated CSCW.

[0023] A further object of this invention is to provide an improved apparatus, system, and method for computer collaboration over a network.

[0024] According to the invention, there is provided an agent mediated CSCW system that can create a sense of group work and at the same time keep the privacy and maintain the security of each user. A multi-agent negotiation process is used in the system to reduce fractions among group members during the geographically distributed team work. A markup language, such as XML (eXtended Markup Language), is used in the system to encode communication messages. (See, for example, http://www.ibm.com/developer/xml.) Event perception is an important task in agent mediated CSCW system which uses an eigen space to perform the event perception task. For the case where the number of devices is large, an eigen pyramid is constructed which can be used to discriminate different events. For more information on the eigen space approach, see Numerical Recipes in C by William H. Press, Saul Teukolsky, William T. Vetterling, and Brian P. Flannery.

[0052] In this disclosure, we will present the system architecture of an agent mediated CSCW system and study in detail its event perception issues. Referring now to the drawings, and more particularly to FIG. 1, there is shown a block diagram of one preferred system architecture. Block 101 is the server, which handles the messages among agents. Block 105 is the network channel between the agent 115 and the server 101. 155 denotes the communication channels between the devices 165 and the agent 115. It can be in any form. For example, a device 165 may be a camera which, after it analyzes an image, can divide the results into several categories and send the message through a TCP/IP (Transfer Control Protocol/Internet Protocol) channel 155. Another example, a device 165 may be a keyboard which, after the user hits a key, generates a signal that can be grabbed and sent to the agent 115 via a TCP/IP channel 155. In general, the devices 165 are used for the system to perform event perception. They can be a camera, a keyboard, a sensitive touch screen, a weight sensor, a motion sensor, and many other devices that can sense the environment. The user 195 uses a multi-modal communication channel 175 for communicating with devices in the environment. Channel 175 is actually the sensing channel for the devices to sense the environment and the user and various activities. The devices 165 keep sensing the environment and user 195 through multi-modal channels 175. The sensing results are categorized and send to the agent 115. The agent 115 analyzes the data and keeps detecting what has happened. When other agents ask about the status of its user, after a negotiation process, the agent provides relevant information.

[0053] FIG. 2 is a preferred diagram on a single agent component. Negotiation module 205 is responsible for negotiations with another agent or agents. It is responsible for parsing messages back and forth between itself and another agent. When it wants to pass information to other agents, it first sends the information to the server 101 (FIG. 1), then the server passes the information to the other agent or agents. The communications between them can be through a TCP/IP channel. When another agent or agents send message to the current agent, they also go through the server. They first send the message to the server, then the server transfers the message to the negotiation module 205 of the current agent. The negotiation module 205 passes messages back and forth between plan generation module 215. The plan generation module 215 generates plans for the agent to negotiate with other agents, or to transmit information through multi-modal user interface module 257 and finally received by the user. The plan generation module 215 consults the event perception module 255 and the knowledge data base 207 so as to generate plans. The knowledge data base 207 stores various data bases for the user, such as the user's day-to-day calendar, appointment schedule, and the like. The calendar can be meeting schedules, teleconferencing schedules, telephone call schedules, and many others. It also has some inference rule in one way or another so as to generate plans. For example, if the user is going to having a meeting with John at 1:00 o'clock, then he should not have any other appointments with Mary or other people in his department at this time. But when the CEO of the company want to meet him at this time, the knowledge data base should overwrite John's meeting. There are many ways to store data and relevant inference rules, as well understood by those skilled in the art of expert systems. Knowledge data base 207 can use any of them. Event perception module 255 perceives events and provide the results to plan generation module 215 when queried. Plan generation module 215 generates plans by various intelligent ways based on the content of event perception module 255 and knowledge data base 207.

[0054] FIG. 3 illustrates the situation for the user within his or her working environment. The user 195 is surrounded by various devices such as screen 301, keyboard 357, mouse 375, and other devices 305, 315 and 355 (any of which may be, for example, a touch sensitive screen, a motion sensor and a weight sensor). Some devices are only for output of information to the user, such as, say, devices 301 and 315. Some devices are only for input from the user or environment, such as, say, devices 355, 357 and 375. Other devices can be bi-directional. For example, the screen 301 could be a touch screen both displaying information to the user and receiving input from the user.

[0055] FIG. 4 is a flow chart on the data collection process. Function block 401 determines the data collection time intervals, such as one second intervals. This means every one second, the agent will collect the readings from all the devices. Function block 405 determines the total time for data collection, such as one minute. If for example the data collection time is one minute, then the total time for collecting data is 60+1=61 times. The first reading is obtained at time instant 0. The second reading is obtained for the time period between time instant 0 and time instant 1. In general, the r^th reading is obtained between the time instants r−1 and r.

[0056] Function block 415 determines the sensing categories for each available devices. For example, a weight sensor within the environment can divide its weights for every 10 pounds. Thus, if the total weight range can be sensed is 100 pounds, then we can divide the sensed range into ten categories. If the average weight sensed within the time instant r−1 and r belongs to category c, we say that the r^th reading belongs to category c. A camera can analyze images by divided the images into several categories based on the image analyzing results. As for a keyboard, we can divide the category like this. Suppose the keys on the keyboard are 0, 1, . . . , 9, a, . . . , z, A, . . . , Z. If within the time interval, the user has done nothing, then the category is 0, if he typed “0”, then it is 1. If he typed “9” then it is 10, if he typed “a”, then it is 11, etc. In general, suppose the keys on the keyboard are k₁, . . . k_n. Then, if the user typed on key k_i within the time period, then the category is i. If the user touched two keys k_i and k_j, then the category is (i−1)×n+j. If the user touches r keys, k₁, . . . , k_r, then the category is (i₁−1)n^r−1+(i₂−1)n^r−2+ . . . +(i_r−1)n+i_r.

[0057] We can also group based on the commands the user types. For example, if the user typed “Is” in the Unix® operating system (OS) environment, we might categorize it as 1. In general, we have different ways of doing the categorization. Our goal here is to categorize the input in such a way that the categorization can be easily used for the eigen space method to perform the event perception task. Function block 455 is the module for collecting readings for all the devices at every time instant. At every time instant, data is collected for each device, until the time is used up. The details will be described in FIGS. 6 to 11.

[0058] FIG. 5 is a flow chart of a data collection process for model building of a data matrix A. In function block 501, the different events to be perceived are determined. Function block 505 collects exemplar within the training time, and then a determination is made in decision block 515 to determine if we need more data for training. If so, the process loops back to function block 505; otherwise, the process uses collected exemplar to construct the sampling readings to form matrix A in function block 555. Here r₁^ij(0) is the readings of device 1 at time instant 0 for the i^th event'j^th exemplar. The column vector [r₁^if(0)] is the readings of all the devices at the time instant 0.

[0059] To model events, we collect N_exemplar exemplar for each events. Each exemplar collects readings from devices within a time interval [0,T]. These readings are discretized into N_readings+1 readings at time instant 0, $\frac{1}{N_{\mathrm{readings}}}T,\dots ,\frac{N_{\mathrm{readings}}^{-1}}{N_{\mathrm{readings}}}T,T.$
For device h (0≦h≦M), we denote its k^th (0≦k≦N_readings) readings for the j^th (1≦j≦N_exemplar) exemplar from the i^th (1≦i≦N) event as r_h^ij(k).

[0060] Let [r_h^ij(k)]=(r₁^ij(k), . . . , r_M^ij(k))^T be a column vector of k^th readings of all the devices for the j^th exemplar from the i^th event. Let [r^ij] represent the column vector obtained by simply concatenating the [r^ij(k)] column vector for all the k readings, [r^ij]=(r₁^ij(0), . . . , r_M^ij(0), r₁^ij(1), . . . , r_M^ij(1), . . . , r₁^ij(N_readngs), . . . , r_M^ij(N_readings))^T. Here [r^ij] gives the readings for the j^th exemplar of the i^th event. These are the readings of all the devices with respect to an exemplar in the model training phase. The length of the vector [r^ij] is M×(N_readings+1)

[0061] FIG. 6 is a graphical explanation of the reading of data at time 0 for the j^th exemplar of the i^th event. FIG. 7 illustrates the reading data for the event's j^th exemplar. FIG. 8 illustrates a more detailed explanation of the reading data for the i^th event's j^th exemplar.

[0062] The sampling readings matrix A for all the events and their associated exemplar can be created by the set of all j and i of [r^ij]. A=([r^1,1], . . . , [r^1,Nexemplar], . . . , [r^N,1], . . . , [r^N,Nexemplar]). The dimension of matrix A is (N×N_exemplar) columns and (M×(N_readings+1)) rows, N is the number of events and M is the number of devices. The N×N_exemplar columns of matrix A give readings for all the exemplar of all the events. This is the total number of training sets of data. The M×(N_readings+1) elements of each column gives the readings from all the devices at all the discrete time instants for an exemplar. Thus, each column of matrix A refers to a given training set, the elements of the column refers to the readings for this set. Usually M×(N_readiings+1)>>N×N_exemplar.

[0063] FIG. 9 illustrates the readings for all the exemplar of event i. FIG. 10 illustrates the data matrix A. FIG. 11 illustrates a more detailed data matrix A.

[0064] Matrix A can be decomposed using singular value decomposition (SVD) as:
A=UWV^T,
where U=(U₁, . . . , U_N×Nexemplar) is an orthogonal matrix of the same size as matrix A representing the principal component directions U_i(1≦i≦N×N_exemplar) in the training set. These are best directions that can clearly distinguish the training data. W is a diagonal matrix with singular values λ₁, . . . , λ_N×Nexemplar, sorted in decreasing order along the diagonal. The virtue of these values is that they rank the dimensions of the space in terms of variations along the principal component directions, and that this ranking is very often related to their importance. V^T is a (N×N_exemplar)×(N×N_exemplar) matrix that encodes the coefficients to be used in expanding each column of matrix A in terms of principal component directions.

[0065] FIG. 12 is a flow chart of a co-efficient generating process. In function block 1201, exemplar are collected to form the data matrix A. In function block 1205, the eigen vectors are generated based on the exemplar matrix. Function block 1215 generates the co-efficient for the corresponding data matrix. More particularly, the readings from the j^th exemplar of the ith event [r^ij] can be approximated according to the q singular values λ₁≧, λ₂≧ . . . _.≧λ_q as: $\left[r^{i,j}\right]=\sum _{l=1}^q{c}_l^{i,j}{U}_l,$
where q=N×N_exemplar, c_l^ij are scalar values that can be calculated by taking the dot product of [r^ij] and U_l, c_l^ij=[r^ij]^TU_l. This is the process of projecting the reading vector [r^ij] onto the subspace spanned by the q basis vectors U₁, . . . , U_q with parameters c₁^ij, . . . , c_q^ij. Thus, for a given i and j, we can obtain a vector C^ij=(c₁^ij, . . . , c_q^ij)^T that gives the coefficients of the corresponding readings. For all the possible i and j, we can get a coefficient matrix C(C^1,1, . . . , C^1,Nexemplar, . . . , C^N,1, . . . , C^N,Nexemplar).

[0066] Now, we transform matrix C into a matrix that represents the average coefficient for each event. For any event i, matrix C contain the coefficient vectors of all its exemplar: (C^i,1, . . . , C^i,Nexemplar). The average coefficient vector for these exemplar vectors can be calculated by: $\overrightarrow{C}=\frac{\sum _{j=1}^{N_{\mathrm{exemplar}}}{C}^{i,j}}{N_{\mathrm{exemplar}}}.$
The average coefficient matrix becomes {right arrow over (C)}=({right arrow over (C)}¹, . . . , {right arrow over (C)}^N). Each column i of matrix C corresponds to the average coefficient vector {right arrow over (C)}=({right arrow over (c)}₁ⁱ, . . . , {right arrow over (c)}_qⁱ) of event i. {right arrow over (C)} is the model of events learned from the training phase and will be used in event perception.

[0067] FIG. 13 is a flow chart of an even perception process without data abstraction. In function block 1301, device readings are collected, and in function block 1305, coefficients are generated. These processes are quite similar to those of function blocks 1201 and 1205, respectively, in FIG. 12. Function block 1315 performs the event perception task. More particularly, the perception of events involves matching readings from all the devices in a real situation against learned models of all the events. For the same event, readings in a real application may differ from those of its exemplar because of various reasons such as noise, etc. However, they may share some commonalities or signatures. The use of eigen space approach for event perception assumes that these commonalities or signatures of a given event is captured by the coefficients of the readings along principal component directions.

[0068] Suppose R(t)=(R₁(t), . . . , R_M(t))^T is the readings from the M devices within time period [0,T]. We discretize [R(t)] into N_reading+1 at time instant 0, $\frac{1}{N_{\mathrm{readings}}},\dots ,\frac{N_{\mathrm{readings}}^{-1}}{N_{\mathrm{readings}}}.$
Let [R(k)] denote the k^th readings of all the devices at k^th time instant. By concatenating readings from all the time instants, we obtain the column vector of matrix R which gives readings of all the devices at all the time instants.

[0069] By projecting this vector on the principal component directions, we recover a vector of coefficients, {right arrow over (c)}=(c₁, . . . , c_q), that approximate the event to be perceived as a linear combination of eigen event basis. Upon the recovery of the real situation coefficient vector, the normalized distance Δ_i between {right arrow over (c)} and model coefficients {right arrow over (C)}¹ is used to perceive the observed event. Here ${\Delta }_i=\sum _{k=1}^q{\left(c_k-{\overrightarrow{c}}_k^i\right)}^2.$
The event i with the smallest distance Δ_i is considered the best match of the observed event. The above is the process when we need to distinguish which event has happened among several possible events.

[0070] During the model formulation process, we can get the following. For any event i, matrix C contains the coefficient vectors of all its exemplar, (C^i,1, . . . , C^i,Nexemplar). The average coefficient vector for these exemplar vectors can be calculated by ${\overrightarrow{C}}^i=\frac{\sum _{j=1}^{N_{\mathrm{exemplar}}}{C}^{i,j}}{N_{\mathrm{exemplar}}}.$
The average coefficient matrix becomes C=({right arrow over (C)}¹, . . . , {right arrow over (C)}^N). Each column i of matrix C corresponds to the average coefficient vector {right arrow over (C)}=({right arrow over (c)}₁ⁱ, . . . , {right arrow over (c)}_qⁱ) of event i.

[0071] For event i, for coefficient vector C^i,1, we can obtain an offset difference: ${\eta }^{i,1}=\sqrt{\sum _{l=1}^q{\left[{\overrightarrow{c}}_l^i-{\overrightarrow{c}}_l^{i,1}\right]}^2}.$
Similarly, for C^i,2, . . . , C^i,Nexemplar, we can obtain η^i,2, . . . , η^i,Nexemplar as follows: ${\eta }^{i,j}=\sqrt{\sum _{l=1}^q{\left[{\overrightarrow{c}}_l^i-{\overrightarrow{c}}_l^{i,j}\right]}^2}.$
Here, j is between 1, . . . , N_exemplar. Thus, we get η^i,1, . . . η^i,Nexemplar. We reorder the above list such that they are in the increasing order, η^i,1≦ . . . ≦η^i,Nexemplar. These are the errors of the training exemplar sample with respect to their average case. We define an acceptance threshold such as 0.95 (or other values based on the experience).

[0072] During the event perception situation, we can collect data and calculate the coefficients {right arrow over (c)}=(c₁, . . . , C_q). We then calculate the difference $\eta =\sqrt{\sum _{l=1}^q{\left[{\overrightarrow{c}}_l^i-{\overrightarrow{c}}_l^{i,j}\right]}^2}.$
We then find the value of k such that
If $\frac{k}{N_{\mathrm{exemplar}}}\le 0.95,$
then we believe that event i has happened. Otherwise, we believe that event i has not happened. The above is the process when we need to figure out whether a given event has happened.

[0073] We can image that in the inter-connected world, a huge amount of devices will be involved. Events like what a person is doing can be perceived by considering only devices within an office. However, events like whether people within a building are having a meeting should be considered with all the devices within the building. In general, the bigger the scale of the events to be perceived, the more devices need to be considered. When the number of devices exceeds a certain threshold, the strategy above will not work because too much computational time is needed.

[0074] FIG. 14 is an illustration for eigen pyramid construction used in the explanation for function block 1601 in FIG. 16. FIG. 15 is an illustration for an eigen-pyramid's data matrix change.

[0075] FIG. 16 is a flow chart illustrating the eigen pyramid model construction process. Function block 1601 divides the device readings into k groups as illustrated in FIG. 14, and in FIG. 15, 1505. In order to perform event perception when the number of devices is huge, we propose a new strategy called “pyramid eigen space”. Suppose n_acceptable is the number of devices that can be handled by the above eigen space approach, shown as reference numeral 1401 in FIG. 14. Suppose N_total=M×N_readings+1 is the total number of readings to be considered. Our strategy is to first divide uniformly these N_total devices into different groups such that each group can be handled by the above eigen space method. Suppose N_total=k(n_acceptable−1)+u₁, where 0≦u<n_acceptable−1. If u=0, then we can divide the devices into n_acceptable−1 groups. Otherwise, we can divide the first k(n_acceptable−1) devices into k groups where each group has n_acceptable−1 members. Then we distribute the rest r devices into the first r groups obtained. Thus, we have divided devices into k groups, where each group has either n_acceptable−1 or n_acceptable members.

[0076] Function block 1605 obtains the co-efficient for each group. This is illustrated in Mapping 1515, 1557 and 1575 of FIG. 15. For each group, we run the training data and detect their principal directions. Now, we collect the coefficient vector with respect to the principal directions for each training exemplar. The length of the coefficient vector is N_exemplar×N_events.

[0077] Function block 1615 forms the next layer of the pyramid. This is illustrated in 1557 and 1565 of FIG. 15. Since the coefficients capture the differences in the training data, we will take coefficients of each exemplar 1555 as the input to the second level of the pyramid. By concatenating the coefficients of each group, we can obtain the new “exemplar” column vector, e.g., reference numeral 1405 in FIG. 14, which has length
k×N_exemplar×N_events
and which should be much smaller than the original length of the “exemplar” column vector. For every old exemplar, we can get a new exemplar. Each new exemplar will be a column in the new matrix. After we put all the new exemplar together, we get a new data matrix that acts as the second layer of the pyramid.

[0078] In decision block 1617, a determination is made as to whether to continue the data abstraction process. If so, the process loops back to function block 1601; otherwise, the process ends.

[0079] At the first stage, the length for each training vector (exemplar) is N_total. During the process above, they are divided into k groups. Each group generates N_exemplar×N_events coefficients. Thus, the total length for the second level of input will be k×N_exemplar×N_events, which is much less than N_total−k(n_acceptable−1)+r.

[0080] If k×N_exemplar×N_events>n_acceptable, we take this new data as input and repeat the above data abstraction process to further reduce the amount of data.

[0081] If k×N_exemplar×N_events is much less than n_acceptable and a further eigen coefficient extraction is meaningless, then we take these k×N_exemplar×N_events numbers as the final coefficients of the training exemplar.

[0082] If k×N_exemplar×N_events is less than n_acceptable and a further eigen coefficient extraction is meaningful, then we take another round of eigen coefficient extraction. These newly generated coefficients 1577 will be taken as the final coefficients of the exemplar of the training events.

[0083] These new coefficients will be taken as the final coefficients of the exemplar of the training events. After extracting the final coefficients of all the exemplar of all the events, the average of the final coefficients of all the exemplar with respect to a given event are taken as the model for the corresponding event. Just like what we did in the single layer case, these models are generated and used to perform the event perception task.

[0084] During the event perception phase, we can first get readings from all devices. FIG. 17 is a flow chart of an extraction process for extracting the signature for each divided group of the real measured data during the event perception process. Function block 1701 collects data from devices during the event perception process. Then according to the division when the pyramid is built, function block 1705 divides the initial data into different groups the same way as the pyramid model generation process when this corresponding layer is being built. We extract coefficients for the data of each group with respect to the principal directions of the first layer of the pyramid formed during the training phase in function block 1707 and, in function block 1715, the coefficients are connected to form the next layer. A determination is made in decision block 1755 as to whether there are more layers (e.g., 1579) in the pyramid. If so, the process loops back to function block 1705; otherwise, the event perception is output at output block 1775.

[0085] FIG. 18 is flow chart of a start negotiation process. In function block 1801, the user sends his negotiation request via the multi-modal interface to its agent. In function block 1805, the agent analyzes the request and determines, based on the input, which agents are to be contacted. Decision block 1815 checks to determine if the total number of agents to be contacted is more than one or not. If so, the process goes to function block 1855 to start the multi-agent negotiation process. Otherwise, the process goes to function block 1875 to start single agent negotiation process.

[0086] FIG. 19 is a flow chart of a start agent process run during the single agent negotiation process. Function block 1901 identifies the other agents to be contacted. Function block 1905 identifies the topics to be negotiated, such as making a phone call or make an appointment. Function block 1907 checks the knowledge data base 207 (FIG. 2) so as to identify parameters of the topic. For example, for scheduling a time, there should be a start time, an end time, and the attendees of the meeting, etc. Function block 1911 examines the knowledge data base to identify the set of acceptable choices. In decision block 1915, a check is made to determine whether there is at least one choice left. If so, function block 1917 asks the negotiation module 205 (FIG. 2) to construct a negotiation message and send the message. It can be in the form of XML (eXtended Markup Language) or other protocols. At function block 1951, a message is received from responding agent. Then, in decision block 1957, a determination is made as to when the agent received messages from the responding agent about whether the request is approved or not. If not, the process loops back to function block 1915; but if so, the process exits. Returning now to decision block 1915, if there are no choices left, a conflict dialogue is started with its own user in function block 1955 to determine a new request.

[0087] FIG. 20 is a flow chart of a responding agent process run during the negotiation process. The agent receives the negotiation request from the starting agent in function block 2001. In function block 2005, the responding agent identifies the starting agent, the topic, and the parameters. In function block 2011, the responding agent check the knowledge base to see whether there are any conflicts. If there is no conflicts, as determined in decision block 2015, the starting agent simply sends a message “approved” in function block 2017. Otherwise, the preference level is checked in function block 2019 to see whether there are any possibility of updating. For example, although at a certain time the user has a meeting with a colleague, but since the starting agent works for the CEO of the company, the original appointment should be replaced by the new appointment. If after checking the knowledge data base in decision block 2055 it is determined that it can be updated, then simply update in function block 2077. Otherwise, a “not approved” message is sent to the starting agent in function block 2057.

[0088] FIG. 21 is a flow chart of a dialogue process for the start agent to handle conflict with its user. Function block 2101 displays information to the user via multi-modal device such as a screen or a voice channel; i.e., “request is not approved”. After receiving the message, it is the user who needs to determine whether to negotiate directly with the other user in function block 2105. If the user decides to directly negotiate with the other user, as determined in decision block 2115, the agent system is bypassed in function block 2155. If not, in function block 2175, the user can propose an alternative negotiation request and send the request to the agent via multi-model interface.

[0089] FIG. 22 is a flow chart of the start agent during the group agent negotiation process. In function block 2201, the group of agents to be contacted is identified. According to the input of the user, function block 2205 identifies the topic to be negotiated (e.g., phone, schedule time, etc.). A check of the knowledge data base is made in function block 2207 to identify parameters of the topic. Function block 2211 examines the knowledge data base to identify the set of acceptable choices. A determination is made in decision block 2215 if there are any choices left and, if so, in function block 2251, the agent constructs a negotiation message (can be in XML form) and sends the message to all of the responding agents. In function block 2255, the starting agent receives all the messages from all the responding agents. Decision block 2271 checks whether the request is approved by every responding agent. If so, the process goes to function block 2277 to inform all the other agent about the final negotiation results; but if not, the process loops back to decision block 2215. Returning to decision block 2215, if there are no more choices left, then a conflict dialogue is stared with its own user in function block 2255 to determine the new request, and the process loops back to function block 2201.

[0090] FIG. 23 is a flow chart of a privacy guarding process for the start agent during the group awareness process. This is an idea to keep the privacy of its user so that it can not be invaded. In other words, “I only let you know what I want you to know”. In function block 2301, the user sends an inquiry to his agent about the status of the other users, such as whether they are making a phone call, they are working right now, etc. In function block 2305, the agent identifies the number of users and the corresponding status parameters. In decision block 2315, the agent selects one agent from the agent pools to be queried and send the request. If every agent has been queried, the process goes to function block 2357; otherwise, in function block 2355, the starting agent receives the response from the agent to be queried, and the process loops back to decision block 2315. The response for the queried agent can contain what information you can display and using which channel. Returning to function block 2357, a determination is made as to the display strategies for different users and the corresponding channels; for example, how to display the corresponding message, what content to display, etc. In function block 2375, the starting agent keeps receiving messages from the other agents about the status of their users, and the starting agent will keep displaying the status of the other users.

[0091] FIG. 24 is a flow chart of the procedure for the responding agent during the privacy guarding process. Function block 2401 receives the message from the starting agent on the status query. Function block 2405 checks its knowledge data base and the value of the agent-event matrix. This matrix encodes the privacy concerns of the agent about its user. Function block 2415 generates the list of events to be transmitted and the way of transmitting them. Function block 2455 keeps performing the event perception task and sends messages to the start agent about the status of its user. The agent-event matrix is used to determine what to send.

[0092] FIG. 25 graphically illustrates the Agent-Event matrix for a given agent used to guard the privacy of the agent's user. The rows represent different events. The columns represent agents, ae, represent the situation for event i to be sent to agent j. For each event, the agent divides them into different categories. Thus, each event is associated with a set of different values. If ae_if=−1, then the responding agent will not give any information about event i to the agent j. If ae_if=0, then the responding agent will display exactly what it perceived. If ae_if=t>0, then the responding agent will always transmit the value of the t to the starting agent.

[0093] FIG. 26 is a flow chart illustrating the awareness process for an agent's own user. The goal is to provide its own user about who is monitoring him. Function block 2601 accesses the knowledge data base to check which agent requested its agent's info. Function block 2605 identifies the information other agent requested, and which information was given to them. Function block 2615 displays the corresponding information to its user when asked using a proper user interface.

[0094] While the invention has been described in terms of a single preferred embodiment, those skilled in the art will recognize that the invention can be practiced with modification within the spirit and scope of the appended claims.