Operator Mental Condition Detector

1. Field of the Invention

The present invention relates to an information processing apparatus.

2. Related Background Art

In recent years, office automation equipment (to be referred to as OA equipment hereinafter) has been used in various application fields, and its importance has increased. Along with the "Second Industrial Revolution", the importance of an MMI (Man-Machine Interface) is recognized, and the need therefore has increasingly arisen.

The MMI results from considering how a machine looks from man's view point, and hence aims at a user friendly system.

However, the Second Industrial Revolution is an innovation centering around the computer replacing some functions performed by man's brain, in particular, a microcomputer comprising a high-performance semiconductor computer. In a sense, a machine is superior to man in performing some functions, and in the present state of the MMI, the intelligent machine is difficult for man to use.

In the early days of the First Industrial Revolution machines performed labor previously performed by man and brought about an alienation of man from machine. Charlie Chaplin's "MODERN TIMES" cynically satirized this alienation. There is therefore a need for a human MMI in order to avoid a second "MODERN TIMES".

The intelligent machine represented by a microcomputer causes greater alienation of man than that caused by the First Industrial Revolution. The present innovation centering around the computer developed from the use of computers in a computer room where only specialists used the computer, into OA, FA (factory automation), and HA (home automation) and has spread from the factory to the home. However, on the whole, this development is merely the leading wave of a much larger wave of new developments in computer technology. When emotionally maladjusted individuals use intelligent machines, the sense of alienation normally experienced can develop into stress or neurosis. At present, a Third Industrial Revolution is arising due to the development of the fifth generation computer.

For example, in recent information processing apparatuses, an icon displayed on a screen is moved by a mouse (a coordinate input devices) to select a desired menu or processing. Thus, recent information processing apparatuses tend to be easy to use.

However, since the keyboard and mouse must be selectively used, the mouse and icon are intermediary constructs through which the machine and man communicate. Machine and man do not communicate directly. Recent information processing apparatus have not yet reached the level at which man communicates directly with man.

Another problem is caused by the fact that man is managed by machines. To make the machine user friendly or to develop a cyborg or a high-performance robot is the ultimate object. in producing machines capable of efficient interaction with man Personification of the operation of various information equipment is a large step in achieving this goal. However, at present, almost no study or development is being in this area.

It is an object of the present invention to provide an information processing apparatus with which an MMI is improved to become a PMI (Personalized Machine Interface) by introducing a friendly character such as a second generation icon to information equipment, software, and the like, on the basis of the psychological inclinations of man.

Note that the PMI aims at improving the MMI, and personalization has both the meanings of personification and individualism.

It is therefore another object of the present invention to provide an information processing apparatus satisfying at least one of the following conditions:

(1) a user is individually identified to automatically select an MMI suitable for him or her;

(2) skill correspondence:

the apparatus interacts with the user according to the user's total access time, frequency of use, preferences, and actual results;

(3) PMI monitor:

the apparatus must comprise hardware and an OS capable of realizing the above items;

(4) hardware must be constituted by means for achieving the PMI and means for controlling equipment, and must be implemented within an appropriate cost range;

(5) the apparatus need only have an information volume sufficient to induce an association on the the part of the user between the output of the apparatus and tasks or behavior the user is to perform since the apparatus appeals to the psychology of the user. Therefore, the display need not always be a high-quality and full-color image and a small icon-like character of a line image need only be displayed; and

(6) the apparatus must appeal to the psychology of the user to cause a user to understand and respond to the status of equipment, the type of operation, a self failure diagnosis, and commands while personificating a machine without applying a heavy load to the equipment.

It is still another object of the present invention to eliminate the conventional drawbacks in consideration of the above situation.

It is still another object of the present invention to provide an apparatus for causing detection means to detect a mental condition of an operator and for causing output means to output information based on the detected mental condition.

It is still another object of the present invention to provide an apparatus for causing selection means to select basic information of an operator from those corresponding to a plurality of different operators stored in storage means and for causing output means to output information corresponding to a mental condition of the operator detected by detection means based on the selected basic information.

FIG. 1 is a view showing a state wherein an operator uses an information processing apparatus of this embodiment;

FIG. 2 is a view showing a state wherein an operator wears OA spectacles;

FIG. 3 is a view showing a pupil movement range detected by the OA spectacles;

FIG. 4 illustrates an example of output information of this embodiment;

FIG. 5 is a view showing architecture of a controller constituting the information processing apparatus of this embodiment;

FIG. 6 is a view showing the relationship between tasks and an operating system;

FIG. 7 is a view showing a basic arrangement of the information processing apparatus of this embodiment; and

FIG. 8 is a table showing operating states which are changed over time according to this embodiment.

An embodiment according to the present invention will now be described with reference to the accompanying drawings. The characteristic feature of the present invention may be achieved by a single piece of equipment or by a system consisting of a plurality of pieces of equipment.

At present, a speech input apparatus is proposed as a simple MMI method and device. However, even this apparatus can only release a user from a cumbersome key input operation; it is not an ultimate MMI means. In particular, it is difficult for current techniques to realize speech recognition of a plurality of users.

As the characteristic feature of the present invention, the psychological inclination of an operator is sensed by a machine, and the communication is requested from the machine to man.

For this purpose, a method and apparatus are proposed wherein access to a machine or a rest state of an operator is detected by the machine by the operator responding to speech from the machine or by the machine discriminating whether or not the operator wishes to use the machine. The machine waits for a set-up (e.g. power-on or initialization), and a speech recognition function is actuated to permit the machine to communicate with an operator by means of speech. However, this method is no better than the MMI, and simplifies the interface with the machine. The PMI cannot be realized unless the machine accesses the mental condition of the operator.

In this embodiment, the information processing apparatus is applied to a PC (personal coumputer) or a WP (word processor). However, the present invention is not limited to this but can be applied to the equivalent relationship between an NC machine, large-scale equipment, and the like, with an operator.

An operator normally operates a keyboard while watching a CRT (display apparatus), and the machine executes various processing operations in accordance with key data or data stored in a memory based on data or an instruction inputted from the keyboard. However, how does the machine sense the mental condition of the operator?

In a first method, data indicating personality, blood type, and the like of an operator is input beforehand to the machine together with a self ID. No. (identification number). In a second method, the mental condition of the operator can be checked based on the pupil movement of the operator, body temperature, room temperature, pulsation, blood pressure, and input errors sensed by a sensor of the machine. This is done only to sense ordinary events. However, with this method, the machine can grasp the personality and the mental condition of the operator, and may display a message on a screen or announce a message by voice as if it were a good friend. Note that the above data can also be detected by monitoring the access time of the operator.

FIG. 1 shows a detection example of bioinformation. FIG. 1 shows a state wherein an operator 1 operates OA equipment (e.g., a word processor).

In this state, the mental condition of the operator 1 is detected as described above. First, OA spectacles 4 will be described.

The OA spectacles 4 have a far-infrared sensor capable of continuously tracing the pupil movement of the operator 1. In practice, a light generating element for generating a weak far-infrared beam is arranged at one end of the OA spectacles 4, and transmit the beam toward the pupil. Since the operator's pupil is blackish, no light reflected thereby is obtained (or light is attenuated). When the beam falls outside the pupil, light reflected thereby is received by a light sensing element. The directivity of the light generating element can be adjusted so that no light reflected by the pupil is obtained when the pupil is moved within the area in which the operator watches the screen of a display apparatus 2. Note that in FIG. 3, an area of pupil movement upon watching a keyboard 3 and an area of pupil movement upon watching the CRT can be identified. When the pupil is out of this area or frequently out of the area, it is discriminted that the degree of fatigue of the operator 1 becomes considerable or that the operator cannot concentrate himself or herself on the operation, and a message "Short Break?" or an image that an operator likes or prefers is displayed on a portion of the screen. FIG. 4 shows this state. In FIG. 4, a landscape that the operator likes is displayed on a portion of the screen, and at the same time, an operator's favorite BGM (background music) can be reproduced. Therefore, since different individuals prefer the displayed image or the BGM, a self ID. No. must be input to the apparatus before the apparatus is started. Thus, the apparatus can discriminate the mental condition of the operator based on data corresponding to the input ID. No. with high precision.

Means for detecting the mental condition of the operator is not limited to the OA spectacles described above. For example, as has been described above, the mental condition of the operator can be detected based on an environment condition when the operator uses the apparatus and on the body temperature, pulsation, blood pressure, and input errors of the operator. For this purpose, a temperature sensor, a pulsation sensor, a blood pressure sensor, and a sensor for detecting depression pressure are arranged in some keys of the keyboard 3 (a carriage return key and a space key which are frequently used are preferable) to detect a change in mental condition. The keyboard 3 having the above function can detect that a depression pressure depressing the keyboard 3 becomes higher than an ordinary pressure or can count the number of correction errors upon depression of wrong keys. The apparatus can communicate with the operator in accordance with the frequency while the realizing a man-machine interaction.

Information indicating the mental condition of the operator such as changes in pupil movement, input error frequency, pulsation, body temperature, blood pressure, and the like is sensed by the apparatus, and the apparatus learns when the user experiences a degree of fatigue, so that communication depending on the detection state can be performed.

FIG. 5 shows the architecture of a controller constituting OA equipment according to the present invention. Along with the recent development of semiconductor process techniques, the controller can be formed on a single chip. As the feature of FIG. 5, a plurality of channels of co-processors (COP) for controlling an actual task are provided to a single chip as needed in addition to a CPU. Therefore, the CPU need only execute the OS (operating system) for managing the COPs. As in this embodiment, the fact that the machine has a function for detecting and learning the mental condition of the operator means that the machines has AI (artificial intelligenece). Therefore, a plurality of processors capable of high-speed parallel processing are preferably arranged.

In FIG. 5, layer-2 incorporates a microprogram allowing high-speed operation of the CPU. This program is adopted to execute at high speed a complicated program used for understanding an operator upon interaction with the operator.

Layer-3 includes a COP group. Processors of channel 0 to channel 16 can be independently parallel-operated. These processors are not directly related to the scope of the present invention, and a detailed description thereof is omitted.

Layer-4 includes a timer/counter array which can be accessed by the CPU and the COPs. Note that each COP incorporates 14 channels of timer/counters. A DPRAM (dual port RAM) is used for data exchange between the CPU and the COP and between the COPs. A RAM (2 clutch band RAM) can be accessed by the COPs and CPU. Layer-5 includes a bus line used for an interface with an external circuit (outside the controller). An S bus is used for expansion of an external memory and I/O ports.

An I bus is specially used for an ICE. Upon debugging of the system, the content of a register RAM (not shown) can be extracted through this bus, and can be checked.

An O bus is an I/O bus, and can be used for expansion for an I/O element.

In a serial I/O interface LAN, a predetermined protocol is constituted by a program, and the COPs are assigned thereto to perform serial communication.

FIG. 6 shows tasks for realizing the information processing apparatus of this embodiment and layers of the OS. As described above, in this embodiment, a plurality of COPs which can be parallel-executed are arranged around the CPU, and some COPs can be assigned to PMI control. This OS includes a normal machine control OS and a PMI OS. The PMI OS can be considered as an OS having an AI-like function for gradually learning the habits or personality of an operator through an interaction with the operator and for allowing better understanding. The COPs are arranged around the PMI OS, and parallel-execute corresponding tasks 0 to 15.

Assignment of the PMI tasks 0 to 15 can be determined depending on the software architecture. The number of COPs can be increased or decreased in accordance with one's work load.

In order to execute the above-mentioned processing, a special OS is required. In order to execute the AI-like PMI-OS having a learning function, high-speed parallel-processing processors are necessary. The architecture of this embodiment is suitable for these OS and processors.

FIG. 7 shows an arrangement when this system is applied to OA equipment. For the PMI arrangement, a bioinformation sensor 6, a display 2 for performing a PMI display on a portion of its screen, and a loudspeaker 7 for aural alarming, are arranged. A controller 5 controls the entire apparatus, and comprises a basic data storage 5a for storing basic data (data indicating personality and favor) for each operator.

With this arrangement, the low-cost PMI of a higher level than a conventional MMI can be easily realized.

FIG. 8 shows messages and announcements of the PMI in accordance with the lapse of time in this embodiment. When the OA equipment is turned on, "Good Morning: is first displayed or announced. Since the equipment is not always turned on in the morning, greeting data corresponding to turn-on time may be output. If necessary, a password or an ID. No. is input. When the ID. No. is input, the apparatus can identify the operator. Therefore, when the following mental condition is detected to output a message, output data best suitable for the operator is output. For example, in FIG. 8, a message "Tea Break?" (at 10 (AM) and 3 (PM)) is displayed. If the operator is fond of coffee, a message can be modified like "Coffee Break?". Since an image output at this time varies depending on the personal preferences or differences, the image displayed on the display is not always the same. For example, images of a mountain, a sea, a popular singer, an idol, a Buddha, and the like can be output depending on the preferences of the operator. In addition, sounds such as classic music, popular songs, bird's songs, rippling waves, and the like can also be output. Therefore, an image and sound that can confort the operator can be output. Therefore, the input of the ID. No. is necessary for the device to select personal basic data prestored in the apparatus, and the content of the output messages can be modified as needed. Of course, if no ID. No. is used, general output data can be output.

According to this embodiment, the machine and man can understand each other, an operator does not suffer from mental diseases such as neurosis due to alienation from the device, and interaction with the machine can be smoothly performed.

Since the machine detects the bioinformation, and accesses information for the operator, a good relationship between the operator and the machine can be assured, and the machine can make an announcement corresponding to a condition at that time to eliminate errors.

If the apparatus prestores personal basic data of operators, a still appropriate message output can be made.