Legged robot, legged robot behavior control method, and storage medium

[0001] The present invention relates to polyarticular robots, such as legged robots having at least limbs and a trunk, to action control methods for legged robots, and to storage media. Particularly, the present invention relates to a legged robot which executes various action sequences using limbs and/or a trunk, to an action control method for the legged robot, and to a storage medium.

[0002] More specifically, the present invention relates to a legged robot of a type which autonomously forms an action plan in response to external factors without direct command input from an operator and which performs the action plan to the world, to an action control method for the legged robot, and to a storage medium. More particularly, the present invention relates to a legged robot which detects external factors, such as a change of time, a change of season, or a change in a user's mood, and transforms the action sequence while operating in cooperation with the user in a work space shared with the user, to an action control method for the legged robot, and to a storage medium.

[0003] Machinery which operates in a manner similar to human behavior by electrical or magnetic operation is referred to as a “robot”. The etymology of the word robot is “ROBOTA (slave machine)” in Slavic. In Japan, robots became widely used in the end of the 1960s. Many of these robots are industrial robots, such as manipulators and transfer robots, designed automation and for unmanned production in factories.

[0004] Recently, research and development of the structure of legged mobile robots, including pet robots emulating the physical mechanism and the operation of quadripedal walking animals, such as dogs, cats, and bear cubs, and “human-shaped” or “human type” robots (humanoid robots) which emulate the physical mechanism and the operation of bipedal orthograde animals, such as human beings and monkeys, and stable walking control thereof have advanced. There is a growing expectation for practical applications. Although these legged mobile robots are unstable and posture control and walking control thereof are difficult compared with crawling-type robots, the legged mobile robots are superior in that they can walk and run flexibly, such as climbing up and down stairs and jumping over obstacles.

[0005] Stationary robots, such as arm robots, which are installed and used at a specific location, operate only in a fixed, local work space where they assemble and select parts. In contrast, the work space for mobile robots is limitless. Mobile robots move along a predetermined path or move freely. The mobile robots can perform, in place of human beings, predetermined or arbitrary human operations and can offer various services replacing human beings, dogs, or other living things.

[0006] One use of the legged mobile robots is to replace human beings in executing various difficult tasks in industrial and production activities. For example, the legged mobile robots can replace human beings in doing dangerous and difficult tasks, such as the maintenance of nuclear power generation plants and thermal power plants, the transfer and assembly of parts at production factories, cleaning skyscrapers, and rescue from fires.

[0007] Rather than supporting human beings in executing the foregoing tasks, another use of the legged mobile robots is to “live together” with human beings or to “entertain” human beings. This type of robot emulates the operation mechanism of a legged walking animal which has a relatively high intelligence, such as a human being, a dog, or a bear cub (pet), and the rich emotional expressions thereof. Instead of accurately executing operation patterns which are input in advance, this type of robot can make lively responsive expressions which are generated dynamically in accordance with the user's words and mood (“praising”, “scolding”, “hitting”, etc).

[0008] In known toys, the relationship between the user operation and the response operation is fixed. The operation of the toy cannot be changed in accordance with the user's preferences. As a result, the user will become bored with a toy which only repeats the same operation.

[0009] In contrast, an intelligent robot has an action model and a learning model which depend on the operation thereof. In accordance with input information including external sounds, images, and tactile information, the models are changed, thus determining the operation. Accordingly, autonomous thinking and operation control can be realized. By preparing the robot with an emotion model and an instinct model, autonomous actions based on the robot's emotions and instincts can be exhibited. When the robot has an image input device and a speech input/output device, the robot can perform image recognition processing and speech recognition processing. Accordingly, the robot can perform realistic communication with a human being at a higher level of intelligence.

[0010] By changing the model in response to detection of an external stimulus including a user operation, that is, by adding a “learning model” having a learning effect, an action sequence which is not boring to the user or which is in accordance with each user's preferences can be performed.

[0011] Even without direct command input from an operator, a so-called autonomous robot can autonomously form an action plan taking into consideration external factors input by various sensors, such as a camera, a loudspeaker, and a touch sensor, and can perform the action plan through various mechanical output forms, such as the operation of limbs, speech output, etc.

[0012] When the action sequence is changed in accordance with the external factors, the robot takes an action which is surprising to and unexpected by the user. Thus, the user can continue to be together with the robot without getting bored.

[0013] While the robot is operating in cooperation with the user or another robot in a work space shared with the user, such as a general domestic space, the robot detects a change in the external factors, such as a change of time, a change of season, or a change in the user's mood and transforms the action sequence. Accordingly, the user can have a stronger affection for the robot.

[0014] It is an object of the present invention to provide a superior legged robot which can execute various action sequences utilizing limbs and/or a truck, an action control method for the legged robot, and a storage medium.

[0015] It is another object of the present invention to provide a superior legged robot of a type which can autonomously form an action plan in response to external factors without receiving direct command input from an operator and which can perform the action plan, an action control method for the legged robot, and a storage medium.

[0016] It is yet another object of the present invention to provide a superior legged robot which can detect external factors, such as a change of time, a change of season, a change in a user's mood, while operating in cooperation with a user in a work space shared with the user or another robot and which can transform an action sequence; an action control method for the legged robot; and a storage medium.

[0017] In view of the foregoing objects, according to a first aspect of the present invention, a legged robot which operates in accordance with a predetermined action sequence or an action control method for the legged robot is provided including:

[0018] input means or step for detecting an external factor;

[0019] option providing means or step for providing changeable options concerning at least a portion of the action sequence;

[0020] input determination means or step for selecting an appropriate option from among the options provided by the option providing means or step in accordance with the external factor detected by the input means or step; and

[0021] action control means or step for performing the action sequence, which is changed in accordance with a determination result by the input determination means or step.

[0022] The legged robot according to the first aspect of the present invention performs an action sequence, such as reading aloud a story printed in a book or other print media or recorded in recording media or a story downloaded through a network. When reading a story aloud, the robot does not simply read every single word as it is written. Instead, the robot uses external factors, such as a change of time, a change of season, or a change in a user's mood, and dynamically alters the story as long as the changed contents are substantially the same as the original contents. As a result, the robot can read aloud the story whose contents would differ every time the story is read.

[0023] Since the legged robot according to the first aspect of the present invention can perform such unique actions, the user can be with the robot for a long period of time without getting bored. Also, the user can have a strong affection for the robot.

[0024] The world of the autonomous robot extends to the world of reading. Thus, the robot's understanding of the world can be enlarged.

[0025] The legged robot according to the first aspect of the present invention may include content obtaining means for obtaining external content for use in performing the action sequence. For example, content can be downloaded through information communication media, such as the Internet. Also, content can be transferred between two systems or greater through content storage media, such as a CD and a DVD. Alternatively, other content distribution media can be used.

[0026] The input means or step may detect an action applied by a user, such as “patting”, as the external factor, or may detect a change of time or season or reaching a special date as the external factor.

[0027] The action sequence performed by the legged robot may be reading aloud a text supplied from a book or its equivalent, such as a printed material/reproduction, or a live performance of a comic story. Also, the action sequence may include playback of music data which can be used as BGM.

[0028] For example, in the action sequence, a scene to be read aloud may be changed in response to an instruction from a user, the instruction being detected by the input means or step.

[0029] The legged mobile robot may further include display means, such as eye lamps, for displaying a state. In such a case, the display means may change a display format in accordance with a change of scene to be read aloud.

[0030] According to a second aspect of the present invention, a robot apparatus with a movable section is provided including:

[0031] external factor detecting means for detecting an external factor;

[0032] speech output means for outputting a speech utterance by the robot apparatus;

[0033] storage means for storing a scenario concerning the contents of the speech utterance; and

[0034] scenario changing means for changing the scenario,

[0035] wherein the scenario is uttered by the speech output means while the scenario is changed by the scenario changing means in accordance with the external factor detected by the external factor detecting means.

[0036] The robot apparatus according to the second aspect of the present invention may actuate the movable section in accordance with the contents of the scenario when uttering the scenario.

[0037] The robot apparatus according to the second aspect of the present invention may perform speech output of the scenario concerning the contents of the speech utterance stored in advance. Instead of simply reading every single word as it is written, the robot apparatus can change the scenario using the scenario changing means in accordance with the external factor detected by the external factor detecting means.

[0038] Specifically, the scenario is dynamically changed using external factors, such as a change of time, a change of season, or a change in the user's mind, as long as the changed contents are substantially the same as the original contents. As a result, the contents to be uttered would differ every time the scenario is uttered. Since the robot apparatus according to the second aspect of the present invention can perform such unique actions, the user can be with the robot for a long period of time without getting bored. Also, the user can have a strong affection for the robot.

[0039] When uttering the scenario, the robot apparatus adds interaction, that is, actuating the movable section in accordance with the contents of the scenario. As a result, the scenario becomes more entertaining.

[0040] According to a third aspect of the present invention, there is provided a storage medium which has physically stored therein computer software in a computer-readable format, the computer software causing a computer system to execute action control of a legged robot which operates in accordance with a predetermined action sequence. The computer software includes:

[0041] an input step of detecting an external factor;

[0042] an option providing step of providing changeable options concerning at least a portion of the action sequence;

[0043] an input determination step of selecting an appropriate option from among the options provided in the option providing step in accordance with the external factor detected in the input step; and

[0044] an action control step of performing the action sequence, which is changed in accordance with a determination result in the input determination step.

[0045] The storage medium according to the third aspect of the present invention provides, for example, computer software in a computer-readable format to a general computer system which can execute various program code. Such a medium includes, for example, a removable, portable storage medium, such as a CD (Compact Disc), an FD (Floppy Disk), and an MO (Magneto-Optical disc). Alternatively, it is technically possible to provide the computer software to a specific computer system through a transmission medium, such as a network (without distinction between wireless networks and wired networks). Needless to say, the intelligent legged mobile robot has a high information processing capacity and has an aspect as a computer.

[0046] The storage medium according to the third aspect of the present invention defines the structural or functional cooperative relationship between predetermined computer software and a storage medium for causing a computer system to perform functions of the computer software. In other words, by installing predetermined computer software into a computer system through the storage medium according to the third aspect of the present invention, the cooperative operation can be performed by the computer system. Thus, the operation and advantages similar to those of the legged mobile robot and the action control method for the legged mobile robot according to the first aspect of the present invention can be achieved.

[0047] According to a fourth aspect of the present invention, a recording medium is provided including a text to be uttered by a robot apparatus; and identification means for enabling the robot apparatus to recognize an utterance position in the text when the robot apparatus utters the text.

[0048] The recording medium according to the fourth aspect of the present invention is formed as, for example, a book formed by binding a printed medium containing a plurality of pages at an edge thereof so that the printed medium can be opened and closed. When reading aloud a text in such a recording medium while looking at it, the robot apparatus can detect an appropriate portion to read aloud with the assistance of the identification means for enabling the robot apparatus to recognize the utterance position.

[0049] As the identification means, for example, the left and right pages when a book is opened are in different colors (that is, printing or image formation processing is performed so that the combination of colors differs for each page). Alternatively, a visual marker, such as a cybercode, can be pasted to each page. Accordingly, the identification means can be realized.

[0050] Further objects, features, and advantages of the present invention will become apparent from the following description of the embodiments of the present invention with reference to the attached drawings.

[0065] Embodiments of the present invention will now be described in detail with reference to the drawings.

[0066] In FIG. 1, according to an embodiment of the present invention, the external configuration of a mobile robot 1 which performs legged walking using four limbs is shown. As shown in the drawing, the robot 1 is a polyarticular mobile robot which is modeled after the shape and the structure of a four-legged animal. In particular, the mobile robot 1 of this embodiment is a pet robot which is designed after the shape and the structure of a dog, which is a typical example of a pet animal. For example, the mobile robot 1 can live together with a human being in a human living environment and can perform actions in response to user operations.

[0067] The mobile robot 1 contains a body unit 2, a head unit 3, a tail 4, and four limbs, that is, leg units 6A to 6D.

[0068] The head unit 3 is arranged on a substantially front top end of the body unit 2 through a neck joint 7 which has degrees of freedom in each axial direction, namely, roll, pitch, and yaw (shown in the drawing). The head unit 3 also includes a CCD (Charge Coupled Device) camera 15, which corresponds to the “eyes” of the dog, a microphone 16, which corresponds to the “ears”, a loudspeaker 17, which corresponds to the “mouth”, a touch sensor 18, which is arranged at a location such as on the head or the back and which senses the user's touch, and a plurality of LED indicators (eye lamps) 19. Apart from these components, the robot 1 may have sensors forming the senses of a living thing.

[0069] In accordance with a display state, the eye lamps 19 feed back to a user information concerning the internal state of the mobile robot 1 and an action sequence being executed. The operation will be described hereinafter.

[0070] The tail 4 is arranged on a substantially rear top end of the body unit 2 through a tail joint 8, which has degrees of freedom along the roll and pitch axes, so that the tail 4 can bend or swing freely.

[0071] The leg units 6A and 6B form front legs, and the leg units 6C and 6D form back legs. The leg units 6A to 6D are formed by combinations of thigh units 9A to 9D and calf units 10A to 10D, respectively. The leg units 6A to 6D are arranged at front, back, left, and right corners of the bottom surface of the body unit 2. The thigh units 9A to 9D are connected at predetermined locations of the body unit 2 by hip joints 11A to 11D, which have degrees of freedom along the roll, pitch, and yaw axes. The thigh units 9A to 9D and the calf units 10A to 10D are interconnected by knee joints 12A to 12D, which have degrees of freedom along the roll and pitch axes.

[0072] In FIG. 11, the mobile robot is shown viewed from the bottom surface. As shown in the drawing, pads are attached to the soles of four limbs. These pads are formed as switches which can be pressed. Along with the camera 15, the loudspeaker 17, and the touch sensor 18, the pads are important input means for detecting a user command and changes in the external environment.

[0073] By driving each joint actuator in response to a command from a controller described below, the mobile robot 1 arranged as described above moves the head unit 3 vertically and horizontally, moves the tail 4, and drives the leg units 6A to 6D in synchronization and in cooperation, thereby realizing an operation such as walking and running.

[0074] The degrees of freedom of the joints of the mobile robot 1 are provided by rotational driving of joint actuators (not shown), which are arranged along each axis. The number of degrees of freedom of the joints of the legged mobile robot 1 is arbitrary and does not limit the scope of the present invention.

[0075] In FIG. 2, a block diagram of an electrical control system of the mobile robot 1 is schematically shown. As shown in the drawing, the mobile robot 1 includes a controller 20 for controlling the overall operation and performing other data processing, an input/output unit 40, a driver section 50, and a power source 60. Each component will now be described below.

[0076] As input units, the input/output unit 40 includes the CCD camera 15, which corresponds to the eyes of the mobile robot 1, the microphone 16, which corresponds to the ears, the touch sensor 18, which is arranged at a predetermined location, such as on the head or the back, and which senses user's touch, the pad switches, which are arranged on the soles, and various other sensors corresponding to the senses. As output units, the input/output unit 40 includes the loudspeaker 17, which corresponds to the mouth, and the LED indicators (eye lamps) 19, which generate facial expressions using combinations of flashing and illumination of the LED indicators at specific times. These output units can represent user feedback from the mobile robot 1 in formats other than mechanical motion patterns using the legs or the like.

[0077] Since the mobile robot 1 includes the camera 15, the mobile robot 1 can recognize the shape and color of an arbitrary object in the work space. In addition to visual means including the camera, the mobile robot 1 can contain a receiver for receiving transmitted waves, such as infrared rays, sound waves, ultrasonic waves, and electromagnetic waves. In this case, the position and the direction from the transmitting source can be measured in accordance with the output of each sensor for sensing the corresponding transmission wave.

[0078] The driver section 50 is a functional block for implementing mechanical motion of the mobile robot 1 in accordance with a predetermined motion pattern instructed by the controller 20. The driver section 50 is formed by drive units provided for each axis, namely, roll, pitch, and yaw, at each of the neck joint 7, the tail joint 8, the hip joints 11A to 11D, and the knee joints 12A and 12D. In an example shown in the drawing, the mobile robot 1 has n joints with the corresponding degrees of freedom. Thus, the driver section 50 is formed by n drive units. Each drive unit is formed by a motor 51 which rotates in a predetermined axial direction, an encoder 52 for detecting the rotational position of the motor 51, and a driver 53 for appropriately controlling the rotational position and the rotational speed of the motor 51 in accordance with the output of the encoder 52.

[0079] Literally speaking, the power source 60 is a functional module for feeding power to each electrical circuit in the mobile robot 1. The mobile robot 1 according to this embodiment is an autonomous driving-type using a battery. The power source 60 is formed by a rechargeable battery 61 and a charging and discharging controller 62 for controlling the charging and discharging state of the rechargeable battery 61.

[0080] The rechargeable battery 61 is formed as a “battery pack”, which is formed by packaging a plurality of nickel cadmium battery cells in a cartridge.

[0081] The charging and discharging controller 62 detects the remaining capacity of the battery 61 by measuring the terminal voltage across the battery 61, the charging/discharging current, and the ambient temperature of the battery 61 and determines the charge start time and end time. The charge start and end time determined by the charging and discharging controller 62 are sent to the controller 20, and this triggers the mobile robot 1 to start and end the charging operation.

[0082] The controller 20 corresponds to a “brain” and is provided in the head unit 3 or the body unit 2 of the mobile robot 1.

[0083] In FIG. 3, the configuration of the controller 20 is shown in further detail. As shown in the drawing, the controller 20 is formed of a CPU (Central Processing Unit) 21, functioning as a main controller, which is interconnected with a memory, other circuit components, and peripheral devices by a bus. A bus 27 is a common signal transmission line including a data bus, an address bus, and a control bus. A unique address (memory address or I/O address) is assigned to each device on the bus 27. By specifying the address, the CPU 21 can communicate with a specific device on the bus 28.

[0084] A RAM (Random Access Memory) 22 is a writable memory formed by a volatile memory, such as a DRAM (Dynamic RAM). The RAM 22 loads program code to be executed by the CPU 21 and temporarily stores working data used by the executed program.

[0085] A ROM (Read Only Memory) 23 is a read only memory for permanently storing programs and data. Program code stored in the ROM 23 includes a self-diagnosis test program executed when the mobile robot 1 is turned on and an operation control program for defining the operation of the mobile robot 1.

[0086] Control programs for the robot 1 include a “sensor input processing program” for processing sensor input from the camera 15 and the microphone 16, an “action command program” for generating an action, that is, a motion pattern, of the mobile robot 1 in accordance with the sensor input and a predetermined operation model, a “drive control program” for controlling driving of each motor and speech output of the loudspeaker 17 in accordance with the generated motion pattern, and an application program for offering various services.

[0087] Besides normal walking and normal running, the motion pattern generated by the drive control program can include entertaining operations, such as “shaking a paw”, “leaving it”, “sitting”, and barking such as “bow-wow”.

[0088] The application program is a program which offers a service including an action sequence for reading a book aloud, giving a live Rakugo (comic story) performance, and playing music in accordance with external factors.

[0089] The sensor input processing program and the drive control program are hardware-dependent software layers. Since program code is unique to the hardware configuration of the body, the program code is generally stored in the ROM 23 and is integrated and provided with the hardware. In contrast, the application software such as an action sequence is a hardware-independent layer, and hence the application software need not be integrated and provided with the hardware. In addition to a case where the application software is stored in advance in the ROM 23 and the ROM 23 is provided in the body, the application software can be dynamically installed from a storage medium, such as a memory stick, or can be downloaded from a server on a network.

[0090] As in an EEPROM (Electrically Erasable and Programmable ROM), a non-volatile memory 24 is formed as a memory device which is electrically erasable/writable and is used to store data to be sequentially updated in a non-volatile manner. Data to be sequentially updated includes, for example, security information including a serial number or a cryptographic key, various models defining the action patterns of the mobile robot 1, and program code.

[0091] An interface 25 interconnects with external devices outside the controller 20, and hence data can be exchanged with these devices. The interface 25 inputs/outputs data from/to, for example, the camera 15, the microphone 16, and the loudspeaker 17. The interface 25 also inputs/outputs data and commands from/to each driver 53-1 . . . in the driver section 50.

[0092] The interface 25 includes general interfaces with computer peripheral devices. Specifically, the general interfaces include a serial interface such as RS (Recommended Standard)-232C, a parallel interface such as IEEE (Institute of Electrical and electronics Engineers) 1284, a USB (Universal Serial Bus) interface, an i-Link (IEEE 1394) interface, an SCSI (Small Computer System Interface) interface, and a memory card interface (card slot) which receives a memory stick. The interface 25 may exchange programs and data with locally-connected external devices.

[0093] As another example of the interface 25, an infrared communication (IrDA) interface can be provided, and hence wireless communication with external devices can be performed.

[0094] The controller 20 further includes a wireless communication interface 26 and a network interface card (NIC) 27 and performs short-range wireless data communication such as “Bluetooth” and data communication with various external host computers 100 via a wireless network such as “IEEE.802.11b” or a wide-area network (WAN) such as the Internet.

[0095] One purpose of data communication between the mobile robot 1 and each host computer 100 is to compute complicated operation control of the mobile robot 1 using (remote) computer resources outside the robot 1 and to perform remote control of the mobile robot 1.

[0096] Another purpose of the data communication is to supply data/content and program code, such as the action model and other program code, which are required for controlling the operation of the robot 1 from a remote apparatus via a network to the mobile robot 1.

[0097] The controller 20 may include a keyboard 29 formed by a numeric keypad and/or alphabet keys. In the work space of the robot 1, the keyboard 29 is used by the user to directly input a command and to input owner authentication information such as a password.

[0098] The mobile robot 1 according to this embodiment can operate autonomously (that is, without requiring people's help) by executing, in the controller 20, a predetermined operation control program. The mobile robot 1 contains input devices corresponding to the senses of a human being or an animal, such as an image input device (which is the camera 15), a speech input device (which is the microphone 16), and the touch sensor 18. Also the mobile robot 1 has the intelligence to execute a rational or an emotional action in response to external input.

[0099] The mobile robot 1 arranged as shown in FIGS. 1 to 3 has the following characteristics. Specifically:

• [0100] (1) When the mobile robot 1 is instructed to change from a first posture to a second posture, instead of directly changing from the first posture to the second posture, the mobile robot 1 can smoothly change from the first posture to the second posture through an intermediate position which is prepared in advance;
• [0101] (2) When the mobile robot 1 reaches an arbitrary posture while changing posture, the mobile robot 1 can receive a notification;
• [0102] (3) The mobile robot 1 can perform posture control while independently controlling the position of each unit, such as the head, the legs, and the tail. In other words, in addition to controlling the overall posture of the robot 1, the position of each unit can be controlled; and
• [0103] (4) The mobile robot 1 can receive parameters showing the detailed operation of an operation command.

[0104] The operation control of the mobile robot 1 is effectively performed by executing a predetermined software program in the CPU 21. In FIG. 4, the software control configuration running on the mobile robot 1 is schematically shown.

[0105] As shown in the drawing, the robot control software has a hierarchical structure formed by a plurality of software layers. The control software can employ object-oriented programming. In this case, each piece of software is treated as a modular unit, each module being an “object” integrating data and processing of the data.

[0106] A device driver in the bottom layer is an object permitted to gain direct access to the hardware, such as to drive each joint actuator and to receive a sensor output. The device driver performs corresponding processing in response to an interrupt request from the hardware.

[0107] A virtual robot is an object which acts as an intermediary between various device drivers and an object operating in accordance with a predetermined inter-object communication protocol. Access to each hardware item forming the robot 1 is gained through the virtual robot.

[0108] A service manager is a system object which prompts each object to establish connection based on inter-object connection information described in a connection file.

[0109] Software above a system layer is modularized according to each object (process). An object is selected according to each function required. Thus, replacement can be performed easily. By rewriting the connection file, input/output of objects of the same data type can be freely connected.

[0110] Software modules other than the device driver layer and the system layer are broadly divided into a middleware layer and an application layer.

[0111] In FIG. 5, the internal configuration of the middleware layer is schematically illustrated.

[0112] The middleware layer is a collection of software modules which provide the basic functions of the robot 1. The configuration of each module is influenced by hardware attributes, such as mechanical/electrical characteristics, specifications, and the shape of the robot 1.

[0113] The middleware layer can be functionally divided into recognition-system middleware (the left half of FIG. 5) and output-system middleware (the right half of FIG. 5).

[0114] In the recognition-system middleware, raw data from the hardware, such as image data, audio data, and detection data obtained from the touch sensor 18, the pad switches, or other sensors, is received through the virtual robot and is processed. Specifically, processing such as speech recognition, distance detection, posture detection, contact, motion detection, and image recognition is performed in accordance with various pieces of input information, and recognition results are obtained (for example, a ball is detected; falling down is detected; the robot 1 is patted; the robot 1 is hit; a C-E-G chord is heard; a moving object is detected; something is hot/cold (or the weather is hot/cold); it is refreshing/humid; an obstacle is detected; an obstacle is recognized; etc.). The recognition results are sent to the upper application layer through an input semantics converter and are used to form an action plan. In this embodiment, in addition to the sensor information, information downloaded through WAN, such as the Internet, and the actual time indicated by a clock or a calendar is employed as input information.

[0115] In contrast, the output-system middleware provides functions such as walking, reproducing motion, synthesizing an output sound, and illumination control of the LEDs corresponding to the eyes. Specifically, the action plan formed by the application layer is received and processed through an output semantics converter. According to each function of the robot 1, a servo command value for each joint, an output sound, output light (eye lamps formed by a plurality of LEDs), and output speech are generated, and they are output, that is, performed by the robot 1 through the virtual robot. As a result of such a mechanism, the operation performed by each joint of the robot 1 can be controlled by giving a more abstract command (such as moving forward or backward, being pleased, barking, sleeping, exercising, being surprised, tracking, etc.).

[0116] In FIG. 6, the internal configuration of the application layer is schematically illustrated.

[0117] The application uses the recognition results, which are received through the input semantics converter, to determine an action plan for the robot 1 and returns the determined action plan through the output semantics converter.

[0118] The application includes an emotion model which models the emotions of the robot 1, an instinct model which models the instincts of the robot 1, a learning module which sequentially stores the causal relationship between external events and actions taken by the robot 1, an action model which models action patterns, and an action switching unit which switches an action output destination determined by the action model.

[0119] The recognition results input through the input semantics converter are input to the emotion model, the instinct model, and the action model. Also, the recognition results are input as learning/teaching signals to the learning module.

[0120] The action of the robot 1, which is determined by the action model, is transmitted to the action switching unit and to the middleware through the output semantics converter and is executed on the robot 1. Alternatively, the action is supplied through the action switching unit as an action history to the emotion model, the instinct model, and the learning module.

[0121] The emotion model and the instinct model receive the recognition results and the action history and manages an emotion value and an instinct value. The action model can refer to the emotion value and the instinct value. The learning module updates an action selection probability in accordance with the learning/teaching signal and supplies the updated contents to the action model.

[0122] The learning module according to this embodiment can associate time-series data, such as music data, with joint angle parameters and can learn the associated time-series data and the joint angle parameters as time-series data. A neural network can be employed to learn the time-series data. For example, the specification of Japanese Patent Application 2000-252483, which has been assigned to the applicant of the present invention, discloses a learning system of a robot using a recurrent neural network.

[0123] The robot, which has the foregoing control software configuration, includes the action model and the learning model which depend on the operation thereof. By changing the models in accordance with input information, such as external speech, images, and contact, and by determining the operation, autonomous thinking and operation control can be realized. Since the robot is prepared with the emotion model and the instinct model, the robot can exhibit autonomous actions based on the robot's own emotions and instincts. Since the robot 1 has the image input device and the speech input device and performs image recognition processing and speech recognition processing, the robot can perform realistic communication with a human being at a higher level of intelligence.

[0124] Even without direct command input from an operator, the so-called autonomous robot can obtain external factors from inputs of various sensors, such as the camera, the loudspeaker, and the touch sensor, autonomously form an action plan, and performs the action plan through various output forms such as the movement of limbs and the speech output. By changing the action sequence in accordance with the external factors, the robot takes an action which is surprising to and unexpected by the user. Thus, the user can continue to be with the robot without getting bored.

[0125] Hereinafter, a process of transforming, by the autonomous robot, an action sequence in accordance with external factors will be described by illustrating a case where the robot executes the action sequence in which the robot “reads aloud” a book.

[0126] In FIG. 7, the functional configuration for transforming the action sequence is schematically illustrated.

[0127] As shown in the drawing, transformation of the action sequence is performed by an input unit for inputting external factors, a scenario unit for providing scenario options forming the action sequence, and an input determination unit for selecting an option from the scenario unit in accordance with the input result.

[0128] The input unit is formed by, for example, an auditory sensor (such as a microphone), a touch sensor, a visual sensor (such as a CCD camera), a temperature sensor, a humidity sensor, a pad switch, a current-time timer such as a calendar function and a clock function, and a receiver for receiving data distributed from an external network, such as the Internet. The input unit is formed by, for example, recognition-system middleware. Detection data-obtained from the sensors is received through the virtual robot, and predetermined recognition processing is performed. Subsequently, the detection data is transferred to the input determination unit.

[0129] The input determination unit determines external factors in the work space where the robot is currently located in accordance with a message received from the input unit. In accordance with the determination result, the input determination unit dynamically transforms the action sequence, that is, the story of the book to be read aloud. The scenario forming the transformed contents to be read aloud can only be changed as long as the transformed contents are substantially the same as the original contents, because changing the story of the book itself no longer means “reading aloud” the book.

[0130] The scenario unit offers scenario options corresponding to external factors. Although each option is generated by modifying or changing the original text, that is, the original scenario, in accordance with external factors, the changed contents have substantially the same meaning as the original contents. In accordance with a message from the input unit, the input determination unit selects one from a plurality of selection results offered by the scenario unit and performs the selected result, that is, reads the selected result aloud.

[0131] The changed contents based on the determination result are assured to have the same meaning as the original story as long as they are offered by the scenario unit. When viewed from the user side, the story whose meaning is preserved is presented in a different manner in accordance with the external factors. Even when the same story is read aloud to the user many times, the user can always listen to the story with a fresh sense. Thus, the user can be with the robot for a long period of time without getting bored.

[0132] FIG. 8 illustrates that, in the functional configuration shown in FIG. 7, the script “I'm hungry. I'm going to eat.” from the original scenario is changed in accordance with external factors.

[0133] As shown in the drawing, of the original scenario, the script “I'm hungry. I'm going to eat.”, which is permitted to be transformed in accordance with external factors, is input to the input determination unit.

[0134] The input determination unit is always aware of the current external factors in accordance with the input message from the input unit. In an example shown in the drawing, for example, the input determination unit is informed of the fact that it is evening based on the input message from the clock function.

[0135] In response to the script input, the input determination unit executes semantic interpretation and detects that the input script is related to “meals”. The input determination unit refers to the scenario unit and selects the optimal scenario from branchable options concerning “meals”. In the example shown in the drawing, the selection result indicating “dinner” is returned to the input determination unit in response to the time setting indicating “evening”.

[0136] The input determination unit transforms the original script in accordance with the selection result as a returned value. In the example shown in the drawing, the original script “I'm hungry. I'm going to eat.” is replaced by the script “I'm hungry. I'm going to have dinner,” which is modified in accordance with external factors.

[0137] The new script replacing the old script is transferred to the middleware through the output semantics and executed in the form of reading by the robot through the virtual robot.

[0138] When the autonomous robot reads a book (story) aloud, the robot does not read the book exactly as it is written. Instead, using various external factors, the robot dynamically alters the story and tells the story so that, every time the story is told, the contents would differ as long as the story is not greatly changed. It is thus possible to provide a unique, autonomous robot.

[0139] The elements of a story include, for example, scripts of characters, stage directions, and other text. These elements of a story can be divided into elements which do not influence the meaning of the entire story when modified/changed/replaced in accordance with external factors (for example, elements within the allowable range of ad lib even when modified/changed) and elements which cause the meaning of the story to be changed when modified/changed.

[0140] FIG. 9 schematically illustrates how the story is changed in accordance with external factors.

[0141] The story itself can be regarded as time-series data whose state changes as time passes (that is, the development of the story). Specifically, the elements including scripts, stage directions, and other text to be read aloud are arranged along the time axis.

[0142] The horizontal axis of FIG. 9 is the time axis. Points P₁, P₂, P₃, . . . on the time axis indicate elements which are not permitted to be changed in accordance with external factors. (In other words, the meaning of the story is changed when these elements are changed.) These elements are incapable of branching in accordance with external factors. In the first place, the scenario unit shown in FIG. 7 does not prepare options for these elements.

[0143] In contrast, regions other than the points P₁, P₂, P₃, . . . on the time axis include elements which are permitted to be changed in accordance with external factors. The meaning of the story is not changed even when these elements are changed in accordance with external factors, such as the season, the time, and the user's mood. Specifically, these elements are capable of branching in accordance with external factors. It is preferable that the scenario unit prepare a plurality of options, that is, candidate values.

[0144] In FIG. 9, points away from the time axis are points changed from the original text in accordance with external factors. The user, who will be the listener, can recognize these points as, for example, ad lib. Thus, the meaning of the story is not changed. Specifically, since the robot according to the embodiment of the present invention can read the book aloud while dynamically changing the story in accordance with external factors, the robot can tell a story which differs slightly every time it is told to the user. Needles to say, the story at points at which elements are changed from the original text in accordance with external factors does not change the meaning of the entire story because of the context between the original scenario before and after the changed portion and unchanged portions.

[0145] The robot according to this embodiment reads aloud a story from a book or the like. The robot can dynamically change the contents to be read in accordance with the time of day or the season when the story is being read aloud and other external factors applied to the robot.

[0146] The robot according to this embodiment can read a picture book aloud while looking at it. For example, even when the season set to a story in the picture book being read is spring, when the current season during which the picture book is being read is autumn, the robot reads the story as if the season is autumn. During the Christmas season, Santa Claus appears as a character. At Halloween, the town is full of pumpkins.

[0147] FIG. 10 shows the robot 1 reading the picture book aloud while looking at it. When reading a text, the mobile robot 1 according to this embodiment has a “reading aloud mode” in which the operation of the body stops and the robot 1 reads the text aloud and a “dynamic mode” in which the robot 1 reads the text aloud while moving the front legs in accordance with the story development (described below). By reading the text aloud in the dynamic mode, the sense of realism is improved, and the text becomes more entertaining.

[0148] For example, the left and right pages are in different colors (that is, printing or image formation processing is performed so that the combination of colors differs for each page). The mobile robot 1 can specify which page is open by performing color recognition and can detect an appropriate passage to be read. Needless to say, by pasting a visual marker, such as a cybercode, to each page, the mobile robot 1 can identify the page by performing image recognition.

[0149] In FIGS. 12 to 17, examples of a story consisting of scenes 1 to 6 are shown. As is clear from the drawings, for scene 1, scene 2, and scene 6, a plurality of versions is prepared in accordance with the outside world, such as the time of day. The remaining scenes, namely, scene 3, scene 4, and scene 5, are not changed in accordance with the time of day or other external factors. Needless to say, even when a version of a scene seems to be greatly different from the original scenario in accordance with external factors, this version does not change the meaning of the entire story because of the context between the original scenario before and after the changed portion and unchanged portions.

[0150] In the robot, which reads the story aloud, external factors are recognized by the input unit and the input determination unit, and the scenario unit sequentially selects a scene in accordance with each external factor.

[0151] The mobile robot 1 can store beforehand the content to be read aloud in the ROM 23. Alternatively, the content to be read aloud can be externally supplied through a storage medium, such as a memory stick.

[0152] Alternatively, when the mobile robot 1 has means for connecting to a network, the content to be read aloud can be appropriately downloaded from a predetermined information distributing server. The use of the most recent content is facilitated by a network connection. Data to be downloaded includes not only the contents of a story, but also an operation program for operating the body in the dynamic mode and a display control program for controlling display by the eye lamps 19. Needless to say, a preview of the subsequent story can be inserted into the content or advertising content from other suppliers can be inserted.

[0153] The mobile robot 1 according to this embodiment can control switching of the scene through input means such as the pad switch. For example, the pad switch on the left-rear leg is pressed, and then the touch sensor on the back is pressed, thereby skipping to the subsequent scene. In order to proceed to further subsequent scenes, the pad switch on the left-rear leg is pressed by the number of proceeding steps, and then the touch sensor on the back is pressed.

[0154] In contrast, when returning to the previous scene, the pad switch on the right-rear leg is pressed, and then the touch sensor on the back is pressed. When returning to further previous scenes, the pad switch on the right-rear leg is pressed by the number of returning steps, and then the touch sensor on the pack is pressed.

[0155] When reading a text aloud, the mobile robot 1 according to this embodiment has the “reading aloud mode” in which the operation of the body stops and the mobile robot 1 reads the text aloud and the “dynamic mode” in which the mobile robot 1 reads the text aloud while moving the front legs in accordance with the story development. By reading the text aloud in the dynamic mode, the sense of realism is improved, and the text becomes more entertaining.

[0156] The mobile robot 1 according to this embodiment changes the display by the eye lamps 19 in accordance with a change of scene. Thus, the user can apocalyptically confirm which scene is being read aloud or that there is a change of scene in accordance with the display by the eye lamps 19.

[0157] In FIG. 18, an example of the display by the eye lamps 19 in the reading aloud mode is shown. In FIG. 19, an example of the display by the eye lamps 19 in the dynamic mode is shown.

[0158] Examples of changes of a scenario (or versions of a scene) according to the season are shown as follows:

• • [0159] Spring
    • [0160] A butterfly is flitting around somebody walking.
  • [0161] Summer
    • [0162] Instead of the butterfly, a cicada is flying.
  • [0163] Autumn
    • [0164] Instead of the butterfly, a red dragonfly is flying.
  • [0165] Winter
    • [0166] Instead of the butterfly, it starts to snow.

[0167] Examples of changes of a scenario (or versions of a scene) according to the time are shown as follows:

• • [0168] Morning
    • [0169] The morning sun is dazzling. Eat breakfast.
  • [0170] Noon
    • [0171] The sun strikes down. Eat lunch.
  • [0172] Evening
    • [0173] The sun is almost setting in. Eat dinner.
  • [0174] Night
    • [0175] Eat late-night snack (noodles, pot noodles, etc.).

[0176] Examples of changes of a scenario (or versions of a scene) due to a public holiday or other special dates based on special events are shown as follows:

• • [0177] Christmas
    • [0178] Santa Claus is on his sleigh, which is pulled by reindeers, and the sleigh is crossing the sky.
    • [0179] People encountered say, “Merry Christmas!”
    • [0180] It may snow.
  • [0181] New Year
    • [0182] The robot greets the user with a “Happy New Year.”
  • [0183] User's birthday
    • [0184] The robot writes and sends a birthday card to the user, and the robot reads the birthday card aloud.

[0185] By incorporating changes according to the season and the time and timely information into the story, it is possible to provide content having real-time features.

[0186] The robot may be in a good mood or a bad mood. When the robot is in a bad mood, the robot may not read a book. Instead of changing the story at random, reading is performed in accordance with autonomous external factors (the time, sense of the season, biorhythm, the robot's character, etc).

[0187] In this embodiment illustrated in the specification, examples of available events which can be used as external factors for the robot are summarized as follows:

• [0188] (1) Communication with the user through the robot's body
• [0189] (Ex) Patted on the head
  • [0190] When the robot is patted on the head, the robot obtains information about user's likes and dislikes and mood.
• [0191] (2) Conceptual representation of the time and the season
• [0192] (Ex. 1) Morning, noon, and evening; and types of meals (breakfast, lunch, and dinner)
• [0193] (Ex. 2) Four seasons
  • [0194] Spring→Warm temperature, cherry blossoms, and tulips
  • [0195] Summer→Rain, hot
  • [0196] Autumn→Fallen leaves
  • [0197] Winter→New Year greeting
    • [0198] →At Christmas, Santa Claus appears.
    • [0199] →Rain changes to snow.
• [0200] (3) Brightness/darkness of user's room
• [0201] (Ex) When it is dark, a ghost appears.
• [0202] (4) The robot's character, emotion, age, star sign, and blood type
• [0203] (Ex. 1) The robot's way of speaking is changed in accordance with the robot's character.
• [0204] (Ex. 2) The robot's way of speaking is changed to adult-like speaking or childlike speaking in accordance with the robot's age.
• [0205] (Ex. 3) Tell the robot's fortune.
• [0206] (5) Visible objects
• [0207] (Ex. 1) The condition of the room
• [0208] (Ex. 2) The user's location and posture (standing, sleeping, or sitting)
• [0209] (Ex. 3) The outdoor landscape
• [0210] (6) The region or country where the robot is.
• [0211] (Ex) Although a picture book is written in Japanese, when the robot is brought to a foreign country, the robot automatically reads the picture book in that country's official language. For example, an automatic translation function is used.
• [0212] (7) The robot's manner of reading aloud is changed in accordance with information input via a network.
• [0213] (8) Direct speech input from a human being, such as the user, or speech input from another robot.
• [0214] (Ex) In accordance with a name called out by the user, the name of a protagonist or another character is changed.

[0215] Text to be read aloud by the robot according to this embodiment can include books other than picture books. Also, rakugo (comic stories) and music (BGM) can be read aloud. The robot can listen to a text read aloud by the user or another robot, and subsequently the robot can read that text aloud.

(1) When Reading a Comic Story Aloud

[0216] A variation can be added to the original text of a classical comic story, and the robot can read this comic story aloud. For example, changes of expressions (motions) of heat or coldness according to the season can be expressed. By implementing billing and downloading through the Internet, an arbitrary piece of comic story data from a collection of classical comic stories can be downloaded, and the downloaded comic story can be told by the robot. The robot can obtain content to be read aloud using various information communication/transfer media, distribution media, and providing media.

(2) When Playing Music (BGM)

[0217] A piece of music BGM can be downloaded from a server through the Internet, and the downloaded music can be played by the robot. By learning user's likes and dislikes or by determining the user's mood, the robot can select and play an appropriate piece of BGM in the user's favorite genre or a genre corresponding to the current state. The robot can obtain content to be read aloud using various information communication/transfer media, distribution media, and providing media.

(3) When Reading Aloud a Text or a Text Which has been Read Aloud by Others

[0218] The robot reads aloud a novel (for example, Harry Potter series or a detective story).

[0219] The reading frequency interval (for example, everyday) and the reading unit per single reading (one chapter) are set. The robot autonomously obtains the necessary amount of content to be read at the required time.

[0220] Alternatively, a text read by the user or another robot can be input to the robot, and at a future date the robot can read the input text aloud. The robot may play a telephone game or a word-association game with the user or another robot. The robot may generate a story through a conversation with the user or another robot.

[0221] As shown in this embodiment, while the robot is operating in cooperation with the user in a work space shared with the user, such as a general domestic space, the robot may detect a change in the external factors, such as a change of time, a change of season, or a change in the user's mood, and may transform an action sequence. Accordingly, the user can have a stronger affection for the robot.

[0222] Although the present invention has been described with reference to the specific embodiment, it is evident that modifications and substitutions can be made by those skilled in the art without departing from the scope of the present invention.

[0223] In this embodiment, an authoring system according to the present invention has been described in detail by illustrating a four-legged walking pet robot which is modeled after a dog. However, the scope of the present invention is not limited to this embodiment. For example, it should be fully understood that the present invention is similarly applicable to a two-legged mobile robot, such as a humanoid robot, or a mobile robot which does not use a legged formula.

[0224] In short, the present invention has been described by illustrative examples, and it is to be understood that the present invention is not limited to the specific embodiments thereof. The scope of the present invention is to be determined solely by the appended claims.

[0225] According to the present invention, it is possible to provide a superior legged robot which can perform various action sequences using limbs and/or a trunk, an action control method for the legged robot, and a storage medium.

[0226] According to the present invention, it is possible to provide a superior legged robot of a type which can autonomously form an action plan in response to external factors without direct command input from an operator and which can perform the action plan; an action control method for the legged robot; and a storage medium.

[0227] According to the present invention, it is possible to provide a superior legged robot which can detect external factors, such as a change of time, a change of season, or a change in a user's mood, and which can transform an action sequence while operating in cooperation with the user in a work space shared with the user; an action control method for the legged robot; and a storage medium.

[0228] When reading a story printed in a book or other print media or recorded in recording media or when reading a story downloaded through a network, an autonomous legged robot realizing the present invention does not simply read every single word as it is written. Instead, the robot dynamically alters the story using external factors, such as a change of time, a change of season, or a change in the user's mood, as long as the altered story is substantially the same as the original story. As a result, the robot can read aloud the story whose contents would differ every time the story is told.

[0229] Since the robot can perform unique actions, the user can continue to be with the robot without getting bored.

[0230] According to the present invention, the world of the autonomous robot extends to the world of reading. Thus, the robot's understanding of the world can be enlarged.