Information processing device for controlling movement of a game character by using a player image and game character movement control method

[0001] This application is a National Phase Application of International Application No. PCT/JP2005/011778, filed Jun. 28, 2005, which claims the benefit under 35 U.S.C. 119(a-e) of JP 2004-194814 filed Jun. 30, 2004, which is herein incorporated by reference.

[0002] The present invention relates to an information processing system that allows a game scenario to proceed, by using a player image that is taken by a camera.

[0003] An information processing system has been proposed that uses a player image taken by a camera, as an input interface to enter a command and the like.

[0004] For example, Patent Document 1 discloses an image processor that generates a combined moving image obtained by combining a mirrored moving image of a player and a computer image such as a menu image, and a hand of the player displayed on the image is assumed as a pointing device to accept a directive such as menu selection.

• [0005] Patent Document 1: Japanese Patent Laid-Open Publication No. 2002-196855

[0006] Motion of a player and an operation for the image processor are directly related, in the input interface disclosed by the Patent Document 1, thereby showing a universally accepted operability. Therefore, this technique is, in particular, expected to be applied in entertainment system fields, targeting a wide range of users at all ages. On the other hand, in applying such a technique in the entertainment system, design ideas to further improve the game quality are required.

[0007] An object of the present invention is to provide a technique to further improve game quality in information processing systems that use a player image taken by a camera, as an input interface.

[0008] An information processing system according to a first aspect of the present invention to address the above subject, includes an area extraction means that extracts an area occupied by a player, from a mirrored moving image containing the player, a control means that generates a computer image containing a game character, and a display control means that allows a display device to display a superimposed moving image obtained by superimposing the mirrored moving image and the computer image, wherein, the control means moves the game character along the area occupied by the player in the superimposed moving image.

[0009] The information processing system may further include a player moving image accepting means that accepts an input of a player moving image obtained by taking images of the player by an image pickup device, and a mirrored moving image processing means that subjects the player moving image to mirroring.

[0010] Here, the area extraction means is capable of extracting an area occupied by the player, on the basis of a difference between an image not containing the player, previously obtained, and the player moving image.

[0011] At this time, it is desirable that the display control means allows the display device to display an image prompting the player to get out of an image frame of the image pickup device, so as to obtain an image that does not contain the player.

[0012] The control means allows the game character to appear from one or multiple starting points provided on one side of a screen, and to move the game character toward one or multiple goals provided on another side of the screen, thereby further improving the game quality.

[0013] In addition, the control means determines whether or not an area occupied by the player exists in front of the game character in the moving direction thereof and below the game character. When there is no existence of an area occupied by the player below the game character, the game character is made to fall, and when an area occupied by the player exists in front of the game character, the game character is made to move obliquely upward in the moving direction of the game character.

[0014] A game character movement control method according to a second aspect of the present invention to address the above subject is a game character movement control method in an information processing device connected to an image pickup device and a display device, including an area extraction process that extracts an area occupied by a player from a mirrored moving image based on a moving image containing the player taken by the image pickup device, a control process that generates a computer image containing a game character, and a movement control process that allows the display device to display a superimposed moving image obtained by superimposing the mirrored moving image and the computer image, and moves the game character regarding an area occupied by the player in the superimposed moving image as a route.

• [0030] 1 VIDEO CAMERA
• [0031] 2 ENTERTAINMENT APPARATUS
• [0032] 3 DISPLAY DEVICE
• [0033] 101 VIDEO IMAGE INPUT SECTION
• [0034] 102 IMAGE INVERTING SECTION
• [0035] 103 AREA EXTRACTING SECTION
• [0036] 104 MAIN CONTROLLER
• [0037] 105 CG GENERATOR
• [0038] 106 SUPERIMPOSED IMAGE GENERATOR
• [0039] 107 DISPLAY CONTROLLER

[0040] Hereinafter, an exemplary embodiment of the present invention will be explained.

[0041] FIG. 1 shows an example of a configuration of an entertainment system relating to the present embodiment. This entertainment system takes an image of player 4 who is at a position facing a display device 3, by using an analogue or digital video camera 1. Entertainment apparatus 2 continuously captures the obtained moving image, and the computer image (CG) generated by the entertainment apparatus 2 and a mirrored moving image of the moving image captured from the video camera 1 are superimposed, to be displayed in real time on the display device 3. Therefore, a motion of the player 4 is reflected on the superimposed image in real time, and the player can enjoy playing with this superimposed image. The entertainment apparatus 2 can generate the mirrored moving image by mirroring the moving image captured from the video camera 1 (right-left inversion of the image), but another method may also be available. For example, the mirroring may be performed on the video camera 1.

[0042] The entertainment apparatus 2 is implemented by an information processing device that forms required functions using a computer program. For example, as illustrated in the hardware configuration shown in FIG. 2, the entertainment apparatus 2 according to the present embodiment includes two buses, main bus B1 and sub bus B2, to which multiple semiconductor devices, each having a specific function, are connected. Those buses B1 and B2 are connected to each other or disconnected from each other via a bus interface INT.

[0043] The main bus B1 is connected to a main CPU 310 that is a primary semiconductor device, a main memory 311 made up of RAM, a main DMAC (Direct Memory Access Controller) 312, a MPEG (Moving Picture Experts Group) decoder (MDEC) 313, and a Graphic Processing Unit (hereinafter, referred to as “GPU”) 314 that incorporates a frame memory 315 serving as a drawing memory. The GPU 314 is connected to a CRTC (CRT Controller) 316 that generates a video signal allowing data drawn in the frame memory 315 to be displayed on the display device 3.

[0044] The main CPU 310 reads a startup program via the bus interface INT from the ROM 323 on the sub bus B2 at the time of starting the computer, and executes the startup program to activate the operating system. In addition, the main CPU 310 controls the media drive 327 to read an application program and data from the media 328 mounted on this media drive 327, and stores the read-in application program and data in the main memory 311. Furthermore, the main CPU 310 applies geometric processing (coordinate value calculation processing) to represent shape, movement, and the like, of an object, on various data read from the media 328, as in, for example, three-dimensional object data (coordinate values of vertices (representative points) of a polygon, etc.), made up of multiple basic graphics (polygons). Then, a display list is generated, which contains polygon definition information by the geometric processing. The polygon definition information includes as contents; a shape and a drawing position of the polygon used, and specifications of components constituting the polygon, such as type, color tone, and texture.

[0045] The GPU 314 is a semiconductor device having the functions of storing drawing context (drawing data containing polygon components), carrying out rendering processing (drawing processing) by reading necessary drawing context according to the display list notified from the main CPU 310, and drawing polygons in the frame memory 315. The frame memory 315 may also be used as a texture memory. Therefore, a pixel image in the frame memory 315 can be pasted as texture to a polygon to be drawn.

[0046] The main DMAC 312 is a semiconductor device having the functions of carrying out DMA transfer control over each of the circuits connected to the main bus B1, and also carrying out DMA transfer control over each of the circuits connected to the sub bus B2 according to the condition of the bus interface INT. The MDEC 313 is a semiconductor device having the functions of operating in parallel with the main CPU 310, and extending data having been compressed according to MPEG (Moving Picture Experts Group) format, JPEG (Joint Photographic Experts Group) format, or the like.

[0047] The sub bus B2 is connected to sub CPU 320 made up of microprocessor and the like, sub memory 321 made of RAM, sub DMAC 322, ROM 323 that stores a control program such as operating system, a sound processing-use semiconductor device (SPU: Sound Processing Unit) 324 that reads sound data accumulated in the sound memory 325 and outputs the data as audio output, a communication controller (ATM) 326 that sends information to and receives information from an external device via a network, not illustrated, and a media drive 327 to mount media 328 such as a CD-ROM and a DVD-ROM, and an input unit 331.

[0048] The sub CPU 320 carries out various operations according to the control program stored in the ROM 323. The sub DMAC 322 is a semiconductor device that carries out control such as DMA transfer over each of the circuits connected to the sub bus B2, only in the state in which the bus interface INT disconnects the main bus B1 from the sub bus B2. The input unit 331 is provided with a connection terminal 332 through which an input signal from the operation device 335 is inputted, a connection terminal 333 through which an image signal from the video camera 1 is inputted, and a connection terminal 334 through which a sound signal is inputted from video camera 1, or the like. It is to be noted that in the present embodiment, an explanation will be given regarding images only, and explanations regarding sound will be omitted for convenience.

[0049] The computer as thus configured operates as the entertainment apparatus 2, when the main CPU 310, the sub CPU 320, and the GPU 314 read necessary computer programs from a recording medium such as the ROM 323 and the media 328, and executes those computer programs.

[0050] FIG. 3 is a functional block diagram of this entertainment apparatus 2. In other words, a video image input section 101, an image inverting section 102, an area extracting section 103, a main controller 104, a CG generator 105, a superimposed image generator 106, and a display controller 107 are formed in the entertainment apparatus 2.

[0051] The video image input section 101 captures a video image taken by the video camera 1. The video image is a moving image, and the video image input section 101 continuously captures the images transferred from the video camera 1.

[0052] The image inverting section 102 subjects the video image captured by the image input section 101 to mirroring, that is, right-left inversion process. FIG. 4A shows one example of a video image 200 of a player, which is obtained by taking an image of the player and subjected to mirroring. FIG. 4A includes the mirrored image 5 of the player. Subsequent processing is performed on this mirrored video image. If the mirroring of the video image is set to be performed on the video camera 1, it is possible to eliminate the image inverting section 102.

[0053] The area extracting section 103 carries out processing to extract an area where the image of the player is placed from the video image inverted by the image inverting section 102.

[0054] In the present embodiment, the area extracting section 103 stores in advance, as a reference image 103a, one frame of the video image in which the image of the player is not placed. One example of the reference image 103a being associated with FIG. 4A is shown in FIG. 4B. On the basis of a difference between the reference image and the video image, an area where the image of the player 5 is placed is extracted. The area where the image of the player 5 is placed, being associated with FIG. 4A and FIG. 4B, is shown in FIG. 4C. FIG. 4C shows the area 6 that is obtained as a difference between FIG. 4A and FIG. 4B.

[0055] Therefore, in order to take an image in which the player is not contained, the entertainment apparatus 2 prompts the player to get out of the screen. In this condition, the area extracting section 103 extracts one frame of the video image that the image inverting section 102 has inverted, and then stores the extracted image as the reference image 103a.

[0056] Specifically, before starting play on the entertainment apparatus 2 according to a directive from the player, the entertainment apparatus 2 makes a display prompting the player to get out of the screen, and extracts one frame of the video image after a lapse of a predetermined period of time. On this occasion, it is desirable that a countdown is displayed so as to allow the player to be aware of the time when the image is taken.

[0057] It is to be noted that a method to extract, from the video image, the area where the image of the player is placed is not limited to the above procedure. For example, it is also possible to detect a difference between a before-frame and an after-frame based on the motion of the player, or to extract an area where the image of the player is placed in the video image by the use of pattern recognition or the like.

[0058] The processing by the area extracting section 103, to extract the area where the image of the player is placed, is performed continuously in real time. That is, along with the motion of the player, the area shown in FIG. 4C varies sequentially.

[0059] The main controller 104 conducts overall control of the entertainment system. For example, when the entertainment apparatus 2 is executing a game program, the main controller 104 decides a story for the game according to the program. In addition, when the main controller 104 decides the story, it may refer to a result of determination of a determining section 103. Details of this procedure will be explained later. In other words, the entertainment apparatus 2 executes a program recorded in a recording medium such as a DVD-ROM or a CD-ROM, thereby configuring the functional block as shown in FIG. 3, and executes various processes of the present embodiment.

[0060] The CG generator 105 generates various computer images along with the game story line according to a directive from the main controller 104. In the present embodiment, the CG generator 105 displays, on the screen, a character 10 as shown in FIG. 5 according to a predetermined rule. The character 10 moves in the screen under control of the main controller 104. At this time, it is desirable that the CG generator 105 changes the mode of how the character 10 is displayed, such as, for example, normal walking, climbing, or falling. In the present example, the character 10 is designed to represent a human being who is wearing a hat, but it is not limited to this example. The character may be an animal, a vehicle, or the like.

[0061] In addition, the CG generator 105 generates a computer image of an entrance that is a starting point for the character 10 and a computer image of an exit that is a goal for the character 10. The entrance and the exit may be treated as fixed images. Furthermore, there may be more than one entrance and exit. In the present embodiment, two entrances are provided on the right edge of the screen and two exits are provided on the left edge of the screen. The two entrances are separately arranged one above the other, respectively, and the two exits are separately arranged one above the other, respectively, as well.

[0062] The superimposed image generator 106 superimposes the video image having been subjected to mirroring by the image inverting section 102, and the computer image generated by the CG generator 105. For example, the superimposed image generator 106 superimposes the video image 200 as shown in FIG. 4A, the character 10 as shown in FIG. 5, and the fixed images of the entrance 21 and the exit 22, whereby the superimposed image 400 as shown in FIG. 6 is generated. For ease of understanding, in FIG. 6, the background to the rear of the player 5 is omitted in the video image that has been subjected to mirroring.

[0063] The area where the image of the player is placed in FIG. 6 corresponds to the area 6 extracted by the area extracting section 103 (see FIG. 4C). In other words, the main controller 104 is able to figure out the area where the image of the player 5 is placed in real time.

[0064] The display controller 107 displays on the display device 3 the superimposed image generated by the superimposed image generator 106. In other words, the player is allowed to conduct various operations in the entertainment system, while the player is viewing the superimposed image obtained by superimposing the mirrored moving image of himself or herself and the computer image.

[0065] Next, control of movement of the character 10, which is carried out by the main controller 104, will be explained. In the present embodiment, it is assumed that the character 10 moves along the area of the player extracted by the area extracting section 103. Specifically, the character 10 moves assuming the area of the player as a route.

[0066] In other words, as shown in FIG. 6, the characters 10 appearing from the entrances 21a and 21b assume the area of the player 5 as a route, and move from the right side to the left side toward the exits 22a and 22b. In this situation, if a route does not exist, the character 10 may fall. Hereinafter, an explanation will be given assuming that the horizontal direction of the screen is the X-axis and the vertical direction of the screen is the Y-axis.

[0067] Here, since the main controller 104 is capable of recognizing the area of the player in real time, the main controller 104 controls the action of the character 10 as appropriate in response to a route change due to the motion of the player. As thus controlled, the character 10 looks as if it is moving along with the player.

[0068] On the other hand, by the player changing his or her own posture on the screen to define the route of the character 10, a destination of each of the characters 10 is controlled in real time. For example, by getting onto the arm of the character 10 who has appeared from the entrance 21a, it is possible to make a display in which the character moves while passing over the arm. If nothing is done about the character 10 who has appeared from the entrance 21b, the character 10 ends up falling.

[0069] The main controller 104 may be able to further improve the game quality in the entertainment apparatus 2 by setting the exit 22 as a destination with respect to each character 10 according to the game scenario, making various types of characters appear, giving a bonus point in response to the player's control, and the like.

[0070] Next, specific examples of controlling the movement of the character 10 in the present embodiment will be explained. As shown in FIG. 7A, the character 10 basically proceeds in one direction, from right to left, in the horizontal direction. On the other hand, as for the vertical direction, if there is a route having a wall in front, the character 10 is controlled to proceed in the direction obliquely upward. If there is no route below, the character is controlled so as to go down. It is to be noted that if the character 10 falls on another route in mid-course while going down, the character 10 is controlled to move horizontally again. This may correspond to a motion of the player picking up the character 10 that is falling, a movement of the character 10 that is slipping down a slope, or the like.

[0071] Since the character 10 is required to be moved in real time, together with the player's area extraction process and the like, it is desirable that the control of the movement of the character 10 is performed in a simple process.

[0072] Therefore, in the present embodiment, conditions of the route in the foreside and in the downside of the character 10 are determined, thereby defining a subsequent moving direction. In other words, as shown in FIG. 7A, there are provided a forward detection area 11 in the foreside of the moving direction, and a downward detection area 12 in the downside. It is desirable that some parts of both the detection areas 11 and 12 may overlap the area where the character 10 occupies.

[0073] These detection areas 11 and 12 are areas to detect to what extent a part accepted as a route is contained in those areas. For example, each detection result may be represented as a ratio, such as 40%. This result indicates that a part accepted as a route occupies 40% of the detection area. This detection result can be obtained, for example, by dividing the detection areas 11 and 12 into a matrix with a predetermined size, determining whether or not each element of the matrix fits into the route, and then calculating a ratio of a part of the matrix corresponding to the route.

[0074] FIG. 7B indicates an example in which the character 10a and the character 10b are passing along the route 13. At this time, the character 10a is detected as occupying around 50% of both the forward detection area 11a and the downward detection area 12a The character 10b is detected as occupying around 5% of the forward detection area 11b, and 50% of the downward detection area 12b.

[0075] On the basis of the detection results regarding the route in the detection areas 11 and 12, the main controller 104 controls the movement of the character 10 according to the flow diagram as shown in FIG. 8. The main controller 104 allows multiple characters 10 to appear in the screen, and each of the characters 10 to move continuously and independently in real time according to the motion of the player, by repeatedly executing the process of the flow diagram as shown in FIG. 8 for each character 10.

[0076] Variable J is employed in the flow diagram above. In the case where there is no route below, that is, the player does not give a route to the character 10, this variable J is used so that the character 10 starts falling after a lapse of a certain period of time, without the character 10 falling immediately.

[0077] The main controller 104 allows the character 10 to appear from any of the starting points (entrances 21), and to conduct control as described below. It is to be noted that making the character 10 walk or run, according to the game scenario is also possible.

[0078] Firstly, the variable J is initialized to zero (S101).

[0079] Then, it is determined whether or not the current position is at a goal (exit 22) (S102). As a result of the determination, if the character reaches the goal (S102: Y), a goal process is performed according to the game scenario (S103). Here, in the goal process, for example, the character 10 is erased from the screen, and the player can acquire a point being added according to a predetermined rule.

[0080] On the other hand, if the character 10 has not reached the goal yet (S102: N), it is determined whether or not there exists a route in front, on the basis of the detection result of the forward detection area 11 (S104). It is possible to determine whether or not there exists a route in front, for example, by the ratio of the route occupying the forward detection area 11. By way of example, if the ratio is equal to or more than a predetermined value, for instance, 30%, it is possible to determine that a route exists in front.

[0081] As a result, if it is determined that a route exists in front (S104: Y), this route is assumed as an upward slope, the variable J is reset (S105), and climbing process is carried out (S106).

[0082] As shown in FIG. 9A, the climbing process allows the character 10 to move mostly in the upper direction with a larger Y-axis shift amount, and to move in the left direction with an X-axis shift amount being smaller than ordinary horizontal movement. The shift amount is set to a value suitable for the game scenario. Then, the process returns to step S102, and the subsequent process is repeated.

[0083] If it is determined that a route does not exist in front (S104: N), it is determined whether or not there exists a route below, on the basis of the detection result of the downward detection area 12 (S107). It is possible to determine whether or not there exists a route below, for example, by the ratio of the route occupying the downward detection area 12. As a way of example, if the ratio is equal to or more than a predetermined value, for instance, 10%, it is determined that a route exists in front.

[0084] Consequently, if it is determined that a route exists below (S107: Y), this route is assumed as being horizontal, and the variable J is reset (S108).

[0085] It is further determined whether or not the ratio of the route occupying the downward detection area 12 is equal to or more than a predetermined value, for instance, 90% (S109). This is determined in order to know the condition of the character 10, such as being buried in the ground as shown in FIG. 9B. In other words, if the ratio of the route occupying the downward detection area 12 is over a predetermined reference value, it is determined that the character is in a condition such as being buried in the route.

[0086] As a result, if the ratio of the route occupying the downward detection area 12 is less than the predetermined ratio (S109: N), the character 10 is moved in the left direction with respect to the X-axis, as a normal horizontal movement (S110). The shift amount is set to a value suitable for the game scenario. For example, if the character is running, the shift amount is set to be larger than the case where the character is walking. In the situation above, it is desirable to display the character 10 in such a manner as “walking”, “skipping”, and the like, according to the game scenario.

[0087] On the other hand, if the ratio of the route occupying the downward detection area 12 is equal to or more than a predetermined ratio (S109: Y), the character 10 is moved somewhat upwardly with respect to Y-axis, in addition to the above horizontal movement (S111). The character 10 is moved in this manner in order to avoid appearing to move while being buried in the route.

[0088] If it is determined that a route does not exist below (S107: N), 1 is added to J (S112). Then, it is determined whether or not J reaches a predetermined value (S113). That is, it is determined whether or not the condition that the route does not exist below, is still continuing.

[0089] As a result of the determination, if the value of J does not reach the predetermined value (S113: Y), the processing returns to step S102, and subsequent processing is repeated.

[0090] On the other hand, when J reaches the predetermined value (S113: N), a falling process is performed (S114). In the falling process, the character 10 is moved downwardly at a predetermined speed. At this time, it is desirable to display a falling motion.

[0091] It is to be noted that processing as below is performed for the character 10 after it has fallen. That is, if the character 10 goes out of the screen after falling, the character 10 is eliminated from a control target. In addition, detection of the downside by the downward detection area is continued during the falling motion (S115), and if a route is detected on the downside (S115: Y), the process returns to step S108 and horizontal movement is restarted. For example, since the downside is not detected on the left end of the player's head, the character 10 starts falling. Then, upon falling on the shoulder part, a route on the downside is detected, and horizontal movement is restarted. Along the downward slope, falling and horizontal movement are repeated alternately in a short cycle, whereby the character 10 goes down stepwise.

[0092] It is to be noted that specific moving rules of the character 10 are not limited to the above example, and various methods can be employed. In addition, the route of the character 10 is not limited to the area where the image of the player is placed. For example, it is also possible to move the character 10 along an item such as a stick, operated by the player, that is regarded as the route.