Sensor calibration apparatus, sensor calibration method, program, storage medium, information processing method, and information processing apparatus

[0002] The present invention relates to a technique for calibrating a position/posture sensor, and a technique for obtaining parameters used upon transforming an output value of the position/posture sensor for the purpose of measuring the position and posture of an image sensing device using the position/posture sensor.

[0003] In recent years, studies about mixed reality that aims at seamless joint of real and virtual spaces have been extensively made. An image display apparatus which presents mixed reality is implemented by superimposing an image of a virtual space (e.g., a virtual object, text information, and the like rendered by computer graphics) onto an image of a real space photographed by an image sensing device such as a video camera or the like.

[0004] As applications of such image display apparatus, new fields different from conventional virtual reality such as operation assistance that superimposes the state in a body onto the body surface of a patient, a mixed reality game in which a player fights against virtual enemies that swim on the real space, and the like are expected.

[0005] A common requirement for these applications involves the precision level of alignment between the real and virtual spaces, and many efforts have been conventionally made in this respect.

[0006] A problem of alignment in mixed reality amounts to obtaining the three-dimensional (3D) position and posture of an image sensing device on a world coordinate system set on the real space (to be simply referred to as a world coordinate system hereinafter). As a method of solving these problems, it is a common practice to use a 3D position/posture sensor such as a magnetic sensor, ultrasonic wave sensor, and the like.

[0007] In general, the output value of a 3D position/posture sensor indicates the position and posture of a measurement point on a sensor coordinate system which is uniquely defined by the sensor, but is not that of the image sensing device on the world coordinate system. Taking the Polhemus FASTRAK (magnetic sensor) as an example, the position and posture of a receiver on a coordinate system defined by a transmitter are obtained as the sensor output. Therefore, the sensor output value cannot be directly used as the position and posture of the image sensing device on the world coordinate system, and must undergo some calibration processes. More specifically, coordinate transformation that transforms the position and posture of a measurement point into those of the image sensing device, and coordinate transformation that transforms the position and posture on the sensor coordinate system into those on the world coordinate system are required. In this specification, information used to transform the sensor output value into the position and posture of the image sensing device on the world coordinate system will be referred to as calibration information.

[0008] FIG. 1 is a block diagram showing the functional arrangement of a general image display apparatus which presents mixed reality.

[0009] A display screen 110 and video camera 120 are fixed to a head-mount unit 100. When the user (not shown) wears the head-mount unit 100 so that the display screen 110 is located in front of the user's eye, a scene in front of the user's eye is captured by the video camera 120. Therefore, if the image captured by the video camera 120 is displayed on the display screen 110, the user observes a scene in front of the eye, which the user may observe by the naked eye if he or she does not wear the head-mount unit 100, via the video camera 120 and display screen 110.

[0010] A position/posture sensor 130 is a device for measuring the position and posture of a measurement point, fixed to the head-mount unit 100, on the sensor coordinate system, and comprises, e.g., the Polhemus FASTRAK as a magnetic sensor including a receiver 131, transmitter 133, and sensor controller 132. The receiver 131 is fixed to the head-mount unit 100 as a measurement point, and the sensor controller 132 measures and outputs the position and posture of the receiver 131 on the sensor coordinate system with reference to the position and posture of the transmitter 133.

[0011] On the other hand, an arithmetic processing unit 170 comprises a position/posture information transformer 140, memory 150, and image generator 160, and can be implemented by, e.g., a single versatile computer. The position/posture information transformer 140 transforms a measurement value input from the position/posture sensor 130 in accordance with calibration information held by the memory 150 so as to calculate the position and posture of the video camera 120 on the world coordinate system, and outputs the calculated position and posture as position/posture information. The image generator 160 generates a virtual image in accordance with the position/posture information input from the position/posture information transformer 140, superimposes that virtual image on an actual image captured by the video camera 120, and outputs superimposed that. The display screen 110 receives an image from the image generator 160, and displays it. With the above arrangement, the user (not shown) can experience as if a virtual object were present on the real space in front of the user's eye.

[0012] A method of calculating the position and posture of the video camera on the world coordinate system by the position/posture information transformer 140 will be described below using FIG. 2. FIG. 2 is a view for explaining the method of calculating the position and posture of the video camera on the world coordinate system.

[0013] In FIG. 2, let M_TWbe the position and posture of a sensor coordinate system 210 (a coordinate system having the position of the transmitter 133 as an origin) on a world coordinate system 200, M_STbe the position and posture of the measurement point (i.e., the receiver 131) of the position/posture sensor 130 on the sensor coordinate system 210, M_CSbe the position and posture of the video camera 120 viewed from the measurement point of the position/posture sensor 130, and M_CWbe the position and posture of the video camera 120 on the world coordinate system 200. In this specification, the position and posture of object B on coordinate system A are expressed by a viewing transformation matrix M_BA(4×4) from coordinate system A to coordinate system B (local coordinate system with reference to object B).

[0014] At this time, M_CWcan be given by:

[0015] M_CW=M_CS·M_STM_TW  (A)

[0016] In equation (A), M_STis the input from the position/posture sensor 130 to the position/posture information transformer 140, M_CWis the output from the position/posture information transformer 140 to the image generator 160, and M_CSand M_TWcorrespond to calibration information required to transform M_STinto M_CW. The position/posture information transformer 140 calculates M_CWbased on equation (A) using M_STinput from the position/posture sensor 130, and M_CSand M_TWheld in the memory 150, and outputs it to the image generator 160.

[0017] In order to attain accurate alignment between the real and virtual spaces, accurate calibration information must be set in the memory 150 by some means. A virtual image which is accurately aligned in the real space can be displayed only when the accurate calibration information is given.

[0018] Note that the holding form of the calibration information in the memory 150 is not limited to the viewing transformation matrix, and any other forms may be adopted as long as information can define the position and posture of one coordinate system viewed from the other coordinate system. For example, the position and posture may be expressed by a total of six parameters, i.e., three parameters that describe the position, and three parameters which express the posture using an Euler angle. Also, the posture may be expressed by four parameters, i.e., a three-valued vector that defines the rotation axis, and a rotation angle about that axis, or may be expressed by three parameters that express the rotation angle by the magnitude of the vector which defines the rotation axis.

[0019] Furthermore, the position and posture may be expressed by parameters which represent their inverse transformations (e.g., the position and posture of the world coordinate system 220 on the sensor coordinate system 210). In any of these cases, the position and posture of an object on a 3D space have only six degrees of freedom (three degrees of freedom for the position, and three degrees of freedom for the posture). Hence, unknown parameters required for calibration for this image display apparatus are a total of 12 parameters, i.e., six parameters required for transformation from the world coordinate system to the sensor coordinate system, and six parameters required for transformation from the position and posture of the measurement point to those of the video camera.

[0020] As one of known methods for setting the calibration information, the user or operator interactively changes 12 parameters (or 12 or more equivalent parameters) used to define M_CSand M_TWstored in the memory 150 via an input means (not shown), and makes adjustment by trial and error until accurate alignment is achieved.

[0021] Also, according to a calibration method proposed by Japanese Patent Application No. 2001-050990 (US AA 2002-95265), if one of M_CSand M_TWis obtained by some method, the remaining unknown parameters can be easily derived using a virtual image generated based on position/posture information fixed to a given value as a visual queue.

[0022] However, in the former method, since 12 unknown parameters must be adjusted at the same time, adjustment takes much time, and accurate calibration information cannot always be obtained. In the latter method, trial & error operations or operations using some calibration tool must be done upon deriving a parameter as a known one. Hence, these methods still have room for improvement.

[0023] The present invention has been made in consideration of the aforementioned problems, and has as its object to easily acquire calibration information required to transform the position and posture of an image sensing device measured by a sensor into those on a world coordinate system without using any special calibration tool.

[0024] In order to achieve the above object, for example, a sensor calibration method of the present invention comprises the following arrangement.

[0025] That is, a sensor calibration method for calculating calibration information of a position/posture sensor used to measure a position and posture of an image sensing device, characterized by comprising:

[0026] a sensor measured value input step of inputting measured values of the position/posture sensor obtained when the image sensing device is located at a plurality of image sensing positions and postures;

[0027] an image coordinate position input step of inputting values of image coordinate positions of a plurality of feature points on captured images sensed by the image sensing device at the plurality of image sensing positions and postures;

[0028] a world coordinate position input step of inputting values of world coordinate positions of the plurality of feature points on a world coordinate system; and

[0029] a calibration information calculation step of calculating the calibration information on the basis of the sensor measured values, actually measured values of the image coordinate positions of the feature points, and the world coordinate positions of the feature points.

[0030] In order to achieve the above object, for example, a sensor calibration apparatus of the present invention comprises the following arrangement.

[0031] That is, a sensor calibration apparatus for calculating calibration information of a position/posture sensor used to measure a position and posture of an image sensing device, characterized by comprising:

[0032] a sensor measured value input unit adapted to input measured values of the position/posture sensor obtained when the image sensing device is located at a plurality of image sensing positions and postures;

[0033] an image coordinate position input unit adapted to input values of image coordinate positions of a plurality of feature points on captured images sensed by the image sensing device at the plurality of image sensing positions and postures;

[0034] a world coordinate position input unit adapted to input values of world coordinate positions of the plurality of feature points on a world coordinate system; and

[0035] a calibration information calculation unit adapted to calculate the calibration information on the basis of the sensor measured values, actually measured values of the image coordinate positions of the feature points, and the world coordinate positions of the feature points.

[0036] In order to achieve the above object, for example, a sensor calibration method of the present invention comprises the following arrangement.

[0037] That is, an information processing method for calculating parameters required to transform a position and posture of an image sensing unit measured by a sensor into a position and posture on a world coordinate system, comprising the step of:

[0038] acquiring a plurality of captured images obtained by sensing images of a real space on which a plurality of markers whose world coordinate positions are known are arranged, at a plurality of positions and postures using the image sensing unit, and measured values of the sensor upon sensing the respective captured images;

[0039] detecting positions of the markers included in the respective captured images; and

[0040] calculating the parameters using the measured values of the sensor, the detected marker positions, and the detected marker world coordinate positions,

[0041] characterized by selecting captured images used upon calculating the parameters from the plurality of acquired captured images in accordance with a user's instruction.

[0042] In order to achieve the above object, for example, a program of the present invention comprises the following arrangement.

[0043] That is, a program for making a computer implement an information processing method for calculating parameters required to transform a position and posture of an image sensing unit measured by a sensor into a position and posture on a world coordinate system, characterized by comprising:

[0044] a program of a step of acquiring a plurality of captured images obtained by sensing images of a real space on which a plurality of markers whose world coordinate positions are known are arranged, at a plurality of positions and postures using the image sensing unit, and measured values of the sensor upon sensing the respective captured images;

[0045] a program of a step of detecting positions of the markers included in the respective captured images;

[0046] a program of a step of calculating the parameters using the measured values of the sensor, the detected marker positions, and the detected marker world coordinate positions; and

[0047] a program of a step of selecting captured images used upon calculating the parameters from the plurality of acquired captured images in accordance with a user's instruction.

[0048] In order to achieve the above object, for example, an information processing apparatus of the present invention comprises the following arrangement.

[0049] That is, an information processing apparatus for calculating parameters required to transform a position and posture of an image sensing unit measured by a sensor into a position and posture on a world coordinate system, characterized by comprising:

[0050] an image sensing unit for sensing images of a real space on which a plurality of markers whose world coordinate positions are known are arranged;

[0051] a sensor for measuring a position and posture of the image sensing unit;

[0052] input unit adapted to input a plurality of captured images obtained by sensing the images of the real space on which the plurality of markers whose world coordinate positions are known are arranged, at a plurality of positions and postures using the image sensing unit, and measured values of the sensor upon sensing the respective captured images;

[0053] detection unit adapted to detect positions of the markers included in the respective captured images;

[0054] selection unit adapted to select captured images used upon calculating the parameters from the plurality of acquired captured images in accordance with a user's instruction; and

[0055] calculation unit adapted to calculate the parameters using the measured values of the sensor, the detected marker positions, and the detected marker world coordinate positions in the selected captured images.

[0056] Other features and advantages of the present invention will be apparent from the following description taken in conjunction with the accompanying drawings, in which like reference characters designate the same or similar parts throughout the figures thereof.

[0057] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0058] FIG. 1 is a block diagram showing the functional arrangement of a general image display apparatus which presents mixed reality;

[0059] FIG. 2 is a view for explaining a method of calculating the position and posture of a video camera on a world coordinate system;

[0060] FIG. 3 is a block diagram showing the functional arrangement of a calibration information generation system which includes a calibration apparatus according to the first embodiment of the present invention;

[0061] FIG. 4 is a flow chart of processes executed by the calibration apparatus of the first embodiment of the present invention upon calculating calibration information;

[0062] FIG. 5 is a schematic block diagram for explaining the arrangement of a calibration apparatus according to the second embodiment of the present invention;

[0063] FIG. 6 is a block diagram for explaining the functional arrangement of the calibration apparatus according to the second embodiment of the present invention;

[0064] FIG. 7 is a view for explaining a GUI presented by the calibration apparatus according to the second embodiment;

[0065] FIG. 8 is a view for explaining a file menu of the GUI presented by the calibration apparatus according to the second embodiment;

[0066] FIG. 9 is a view for explaining a state wherein a mark indicating the image coordinate position of a marker, and additional information are superimposed on a real image on the GUI presented by the calibration apparatus according to the second embodiment;

[0067] FIG. 10 is a view for explaining a state wherein the mark indicating the image coordinate position of the marker, and additional information are superimposed on a real image on the GUI presented by the calibration apparatus according to the second embodiment, when the marker is not identified;

[0068] FIG. 11 is a view for explaining a state wherein the world coordinate position of the marker is rendered as CG, and is superimposed on a real image on the GUI presented by the calibration apparatus according to the second embodiment;

[0069] FIG. 12 is a view for explaining a data acquisition dialog of the GUI presented by the calibration apparatus according to the second embodiment;

[0070] FIG. 13 is a view for explaining a selected data manipulation area on the dialog shown in FIG. 12;

[0071] FIG. 14 is a view for explaining an initial value setup dialog of the GUI presented by the calibration apparatus according to the second embodiment;

[0072] FIG. 15 is a view for explaining a detection parameter setup dialog of the GUI presented by the calibration apparatus according to the second embodiment;

[0073] FIG. 16 is a view for explaining a detection parameter setup dialog of the GUI presented by the calibration apparatus according to the second embodiment;

[0074] FIG. 17 is a view for explaining a state wherein a color extraction result image is displayed on an image display area of the GUI shown in FIG. 7 in the calibration apparatus according to the second embodiment;

[0075] FIGS. 18A to 18C are views for explaining a sequence for automatically calculating approximate Local Transform values using a sensor measurement value in the calibration apparatus according to the second embodiment;

[0076] FIG. 19 is a flow chart showing the processing sequence of a sensor calibration apparatus according to the second embodiment;

[0077] FIG. 20 is a flow chart showing the sequence when the user of the sensor calibration apparatus uses the sensor calibration apparatus in the second embodiment;

[0078] FIG. 21 is a schematic block diagram for explaining the arrangement of a sensor calibration apparatus according to the third embodiment; and

[0079] FIG. 22 is a schematic block diagram for explaining the arrangement of a sensor calibration apparatus according to the fourth embodiment.

[0080] Preferred embodiments of the present invention will now be described in detail in accordance with the accompanying drawings.

[0081] The embodiments to be described hereinafter will explain a case wherein the calibration apparatus and method of the present invention are applied to calibration of the image display apparatus shown in FIG. 1.

[0082] [First Embodiment]

[0083] In order to calibrate the image display apparatus using the calibration apparatus of this embodiment, at least three points of landmarks (feature points), which are known to the world coordinate system, must be arranged in a real space as a display target of the image display apparatus. Assume that the landmarks have, e.g., different colors, so that the image coordinate positions of their images projected onto a captured image can be detected, and these landmarks can be identified from each other. The world coordinate system has a predetermined point as an origin, and specifies x-, y-, and z-axes in orthogonal directions from this origin. Also, assume that the coordinate positions of the three points of landmarks are known on this coordinate system. That is, the distances to the three points of landmarks in the x-, y-, and z-directions from the origin are measured in advance.

[0084] FIG. 3 is a block diagram showing the functional arrangement of a calibration information generation system including the calibration apparatus of this embodiment. As shown in FIG. 3, a calibration apparatus 300 of this embodiment comprises a world coordinate holding unit 310, image coordinate acquisition unit 320, data management unit 330, calibration information calculation unit 340, and instruction unit 350. A head-mount unit 100 and position/posture sensor 130 of the image display apparatus to be calibrated are connected to the calibration apparatus 300.

[0085] The world coordinate holding unit 310 holds coordinate data (data measured in advance by the above method or the like) of the respective landmarks on the world coordinate system, and outputs these data in accordance with a request from the data management unit 330. Also, the world coordinate holding unit 310 holds information (landmark color, identification number) unique to each landmark in association with the coordinate data of the landmarks.

[0086] In accordance with a request from data management unit 330, the image coordinate acquisition unit 320 receives an image captured by a video camera 120, specifies the image coordinate position and identification number (identifier) of (a) landmark(s) which appear(s) in that image, and outputs such information to the data management unit 330. As a method of specifying the coordinate position and identification number of a landmark in an image, a region having a color of each landmark is extracted from an image with reference to the color information of the landmarks held by the world coordinate holding unit 310, and the barycentric position in the extracted region is obtained. In this way, the coordinate position of each landmark in the image can be obtained. Furthermore, an identification number corresponding to the color of each region is specified with reference to the identification number information of each landmark color held by the world coordinate holding unit 310. As a result, the landmark itself and its position in the image can be specified.

[0087] Upon receiving a “data acquisition” instruction from the instruction unit 350, the data management unit 330 receives the image coordinate position and identification number of a given landmark from the image coordinate acquisition unit 320, receives the world coordinate position of the corresponding landmark from the world coordinate holding unit 310, receives a sensor measurement value at the same time with a captured image from the position/posture sensor 130, adds a set of [world coordinate—sensor measurement value—image coordinate] in a data list, and holds that list. Also, the data management unit 330 outputs the generated data list to the calibration information calculation unit 340 in accordance with a request from the calibration information calculation unit 340.

[0088] Upon reception of a “calibration information calculation” instruction from the instruction unit 350, the calibration information calculation unit 340 receives the data list from the data management unit 350, calculates calibration information based on the data list, and outputs the calculated calibration information.

[0089] The instruction unit 350 sends the “data acquisition” instruction to the data management unit 330 when the operator (not shown) inputs a data acquisition command, and the “calibration information calculation” instruction to the calibration information calculation unit 340 when the operator inputs a calibration information calculation command. The operator can input a command to the instruction unit 350 by pressing a key assigned a specific command using, e.g., a keyboard. Also, a command may be input on a GUI displayed on a display. That is, the command input method is not particularly limited.

[0090] FIG. 4 is a flow chart showing processes of the calibration apparatus of this embodiment upon calculating calibration information. Note that a program code according to this flow chart is stored in a memory such as a RAM, ROM, or the like (not shown) in the apparatus of this embodiment, and is read out and executed by a CPU (not shown).

[0091] The instruction unit 350 determines in step S1010 whether or not the operator has input a data acquisition command. If the data acquisition command has been input, the flow advances to step S1020. In the following description, let V_jbe the viewpoint position where the video camera 120 is located at the input timing of the data acquisition command.

[0092] In step S1020, the data management unit 330 receives, from the position/posture sensor 130 (sensor controller 132), measurement value M^Vj_STobtained when the video camera 120 is located at viewpoint position V_j.

[0093] In step S1030, the data management unit 330 receives, from the image coordinate acquisition unit 320, identification number k and image coordinate position u^VjQkof landmark Q_k(k=1, 2, 3, . . . ) on an image captured by the video camera 120 located at viewpoint position V_j. If a plurality of landmarks appear in the captured image, this input is repeated for respective landmarks.

[0094] In step S1040, the data management unit 330 receives, from the world coordinate holding unit 310, world coordinate position X^Qk_Wof each landmark Q_k(corresponding to identification number k) input from the image coordinate acquisition unit 320.

[0095] In step S1050, the data management unit 330 adds input data to data list L_ifor each detected landmark. More specifically, the image coordinate position of landmark Q_kis given by uⁱ=u^VjQk=[uⁱ_x, uⁱ_y]^T, its world coordinate position is given by Xⁱ_W=X^Qk_W=[xⁱ_W, yⁱ_W, zⁱ_W, l]^T, and the sensor output at that time is given by Mⁱ_ST=M^Vj_ST. Then, a set of [uⁱ, Xⁱ_W, Mⁱ_ST] is registered in the list as the i-th data. Note that i indicates a value obtained by adding 1 to the total number of data currently registered in the list.

[0096] With the above process, data acquisition is made.

[0097] The instruction unit 350 checks in step S1060 if the data list acquired so far has information that suffices to calculate calibration information. If the data list does not meet a given condition, the flow returns to step S1010 to wait for input of the next data acquisition command. On the other hand, if the data list meets a calibration information calculation condition, the flow advances to step S1070. The calibration information calculation condition may include, for example, that data associated with three or more points of different landmarks be acquired, data be acquired at a plurality of viewpoint positions, and the total number of data be 6 or more. However, since the precision of calculated calibration information improves with increasing variety of input data, a condition that requires more data may be set.

[0098] It is checked in step S1070 if the operator has input a calibration information calculation command. If the calibration information calculation command has been input, the flow advances to step S1080; otherwise, the flow returns to step S1010 to wait for input of the next data acquisition command.

[0099] The calibration information calculation unit 340 internally expresses information that pertains to the posture of the calibration information to be calculated using a three-valued vector, which defines a rotation angle by the magnitude of the vector, and defines the direction of a rotation axis by the direction of the vector. M_CSis expressed by the position (x_cs, y_cs, z_cs) and posture (ξ_cs, ψ_cs, ζ_cs) of the receiver 131 on the coordinate system defined by the video camera 120, and M_TWis expressed by the position (x_TW, y_TW, z_TW) and posture (ξ_TW, ψ_TW, ζ_TW) of the world coordinate system 200 on the sensor coordinate system 210. Twelve unknown parameters are expressed by vector s=[x_csy_csz_csξ_csψ_csζ_csx_TWy_TWz_TWξ_TWζ_TW].

[0100] In step S1080, the calibration information calculation unit 340 gives an appropriate initial value (e.g., s=[0 0 0 2/π0 0 0 0 0 2/π0 0]^T) to vector s.

[0101] In step S1090, the calibration information calculation unit 340 calculates a theoretical value u′ⁱ=[u′ⁱ_x, u′ⁱ_y]^Tof the image coordinate position of a given landmark for respective data L_i(i=1, 2, . . . , N) in the list on the basis of sensor output Mⁱ_ST, world coordinate position Xⁱ_W, and current s. Note that the theoretical value of the image coordinate position of the landmark indicates data of the position (coordinates) where the landmark is to appear in the image, which position is calculated from the obtained sensor output and the world coordinate position of the landmark. u′ⁱcan be obtained by a function:

[0102] u′ⁱ=Fⁱ(s)  (B)

[0103] which is defined by sensor output Mⁱ_STand world coordinate position Xⁱ_Wusing s as a parameter.

[0104] More specifically, viewing transformation matrix M_csis calculated from (x_CS, y_CS, z_CS) and (ξ_CS, ψ_CS, ζ_CS) as elements of s, and viewing transformation matrix M_TWis similarly calculated from (x_TW, y_TW, z_TW) and (ξ_TW, ψ_TW, ζ_TW) on the basis of: $\begin{array}{cc}M\left(x,y,z,\xi ,\psi ,\zeta \right)=\left[\begin{array}{cccc}1& 0& 0& x\\ 0& 1& 0& y\\ 0& 0& 1& z\\ 0& 0& 0& 1\end{array}\right]·\left[\begin{array}{cccc}\frac{{\xi }^2}{{\theta }^2}\left(1-\cos \theta \right)+\cos \theta & \frac{\mathrm{\xi \psi }}{{\theta }^2}\left(1-\cos \theta \right)-\frac{\zeta }{\theta }\sin \theta & \frac{\mathrm{\xi \zeta }}{{\theta }^2}\left(1-\cos \theta \right)+\frac{\psi }{\theta }\sin \theta & 0\\ \frac{\mathrm{\psi \xi }}{{\theta }^2}\left(1-\cos \theta \right)+\frac{\zeta }{\theta }\sin \theta & \frac{{\psi }^2}{{\theta }^2}\left(1-\cos \theta \right)+\cos \theta & \frac{\mathrm{\psi \zeta }}{{\theta }^2}\left(1-\cos \theta \right)-\frac{\zeta }{\theta }\sin \theta & 0\\ \frac{\mathrm{\zeta \xi }}{{\theta }^2}\left(1-\cos \theta \right)-\frac{\psi }{\theta }\sin \theta & \frac{\mathrm{\xi \psi }}{{\theta }^2}\left(1-\cos \theta \right)+\frac{\xi }{\theta }\sin \theta & \frac{{\zeta }^2}{{\theta }^2}\left(1-\cos \theta \right)+\cos \theta & 1\\ 0& 0& 0& 0\end{array}\right]\left(\mathrm{for}\theta =\sqrt{{\xi }^2+{\psi }^2+{\zeta }^2}\right)& \left(C\right)\end{array}$

[0105] Then, camera coordinate position Xⁱ_cof that landmark is calculated by:

[0106] Xⁱ_C=[xⁱ_cyⁱ_czⁱ_C1]^T=M_CS·Mⁱ_ST·M_TW·Xⁱ_W  (D)

[0107] Finally, the theoretical value of the image coordinate position is calculated by: $\begin{array}{cc}{u}_x^{\prime i}=-f\times x_c^i/z_c^i{u}_y^{\prime i}=-f\times y_c^i/z_c^i& \left(E\right)\end{array}$

[0108] where f is the focal length of the video camera 120.

[0109] In step S1100, the calibration information calculation unit 340 calculates error Δuⁱbetween theoretical value u′ⁱand measured value uⁱof the image coordinate position of the corresponding landmark for each data L_iin the list by:

[0110] Δuⁱ=uⁱ−u′ⁱ  (F)

[0111] In step S1110, the calibration information calculation unit 340 calculates Jacobian matrix Jⁱ_us(=∂u/∂s) of 2 rows×12 columns, which has, as elements, solutions obtained by partially differentiating the right-hand side of equation (B) by respective elements of vector s for each data L_iin the list. More specifically, the unit 340 calculates, on the basis of equations (E) and (D), Jacobian matrix Jⁱ_ux(=∂u/∂x) of 2 rows×3 columns, which has, as elements, solutions obtained by partially differentiating the right-hand side of equation (E) by respective elements of camera coordinate position Xⁱ_c, and Jacobian matrix Jⁱ_xs(=∂x/∂s) of 3 rows×12 columns, which has, as elements, solutions obtained by partially differentiating the right-hand side of equation (D) by respective elements of vector s, and then calculates them as:

[0112] Jⁱ_us=Jⁱ_uxJⁱ_xs  (G)

[0113] In step S1120, the calibration information calculation unit 340 calculates correction value Δs of s on the basis of errors Δuⁱand Jacobian matrices Jⁱ_uscalculated for all data L_iin the list calculated in steps S1100 and S1110. More specifically, the unit 340 generates vectors U=[Δu¹Δu². . . Δu^N]^Tand Φ=[J¹_usJ²_us. . . . J^N_us]^Tby vertically arranging errors Δuⁱand Jacobian matrices Jⁱ_uscalculated for all data, and calculates them using a generalized inverse matrix of Φ as:

[0114] Δs=(φ^Tφ)⁻¹φ^TU  (H)

[0115] In step S1130, the calibration information calculation unit 340 corrects s using correction value Δs calculated in step S1120 by

[0116] s+Δs→s  (I)

[0117] The calibration information calculation unit 340 determined in step S1140 whether or not the calculation converges, using some criteria, e.g., by seeing if elements of U are sufficiently small or if Δs is sufficiently small. If the calculation does not converge, processes in step S1090 and subsequent steps are repeated using corrected s.

[0118] In step S1150, the calibration information calculation unit 340 outputs obtained s as calibration information. As the calibration information, for example, two viewing transformation matrices calculated from s are output. The output form may be s itself, or information described by any other position/posture description methods may be output.

[0119] Note that the world coordinate holding unit 310, image coordinate acquisition unit 320, data management unit 330, calibration information calculation unit 340, and instruction unit 350 can be implemented by, e.g., a single versatile computer.

[0120] <Modification 1>

[0121] In this embodiment, the aforementioned calibration apparatus and method are applied to calibration of the image display apparatus which uses the magnetic sensor FASTRAK available from Polhemus as the position/posture sensor 130. However, the position/posture sensor 130 that can be calibrated by the calibration apparatus and method of this embodiment is not limited to the FASTRAK. That is, the calibration apparatus and method of this embodiment can calibrate image display apparatuses that use any other position/posture sensors (e.g., Flock of Birds available from Ascension Technology, OPTOTRAK available from Northern Digital Inc., and the like) in measurement of the position and posture.

[0122] <Modification 2>

[0123] In the above embodiment, the image coordinate position and identification number of each landmark are acquired using a landmark which has a specific color as identification information by extracting a specific color region from an image by a threshold value process, and detecting the barycentric position of that region as the image coordinate position of the landmark having that color. However, any other methods may be used as long as they can specify the projection coordinate position of a landmark on an image and its identification number. For example, using a landmark which has a specific pattern as identification information, a region of the specific pattern may be extracted from an image by pattern matching, and its detection position may be output as the image coordinate position of the landmark having that pattern. Also, such detection process need not always be implemented by an image process, and the operator may manually input the image coordinate position and identification number of a landmark. In this case, the image coordinate acquisition unit 320 preferably has an arbitrary GUI that allows the operator to easily input a landmark position by, e.g., designating a landmark position on a captured image displayed on a work display by clicking a mouse button.

[0124] When a plurality of landmarks which have identical features that cannot be identified by an image process are used, the image coordinate position of each landmark may be acquired by the image process, and its identification number may be manually input. The landmark may be identified by any other methods. For example, when approximate calibration information is obtained in advance, the landmark may be identified by comparing the theoretical value of the image coordinate position of each landmark calculated by equation (B) with the image coordinate position of the detected landmark.

[0125] In either of the acquisition process using the image process or manual acquisition, the landmark need not always be an artificial (man-made) one, but may use a natural feature.

[0126] <Modification 3>

[0127] In the above embodiment, an appropriate value is set as the initial value of the calibration information in step S1080. However, if the initial value is considerably different from an actual value, the solution cannot converge and calibration information cannot be calculated in the above embodiment. To cope with such situation, combinations of different positions/postures may be set in advance, the processes in steps S1090 to S1140 may be executed using these combinations in turn as the initial value, and s upon convergence of the solution may be selected. Also, an initial value input unit at which the operator inputs initial values (or information of the position and posture required to generate initial-values) may be prepared, and a calibration information calculation process may be executed using the input initial values.

[0128] <Modification 4>

[0129] In a preferred modification of the present invention, the calibration apparatus further has an image generation unit. The image generation unit superimposes the image coordinate position of the landmark detected by the image coordinate acquisition unit 320 on a captured image and outputs to the display screen. According to this modification, the operator can input a data acquisition instruction while confirming the landmark detection state.

[0130] The image generation unit calculates the theoretical value of the image coordinate position of each landmark on the basis of the calibration information calculated by the calibration information calculation unit 340, and superimposes the calculated value on the captured image and outputs to the display screen. According to this modification, the operator can verify the calibration process result by comparing an actual landmark and the calculated position displayed there.

[0131] <Modification 5>

[0132] In the above embodiment, the apparatus has the instruction unit 350 used to input a control command by an operator. However, this input is not always necessary. For example, every time the image coordinate acquisition unit 320 detects a landmark, data may be added to the data list, and when the data list meets a given condition, the calibration information calculation unit 340 may calculate calibration information.

[0133] <Modification 6>

[0134] In the above embodiment, the calibration apparatus is independent from the image display apparatus. Of course, an image display apparatus may be designed to have the function of the calibration apparatus.

[0135] <Modification 7>

[0136] In the above embodiment, the position/posture sensor of the image display apparatus that presents mixed reality is calibrated. However, the present invention is not limited to such specific application range, and may be applied to any other applications that measure the position and posture of an image sensing device using a position/posture sensor.

[0137] As described above, according to the first embodiment, calibration information required to transform the position and posture of the image sensing device measured by the sensor into those on the world coordinate system can be easily acquired without using any special calibration tool.

[0138] [Second Embodiment]

[0139] A sensor calibration apparatus of this embodiment calculates parameters required to transform the position and posture of an image sensing unit measured by a sensor into those on the world coordinate system.

[0140] For this purpose, the sensor calibration apparatus of this embodiment calculates parameters using:

[0141] (1) the world coordinate positions of four or more markers which are not located on an identical line;

[0142] (2) the image coordinate positions of markers on images captured at a plurality of positions/postures;

[0143] (3) sensor measurement values upon capturing the images of (2); and

[0144] (4) initial values (approximate values) of sensor layout information.

[0145] Of these data, (1) the world coordinate positions of the markers are data which must be prepared as known information in a preparation process of calibration. At least three points of markers (landmarks, feature points), which are known to the world coordinate positions, must be arranged in a real space as a display target of the image display apparatus. Assume that the markers have, e.g., different colors, so that the image coordinate positions of their images projected onto a captured image can be detected, and these markers can be identified from each other. The world coordinate system has a predetermined point as an origin, and specifies X-, Y-, and Z-axes in orthogonal directions from this origin. Also, assume that the coordinate positions of the three points of markers are known on this coordinate system. That is, the distances from the origin to the three points of markers in the X-, Y-, and Z-directions are measured in advance.

[0146] (2) and (3) mean data acquired upon calibration. The initial values of the sensor layout information (4) are not always required, but an appropriate solution cannot often be often if they are not set.

[0147] A theoretical value of the image coordinate position of each marker is calculated on the basis of the world coordinate position of that marker, sensor measurement value, and sensor layout information using these data as inputs, and calibration information (parameters) that can minimize the sum of errors from actually measured values is calculated.

[0148] Note that parameters may be calculated using other methods that use other kinds of information.

[0149] FIG. 5 is a schematic block diagram of the sensor calibration apparatus of this embodiment.

[0150] Referring to FIG. 5, reference numeral 5100 denotes an arithmetic processing unit which comprises a computer or the like. The arithmetic processing unit 5100 includes a CPU 5101, RAM 5102, image generation device 5103, system bus 5104, disk device 5105, input device 5106, and image capture device 5107.

[0151] The CPU 5101 controls a calibration process on the basis of a calibration program. The CPU 5101 is connected to the system bus 5104, and can communicate with the RAM 5102, image generation device 5103, disk device 5105, input device 5106, and image capture device 5107.

[0152] The RAM 5102 is implemented by a main storage device such as a memory or the like. The RAM 5102 temporarily holds a program code of the calibration program, program control information, the world and image coordinate positions of the markers, calibration information calculated by this apparatus, and the like via the system bus 5104.

[0153] The image generation device 5103 is implemented by a device such as a graphics card or the like. The image generation device 5103 outputs image information generated by a program which is executed by the CPU 5101 to a display unit 5200 via the system bus 5104.

[0154] The system bus 5104 serves as a communication path to which respective devices which form the arithmetic processing unit 5100 are connected to communicate with each other.

[0155] The disk device 5105 is implemented by an auxiliary storage device such as a hard disk or the like. The disk device 5105 holds the program code of the calibration program, the program control information, the world and image coordinate positions of the markers, the calibration information calculated by this apparatus, and the like.

[0156] The input device 5106 is implemented by various interface devices. The input device 5106 receives signals from devices externally connected to the arithmetic processing unit 5100 as data, and writes the data in the RAM 5102 via the system bus 5104.

[0157] The image capture device 5107 is implemented by a device such as a capture card or the like. The image capture device 5107 receives an image output from an image sensing device 5302, and writes image data in the RAM 5102 via the system bus 5104.

[0158] Reference numeral 5200 denotes a display unit which is implemented by a display device such as a CRT monitor, liquid crystal monitor, or the like. The display unit 5200 is used to display a video signal output from the image generation device 5103, and to present the result to the user of this apparatus.

[0159] Reference numeral 5300 denotes a head-mount unit, which is to be calibrated by this apparatus. The head-mount unit 5300 comprises a receiver 5301 and the image sensing device 5302.

[0160] The receiver 5301 is implemented by a device for measuring a magnetic field generated by a transmitter 5600 in, e.g., a magnetic sensor. The magnetic field value measured by the receiver 5301 is output to a sensor control unit 5500, which converts the measured value into parameters that represent a 3D position and posture.

[0161] The image sensing device 5302 is implemented by an image sensing device such as a CCD camera or the like. A video signal sensed by the image sensing device 5302 is sent to the image capture device 5107.

[0162] An operation input unit 5400 is implemented by input devices such as a keyboard, mouse, and the like, which are used to control the arithmetic processing unit 5100. The operation input unit 5400 outputs an operation signal to the input device 5106. The user (not shown) of this apparatus gives a control instruction of this apparatus by operating the operation input unit 5400.

[0163] The sensor control unit 5500 controls the receiver 5301 and transmitter 5500 in, e.g., a magnetic sensor, and calculates the 3D position and posture information of the receiver 5301 on the basis of information received from the receiver 5301. The 3D position and posture information calculated by the sensor control unit 5500 are sent to the input device 5106.

[0164] The transmitter 5600 generates a magnetic field to make the sensor control unit 5500 calculate the 3D position and posture of the receiver 5301 in, e.g., the magnetic sensor.

[0165] In this embodiment, the magnetic sensor is used. However, the sensor to be used is not limited to the magnetic sensor. For example, an optical sensor, ultrasonic wave sensor, or the like may be used, and the types of sensors are not particularly limited as long as they can measure the 3D position and posture. In this case, the receiver 5301 serves as an object to be measured by the sensor, and the transmitter 5600 serves as the origin of a sensor coordinate system when the sensor control unit 5500 calculates the 3D position and posture of the receiver 5301.

[0166] FIG. 6 is a block diagram showing the functional arrangement of the calibration apparatus of this embodiment. Processes of respective units in FIG. 6 are executed inside the arithmetic processing unit 5100 shown in FIG. 5.

[0167] A world coordinate holding unit 5110 holds coordinate data of the respective markers on the world coordinate system, and outputs these data in accordance with a request from a data management unit 5111. Also, the world coordinate holding unit 5110 holds (marker color information, identification information) data of unique information to each marker in association with the coordinate data of the markers.

[0168] Upon receiving a data acquisition request from an instruction unit 5115, the data management unit 5111 receives the image coordinate position and identification information of a given marker from an image coordinate acquisition unit 5112, receives the world coordinate position of the marker corresponding to the identification information from the world coordinate holding unit 5110, adds a set of the image coordinate position, world coordinate position, and identification information in a data list, and holds that list. If the data management unit 5111 receives the image coordinate position of a given marker alone from the image coordinate acquisition unit 5112 but does not receive any identification information, only the image coordinate position is added to the data list. Upon receiving a data delete request from the instruction unit 5115, the data management unit 5111 deletes data from the data list. Furthermore, upon receiving a data identification request from the instruction unit 5115, the data management unit 5111 changes data in the data list by changing the combinations of world coordinate positions, image coordinate positions, and identification information of markers. Moreover, the data management unit 5111 outputs the generated data list to a calibration information calculation unit 5113 in accordance with a request from the calibration information calculation unit 5113.

[0169] The image coordinate acquisition unit 5112 specifies the coordinate position and identification information of a marker which appears in an image (to be referred to as a real image hereinafter), which is sensed by the image sensing device 5302 (FIG. 5) and is captured by the image captured device 5107 (FIG. 5), and outputs such information to the data management unit 5111 in accordance with a request from the data management unit 5111.

[0170] Upon receiving a calibration information calculation instruction from the instruction unit 5115, the calibration information calculation unit 5113 receives the data list from the data management unit 5111, calculates calibration information based on the data list, and outputs the calculated calibration information to a calibration information holding unit 5114.

[0171] The calibration information holding unit 5114 holds the calibration information calculated by the calibration information calculation unit 5113, and outputs or changes the held calibration information, or saves the calibration information in a file in accordance with a request from the instruction unit 5115. Upon receiving an output request from the instruction unit 5115, the unit 5114 outputs the calibration information. Upon receiving a calibration information change request from the instruction unit 5115, the unit 5114 changes the value of the held calibration information. Upon receiving a file save request from the instruction unit 5115, the unit 5114 generates a file on the disk device 5105, and saves the calibration information in that file. Upon receiving a file load request from the instruction unit 5115, the unit 5114 discards the currently held calibration information, loads the designated file from the disk device 5105, and sets the loaded value as new current calibration information. Upon receiving a reset request from the instruction unit 5115, the unit 5114 discards the currently held calibration information, and sets a default value of calibration information held upon launching this apparatus as new current calibration information. Also, the calibration information holding unit 5114 outputs held calibration information to the calibration information calculation unit 5113 in accordance with a request from the calibration information calculation unit 5113. The calibration information output to the calibration information calculation unit 5113 is used as an initial value when the calibration information calculation unit 5113 calculates calibration information.

[0172] The instruction unit 5115 outputs a data acquisition request, data delete request, and data identification request to the data management unit 5111 in response to a data acquisition command, data delete command, and data identification command input from the user of this apparatus, respectively. Upon receiving a calibration information calculation command, the unit 5115 outputs a calibration information calculation request to the calibration information calculation unit 5113.

[0173] In this embodiment, the user of this apparatus gives an instruction to the instruction unit 5115 using a GUI shown in FIG. 7. The GUI shown in FIG. 7 is formed by a main window 51000, image display area 51010, calibration information display area 51020, operation buttons 51030, and file menu 51040. As shown in FIG. 8, the file menu 51040 includes a save sub-menu 51050, and quit sub-menu 51060.

[0174] On the image display area 51010, a real image is displayed. Also, a mark indicating the coordinate position of a marker specified by the image coordinate acquisition unit 5112, and its identification information are superimposed on the real image.

[0175] FIG. 9 shows a state wherein the mark which indicates the coordinate position of a marker specified by the image coordinate acquisition unit 5112, and its identification information are superimposed on the real image displayed on the image display area 51010. In FIG. 9, an ellipse indicates a marker which appears in the real image. Also, the image coordinate position of the marker specified by the image coordinate acquisition unit 5112 is indicated by a boxed mark X. Furthermore, text which indicates a marker name is superimposed in association with the identification information of the marker specified by the image coordinate acquisition unit 5112.

[0176] By superimposing the mark at the image coordinate position of the marker, the user of this apparatus can confirm the coordinate position specified by the image coordinate acquisition unit 5112 at a glance.

[0177] Also, by superimposing the identification information of the marker, the user of this apparatus can confirm, at a glance, the marker specified by the image coordinate acquisition unit 5112.

[0178] In this example, the boxed mark X is superimposed to indicate the image coordinate position of the marker. However, the mark to be superimposed is not limited to the boxed mark X. For example, various symbols such as arrows ◯, x, □, and the like or icons and the like may be used, and any other marks may be used as long as the user can visually confirm the image coordinate position of the marker specified by the image coordinate acquisition unit 5112.

[0179] Also, text indicating the name is superimposed as the identification information of the marker. However, the identification information to be superimposed is not limited to text. For example, the user may identify the marker using an icon, image, or the like, or by changing the color of text, icon, image, or the like or that of the marker itself. That is, any other types of identification information may be used as long as the user of this apparatus can confirm correspondence between the marker which appears in the real image, and the actual marker.

[0180] FIG. 10 shows a display example on the image display area 51010 when the image coordinate acquisition unit 5112 specifies the image coordinate position of the marker, but cannot specify identification information, and the data management unit 5111 does not generate a set of the world coordinate position, image coordinate position, and identification information of the marker. In the following description, determining correspondence among the world coordinate position, image coordinate position, and identification information of the marker, and generating a set of these data by the data management unit 5111 will be expressed by “identifying a marker”.

[0181] In FIG. 10, the mark X is superimposed on the real image at the image coordinate position of the marker. Also, text “?” is superimposed on the real image. When the marker has been identified, the boxed mark X and identification information of the marker are superimposed on the image display area 51010; when the marker has not been identified, the display method is switched.

[0182] The display method used depending on whether or not the marker has been identified need not be switched on the entire image display area 51010, but may be independently switched for each marker which appears on the image display area 51010.

[0183] FIG. 11 shows a state wherein the coordinate position of the marker on the world coordinate system, which is held by the world coordinate holding unit 5110, is rendered by CG from the viewpoint of the image sensing device 5302, on the basis of the current calibration information and the position/posture information of the receiver 5301, and that CG image is superimposed on the real image displayed on the image display area 51010.

[0184] If calibration is correctly done by this apparatus, the coordinate position of the marker specified by the image coordinate acquisition unit 5112 theoretically matches the world coordinate position of that marker superimposed on the real image. The user of this apparatus can visually confirm the precision of calibration done by this apparatus by checking the deviation between the image and world coordinate positions of the marker.

[0185] In this embodiment, the world coordinate position of the marker is rendered by CG. However, an object to be rendered by CG is not limited to the world coordinate position of the marker.

[0186] For example, in order to allow the user to confirm the appearance of the world coordinate system on the real image, the coordinate axes, plane, world coordinate origin, and the like, which define the world coordinate system, may be rendered.

[0187] When some functions of this calibration apparatus are implemented using an image display apparatus that presents mixed reality, a virtual world and virtual object used upon presenting mixed reality may be rendered by CG.

[0188] When the image coordinate acquisition unit 5112 executes a process for extracting a color region from an image so as to specify the marker coordinate position, the image display area 51010 may display an image indicating result of the color region extraction process in place of the real image. In this way, the user of this apparatus can easily adjust parameters for the color region extraction process.

[0189] The user of this apparatus can designate the image coordinate position in the image display area 51010 using a pointing device such as a mouse or the like. For example, the user himself or herself of this apparatus can specify the coordinate position of a marker (in the real image), which is to be done by the image coordinate acquisition unit 5112. Also, the user of this apparatus can select a desired marker which appears in the real image, can change processing parameters for that marker, and can manually input identification information of the marker.

[0190] The calibration information display area 51020 displays the current calibration information held by the calibration information holding unit 5114.

[0191] The operation buttons 51030 are used to control the behavior of this apparatus and GUI when the user of this apparatus selects each button region using a pointing device such as a mouse or the like. In this embodiment, when the user of this apparatus selects one of the operation buttons 51030, a data acquisition dialog, initial value setting dialog, or detection parameter setting dialog is newly displayed.

[0192] The file menu 51040 displays the save sub-menu 51050 and quit sub-menu 5160 when the user of this apparatus selects a menu region using a pointing device such as a mouse or the like.

[0193] The save sub-menu 51050 gives an instruction to this apparatus when the user of this apparatus selects a sub-menu region using a pointing device such as a mouse or the like. Upon selection of the save sub-menu 51050, the instruction unit 5115 issues a file save command, and outputs a file save request to the calibration information holding unit 5114.

[0194] The quit sub-menu 51060 gives an instruction to this apparatus when the user of this apparatus selects a sub-menu region using a pointing device such as a mouse or the like. Upon selection of the quit sub-menu 51060, this calibration apparatus quits. When the calibration information holding unit 5114 holds calibration information which is not saved yet, the instruction unit 5115 issues a file save command, and outputs a file save request to the calibration information holding unit 5114. The instruction unit 5115 waits until the calibration information holding unit 5114 completes a file save process, and then quits this calibration apparatus.

[0195] FIG. 12 shows an example of the data acquisition dialog. A data acquisition dialog 51100 shown in FIG. 12, includes a data list display area 51110, data acquisition operation button 51120, display image switch button 51130, calibration information calculation button 51140, selected data deactivate/activate button 51150, selected data delete button 51160, selected data display area 51170, selected data manipulation area 51180, error display area 51190, and data acquisition dialog close button 51195.

[0196] The data list display area 51110 displays thumbnails of real images corresponding to the data list which has been acquired so far. When the user of this apparatus selects one of these thumbnails, data corresponding to that thumbnail is selected from the data list, and is displayed on the selected data display area 51170. Also, the data corresponding to that thumbnail is activated. The active data is to be processed upon depression of the selected data deactivate/activate button 51150 or selected data delete button 51160.

[0197] At this time, the image display area 51010 displays a real image corresponding to the data list. Since the data list display area 51110 displays thumbnails of real images, and it is difficult for the user of this apparatus to confirm details from the thumbnail, the image display area 51010 displays the real image without reducing. Also, the image coordinate position, world coordinate position, and identification information of a marker selected from the data list are superimposed on the real image by the method described in the paragraphs of the image display area 51010.

[0198] The data acquisition button 51120 gives an instruction to this apparatus when the user of this apparatus selects a button region using a pointing device such as a mouse or the like. Upon depression of the data acquisition button 51120, the instruction unit 5115 issues a data acquisition command, and outputs a data acquisition request to the data management unit 5111. Also, a real image at that time is additionally displayed on the data list display area.

[0199] The display image switch button 51130 gives an instruction to this apparatus when the user of this apparatus selects a button region using a pointing device such as a mouse or the like. Upon depression of the display image switch button 51130, a mode of an image to be displayed on the image display area 51010 is switched. The user of this apparatus can select one of a “live video display mode” for displaying the currently acquired real image, and an “acquired data display mode” for displaying a real image currently selected on the data list display area 51110. Upon depression of the display image switch button 51130, if the current mode is the live video display mode, it is switched to the acquisition data display mode, and vice versa.

[0200] The calibration information calculation button 51140 gives an instruction to this apparatus when the user of this apparatus selects a button region using a pointing device such as a mouse or the like. Upon depression of the calibration information calculation button 51140, the instruction unit 5115 issues a calibration information calculation command, and outputs a calibration information calculation request to the calibration information calculation unit 5113.

[0201] The selected data deactivate/activate button 51150 gives an instruction to this apparatus when the user of this apparatus selects a button region using a pointing device such as a mouse or the like. Upon depression of the selected data deactivate/activate button 51150, when data (captured image) currently selected on the data list display area 51110 is active, that data is deactivated and is excluded from an object of the calibration information calculation process. If the currently selected data is inactive, that data is activated.

[0202] The selected data delete button 51160 gives an instruction to this apparatus when the user of this apparatus selects a button region using a pointing device such as a mouse or the like. Upon depression of the selected data delete button 51160, the instruction unit 5115 issues a data delete command, and outputs a data delete request to the data management unit 5111. Also, the thumbnail of the currently selected real image is erased from the data list display area 51110.

[0203] The selected data display area 51170 displays data selected by the user of this apparatus from the data list display area 51110.

[0204] The selected data manipulation area 51180 is used when the user of this apparatus manipulates the selected data. FIG. 13 shows details of the selected data manipulation area 51180.

[0205] The selected data manipulation area 51180 includes a marker select button 51181, marker add button 51182, marker identification information select menu 51183, and marker delete button 51184.

[0206] The marker select button 51181 gives an instruction to this apparatus when the user of this apparatus selects a button region using a pointing device such as a mouse or the like. Upon depression of the marker select button 51181, a manipulation mode is set in a “marker select mode”. When the user of this apparatus selects the vicinity of the mark which indicates the coordinate position of the marker displayed on the image display area 51010 while the marker select mode is set, that marker is activated.

[0207] The marker add button 51182 gives an instruction to this apparatus when the user of this apparatus selects a button region using a pointing device such as a mouse or the like. Upon depression of the marker add button 51182, the manipulation mode is set in a “marker add mode”. When the user of this apparatus selects an arbitrary portion of the image display area 51010 while the marker add mode is set, a new marker having that image coordinate position as the marker coordinate position is set, and is added to the data list. Furthermore, the added marker is activated.

[0208] The marker identification information select menu 51183 gives an instruction to this apparatus when the user of this apparatus selects a menu region using a pointing device such as a mouse or the like. Upon selection of the marker identification information select menu 51183, a list of marker identification information saved in the data list is displayed. In this embodiment, a character string indicating the marker name is used as the marker identification information. The user of this apparatus selects one marker identification information from the displayed list using a pointing device such as a mouse or the like. Upon selection of the marker identification information, the instruction unit 5115 issues a data identification command, and outputs the selected identification information and active marker information to the data management unit 5111 together with a data identification request.

[0209] The marker delete button 51184 gives an instruction to this apparatus when the user of this apparatus selects a button region using a pointing device such as a mouse or the like. Upon depression of the marker delete button 51184, the currently selected marker is deleted from the data list.

[0210] The error display area 51190 displays calibration errors obtained when the calibration information calculation unit 5113 calculates calibration information. In this embodiment, two different types of calibration errors, i.e., errors with respect to respective markers in the data list, and the average of errors of all data are displayed.

[0211] When calibration information calculated by the calibration information calculation unit 5113 has insufficient precision, or when the solution does not converge upon calculation of calibration information, the user of this apparatus can easily specify a marker as that cause by confirming calibration errors with respect to respective markers.

[0212] By confirming the average of errors of all data, the activation/deactivation results of data used in calculation of calibration information using the selected data deactivate/activate button can be evaluated.

[0213] The data acquisition dialog close button 51195 gives an instruction to this apparatus when the user of this apparatus selects a button region using a pointing device such as a mouse or the like. Upon selection of the data acquisition dialog close button 51195, the data acquisition dialog is closed. Even when the data acquisition dialog is closed, the contents of the data list are held.

[0214] FIG. 14 shows an example of an initial value setting dialog 51200. The initial value setting dialog includes parameter type select tabs 51210, a parameter setting area 51220, standard configuration designation button 51230, transmitter observation button 51240, reset button 51250, and initial value setting dialog close button 51260.

[0215] The parameter type select tabs 51210 are used to select the type of calibration information for which an initial value is to be set when the user of this apparatus selects one of regions using a pointing device such as a mouse or the like. There are two regions “Local Transform” and “World Transform”. Upon selection of the “Local Transform” region, a calibration information mode is set in an “image sensing device—measurement point calibration information mode” that calculates first parameters required to transform the measurement value of the sensor into the position and posture of the image sensing unit. Upon selection of the “World Transform” region, the calibration information mode is set in a “world coordinate system—sensor coordinate system calibration information mode”.

[0216] On the parameter setting area 51220, the user of this apparatus can set calibration information parameters corresponding to the current calibration information mode by various means such as buttons, sliders, spin buttons, numerical value input from a keyboard, and the like. In this embodiment, calibration information parameters are set by using one or a plurality of the following methods solely or in combination.

[0217] First, a vector that defines the position and rotation axis, and a rotation angle about that axis are designated.

[0218] Second, the position and Euler angles are designated.

[0219] Third, rotation angles about the X-, Y-, and Z-axes of the receiver 5301 viewed from the image sensing device 5302, and those about the X-, Y-, and Z-axes of the image sensing device 5302 viewed from the receiver 5301 are designated.

[0220] Upon completion of the setups of the calibration information parameters, the instruction unit 5115 issues a calibration information change command, and outputs a calibration information change request to the calibration information holding unit 5114.

[0221] The standard configuration designation button 51230 gives an instruction to this apparatus when the user of this apparatus selects a button region using a pointing device such as a mouse or the like. Upon designation of the standard configuration designation button 51230, the instruction unit 5115 issues a calibration information load command, and outputs a calibration information load request to the calibration information holding unit 5114.

[0222] The transmitter observation button 51240 gives an instruction to this apparatus when the user of this apparatus selects a button region using a pointing device such as a mouse or the like.

[0223] Upon selection of the transmitter observation button 51240, approximate values of Local Transform can be automatically calculated using the sensor measured value.

[0224] After the user adjusts the position and posture of a camera so that the image of the transmitter of the sensor is captured at nearly the center of the captured image, as shown in FIG. 18A, he or she clicks the transmitter observation button 51240. Based on the sensor measured value at that time, approximate values of Local Transform are calculated.

[0225] Upon clicking the transmitter observation button 51240, the instruction unit 5115 issues a calibration information calculation command, and outputs a calibration information calculation request to the calibration information calculation unit 5113. At this time, the calibration information calculation unit 5113 calculates calibration information (Local Transform) between the image sensing device 5302 and receiver 5301 using the only current sensor measured value without using the world coordinate position, image coordinate position, and identification information in the data list managed by the data management unit 5111.

[0226] Theoretical values of the position and posture of the transmitter are calculated using the calculated approximate values, and a virtual image of the transmitter is generated based on the theoretical values. In addition, the virtual image of the transmitter is superimposed on the captured image at an image position corresponding to the theoretical values (FIG. 18B).

[0227] In Local Transform set using the transmitter observation button 51240, a rotation angle of the image sensing device in the Z-direction is indetermination. Hence, this parameter (a Z-posture component of Local Transform) is roughly adjusted using an azimuth (z-axis) slider bar (parameter setting area 51220). The virtual image of the transmitter is updated in real time in accordance with the adjustment result (FIG. 18C). Note that other parameters can be adjusted.

[0228] The reset button 51250 gives an instruction to this apparatus when the user of this apparatus selects a button region using a pointing device such as a mouse or the like. Upon designation of the reset button 51250, the instruction unit 5115 issues a reset command, and outputs a reset request to the calibration information holding unit 5114.

[0229] The initial value setting dialog close button 51260 gives an instruction to this apparatus when the user of this apparatus selects a button region using a pointing device such as a mouse or the like. Upon selection of the initial value setting dialog close button 51260, the initial value setting dialog is closed. Even after the initial value setting dialog is closed, calibration information can be held.

[0230] In this manner, by selecting the transmitter observation button 51240, approximate values of Local Transform can be easily and automatically calculated using the sensor measured value without requiring any special preparation processes, using a transmitter. Since the transmitter is an indispensable component in this system, the user need not execute any special processes for this process.

[0231] FIGS. 15 and 16 show an example of a marker detection parameter setting dialog (marker detection parameters dialog) 51300. The marker detection parameter setting dialog 51300 includes parameter type select s 51310, a marker detection parameter setting area 51320, marker detection result display button 51330, marker detection parameter reset button 51340, marker detection disable button 51350, marker identification means switch button 51360, marker identification parameter setting area 51370, marker identification parameter reset button 51380, and marker detection parameter setting dialog close button 51390.

[0232] The parameter type select s 51310 are used to select and display one of a marker detection parameter setting dialog and marker identification parameter setting dialog, when the user of this apparatus selects one of regions using a pointing device such as a mouse or the like.

[0233] The marker detection parameter setting area 51320 is displayed when the marker detection parameter setting dialog is selected by the parameter type select 51310. On the marker detection parameter setting area 51320, the user of this apparatus sets threshold values of colors used to detect markers, and a minimum value of the number of pixels of a color region to be recognized as a marker using sliders, numerical value input boxes, and the like. The threshold values of colors are used to determine an arbitrary pixel as a marker candidate when color component values of that pixel exceed the threshold values.

[0234] In this embodiment, the threshold values are set for R, G, and B which indicate the luminance values of red, green, and blue. Alternatively, threshold values may be set for Y, Cr, and Cb components of a pixel. The minimum value of the number of pixels of a color region is used to inhibit a given color region from being recognized as marker, if the number of pixels of that color region is smaller than the set minimum value. In this embodiment, only the minimum value of the number of pixels of a color region is set. Also, a maximum value may be set at the same time, and the number of pixels which are recognized as marker candidates may be set to fall within a given range.

[0235] The marker detection result display button 51330 is displayed when the marker detection parameter setting dialog is selected by the parameter type select 51310. The marker detection result display button 51330 gives an instruction to this apparatus when the user of this apparatus selects a button region using a pointing device such as a mouse or the like. Upon designation of the marker detection result display button 51330, one of a real image display mode and color extraction result image display mode is selected. In the real image display mode, a real image is displayed on the image display area 51010. In the color extraction result image display mode, a color extraction result image is displayed on the image display area 51010. In a color extraction result image shown in FIG. 17, only color regions extracted from the real image are displayed.

[0236] The marker detection parameter reset button 51340 is displayed when the marker detection parameter setting dialog is selected by the parameter type select 51310. The marker detection parameter reset button 51340 gives an instruction to this apparatus when the user of this apparatus selects a button region using a pointing device such as a mouse or the like. Upon designation of the marker detection parameter reset button 51340, the currently held marker detection parameters are discarded, and default values of marker detection parameters, which are set upon launching this apparatus, are set as current marker detection parameters.

[0237] The marker detection disable button 51350 is displayed when the marker detection parameter setting dialog is selected by the parameter type select 51310. The marker detection disable button 51350 gives an instruction to this apparatus when the user of this apparatus selects a button region using a pointing device such as a mouse or the like. Upon selection of the marker detection disable button 51350, a marker detection process is disabled.

[0238] The marker identification means switch button 51360 is displayed when the marker identification parameter setting dialog is selected by the parameter type select 51310. The marker identification means switch button 51360 gives an instruction to this apparatus when the user of this apparatus selects a button region using a pointing device such as a mouse or the like. Upon selection of the marker identification means switch button 51360, a marker identification function using the sensor measured value is selectively activated/deactivated. The marker identification function calculates the distance between the image coordinate position of the detected marker, and a coordinate position obtained by projecting the world coordinate position of the marker held by the world coordinate holding unit 5110 onto the image sensing surface of the image sensing device 5302, on the basis of the current sensor measured value and calibration information, and automatically identifies the marker when the minimum value of the distance is equal to or smaller than a maximum allowable value.

[0239] The marker identification parameter setting area 51370 is displayed when the marker identification parameter setting dialog is selected by the parameter type select 51310. On the marker identification parameter setting area 51370, the user of this apparatus sets a maximum allowable distance used to identify a maker using a slider, numerical value input box, and the like.

[0240] The marker identification parameter reset button 51380 is displayed when the marker identification parameter setting dialog is selected by the parameter type select 51310. The marker identification parameter reset button 51380 gives an instruction to this apparatus when the user of this apparatus selects a button region using a pointing device such as a mouse or the like. Upon designation of the marker identification parameter reset button 51380, the currently held marker identification parameters are discarded, and default values of marker identification parameters, which are set upon launching this apparatus, are set as current marker identification parameters.

[0241] The marker detection parameter setting dialog close button 51390 gives an instruction to this apparatus when the user of this apparatus selects a button region using a pointing device such as a mouse or the like. Upon selection of the marker detection parameter setting dialog close button 51390, the detection parameter setting dialog 51390 is closed. Even after the detection parameter setting dialog is closed, marker detection parameters and marker identification parameters are held.

[0242] FIG. 19 is a flow chart showing processes to be executed by the sensor calibration apparatus of this embodiment. Note that a program code that implements processes according to this flow chart is stored in a storage device such as the disk device 5105, RAM 5102, or the like in the apparatus of this embodiment, and is read out and executed by the CPU 5101.

[0243] In step S5000, the sensor calibration apparatus of this embodiment is launched.

[0244] The instruction unit 5115 checks in step S5100 if the user of this apparatus has input a data acquisition command. If the data acquisition command has been input, the flow advances to step S5110.

[0245] In step S5110, the data management unit 5111 receives the measured values of the current position and posture of the receiver 5301 from the sensor control unit 5500.

[0246] In step S5120, the data management unit 5111 receives the identification information and image coordinate position of a marker which appears on a captured image sensed by the image sensing device 5302 via the image coordinate acquisition unit 5112. If a plurality of markers appear on the captured image, this step is repeated for these markers.

[0247] In step S5130, the data management unit 5111 adds input data to the data list for each detected marker.

[0248] The instruction unit 5115 checks in step 5200 if the user of this apparatus has made an edit operation of the data list. If the edit operation of the data list has been made, the flow advances to step S5210.

[0249] In step S5210, the instruction unit 5115 issues a command corresponding to the edit operation of the data list made by the user of this apparatus, thereby editing the data list. For example, the data list edit operation includes an operation for selecting an element of the data list, an operation for deleting the selected element from the data list, an operation for adding a new marker to the data list, an operation for deleting an existing marker from the data list, and an operation for giving identification information to the existing marker in the data list to identify that marker.

[0250] It is checked in step S5300 if the data list acquired so far by the calibration information calculation unit 5113 has information that suffices to calculate calibration information. If the data list does not meet a given condition, the flow returns to step S5100 to wait for input of the next data acquisition command. On the other hand, if the data list meets a calibration information calculation condition, the flow advances to step S5400. The calibration information calculation condition may include, for example, that data associated with three or more points of different markers be acquired, data be acquired at a plurality of viewpoint positions, and the total number of data be 6 or more. However, since the precision of calculated calibration information improves with increasing variety of input data, a condition that requires more data may be set.

[0251] It is checked in step S5400 if the user of this apparatus has input a calibration information calculation command. If the calibration information calculation command has been input, the flow advances to step S5410; otherwise, the flow returns to step S5100 to wait for input of the next data acquisition command.

[0252] In step S5410, the calibration information calculation unit 5113 executes a calibration information calculation process.

[0253] The instruction unit 5115 checks in step S5500 if the user of this apparatus has made an edit operation of calibration information. If the edit operation of calibration information has been made, the flow advances to step S5510.

[0254] In step S5510, the instruction unit 5115 issues a command corresponding to the calibration information edit operation made by the user of this apparatus, thus editing calibration information. For example, the calibration information edit operation includes an operation for changing parameters of calibration information, an operation for loading calibration information, and an operation for resetting calibration information.

[0255] The instruction unit 5115 checks in step S5600 if the user of this apparatus has made a save operation of calibration information. If the save operation of calibration information has been made, the flow advances to step S5610.

[0256] In step S5610, the instruction unit 5115 outputs a file save request to the calibration information holding unit 5114 to output the currently held calibration information to the disk device 5105.

[0257] The instruction unit 5115 checks in step S5700 if the user of this apparatus has made a quit operation of this apparatus. If the quit operation of this apparatus has been made, the flow advances to step S5800; otherwise, the flow returns to step S5100.

[0258] In step S5800, a process for quitting this apparatus is executed.

[0259] FIG. 20 is a flow chart of standard operations made when the user of the sensor calibration apparatus of this embodiment calibrates the sensor using this apparatus.

[0260] In step S51000, the user of this apparatus launches the sensor calibration apparatus of this embodiment.

[0261] In step S51100, the user of this apparatus senses an image of a given marker using the image sensing device 5302. The user of this apparatus confirms whether or not the image coordinate acquisition unit 5112 of this apparatus correctly specifies the image coordinate position and identification information of the marker. If the image coordinate position and identification information can be correctly specified, the flow advances to step S51200; otherwise, the flow advances to step S51110.

[0262] In step S51110, the user of this apparatus gives instructions for adjusting marker detection parameters and marker identification parameters to the instruction unit 5115 using, e.g., the marker detection parameter setting dialog 51300.

[0263] In step S51200, the user of this apparatus makes the instruction unit 5115 issue a data acquisition command using, e.g., the data acquisition button 51120.

[0264] The user of this apparatus confirms in step S51300 whether or not respective data of the acquired data list are satisfactory. If the data are satisfactory, the flow advances to step S51400; otherwise, the flow advances to step S51310 or returns to step S51100 depending on circumstances.

[0265] In step S51310, the user of this apparatus gives an instruction for a data list edit operation to the instruction unit 5115 using, e.g., the data acquisition dialog 51100.

[0266] In step S51400, the user of this apparatus makes the instruction unit 5115 issue a calibration information calculation command using, e.g., the calibration information calculation button 51140.

[0267] The user of this apparatus confirms in step S51500 whether or not the calibration information calculation unit 5113 correctly calculates calibration information. If the calibration information is calculated correctly, the flow advances to step S51600; otherwise, the flow returns to step S51100.

[0268] The user of this apparatus confirms in step S51600 whether or not the calibration information calculated by the calibration information calculation unit 5113 is satisfactory. If the calibration information is satisfactory, the flow advances to step S51700; otherwise, the flow advances to step S51610 or returns to step S51100 depending on circumstances.

[0269] In step S51610, the user of this apparatus gives an instruction for editing calibration information to the instruction unit 5115 using, e.g., the initial value setting dialog 51200.

[0270] In step S51700, the user of this apparatus makes the instruction unit 5115 issue a file save command using, e.g., the save sub-menu 51050.

[0271] Instep S51800, the user of this embodiment makes this apparatus quit using, e.g., the quit sub-menu 51060.

[0272] [Third Embodiment]

[0273] FIG. 21 is a schematic block diagram showing the arrangement of a sensor calibration apparatus according to the third embodiment. In the second embodiment, the display unit 5200 is arranged outside the head-mount unit 5300. In the third embodiment, the display unit 5200 is included in the head-mount unit 5300. Such arrangement can be implemented when a display device such as an HMD or the like that the user can wear on the head is used as the display unit 5200. In the third embodiment, since the head-wearable display device is used, the user of this apparatus can calibrate under the same condition as that upon using an image display apparatus which presents mixed reality.

[0274] [Fourth Embodiment]

[0275] FIG. 22 is a schematic block diagram showing the arrangement of a sensor calibration apparatus according to the fourth embodiment. In the fourth embodiment, the display unit 5200 comprises both a display device which is arranged outside the head-mount unit 5300 in the second embodiment, and a display device that the user of this apparatus wears on the head in the third embodiment. For example, an HMD normally has lower resolution than a normal display device such as a CRT monitor, liquid crystal monitor, or the like. When such display device is used, the user of this apparatus can hardly gives instructions to this apparatus using the GUI that has been explained in the second embodiment. In such case, for example, this apparatus may be used by two users.

[0276] That is, the display device which is observed by the first user of this apparatus and is arranged outside the head-mount unit 5300 (the display unit in the second embodiment) displays the GUI that has been explained in the second embodiment. The display device that the second user of this apparatus wears on the head (the display unit in the third embodiment) displays a real image, a composite image formed by superposing various kinds of information and CG on the real image, marker color extraction images, and the like except for the GUI that has been explained in the second embodiment. Since the first user operates this apparatus, and the second user adjusts the positions and postures of the image sensing device 5302 and receiver 5301, a calibration work can be efficiently done.

[0277] [Another Embodiment]

[0278] It goes without saying that the objects of the present invention are also achieved by supplying a storage medium (or recording medium), which records a program code of a software program that can implement the functions of the above-mentioned embodiments to the system or apparatus, and reading out and executing the program code stored in the storage medium by a computer (or a CPU or MPU) of the system or apparatus. In this case, the program code itself read out from the storage medium implements the functions of the above-mentioned embodiments, and the storage medium which stores the program code constitutes the present invention. It goes without saying that the functions of the above-mentioned embodiments may be implemented not only by executing the readout program code by the computer but also by some or all of actual processing operations executed by an operating system (OS) running on the computer on the basis of an instruction of the program code.

[0279] Furthermore, it goes without saying that a case is included within the scope of the invention where after a program code, read out from storage medium, has been stored in memory provided on a function expansion board of a computer or a function expansion unit connected to a computer, the CPU or the like on the feature expansion board or unit executes some or all of the actual processing based on the designation of the program code to implement the embodiments.

[0280] When the present invention is applied to the storage medium, that storage medium stores the program codes corresponding to the aforementioned flow charts (shown in FIG. 4 and/or FIG. 15).

[0281] As described above, according to the present invention, parameters (calibration information) required to transform the position and posture of an image sensing unit measured by a sensor into those on the world coordinate system can be easily and accurately acquired without using any special calibration tool.

[0282] Especially, since the user can select a captured image used upon calculating parameters from a plurality of captured images, he or she can exclude an inappropriately captured image upon calculating parameters.

[0283] According to the invention of claim 14, since errors are displayed for respective detected markers, an inappropriately captured image can be appropriately selected.

[0284] As many apparently widely different embodiments of the present invention can be made without departing from the spirit and scope thereof, it is to be understood that the invention is not limited to the specific embodiments thereof except as defined in the appended claims.