Image processing method and image processing device for separating the background area of an image

[0001] The present invention contains subject matter related to Japanese Patent Application JP 2004-294241 filed in the Japanese Patent Office on Oct. 6, 2004, the entire contents of which being incorporated herein by reference.

[0002] The present invention relates to an image processing method and an image processing device which are capable of separating a background area of an image.

[0003] In recent years, a visual communication apparatus with a camera has begun to come into wide use. For example, a system for a television conference which a plurality of persons join is known in the field of business, while a communication system which utilizes a Web camera is known in the personal field. In such systems, a display image of the other party is displayed on a display device, and an image of a user who visually observes the screen of the display device is captured as an image of a photographing object. The resulting image signal is transmitted to an image processing device on the other party side through a network including a public line, a dedicated line or the like, whereby it becomes possible to make both the users have the realistic sensations.

[0004] However, a problem common to those systems is such that information on an area other than a person is desired not to be transmitted to the other party. The present system with which all the information on an image captured with a camera is transmitted to the other party involves such a limitation that the system must be installed in a place an image of which may be shown to the other party.

[0005] Then, a method is expected in which an image part of a person (foreground) and an image part of a background are separated from each other, the image part of the background is subjected to some processing or other, and so forth.

[0006] That is, a foreground area containing an image of moving objects such as a user is cut down from a moving image which is captured in a camera system, and the cut-down image of the moving objects is subjected to predetermined processing, whereby robust image processing becomes possible which is hardly influenced by the feature of the background. In addition, when information on the background is desired not to be sent to the other party, the foreground and the background are separated from each other and only information on the foreground is delivered, thereby realizing the robust image processing.

[0007] Here, as regards a method of separating a background and a foreground from each other, for example, there is known a method in which a person area is specified by using a heat sensor, and a background area is subjected to predetermined image processing (see Patent document 1 for example), a method in which a movement area is detected based on a background difference, an interframe difference, and a stereo method to be made a foreground (see Patent document 2 for example), a method in which a distance region is specified and information on an image falling within the distance region is outputted (see Patent document 3 for example), or the like.

[0008] Patent document 1:

[0009] Japanese Patent Laid-open No. Hei 9-200714

[0010] Patent document 2:

[0011] Japanese Patent Laid-open No. 2000-20722

[0012] Patent document 3:

[0013] Japanese Patent Laid-open No. 2000-200359

[0014] However, in case of the method disclosed in Patent document 1 in which the person area is specified by using the heat sensor and the background area is subjected to the predetermined image processing, the heat sensor is generally expensive and a sensor, other than a camera, required for the visual communication apparatus must be provided. In addition, in case of the method disclosed in Patent document 2 in which the movement region is determined based on the background difference, the interframe difference, and the stereo method to be made the foreground, it-is necessary to acquire the image having no person image taken therein in advance due to adoption of the background difference. Thus, this method involves a problem in terms of convenience. Moreover, in case of the method disclosed in Patent document. 3 in which the distance range is specified and the information on the image falling within that range is outputted, when the distance is measured by utilizing the stereo method, if many repetitive patterns or occlusions are contained in the input scene, it is difficult to perform the distance measurement with high accuracy. Furthermore, as regards other techniques, there are known a method of dividing an area of a captured image, a method of recognizing a person, and the like. However, those methods are actually yet established in terms of a robust property and accuracy.

[0015] In the light of the above-mentioned problems associated with the prior art, the present invention therefor is desired to provide an image processing method and an image processing device which are capable of combining a stereo method and a method of estimating a background with each other by using images captured with a plurality of cameras in order to separate a background area of an image.

[0016] Other objects of the present invention and concrete advantages obtained by the present invention will be made clearer from the preferred embodiments which will be described below.

[0017] According to an embodiment of the present invention, there is provided an image processing method including: a capturing step of capturing images of a subject from different points of sight with at least two cameras to obtain input images synchronous with one another; a background area estimating step of estimating a background area based on a degree of similarity after global image superimposition is performed on the assumption that the background area is in the distance and has a flat surface with respect to each of the input images obtained in the capturing step; a distance image generating step of performing distance measurement by utilizing a stereo method to generate distance images with respect to points other than those in the background area estimated in the background area estimating step; and a background separated image generating step of removing the image of the area in a predetermined distance or more from each of the input images by referring to the distance images generated in the distance image generating step to generate a background separated image.

[0018] Preferably, in the image processing method according to the embodiment of the present invention, in the background area estimating step, an amount of shift among the input images such that a luminance error of the overall image becomes the minimum among the input images, the amount of global shift not exceeding a threshold determined depending on a separation boundary based on which the background area is separated is calculated, thereby performing global image superimposition.

[0019] Preferably, in the background area estimating step, the estimation is performed with an area having the degree of local similarity higher than a predetermined threshold as the background area.

[0020] In addition, preferably, in the image processing method according to the embodiment of the present invention, for example, in the background area estimating step, a background template used to estimate the background area based on the degree of local similarity is updated/added whenever necessary.

[0021] According to another embodiment of the present invention, there is provided an image processing device including: storage means for storing therein data on input images synchronous with one another, the input images being obtained by photographing a subject from different points of sight with at least two cameras; background area estimation processing unit for estimating a background area based on a degree of local similarity after global image superimposition is performed on the assumption that the background area is in the distance and has a flat surface with respect to each of the input images synchronous with one another, the data on the input images being stored in the storage means; distance image generation processing unit for performing distance measurement by utilizing a stereo method with respect to points other than those in the background area estimated in the background area estimation processing unit to generate distance images; and background separated image generation processing unit for removing the image of the area in a predetermined distance or more from each of the input images by referring to the distance images generated in the distance image generation processing unit to generate a background separated image.

[0022] Preferably, the background area estimation processing unit calculates an amount of shift among the input images such that a luminance error of the overall image becomes the minimum among the input images, the amount of global shift not exceeding a threshold determined depending on a separation boundary based on which the background is separated, thereby performing global image superimposition.

[0023] In addition, preferably, the background area estimation processing unit estimates an area having the degree of local similarity higher than a predetermined threshold as the background area.

[0024] Also, preferably, in the image processing device according to the another embodiment of the present invention, for example, the background area estimation processing unit, whenever necessary, updates/adds a background template used to estimate the background area based on the degree of local similarity.

[0025] According to the present invention, performing the global and local background estimation makes it possible to more accurately generate the distance images. Thus, the foreground and the background can be separated from each other. The superimposition of the overall image is used and the rough background area is estimated, whereby the robust results can be obtained for a scene as well having many repetitive patterns and occlusions. In addition, the background template stacks are updated/added whenever necessary, which becomes robust against an environment change due to a lapse of time.

[0041] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. Incidentally, it is to be understood that the present invention is not intended to be limited to the following embodiments, and thus the various changes will arbitrarily occur to those skilled in the art without departing from the gist of the present invention.

[0042] The present invention, for example, is applied to a visual communication apparatus 10 having a configuration as shown in FIG. 1.

[0043] The visual communication apparatus 10 has a function of estimating a background area by using a plurality of images captured with a plurality of cameras and of separating a background and a foreground from each other by utilizing distance measurement based on a stereo method. The visual communication apparatus 10 includes a reference camera 1, a detecting camera 2, a frame memory 3, a background area estimation processing unit 4, a distance image generation processing unit 5, and a background separation image generation processing unit 6.

[0044] It is supposed that the calibration is previously performed for a plurality of cameras used in the visual communication apparatus 10. While two or more cameras must be used, in this embodiment, the following description will be given with respect to a case where the two cameras (the reference camera 1 and the detecting camera 2) are used. The calibration is such that a point m for the reference camera 1 and a point m′, corresponding to the point m, for the detecting camera 2 are obtained, and a constraint condition between the cameras for distance measurement is calculated.

[0045] In the visual communication apparatus 10, data on color images which are simultaneously captured with the reference camera 1 and the detecting camera 2 is accumulated in the frame memory 3 and the background area estimation processing unit 4 estimates a background area. Subsequently, the distance image generation processing unit 5 performs the distance measurement by utilizing the stereo method, which will be described later, with respect to points other than those in the background area estimated by the background area estimation processing unit 4. The background separated image generation processing unit 6 determines the final background area, performs the processing if necessary, and outputs data on a background separated image.

[0046] Here, the general camera calibration and distance measuring method will be described.

[0047] In this case, the point m on the image and a point M in the space are assigned m=[u, v]^T and M=[x, y, z]^T, respectively, and descriptions of those points in a homogeneous coordinate system are assigned m^˜=[u, v, 1]^T and M^˜=[x, y, z, 1]^T, respectively. A relationship between the point M in the space and the point m on the image is given by Expression 1:

[0048] $\begin{array}{cc}\begin{array}{c}{\mathrm{sm}}^{\sim }=A·\left[\mathrm{Rt}\right]·M^{\sim }\\ =P·M^{\sim }\end{array}& \left(\mathrm{Expression}1\right)\end{array}$

[0049] where s is a scale factor, a matrix [R t] is called an external parameter and represents a position of a camera within the space, and R and t represent a rotation matrix and a translation matrix, respectively. A matrix A is called an internal parameter of the camera and given by Expression 2:

[0050] $\begin{array}{cc}A=\left[\begin{array}{ccc}\alpha & \gamma & u0\\ 0& \beta & v0\\ 0& 0& 1\end{array}\right]& \left(\mathrm{Expression}2\right)\end{array}$

[0051] where (u0, v0) represents an image center, α=−f/ku and β=−f·kv/sin θ represent a scale factor of a u-axis and a scale factor of a v-axis, respectively, and γ=f·ku·cos θ represents twisting of the two axes. In addition, a matrix P=A·[R t] is a projection matrix of 3×4 with which a point within the space is projected on an image surface. The above-mentioned description has shown transformation under the ideal condition without taking an influence of the lens distortion into consideration. However, since the influence of the distortion cannot be actually disregarded, the lens distortion is added as the camera parameter. A relationship between a point m_d=[ud, vd]^T and a point m=[u, v]^T in a distortion coordinate system can be expressed by Expression 3:
u=ud+(ud−cu)·k·rd²
v=vd+(vd−cv)·k·rd²   (Expression 3)

[0052] where rd² is expressed by ((ud−cu)²+(vd−cv)²sv²), (cu, cv) represents a distortion center, and sv represents an aspect ratio.

[0053] Subsequently, a distance measuring method based on the stereo method will be described.

[0054] As shown in FIG. 2, projection matrices on the reference camera 1 and the detection camera 2 are assigned P and P′, respectively, and points on the images captured with the reference camera 1 and the detection camera 2 are assigned m and m′, respectively. But, it is supposed that an influence of the distortion is removed from those points on the respective images as shown in Expression 3. When a distance to a point M is measured, it is necessary to detect the point m′, on the detected image, corresponding to the point m on the reference image. Since the point m′, as shown in FIG. 2, is located on a straight line 1 called an epipola line, in order to detect the point, on the detected image, corresponding to the point m, a search on this line must be performed. The epipola line is a set of points which are obtained by projecting points on a straight line between the camera center and the point m on the detecting camera 2. For example, when the measurement range in a Z-axis direction within the space is assigned (Z0−Zn), the epipola line becomes a straight line between points m₀′ and m_n′ which are obtained by projecting points M₀ and M_n corresponding to the respective distances. In the actual search, a point located in a distance Zi is projected on the detected image, the degree of similarity with a point on the reference image is measured, and the correspondence between the points between the images is determined.

[0055] A procedure for projecting a point Mi, in a distance Zi, which is projected on the point m on the reference camera 1 on a point m_i′ on the detected image will be shown as follows.

[0056] A point on a line of sight passing through the point m can be expressed from Expression 1 by Expression 4:
M^˜=sP⁺·m^˜+P^⊥  (Expression 4)

[0057] where P⁺ is a pseudo inverse matrix. The pseudo inverse matrix P⁺ is defined by Expression 5:
P⁺=P^T(PP^T)⁻¹   (Expression 5)

[0058] In addition, since P^⊥ becomes a vector fulfilling Expression 6 and is always projected on the origin, it represents actually the optical center and can be calculated from Expression 6.
P·P^⊥=0
P^⊥=(I−PP⁺)ω  (Expression 6)

[0059] where ω is an arbitrary four-dimensional vector.

[0060] Expression 4 represents all the points which pass through the optical center and the point m on the reference image. However, setting the distance as Zi makes it possible to determine the scale factor and determine the point Mi within the space. When this point is projected by using the projection matrix P′, the point m_i′ on the detected image can be calculated.

[0061] From the foregoing, in order to obtain the point within the space, the camera parameters A, R, t, A′, R′, and t′ of the respective cameras a and 2 must be individually obtained, or the projection matrices P and P′ must be directly calculated and the distortion parameters k1, cu1, cv1, sv1, k2, cu2, cv2, and sv2 of the respective cameras 1 and 2 must be further calculated. Thus, calculating those parameters becomes the camera calibration.

[0062] FIG. 3 shows a flow of processing for separating a background and a foreground from each other in the visual communication apparatus 10.

[0063] That is, in the visual communication apparatus 10, firstly, a subject is photographed from the different points of sight with the reference camera 1 and the detecting camera 2 to acquire input color images which are synchronous with each other (Step S1). Data on the acquired input images is stored in the frame memory 3.

[0064] Subsequently, the background area estimation processing unit 4 removes the influence of the lens distortion from each of the input images the data on which is stored in the frame memory 3 by using the above-mentioned calibration parameters, and performs the image normalization (Step S2).

[0065] That is, since each of the input images is influenced by the lens distortion, the influence of the lens distortion is removed therefrom by using Expression 3. In addition, the image normalization is such that as shown in FIG. 4, images I_L and I_R which are obtained by photographing a subject from different points c1 and c2 of sight with the two cameras 1 and 2 are transformed into images I_L an I_R which are obtained by photographing the subject with just like parallel cameras (left-hand and right-hand side cameras having parallel optical axes) As a result, the epipola lines which are not originally parallel with each other are parallelized and become parallel with a straight line between the optical centers (the points c1 and c2 of sight). When the projection matrix P and P′ are obtained as the calibration parameters, transformation matrices TH1 and TH2 shown in FIG. 4 can be obtained by utilizing a method disclosed in the following reference literature for example. The input images are transformed by using the resulting transformation matrices TH1 and TH2 to generate a reference camera image Ib and a detecting camera image Id. [Reference literature: Andrea Fusiello, Emanuele Trucco and Alessandro Verri: A compact algorithm for rectification of stereo pairs, Machine Vision and Applications (2000)12, pp. 16 to 22]

[0066] That is, the transformation matrices are calculated from the projection matrices P and P′ of the left-hand and right-hand side cameras.

[0067] Actually, the projection matrices P and P′ are converted so that the X-axis of the XYZ coordinate system of the left-hand side camera which is used when the projection matrix P is obtained becomes parallel with the straight line between the optical centers c1 and c2 of the respective cameras 1 and 2, and the Y′-axis becomes perpendicular to a plane containing the straight line (X′-axis) between the optical centers c1 and c2 and the Z-axis. The transformation matrices are calculated from the corrected projection matrices P_n and P_n′.

[0068] Hereinafter, a procedure for calculating the corrected projection matrices P_n and P_n′ will be described. Also, a description will be given below with respect to a procedure for calculating the transformation matrices TH1 and TH2 from the resulting matrices P_n and P_n′ and the projection matrices P and P′.

(i) Calculation of Optical Centers

[0069] When the projection matrices P and P′ are described by Expression 7, the optical centers c1=[xc1, yc1, zc1]^T and c2=[xc2, yc2, zc2 ]^T are given by Expression 8:

[0070] $\begin{array}{cc}\begin{array}{c}P=\left[\begin{array}{cccc}{q}_{11}& q_{12}& q_{13}& q_{14}\\ q_{21}& q_{22}& q_{23}& q_{24}\\ q_{31}& q_{32}& q_{33}& q_{34}\end{array}\right]=[\begin{array}{c}{q}_1^T\\ {q_2^T}_{}\\ q_3^T\end{array}\begin{array}{c}{q}_{14}\\ q_{24}\\ q_{34}\end{array}]=\left[Q-\mathrm{Qc}1]=[Qq\right]\\ P^{\prime }=\left[\begin{array}{cccc}{q}_{11}^{\prime }& q_{12}^{\prime }& q_{13}^{\prime }& q_{14}^{\prime }\\ q_{21}^{\prime }& q_{22}^{\prime }& q_{23}^{\prime }& q_{24}^{\prime }\\ q_{31}^{\prime }& q_{32}^{\prime }& q_{33}^{\prime }& q_{34}^{\prime }\end{array}\right]=[\begin{array}{c}{q}_1^{\mathrm{\prime T}}\\ q_2^{\mathrm{\prime T}}\\ q_3^{\mathrm{\prime T}}\end{array}\begin{array}{c}{q}_{14}^{\prime }\\ q_{24}^{\prime }\\ q_{34}^{\prime }\end{array}]=\left[Q^{\prime }-Q^{\prime }c2]=[Q^{\prime }q^{\prime }\right]\\ \end{array}& \left(\mathrm{Expression}7\right)\end{array}$
c1=−Q⁻¹q
c2=−Q′⁻¹q′  (Expression 8)
(ii) Calculation of X′-, Y′- and Z′-Axes

[0071] As described above, X′-axis is the vector linking the optical centers. Thus, when being expressed by v1=[x1, y1, z1]^T, the vector v1 of the X′-axis is obtained by Expression 9:
v1=(c1−c2)   (Expression 9)

[0072] In addition, the vector v2=[x2, y2, z2]^T of the Y′-axis, as described above, is the vector perpendicular to the X′-axis and the Z-axis. Thus, when the projection matrix P is expressed by Expression 10, the vector v2 of the Y′-axis is expressed by v2=r₃Λv1:

[0073] $\begin{array}{cc}P=A\left[\mathrm{Rt}\right]=A[\begin{array}{c}{r}_1^T\\ r_2^T\\ r_3^T\end{array}\begin{array}{c}\\ t\\ \end{array}]& \left(\mathrm{Expression}10\right)\end{array}$

[0074] Moreover, the vector v3=[x3, y3, z3]^T of the Z′-axis is expressed as the vector perpendicular to the X′-axis and the Y′-axis by Expression 11:
v3=v1Λv2   (Expression 11)
(iii) Calculation of Projection Matrices P_n and P_n′

[0075] The projection matrices P_n and P′_n are described by Expression 12:
P_n=A_n·┌R_n t_n┘
P_n′=A_n·[R_n t_n′]  (Expression 12)

[0076] The rotation components of the internal parameters and the external parameters become identical to each other in the new projection matrices. Thus, in order to calculate the projection matrices P_n and P_n′, the internal and external parameters A_n, R_n, t_n, and t_n′ must be calculated. The internal parameter A_n is calculated by using the internal parameters of the projection matrices P and P′ by Expression 13:

[0077] $\begin{array}{cc}{A}_n=\frac{1}{2}\left(A+A^{\prime }\right)& \left(\mathrm{Expression}13\right)\end{array}$

[0078] In addition, the external parameter R_n is calculated by using the vectors v1, v2, and v3 of the X′-axis, the Y′-axis, and the Z′-axis which are calculated in accordance with the procedure as described above by Expression 14:

[0079] $\begin{array}{cc}{R}_n=\left[\begin{array}{c}v{1}^{\mathrm{\prime T}}\\ v{2}^{\mathrm{\prime T}}\\ v{3}^{\mathrm{\prime T}}\end{array}\right]& \left(\mathrm{Expression}14\right)\end{array}$
v1′=v1/∥v1∥
v2′=v2/∥v2∥
v3′=v3/∥v3∥  (Expression 14)

[0080] Also, the external parameters t_n and t_n′ are calculated by using the external parameter R_n and the optical centers c1 and c2 from Expression 7 by Expression 15:
t_n=−R_nc1
t_n′=−R_nc2   (Expression 15)

[0081] The projection matrices P_n and P_n′ can be calculated from Expression 12 based on the above-mentioned calculation of the internal and external parameters A_n, R_n, t_n, and t_n′.

(iv) Calculation of Transformation Matrices TH1 and TH2

[0082] In this chapter, a description will be given below with respect to a method of calculating the transformation matrices TH1 and TH2 by using the projection matrices P, P′, P_n, and P_n′.

[0083] When a point m^˜=[u, v, 1]^T of the captured image is assumed to be a point m^˜_n [u, v, 1]^T on the normalized image, a relationship between the point m^˜=[x, y, z, 1]^T and the point m^˜_n within the space can be described by Expression 16:
m^˜=PM^˜=[Q|−Qc1]M^˜
m^˜_n=P_nM^˜=[Q_n|−Q_nc1]M^˜  (Expression 16)

[0084] A straight line passing through the point m^˜ and the point m^˜_n can be expressed from Expression 16 by Expression 17:
M^˜=sQ⁻¹m^˜+c1
M^˜=S_nQ_n⁻¹m^˜_n+c1   (Expression 17)

[0085] Hence, the relationship between the point m^˜ and the point m^˜_n can be described by Expression 18:
m^˜_n=λQ_nQ⁻¹m^˜  (Expression 18)

[0086] Since the transformation matrix is the matrix through which the point m on the captured image is projected on the point m_n on the normalized image, a matrix Q_nQ⁻¹ becomes a transformation matrix. In other words, the transformation matrices TH1 and TH2 are calculated from Expression 19:
TH1=Q_nQ⁻¹
TH2=Q_n′Q′⁻¹   (Expression 19)

[0087] Subsequently, the background area estimation processing unit 4 calculates background area masks Rb_bg (u, v) and Rd_bg (u, v) (Step S3).

[0088] In this case, it is assumed that the background area is in the distance and near a flat surface. FIG. 5 shows a flow of processing for actually calculating the background area masks Rb_bg (u, v) and Rd_bg (u, v). In the background area masks Rb_bg (u, v) and Rd_bg (u, v), the background part is assigned “1” and other parts are assigned “0”.

[0089] For example, in case of input images Ib (u, v) and Id (u, v) as shown in FIG. 6, the background area has many repetitive patterns and the large occlusion area. Thus, when the normal local area-based matching method is utilized, the corresponding unit cannot be determined with high accuracy. Then, firstly, the background area is estimated by using a global image superimposing technique.

[0090] Here, the background area processing unit, as shown in a functional block diagram of FIG. 7, includes a global shift amount calculating unit 41, a background pixel estimating unit 42, and an update processing unit 43.

[0091] Then, in the processing of Step S3, as shown in FIG. 5, firstly, in processing of first Step S3-1, the global shift amount calculating unit 41 executes global shift amount calculation processing for obtaining a shift amount bg_shift such that a luminance error of the overall image becomes the minimum as shown in Expression 20 for the left-hand and right-hand side input images Ib (u, v) and Id (u, v) the data on which is stored in a frame memory FM1. Then, the global shift amount calculating unit 41 stores data on the resulting shit amount bg_shift in a memory M1.

[0092] $\begin{array}{cc}E\_g=\sum _i\mathrm{lb}\left(\mathrm{ui},\mathrm{vi}\right)-\mathrm{ld}\left(\mathrm{ui}\text{-}\mathrm{bg\_}\mathrm{shift},vi\right)& \left(\mathrm{Expression}20\right)\end{array}$

[0093] But, in this case, it is assumed that the background area is in the distance, and 0≦bg_shift<SHIFT_THRESH is established. SHIFT_THRESH is a threshold which is adaptively determined depending on a separation boundary between the foreground and the background. An area in and after a certain distance is desired not to be watched, i.e., this area is made the background, thereby determining the threshold. In addition, since the image processing is executed on the assumption that the background has a flat surface, this reduces the resolution of the input images Ib (u, v) and Id (u, v). Thus, the shift amount bg_shift is calculated from Expression 20. Since the input images Ib (u, v) and Id (u, v) are normalized in the processing of Step S2, a shift amount in a direction of a u-axis has only to be obtained in this case.

[0094] FIG. 8 shows an example of the image superimposition, and FIG. 9 shows a concrete flow of global shift amount calculating processing of Step S3-1.

[0095] That is, in the global shift amount calculating processing of Step S3-1, firstly, shift and E_min are initialized to zero and MAX, respectively (Step S3-1A). Moreover, E_g=0, i=0 are obtained, i.e., variables E_g and i are reset to “0” (Step S3-1B).

[0096] Next, E_gt⁺=|Ib (ui, vi)−Id (ui-shift, vi)| is obtained, i.e., a sum of absolute values of differences between Ib (ui, vi) and Id (ui-shift, vi) is obtained with respect to the input images Ib (u, v) and Id (u, v) (Step S3-1C). The variable i is then incremented (Step S3-1D). It is then determined whether or not the incremented variable i becomes larger than an image size (Step S3-1E).

[0097] When the determination results in Step S3-1E show NO, the operation returns back to the processing of Step S3-1C, and the processing from Step S3-1C to Step S3-1E is repeatedly executed. As a result, when the determination results in Step S3-1E come to show YES, it is determined whether or not the variable E_g representing the sum of the absolute values of the differences between Ib (ui, vi) and Id (ui-shift, vi) is smaller than a variable E min (Step S3-1F).

[0098] When the determination results in Step S3-1F show YES, the variable E_min is made equal to the value of the variable E_g representing the above-mentioned sum of the absolute values, and a shift amount bg_shift is made equal to the value of the variable shift (Step S3-1G). Thereafter, the variable shift is incremented (Step S3-1H). On the other hand, when the determination results in Step S3-1F show NO, the variable shift is immediately increased (Step S3-1H).

[0099] Then, it is determined whether or not the variable shift incremented in Step S3-1H is larger than a threshold SHIFT_THRESH (Step S3-1I).

[0100] When the determination results in Step S3-1I show NO, the operation returns back to the processing of Step S3-1B. Then, the processing from Step S3-1B to Step S3-1I is repeatedly executed until the incremented variable shift becomes larger than the threshold SHIFT_THRESH. As a result, when the determination results in Step S3-1I come to show YES, the global shift amount calculating processing is completed.

[0101] Next, in the processing of Step S3-2, the background pixel estimating unit 42 performs the local matching with respect to the left-hand and right-hand side input images Ib (u, v) and Id (u, v) both the data on which is stored in the frame memory FM1 by using the shift amount bg_shift obtained in the global shift amount calculating processing of Step S3-1. More specifically, as shown in FIG. 10, a luminance error E_l within a Window size of M×N is calculated. When E_l<THRESH_L, points (ui, vi) and (ui−bg_shift, vi) of the images Ib and Id are made the background area. In other words, Rb_bg (vi, ui)=1 and Rd_bg (ui-bg_shift, vi)=1 are obtained. The data on the background area estimated through the local matching is stored as the background area masks Rb_bg and Rd_bg in the frame memory FM2 and the memory M2, respectively. FIG. 11 shows a concrete flow of the processing of Step S3-2.

[0102] That is, in the processing of Step S3-2, firstly, the initialization for obtaining i=0, Rb_bg≦all0, and Rd_bg≦all0 is performed (Step S3-2A). Then, j=0 and E_l=0 are obtained, i.e., the variables j and E_l are reset to “0” (Step S3-2B).

[0103] Next, E_l⁺=|Ib (uij, vij)−Id (uij−bg_shift, vij)| is obtained, i.e., a sum of absolute values of differences between Ib (uij, vij) and Id (uij−bg_shift, vij) is obtained as the luminance error E_l (Step S3-2C). The variable j is then incremented (Step S3-2D). It is determined whether or not the incremented variable j becomes larger than the Window size of M×N (Step S3-2E).

[0104] When the determination results in Step S3-2E show NO, the operation returns back to the processing of Step S3-2C, and the processing from Step S3-2C to Step S3-2E is repeatedly executed. As a result, when the determination results in Step S3-2E come to show YES, it is determined whether or not the resulting luminance error E_l within the Window size of M×N is smaller than a threshold THRESH_L (Step S3-2F).

[0105] When the determination results in Step S3-2F show YES, after Rb_bg (ui, vi)=1 and Rd_bg (ui−bg_shift, vi)=1 are obtained, i.e., the points (ui, vi) and (ui−bg_shift, vi) of the images Ib and Id are made the background area (Step S3-2G), the variable i is incremented (Step S3-2H). On the other hand, when the determination results in Step S3-2F show NO, the variable i is immediately incremented (Step S3-2H).

[0106] It is then determined whether or not the variable i incremented in Step S3-2H is larger than the image size (Step S3-2I).

[0107] When the determination results in Step S3-2I show NO, the operation returns back to the processing of Step S3-2B. The processing from Step S3-2B to Step S3-2I is then repeatedly executed until the incremented variable i becomes larger than the image size. As a result, when the determination results in Step S3-2I come to show YES, the local similarity degree calculating processing is completed.

[0108] Next, in the processing of Step S3-3, the image Ib captured with the reference camera 1 and the image Id captured with the detecting camera 2 are compared with the background template stacks bgBufb and bgBufd, respectively. Each of the background template stacks bgBufb and bgBufd is a stack of a local image which is obtained by cutting down a peripheral area (M×N) of the points made the background in the frame by a time (T−1). Also, each of the background template stacks bgBufb and bgBufd has luminance information having a size of (M×N) and its central coordinates (ui, vi). The matching between each template stacked on the background template stack bgBufb and the image Ib (ui, vi) captured with the reference camera 1 is performed and a luminance error E_T is calculated. When E_T<THRESH_T, the point (ui, vi) is made the background area, and Rb_bg (ui, vi)=1 is obtained. The background template stack bgBufd is used for the image Id captured with the detecting camera 2, and the same processing as that for the image Ib is executed to generate the background area mask Rb_bg (u, v) FIG. 12 shows an outline of the processing of Step S3-3 for the reference camera 1, and FIG. 13 shows a concrete flow of the processing of Step S3-3.

[0109] That is, in the processing of Step S3-3, firstly, the variable i is initialized to “0” (Step S3-3A), and j=0 is then obtained, i.e., the variable j is then reset to “0” (Step S3-3B).

[0110] Next, E_T=Σ|Ibj−bgBufb[i]| is obtained, i.e., a sum of absolute values of differences between Ibj and bgBufb[i] is obtained as the luminance error E_T with respect to the input images Ib (u, v) and Id (u, v) (Step S3-3C). It is then determined whether or not the resulting luminance error E_T is smaller than a threshold E_T<THRESH_T (Step S3-3D).

[0111] When the determination results in Step S3-3D show YES, after Rb_bg (ui, vi), i.e., the point (ui, vi) is made the background area (Step S3-3E), the variable j is incremented (Step S3-3F). It is then determined whether or not the incremented variable j becomes larger than the image size (Step S3-3G).

[0112] When the determination results in Step S3-3G show NO, the operation returns back to the processing of Step S3-3C, and the processing from Step S3-3C to Step S3-3G is repeatedly executed. As a result, when the determination results in Step S3-3G come to show YES, the variable j is incremented (Step S3-3H). On the other hand, when the determination results in Step S3-3D show NO, the variable i is immediately incremented (Step S3-3H).

[0113] It is then determined whether or not the variable i incremented in Step S3-3H is larger than the stack size (Step S3-3I).

[0114] When the determination results in Step S3-3I show NO, the operation returns back to the processing of Step S3-3B. The processing from Step S3-3B to Step S3-3I is then repeatedly executed until the incremented variable i becomes larger than the stack size. As a result, when the determination results in Step S3-3I come to show YES, the processing for comparison with the templates is completed.

[0115] Subsequently, in the processing of Step S3-4, the update processing unit 43 updates the background template stacks by using the background area masks Rb_bg (u, v) and Rd_bg (u, v) which are generated in the above-mentioned processing. More specifically, an area R_n of the peripheral pixels of M×N of the point (ui, vi) which has the background area mask value of 1, i.e., which is estimated to be the background is cut down, and reference is then made to the background template stacks. When the background template for the point (ui, vi) previously exists, the existing template and the new template R_n are averaged. On the other hand, when no background template for the point (ui, vi) previously exists, the new template is stacked. FIG. 14 shows a concrete flow of the update processing for the reference camera 1.

[0116] That is, in the processing of Step S3-4, firstly, the variables i and j are initialized to “0” (Step S3-4A)

[0117] Next, val=Rb_bg (ui, vi) is obtained, i.e., a variable val is made equal to the value of Rb_bg (ui, vi) (Step S3-4B), and it is determined whether or not the value of the variable val is “1” (Step S3-4C).

[0118] When the determination results in Step S3-4C show YES, reference is made to coordinates (bfuj, bfvj) of bgBufb[j] (Step S3-4D). It is then determined whether or not the coordinates (bfuj, bfvj) agree with a point (uj, vj) (Step S3-4E).

[0119] When the determination results in Step S3-4E show YES, after the template for bgBufb[j] is updated (Step S3-4F), the variable j is incremented (Step S3-4G). On the other hand, when the determination results in Step S3-4E show NO, the variable j is immediately incremented (Step S3-4G).

[0120] Next, it is determined whether or not the variable j incremented in Step S3-4G becomes larger than the stack size (Step S3-4H).

[0121] When the determination results in Step S3-3H show NO, the operation returns back to the processing of Step S3-4D, and the processing from Step S3-4D to Step S3-4H is then repeatedly executed. As a result, when the determination results in Step S3-4H come to show YES, the template for the point (uj, vj) is added to bgBufb (Step S3-4I), and the variable i is incremented (Step S3-4J). On the other hand, when the determination results in Step S3-4C show NO, the variable i is immediately incremented (Step S3-4J).

[0122] It is then determined whether or not the variable i incremented in Step S3-4J is larger than the image size of the variable i (Step S3-4K).

[0123] When the determination results in Step S3-4K show NO, the operation returns back to the processing of Step S3-4B, and the processing from Step S3-4B to Step S3-4K is then repeatedly executed until the incremented variable i becomes larger than the image size. As a result, when the determination results in Step S3-4K come to show YES, the processing for updating the background templates is completed.

[0124] Next, the distance image generation processing unit 5 calculates distance information by using the background area masks Rb_bg (u, v) and Rd_bg (u, v) which are generated in the processing of Step S3 (Step S4).

[0125] Here, in the processing of Step S4, distance images Db (u, v) and Dd (u, v) are generated based on the information of the background area masks Rb_bg (u, v) and Rd_bg (u, v) which are generated in the processing of Step S3. The values of the distance images in the processing of Step S4 are set as shift amounts in a u-axis direction since the input images Ib (u, v) and Id (u, v) are normalized in the processing of Step S2. In other words, when the point corresponding to the point Ib (ui, vi) on the reference camera image Ib (u, v) is made the point Id (ui-shift, vi) on the detecting camera image Id (u, v), Db (ui, vi)=shift is obtained. The various methods are known-as the method of searching for a corresponding point. In this case, however, the corresponding point is obtained by searching for a point in which a luminance error between the peripheral area of M×N of the point (ui, vi) on the reference camera image Ib (u, v) and the peripheral area of M×N of the point (ui-shift, vi) on the detecting camera image Id (u, v) becomes the minimum. At this time, as shown in FIG. 15, any of the points made the background area (background area mask takes 1) is excluded in search by utilizing the background area mask information. As a result, the matching can be more accurately performed than the matching for the area base for the overall ordinary input image can be done.

[0126] The background separated image generation processing unit 6 removes any of the points (ui, vi) each having a distance equal to or smaller than a-threshold from the input images Ib (u, v) and Id (u, v) by referring to the distance area images Db (u, v) and Dd (u, v) which are generated in the processing of Step S4 Then, the background separated image generation processing unit 6 generates background separated images Ib′ (u, v) and Id′ (u, v) (Step S5), and outputs the resulting background separated images Ib′ (u, v) and Id′ (u, v) (Step S6). In the threshold processing in this case, the points each in a certain distance or more are made the background.

[0127] For the background separated images Ib′ (u, v) and Id′ (u, v) which are generated in the above-mentioned processing, the global and local background estimation is performed for the input images Ib (u, v) and Id (u, v), the distance images are generated, and the background separation is performed for the distance images. As a result, the background separation becomes possible which is more accurate than that utilizing the method in which the distance images are generated for the overall input images Ib (u, v) and Id (u, v), and the background separation is then performed. In addition, the background template stacks are updated/added whenever necessary, which becomes robust against the environment change due to a lapse of time. Moreover, the superimposition of the overall image is used and the rough background area is estimated, whereby the robust results can be obtained for a scene as well having many repetitive patterns and occlusions.

[0128] While preferred embodiments of the present invention have been described using specific terms, such description is for illustrative purposes only, and it is to be understood that changes and variations may be made without departing from the spirit or scope of the following claims.