Method and apparatus for segmenting an object region of interest from an image

[0001] The present application claims priority to Chinese Patent Application Number 200710195724.6, filed Dec. 13, 2007, the entirety of which is hereby incorporated by reference.

[0002] The invention relates to object detection based on computer vision, and particularly to a method and apparatus for segmenting an object region of interest from an image.

[0003] An existing method for detecting vehicles based on computer vision segments an image region that probably includes a vehicle as a vehicle region of interest from an image captured by an imaging device, such as a video camera, according to features such as the vehicle shadow and vehicle edges.

[0004] However, in a bad environment with low contrast, such as when it is raining, snowing, or foggy, the features, such as vehicle shadow and vehicle edges, will be significantly diminished (as shown in FIG. 1a), or may even vanish (as shown in FIG. 1b). In this case, segmenting a vehicle region of interest from an image according to the vehicle shadow and vehicle edges may not be possible.

[0005] In view of the above the problem, it is an object of the present invention to provide a method and apparatus for segmenting an object region of interest from an image, which can segment an object region of interest from an image without utilizing features such as vehicle shadow and vehicle edges.

[0006] To achieve this goal, a method for segmenting an object region of interest from an image according to one embodiment of the invention comprises detecting each corner in at least a portion of a captured image; acquiring edges of the object from the captured image based on the detected corners; clustering the detected corners based on corners on the acquired edges; and segmenting the image region in the captured image formed by the clustered corners as a region of interest that likely includes the object.

[0007] To achieve the goal, an apparatus for segmenting an object region of interest from an image according to one embodiment of the invention comprises a detecting means for detecting each corner in at lease a portion of a captured image; an acquiring means for acquiring edges of the object from the captured image based on the detected corners; a clustering means for clustering the detected corners based on corners on the acquired edges; and a segmenting means for segmenting the image region in the captured image formed by the clustered corners as a region of interest that likely includes the object.

[0023] According to one aspect, first, corners are detected from a captured image, wherein corners refer to points whose intensity varies more significantly compared with other points in any direction (horizontal direction, vertical direction, or directions of any angles as shown in FIG. 2a), and a corner is different from an edge in that the intensity of the edge only varies with respect to other points in the direction perpendicular to the edge (as shown in FIG. 2b). Second, edges of an object are acquired from the captured image by utilizing the detected corners. Next, the detected corners are clustered based on the corners in the acquired edges, and lastly, the image region in the captured image formed by the clustered corners is segmented as a region of interest that likely includes the object.

[0024] The method and apparatus for segmenting an object region of interest from an image according to various embodiments of the invention will be described in more detail below, taking a vehicle as an example of the object in conjunction with the accompanying drawings.

(First Embodiment)

[0025] FIG. 3 is a flowchart of a method for segmenting a vehicle region of interest from an image according to the first embodiment of the invention. As shown in FIG. 3, first, in step S10, corners in the portion below the vanishing line in the image captured by an imaging device such as a video camera are detected by using a corner detecting method (as illustrated in FIG. 4). The corner detecting method may be any one of the various existing or forthcoming corner detecting methods, such as the Harris corner detecting method, or the SUSAN corner detecting algorithm.

[0026] Then, in step S20, the number of corners included in each column (as illustrated in FIG. 5) of the captured image are calculated, so as to obtain a statistical histogram of corners of the captured image (as shown in FIG. 6). The width ΔT of each column can be set according to actual requirements, which in this embodiment is ΔT=2.

[0027] In step S30, the column that includes the greatest number of corners is determined from the captured image according to the statistical histogram of corners. It is known from the principle of corner detecting methods that, generally in an image, more corners appear at the vehicle edges and fewer corners appear at the location of the vehicle body itself, and the fewest number of corners appear in plane objects such as even road surfaces. Therefore, the column that includes the greatest number of corners in the captured image usually includes vehicle edges.

[0028] In step S40, line segments that include vehicle edges are extracted from the column that includes the greatest number of corners. The process for extracting line segments that include vehicle edges according to one embodiment of the invention will be explained in detail in conjunction with FIG. 9.

[0029] In step S50, the detected corners in the captured image are clustered with a predetermined clustering radius based on the corners in the extracted line segments. Specifically, the process of clustering is as follows: it is supposed that N corners are detected from the image, which compose a set C_S, C_S={C₁, C₂, . . . , C_N}. Any one of the corners C_i on the extracted line segment is selected as a clustering origin with T as a clustering radius to start clustering, such that the corners C_j(j≠i, C_jεC_s) falling within a circle with C_i as the clustering origin and T as the clustering radius constitute a cluster Z. Next, the above process is repeated with all the other corners in cluster Z with C_i as the clustering origin and with T as the clustering radius, until no corner complying with this condition is added into cluster Z. Thus, a cluster Z is formed. In this embodiment, the clustering radius is equal to the length of the extracted line segment. However, the invention is not limited to that, and in other embodiments the clustering radius may have other values according to the actual requirements.

[0030] In step S60, an image region formed by the corners obtained by clustering is determined from the captured image, and a bounding box of this image region is computed, as shown in FIG. 7.

[0031] The method of computing the bounding box is as follows:

[0032] top border of the box

[0033] ${\mathrm{BoundingBox}}_{\mathrm{top}}=\stackrel{N}{\underset{i=1}{\min }}{y}_1,\left(x_1,y_1\right)\in C$

[0034] bottom border of the box

[0035] ${\mathrm{BoundingBox}}_{\mathrm{bottom}}=\underset{i=1}{\stackrel{N}{\max }}{y}_1,\left(x_1,y_1\right)\in C$

[0036] left border of the box

[0037] ${\mathrm{BoundingBox}}_{\mathrm{left}}=\underset{i=1}{\stackrel{N}{\min }}{x}_1,\left(x_1,y_1\right)\in C$

[0038] right border of the box

[0039] ${\mathrm{BoundingBox}}_{\mathrm{right}}=\underset{i=1}{\stackrel{N}{\max }}{x}_1,\left(x_1,y_1\right)\in C$

[0040] Here, C is the image region formed by the corners obtained by clustering, x_i and y_i are the coordinates of the corners in C, and BoundingBox is the obtained bounding box, as illustrated in FIG. 8.

[0041] In step S70, the height of the bounding box is adjusted according to the aspect ratio of the actual vehicle, as shown in FIG. 9.

[0042] In step S80, the adjusted bounding box is segmented from the captured image as a region of interest that likely includes a vehicle.

[0043] In step S90, it is judged whether all regions of interest that may include a vehicle have been segmented from the captured image. If the judging result is NO, the process returns to step S30 to determine other regions of interest that may include a vehicle based on each column in the captured image except for the columns covered by the image regions that have been segmented as regions of interest previously. If the judging result is YES, the process is ended.

[0044] FIG. 10 is a flowchart of a process for extracting line segments that include vehicle edges according to one embodiment of the invention.

[0045] First, in step S402, the vertical coordinate Y_min of the topmost corner and the vertical coordinate Y_max of the lowermost corner in the column that includes the greatest number of corners are computed according to coordinate information of each corner in the column in the image.

[0046] $\begin{array}{cc}{Y}_{\min }=\stackrel{\mathrm{CN}}{\underset{i=1}{\min y_1}},x_i\in E_x,& Y_{\max }=\stackrel{\mathrm{CN}}{\underset{i=1}{\max y_i}},x_i\in E_x\end{array}$

[0047] Here, x_i and y_i are the horizontal coordinate and the vertical coordinate of the ith corner C(x_i, y_i) in the column that includes the greatest number of corners, respectively, CN is the number of the corners in that column, and E_x is the column that includes the greatest number of corners.

[0048] In step S404, the midpoint Y_mid=(Y_min+Y_max)/2 of line segment Y_min Y_max is computed according to Y_min and Y_max. The midpoint Y_mid equally divides the line segment [Y_min, Y_max] into two line segments Sub_A=[Y_min, Y_mid] and Sub_B [Y_mid, Y_max].

[0049] In step S406, the densities of corners in line segments Sub_A and Sub_B are computed. The definition of the density of corners is as follows:

[0050] $D_A=\frac{\sum _{i=1}^{\mathrm{CN}}{S}_A\left(i\right)}{{\mathrm{Len}}_A},D_B=\frac{\sum _{i=1}^{\mathrm{CN}}{S}_B\left(i\right)}{{\mathrm{Len}}_B}$

[0051] Here, Len_A and Len_B are the lengths of line segments Sub_A and Sub_B, respectively, and D_A and D_B are the densities of corners in line segments Sub_A and Sub_B, respectively.

[0052] $\begin{array}{c}{\mathrm{Len}}_A=Y_{\mathrm{mid}}-Y_{\min }+1,{\mathrm{Len}}_B=Y_{\max }-Y_{\mathrm{mid}}\\ S_A\left(i\right)=\{\begin{array}{cc}1,& x_i\in E_x\bigcap y_i\in \left[Y_{\min },Y_{\mathrm{mid}}\right]\\ 0,& \mathrm{others}\end{array}\\ S_B\left(i\right)=\{\begin{array}{cc}1,& x_i\in E_x\bigcap y_i\in \left[Y_{\mathrm{mid}},Y_{\max }\right]\\ 0,& \mathrm{others}\end{array}\end{array}$

[0053] In step S408, the line segment with a larger density of corners is selected from the line segments Sub_A and Sub_B computed at that time. If the densities of corners of line segments Sub_A and Sub_B are the same, line segment Sub_A is selected.

[0054] If the selected line segment with a larger density of corners is line segment Sub_A, in step S410 it is judged whether its density D_A is larger than D_max (with an initial value 0). If the judging result is positive, in step S412 it is set that D_max=D_A and Y_max=Y_mid, and then the process returns to step S404. If the judging result is negative, at step S414 the current line segment Sub_A is extracted as a line segment that includes vehicle edges.

[0055] If the selected line segment with a larger density of corners is line segment Sub_B, in step S416 it is judged whether its density D_B is larger than D_max. If the judging result is positive, in step S418 it is set that D_max=D_B and Y_min=Y_mid+1, then the process returns to step S404. If the judging result is negative, at step S420 the current line segment Sub_B is extracted as a line segment that includes vehicle edges.

(Second Embodiment)

[0056] In the second embodiment, after a region of interest is obtained by segmentation according to the procedure of the first embodiment, the region of interest obtained by segmentation is further processed to obtain a more accurate region of interest.

[0057] FIG. 11 is a flowchart of a method for segmenting a vehicle region of interest from an image according to the second embodiment of the invention. In order to avoid repeated explanation, only the steps different from that of the first embodiment are shown in FIG. 11.

[0058] As shown in FIG. 11, in step S100, in the captured image, the left border and the right border of the region of interest obtained by segmentation in step S80 are moved inward and outward by a certain width, respectively, and the top border and the bottom border of the region of interest are moved inward and outward by a certain height, respectively, so as to obtain four extended regions, i.e., a left extended region, a right extended region, a top extended region and bottom extended region. Here, the certain width is ΔW=0.1×(width of the region of interest), and the certain height is ΔH=0.1×(height of the region of interest). FIG. 12 is a schematic drawing of an extension of the region of interest according to one embodiment of the invention, wherein abcd represents a region of interest before extension, ABIJ represents a left extended region, CDGH represents a right extended region, ADEL represents a top extended region and FGJK represents a bottom extended region.

[0059] In step S110, corners within the left extended region, right extended region, top extended region, and bottom extended region are detected by using a corner detecting method.

[0060] In step S120, the number of corners of each column in the left extended region and the right extended region are calculated, as well as the number of corners of each row in the top extended region and the bottom extended region.

[0061] In step S130, the column in the left extended region that includes the greatest number of corners, the column in the right extended region that includes the greatest number of corners, the row in the top extended region that includes the greatest number of corners, and the row in the bottom extended region that includes the greatest number of corners are determined.

[0062] In step S140, positions of the four borders of the region of interest are adjusted, that is to say, the left border of the region of interest is adjusted to the column in the left extended region that includes the greatest number of corners, the right border of the region of interest is adjusted to the column in the right extended region that includes the greatest number of corners, the top border of the region of interest is adjusted to the row in the top extended region that includes the greatest number of corners, and the bottom border of the region of interest is adjusted to the row in the bottom extended region that includes the greatest number of corners.

[0063] Although, in the first embodiment, corners in the portion below the vanishing line in a captured image are detected, the invention is not limited to that. In other embodiments of the invention, corners in the entire captured image may be detected.

[0064] Although, in step S70 of the first embodiment, the height of the bounding box is adjusted according to the aspect ratio of an actual vehicle, the invention is not limited to that. In other embodiments of the invention, the height of the bounding box may not be adjusted.

[0065] Although, in the second embodiment, four borders of the region of interest are adjusted concurrently, in other embodiments of the invention, only one border, two borders, or three borders of the region of interest may be adjusted.

[0066] Although, in the second embodiment, the borders of the region of interest are moved inward and outward concurrently by a predetermined width, respectively, so as to obtain extended regions, the invention is not limited to that. In other embodiments of the invention, it is possible to extend the borders of the region of interest only inward or outward by a predetermined width to obtain the extended regions.

[0067] Although, in the second embodiment, corners in each of the four extended regions of the region of interest are re-detected, in other embodiments of the invention, if the corners in any of the four extended regions have been detected previously, the previously detected corners may be directly used to calculate the number of corners in each column or row of that extension region.

[0068] Although, in the first and second embodiments, the image region that likely includes a vehicle is segmented from an image, in other embodiments of the invention, an image region that likely includes another object such as a passerby, etc. may be segmented.

[0069] The method for segmenting an object region of interest from an image according to the invention may be implemented by means of software or hardware, as well as a combination thereof.

[0070] FIG. 13 is a schematic drawing of an apparatus for segmenting a vehicle region of interest from an image according to the first embodiment of the invention. Only the modules related to the invention are shown.

[0071] As shown in FIG. 13, an apparatus 100 for segmenting a vehicle region of interest from an image includes:

[0072] a detecting means 101 for detecting each corner in at least a portion of a captured image, e.g., for detecting each corner in the portion below the vanishing line of the captured image;

[0073] an acquiring means 103 for acquiring edges of an object from the captured image based on the detected corners, wherein the acquiring means 103 further includes an extracting unit 1031 for extracting the column that includes the greatest number of corners from each column except for the columns covered by the image regions that have been segmented as regions of interest previously from the captured image; and a determining unit 1033 for determining a line segment with the density of corners larger than a predetermined value as an edge of the object from the extracted column;

[0074] a clustering means 105 for clustering the detected corners based on corners on the acquired edge;

[0075] a computing means 107 for computing a bounding box of the image region;

[0076] a first adjusting means 109 for adjusting the height of the bounding box according to the aspect ratio of the object;

[0077] a segmenting means 111 for segmenting the bounding box as the region of interest from the captured image;

[0078] a first extending means 113 for moving at least one of the left border and the right border of the region of interest inward and/or outward by a predetermined width, respectively;

[0079] a second extending means 115 for moving at least one of the top border and the bottom border of the region of interest inward and/or outward by a predetermined height, respectively, so as to obtain extended regions;

[0080] a first determining means 117 for determining a column that includes the greatest number of corners in the extended regions obtained by the first extending means 113;

[0081] a second determining means 119 for determining a row that includes the greatest number of corners in the extended regions obtained by the second extending means 115;

[0082] a second adjusting means 121 for adjusting the corresponding border of the left border or the right border of the region of interest to the column found by the first determining means 117; and

[0083] a third adjusting means 123 for adjusting the corresponding border of the top border or the bottom border of the region of interest to the row found by the second determining means 119.

[0084] While there has been illustrated and described what is at present contemplated to be preferred embodiments of the present invention, it will be understood by those skilled in the art that various changes and modifications may be made, and equivalents may be substituted for elements thereof without departing from the true scope of the invention. In addition, many modifications may be made to adapt a particular situation to the teachings of the invention without departing from the central scope thereof. Therefore, it is intended that this invention not be limited to the particular embodiments disclosed, but that the invention will include all embodiments falling within the scope of the appended claims.