METHOD FOR VIDEO OBJECT SEGMENTATION

[0001] This application claims the provisional application, which is U.S. Provisional Application No. 60/766,973, filed Feb. 22, 2006 and is included herein by reference.

[0002] 1. Field of the Invention

[0003] The present invention relates to a method for video object segmentation, and more particularly relates to a method for video object segmentation that updates a static object to the background by updating background.

[0004] 2. Description of the Prior Art

[0005] In recent years, computer and communication techniques have improved gratefully, as well as digital media techniques. Since digital media techniques can directly transmit to users by utilizing reduced data amounts, these improvements deeply affect users of such technology. The concept of media comprises not only normally used music and images, but also other media such as spoken words, images and diagrams. 3C (computer, communication, and consuming) integrated electronic products such as MP3 video, mobile phones with cameras, and electronic clothes also popularize media techniques. In this broad field, video technique is a popular research object. Different from single static image, video also considers time, thus video can be regarded as a single image continuously varying in a specific period of time. Therefore, the data amount increases rapidly as time increases, and it becomes a serious problem of media signal processing.

[0006] Prior art video object segmentation methods can be classified into two kinds, one kind of which is automatic video object segmentation, and the other kind is semi-automatic video object segmentation. Automatic video object segmentation is also called unsupervised video object segmentation method, which automatically segments the video object through a computer. Semi-automatic video object segmentation method is also called supervised video object segmentation method, which defines the object regions to be segmented prior to segmentation, and frames are segmented according to defined object regions. The two kinds of segmentation methods are described below. Normally, automatic video object segmentation method comprises the video object segmentation method based on object edge information, the video object segmentation method based on time and spatial domain, and the semi-automatic video object segmentation method comprising video object segmentation methods which interacts with the user.

[0007] The video object segmentation method based on object edge information always utilizes Canny edge detecting operators to get object outline information. In the first step, the method computes the difference between two nearby frames, and performs Canny edge detection to generate a double object edge map. After that, object edge of the current frame is subtracted. Next, the edge of static and dynamic edge can be merged to obtain the object outline. Finally, a more complete outline can be obtained by jointly detecting horizontal and vertical regions and type operating. Such a method utilizes a Canny operator, which determines reliability of the edges according to the gradient, to obtain a static object. The difference gradient is near to zero when the grains of object region and background region are similar, thus the object edge may miss. Also, noises may be causes of error in determining dynamic and static edges. The segmented object outlines therefore, may be different from the original object edge. Additionally, another disadvantage of this method is that an absolute background is obtained from video, that is, no moving objects, but such situation rarely happens in real word.

[0008] The video object segmentation method based on time domain and spatial domain is the most popular segmentation algorithm, and utilizes segmentation region in spatial domain to assist segmentation defect to provide an accurate outline location. The most popular video object segmentation method based on time domain and spatial domain always comprises watershed techniques and K-Mean grouping techniques. Such algorithms always comprise: time domain, spatial domain, and the combination of time domain and spatial domain. The time domain further comprises three steps: all field motion estimation and compensation, environment change detection and core segmentation. The all field motion estimation and compensation is used for compensate the movement of camera, and the environment change detection is used for detecting if there is environment change in inputted video frames. These two methods are preprocessing steps for obtaining time domain. After that, the core segmentation of time domain is performed, and a change detecting method is used to determine if there is any pixel changing in the obtained motion information by utilizing probability estimation method. After that, spatial domain segmentation is performed, which utilizes image content to define the images as meaningful or non-meaningful. However, the regions defined by a computer are different from which defined by human observation. That is, some regions are defined as different through human eyes but are defined as similar with a computer.

[0009] The watershed technique classifies the pixels with similar gray levels as the same region. Of all methods of watershed technique, an immersion-based method is the most popular, which starts operation from the minimum value of the region, that is, the location with a minimum value of image gradient. It is imagined that a hole is formed, and the water rises from this hole, and pixels with similar gradient values are classified as the same region, and the region is enlarged until it reaches the maximum value of the image gradient value that is the highest point of the plane. Then, a dam is build to prevent the water of different regions. Finally, the information in time domain and spatial domain is merged to obtain final object region outline.

[0010] The watershed technique is sensitive to noise however, and the problem of over-segmentation may occur. Though most effects can be omitted via image processing, apparent grains may happen on face region, that is, the face regions should be determined as the region but still be divided). Thus, a region merging method is needed to solve the problem of over-segmenting. Such methods increase the loading of the system. However, the complexity of merging it to a real time system is also increased.

[0011] Additionally, K-Mean grouping technique is used to divide the region into k similar regions. After that, an AND operation is performed to the divided region and the corresponding change detection result region, and if the result of AND operation divided by the pixel numbers of the region is larger than predetermined threshold value, then the region is determined as a moving region. Afterwards, a region description operator is used to obtain a moving object. However, such methods do not utilize a threshold value adaptive to video, and thus the obtained moving region is not perfect. Since the detection result of most of the foreground object with static parts are ignored, the foreground object is barely obtained after the region dividing operation. In this case, though the complete object outline can be obtained from the region description operator, more computing time for comparing front and current frames is needed, which is hard for a real time system. Furthermore, it is also an important issue that the K-Mean grouping technique determines the number of classified groups.

[0012] The method based on change detection is used for detecting the variance between two nearby frames. The most popular detection method is to subtract nearby frames, to thereby obtain the difference indicating the variance level of the object in nearby frames to be a reference for changing of nearby frames.

[0013] After change detecting, the result of object masking is utilized next to obtain a complete object outline. Since the difference of frames is directly utilized to determine moving information, this method has low resistance to outside interference such as light change shadow or noise, but has a low computing amount. Additionally, such methods may lose useable motion information if the moving object suddenly stops or moves slowly after moving for a period of time. Some inventions disclose how to solve this problem, but still cannot solve the problems of changing brightness and shadow effect, and the uncovered background of which may incorrectly determine the background region as a foreground region. One of the prior art methods utilizes the motion estimation method to determine if the displacement vector in the change region at time k corresponds to the change region at time k+1. If it does, the region is determined as a foreground object; otherwise it is a background region. Such a method cannot have a high accuracy of edges due to the use of motion estimation. Otherwise, the method can only process translation type objects, and may wrongly determine the movement of other types, thus the computing complexity of the system may increase.

[0014] Finally, the video object segmentation method, which interacts with the user, allows a user to circle the object outline of the object to be segmented first. Then, it tracks the circled region in following frames and updates the relative information, and combines the image content to obtain object region. Such method may have finer objects outlined, but may incorrectly update outlines if the obtained characteristics do not have high relative characteristics. Thus efficiency is decreased, and the application of such methods is limited.

[0015] As described-above, each method presents different disadvantages. A new invention is therefore needed to solve above mentioned problems.

[0016] Therefore, one objective of the present invention is to provide an image object segmentation method for detecting changes and updating processes to solve the above-mentioned problems.

[0017] An embodiment of the present invention discloses a method for video object segmenting, comprising: (a) detecting a changing state between a current frame and a previous frame; (b) categorizing the current frame as a moving region, a still region, or a background region and an uncovered background region according to the detecting result from the step (a); (c) determining a moving object in the current frame according to the sorting result from the step (b); and (d) updating a background component.

[0018] Step (d) can also further comprise: (d1′) removing non-static components from the still region; and (d2) updating the still region to part of the background component if the still region determined in step (c) keeps static for more than a predetermined amount of time. The step (d1′) utilizes a region adjacent graphic to determine if there is any moving region adjacent to the still region to remove the non-static component.

[0019] The step (a) can also further comprise: (a1) detecting the pixel difference between the current frame and the previous frame; and (a2) detecting the pixel value difference between the current frame and the background component; wherein step (b) processes according to step (a1) and (a2). Step (a1) can comprises: analyzing the gray level distribution of the pixel value difference to determine a specific gray level difference value with the most pixel numbers; determining a background standard deviation corresponding to a specific background model, according to the specific gray level difference value and a pixel number corresponding to the specific gray level difference value; and classifying the pixels in the current frame according to the background standard deviation.

[0020] The method of the present invention can further comprise: (e) using hierarchy block segmentation method to find a plurality of edge blocks corresponding to the moving object, to amend the outline of the moving object. Step (e) can also comprise: (e1) segmenting the moving object to a plurality of blocks by using the size of a block; (e2) determining if each one of the blocks is a moving block or not, and if yes, then keeping the block as an edge block. If no, then removing the block; and (e3) reducing the size of the block; wherein the steps (e1), (e2) and (e3) are repeated until the block size reaches a predetermined block size.

[0021] Alternatively, the method can further comprise: (e′) estimating background component of the outline of the moving object. The step (e′) comprises: obtaining a plurality edge region growing seeds; performing region growing according to the region growing seeds to generate an edge region corresponding the background component; and removing the outline of the moving object to estimate the edge region.

[0022] These and other objectives of the present invention will no doubt become obvious to those of ordinary skill in the art after reading the following detailed description of the preferred embodiment that is illustrated in the various figures and drawings.

[0033] Some concepts of the present invention have already been published in the proceedings of conference, as described below: Thou-Ho (Chao-Ho) Chen, Tsong-Yi Chen and Yung-Chuen Chiou, “An Efficient Real-Time Video Object Segmentation Algorithm Based On Change Detection And Background Updating”, IEEE 2006 International Conference on Image Processing (ICIP), Atlanta, Ga., USA, Oct. 8-11, 2006. (pp. 1837-1840)

[0034] FIG. 1 is a flow chart illustrating the video object segmentation method according to a preferred embodiment of the present invention. As shown in FIG. 1, the video object segmentation method comprises:

[0035] Step 101 :Detect a change of state between a current frame and a previous frame.

[0036] Step 103:

[0037] Categorize the current frame as a moving region, a still region, a background region, or an uncovered background region according to the detecting result from the step 101.

[0038] Step 105:

[0039] Determine a moving object in the current frame according to the sorting result from the step 103.

[0040] Step 107:

[0041] Update a background component in the frame.

[0042] Detailed steps of the above will be further discussed below.

[0043] FIG. 2(a) is a flow chart illustrating further detail of step 101 as illustrated in FIG. 1. As shown in FIG. 2, step 101 further comprises:

[0044] Step 201:

[0045] Histogram analysis.

[0046] Step 203

[0047] Parameter estimation.

[0048] Step 205

[0049] Pixel classification.

[0050] Step 201 utilizes histograms to provide the distribution of the image gray levels or color (not illustrated). As shown in FIG. 2(b), the horizontal axis is gray level value, and the vertical axis indicates the number of pixels corresponding to different gray values. By this method, the gray level having the most pixels can be found.

[0051] Normally, differences between images comprise static backgrounds and moving objects, with moving objects always generating larger intensity changes. Otherwise, subjects with weaker intensities always correspond to background regions as the static part. As shown in FIG. 2(b), most part of the gray level distribution centers on gray level 0, therefore these gray levels are the most suitable ones for indicating the background region in the difference value of frames. Thus a gray level with the greatest pixel numbers in a gray level range r is selected as a reference for the following parameter estimation. The histogram of difference value image DI can be shown as Equation 1, where p indicates the gray level value of pixels:

His(p)=Number{(i,j)|DI(i,j)=p,(i,j)∈ DI}  Equation 1

[0052] If the gray level with the maximum number of pixels p′ in the range of r, then the number of which is His(p′). If the gray level number is the same, then the larger gray level is selected. In this embodiment, the preferred value of parameter r is 100.

[0053] The main purpose of step 203 is to estimate a background model from the difference value of the image. In this case, if the difference value of the static background region is caused by camera noise, then the distribution can be regarded as Gaussian distribution. If the pixel of the background in the difference value image is a random variable d, the probability density function of Gaussian distribution can be shown as Equation 2:

$\begin{array}{cc}p\left(d\right)=\frac{1}{\sqrt{2{\mathrm{\pi \sigma }}_b^2}}{}^{\left(\frac{{\left(d-u_b\right)}^2}{2{\sigma }_b^2}\right)}& \mathrm{Equation}\left(2\right)\end{array}$

[0054] wherein μ_b and μσ_b² respectively represent the mean and variance of the background region.

[0055] Next, when estimating mean and variance of the background region. First, the location of pixels having gray level p′ in the difference value image is found. An estimation window w with size N is used for consideration. After that, the mean u and standard deviation std, as shown in Equation (3) is computed.

$\begin{array}{cc}{\mu }_{w_i}\left(p^{\prime }\right)=\frac{1}{N}\sum _{j=1}^Nw_i\left(j\right){\mathrm{std}}_{w_i}\left(p^{\prime }\right)=\sqrt{\frac{1}{N}\sum _{j=1}^N{\left[w_i\left(j\right)-{\mu }_{w_i}\left(p^{\prime }\right)\right]}^2}\mathrm{for}i=1,2\dots \mathrm{His}\left(p^{\prime }\right)& \mathrm{Equation}\left(3\right)\end{array}$

[0056] Since p′ and nearby pixels are considered, the accuracy of the estimation references is higher. Also. In an embodiment of the present invention, the size of estimation window is 3×3. After, the estimated mean and variance are equalized to indicate the parameter of background, as shown in Equation 4:

$\begin{array}{cc}{\mu }_b=\frac{1}{\mathrm{his}\left(p^{\prime }\right)}\sum _{i=1}^{\mathrm{his}\left(p^{\prime }\right)}{\mu }_{w_i}\left(p^{\prime }\right){\sigma }_b=\frac{1}{\mathrm{his}\left(p^{\prime }\right)}\sum _{i=1}^{\mathrm{his}\left(p^{\prime }\right)}{\mathrm{std}}_{w_i}\left(p^{\prime }\right)\mathrm{for}i=1,2\dots \mathrm{His}\left(p^{\prime }\right)& \mathrm{Equation}\left(4\right)\end{array}$

[0057] Step 205 utilizes the background variance σ_b obtained by above-mentioned parameter estimation, and multiplies a constant c to indicate the threshold value Th (=cσ_b). Next, if the difference value between the pixel value and the background mean value μ_b of the difference value image is larger than this threshold value, than the pixel is classified as a foreground part (that is, gray level value 255), otherwise it is background part (that is, the grey level 0), which can be shown as Equation (5):

If (|DI(i,j)−μ_b|>Th) foreground pixel else background pixel   Equation (5)

[0058] Wherein the constant c is used for controlling the intensity of pixel classification, and noise may be incorrectly determined to the foreground while c is small. Otherwise, the increasing of c may regard the object component with smaller object moving amount as background. Therefore the constant c is mainly determined by the level of change of the frame concept. C needs a higher value if the concept has a larger change. Otherwise, c has a smaller value if c has smaller change. In this embodiment, the better value of constant c is located between 0 to 10. The main purpose of step 105 is to combine the frame difference mask, which can change the detection result, and the background subtraction mask is for detecting the situation of object region.

[0059] FIG. 3(a) is a schematic diagram illustrating detail of the step 103 illustrated in FIG. 1. As shown in FIG. 3(a), four regions are generated while the object 300 moves from left to the right. Four regions comprise: background region 301, still region 303, uncovered background region 305 and moving region 307.

[0060] The issue that the video segmentation method based on change detection always has is that the still region 303 is incorrectly determined to the background region 301, and the uncovered background region 305 is wrongly determined to part of the moving object 307. Though some prior art is disclosed to solve such problems, the computing amount is cumbersome since motion estimation is used to detect the uncovered background region.

[0061] Therefore, the present invention utilizes a frame difference mask and background subtraction mask to detect every region, as shown in FIG. 3(b). In FIG. 3(b), the ON state indicates the pixel is determined to “changed”, and OFF indicates “unchanged”. After that, the detected background region 301 and uncovered region 305 can be deleted from the mask, thus the uncovered background region only outputs moving region 307 and still region 303 for following processes.

[0062] The main purpose of step 107 is to establish reliable background information from inputted frames, such that the above-mentioned uncovered background region step can be correctly performed, and allow the segmentation region of the object to be more complete. For example, if background updating is not used, then the original moving object region will be wrongly determined for segmentation if the original moving object in the frame moves in few frames. Alternatively, an object can be regarded as part of the background region by viewers if an original moving object becomes a still object for a period of time. Thus the method for updating background is necessary, or else the still object will be regarded as a moving object and become segmented.

[0063] Many prior art background updating algorithms manage to establish a background frame, which utilizes the difference value between the frames to be as the input of a Kalman filter to estimate background updating information. Such methods should update the Kalmen gaining matrix. However, this requires a huge computer processing amount. Also, such a method cannot process the first frame if the moving object appears in video movies, thus it is hard to employ in practical application.

[0064] In the background updating method of the present invention, reliable background information is necessary to allow for segmentation of moving objects to fit an actual situation.The utilized operator should be simplified to increase processing speed. The idea of which, is since the object keeping still for a period of time does not change, it should be updated into the background. Thus an accumulator is used to record the still level of moving. The still level of the moving object is regarded as high if the accumulated amount of the accumulator is high, otherwise it is low. Then, if the accumulated value is higher than the predetermined threshold value, then the corresponding pixels of the present frame are updated to background image.

[0065] FIG. 4 is a flow chart illustrating the detail of step 107 illustrated in FIG. 1, which comprises:

[0066] Step 401:

[0067] Remove non-still regions.

[0068] Step 403:

[0069] Detect an absolute background.

[0070] Step 405:

[0071] Update selector.

[0072] Step 407:

[0073] Update the background.

[0074] Step 409:

[0075] Buffer the background.

[0076] The above-mentioned steps are further described as below. The background updating mechanism of the present invention uses the static region mask of the object region detecting result as an input. Normally, still regions not only comprise real still regions, but also other regions, including the small region noise from outside interference, or the frame difference value inside the moving object. Such regions are regarded as still regions while object region detecting is performed, but the regions are part of the moving regions, in fact.

[0077] Therefore, step 401 should be performed to remove actual still regions from still regions, such that only the actual (real) still regions can enter the background upgrading process. Normally, the still region will connect the moving region generating region if the still region belongs to part of the moving region, but the real still region doesn't. Therefore, region adjacent graphics can be used to determine if there is any moving region nearby the still region. If yes, then the still region is removed and real still regions are left.

[0078] Step 401 will fail if the first frame has a moving object. Since the region generated from the moving object leaves the original location, the connected region will be regarded as a moving object and removed. Therefore, no real still region is transmitted to the background updating module. Step 403 is therefore used first to determine if the first frame of inputted video comprises moving objects or not. If not, then step 401 is performed rather than step 405. step 405 is performed without going through step 401. Step 403 utilizes higher-order-statistics to determine if the changing region of the first frame and the n-th frame is larger than the predetermined threshold value.If so, then it means a moving object appears in the first frame, otherwise, it means no moving object appears in the first frame.

[0079] Equation (6) refers to a mean absolute background determination computing of four-order moments, and Equation (7) refers to an updating selector of region selection.

$\begin{array}{cc}{\mu }_{1,n}=\frac{1}{W\times H}\sum _{i\in H}\sum _{j\in W}f_n\left(i,j\right)-f_1\left(i,j\right)\rho =\sum _{i\in H}\sum _{j\in W}{\left(d^{\prime }\left(i,j\right)-u_{1,n}\right)}^4& \mathrm{Equation}\left(6\right)\\ \mathrm{RS}=\mathrm{sr},\mathrm{if}\rho >\mathrm{Sth}\mathrm{NSRE}\left(\mathrm{sr}\right),\mathrm{otherwise}& \mathrm{Equation}\left(7\right)\end{array}$

[0080] In which the W and H indicates the size of frame and d′(i, j) indicates the difference value between the first frame and the n-th frame. In the present invention, Sth is set to 150, and n is determined by the change of frame.

[0081] According to the preferred embodiment of the present invention, the change of RSt in a period of time should be considered for updating of the background frame. Since the still object keeps still for a period of time, and belongs to part of the background. An accumulator is used for accumulating the pixel changing times for a period of time, and the pixels are updated to the background frame while the accumulated value reaches a predetermined threshold value. The process of background updating is shown as equation (8).

$\begin{array}{cc}{\mathrm{SM}}_t\left(i,j\right)=\{\begin{array}{cc}{\mathrm{SM}}_{t-1}\left(i,j\right)+1,& \mathrm{if}{\mathrm{RS}}_t\left(i,j\right)\mathrm{is}\mathrm{active}\mathrm{pixel}\\ 0,& \mathrm{otherwise}\end{array}\mathrm{if}\left(t=1\right){\mathrm{BI}}_t\left(i,j\right)=f_t\left(i,j\right)\mathrm{else}{\mathrm{BI}}_t\left(i,j\right)=\{\begin{array}{cc}{f}_t\left(i,j\right),& \mathrm{if}{\mathrm{SM}}_t\left(i,j\right)=F_{\mathrm{th}}\\ {\mathrm{BI}}_{i-1}\left(i,j\right),& \mathrm{otherwise}\end{array}& \left(8\right)\end{array}$

[0082] In equation (8), RS_t(i,j) indicates the outputting result for updating the selector at time t, and the pixels of which comprise two states of changing and unchanging. SM_t(i,j) is an accumulator for recording the pixel changing times in the current frame RS_t(i,j), and F_th is a predetermined threshold value. Therefore, the first frame is used originally as an initial background, and the initial value of the accumulator is set to 0. Then, the pixels of the current frame are updated to current background BI_t(i,j) if the accumulated times reach Fth, otherwise the pixels of the previous background BI_t-1(i,j) are updated to the current background.

[0083] Besides the above-mentioned steps, the present invention further provides an object outline improvement method for eliminating noise regions, such that the lacking or protruding parts of the object outline. This can be repaired, and accurate outline information can be obtained as a result. FIG. 5 is a flow chart illustrating the detailed steps of the object outline improvement method according to an embodiment of the present invention. The inputted data is the output from the above-mentioned video object segmentation method, that is, the union of the object region detecting results. As shown in FIG. 5, the method comprises:

[0084] Step 501:

[0085] Time unity compensation.

[0086] Step 503:

[0087] Hierarchical outline segmentation.

[0088] Step 505:

[0089] Post processing filter.

[0090] Since the moving object may move slowly or part of which is silent in a particular time period, the object mask may be incomplete after changing detection, such that the segmented object region has a defect issue. Since each frame has a high relation in the signal process, step 501 utilizes this idea to compensate for incomplete regions. Such a method is well-known by persons skilled in the art, and is omitted for brevity. In the preferred embodiment of the present invention, step 501 is not necessary to perform step 503, because Time unity compensation will decrease the quality of video object segmentation.

[0091] The step 503 segments the original object mask to blocks and uses each block as processing unit and determine if the edge block belongs to a moving block or not. If it does then it is retained, otherwise the block is removed. Afterwards, the block sizes are gradually reduced, and the above-mentioned operations are repeated until the object outline is stable. Finally, object edge improvement methods are utilized to obtain complete object outlines. Step 503 can comprise:

[0092] Step 502:

[0093] Segment and fill the object mask block.

[0094] Step 504:

[0095] Hierarchical outline segmentation.

[0096] Step 506 Amend object outline.

[0097] Detailed operations of each step will be further described as below.

[0098] Step 503 originally utilizes a 16×16 block. Therefore, step 502 is performed to block-segment the original object mask. Also, since the edge block is utilized for subject operation, the un-filled edge block should be filled again.

[0099] After block segmentation and filling, the object edge block should be processed. The function is shown as equation (9) to determine if it is a moving block or not. If yes, retain the block, otherwise remove it. Afterwards, the next layer edge block is checked until no moving block is found. Then, the above-mentioned steps are repeated until the block size reaches 2×2. FIG. 6 is a flow chart illustrating the detailed steps of the hierarchy block segmentation method according to the present invention. FIG. 6(a) indicates the original object mask block, FIG. 6(b) indicates 16×16 hierarchical segmentation, FIG. 6(c) indicates 8×8 hierarchical segmentation, FIG. 6(d) indicates 4×4 hierarchical segmentation, and FIG. 6(e) indicates 2×2 hierarchical segmentation.

If (VBi<V_{Diff(G(ft),G(ft-1)}+λ) the block is removed else the block is kept   Equation (9)

[0100] wherein the V_{Diff(G(ft),G(ft-1)} indicates the difference in variance of the adjacent frames and the gradient images, V_Bi indicates the variance of the i-th object edge block, and λ indicates a constant as shown in equation (10).

if (blocksize=16) λ=2 else if (blocksize=16) λ=1.5 else if (blocksize=4) λ=1 else if (blocksize=2) λ=0.5   equation (10)

[0101] Though better object outlines can be obtained after step 504, the effect of hierarchical outline segmentation may still be remain however The inner of object can thus be corrupted and background region may still exist surrounding the object edge. Thus, step 506 is needed to improve these defects.

[0102] According to a preferred embodiment of the present invention, original object mask and hierarchical outline segmentation gradient changing are used to eliminate the effect of hierarchical outline segmentation. If OM_i and OM_h are assumed to respectively indicate the original object mask and the mask after hierarchical outline segmentation, and G_i and G_h are assumed to indicate the mask corresponding to original frames, then the gradient frame after type gradient operation can be shown as FIG. 7(a) and FIG. 7(b). FIG. 7(a) indicates the gradient frame of the original object mask, and FIG. 7(b) indicates the gradient frame of the hierarchical segmentation mask.

[0103] After that, the following steps are performed:

[0104] Step1: Eliminate the gradient pixel of the most outside layer of G_i and G_h, such that the pixel gradient of the most outside layer will not be considered in the following steps.

[0105] Step2: Find different regions from OM_i and OM_h and utilize R_i.h to indicate the union of the pixels in this region, as shown in equation (11).

R_i,h={( )m_r|( )m_r∩( )m_j=( )∀( )m_r ∈( )M_i,( )m_j ∈( )M_h}  Equation 11

[0106] Step3: Find pixels with pixel gradients larger than threshold value φ from R_i.h, such as equation (12), wherein p indicates the union of the pixels larger than the threshold value. Finally, compensate the pixels to the frames affected by the hierarchical outline segmentation effect.

P={G_i(p′)>φ|p′∈R_i,h}  Equation 12

[0107] Background always exists at the outline surrounding of the segmented objects, however, the compression ratio based on object compression may be decreased. Therefore, the present invention further utilizes region growing methods to eliminate the background part of the object outline surrounding, to make the object outline more accurate. Finally, the post processing filter can be used for performing the final process of the outline improvement technique to eliminate noise regions and perform type smoothing procession. Since the edge region growing and post processing filter are well-known by persons skilled in the art, it is omitted for brevity.

[0108] FIG. 8 is a flow chart illustrating video object segmentation method according to a preferred embodiment of the present invention, which comprises:

[0109] Step 801:

[0110] Guassian smooth processing.

[0111] Step 803:

[0112] Frame buffering.

[0113] Step 805:

[0114] Frame difference.

[0115] Step 807:

[0116] Change detection.

[0117] Step 809:

[0118] Object region detection.

[0119] Step 811:

[0120] Object outline arrangement.

[0121] Step 813:

[0122] Background buffering.

[0123] Step 815:

[0124] Background subtracting.

[0125] Step 817:

[0126] Change detection.

[0127] Step 819:

[0128] Remove non-static region:

[0129] Step 821:

[0130] Update background.

[0131] Step 823:

[0132] Detect absolute background.

[0133] Step 825:

[0134] Update selector.

[0135] Those skilled in the art will readily observe that numerous modifications and alterations of the device and method may be made while retaining the teachings of the invention. Accordingly, the above disclosure should be construed as limited only by the metes and bounds of the appended claims.