Shadow and highlight detection system and method of the same in surveillance camera and recording medium thereof

[0001] This application claims the benefit of Chinese Patent Application No. 200710151767.4, filed on Sep. 29, 2007, in the Chinese States Intellectual Property Office, and the benefit of Korean Patent Application No. 10-2007-0135842, filed on Dec. 21, 2007, in the Korean Intellectual Property Office, the disclosures of which are incorporated herein by reference.

[0002] 1. Field

[0003] One or more embodiments of the present invention relate to a method and system for detecting a shadow and a highlight, and a recording medium thereof, and more particularly, to a method and system capable of detecting a shadow and a highlight in a surveillance camera, and recording medium thereof.

[0004] 2. Description of the Related Art

[0005] Surveillance cameras are widely used in public places for security purposes. However, additional labor is required to inspect and determine whether a critical incident occurred or is occurring. Therefore, a surveillance system with an intelligent camera that can detect, track, and analyze actions of a human being and other objects is a focus of much research.

[0006] Detecting and tracking of a moving object is at the core of many applications that process image sequences. The applications generally attempt to identify an appearance of the moving object and a shadow that moves along with the moving object in a scene. When a foreground is misclassified as a shadow point and thereby the moving object is separated and extracted, the shadow may cause a serious problem. Specifically, due to the shadow cast over another object, the merging of objects, shape distortion of the objects, and loss of the objects may occur.

[0007] Since a shadow and a related object generally share two important visual features outlined below, detection of the shadow may be difficult. First, the shadow point is significantly different from a background and thus is detectable as the foreground. Second, the shadow has the same motion as its corresponding object. For this reason, identifying the shadow is critical for still images and image sequences, for example, a video. Currently, the identification of shadows has become an active research area.

[0008] Generally, there are three types of schemes for identifying a shadow region and an object region.

[0009] A first scheme is based on the assumption that a system is aware of each of a light source, an object shape, and a ground plane. Therefore, when there is a combined region of shadow and object, the first scheme may predict which region is the object and which region is the shadow cast by the object.

[0010] A second scheme is based on image constancy. Color constancy indicates that a luminance is reduced in a shadow region to thereby decrease the intensity of a corresponding pixel. However, there is a relatively small change in the hue and saturation of the pixel. Therefore, the second scheme tests hue and saturation information to identify the shadow and the object. Also, texture constancy indicates that even though the intensity is reduced in the shadow region, an edge of the shadow region does not move. However, in the case of a foreground object, both edge and intensity are changed.

[0011] A third scheme is based on the edge width. The shadow region includes a complete shadow region and a partial shadow region. The partial shadow region corresponds to a narrow ribbon between the complete shadow region and the background. Also, the partial shadow region corresponds to a portion that gradually changes from the background to the complete shadow region. Therefore, the edge of the shadow region is wide and smooth, whereas the edge of the object region is narrow and sharp.

[0012] One or more embodiments of the present invention provide a method and system that can identify a shadow region and a highlight region from a foreground region in a surveillance camera system, without restrictions such as assumptions about a clean background, a texture background, a light source location, and the like, and a recording medium thereof.

[0013] Additional aspects and/or advantages will be set forth in part in the description which follows and, in part, will be apparent from the description, or may be learned by practice, of the invention.

[0014] According to an aspect of the present invention, a system for detecting a shadow region and a highlight region is provided The system includes an image capturing unit to capture a new image, a background model unit to receive the new image and update a stored background model with the new image, a difference image obtaining unit to compare the new image with the background model and to obtain a difference image between the new image and the background model, a penumbra region extraction unit to extract a partial shadow region or a partial highlight region by measuring a sharpness of an edge of the difference image and expanding a background region, and an umbra region extraction unit to extract a complete shadow region or a complete highlight region based on the result of the extraction.

[0015] According to another aspect of the present invention, a method of detecting a shadow region and a highlight region is provided. The method includes capturing a new image, comparing the new image with a background model and updating the background model with the new image, obtaining a difference image between the new image and the background image, extracting a partial shadow region or a partial highlight region by measuring a sharpness of a edge of the difference image and expanding a background region, and extracting a complete shadow region or a complete highlight region based on the result of the extraction.

[0016] In an aspect of the present invention, the present invention provides at least one computer medium recording the method of detecting the shadow region and the highlight region.

[0029] Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the like elements throughout. Embodiments are described below to explain the present invention by referring to the figures.

[0030] FIG. 1 is a block diagram illustrating a shadow detection system 111 according to an exemplary embodiment of the present invention. One or more embodiments of the present invention describe detection of a shadow region, and embodiments may also be applicable to detection of a highlight region.

[0031] Referring to FIG. 1, the shadow detection system 111 includes, for example: an image capturing unit 110 to capture a new image and input the new image in a background model unit 120; the background model unit 120 to include the background model and update the background model with the new image; a difference image obtaining unit 100 to compare the new image with the background model and to obtain a difference image between the new image and the background model; a penumbra region extraction unit 200 to extract a partial shadow region; and an umbra region extraction unit 300 to extract a complete shadow region.

[0032] According to an aspect of the present invention, the penumbra region extraction unit 200 may extract a partial highlight region. Also, the umbra region extraction unit 300 may extract a complete highlight region.

[0033] The image capturing unit 110 captures the new image in the shadow detection system. The new image is input in the background model unit 120 and is compared with the background model to obtain the difference image.

[0034] The background model is a statistical model about the history of a scene. In an exemplary embodiment, the background model is only a reference image of a background without any foreground object. In another implementation, the background model is a collection of pixel distributions. In a Gaussian Mixture model, each pixel distribution is given by Equation 1 below:
Σw_i·N(u_i,σ_i)  Equation 1

[0035] where w_i is a weight of each unit Gaussian, and N(u_i, σ_i) is a unit Gaussian distribution with a center u_i and a variance σ_i.

[0036] The background model unit 120 updates the background model. In order to update the mixture Gaussian background model, there is a need to compare a pixel of the new image (hereinafter, referred to as a new pixel) with each unit Gaussian distribution. When the new pixel belongs to a unit Gaussian, the background model unit 120 updates a center, a variance, and a weight of the unit Gaussian. The background model unit 120 updates the center of the unit Gaussian by assigning a weight to an original center of the unit Gaussian and a color of the pixel and obtains the average thereof. The background model unit 120 further updates the variance by obtaining the average of an original variance of the unit Gaussian and a distance from the new pixel to the original center of the unit Gaussian, and updates the weight of the unit Gaussian by increasing the weight of the unit Gaussian and decreasing weights of the other unit Gaussians.

[0037] Conversely, when the pixel is excluded from all unit Gaussians, the background model unit 120 deletes a unit Gaussian with a smallest weight and adds a new unit Gaussian centered at the pixel.

[0038] Based on the assumption that the background occupies a pixel for a relatively long period of time, when a pixel belongs to a unit Gaussian with a great weight, the pixel belongs to the background. Otherwise, the pixel belongs to a foreground, a shadow region, or a highlight region. Embodiments of the present invention may identify which type of region each changed pixel belongs to.

[0039] In the case of the highlight region, the situation is similar. As shown in FIG. 11, the border of the highlight region changes gradually. The only difference is that the highlight region is brighter than the original background, whereas the shadow region is darker than the background. Based on this, the shadow detection system detects the shadow and the highlight using the same scheme. In a latter part of the present specification, embodiments of the present invention will be described using the shadow region as an example. However, as described above, the highlight region also may be detected using the same scheme.

[0040] Hereinafter, a partial shadow region phenomenon will be described prior to describing the shadow detection. FIG. 2 illustrates arrangement of a partial shadow region and a complete shadow region as described in embodiments of the present invention.

[0041] Generally, a light source is not a point of light. When an object blocks all the light radiated from the light source, an occluded region becomes a complete shadow region. In the complete shadow region, pixels become darker at nearly the same ratio. As shown in FIG. 2, when the light source is partially blocked by the object, a partially lit region corresponding to a right-hand side of the complete shadow region becomes a partial shadow region. In the partial shadow region, the occluded light gradually increases. Therefore, the difference of darkness of pixels between the complete shadow region and the partial shadow region gradually changes from zero near the complete shadow region to a maximum value at a point in the partial shadow region furthest from the complete shadow region.

[0042] The difference image obtaining unit 100 obtains the difference image between the new image and the background image and stores the difference image. The difference image obtaining unit 100 compares a pixel of the new image with a center of each unit Gaussian of the background model, and finds a closest unit Gaussian of the background region. Next, the difference image obtaining unit 100 determines whether a distance from the closest unit Gaussian of the background region is less than a variance of the closest unit Gaussian of the background region.

[0043] When the distance is less than the variance, the difference image obtaining unit 100 labels the pixel as background and sets a corresponding difference of the difference image as zero. Conversely, when the distance is greater than or equal to the variance, the difference image obtaining unit 100 stores the distance as the corresponding difference in the difference image.

[0044] FIG. 3 illustrates a method of computing the difference image. A space is a color space 101 such as a red-green-blue (RGB) space. A first region is a unit Gaussian 102 of a foreground region or a shadow region. A second region is a unit Gaussian 103 of a background region and a point is an input pixel 104. The input pixel 104 is outside of unit Gaussians 103 of the background region. A distance from the input pixel 104 to the closest unit Gaussian 103 of the background region is stored in the difference image. Another input pixel 105 is within the range of the unit Gaussian 103 of the background region and thus is labeled as the background. In this manner, the difference image obtaining unit 100 stores the difference image.

[0045] FIG. 4 illustrates a structure of the penumbra region extraction unit 200 of FIG. 1.

[0046] The penumbra region extraction unit 200 includes, for example, a sharpness measurement unit 210 and a background expansion unit 220. The sharpness measurement unit 210 measures a sharpness of the edge of the difference image. When the sharpness is relatively large, it indicates that the difference between adjacent pixels in the edge is also relatively large. The background expansion unit 220 expands the background region through a horizontal and vertical scan. When a foreground pixel is reachable from the background without crossing any sharp edge, the foreground pixel is labeled as the partial shadow region.

[0047] Generally, the partial shadow region has a relatively wider and smoother edge, whereas the border of the object has a relatively narrow and sharp edge. According to an aspect of the present invention, the sharpness measurement unit 210 is used to measure the sharpness of the edge. For a pixel p, the sharpness measurement unit 210 computes gradients of the pixel in a small scale and a large scale, and obtains gradient g1 of the small scale and gradient g2 of the large scale. By dividing g1 by g2, the sharpness of the pixel p is set to sharp(p)=g1/g2. For example, FIG. 5 illustrates a smooth edge and FIG. 6 illustrates a sharp edge. Although the absolute gradient of FIG. 5 is similar to the absolute gradient of FIG. 6, the total gradient g2, according to a measurement scheme of an embodiment of the present invention, is relatively small and thus the sharpness, sharp(p)=g1/g2, is relatively large.

[0048] In order to compute the sharpness, the sharpness measurement unit 210 uses a filter shown in FIG. 7. A horizontal filter 710 and a vertical filter 720 are used to compute the sharpness. Initially, vi for calculating gradient(g1, g2), as suggested in Equation 2 below, is computed by summing up grey values of pixels in region i. Here, i indicates regions 1, 2, 3, and 4 shown in FIG. 7. S denotes the pixel p. Next, the sharpness using gradients(g1, g2) is computed by Equation 2 below. A is a constant used to control a small gradient. Also, if |v2−v3|<thr_grad, it indicates the gradient at the pixel p is very small. In this instance, the sharpness of the pixel p is set as 0. The threshold thr_grad is a predetermined system parameter. The sharpness is an absolute value and is computed by Equation 2 below:
Sharp(p)=|v2−v3|/(|v1−v4|+A)
where g1=|v2−v3|,g2=|v1v4|+A.  Equation 2

[0049] The background expansion unit 220 extracts the partial shadow region by applying the horizontal and vertical scan. For each horizontal scan line, the background expansion unit 220 finds every background region and expands the background region towards a left-hand side direction. When it is assumed that a left-hand side pixel of the background region is p_L, and if sharp(p_L)<thr_sharp(Shadow) and diff(p_L)<thr_diff(Shadow), the pixel is labeled as the partial shadow region. In this instance, thr_sharp(Shadow) is a sharpness threshold of the shadow and thr_diff(Shadow) is a difference threshold of the shadow. Also, if sharp(p_L)<thr_sharp(Highlight) and diff(p_L)>thr_diff(Highlight), the pixel is labeled as the partial highlight region. In this instance, thr_sharp(Highlight) is a sharpness threshold of the highlight and thr_diff(Highlight) is a difference threshold of the highlight.

[0050] Here, diff(p_L) is a difference, absolute value, between the pixel p_L and the background model and is a difference between a current pixel value and a middle point in the background model in a corresponding pixel location. This is stored in the difference image. The sharpness threshold thr_sharp and the difference threshold thr_diff are two system parameters.

[0051] The above operation is horizontally repeated until the above conditions are not satisfied. When the expanding operation to a left-hand side direction is completed, the expanding operation is performed to a right-hand side direction in the same way.

[0052] After the horizontal scan, most of the partial shadow region is detected. However, due to image noise, some sharp edges may block the horizontal expansion. Therefore, each vertical line is scanned in a similar way. If the sharpness of a nearest upper pixel p in the background region is less than thr_sharp(Shadow) and diff(p)<thr_diff(Shadow), the pixel is labeled as a partial shadow region. Also, when the sharpness is less than thr_sharp(Highlight) and diff(p)>thr_diff(Highlight), the pixel is labeled as a partial highlight region. The above operation is repeated until the above conditions are not satisfied and the expanding operation is performed with respect to a lower direction in the same way. In this instance, diff(p) is a difference between the pixel p and the background model. The sharpness threshold thr_sharp and the difference threshold thr_diff are two system parameters.

[0053] After completing the operation, the partial shadow region is extracted using the sharpness measurement unit 210 and the background expansion unit 220.

[0054] The umbra region extraction unit 300 extracts a completed shadow using a spatial constraint. FIGS. 8 and 9 illustrate two typical situations.

[0055] FIG. 8 illustrates locating a complete shadow region, according to an extraction result of a partial shadow region.

[0056] Referring to FIG. 8, according to an aspect of the present invention, when a predetermined region is surrounded by a partial shadow region, the entire region is labeled as a complete shadow region. In FIG. 8, a region 802 is the complete shadow region and another region 803 is the partial shadow region. Specifically, since the region 802 is surrounded by the partial shadow region 803, the region 802 is labeled as the complete shadow region.

[0057] FIG. 9 illustrates locating a complete shadow region, according to an extraction result of a partial shadow region, when the complete shadow region approaches a foreground object.

[0058] Referring to FIG. 9, a region 901 is an object region, another region 902 is the complete shadow region, and still another region 903 is a partial shadow region. According to an aspect of the preset invention, when a part of a foreground region is surrounded by a partial shadow region, the part also must be labeled as the complete shadow region, like the region 902. Embodiments of the present invention apply a horizontal and vertical scan to clarify whether the part of the foreground is surrounded by the partial shadow region.

[0059] When a scanned line region includes a partial shadow region-object region-partial shadow region pattern and the length of the object region is not too long, that is, when the length of an object corresponding to a system parameter is less than Thr_length(Shadow), the object region is labeled as the complete shadow region. Also, when the scanned line region includes a partial highlight region-object region-partial highlight region pattern and the length of the object region is not too long, that is, when the length of the object region that is the system parameter is less than Thr_length(highlight), the object region is labeled as the complete highlight region.

[0060] FIG. 10 illustrates a method of identifying a shadow region and a highlight region from a foreground region, according to an exemplary embodiment of the present invention.

[0061] In operation S1010, the image capturing unit 110 captures a new image. In operation S1020, the background model unit 120 compares the new image with a background model, and updates the background model with the new image. In operation S1030, the difference image obtaining unit 100 obtains a difference image between the new image and the background model. In operation S1040, the penumbra region extraction unit 200 measures a sharpness of an edge in the difference image, extracts a background image, and thereby extracts a penumbra region, which is a partial shadow region or partial highlight region. In operation S1050, the umbra region extraction unit 300 extracts an umbra region, which is a complete shadow region or a complete highlight region, based on the result of the extraction by the penumbra region extraction unit 200.

[0062] FIG. 11, images A, B, C, and D, illustrate two experimental results. One is a highlight extraction result and another is a shadow extraction result.

[0063] In FIG. 11, two experimental results are shown. From left to right, there are original images A, difference images B, sharpness images C, and object and shadow/highlight extraction results D, respectively.

[0064] The top images show the highlight extraction result. In this instance, a road is illuminated by a headlight of a vehicle. The bottom images show an indoor scene. Here, a background light was relatively bright and made a large shadow region on the ground. However, it can be seen that the border of the light region is very smooth. When using a sharpness measurement scheme according to embodiments of the present invention, the sharpness of the border is less than the border of the general object. Therefore, the shadow region is successfully extracted.

[0065] In addition to the above described embodiments, embodiments of the present invention can also be implemented through computer readable code/instructions in/on a medium, e.g., a computer readable medium, to control at least one processing element to implement any above described embodiment. The medium can correspond to any medium/media permitting the storing and/or transmission of the computer readable code.

[0066] The computer readable code can be recorded/transferred on a medium in a variety of ways, with examples of the medium including recording media, such as magnetic storage media (e.g., ROM, floppy disks, hard disks, etc.) and optical recording media (e.g., CD-ROMs, DVDs, Blu Ray disks), transmission media, and elements of the Internet, for example. Thus, the medium may be such a defined and measurable structure including or carrying a signal or information, such as a device carrying a bitstream, for example, according to embodiments of the present invention. The media may also be a distributed network, so that the computer readable code is stored/transferred and executed in a distributed fashion. Still further, as only an example, the processing element could include a processor or a computer processor, and processing elements may be distributed and/or included in a single device.

[0067] According to one or more embodiments of the present invention, a surveillance camera system may identify a shadow region from a foreground region, without restrictions such as a clean background, a texture background, assumptions about a light source location, and the like.

[0068] Although a few embodiments have been shown and described, it would be appreciated by those skilled in the art that changes may be made in these embodiments without departing from the principles and spirit of the invention, the scope of which is defined in the claims and their equivalents.