Device for and method of automatically tracking a moving object

[0001] This application claims the benefit of Korean Patent Application No. 2001-47410, filed Aug. 7, 2001, in the Korean Industrial Property Office, the disclosure of which is incorporated herein by reference.

[0002] 1. Field of the Invention

[0003] The present invention generally relates to a device for and a method of tracking a moving object, and more particularly, to a device for and a method of automatically detecting a moving object through signal processing on input images and continuously tracking the moving object until the moving object escapes out of a monitoring area.

[0004] 2. Description of the Related Art

[0005] There are various types of tracking systems worldwide for automatically detecting and tracking a moving object based on input image information.

[0006] The following are a few representative examples of tracking methods applied in such tracking systems that track the moving object through the input image information.

• • [0007] 1. Tracking based on a difference signal between two frames
  • [0008] 2. Tracking based on a correlation (block matching algorithm)
  • [0009] 3. Tracking based on a disturbance map
  • [0010] 4. Tracking based on a color distribution
  • [0011] 5. Tracking based on an optical flow

[0012] Despite some advantages, most of the currently available tracking methods including the above are difficult to commercialize due to problems especially in terms of speed and accuracy.

[0013] The Korean Patent Publication No. 10-20010000107 published Jan. 5, 2001 discloses a multi moving object tracking and monitoring system, which is a combined form of various tracking methods.

[0014] The multi moving object tracking and monitoring system mainly includes a general monitoring device and a local monitoring device. The general monitoring device has a fixed camera monitoring a wide area, and registers and tracks all the moving object within a predetermined monitoring range. For a certain moving object among the registered moving objects, the general monitoring device transmits a tracking command to the local monitoring device to track the certain moving object, and thus, the local monitoring device keeps tracking the certain moving object until the certain moving object escapes out of the monitoring range.

[0015] FIG. 1 is a block diagram showing a local monitoring device of a multi moving object tracking and monitoring system.

[0016] A local monitoring device 100 includes a camera 10, an image input unit 20 receiving input signal corresponding to images from the camera 10, an image compensation unit 30 eliminating a noise factor from the input image signal, a motion detecting unit 40 detecting a movement of the moving object based on a disturbance map from the compensated input image signal, a moving object extracting unit 50 separating the moving object from the input image signal, a moving object tracking unit 60 tracking the moving object based on extracted position and speed information of the moving object, and a camera movement compensation unit 70 compensating for the input image signal in accordance with a movement of the camera 10.

[0017] An operation of the local monitoring device 100 constructed as above will be described below with reference to a flowchart of FIG. 2.

[0018] First, when the images taken by the camera 10 are input to the image input unit 20 in operation S1, the image compensate unit 30 filters the input image signal to eliminate noise factors therefrom in operation S2, and the camera movement compensation unit 70 compensates for the input image signal according to the movement of the camera 10 in operation S3. Next, with respect to the compensated input image signal, the motion detecting unit 40 detects the movement of the moving object through the disturbance map algorithm in operation S4. The moving object extracting unit 50 extracts an actual image of the moving object through division and composition of the images in a motion area extracted through a disturbance map in operation S5. Then, the moving object extracting unit 50 estimates a next location of the moving object by applying the extracted location and speed information of the moving object to an estimating algorithm that uses a Kalman filter, and then tracks the moving object in operation S6. When the moving object stops and thus there is no further movement, or when it becomes hard to track the moving object because many objects crossed one another, the tracking is continued by using a block matching process in operation S7. As the movement of the moving object is detected, the camera 10 is moved in accordance with the movement of the moving object in operation S8. After that, the tracking of the moving object includes repeating the above processes.

[0019] In this multi moving object tracking and monitoring system, many algorithms are used, and among them, the disturbance map algorithm is representative example that is used to automatically acquire or extract the information about the moving object. For the purpose of the tracking of the moving object, a block matching algorithm is used.

[0020] In a conventional moving object tracking system, an extracting performance based on a disturbance map algorithm is satisfactory when a camera is in a static state. However, when the camera is in motion, image compensation is inevitably required. For this purpose, the conventional art uses a camera compensation algorithm, which results in an increased amount of data to be processed by the system and consequent overload in data processing.

[0021] A block matching algorithm, which has been used for tracking the moving object, also degrades a data processing speed because a size of a block increases as the moving object is moved, and because the size, shape or brightness changes.

[0022] Accordingly, it is an object of the present invention to provide an automatic moving object tracking device and a method thereof offering a new concept of tracking which does not require a compensation process for a movement of the camera while utilizing advantages of a disturbance map algorithm, i.e., excellent tracking performance on a moving object with a fixed camera. With an improved processing speed and tracking performance, which is respectively resulted from an acquisition of information about the moving object using a least number of image processes and a proper adaptation to a movement of the moving object and to the changes of a surrounding environment.

[0023] Additional objects and advantageous of the invention will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the invention.

[0024] The above and other objects are accomplished by providing a moving object tracking device according to an embodiment of the present invention. The moving object tracking device includes a photographing unit photographing a monitoring area, a binary disturbance image generating unit generating a binary disturbance image signal from an input image signal acquired through the photographing unit, a moving object acquiring unit acquiring from the binary disturbance image signal information about the moving object by using a moving window through use of an initial tracking window of a predetermined size and adjusting the size of the initial tracking window so that the binary disturbance image signal contains the moving object therein, a moving object automatic tracking unit estimating a location of the moving object in a following frame to which a center of the moving object is to be moved based on currently acquired information and previously acquired information about the center of the moving object, moving the center of the tracking window to the estimated location where the center of the moving object is estimated to be moved to, and acquiring information about an actual center of the moving object in the tracking window and the size of the tracking window, and a tracking status determining unit comparing actual information about the moving object with the estimated information about the moving object, and determining tracking status based on a resultant error range, the actual information about the moving object being acquired through the following frame.

[0025] The moving object tracking device includes an input image pre-processing unit improving a processing speed of the input image signal that is input through a photographing process, and then pre-processing the input image signal to eliminate noises from the input image signal.

[0026] The moving object tracking device includes a photographing control unit for moving the photographing direction of the photographing unit to the estimated direction to which the moving object is estimated to be moved, when the center of the tracking window is located outside of the predetermined range with respect to the entire input image area. The photographing control and the signal processing on the input image are both performed in the basis of time-division.

[0027] The above and other objects are also accomplished by providing a moving object tracking method according to another embodiment of the present invention. The moving object tracking method includes photographing a monitoring area, generating a binary disturbance image signal from an input image signal acquired through the photographing unit, acquiring from the binary disturbance image signal information about the moving object by using a moving window through use of an initial tracking window of a predetermined size, adjusting the size of the initial tracking window so that the binary disturbance image signal contains the moving object therein, estimating a location in a following frame to which a center of the moving object is to be moved based on currently acquired information and previously acquired information about the center of the moving object, moving the center of the tracking window to the estimated location where the center of the moving object is estimated to be moved to, acquiring information about an actual center of the moving object in the tracking window and the size of the tracking window, and comparing actual information about the moving object with the estimated information about the moving object, and determining tracking status based on a resultant error range, the actual information about the moving object being acquired through the following frame.

[0028] The moving object tracking method includes improving a speed of processing the image that is input through the photographing, and then pre-processing the input image to eliminate noises.

[0029] The moving object tracking method includes a photographing control operation of moving (changing) a photographing direction of the photographing unit to the estimated direction to which the moving object is estimated to be moved, when the center of the tracking window is located outside of the predetermined range with respect to the entire input image area. The photographing control and the signal process on the input image signal are both performed on a basis of time-division.

[0030] According to the moving object tracking device and method thereof in accordance with the present invention, even a smallest image in size is processed during tracking the moving object. Accordingly, the process speed for the image increases. Further, as the controlling of the photographing direction and the processing of the input image signal are performed separately on the time-division basis during the change of the photographing direction to track the moving object, a compensation for the movement of camera is not required, and thus, burden for a data processing operation is lessened.

[0044] Reference will now be made in detail to the present preferred embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the like elements throughout. The embodiments are described in order to explain the present invention by referring to the figures.

[0045] Hereinafter, the preferred embodiments of the present invention will be described in greater detail with reference to the accompanying drawings.

[0046] FIG. 3 is a block diagram of a moving object tracking device 200 according to an embodiment of the present invention.

[0047] The moving object tracking device 200 includes a photographing unit 110, an input image pre-processing (post-processing) unit 120, a binary disturbance image generating unit 130, a moving object acquiring unit 140, a moving object automatic tracking unit 150, a tracking status determining unit 160 and a photographing control unit 170.

[0048] The photographing unit 110 photographs a monitoring area and outputs an analogue image signal corresponding to an input image of the monitoring area.

[0049] The input image pre-processing unit 120 includes an analog-to-digital (A/D) converter 121 converting the analog image signal into a digital input image signal so as to acquire desired information, like a center value and a size of the moving object, from the entire input image signal within a shortest time period, a resolution adjusting unit 122 adjusting resolution of the input image signal, and a filter 123 filtering various noises from the input image having the adjusted resolution.

[0050] The binary disturbance image generating unit 130 transforms the input image signal into a binary disturbance image signal through a disturbance map algorithm so as to separate from a pre-processed input image a background image and a moving object image, respectively.

[0051] FIG. 4 is a detailed block diagram of the binary disturbance image generating unit 130. The binary disturbance image generating unit 130 includes a recursive filter 132 obtaining a disturbance field through a currently input image signal and an average of previously input image signals, and an image division unit 136 outputting a binary disturbance image signal with respect to each pixel upon comparing an absolute value of the disturbance field obtained by the recursive filter 132 with a first disturbance threshold. The binary disturbance image generating unit 130 further includes a headlight preventing unit 134 modifying a determination of a binary value of the binary disturbance image signal for each pixel so as to prevent particular surrounding factors, like a headlight signal of an automobile from causing an incorrect determination on the moving object.

[0052] The headlight preventing unit 134 includes a first comparator 134-1 and a second comparator 134-2 comparing the input image signal and the disturbance field of the current frame with a second disturbance threshold, respectively, and an OR gate 134-3 performing an OR-operation with respect to an output value from the first and second comparators 134-1, 134-2.

[0053] When the moving object acquiring unit 140 acquires the information about the moving object in a first tracking operation, or in a re-attempt after the first tracking operation has failed, the moving object acquiring unit 140 acquires the center value and the size of the moving object by using a tracking window. For this purpose, the moving object acquiring unit 140 includes a motion disturbance function (MDF) calculating unit 142 calculating a motion disturbance function (MDF) of each of the pixels in a predetermined range based on a moving window concept that uses an initial tracking window having a predetermined size, and a tracking window adjusting unit 141 that searches for a pixel having a greatest MDF and adjusts the size of the initial tracking window by calculating the center and the size of the moving object within an actual tracking window with reference to the pixel of the greatest MDF. The MDF calculating unit 142 adds binary values of coordinates of the initial tracking window from first to final coordinates and determines a sum of the coordinates as the MDF of the pixel serving as a center coordinate. The information about the center of the moving object is used for a fan/tilt control signal to control the photographing unit 110 while the information about the size of the moving object is used for a zoom-in/zoom-out control signal to control the photographing unit 110.

[0054] Based on current and previous acquired information, the moving object automatic tracking unit 150 estimates a next location in a following frame where the center of the moving object is to be moved, moves the center of the size-adjusted tracking window to the estimated next location, and acquires the information of the center of the moving object in the following frame in the size-adjusted tracking window and the information of the size of the size-adjusted tracking window. In other words, only the image signal corresponding to the moving object within the size-adjusted tracking window is processed in the following frame.

[0055] The tracking status determining unit 160 compares the information of an actual image of the moving object acquired from the following frame with the estimated information of the moving object to generate a resultant error range, and determines a tracking status of the moving object based on the resultant error range. Alternatively, the tracking status determining unit 160 can determines the tracking status by monitoring a number of binary disturbance image signal of pixels disposed in the size-adjusted tracking window. When an error of the tracking status is within a predetermined range, the moving object automatic tracking unit 150 continues tracking the moving object using exclusive signal processing on an area of the size-adjusted tracking window. When the error exceeds the predetermined range, the moving object acquiring unit 140 re-acquires information about the moving object of the entire input image and repeats automatic tracking processes.

[0056] The photographing control unit 170 controls a photographing direction of the photographing unit 110 in an estimated moving direction of the moving object when the center of the tracking window with respect to the entire input image is located outside the predetermined range. The photographing control unit 170 also controls zoom-in/zoom-out of the photographing unit 110 so that the size of the tracking window at the estimated pixel and the size of the moving object acquired from the following frame can be maintained at a certain rate. Both the photographing controlling like the fan/tilt control, the zoom-in/zoom-out control, and a movement of the photographing control unit 170, and the signal processing of the input image are performed on a time-division basis.

[0057] A method of tracking the moving object in the moving object tracking device 200 of FIGS. 3 and 4 will be described below with reference to a flowchart of FIG. 5.

[0058] First, a system acquires the analog image signal by photographing a monitoring area through the photographing unit 110 in operation S10. Then the system pre-processes the acquired image signal in operation S20 to generate the input image signal. The pre-processing in operation S20 includes adjusting the resolution of the input image signal so as to increase a signal processing speed in operation S21, and filtering the input image signal having the adjusted resolution so as to eliminate the noises from the input image signal in operation S22. After the pre-processing operation in operation S20, the input image signal (post-processed input image signal) is binarized (transformed) to the binary disturbance image signal though the disturbance map algorithm in operation S30.

[0059] The disturbance map algorithm generates a disturbance field Δ_t by using a difference of the currently input image I_t and the background image A_t (average image A_t−1 of previously input images) and detects a motion area of the moving object based on the disturbance field D_t. For this purpose, the background image A_t is generated when a historical weight ω is applied on the average image A_t−1 of the previously input images. The binary disturbance image signal and the background image are obtained with respect to the current frame by:

• • [0060] Mathematical expressions 1
    Δ_t=I_t−A_t−1
    A_t=I_t−K·LIM(I_t−A_t−1)
    where
  • [0061] Δ_t is a disturbance of a current frame,
  • [0062] A_t is either a background image or an average image of current frames,
  • [0063] A_t−1 is either a background image or an average image of previous frames,
  • [0064] I_t is a currently input image,
  • [0065] K is a gain, and
  • [0066] LIM is a limiter.

[0067] The disturbance map algorithm has a value ‘1’ in the motion area where the object is moving, and has a value ‘0’ in a motionless area. In other words, when the absolute value of the binary disturbance image signal for each pixel exceeds a predetermined threshold, it is determined as the motion area, and if the absolute value is below the predetermined threshold, it is determined as the motionless area.

[0068] Then each signal of the disturbance field of the binary disturbance image signal from the mathematical expressions 1 is image-divided based on first and second thresholds. The image division is performed such that the motion area has the value of ‘1’ while the motionless area has the value of ‘0’.

[0069] The transformed binary disturbance image signal is displayed in white in the motion area and in black in the background area (motionless area). Meanwhile, according to this embodiment of the present invention, the output from the headlight preventing unit 136 is used as a parameter that determines the binary values of the respective pixels so as to prevent the inaccurate (incorrect) determination with respect to the motion area due to the headlight from the automobile, etc.

[0070] When the input image signal is converted into the binary disturbance image signal, the moving object acquiring unit 140 acquires from the binary disturbance image signal the information about the center and the size of the moving object through the initial tracking window of a predetermined size in operation S40. In other words, the moving object acquiring unit 140 acquires the information about the center of the moving object and the size of the tracking window in the area of the entire image where the moving object is located. For the information about the moving object, MDFs of the respective pixels are calculated in the binary disturbance image signal based on a concept of a moving window in operation S42.

[0071] FIG. 6 shows the concept of the moving window according to the moving object tracking device 200 of FIG. 3 and the method of FIG. 5.

[0072] The moving window, which is formed of a predetermined size, has each pixel as a center coordinate and searches the MDF corresponding to each pixel of the input image. The sum of the binary values of pixels in the moving window is determined as a certain value, i.e., the MDF of the pixel located at the center coordinate.

[0073] The moving windows are identical to the initial tracking windows in size, and are generated in number less than a total number of the pixels. The sums obtained by adding up the binary values of the coordinates from the initial to the final coordinates within the respective initial tracking windows, are determined as the MDFs of respective moving windows and the respective pixels located at the center coordinates. Meanwhile, for a real-time processing operation, the searching of the MDF with respect to each pixel is performed in parallel.

[0074] The MDF of each pixel through the moving window is calculated by: $\begin{array}{cc}\mathrm{MDF}\left(p,q\right)=\sum _{x\in MW}\sum _{y\in MW}B\left(x+p,y+q\right)& \mathrm{Mathematical}\mathrm{expression}2\end{array}$
where

• • [0075] M1_xc≦p≦Mn_xc, M1_yc≦p≦Mn_yc,
  • [0076] MDF is a motion disturbance function,
  • [0077] p, q is a coordinate of a certain pixel in the entire input image,
  • [0078] x, y is a coordinate of a certain pixel within a moving window,
  • [0079] MW is a moving window,
  • [0080] M1_xc, M1_yc is a center coordinate in the moving window containing an initial coordinate of the entire input image,
  • [0081] Mn_xc, Mn_yc is a center coordinate in the moving window containing a final coordinate of the entire input image, and
  • [0082] B(Binary) is a binary disturbance image signal.

[0083] FIG. 7 shows a process of acquiring the information about the moving object.

[0084] Referring FIG. 7, the MDFs of respective pixels (c) are calculated through a plurality of initial tracking windows (b) with respect to the binary disturbance image (a). Then the MDF field with respect to the moving object (d) is detected. If the maximum (greatest) MDF equals to, or greater than a third threshold, the center of the initial tracking window is positioned at the pixel having the maximum MDF. Then the size of the initial tracking window is adjusted so that the initial tracking window includes the moving object, and then the center coordinate of the moving object in the adjusted-sized tracking window and the size of the tracking window (e) are calculated. If the MDF is below the third threshold, it is determined that there is no moving object in operation S43 of FIG. 5. The center coordinate of the moving object in the tracking window is obtained by: $\begin{array}{cc}\begin{array}{c}{x}_e=\frac{\sum _{\left(x,y\right)\in \mathrm{TW}}x·B\left(x-P_L,y-q_L\right)}{\sum _{\left(x,y\in \mathrm{TW}\right)}B\left(x-p_L,y-q_L\right)}\\ y_e=\frac{\sum y·B\left(x-p_L,y-q_L\right)}{\sum _{\left(x,y\in \mathrm{TW}\right)}B\left(x-p_L,y-q_L\right)}\end{array}& \mathrm{Mathematical}\mathrm{expressions}3\end{array}$
where

• • [0085] x_e, y_e is a center coordinate of the moving object in the tracking window,
  • [0086] TW is a tracking window,
  • [0087] x, y is the certain coordinate in the tracking window,
  • [0088] B is the binary disturbance image signal, and
  • [0089] p_L, q_L is a coordinate of a spot where the MDF is a maximum value in the entire input image.

[0090] As described above, the size of the moving object can be calculated by calculating a center value of the actual moving object in the tracking window. The size of the tracking window can also be adjusted based on a size rate of the moving object and the tracking window. An occupancy rate of the moving object to adjust the size of the tracking window is calculated by the following expression: $\begin{array}{cc}\mathrm{Occupancy}\mathrm{rate}=\frac{\mathrm{Moving}\mathrm{Object}\mathrm{Size}}{\mathrm{Tracking}\mathrm{Window}\mathrm{Size}}& \mathrm{Mathematical}\mathrm{expression}4\end{array}$

[0091] When the moving object is detected and the size of the tracking window is adjusted, the location of the moving object is estimated and the moving object is tracked automatically in operation S50 of FIG. 5. In other words, based on the acquired information about the current and previous center of the moving object, the next location of the moving object is estimated in the following frame in operation S51, and the center of the tracking window having the adjusted size is disposed at the estimated location in operation S52. FIG. 8 shows the tracking window being re-arranged to the estimated location of the moving object in the following frame based on the previously acquired information. Next, the information about the true center of the moving object of the following frame in the tracking window and the information about the size of the tracking window are acquired in operation S53. When the information about the actual moving object is acquired in the following frame, the acquired information is compared with the estimated information, and the tracking status is determined according to the error range of the comparison in operation S60. When the error range is within a predetermined limit, the moving object automatic tracking unit 150 keeps tracking the moving object while performing the signal process only with respect to the area of the tracking window. If the error rate exceeds the predetermined limit, the moving object acquiring unit 140 re-acquires the information about the moving object in the entire input image.

[0092] Meanwhile, if the error range of the tracking status is within the predetermined limit, it is determined as the tracking of the moving object is performed well, so that the system controls the movement, the zoom-in/zoom-out and the fan/tilt of the photographing unit 110 in operation S70 . When the center of the tracking window is located outside the predetermined range in the entire input image, the system changes the photographing direction of the photographing unit 110 to the estimated direction of the moving object. Further, when the center of the tracking window is located within a predetermined range A of the entire input image area B as shown in FIG. 11, the system controls so that the fan/tilt control is not performed. The system also controls the zoom-in/zoom-out automatically, so that the size of the tracking window and the moving object can be maintained at a certain rate according to the size of the actual moving object that is acquired from the following frame of the tracking window. As shown in FIG. 12, the system can either perform or omit the process of the zoom-in/zoom-out control depending on the location of the moving object in the input image. Accordingly, unnecessary shaky image due to an excessive control can be prevented while reliability of the system tracking the moving object improves.

[0093] The controlling of the photographing unit 110 and the signal process with respect to the input image are performed in the time-division basis as shown in FIG. 9. Accordingly, no image compensation is required for the movement of the photographing unit 110 (camera). FIG. 10 shows the controlling of the photographing unit 110 and the image signal process according to another embodiment of the present invention, which provides a sufficient time for the initial image signal process of acquiring information about the moving object.

[0094] As a result, the automatic moving object tracking device and method thereof processes even a smallest sized image through the use of the tracking window.

[0095] Further, by using the time-division for the control of the camera such as camera movement, the zoom-in/zoom-out and the fan/tilt, and also for the image signal process, image compensation for the movement of the camera can be avoided.

[0096] Further, the inaccurate determination on the motion area due to the headlight of the automobile can be prevented through the use of the headlight preventing unit 134.

[0097] Further, an accurate fan/tilt control can be achieved by the center value of the moving object, and the moving object can be automatically zoomed-in/zoomed-out based on the size of the moving object and the size of the tracking window. Such zoom-in/zoom-out control can be very useful in circumstances like when a numeral and a character in a license plate of an automobile parked in a parking lot is checked.

[0098] With the automatic moving object tracking device and method thereof according to the present invention, a minimum amount of images is processed during the tracking of the moving object. Accordingly, the processing speed increases while influences on the moving object due to interferences with the other objects can be minimized.

[0099] Further, since the image compensation for the movement of camera can be omitted according to the present invention, a complex signal process in accordance with the tracking of the moving object is simplified.

[0100] Further, the inaccurate determination on the moving object due to changes in the background such as headlight from the automobile can be prevented.

[0101] Further, since the fan/tilt and zoom-in/zoom-out is controlled very accurately with respect to the moving object, the moving object can be clearly recognized while a monitoring result on the monitoring area is checked.

[0102] Although the preferred embodiments of the present invention have been described, it will be understood by those skilled in the art that the present invention should not be limited to the described preferred embodiments, but various changes and modifications may be made without departing from the principles and spirit of the present invention, the scope of which is defined by the appended claims and equivalents.