Method for detection and recognition of fog presence within an aircraft compartment using video images

[0001] This application is a continuation-in-part of U.S. patent application Ser. No. 10/152,148, filed on May 20, 2002, Attorney Docket No. BFM-01301, which is incorporated by reference herein, and claims priority to Attorney Docket No. BFM-02760, U.S. Provisional Patent Application No. 60/502,099, filed on Sep. 11, 2003, which is incorporated by reference herein, and U.S. Provisional Patent Application No. 60/424,287 filed on Nov. 6, 2002, which is also incorporated by reference herein.

[0002] 1. Technical Field

[0003] This application relates to the detection, isolation, verification and alarming of certain conditions, and more particularly to the detection, isolation, verification and alarming of certain conditions using video images.

[0004] 2. Description of Related Art

[0005] Conventional fire and smoke detection systems, such as may be installed aboard aircraft, often suffer from high rates of false alarms. Currently, once an alarm has been initiated, such as by a smoke detection system, the pilot is obligated to suppress the fire and divert the aircraft to the nearest emergency airfield. In case of extended range operations, such as, for example, with large jetliners over polar regions, the act of diverting to the nearest airfield may itself be dangerous as well as inconvenient. Additionally, false alarms may increase costs and expend resources unnecessarily. Reducing false smoke alarm rate may therefore beneficial both for both safety and economic reasons.

[0006] One cause of false alarms is caused by the formation of fog. Conventional systems may have false alarms due to the inability of a fire and smoke detection system to distinguish between fog and smoke. Accordingly, it may be desirable to reduce the incidence rate of false alarms issued by conventional smoke detection system, such as those installed in aircraft, by detecting and distinguishing between fog and smoke conditions.

[0007] In accordance with one aspect of the invention is a method for detecting fog comprising: determining at least one visual characteristic of image data for a portion of an image including at least one of: a change in overall image intensity for said portion, a change in image contrast for said portion, and a change in image sharpness for said portion; and determining using said at least one visual characteristic whether said change associated with said at least one visual characteristic is approximately uniform with respect to said portion. An amount of said change may be within a predetermined threshold. The method may also include determining that at least one of the following conditions is present for said portion: an increase in overall image intensity, a decrease in image sharpness, and a decrease in image contrast; and determining a positive indicator for fog presence if it is determined that said at least one of the following conditions is present, and it is determined using said at least one visual characteristic that said change associated with said at least one visual characteristic is approximately uniform with respect to the portion. The method may also include detecting that said change occurs rapidly within a predetermined time period. The time period may be a few seconds. The method may further comprise: detecting that fog droplets form throughout a predetermined area at a same point in time. An initial value may be associated with said at least one visual characteristic prior to fog formation and a second value is associated with said at least one visual characteristic during fog formation, and the method may further comprise: detecting that a value associated with said at least visual characteristic returns from said second value toward said initial value indicating fog dispersal. The value may return to said initial value from said second value in approximately linear time. The value returning to said initial value from said second value may be approximately uniform across said portion of the image. The method may include: extracting at least one feature of said image data. The at least one feature may include an average image intensity of said portion of said image indicating image brightness, the average intensity m of said portion X of an image at a time “t” being represented as:

$m\left(t\right)=\frac{1}{N}\sum _{j=1}^N\mathrm{Xj}\left(t\right)$

where N represents the total number of pixels in the portion X of the image at a time “t”. The method may include: determining an average image intensity for a plurality of portions for each of a plurality of images over a period of time to detect a trend in image brightness; and determining, using said average image intensity for each of said plurality of portions for each of said plurality of images, if overall brightness of a region defined by said plurality of portions increases over a predetermined time period above a predetermined threshold. The at least one feature may include intensity of a change image with respect to said portion X of said image, D(t,δ)=X(t)−X(t−δ), which is a difference between time instances t and t−δ of said portion, represented as:

$m_D\left(t,\delta \right)=\frac{1}{N}\sum _{j=1}^N\mathrm{Dj}\left(t,\delta \right)$

where N represents a total number of pixels being analyzed of said portion X. The method may further comprise: determining creation of fog when there are one or more positive values of m_D(t,δ); and determining dispersion of fog when there are one or more negative values of m_D(t,δ). The time interval δ between two compared frames may be fixed. The time interval δ may be adjusted in accordance with at least one system condition or parameter. A reference frame of the portion, X(t−δ), may represent an initial view of a portion of a cargo bay X(0) such that δ is a current time t since the start of a flight from a time of the initial view and wherein the difference image with respect to the portion, D(t,δ), represents a cumulative change of view of the portion since beginning of the flight. A portion of reference image X(t−δ) may be reset periodically to accommodate changes of background. A portion of a reference frame X(t−δ) may be set to a frame immediately preceding a current frame such that δ=1, and wherein the difference image with respect to said portion D(t,δ) represents the instantaneous rate of change of a view of said portion. The at least one feature may include a standard deviation of the image intensity of said portion represented as:

$s\left(t\right)=\sqrt{\frac{1}{N}\sum _{j=1}^N{\left(\mathrm{Xj}\left(t\right)-m\left(t\right)\right)}^2}.$

Large values of s(t) may indicate high variability of intensity and may be related to high contrast, low values of s(t) may indicate lower contrast, and the method may include: detecting creation of fog when there is a rapid decrease of s(t) within a predetermined threshold; and detecting fog dispersal when there is an increase in s(t). The at least one feature may include a mean absolute difference from the mean represented as:

$d\left(t\right)=\frac{1}{N}\sum _{j=1}^N\mathrm{Xj}\left(t\right)-m\left(t\right)$

[0008] Creation of fog may be associated with a rapid drop of d(t) above a predetermined threshold and an increase in d(t) may be associated with fog dispersal. The at least one feature may include an absolute value of the average intensity change represented as:

$d_a\left(t,\delta \right)=\frac{1}{N}\sum _{j=1}^N\left(\mathrm{Xj}\left(t\right)-\mathrm{Xj}\left(t-\delta \right)\right)$

in which t and t−δ represent points in time and X(t) and X(t−δ) are portions of images taken, respectively, at these two points in time. The at least one feature may include a correlation measurement based on the t-Student distribution represented as:

$d_s\left(t,\delta \right)=\frac{\sum _{j=1}^N\left(\mathrm{Xj}\left(t\right)-m\left(t-\delta \right)\right)}{\sqrt{\sum _{j=1}^N{\left(\mathrm{Xj}\left(t\right)-m\left(t-\delta \right)\right)}^2}}$

in which t and t−δ represent points in time and X(t) and X(t−δ) are portions of images taken, respectively, at these two points in time. The correlation measurement may be used in tracking a statistical evolution of a portion of a video stream as compared to a portion of the reference image at time t−δ, and wherein values of said correlation measurement larger than a predetermined threshold indicate fog. The at least one feature may include a correlation based on a likelihood ratio distribution represented as:

$d_1\left(t\right)=1-\underset{j=\mathrm{all}\mathrm{pixels}}{\min }\left({\mathrm{LR}}_j,\frac{1}{{\mathrm{LR}}_j}\right)\mathrm{where}$

${\mathrm{LR}}_j=\frac{{\left[\frac{{\left(s\left(t\right)\right)}^2+{\left(s\left(t-\delta \right)\right)}^2}{2}+\frac{{\left(X_j\left(t\right)-m\left(t-\delta \right)\right)}^2}{2}\right]}^2}{{\left(s\left(t\right)\right)}^2{\left(s\left(t-\delta \right)\right)}^2}.$

[0009] The at least one feature may include an intensity range r(t) at a time t represented as: r(t)=χ_max(t)−χ_min(t) where a maximum (Xmax) and a minimum (Xmin) intensity of a portion X of an image at a time t are used to provide an indication of reduced image contrast for the portion X and are represented as:

${\chi }_{\max }\left(t\right)=\underset{j=\mathrm{all}\mathrm{pixels}}{\max }X_j\left(t\right),{\chi }_{\min }\left(t\right)=\underset{j=\mathrm{all}\mathrm{pixels}}{\min }X_j\left(t\right).$

Creation of fog may be indicated by a rapid drop of r(t), and an increase in r(t) may indicate fog dispersal, and r(t) decreasing below a threshold amount may indicate that fog is present. The at least one feature may include characterizing image sharpness of a portion X of an image using an intensity gradient. X and y gradient components G at pixel i,j of a portion X of an image at a time t may be defined as a left difference represented as: G_i,j^x(t)=X_i,j(t)−X_i−1,j(t) G_i,j^y(t)=X_i,j(t)−X_i−j,1(t). X and y gradient components G at pixel i,j of a portion X of an image at a time t may be defined as a right difference represented as: G_i,j^x(t)=X_i+1,j(t)−X_i,j(t) G_i,j^y(t)=X_i,j+1(t)−X_i,j(t). X and y gradient components G at pixel i,j may be defined as a double-sided difference represented as:

$G_{i,j}^x\left(t\right)=\frac{X_{i+1,j}\left(t\right)-X_{i-1,j}\left(t\right)}{2}G_{i,j}^y\left(t\right)=\frac{X_{i,j+1}\left(t\right)-X_{i,j-1}\left(t\right)}{2}.$

The intensity gradient may define a gradient in terms of differences between pixel locations with time as a constant. The intensity gradient may define a gradient in terms of pixel values between portions of images taken at different points in time. The at least one feature may include characterizing image sharpness using an intensity gradient, and wherein said intensity gradient may be determined using a portion of a change image. A large value of said intensity gradient may indicate sharp edges within a portion of an image. The at least one feature may include a mean absolute gradient value represented as:

$g_a^x\left(t\right)=\frac{1}{N}\sum _{i,j=\mathrm{all}\mathrm{pixels}}G_{i,j}^x\left(t\right),g_a^y\left(t\right)=\frac{1}{N}\sum _{i,j=\mathrm{all}\mathrm{pixels}}G_{i,j}^y\left(t\right)$

such that creation of fog is signified by a rapid drop in at least one of: g_a^x(t) and g_a^y(t). The at least one feature may include an overall average gradient characteristic represented as: g_a(t)=g_a^x(t)+g_a^y(t). The at least one feature may include an average gradient norm, wherein a gradient norm G at pixel i,j is represented as: G_i,jⁿ(t)=√{square root over (G_i,j^x(t)²+G_i,j^y(t)²)}{square root over (G_i,j^x(t)²+G_i,j^y(t)²)} for all “N” pixels within a portion of an image, and the average gradient norm may be represented as:

$g_n\left(t\right)=\frac{1}{N}\sum _{i,j=\mathrm{all}\mathrm{pixels}}G_{i,j}^n\left(t\right).$

Creation of fog may be related to a rapid drop in g_n(t) below a predetermined threshold value, and an increase in g_n(t) may indicate fog dispersal. The at least one feature may include maximum and minimum values of x and y components of a gradient norm G represented as:

$g_{\max }\left(t\right)=\underset{i,j=\mathrm{all}\mathrm{pixels}}{\max }G_{i,j}^n\left(t\right),g_{\min }\left(t\right)=\underset{i,j=\mathrm{all}\mathrm{pixels}}{\min }G_{i,j}^n\left(t\right),$

$g_{\max }^x\left(t\right)=\underset{i,j=\mathrm{all}\mathrm{pixels}}{\max }G_{i,j}^x\left(t\right),g_{\min }^x\left(t\right)=\underset{i,j=\mathrm{all}\mathrm{pixels}}{\min }G_{i,j}^n\left(t\right),\mathrm{and}$ $g_{\max }^y\left(t\right)=\underset{i,j=\mathrm{all}\mathrm{pixels}}{\max }G_{i,j}^y\left(t\right),g_{\min }^y\left(t\right)=\underset{i,j=\mathrm{all}\mathrm{pixels}}{\min }G_{i,j}^y\left(t\right).$

The at least one feature may include a dynamic range of intensity change including a standard deviation s_d(t,δ) of intensity change over some predefined time interval, δ, defined as:

$s_d\left(t,\delta \right)=\sqrt{\frac{1}{N}\sum _{j=\mathrm{all}\mathrm{pixels}}{\left(D_j\left(t,\delta \right)-m_d\left(t,\delta \right)\right)}^2}$

and wherein a value of the standard deviation s_d(t,δ) is close to zero within some predetermined threshold if there is fog. The at least one feature may include a mean absolute deviation of a portion from the mean value of a portion of the change image represented as:

$d_d\left(t\right)=\frac{1}{N}\sum _{j=\mathrm{all}\mathrm{pixels}}D_j\left(t,\delta \right)-m_d\left(t,\delta \right)$

and wherein a value of the mean absolute deviation is close to zero within some predetermined threshold if there is fog. The at least one feature may include a spatial moment of a portion of the change image, and wherein coordinates of a center of mass a portion of the change image D(t,δ) are represented as:

$i_c\left(t,\delta \right)=\frac{\sum _{i,j=\mathrm{all}\mathrm{pixels}}{\mathrm{iD}}_{i,j}\left(t,\delta \right)}{\sum _{i,j=\mathrm{all}\mathrm{pixels}}D_{i,j}\left(t,\delta \right)}j_c\left(t,\delta \right)=\frac{\sum _{i,j=\mathrm{all}\mathrm{pixels}}{\mathrm{jD}}_{i,j}\left(t,\delta \right)}{\sum _{i,j=\mathrm{all}\mathrm{pixels}}D_{i,j}\left(t,\delta \right)}$

and wherein, if the image change for a portion is uniform across the portion of the image, the coordinates are close to the geometric center of the portion of the image indicating presence of fog. The at least one feature may include higher order moments of a portion of a change image represented as:

$M_{2,0}\left(t,\delta \right)=\sum _{i,j=\mathrm{all}\mathrm{pixels}}{\left(i-i_c\left(t,\delta \right)\right)}^2D_{i,j}\left(t,\delta \right)$ $M_{0,2}\left(t,\delta \right)=\sum _{i,j=\mathrm{all}\mathrm{pixels}}{\left(j-j_c\left(t,\delta \right)\right)}^2D_{i,j}\left(t,\delta \right).$

and
The at least one feature may include a moment of inertia of a portion of the change image represented as:

$M_i\left(t,\delta \right)=\sum _{i,j=\mathrm{all}\mathrm{pixels}}\left({\left(i-i_c\left(t,\delta \right)\right)}^2+{\left(j-j_c\left(t,\delta \right)\right)}^2\right)D_{i,j}\left(t,\delta \right).$

The at least one feature may include moments defined using average absolute values of pixels represented as:

$M_{1,0}^a\left(t,\delta \right)=\sum _{i,j=\mathrm{all}\mathrm{pixels}}i-i_c\left(t,\delta \right)D_{i,j}\left(t,\delta \right)$ $M_{0,1}^a\left(t,\delta \right)=\sum _{i,j=\mathrm{all}\mathrm{pixels}}j-j_c\left(t,\delta \right)D_{i,j}\left(t,\delta \right)$ $M_i^a\left(t,\delta \right)=\sum _{i,j=\mathrm{all}\mathrm{pixels}}\left(i-i_c\left(t,\delta \right)+j-j_c\left(t,\delta \right)\right)D_{i,j}\left(t,\delta \right)$

and wherein, if a portion of the change image is uniform, values for these moments are larger than a predetermined threshold indicating a presence of fog. The method may also include: determining that at least one of the following conditions is present for said portion of said image: an increase in overall image intensity, a decrease in image sharpness, and a decrease in image contrast; determining an intermediate positive indicator for fog presence if it is determined that said at least one of the following conditions is present, and it is determined using said at least one visual characteristic that said change associated with said at least one visual characteristic is approximately uniform with respect to the portion of the image; and determining a final positive indicator for fog presence if said intermediate positive indicator indicates that there is fog which is confirmed by at least one other feature. The at least one other feature may be a non-image feature. The at least one other feature may include at least one of temperature, humidity and pressure. A plurality of intermediate positive indicators may be used in determining said final positive indicator. The method may further include: distinguishing fog from one of a plurality of other conditions, wherein said plurality of other conditions includes smoke and an aerosol dispersion. The portion may be an entire image. The portion may include a plurality of regions of said image. Each of the plurality of regions may be a predefined shape in accordance with lighting and camera view. The at least one visual characteristic may be a frequency-based feature. The frequency-based feature may estimate motion of an element of said portion. The method may also include: receiving an image from a CCD camera. The CCD camera may have an operational wavelength sensitivity between approximately 770 and 1200 nanometers blocking visible light. The CCD camera may be a conventional CCD camera with an operational wavelength sensitivity between approximately 400 and 1200 nanometers. The CCD camera may be used when it is determined that a view area is completely obscured except for a predetermined space within which said CCD camera is included. The operational wavelength sensitivity of said CCD camera may exclude a portion of the range between approximately 400 and 1200 nanometers. The at least one excluded portion may have a range corresponding to one of: a light source and a device that emits within said at least one excluded portion to filter out wavelengths within said at least one excluded portion. The method may include: receiving an image from a camera with an associated light source wherein said camera is mounted opposite said associated light source within a viewing area. The viewing area may be an aircraft cargo bay, and said camera and said associated light source may be mounted within a predetermined distance from a ceiling of said aircraft cargo bay. The said camera and said associated light source may be positioned at a same vertical and horizontal location on walls of said cargo bay area. The CCD camera may have an operational wavelength sensitivity approximating that of visible light. The frequency-based feature may be used to monitor an area within a camera view for at least one other condition unrelated to fire. A cargo bay area may be monitored during cargo loading using said frequency-base feature.

[0010] In accordance with another aspect of the invention is a computer program product that detects fog comprising executable code that: determines at least one visual characteristic of image data for a portion of an image including at least one of: a change in overall image intensity for said portion, a change in image contrast for said portion, and a change in image sharpness for said portion; and determines, using said at least one visual characteristic, whether said change associated with said at least one visual characteristic is approximately uniform with respect to said portion. An amount of said change may be within a predetermined threshold. The computer program product may include executable code that: determines at least one of the following conditions is present for said portion: an increase in overall image intensity, a decrease in image sharpness, and a decrease in image contrast; and determines a positive indicator for fog presence if it is determined that said at least one of the following conditions is present, and it is determined using said at least one visual characteristic that said change associated with said at least one visual characteristic is approximately uniform with respect to the portion. The computer program product may include: executable code that detects that said change occurs rapidly within a predetermined time period. The time period may be a few seconds. The computer program product may also include: executable code that detects that fog droplets form throughout a predetermined area at a same point in time. An initial value may be associated with said at least one visual characteristic prior to fog formation and a second value may be associated with said at least one visual characteristic during fog formation, and the computer program product may also include executable code that detects that a value associated with said at least visual characteristic returns from said second value toward said initial value indicating fog dispersal. The value may return to said initial value from said second value in approximately linearly time. The value returning to said initial value from said second value may be approximately uniform across said portion of the image. The computer program product may include: executable code that extracts at least one feature of said image data. The at least one feature may include an average image intensity of said portion of said image indicating image brightness, the average intensity m of said portion X of an image at a time “t” being represented as:

$m\left(t\right)=\frac{1}{N}\sum _{j=1}^N\mathrm{Xj}\left(t\right)$

where N represents the total number of pixels in the portion X of the image at a time “t”. The computer program product may include executable code that: determines an average image intensity for a plurality of portions for each of a plurality of images over a period of time to detect a trend in image brightness; and determines, using said average image intensity for each of said plurality of portions for each of said plurality of images, if overall brightness of a region defined by said plurality of portions increases over a predetermined time period above a predetermined threshold. The at least one feature may include intensity of a change image with respect to said portion X of said image, D(t,δ)=X(t)−X(t−δ), which is a difference between time instances t and t−δ of said portion, represented as:

$m_D\left(t,\delta \right)=\frac{1}{N}\sum _{j=1}^N\mathrm{Dj}\left(t,\delta \right)$

where N represents a total number of pixels being analyzed of said portion X. The computer program product may include executable code that: determines creation of fog when there are one or more positive values of m_D(t,δ); and determines dispersion of fog when there are one or more negative values of m_D(t,δ). The time interval δ between two compared frames may be fixed. The time interval δ may be adjusted in accordance with at least one system condition or parameter. A reference frame of the portion, X(t−δ), may represent an initial view of a portion of a cargo bay X(0) such that δ is a current time t since the start of a flight from a time of the initial view and wherein the difference image with respect to the portion, D(t,δ), may represent a cumulative change of view of the portion since beginning of the flight. A portion of reference image X(t−δ) may be reset periodically to accommodate changes of background. A portion of a reference frame X(t−δ) may be set to a frame immediately preceding a current frame such that δ=1, and wherein the difference image with respect to said portion D(t,δ) may represent the instantaneous rate of change of a view of said portion. The at least one feature may include a standard deviation of the image intensity of said portion represented as:

$s\left(t\right)=\sqrt{\frac{1}{N}\sum _{j=1}^N{\left(\mathrm{Xj}\left(t\right)-m\left(t\right)\right)}^2}.$

Large values of s(t) may indicate high variability of intensity and may be related to high contrast, low values of s(t) may indicate lower contrast, and the computer program product may include executable code that: detects creation of fog when there is a rapid decrease of s(t) within a predetermined threshold; and detects fog dispersal when there is an increase in s(t). The at least one feature includes a mean absolute difference from the mean represented as:

$d\left(t\right)=\frac{1}{N}\sum _{j=1}^N\mathrm{Xj}\left(t\right)-m\left(t\right).$

Creation of fog may be associated with a rapid drop of d(t) above a predetermined threshold and wherein an increase in d(t) may be associated with fog dispersal. The at least one feature may include an absolute value of the average intensity change represented as:

$d_a\left(t,\delta \right)=\frac{1}{N}\sum _{j=1}^N\left(\mathrm{Xj}\left(t\right)-\mathrm{Xj}\left(t-\delta \right)\right)$

in which t and t−δ represent points in time and X(t) and X(t−δ) are portions of images taken, respectively, at these two points in time. The at least one feature may include a correlation measurement based on the t-Student distribution represented as:

$d_s\left(t,\delta \right)=\frac{\sum _{j=1}^N\left(\mathrm{Xj}\left(t\right)-m\left(t-\delta \right)\right)}{\sqrt{\sum _{j=1}^N{\left(\mathrm{Xj}\left(t\right)-m\left(t-\delta \right)\right)}^2}}$

in which t and t−δ represent points in time and X(t) and X(t−δ) are portions of images taken, respectively, at these two points in time. The correlation measurement may be used in tracking a statistical evolution of a portion of a video stream as compared to a portion of the reference image at time t−δ, and wherein values of said correlation measurement larger than a predetermined threshold indicate fog. The at least one feature may include a correlation based on a likelihood ratio distribution represented as:

$d_1\left(t\right)=1-\underset{j=\mathrm{all}\mathrm{pixels}}{\min }\left({\mathrm{LR}}_j,\frac{1}{{\mathrm{LR}}_j}\right)$ $\mathrm{where}$ ${\mathrm{LR}}_j=\frac{{\left[\frac{{\left(s\left(t\right)\right)}^2+{\left(s\left(t-\delta \right)\right)}^2}{2}+\frac{{\left(X_j\left(t\right)-m\left(t-\delta \right)\right)}^2}{2}\right]}^2}{{\left(s\left(t\right)\right)}^2{\left(s\left(t-\delta \right)\right)}^2}.$

The at least one feature may include an intensity range r(t) at a time t represented as: r(t)=X_max(t)−X_min(t) where a maximum (Xmax) and a minimum (Xmin) intensity of a portion X of an image at a time t are used to provide an indication of reduced image contrast for the portion X and may be represented as:

${\chi }_{\max }\left(t\right)=\underset{j=\mathrm{all}\mathrm{pixels}}{\max }X_j\left(t\right),{\chi }_{\min }\left(t\right)=\underset{j=\mathrm{all}\mathrm{pixels}}{\min }X_j\left(t\right)$

Creation of fog may be indicated by a rapid drop of r(t), and an increase in r(t) may indicate fog dispersal, and wherein, r(t) decreasing below a threshold amount may indicate that fog is present. The at least one feature may include characterizing image sharpness of a portion X of an image using an intensity gradient. X and y gradient components G at pixel i,j of a portion X of an image at a time t may be defined as a left difference represented as: G_i,j^x(t)=X_i,j(t)=X_i−1,j(t) G_i,j^y(t)=X_i,j(t)−X_i−j,1(t). X and y gradient components G at pixel i,j of a portion X of an image at a time t may be defined as a right difference represented as: G_i,j^x(t)=X_i+1,j(t)=X_i,j(t) G_i,j^y(t)=X_i,j+1(t)−X_i,j,(t) (t). X and y gradient components G at pixel i,j may be defined as a double-sided difference represented as:

$G_{i,j}^x\left(t\right)=\frac{X_{i+1,j}\left(t\right)-X_{i-1,j}\left(t\right)}{2}$ $G_{i,j}^y\left(t\right)=\frac{X_{i,j+1}\left(t\right)-X_{i,j-1}\left(t\right)}{2}.$

The intensity gradient may define a gradient in terms of differences between pixel locations with time as a constant. The intensity gradient may define a gradient in terms of pixel values between portions of images taken at different points in time. The at least one feature includes characterizing image sharpness using an intensity gradient, and wherein said intensity gradient is determined using a portion of a change image. A large value of said intensity gradient may indicate sharp edges within a portion of an image. The at least one feature may include a mean absolute gradient value represented as:

$g_a^x\left(t\right)=\frac{1}{N}\sum _{i,j=\mathrm{all}\mathrm{pixels}}G_{i,j}^x\left(t\right),g_a^y\left(t\right)=\frac{1}{N}\sum _{i,j=\mathrm{all}\mathrm{pixels}}G_{i,j}^y\left(t\right)$

such that creation of fog is signified by a rapid drop in at least one of: g_a^x(t) and g_a^y(t). The at least one feature may an overall average gradient characteristic represented as: g_a(t)=g_a^x(t)+g_a^y(t). The at least one feature may an average gradient norm, wherein a gradient norm G at pixel i,j may be represented as: G_i,jⁿ(t)=√{square root over (G_i,j^x(t)²+G_i,j^y(t)²)}{square root over (G_i,j^x(t)²+G_i,j^y(t)²)} for all “N” pixels within a portion of an image, and the average gradient norm may be represented as:

$g_n\left(t\right)=\frac{1}{N}\sum _{i,j=\mathrm{all}\mathrm{pixels}}G_{i,j}^n\left(t\right).$

Creation of fog may be related to a rapid drop in g_n(t) below a predetermined threshold value, and an increase in g_n(t) may indicate fog dispersal. The at least one feature may include maximum and minimum values of x and y components of a gradient norm G represented

$\mathrm{as}\text{:}$ $g_{\max }\left(t\right)=\underset{i,j=\mathrm{all}\mathrm{pixels}}{\max }G_{i,j}^n\left(t\right),g_{\min }\left(t\right)=\underset{i,j=\mathrm{all}\mathrm{pixels}}{\min }G_{i,j}^n\left(t\right)$ $g_{\max }^x\left(t\right)=\underset{i,j=\mathrm{all}\mathrm{pixels}}{\max }G_{i,j}^x\left(t\right),g_{\min }^{x}\left(t\right)=\underset{i,j=\mathrm{all}\mathrm{pixels}}{\min }G_{i,j}^n\left(t\right)$ $g_{\max }^y\left(t\right)=\underset{i,j=\mathrm{all}\mathrm{pixels}}{\max }G_{i,j}^y\left(t\right),g_{\min }^y\left(t\right)=\underset{i,j=\mathrm{all}\mathrm{pixels}}{\min }G_{i,j}^y\left(t\right).$

The at least one feature may include a dynamic range of intensity change including a standard deviation s_d(t,δ) of intensity change over some predefined time interval, δ, defined as:

$s_d\left(t,\delta \right)=\sqrt{\frac{1}{N}\sum _{j=\mathrm{all}\mathrm{pixels}}{\left(D_j\left(t,\delta \right)-m_d\left(t,\delta \right)\right)}^2}$

and wherein a value of the standard deviation s_d(t,δ) is close to zero within some predetermined threshold if there is fog. The at least one feature may include a mean absolute deviation of a portion from the mean value of a portion of the change image represented as:

$d_d\left(t\right)=\frac{1}{N}\sum _{j=\mathrm{all}\mathrm{pixels}}D_j\left(t,\delta \right)-m_d\left(t,\delta \right)$

and wherein a value of the mean absolute deviation is close to zero within some predetermined threshold if there is fog. The at least one feature may include a spatial moment of a portion of the change image, and wherein coordinates of a center of mass a portion of the change image D(t,δ) may be represented as:

$i_c\left(t,\delta \right)=\frac{\sum _{i,j=\mathrm{all}\mathrm{pixels}}{\mathrm{iD}}_{i,j}\left(t,\delta \right)}{\sum _{i,j=\mathrm{all}\mathrm{pixels}}D_{i,j}\left(t,\delta \right)}j_c\left(t,\delta \right)=\frac{\sum _{i,j=\mathrm{all}\mathrm{pixels}}{\mathrm{jD}}_{i,j}\left(t,\delta \right)}{\sum _{i,j=\mathrm{all}\mathrm{pixels}}D_{i,j}\left(t,\delta \right)}$

and wherein, if the image change for a portion is uniform across the portion of the image, the coordinates are close to the geometric center of the portion of the image indicating presence of fog. The at least one feature may include higher order moments of a portion of a change image represented as:

$M_{2,0}\left(t,\delta \right)=\sum _{i,j=\mathrm{all}\mathrm{pixels}}{\left(i-i_c\left(t,\delta \right)\right)}^2D_{i,j}\left(t,\delta \right)$ $M_{0,2}\left(t,\delta \right)=\sum _{i,j=\mathrm{all}\mathrm{pixels}}{\left(j-j_c\left(t,\delta \right)\right)}^2D_{i,j}\left(t,\delta \right)$

The at least one feature may include a moment of inertia of a portion of the change image represented as:

$M_i\left(t,\delta \right)=\sum _{i,j=\mathrm{all}\mathrm{pixels}}\left({\left(i-i_c\left(t,\delta \right)\right)}^2+{\left(j-j_c\left(t,\delta \right)\right)}^2\right)D_{i,j}\left(t,\delta \right).$

The at least one feature may include moments defined using average absolute values of pixels represented

$\mathrm{as}\text{:}$ $M_{1,0}^a\left(t,\delta \right)=\sum _{i,j=\mathrm{all}\mathrm{pixels}}i-i_c\left(t,\delta \right)D_{i,j}\left(t,\delta \right)$ $M_{0,1}^a\left(t,\delta \right)=\sum _{i,j=\mathrm{all}\mathrm{pixels}}j-j_c\left(t,\delta \right)D_{i,j}\left(t,\delta \right)$ $M_i^a\left(t,\delta \right)=\sum _{i,j=\mathrm{all}\mathrm{pixels}}\left(i-i_c\left(t,\delta \right)+j-j_c\left(t-\delta \right)\right)D_{i,j}\left(t,\delta \right)$

and wherein, if a portion of the change image is uniform, values for these moments are larger than a predetermined threshold indicating a presence of fog. The computer program product may include executable code that: determines that at least one of the following conditions is present for said portion of said image: an increase in overall image intensity, a decrease in image sharpness, and a decrease in image contrast; determines an intermediate positive indicator for fog presence if it is determined that said at least one of the following conditions is present, and it is determined using said at least one visual characteristic that said change associated with said at least one visual characteristic is approximately uniform with respect to the portion of the image; and determines a final positive indicator for fog presence if said intermediate positive indicator indicates that there is fog which is confirmed by at least one other feature. The at least one other feature is a non-image feature. The at least one other feature may include at least one of temperature, humidity and pressure. A plurality of intermediate positive indicators may be used in determining said final positive indicator. The computer program product may include executable code that: distinguishes fog from one of a plurality of other conditions, wherein said plurality of other conditions includes smoke and an aerosol dispersion. The portion may be an entire image. The portion may include a plurality of regions of said image. Each of the plurality of regions may be a predefined shape in accordance with lighting and camera view. The at least one visual characteristic may be a frequency-based feature. The frequency-based feature may estimate motion of an element of said portion. The computer program product may include executable code that receives an image from a CCD camera. The CCD camera may have an operational wavelength sensitivity between approximately 770 and 1200 nanometers blocking visible light. The CCD camera may be a conventional CCD camera with an operational wavelength sensitivity between approximately 400 and 1200 nanometers. The CCD camera may be used when it is determined that a view area is completely obscured except for a predetermined space within which said CCD camera is included. The operational wavelength sensitivity of said CCD camera may exclude a portion of the range between approximately 400 and 1200 nanometers. The at least one excluded portion may have a range corresponding to one of: a light source and a device that emits within said at least one excluded portion to filter out wavelengths within said at least one excluded portion. The computer program product may include: executable code that receives an image from a camera with an associated light source wherein said camera is mounted opposite said associated light source within a viewing area. The viewing area may be an aircraft cargo bay, and said camera and said associated light source may be mounted within a predetermined distance from a ceiling of said aircraft cargo bay. The camera and said associated light source may be positioned at a same vertical and horizontal location on walls of said cargo bay area. The CCD camera may have an operational wavelength sensitivity approximating that of visible light.

[0011] In accordance with one aspect of the invention is a method for detecting smoke comprising: determining portions of a plurality of images obtained at different times; determining a region for each of the portions in accordance with a predetermined pixel intensity, a minimum number of neighboring pixels having said intensity which are connected to a pixel of interest, and a predetermined connectivity, said predetermined connectivity indicating particular pixel positions adjacent to a pixel of interest; and determining, using regions for said corresponding portions, whether the portions indicate a smoke condition. The method may be executed in a computer system. The portions of said images may include a view of a ceiling gap of a cargo bay area in an aircraft. The method may also include: determining whether the portions indicate a non-smoke condition. The portions may indicate a smoke condition if the portions include a shrinking region of brightness with respect to a change in time. The portion may be entire frames. The portions may be determined in accordance with an image projection of a lighting source for an opposite lighting view wherein a camera obtains an image using only a light source located opposite said camera. The plurality of images may be obtained using at least one camera receiving near infrared light. The camera may be a CCD camera. The lighting may be used to produce said opposite lighting view using a lighting source controlled by a system control unit. The determining whether the portions indicate a smoke condition may set an indicator to indicate one of a smoke and a non-smoke condition. The method may also include: using said indicator as one of a plurality of inputs to determine whether a fire condition is present. The predetermined connectivity may be one of 4 and 8 indicating a number of adjacent pixel positions used to evaluate a pixel of interest, and the method may also include: determining whether said pixel of interest meets said predetermined connectivity by determining if said pixel of interest has said minimum number of neighboring pixels above said predetermined pixel intensity, and whether said pixel of interest meets said predetermined pixel intensity. The pixel and all connected neighboring pixels located in said portion may be included in said region if said pixel of interest meets said predetermined connectivity and said predetermined pixel intensity. The minimum number of neighboring pixels may be less than or equal to said predetermined connectivity. The method may also include: determining, for each of said portions, if said region is shrinking. The method may also include: determining a feature for each of said portions, said feature using a count of bright and connected pixels included in said region for each of said portions. The method may also include: determining for each pixel in a reference frame whether said each pixel meets specified criteria, said specified criteria including whether said each pixel meets said predetermined pixel intensity and has said minimum number of neighboring pixels meeting said predetermined pixel intensity in accordance with neighboring pixel positions indicated by said predetermined connectivity; determining for each pixel in a subsequent frame whether said each pixel meets said specified criteria; determining for each pixel in said subsequent frame whether there has been a state change of said each pixel wherein it is determined that there has been a state change for said each pixel if said each pixel has changed from one of: meeting said criteria in said reference frame to not meeting said criteria in said subsequent frame, and not meeting said criteria in said reference frame to meeting said criteria in said subsequent frame; determining a feature for said subsequent frame using a total number of pixels determined as having a state change; and using said feature in determining whether a smoke condition is present. The total number of pixels having a state change may be compared to a threshold, and a smoke condition may be determined as being present when said total number exceeds said threshold. The feature may be a difference in said total number of pixels determined for two subsequent frames obtained at two points in time. The feature may be a ratio of said total number to a number of bright and connected pixels included in said region associated with said reference frame. The method may also include: determining a first quantity of pixels in a portion of a reference frame meeting specified criteria, said specified criteria including whether each pixel in said portion meets said predetermined pixel intensity and has said minimum number of neighboring pixels meeting said predetermined pixel intensity in accordance with neighboring pixel positions indicated by said predetermined connectivity; determining a second quantity of pixels in a portion of a first subsequent frame meeting said specified criteria; and using said first and second quantities in determining whether a smoke condition is present. The method may also include: determining a third quantity of pixels in a portion of a second subsequent frame meeting said specified criteria; determining a first feature value for said first subsequent frame that is a ratio using said first and second quantities and a second feature value for said second subsequent frame that is a ratio using said first and third quantities; and using said feature values in determining whether a smoke condition is present. The reference frame may be determined using an opposite camera lighting view and determining a projection of a light source of a recorded image. The method may also include: determining said region by using a starting pixel candidate in accordance with said light source. The method may also include: determining said region for each of said portions using a starting pixel candidate located in a first row and column position of said each portion, and wherein each pixel in each of said portions are examined in accordance with specified criteria, said specified criteria including whether each pixel in said each portion meets said predetermined pixel intensity and has said minimum number of neighboring pixels meeting said predetermined pixel intensity in accordance with neighboring pixel positions indicated by said predetermined connectivity. The method may also include: determining region shrinkage of said region of a first frame as compared to a second frame. The region shrinkage may be determined using only pixels initially indicated as meeting said specified criteria in a reference frame. The method may also include: determining said predetermined pixel intensity in accordance with a distribution of pixel intensity values of an opposing camera lighting view of an image. At least the minimum number of neighboring pixels, said predetermined connectivity, and a smoke detection threshold may be selected in accordance with selection criteria including a first criteria that all fire conditions are properly classified. The selection criteria may include a second criteria that smoke detection occurs before at least one conventional fire detection unit. The selection criteria may include a third criteria that minimizes a number of non-fire conditions properly classified. The method may include: determining a third quantity of pixels in a portion of a second subsequent frame meeting said specified criteria; determining a first feature value for said first subsequent frame that is one of: an absolute value of a difference between said first quantity and said second quantity, and an absolute value of a difference between said second quantity and another quantity associated with a frame preceding said first subsequent frame in time; determining a second feature value for said subsequent frame that is one of: an absolute value of a difference between said first quantity and said third quantity, and an absolute value of a difference between said second quantity and said third quantity; and using said first and second feature values in determining whether a smoke condition is present. The predetermined connectivity may include two layers of pixels surrounding a pixel of interest, and the method may further comprise: determining whether said pixel of interest meets said predetermined connectivity by determining if said pixel of interest has said minimum number of neighboring pixels above said predetermined pixel intensity in said two layers, and whether said pixel of interest meets said predetermined pixel intensity. The predetermined connectivity may be a predefined shape with respect to said pixel of interest, and the method may further comprise: determining whether said pixel of interest meets said predetermined connectivity by determining if said pixel of interest has said minimum number of neighboring pixels above said predetermined pixel intensity in said predefined shape, and whether said pixel of interest meets said predetermined pixel intensity. The predefined shape may be one of: a rectangle, a square, at least one horizontal row, and at least one vertical row. The predefined shape may be selected in accordance with a property of data being analyzed. The feature may be a ratio of a feature value associated with a current frame, FeatureValue(t), with respect to a feature value associated with a reference frame, FeatureValue (O), represented as:

$\frac{\mathrm{FeatureValue}\left(t\right)-\mathrm{FeatureValue}\left(0\right)}{\mathrm{FeatureValue}\left(0\right)}.$

The feature value may be a value of one of the following features: number of bright and connected pixels in a connected bright region, mean pixel intensity, pixel intensity standard deviation, gradient mean, and gradient mean standard deviation. The feature may represent a relative comparison of a feature value at a first point in time with respect to a feature values of the feature associated with a reference frame. The region may be determined in accordance with a time connectivity K, K being greater than or equal to zero, K representing whether a pixel has been included in at least K previous regions associated with K other frames at different points in time.

[0012] In accordance with another aspect of the invention is a computer program product that detects detecting smoke comprising executable code that: determines portions of a plurality of images obtained at different times; determines a region for each of the portions in accordance with a predetermined pixel intensity, a minimum number of neighboring pixels having said intensity which are connected to a pixel of interest, and a predetermined connectivity, said predetermined connectivity indicating particular pixel positions adjacent to a pixel of interest; and determines, using regions for said corresponding portions, whether the portions indicate a smoke condition. The portions of said images may include a view of a ceiling gap of a cargo bay area in an aircraft. The computer program product may include: executable code that determines whether the portions indicate a non-smoke condition. The portions may indicate a smoke condition if the portions include a shrinking region of brightness with respect to a change in time. The portions may be entire frames. The portions may be determined in accordance with an image projection of a lighting source for an opposite lighting view wherein a camera obtains an image using only a light source located opposite said camera. The plurality of images may be obtained using at least one camera receiving near infrared light. The camera may be a CCD camera. The lighting used to produce said opposite lighting view may use a lighting source controlled by a system control unit. The executable code that determines may also include executable code that determines whether the portions indicate a smoke condition further comprises executable code that sets an indicator to indicate one of a smoke and a non-smoke condition. The computer program product may also include: executable code that uses said indicator as one of a plurality of inputs to determine whether a fire condition is present. The predetermined connectivity may be one of 4 and 8 indicating a number of adjacent pixel positions used to evaluate a pixel of interest, and the computer program product may also include: executable code that determines whether said pixel of interest meets said predetermined connectivity by determining if said pixel of interest has said minimum number of neighboring pixels above said predetermined pixel intensity, and whether said pixel of interest meets said predetermined pixel intensity. The pixel and all connected neighboring pixels located in said portion may be included in said region if said pixel of interest meets said predetermined connectivity and said predetermined pixel intensity. The minimum number of neighboring pixels may be less than or equal to said predetermined connectivity. The computer program product may also include executable code that determines, for each of said portions, if said region is shrinking. The computer program product may also include: executable code that determines a feature for each of said portions, said feature using a count of bright and connected pixels included in said region for each of said portions. The computer program product may further comprise executable code that: determines for each pixel in a reference frame whether said each pixel meets specified criteria, said specified criteria including whether said each pixel meets said predetermined pixel intensity and has said minimum number of neighboring pixels meeting said predetermined pixel intensity in accordance with neighboring pixel positions indicated by said predetermined connectivity; determines for each pixel in a subsequent frame whether said each pixel meets said specified criteria; determines for each pixel in said subsequent frame whether there has been a state change of said each pixel wherein it is determined that there has been a state change for said each pixel if said each pixel has changed from one of: meeting said criteria in said reference frame to not meeting said criteria in said subsequent frame, and not meeting said criteria in said reference frame to meeting said criteria in said subsequent frame; determines a feature for said subsequent frame using a total number of pixels determined as having a state change; and uses said feature in determining whether a smoke condition is present. The computer program product may also comprise: executable code that compares said total number of pixels having a state change to a threshold; and executable code that determines a smoke condition as being present when said total number exceeds said threshold. The feature may be a difference in said total number of pixels determined for two subsequent frames obtained at two points in time. The feature may be a ratio of said total number to a number of bright and connected pixels included in said region associated with said reference frame. The computer program product may further comprise: executable code that: determines a first quantity of pixels in a portion of a reference frame meeting specified criteria, said specified criteria including whether each pixel in said portion meets said predetermined pixel intensity and has said minimum number of neighboring pixels meeting said predetermined pixel intensity in accordance with neighboring pixel positions indicated by said predetermined connectivity; determines a second quantity of pixels in a portion of a first subsequent frame meeting said specified criteria; and uses said first and second quantities in determining whether a smoke condition is present. The computer program product may further comprise executable code that: determines a third quantity of pixels in a portion of a second subsequent frame meeting said specified criteria; determines a first feature value for said first subsequent frame that is a ratio using said first and second quantities and a second feature value for said second subsequent frame that is a ratio using said first and third quantities; and uses said feature values in determining whether a smoke condition is present. The reference frame may be determined using an opposite camera lighting view and determining a projection of a light source of a recorded image. The computer program product may further comprise: executable code that determines said region by using a starting pixel candidate in accordance with said light source. The computer program product may also include: executable code that determines said region for each of said portions using a starting pixel candidate located in a first row and column position of said each portion, and wherein each pixel in each of said portions are examined in accordance with specified criteria, said specified criteria including whether each pixel in said each portion meets said predetermined pixel intensity and has said minimum number of neighboring pixels meeting said predetermined pixel intensity in accordance with neighboring pixel positions indicated by said predetermined connectivity. The computer program product may also include: executable code that determines region shrinkage of said region of a first frame as compared to a second frame. The region shrinkage may be determined using only pixels initially indicated as meeting said specified criteria in a reference frame. The computer program product may also include: executable code that determines said predetermined pixel intensity in accordance with a distribution of pixel intensity values of an opposing camera lighting view of an image. At least the minimum number of neighboring pixels, said predetermined connectivity and a smoke detection threshold may be selected in accordance with selection criteria including a first criteria that all fire conditions are properly classified. The selection criteria may include a second criteria that smoke detection occurs before at least one conventional fire detection unit. The selection criteria may include a third criteria that minimizes a number of non-fire conditions properly classified. The computer program product may further comprise executable code that: determines a third quantity of pixels in a portion of a second subsequent frame meeting said specified criteria; determines a first feature value for said first subsequent frame that is one of: an absolute value of a difference between said first quantity and said second quantity, and an absolute value of a difference between said second quantity and another quantity associated with a frame preceding said first subsequent frame in time; determines a second feature value for said subsequent frame that is one of: an absolute value of a difference between said first quantity and said third quantity, and an absolute value of a difference between said second quantity and said third quantity; and uses said first and second feature values in determining whether a smoke condition is present. The predetermined connectivity may include two layers of pixels surrounding a pixel of interest, and the computer program product may further comprise: executable code that determines whether said pixel of interest meets said predetermined connectivity by determining if said pixel of interest has said minimum number of neighboring pixels above said predetermined pixel intensity in said two layers, and whether said pixel of interest meets said predetermined pixel intensity. The predetermined connectivity may be a predefined shape with respect to said pixel of interest, and the computer program product may further comprise: executable code that determines whether said pixel of interest meets said predetermined connectivity by determining if said pixel of interest has said minimum number of neighboring pixels above said predetermined pixel intensity in said predefined shape, and whether said pixel of interest meets said predetermined pixel intensity. The predefined shape may be one of: a rectangle, a square, at least one horizontal row, and at least one vertical row. The predefined shape may be selected in accordance with a property of data being analyzed. The feature may be a ratio of a feature value associated with a current frame, FeatureValue(t), with respect to a feature value associated with a reference frame, FeatureValue (O), represented as:

$\frac{\mathrm{FeatureValue}\left(t\right)-\mathrm{FeatureValue}\left(0\right)}{\mathrm{FeatureValue}\left(0\right)}.$

The feature value may be a value of one of the following features: number of bright and connected pixels in a connected bright region, mean pixel intensity, pixel intensity standard deviation, gradient mean, and gradient mean standard deviation. The feature may represent a relative comparison of a feature value at a first point in time with respect to a feature values of the feature associated with a reference frame. The region may be determined in accordance with a time connectivity K, K being greater than or equal to zero, K representing whether a pixel has been included in at least K previous regions associated with K other frames at different points in time.

[0013] In accordance with one aspect of the invention is a method for detection of hotspots comprising: determining a connected bright region for a portion of a frame, said connected bright region including each pixel of said portion which exceed a first predetermined threshold and has a minimum number of neighboring pixels exceeding said first predetermined threshold and being connected to said each pixel; and determining, using a feature based on said connected bright region, whether a hotspot exists in accordance with a detection threshold. The minimum number of neighboring pixels may be determined in accordance with a connectivity configuration indicating particular pixel positions adjacent to said each pixel. The connectivity configuration may specify one of 4 and 8 pixel positions adjacent to a pixel of interest. The method may further comprise: determining a first metric for a first portion of a first frame using a first connected bright region associated with said first frame at a first point in time; determining a second metric for a second portion of a second frame using a second connected bright region associated with said second frame at a second point in time; and determining whether a hotspot exists using said first and said second metrics. The method may also include: performing a trend analysis using said first and said second metrics to determine whether a hotspot exists. The feature may use the number of pixels included in said connected bright region. The portion may be one of: said entire frame and pixels selected from said frame in accordance with a predetermined pattern. The portion may include every other pixel in a designated area of said frame. The frame may be obtained using a thermal lighting view in which no controlled light is switched on. The frame may be obtained using a camera that receives near infrared light. The first predetermined threshold may be determined offline in accordance with an image histogram of gray level distribution obtained using a thermal lighting view with all controlled lighting switch off with a fire condition. The first threshold distribution may be determined in accordance with predetermined criteria including at least one of: minimizing false positives, limitations of a camera used to obtain said frame, image noise, and detection prior to a conventional fire detection system. A reference frame may be subtracted from said frame.

[0014] The reference frame may be used to filter out light sources. A pixel in said connected bright region may meet a time connectivity parameter K, K being greater than or equal to zero, K representing whether a pixel has been included in at least K previous connected bright regions associated with K other frames determined at different points in time. The method may also include: determining a ratio of a number of pixels included in said connected bright region to a total number of pixels in said portion; and determining if a hotspot exists in said frame if said ratio exceeds a hotspot detection threshold. The ratio may be represented as:

$\mathrm{HotSpot}\mathrm{Coverage}\left(t_k\right)=\frac{100}{\left(\left(\frac{M}{2}-1\right)\times \left(\frac{N}{2}-1\right)\right)}\times \sum _{m=1}^{\left(\frac{M}{2}-1\right)}\sum _{n=1}^{\left(\frac{N}{2}-1\right)}\left\{\left[\sum _{i=\left(2m-1\right)}^{i=\left(2m+1\right)}\sum _{j=\left(2n-1\right)}^{j=\left(2n+1\right)}\left(P\left(i,j;t_k\right)>H_0\right)\right]>C_0\right\}$

and (M,N) designates frame dimensions; H₀ is the first predetermined threshold indicating a brightness threshold; C₀ is a minimum number of neighboring pixels based on a selected connectivity type; P(i,j))tk represents a pixel value at time tk; and
a time connectivity of order K applied to a pixel p of coordinates (i,j) means that:

$\left\{\left[\sum _{i=\left(2m-1\right)}^{i=\left(2m+1\right)}\sum _{j=\left(2n-1\right)}^{j=\left(2n+1\right)}\left(P\left(i,j;t\right)>H_0\right)\right]>C_0\right\}$ $\mathrm{for}t=t_k,t_{k-1},\dots ,t_{k-K}$

for K frames at different points in time t. The method may also include: setting an indicator if a hotspot is detected. The indicator may be one of a plurality of inputs to data fusion processing producing as an output a system fire signal in accordance with said plurality of inputs. The determining of a connected bright region may be performed in response to detecting a first precondition using a global metric. The global metric may characterize pixels and associated values of said frame.

[0015] In accordance with another aspect of the invention is a computer program product that detects hotspots comprising executable code that: determines a connected bright region for a portion of a frame, said connected bright region including each pixel of said portion which exceed a first predetermined threshold and has a minimum number of neighboring pixels exceeding said first predetermined threshold and being connected to said each pixel; and determines, using a feature based on said connected bright region, whether a hotspot exists in accordance with a detection threshold. The minimum number of neighboring pixels may be determined in accordance with a connectivity configuration indicating particular pixel positions adjacent to said each pixel. The connectivity configuration may specify one of 4 and 8 pixel positions adjacent to a pixel of interest.

[0016] The computer program product may further comprise executable code that: determines a first metric for a first portion of a first frame using a first connected bright region associated with said first frame at a first point in time; determines a second metric for a second portion of a second frame using a second connected bright region associated with said second frame at a second point in time; and determines whether a hotspot exists using said first and said second metrics. The computer program product may further comprise: executable code that performs a trend analysis using said first and said second metrics to determine whether a hotspot exists. The feature may use the number of pixels included in said connected bright region. The portion may be one of: said entire frame, and pixels selected from said frame in accordance with a predetermined pattern. The portion may include every other pixel in a designated area of said frame. The frame may be obtained using a thermal lighting view in which no controlled light is switched on. The frame may be obtained using a camera that receives near infrared light. The first predetermined threshold may be determined offline in accordance with an image histogram of gray level distribution obtained using a thermal lighting view with all controlled lighting switch off with a fire condition. The first threshold distribution may be determined in accordance with predetermined criteria including at least one of: minimizing false positives, limitations of a camera used to obtain said frame, image noise, and detection prior to a conventional fire detection system. A reference frame may be subtracted from said frame. The reference frame may be used to filter out light sources. A pixel in the connected bright region may meet a time connectivity parameter K, K being greater than or equal to zero, K representing whether a pixel has been included in at least K previous connected bright regions associated with K other frames determined at different points in time. The computer program product may further comprise executable code that: determines a ratio of a number of pixels included in said connected bright region to a total number of pixels in said portion; and determines if a hotspot exists in said frame if said ratio exceeds a hotspot detection threshold. The ratio may be represented as:

$\mathrm{HotSpot}\mathrm{Coverage}\left(t_k\right)=\frac{100}{\left(\left(\frac{M}{2}-1\right)\times \left(\frac{N}{2}-1\right)\right)}\times \sum _{m=1}^{\left(\frac{M}{2}-1\right)}\sum _{n=1}^{\left(\frac{N}{2}-1\right)}\left\{\left[\sum _{i=\left(2m-1\right)}^{i=\left(2m+1\right)}\sum _{j=\left(2n-1\right)}^{j=\left(2n+1\right)}\left(P\left(i,j;t_k\right)>H_0\right)\right]>C_0\right\}$

and (M,N) designates frame dimensions; H₀ is the first predetermined threshold indicating a brightness threshold; C₀ is a minimum number of neighboring pixels based on a selected connectivity type; P(i,j))tk represents a pixel value at time tk; and a time connectivity of order K applied to a pixel p of coordinates (i,j) means that:

$\left\{\left[\sum _{i=\left(2m-1\right)}^{i=\left(2m+1\right)}\sum _{j=\left(2n-1\right)}^{j=\left(2n+1\right)}\left(P\left(i,j;t\right)>H_0\right)\right]>C_0\right\}$ $\mathrm{for}t=t_k,t_{k-1},\dots ,t_{k-K}$

for K frames at different points in time t. The computer program product may also include: executable code that sets an indicator if a hotspot is detected. The indicator may be one of a plurality of inputs to data fusion processing producing as an output a system fire signal in accordance with said plurality of inputs. The executable code that determines a connected bright region may be responsive to executable code that detects a first precondition using a global metric. The global metric may characterize pixels and associated values of said frame.

[0017] In accordance with aspect of the invention is a method for distinguishing a smoke condition from a dust condition comprising: obtaining a first frame of video data; determining at least one feature of a portion of said first frame; and determining a smoke condition if a threshold value for said at least one feature is crossed, and, in the presence of a dust condition, said threshold value is not crossed. A fire indicator may be set if the threshold value is crossed. At least one feature may be an average absolute gray level intensity represented as:

$\mu \left(t\right)=\frac{1}{N_m\times M_m}\sum _{i=1}^{N_m}\sum _{j=1}^{M_m}P\left(i,j;t\right)-P_{\mathrm{Ref}}\left(i,j;t_0\right)$

where: P(i, j; t) is a pixel value at location (i,j) and time t; P_Ref(i, j;t₀) is a pixel value at location (i,j) of a reference frame at time 0; and N_m and M_m are dimensions of said portion. The first frame may be obtained using one of: an overhead lighting view in which only overhead controlled lights are turned on, a diagonal lighting view in which only a light source opposite a camera obtaining said first frame is on, and a pilot lighting view in which only a light source co-located with a camera obtaining said first frame is on. If the threshold value is crossed, a smoke condition may be determined. The threshold value may be determined in accordance with a vibration level of noise in a system. The method may also include: selecting said threshold in accordance with criteria that all fire conditions are properly classified and an acceptable rate of dust cases are improperly classified. The criteria may include detecting a fire condition prior to a selected conventional smoke detector. The feature may have a larger value in a smoke condition than a dust condition. The feature may include a plurality of features each associated with at least one threshold. The plurality of features may include at least two of: a mean intensity value of an average pixel intensity for all pixels in said portion; a maximum pixel intensity value of a maximum pixel intensity for all pixels in said portion; a number of pixels in a bright region, said number representing a number of pixels in said portion that are one of: greater than a threshold intensity, and greater than a threshold intensity and connected to a minimum number of neighboring pixels; a mean norm of intensity gradient vector for all pixels in said portion; and a norm of the second order moment matrix calculated over all pixels in said portion. A fire condition may be determined if any one of the plurality of features crosses an associated threshold. Different features may be determined and used in detecting a smoke condition in accordance with the particular lighting view of a frame. The mean intensity value may be determined for a frame obtained in one of a diagonal and overhead lighting view, said maximum pixel intensity may be determined for a frame obtained in one of diagonal view, and said size of a brightness region may be determined for a frame obtained in one of a diagonal and overhead lighting views. A different threshold may be used for each combination of feature and lighting view for at least one feature. The plurality of features may include a feature that is a mean intensity represented as:

$\mathrm{mean}\left(P\right)=\frac{1}{\mathrm{MN}}\sum _{i=1}^M\sum _{j=1}^NP\left(i,j\right)$

where:M represents a number of pixel rows in said portion; N represents a number of pixel columns in said portion; P is said portion of a frame; P(i,j) is a pixel intensity or gray level at pixel position i,j in P. The plurality of features may include a second feature that is a mean norm of a gradient vector represented as:

$\mathrm{mgrad}\left(P\right)=\frac{1}{\mathrm{MN}}\sum _{i=1}^{M-1}\sum _{j=1}^{N-1}\sqrt{{G_1\left(i,j\right)}^2+{G_2\left(i,j\right)}^2}$

where G₁ and G₂ are given by:

$G_1\left(i,j\right)=X\left(i+1,j\right)-X\left(i,j\right)$ $G_2\left(i,j\right)=X\left(i,j+1\right)-X\left(i,j\right).$

A value of said second feature may decrease as an image decreases in sharpness of detail. A third feature of said plurality of features may be a norm of a second order moment matrix represented as:

$\mathrm{norm}2\left(P\right)=\mathrm{norm}\left(\left[\begin{array}{cc}\mathrm{SM}20\left(P\right)& \mathrm{SM}11\left(P\right)\\ \mathrm{SM}11\left(P\right)& \mathrm{SM}02\left(P\right)\end{array}\right]\right)$

where second order moments S M 20, S M 02 and S M11 are defined via equations:

$\mathrm{Xcenter}\left(P\right)=\frac{1}{\mathrm{MN}}\sum _{i=1}^M\sum _{j=1}^N\mathrm{iP}\left(i,j\right);$ $\mathrm{Ycenter}\left(P\right)=\frac{1}{\mathrm{MN}}\sum _{i=1}^M\sum _{j=1}^N\mathrm{jP}\left(i,j\right);$ $\mathrm{SM}20\left(P\right)=\frac{1}{\mathrm{MN}}\sum _{i=1}^M\sum _{j=1}^N{\left(i-\mathrm{Xcenter}\right)}^2P\left(i,j\right)$ $\mathrm{SM}02\left(P\right)=\frac{1}{\mathrm{MN}}\sum _{i=1}^M\sum _{j=1}^N{\left(j-\mathrm{Ycenter}\right)}^2P\left(i,j\right)$ $\mathrm{and}$ $\mathrm{SM}11\left(P\right)=\frac{1}{\mathrm{MN}}\sum _{i=1}^M\sum _{j=1}^N\left(i-\mathrm{Xcenter}\right)\left(j-\mathrm{Ycenter}\right)P\left(i,j\right).$

The plurality of features may include a fourth feature that is the size of a bright region represented as:
brsize(P)=Σ_i=1^MΣ_j=1^NT(i,j) where binary thresholded image T is defined through at least an intensity threshold. The bright region may be determined in accordance with an intensity threshold and a connectivity factor. A fire condition may be determined if any one value of any one feature crosses a threshold. The method may also include: determining said first, said second and said fourth features if said first frame is obtained using a diagonal lighting view in which said frame is obtained using only an opposing camera light; and determining said first, said second and said third features if said first frame is obtained using an overhead lighting view in which said frame is obtained using only overhead lights. At least one feature may be determined for multiple windows of said portion. At least one of said plurality of features may be determined using a relative change expressed as a ratio of a first value to a second value, said first value representing said feature for said portion and said second value representing a steady state feature value. Each of said multiple windows may be determined in accordance with a location of at least one light relative to location and field of view of a camera acquiring frames. The plurality of features may include a fifth feature that is a maximum pixel intensity value represented as:

$\max \left(P\right)=\underset{j=1,N}{\underset{i=1,M}{\max }}P\left(i,j\right)$

where: M represents a number of pixel rows in said portion; N represents the number of pixel columns in said portion; P is said portion; and P(i,j) is a pixel intensity or gray level at pixel position i,j in said portion. The at least one feature may be a rate of change feature based on an average absolute gray level, wherein the rate of change may be calculated as:

${\delta }_{\mu }=\frac{\mu \left(t\right)-\mu \left(t-\Delta t\right)}{\Delta t}$

where μ(t) and μ(t−Δt) are values of an average absolute intensity feature at times t and t−Δt, respectively, and wherein said average absolute gray level is represented as:

$\mu \left(t\right)=\frac{1}{N_m\times M_m}\sum _{i=1}^{N_m}\sum _{j=1}^{M_m}P\left(i,j;t\right)-P_{\mathrm{Ref}}\left(i,j;t_0\right)$

where: P(i, j; t) is a pixel value at location (i,j) and time t; P_Ref(i, j; t₀) is a pixel value at location (i,j) of a reference frame at time 0; and N_m and M_n, are dimensions of said portion. A value of said rate of change feature may be higher for a smoke condition than for a dust condition.

[0018] In accordance with another aspect of the invention is a method of detecting a fire condition comprising: determining a lighting view associated with a frame; if said lighting view is an opposite lighting view, determining a number of pixels associated with a portion of a frame in accordance with a lighting view of said frame, said number of pixels representing a number of pixels in said portion which increase or decrease by an amount exceeding a brightness threshold with respect to a reference frame, each of said number of pixels being connected to a minimum number of neighboring pixels in terms of space, and determining a ratio of said number of pixels for a current frame with respect to said number of pixels of a reference frame; and if said ratio exceeds a first threshold, determining that a fire condition exists, said first ratio being determined in accordance with the opposite lighting view. For a current frame at a time tk, said ratio may be represented as a percentage of fading with an opposite view configuration:

$\mathrm{Fading}\left(t_k\right)=100\times \frac{\sum _{m=1}^{M/S_s}\sum _{n=1}^{N/S_s}\left\{\left(\begin{array}{c}\sum _{i=-1}^{i=1}\sum _{j=-1}^{j=1}P\left(\begin{array}{c}{S}_sm+i,\\ S_sn+j;t_k\end{array}\right)-\\ P_{\mathrm{Ref}}\left(\begin{array}{c}{S}_sm+i,\\ S_sn+j;t_0\end{array}\right)>D_0\end{array}\right)>C_0\right\}}{\mathrm{Fading}\left(t_0\right)}\%$

wherein Fading(t₀) is a number of bright and connected pixels generated by an opposite light of a reference image represented by:

$\mathrm{Fading}\left(t_0\right)=\sum _{m=1}^{M/S_s}\sum _{n=1}^{N/S_s}\left\{\left(\sum _{i=-1}^{i=1}\sum _{j=-1}^{j=1}P_{\mathrm{Ref}}\left(\begin{array}{c}{S}_sm+i,\\ S_sn+j;t_0\end{array}\right)>D_0\right)>C_0\right\}$

such that Fading(t₀) is non zero; (M,N) designates dimensions of said portion; D₀ is said brightness threshold determined in accordance with an opposite lighting view; S_s is a sweeping step across one of a horizontal and vertical directions; and C₀ is a clustering threshold that is said minimum number of neighboring pixels in terms of space. Each of said number of pixels in said current frame may exceed said brightness threshold and said minimum number of neighboring pixels for a predetermined number of previous samples representing a time connectivity, said predetermined number being greater than zero. The method may also include: determining if said lighting view is one of: an overhead and near view lighting, and if so, determining a feature for said portion as a number of fading/glowing pixels represented

$\mathrm{as}\text{:}\mathrm{FG}\left(t_k\right)=\sum _{m=I}^{M/S_s}\sum _{n=I}^{N/S_s}\left\{\left(\sum _{i=-1}^{i=1}\sum _{j=-1}^{j=1}\begin{array}{c}P\left(S_sm+i,S_sn+j;t_k\right)-\\ P_{\mathrm{Ref}}\left(S_sm+i,S_sn+j;t_0\right)\end{array}>D_0\right)>C_0\right\}$

wherein Do of FG(tk) is determined in accordance with one of said overhead and near lighting views; and determining a fire condition if FG(tk) exceeds a second threshold selected in accordance with one of said overhead and near lighting views. The opposite lighting view and at least one of said overhead and near lighting views may be used in determining whether a fire condition exists. A fire condition may be determined if at least one of said first and second thresholds are exceeded. The method may also include: determining whether Fading(t₀) is non zero; and if Fading(t₀) is not non-zero, determining existence of a system problem. The method may also include: in response to determining an existence of a system problem, selecting a different feature for use with said current frame other than Fading(t_k).

[0019] In accordance with another aspect of the invention is a computer program product that distinguishes a smoke condition from a dust condition comprising executable code that: obtains a first frame of video data; determines at least one feature of a portion of said first frame; and determines a smoke condition if a threshold value for said at least one feature is crossed, and, in the presence of a dust condition, said threshold value is not crossed. The computer program product may include executable cod that sets a fire indicator if said threshold value is crossed. The computer program product may also have said at least one feature be an average absolute gray level intensity represented as:

$\mu \left(t\right)=\frac{1}{N_m\times M_m}\sum _{i=1}^{N_m}\sum _{j=1}^{M_m}P\left(i,j;t\right)-P_{\mathrm{Ref}}\left(i,j;t_0\right)$

where: P(i, j; t) is a pixel value at location (i,j) and time t; P_Ref(i, j; t₀) is a pixel value at location (i,j) of a reference frame at time 0; and N_m and M_m are dimensions of said portion. The first frame may be obtained using one of: an overhead lighting view in which only overhead controlled lights are turned on, a diagonal lighting view in which only a light source opposite a camera obtaining said first frame is on, and a pilot lighting view in which only a light source co-located with a camera obtaining said first frame is on. If said threshold value is crossed, a smoke condition may be determined. The computer program product may also include executable code that determines said threshold value in accordance with a vibration level of noise in a system. The computer program product may also include: executable code that selects said threshold in accordance with criteria that all fire conditions are properly classified and an acceptable rate of dust cases are improperly classified. The criteria may include detecting a fire condition prior to a selected conventional smoke detector. The feature may have a larger value in a smoke condition than a dust condition. The feature may include a plurality of features each associated with at least one threshold. The plurality of features may include at least two of: a mean intensity value of an average pixel intensity for all pixels in said portion; a maximum pixel intensity value of a maximum pixel intensity for all pixels in said portion; a number of pixels in a bright region, said number representing a number of pixels in said portion that are one of: greater than a threshold intensity, and greater than a threshold intensity and connected to a minimum number of neighboring pixels; a mean norm of intensity gradient vector for all pixels in said portion; and a norm of the second order moment matrix calculated over all pixels in said portion. A fire condition may be determined if any one of the plurality of features crosses an associated threshold. The computer program product may also include executable code that determines different features used in detecting a smoke condition in accordance with a particular lighting view of a frame. The mean intensity value may be determined for a frame obtained in one of a diagonal and overhead lighting view, said maximum pixel intensity is determined for a frame obtained in one of diagonal view, and said size of a brightness region is determined for a frame obtained in one of a diagonal and overhead lighting views. A different threshold may be used for each combination of feature and lighting view for at least one feature. The plurality of features may include a feature that is a mean intensity represented as:

$\mathrm{mean}\left(P\right)=\frac{1}{\mathrm{MN}}\sum _{i=1}^M\sum _{j=1}^NP\left(i,j\right)$

where: M represents a number of pixel rows in said portion; N represents a number of pixel columns in said portion; P is said portion of a frame; and P(i,j) is a pixel intensity or gray level at pixel position i,j in P. The plurality of features may include a second feature that is a mean norm of a gradient vector represented as:

$\mathrm{mgrad}\left(P\right)=\frac{1}{\mathrm{MN}}\sum _{i=1}^{M-1}\sum _{j=1}^{N-1}\sqrt{{G_1\left(i,j\right)}^2+{G_2\left(i,j\right)}^2}$

where G₁ and G₂ are given by:

$G_1\left(i,j\right)=X\left(i+1,j\right)-X\left(i,j\right)$ $G_2\left(i,j\right)=X\left(i,j+1\right)-X\left(i,j\right).$

A value of said second feature may decrease as an image decreases in sharpness of detail. A third feature of said plurality of features may be a norm of a second order moment matrix represented as:

$\mathrm{norm}2\left(P\right)=\mathrm{norm}\left(\left[\begin{array}{cc}\mathrm{SM}20\left(P\right)& \mathrm{SM}11\left(P\right)\\ \mathrm{SM}11\left(P\right)& \mathrm{SM}02\left(P\right)\end{array}\right]\right)$

where second order moments S M 20, S M 02 and S M 11 are defined via equations:

$\begin{array}{c}\mathrm{Xcenter}\left(P\right)=\frac{1}{\mathrm{MN}}\sum _{i=1}^M\sum _{j=1}^N\mathrm{iP}\left(i,j\right)\\ \mathrm{Ycenter}\left(P\right)=\frac{1}{\mathrm{MN}}\sum _{i=1}^M\sum _{j=1}^N\mathrm{jP}\left(i,j\right)\\ \mathrm{SM}20\left(P\right)=\frac{1}{\mathrm{MN}}\sum _{i=1}^M\sum _{j=1}^N{\left(i-\mathrm{Xcenter}\right)}^2P\left(i,j\right)\\ \mathrm{SM}02\left(P\right):=\frac{1}{\mathrm{MN}}\sum _{i=1}^M\sum _{j=1}^N{\left(j-\mathrm{Ycenter}\right)}^2P\left(i,j\right)\mathrm{and}\\ \mathrm{SM}11\left(P\right)=\frac{1}{\mathrm{MN}}\sum _{i=1}^M\sum _{j=1}^N\left(i-\mathrm{Xcenter}\right)\left(j-\mathrm{Ycenter}\right)P\left(i,j\right).\end{array}$

The plurality of features may include a fourth feature that is the size of a bright region represented as:
brsize(P)=Σ_i=1^MΣ_j=1^NT(i,j), where binary thresholded image T is defined through at least an intensity threshold. The computer program product may also include executable code that determines said bright region in accordance with an intensity threshold and a connectivity factor. A fire condition may be determined if any one value of any one feature crosses a threshold. The computer program product may also include executable code that determines said first, said second and said fourth features if said first frame is obtained using a diagonal lighting view in which said frame is obtained using only an opposing camera light; and executable code that determines said first, said second and said third features if said first frame is obtained using an overhead lighting view in which said frame is obtained using only overhead lights. At least one feature may be determined for multiple windows of said portion. At least one of said plurality of features may be determined using a relative change expressed as a ratio of a first value to a second value, said first value representing said feature for said portion and said second value representing a steady state feature value. Each of said multiple windows is determined in accordance with a location of at least one light relative to location and field of view of a camera acquiring frames. The plurality of features may include a fifth feature that is a maximum pixel intensity value represented as:

$\max \left(P\right)=\underset{\underset{j=1,N}{i=1,M}}{\max }P\left(i,j\right)$

where: M represents a number of pixel rows in said portion; N represents the number of pixel columns in said portion; P is said portion; and P(i,j) is a pixel intensity or gray level at pixel position i,j in said portion. The at least one feature may be a rate of change feature based on an average absolute gray level, wherein the rate of change is calculated as:

${\delta }_{\mu }=\frac{\mu \left(t\right)-\mu \left(t-\Delta t\right)}{\Delta t}$

where μ(t) and μ(t−μt) are values of an average absolute intensity feature at times t and t−Δt, respectively, and wherein said average absolute gray level is represented as:

$\mu \left(t\right)=\frac{1}{N_m\times M_m}\sum _{i=1}^{N_m}\sum _{j=1}^{M_m}P\left(i,j;t\right)-P_{\mathrm{Ref}}\left(i,j;t_0\right)$

where: P(i, j; t) is a pixel value at location (i,j) and time t; P_Ref(i, j; t₀) is a pixel value at location (i,j) of a reference frame at time 0; and N_m and M_m are dimensions of said portion. A value of said rate of change feature may be higher for a smoke condition than for a dust condition.

[0020] In accordance with another aspect of the invention is a computer program product that detects a fire condition comprising executable code that: determines a lighting view associated with a frame; if said lighting view is an opposite lighting view, determines a number of pixels associated with a portion of a frame in accordance with a lighting view of said frame, said number of pixels representing a number of pixels in said portion which increase or decrease by an amount exceeding a brightness threshold with respect to a reference frame, each of said number of pixels being connected to a minimum number of neighboring pixels in terms of space, and determining a ratio of said number of pixels for a current frame with respect to said number of pixels of a reference frame; and if said ratio exceeds a first threshold, determines that a fire condition exists, said first ratio being determined in accordance with the opposite lighting view. For a current frame at a time tk, said ratio may be represented as a percentage of fading with an opposite view configuration:

$\mathrm{Fading}\left(t_k\right)=100\times \frac{\sum _{m=I}^{M/S_s}\sum _{n=I}^{N/S_s}\left\{\left(\sum _{i=-1}^{i=1}\sum _{j=-1}^{j=1}\begin{array}{c}P\left(S_sm+i,S_sn+j;t_k\right)-\\ P_{\mathrm{Ref}}\left(S_sm+i,S_sn+j;t_0\right)\end{array}>D_0\right)>C_0\right\}}{\mathrm{Fading}\left(t_0\right)}\%$

wherein Fading(t₀) is a number of bright and connected pixels generated by an opposite light of a reference image represented by:

$\mathrm{Fading}\left(t_0\right)=\sum _{m=I}^{M/S_s}\sum _{n=I}^{N/S_s}\left\{\left(\sum _{i=-1}^{i=1}\sum _{j=-1}^{j=1}P_{\mathrm{Ref}}\left(S_sm+i,S_sn+j;t_0>D_0\right)\right)>C_0\right\}\mathrm{such}\mathrm{that}\mathrm{Fading}\left(t_0\right)$