Image processing warning system

[0001] The present invention relates generally to alerting and warning systems, and more specifically to a method and apparatus for capturing and processing images from a portion of a system for monitoring that portion of the system for conditions warranting an operator warning.

[0002] Alerting and warning apparatus are used in a variety of applications to provide warnings to user or operators of a system, for example, a vehicle, an aircraft, a ship, a railroad, a building, or the like of impending operator concerns such as impending mechanical failures, safety issues, and the like. For instance, typical aircraft systems may provide a variety of alerts, cautions and warnings for such operator concerns as engine fire, impending collision, stalling, or low hydraulic pressure. Similarly, automobile systems may provide warnings for open doors, engine overheating, poor traction, or low fuel. However, the operator of such system cannot view the affected portion of the system to determine if the warning is in fact true, or to determine whether the condition causing the warning warrants the operator's immediate attention and action. Further, most existing alerting and warning systems typically do not include internal redundancy that is capable of reliably indicating whether the warning provided to the operator is in fact a true warning condition or is instead a failure of the warning system itself.

[0003] Accordingly, the present invention is directed to a method and apparatus for capturing and processing images from a portion of a system in order to detect a warning condition so that a warning may be provided to users or operators of the system.

[0004] In accordance with one aspect of the invention, an apparatus for providing warnings to a user of a system upon detection of a warning condition within the system is disclosed. In an exemplary embodiment, the apparatus includes at least one image capture device for monitoring at least a portion of the system. The image capture device is capable of capturing an image, for example, a continuous video, of the portion of the system being monitored. An image processing assembly processes the image captured by the image capture device in real time for determining if the warning condition exists. Upon determining that the warning condition exists, the image processing assembly causes a warning device to provide a warning to the user. The captured image (video) may then be displayed to the user or operator if desired.

[0005] In accordance with a further aspect of the invention, a method for providing warnings to a user of a system upon detection of a warning condition is disclosed. In an exemplary embodiment, the method is comprised of the steps of capturing an image of at least a portion of a system; processing the captured image in real time for determining if the warning condition exists; and, if the warning condition is determined to exist, providing a warning to the user. In embodiments of the invention, processing of the captured image comprises correlating the captured image with a reference wherein the difference between the captured image and the reference is used to determine if a warning condition exists.

[0006] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention claimed. The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate an embodiment of the invention and together with the general description, serve to explain the principles of the invention.

[0016] Reference will now be made in detail to the presently preferred embodiments of the invention, examples of which are illustrated in the accompanying drawings.

[0017] Referring now to FIG. 1, an exemplary image processing warning system in accordance with the present invention is described. Typically, the image processing warning system 100 will function as an integrated sub-system of a master system [hereinafter system] such as a vehicle, an aircraft, a ship, a subway or railroad train, a railroad system, a building, or the like. The image processing warning system 100 is comprised of apparatus utilizing image capture and processing for providing warnings to a users or operators of the system upon detection of a condition warranting a warning [hereinafter a “warning condition”] within at least a portion of that system.

[0018] As shown in FIG. 1, the image processing warning system 100 includes one or more image capture devices 102 for monitoring at least a portion of the system. In exemplary embodiments of the invention, the image capture device may be comprised of a video camera such as an analog video camera, a digital video camera, digital camera, or the like. Preferably, the image capture device 102 is capable of capturing an image of the portion of the system being monitored. This captured image may be comprised of a still image, a continuous video, periodically sampled frames of a continuous video, or the like. Further, the captured image may be comprised of an optical image, or alternately images produced using other wavelengths (e.g., infrared, ultraviolet, other non-visible wavelength electromagnetic radiation, and the like). Additionally, the captured image may be comprised a fusion of images captured by two or more image capture devices.

[0019] An image processing assembly 104 processes the image captured by the image capture device 102 in real time for determining if a warning condition exists within the portion of the system being monitored. In the exemplary embodiment shown, the captured image (e.g., a continuous video feed, or sampled frames of a such a video feed) may be correlated with a stored reference 108 of the portion of the system being monitored using a comparison algorithm 110. In one embodiment, this reference 108 may comprise an image of the portion of the system being monitored under normal conditions (i.e., where no warning condition exists). Alternately, the reference may comprise an image of the portion of the system being monitored where a warning is desired (i.e., where the warning condition does exist). In other embodiments of the invention, the reference may be comprised of one or more parameters that are expected to be found in the captured image under normal conditions. For example, under normal conditions the captured image (or portions thereof) may be expected to have a particular parameter or attribute such as a certain color or colors, shape, texture, temperature (in an infrared image), or the like.

[0020] The comparison algorithm 110 correlates the captured image with the stored reference 108 to determine if the captured image deviates from the reference 108. When the deviation of the captured image from the reference reaches a predetermined threshold, a warning condition is determined to exist. Upon determining that a warning condition exists, the image processing assembly 104 causes a warning device 106 to provide a warning to the user. The captured image (e.g., real time continuous video) may further be displayed to the user or operator by the warning device 108 via a display device 110 if so desired by the user or operator of the system.

[0021] In exemplary embodiments, the system being monitored by the image processing warning system 100 of the present invention may also include one or more secondary warning systems 114 suitable for monitoring the portion of the system in order to provide warnings to the operator of the system. The image processing warning system 100 of the present invention functions to supplement or enhance such secondary warning systems 114 to provide redundant warnings of system failure or to provide the operator with desired additional information (for example, real time visual inspection of the portion of the system being monitored via a live video feed) should such a secondary warning system initiate a warning. Additionally, the image processing warning system 100 of the present invention may monitor other physical parameters provided by such secondary warning systems 114 for aiding processing of the captured image to determine if a warning condition exists. For example, such parameters may be fused with the captured images and compared to a stored reference providing normal values for the parameters.

[0022] Turning now to FIG. 2, a method for providing warnings to a user of a system upon detection of a warning condition is described. In an exemplary embodiment, the method 200 employs apparatus such as the apparatus described in the discussion of FIG. 1 to continuously monitor at least a portion of a system for a condition warranting a warning (a warning condition) to the users or operators of that system and then automatically initiates that warning if the warning condition is detected. As shown in FIG. 2, an image of the portion of the system being monitored is captured at step 202. In exemplary embodiments of the invention, capture of the image at step 202 may, for example, involve continuously monitoring the portion of the system utilizing a digital video camera to provide a continuous video that is provided to an image processing assembly. The captured image is then processed to determine if a warning condition exists at step 204. For example, in exemplary embodiments, the captured image (e.g., the continuous video feed or, alternately, periodically sampled frames of a continuous video feed) may be correlated or compared to a reference of the portion of the system using a comparison algorithm. A determination is next made as to whether a warning condition exists at step 206. In exemplary embodiments, wherein the captured image is compared to a reference, this determination may be made by ascertaining whether the captured image either matches the reference within a predetermined tolerance, or, alternately, deviates from the reference beyond a predetermined acceptable threshold. When a determination is made that no warning condition exists, monitoring of the portion of the system continues undisturbed and additional images of the portion of the system being monitored are captured and processed. However, when a warning condition is determined to exist, a warning is provided to the user or operator at step 208.

[0023] The user or operator may then be allowed to view the captured image. In exemplary embodiments, the user or operator may be queried to determine if he or she wishes to view the captured image of the portion of the system being monitored at step 210. For example, the warning may be provided by a multipurpose display (MPD) that may further display a prompt adjacent to the warning querying the user if the image is to be displayed (see FIGS. 3D and 3E). Alternately, the warning may be furnished by a warning light, warning horn, siren, or the like. If the operator chooses to view the image of the portion of the system in which a warning condition exists, the captured image is displayed at step 212. For example, when the warning is provided by a multipurpose display (MPD), the captured image may be displayed by the multipurpose display over or adjacent to the warning. Similarly, when another warning device, such as a warning light, siren or the like is employed, the image may be displayed by an associated display device.

[0024] If the user or operator chooses to not view the image (for example, if the user or operator is aware of the warning condition and has already taken steps to correct the warning condition such that viewing the captured image would only unnecessarily increase his or her workload), the warning may optionally be canceled at step 214 whereupon monitoring of the system may continue. If the warning condition is found to persist, the operator may further override or disable the warning, if such capability is deemed advantageous for the particular application. Similarly, upon viewing the captured image provided at step 212, the operator may determine that provision of the warning is no longer necessary. For example, the operator may have taken necessary remedial steps to correct the warning condition or there may be no corrective action possible. The operator may optionally cancel the warning at step 214, whereupon monitoring of the system may continue. Again, if the warning condition is found to persist, the operator may override or disable the warning, if such capability is deemed advantageous for the particular application. Finally, when the warning condition ceases to exist, the warning provided to the operator may also be automatically terminated at step 214 and monitoring of the system may continue.

[0025] It is anticipated that the image processing warning system of the present invention is suitable for use in a wide variety of systems including, but not limited to, aircraft and spacecraft, automobiles and like vehicles, ships and submarines, building systems, subway trains, railroad trains, railroad systems, patient monitoring systems utilized by health care providers, and security systems. Specific examples of systems employing the present invention are shown in FIGS. 3A through 7B. However, implementation of the image capture warning apparatus and method disclosed herein in other systems by those of skill in the art is anticipated and would not depart from the scope and spirit of the invention.

[0026] Referring now to FIGS. 3A through 3E, a system comprised of an aircraft 300 employing an exemplary image processing warning assembly or system 302 in accordance with the present invention for detecting and warning the flight crew of improper landing gear deployment and/or retraction is described. In FIGS. 3A through 3B, the aircraft 300 includes a landing gear bay 304 into which a landing gear 306 may be retracted during flight by an actuator or like retraction device (not shown). The landing gear bay 304 includes one or more doors 308 & 310 that enclose the landing gear 306 within the bay 304. An image capture device, such as digital video camera 312, of the image processing warning assembly 302 is positioned to monitor the landing gear bay 304 for proper extension and retraction of the landing gear 306. The video camera 312, or alternately, a second video camera, may also monitor the landing gear bay doors 306 for determining if they have properly opened and closed.

[0027] In the exemplary embodiment shown, the video camera 312 captures a continuous video of the landing gear bay 304, landing gear 306, and landing gear bay doors 308 & 310. This video is processed by the image processing assembly (see FIG. 1) in real time for determining if a landing gear warning condition exists so that appropriate warnings may be provided to the flight crew of the aircraft 300. For instance, as previously discussed, the video captured by video camera 312 may be correlated or compared with static reference images of the landing gear bay 304, landing gear 306 and landing gear bay doors 308 & 310 using a comparison algorithm to determine if the captured images deviate from the references. When the deviation of the captured images from the references reaches a predetermined threshold, a warning condition is determined to exist. Upon determining that a warning condition exists, the image processing assembly causes a warning to be provided to the flight crew. For example, as shown in FIG. 3D, a landing gear caution light 314 may be illuminated and/or a warning (for example, “WARNING FRT LND GR”) 316 may be displayed on a multipurpose display (MPD) 318 within the crew station of the aircraft 300.

[0028] In exemplary embodiments, the image processing warning assembly 302 is capable of detecting a variety of landing gear warning conditions for which advisory and cautionary warnings may be provided to the flight crew. For example, the image processing warning system 302 may be capable of detecting when the landing gear 306 is fully retracted as shown in FIG. 3A or fully extended as shown in FIG. 3C, and may provide an advisory warning of such events to the flight crew. Likewise, in exemplary embodiments, the image processing warning system 302 may be capable of detecting when the landing gear bay doors 308 & 310 are fully opened or closed, so as to provide an advisory warning of these events. Similarly, the image processing warning system 302 may be capable of detecting non-normal events such as a hung landing gear 306, as shown in FIG. 3B so that a warning, in this case a cautionary warning, may be provided to the flight crew.

[0029] The image processing warning system may further allow the flight crew to access video of the landing gear bay 304. For example, in the exemplary embodiment shown in FIG. 3D, the multipurpose display (MPD) 318 may further provide a prompt (“Display Video”) 320 adjacent to the warning (“WARNING FRT LND GR”) 316 querying the user if video of the landing gear bay 304 is to be displayed. A member of the flight crew, for example the pilot, copilot, or the like, may select a button 322 of the multipurpose display (MPD) 318 associated with the prompt 320 causing the video 324 to be displayed (FIG. 3E).

[0030] In exemplary embodiments of the invention, the aircraft 300 may also include mechanical warning systems suitable for monitoring the landing gear and landing gear bay doors to provide a warning to the flight crew of improper landing gear or landing gear bay door position. For example, as shown in FIGS. 3A through 3C, the landing gear bay 304 may include a mechanical sensor such as proximity switch 326 positioned to sense proper landing gear retraction. Similarly, additional proximity switches or like mechanical sensors (not shown) may be positioned to sense proper landing gear extension, landing gear bay door closure, and the like. When one or more of the switches detects that the landing gear or a landing gear bay door is not in the desired position, the warning system provides a warning to the flight crew (for example, a landing gear position warning light 314 or a warning message on the multipurpose display (MPD) 318). The image processing warning system 302 may thus function to supplement or enhance such mechanical warning systems to provide redundant warnings of landing gear failure or to provide the flight crew with desired additional information (for example, real time visual inspection of the landing gear bay 304 via a live video feed) should such a mechanical system initiate a warning.

[0031] Referring now to FIG. 4, in an exemplary embodiment, the image processing warning system of the present invention may similarly be utilized by an aircraft 400 for detecting and warning the flight crew or maintenance personnel of abnormal tire conditions such as excessive wear or damage to the tread 402 of the landing gear tires 404. As shown in FIG. 4, an image capture device (for example, a digital video camera, digital camera, or the like) 406 is positioned within the landing gear bay 408 for monitoring the landing gear tire 404 upon its retraction within the bay 408. In accordance with the present invention, the image capture device captures an image of the landing gear tire's tread 402 that is processed by the image processing assembly of the image processing warning system (FIG. 1) for providing a warning if excessive wear in 412 or damage to 414 the tread 402 is found to exist. For example, in an exemplary embodiment, the captured image may be correlated with a reference of an unworn tire tread using a suitable comparison algorithm, whereupon excessive tread wear 412 or damage 414 to the tire 404 would cause a warning to be provided to the flight crew or other maintenance personnel within the crew station of the aircraft 400. In this manner, the image processing warning system may furnish a go/no go indication of tire tread safety. The image processing warning system may further allow the flight crew or maintenance personnel to access the tire tread image to provide in-crew station inspection of the tire 404, or may transmit the captured tire tread image to a central maintenance facility, group (an airline), or agency (such as the Federal Aviation Administration (FAA)) via a wireless radio frequency transmission or a wired ground network connection such as an Intranet or the Internet, or the like for inspection by maintenance personnel, airline or governmental inspectors, or the like.

[0032] Referring now to FIGS. 5A and 5B, a system comprised of an aircraft 500 employing an exemplary image processing warning assembly 502 in accordance with the present invention for detecting and warning the flight crew of wing surface irregularities (for example, ice buildup on the aircraft's wing or other control surfaces 504) is described. As shown, an image capture device (for example, a digital video camera, digital camera, or the like) 506 of the image processing warning assembly (see FIG. 1) is positioned on the aircraft 500 for monitoring the leading edge surfaces of the wing 504. In exemplary embodiments, the image capture device 506 may be capable of detecting electromagnetic radiation within the infrared range for sensing temperature differences along the wing's surface. In this manner, the image capture device 506 may more accurately detect the buildup of ice 508 (which may be transparent or translucent) on the wing surface to provide an image of the wing surface showing any ice buildup. The image captured by the image capture device 506 is then processed by the image processing assembly of the image processing warning system (FIG. 1) to determine whether ice 508 has accumulated on the wing's surface beyond an acceptable amount. For instance, the captured image may be correlated with a reference of the wing's surface without ice buildup. The amount of ice buildup on the wing surface 504 may be measured from the difference between the correlated captured image and the reference. Where the amount of ice 508 on the wing surface exceeds a predetermined threshold level as shown in FIG. 6B, a warning condition is determined to exist, and a warning may be provided to the flight crew (for example, a wing icing warning light or a warning message on the multipurpose display (MPD) as shown in FIGS. 3D and 3E). The image processing warning system may further allow the flight crew to access the captured image to provide in-crew station inspection of the wing surfaces 504, or may transmit the captured image to a central facility such as the flight control tower or ground traffic control tower, or alternately a regulatory body via wireless radio frequency transmission, network connection, or the like, for inspection by air traffic control personnel, ground traffic control personnel, maintenance personnel, airline or regulatory agency inspectors, or the like.

[0033] Referring now to FIGS. 6A and 6B, a system comprised of an aircraft 600 employing an exemplary image processing warning assembly 602 in accordance with the present invention for detecting and warning the flight crew of warning conditions such as, for example, smoke, fuel leaks, fire within the engine compartment 604 or excessive vibration of the engine 606 is described. As shown, one or more image capture devices (for example, digital video cameras, digital cameras, or the like) 608 of the image processing warning assembly (see FIG. 1) are positioned within the engine compartment 604 for monitoring the aircraft's engine 606. In exemplary embodiments, the image capture device 608 may be capable of sensing electromagnetic radiation within the infrared range for detecting differences in temperature within the engine compartment 604. The image captured by the image capture device 608 is then processed by the image processing assembly of the image processing warning system 602 (see also FIG. 1) for determining if there are any outward indications that the engine 606 may be malfunctioning such as smoke, fire, fuel leaks, hydraulic leaks, engine oil leaks, or the like 610. For instance, the captured image may be correlated with a reference of the engine compartment under normal conditions. The presence of warning conditions within the engine compartment may in this manner be detected from the difference between the correlated captured image and the reference. If a warning condition is determined to exist, a warning may be provided to the flight crew (for example, a engine warning light or a warning message on the multipurpose display (MPD) shown in FIGS. 3D and 3E). The image processing warning system may further allow the flight crew to access the captured image to provide in-crew station inspection of the engine compartment 604 and engine 606.

[0034] In exemplary embodiments of the invention, the aircraft 600 may also include secondary warning systems suitable for monitoring the engine compartment for a specific warning condition and providing a warning to the flight crew if this condition is detected. For example, as shown in FIGS. 6A and 6B, the engine compartment 604 may include one or more engine fire detectors 612 positioned to sense a fire within the engine bay 604. When one or more of the sensors 612 senses fire within the engine compartment, the warning system provides a warning to the flight crew (for example, a fire warning light or a warning message on the multipurpose display (MPD)) and may also activate an automatic fire suppression system. The image processing warning system 602 of the present invention may thus function to supplement or enhance such warning systems to provide a redundant warning of warning conditions within the engine compartment 604 and to provide the flight crew with needed additional information (real time visual inspection of the engine compartment 604 via a live video feed) should such a system cause a warning to be provided.

[0035] Referring now to FIGS. 7A and 7B, an image processing warning system in accordance with an exemplary embodiment of the present invention is shown wherein the system is used to provide monitoring of an area such as the entrance to an aircraft, the entrance to the crew station of the aircraft, the aircraft's flight deck, the entrance to a secured area in a building, a cabin of a subway train, a room in a museum, or the like for security purposes. For example, as shown in FIGS. 7A and 7B, an aircraft 700 may utilize the image processing warning system 702 of the present invention to monitor an area 704 within the aircraft such as the entrance 706 to the aircraft 700 or, alternately, the entrance to the crew station of the aircraft 700, the aircraft's flight deck, or the like. As shown, one or more image capture devices (for example, digital video cameras, digital cameras, or the like) 708 of the image processing warning assembly 702 are positioned within the area 704 for monitoring the entryway 706. The image captured by the image capture device 708 is then processed by the image processing assembly of the image processing warning system 702 (see also FIG. 1) for determining if a warning condition, for example, the presence of an intruder, exists within the area 704. For instance, the captured image may be correlated with a static reference of the area to determine if an intruder 710 has entered the area 704. When an intruder 710 is detected within the monitored area 704, as shown in FIG. 7B, a warning condition is determined to exist, and a warning may be provided to the flight crew or security personnel in a central facility via wireless radio frequency transmission, network connection, or the like. The image processing warning system may further allow the flight crew or security personnel to access the captured image to provide verification of the warning.

[0036] It is believed that the present invention and many of its attendant advantages will be understood by the foregoing description, and it will be apparent that various changes may be made in the form, construction and arrangement of the components thereof without departing from the scope and spirit of the invention or without sacrificing all of its material advantages. The form herein before described being merely an explanatory embodiment thereof, it is the intention of the following claims to encompass and include such changes.