Integrated collision avoidance system for air vehicle

[0001] This is a regular application of a provisional application having an application No. 60/874,708 and filing date of Dec. 12, 2006.

[0002] This invention was made with Government support under contract NNL05AA96P awarded by NASA. The Government has certain rights in the invention.

[0003] 1. Field of Invention

[0004] The present invention relates to an integrated positioning/ground proximity warning method and system, and more particularly to a low cost, lightweight and vision based miniaturized Integrated Collision Avoidance Enhanced GNC Micro-System for Smart Air Vehicles and to convert the invention successes into a commercial line of a “low-cost, small, lightweight GNC Micro-System for Aircraft Collision Avoidance.

[0005] 2. Description of Related Arts

[0006] Current properly instrumented aircraft, operating individually or in teams with manned vehicles and unmanned vehicles, in the same airspace for a specific mission, are required to perform many missions. Safety of the aircraft is of paramount importance in all airspace environments. Avoiding collisions between aircraft and obstacles is a basic necessity for various operations. A real-time collision avoidance subsystem detects obstacles and provides warning messages to the pilot or controls the aircraft so that collisions with obstacles are avoided.

[0007] Adverse flight conditions make the autonomous avoidance of collisions a formidable problem for aircraft. First, collision avoidance for aircraft is associated with their sensors environment for detection of objects and for prediction of potential collision within their flight paths. Second, the collision avoidance should provide the aircraft with the capability of escaping from possible collisions. This means that an optimal escape trajectory is required for operational safety of the aircraft.

[0008] NASA is seeking advanced technologies to achieve smart, adaptive aerospace vehicles with intelligence. The innovative Integrated System is related to NASA's interests because based on vision it can become aware of potential collisions with terrain and obstacles and react rapidly to provide guidance for avoiding those disasters while maintaining the flight mission.

[0009] The objective of this invention is to demonstrate a low cost, lightweight and vision based miniaturized Integrated Collision Avoidance Enhanced GNC Micro-System for Smart Air Vehicles and to convert the invention successes into a commercial line of a “low-cost, small, lightweight GNC Micro-System for Aircraft Collision Avoidance: Vision Based Miniaturized Aircraft Integrated Collision Avoidance System” product. This product is referred as “The Integrated System” in this invention. The Integrated System takes advantage of the American GNC Corporation (AGNC), our company, coremicro AHRS/INS/GPS Integration Unit and is based on AGNC patents to achieve a vision based integrated, guidance, navigation, and control (GNC) micro-system for aircraft and UAVs, which is capable of assisting aircraft pilots to avoid approach and collision with ground/water and other near objects in flight, thus, improving safety, reliability and cost effectiveness. The various data from the coremicro AHRS/INS/GPS Integration Unit, terrain data base, and object detection sensors are processed to produce collision warning audio/visual messages and collision detection and avoidance of terrain and obstacles through generation of guidance commands in a closed-loop system.

[0010] This invention investigated the vision based object detection and tracking system. A statistical model based object recognition algorithm was designed which successfully classified different terrain, such as, mountains, trees, etc. A ranging method with stereo cameras was demonstrated. An integrated navigation/object detection and tracking system based on the AGNC coremicro AHRS/INS/GPS Integration Unit 1010 and cameras is demonstrated by the robot platforms. This robot demo system provides navigation and collision avoidance simultaneously.

[0011] AGNC low cost, light weight, vision based miniaturized, Integrated Collision Avoidance Enhanced GNC System is utilized for smart air vehicles, based on coremicro AHRS/INS/GPS Integration Unit and patents. The Integrated System extends AGNC's existing product line and utilizes its patented technologies to direct aircraft in and out of austere airfields and landing pads located in remote deserts, mountains, and forested areas and to lead pilots around obstacles to avoid the terrain, water, and other near obstacles, such as towers, power lines, structures, and trees, and to minimize the annual loss of valuable aircraft and lives.

[0012] Integration of the AGNC products is directed at a unique solution for vision based collision avoidance for general aviation aircraft and unmanned air vehicles, such as, (1) Statistical model based algorithm is designed for terrain and obstacle recognition, and the recognition results are used to aid the decision of the Ground Proximity Warning System (GPWS); (2) Stereo cameras are used to provide range and 3D measurements of a target, which are used for positioning; (3) As a necessary step toward instrumenting the small, lightweight, Vision Based GNC Micro-System into a particular Aircraft Integrated Collision Avoidance System for flight demonstrations and tests, a flexible infrastructure utilizing the AGNC GPS/IMU REALTIME SIMULATOR for closed-loop testing and evaluation of many components of the GNC system has been designed for the Aircraft Integrated Collision Avoidance System; (4) Different modes are designed to fuse the information of the imaging sensors with that of other sensors for adaptive maneuvering target tracking and guidance, such as, sensor level fusion, track level fusion, and partially processed data fusion.

1.1 Identification of the Aircraft Collision Avoidance Problems

[0013] Collision is one of the common causes of severe aircraft accidents. An aircraft may collide with the terrain, another aircraft, towers, powerlines, etc. Although a standalone operated positioning systems, such as, existing civil GPWS and Enhanced Ground Proximity Warning System (EGPWS) have been designed to prevent crashes, they do not provide adequate warning messages of obstacles nor guidance around the obstacles, and are generally installed in a large aircraft. It is urgent to enhance the aviation safety with low-cost, small-sized GNC systems.

[0014] An independent Ground Proximity Warning System became mandatory by the FAA in 1974 for commercial turbojet/turboprop aircraft with more than 10 passenger seats which fly in U.S. airspace, traditional ground proximity warning systems have dramatically reduced the number of Controlled Flight Into Terrain accidents among airlines, by monitoring the aircraft's height above ground as determined by a radio altimeter. Although traditional ground proximity warning systems have saved thousands of lives, they still fail to prevent all Controlled Flight Into Terrain accidents. A traditional GPWS can not really tell what is ahead of the airplane. It can only look down, as the radio altimeter measures the distance from the airplane to the ground. The GPWS is used to predict if a potential terrain problem lies ahead by tracking information from the recent past and projecting any trend into the future. This can warn a pilot, driver or operator when the airplane is flying towards terrain that has a rising slope, but it can not warn in time to avoid, for example, a sheer cliff or extremely steep slope.

[0015] A positioning system is used to provide position, velocity, altitude, altitude rate information, and etc., for an aircraft flight control and management system. Traditional positioning equipment in a civil aircraft includes global positioning system (GPS), inertial navigation system (INS), radio navigation system, and integrated positioning system. Although the INS used in the traditional positioning systems can provide very high accuracy, its cost is huge and its size is large. The availability of low cost inertial sensors, such as, the AGNC (MEMS) coremicro IMU inertial sensors, provide an obvious choice for implementing low cost, highly accurate, continuous positioning of a vehicle by employing a low cost strapdown inertial system with the compensation of an external sensor. The global positioning system receiver is an ideal external sensor for an inertial navigation system.

[0016] To enhance the performance of the GPWS and EGPWS based on standalone GPS/INS positioning systems, it is necessary to incorporate with the Vision Based Miniaturized Aircraft integrated Collision Avoidance System to detect the objects nearby, and avoid collision with these objects by some algorithms. Some sensors, such as, video camera, infrared camera, radar, ladar, are fused to detect nearby objects.

1.2 Functions of the Vision Based Miniaturized Aircraft Integrated Collision Avoidance System

[0017] The function of the vision based aircraft integrated positioning/collision avoidance system is to avoid collision with terrain and obstacles. The vehicle position/velocity/altitude information is provided by the positioning sensors, such as, GPS, INS, baro altimeter, radio altimeter, etc. The potential collision with terrain is predicted by using the vehicle state information and surrounding terrain height from the onboard terrain database.

[0018] Another method for terrain collision avoidance is to detect the terrain, such as, mountains and trees, by using the vision sensors. A statistical model based object recognition algorithm was designed which successfully classified different terrain, such as, mountains, trees, etc. A ranging method with stereo cameras was demonstrated. The vision sensor provides more information for the ground proximity warning system.

[0019] To avoid collision with obstacles, the Integrated System dynamically assesses its own time varying temporal spaces about itself using measurements obtained from the object detection system. Two space areas about the aircraft are defined to trigger Collision Avoidance Action,

[0020] A: Safe space. The aircraft may continue its mission.

[0021] B: Unsafe space. The unsafe space represents the area, which if violated by the obstacle, requires the aircraft to perform an evasive maneuver thus, mitigating a potential collision.

[0022] Radar and vision sensors are used to detect and track the obstacles nearby. The collision detection processing module continues to track the aircraft and predict the projected trajectory of the obstacle. Based on the predicted trajectory and maneuver capability of the aircraft, the collision avoidance guidance modules decide to perform the maneuver type and generate the escape trajectory.

[0023] Many different types of evasive maneuvers are investigated and evaluated for safety and integrity, such as horizontal maneuvers, vertical maneuvers, “inward”, “outward”, pull-up, dive, and inverted pull down.

1.3 Significant Improvement of the Vision Based Miniaturized Aircraft Integrated Collision Avoidance System

[0024] Compared with the conventional positioning and ground proximity warning systems, the enhanced vision based integrated collision avoidance system has a lot of advantages, such as, enhanced ground proximity warning ability by using a worldwide digital terrain database and the vision based terrain recognition system, higher positioning accuracy, higher obstacle avoidance ability by using fusion of different object detection sensors, such as, camera, radar, etc. The vision sensors used in the invention provide more information for the Integrated System, such as, terrain recognition, and ranging of terrain and obstacles, which play an important role in the improvement of the integrated collision avoidance system.

1.3.1 Performance Improvement of EGPWS by the Integrated System

[0025] Terrain airplane accidents have been a worldwide problem. The available improved ground proximity warning system, which is called the Enhanced Ground Proximity Warning System, is the result of an effort to further reduce the controlled flight into terrain risk.

[0026] Our first method of improving the EGPWS is to use a worldwide digital terrain database. Also, the enhanced ground proximity warning system adds two other enhancements to the traditional ground proximity warning system:

[0027] 1. It can provide the flight crew a map-like display of nearby terrain.

[0028] 2. It sounds an audible alert approximately one minute's flight time or more away from terrain. Traditional ground proximity warning systems typically sound a warning from a few seconds to about 30 seconds from terrain, but average 10 to 15 seconds.

[0029] The enhanced ground proximity warning system computer uses information provided by the onboard navigation system and terrain database. The enhanced ground proximity warning system computer uses aircraft position, performance, and configuration data to calculate an envelope along the projected flight path of the aircraft and compares that to the terrain database. Since the enhanced ground proximity warning system display can show nearby terrain, pilots are much less likely to continue flying toward that terrain.

[0030] Our second method of improving the EGPWS is to use a vision based terrain recognition and ranging algorithm. A statistical model based terrain recognition algorithm is designed. This algorithm successfully recognizes different terrain, such as, mountains, trees, etc. Furthermore, a stereo camera is used to detect the range between the aircraft and the terrain. Based on the aircraft states (position/velocity/altitude), and the terrain (mountains/trees) pixel coordinates in the aerial images and the range between the aircraft and the terrain, the EGPWS detects the potential collision with the terrain.

1.3.2 Performance Improvement of Aircraft Positioning by the Integrated System

[0031] Conventional integrated global positioning/inertial navigation systems integrate the information from an inertial measurement unit and a global positioning system receiver to obtain improved navigation solution. Conventional GPWS uses the position information provided by the conventional integrated global positioning/inertial navigation systems or stand-alone global positioning systems and the information from a radio altimeter and a baro altimeter and a terrain database to solve the ground proximity warning problem. The process for both conventional integrated global positioning/inertial navigation systems and conventional ground proximity warning systems is implemented in software in microprocessors.

[0032] As advances in speed and memory of microprocessors occur, it is possible to implement an integrated positioning and ground proximity warning system. Furthermore, an integrated process for position solution and ground proximity warning solution can provide better performance than an independent positioning system and an independent ground proximity warning system, because the information from the vision sensors, radio altimeter, the baro altimeter, and the terrain database which are employed by conventional GPWS has potential capability for improving the accuracy and reliability of conventional positioning systems, and improved position information in turn improve the performance of conventional ground proximity warning systems.

1.3.3 Performance Improvement of Midair Collision Avoidance by the Integrated System

[0033] As many of the aircraft operating in the same airspace fly in close formation and execute highly dynamic maneuvers, pilots have routinely identified a midair collision between aircraft as a most likely mishap. In addition, it is possible to obtain an optimal collision avoidance solution with a near object by further incorporating an object detection system with the results of the integration of the ground proximity warning system and multiple navigation sensors.

[0034] The performance improvement of midair collision avoidance is realized by the following methods: (1) higher positioning accuracy by the Integrated System; (2) detection and tracking by fusion of a stereo camera and other object detection sensors; (3) data link between aircraft.

1.3.4 Other Advantages of the Integrated System Over Traditional Independent Systems

[0035] In order to maximize safety of the vehicle operation for future commercial application, it is highly desirable for an Integrated System capable of providing optimal navigation solution and optimal ground/water/object collision avoidance capabilities.

[0036] Accordingly, compared to existing separate processes and systems for positioning and ground proximity warning systems, it is the objective of the Integrated System to provide an integrated method and system for positioning and preventing Controlled Flight Into Terrain flight accidents for aircraft to obtain improved performance, such as navigation accuracy, high degree of tolerance to loss of a global positioning system signal, global positioning system integrity monitoring, precise and reliable ground proximity warning information, and to make air travel safer in increasingly busy skies, as follows:

[0037] 1. The performance of an integrated positioning system and the ground proximity warning system is unattainable in the two systems alone.

[0038] 2. The positioning accuracy of the Integrated System is higher than the conventional integrated global positioning system/inertial navigation system alone.

[0039] 3. The Integrated System has the ability to detect the malfunction of the global positioning system satellite.

[0040] 4. Prompt and accurate ground proximity warning message is available, due to more accurate positioning solution provided by the Integrated System.

[0041] 5. The Integrated System reduces false ground proximity warning probability, due to a more accurate positioning solution provided by the Integrated System.

[0042] 6. Compared with the conventional enhanced ground proximity warning systems, in the Integrated System, an external navigation system is not required to support the ground proximity warning solution. This is especially affordable for small commercial aircraft vehicles.

[0043] The Integrated System provides a positioning and proximity warning method and system thereof, which are an integrated method and system for positioning and proximity warning for a vehicle in air, land, and water, which uses information from a global positioning system, an inertial navigation system, a baro altimeter, a radio altimeter, a terrain database, a vehicle control and management system, and an object detection system.

[0044] As compared with existing EGPWS, the innovative Integrated System affordably provides integral combination of not only EGPWS function, but also:

• • [0045] Autonomous integrated navigation solution, not dependent on GPS signals being available.
  • [0046] Midair object detection and collision avoidance guidance.
    Autonomous Integrated Navigation of the Integrated System

[0047] The highly accurate and reliable autonomous integrated navigation solution is the result of innovative multi-mode integration filtering approach using the coremicro AHRS/INS/GPS Integration Unit:

[0048] (1) GPS/INS mode with optional DGPS (differential GPS) correction capabilities from a ground reference GPS receiver via a telecommunication subsystem.

[0049] (2) Aided INS mode with onboard aiding navigation sensor data. The onboard aiding navigation sensor measurements that can provide potential aiding of INS include: baro altitude measurements from a baro altimeter or air data computer; radar altitude measurements from a radar altimeter; air speed measurements from the air data computer, terrain data from onboard terrain database (processed together with altitude measurements from a radar altimeter and baro altimeter to provide accurate update of INS), speed measurement from the Doppler radar, and magnetic heading from the magnetometer.

[0050] (3) AHRS (Altitude and Heading Reference System) mode.

Multiple Object Detection Sensors

[0051] The candidate sensors for the object detection system are optimized (possibly combined) from radar, LADAR, millimeter wave (MMW), infrared and video, passive and/or active imagers. Multiple level data fusion methods are used to fuse these sensors for terrain and obstacle recognition and tracking.

1.4 Innovative Accomplishments

[0052] The significant accomplishments are outlined as follows:

[0053] A system architecture is designed which integrates ground proximity warning, midair obstacle collision avoidance, and navigation/positioning by fusion of information from positioning sensors (GPS, MEMS coremicro IMU, vision, altimeter, terrain database), obstacle detection sensors (vision, radar), and data communication.

[0054] A real-time color tracking robot demo system is designed. Feature based vision tracking is to be emphasized and is integrated with the color tracking method.

[0055] Statistical model based algorithm is designed for terrain and obstacle recognition, and the recognition results are used to aid the decision of the collision avoidance.

[0056] Stereo cameras are used to provide range and 3D measurements of a target, which are used for positioning. Stereo camera is fused with other sensors, such as, radar, to provide a more accurate target tracking.

[0057] The cost-effective Interacting Multiple Model (IMM) estimator is implemented to improve the estimation accuracy of target tracking based on radar measurements.

[0058] Information fusion structure is designed for collision avoidance of aircraft with terrain and midair obstacles. The Integrated System is implemented by fusion of positioning information, terrain database, terrain recognition and stereo camera based terrain ranging. The midair collision avoidance is implemented by fusion of both obstacle tracking and recognition.

[0059] As a necessary step toward instrumenting the small, low cost, lightweight, Vision Based GNC Micro-System into a particular Aircraft Integrated Collision Avoidance System for flight demonstrations and tests, a flexible infrastructure utilizing the AGNC GPS/IMU REALTIME SIMULATOR for closed-loop hardware-in-the-loop testing and evaluation of many components of the GNC system has been designed.

[0060] Different modes are designed to fuse the information of the imaging sensors with that of other sensors for adaptive maneuvering target tracking and guidance, such as, sensor level fusion, track level fusion, and partially processed data fusion.

[0061] These and other objectives, features, and advantages of the present invention will become apparent from the following detailed description, the accompanying drawings, and the appended claims.

1 Statistical Model Based Terrain and Obstacle Recognition

[0071] The imaging sensor 100 can provide measurements of the target range, position, and altitude, which are fused with other object detection sensors, such as, radar, for obstacle tracking. Besides the vision based obstacle tracking, the imaging sensors can provide more information which is very useful for the ground proximity warning and obstacle collision avoidance decision. For example, different terrain, such as, mountains and trees, are recognized by our statistical model based image recognition methods. Even if the navigation based ground proximity warning system cannot work, the imaging sensor provides another method for terrain collision avoidance. Recognition of different aircraft nearby is helpful in choosing the best midair collision avoidance strategies.

[0072] Statistical model based image object recognition, identifying specific patterns and structures, is widely used in many engineering and scientific fields, such as, in applications for defense, machine vision and medical imaging. These techniques classify objects through a feature vector in a high-dimensional feature space.

[0073] There are two different classification methods for object recognition based on statistical modeling: supervised classification and unsupervised classification. In supervised classification, a set of features of known classification is provided as a training set which is then used to establish the classification method's parametric criteria that allow the various object classes of interest to be separated in feature space. Unsupervised classification uses clustering algorithms to affect an automated class delineation process in feature space.

[0074] The system architecture of supervised object recognition is shown in FIG. 1. The task of the object recognition system involves the identification of natural objects, such as, mountains, trees and grass, and cultural objects, such as, highways, roads and buildings. From the features of training images, the designed system sets up a knowledge bank of image features to represent the expert knowledge, as utilized by humans for object recognition, by specifying “rules” for image interpretation.

[0075] The supervised system consists of three modules: system training and machine learning, information system and modeling, and cultural objects extraction. The system training and machine learning module is trained for recognition and segmentation of natural objects and cultural objects through feature extraction, feature data analysis and statistical modeling. Information system and modeling consists of data storage (CAIDB: Categorized Aerial Images Database) of sampled object images and their features, and model optimization tools for feature selection to structure an image classifier. After modeling the knowledge-based object recognition system, the application and test module is implemented to extract natural and man-made objects from aerial images. The application module includes an aerial-image objects classifier, natural objects and cultural objects recognition and labeling. In order to improve the system performance, validation and verification of test or actual aerial images is executed.

[0076] We designed a Fuzzy C-Means (FCM) based method to extract cultural and natural objects (mountains, grass, trees and man-made objects, such as, buildings and roads) from an aerial image. FIG. 2 is the image processing pipeline. In the FCM-based classification, we use the Fuzzy-C-Means clustering algorithms to implement an automated class delineation process in feature space. As described above, FCM is an unsupervised classifier based on clustering unlabeled data (pixels in the 2D aerial image) into different groups (yielding, labeled pixels) on the basis of minimizing a distance measure in feature space. The FCM-based unsupervised classifier accomplishes the desired separation of the cultural objects and natural objects. It is different from the supervised classification case, where the data are pre-labeled as to their class membership and the classification process amounts to the establishment of thresholds or parameters that result from the established, by training, class clusters in feature space.

[0077] In FIG. 2 an aerial image is first preprocessed for generalization and contrast adjustment. In feature quantification, several features are extracted to form a high-dimension feature space. The data set of each pixel in feature space then are input into an FCM classifier to be labeled as mountain, tree, grass, building or road. After every pixel is labeled, a series of binary image morphological operations is used to remove small islands or holes on the segmentation masks generated by the FCM classifier. These operations act as a further outlier discriminator to improve the quality of the object segmentation and extraction.

2 Information Fusion Based Avoidance of Collision with Terrain and Midair Obstacles

2.1 Information Fusion Based Terrain Collision Avoidance

[0078] A terrain collision avoidance method is designed based on information fusion of camera, GPS, INS, and terrain database. As shown in FIG. 3, stereo camera 1013 provides terrain ranging information. Different terrain, such as, mountains, and trees, is recognized by the image recognition algorithms discussed above. Based on 3D terrain and 2D image feature matching 2030, the platform altitude is determined. The final estimation of the platform altitude, position, velocity, etc, is obtained by fusion of feature matching based altitude measurements, stereo camera based terrain ranging 2021, and the measurements from the AGNC AHRS/INS/GPS Integration Unit 1011, and other aiding sensors, such as altimeter 1012 and odometer. The terrain collision avoidance algorithm 2061 is based on the platform positioning, flight path prediction 632, terrain recognition 2011, and terrain database. First, the terrain environment is determined by these information sources. Secondly, the flight path is predicted by the platform altitude, position, and velocity. Thirdly, the terrain collision is predicted based on the flight path prediction 632 and terrain environment. Finally, if there is a possibility of terrain collision, visual and voice alert 2070 are provided to the pilot, and the optimal escaping method is designed to avoid collision with terrain nearby.

2.2 Information Fusion Based Midair Collision Avoidance

[0079] With the increase in high air traffic volume, midair collisions become a growing concern throughout the world. A midair collision avoidance method is designed based on information fusion of camera, GPS, INS, terrain database, and midair obstacle detection radar, altimeter, etc. As shown in FIG. 4, the platform altitude, position, and velocity are estimated by fusion of the AGNC coremicro AHRS/INS/GPS Integration Unit 1010, stereo camera 1013, terrain database 50, etc. Midair obstacles are detected and tracked by the obstacle detection sensors. The midair collision avoidance algorithm 2060 is based on the positioning and flight path prediction 632 of the vehicle and the obstacles. If there is a possibility of midair collision, visual and voice alert 2070 is provided to the pilot, and the optimal escaping method is designed.

[0080] Two kinds of obstacle detection sensors are used here for midair collision avoidance: imaging sensor 100 (such as, stereo camera) and radar (such as, Moving Target Indicator (MTI) radar). The stereo camera 1013 can provide ranging information from the obstacle to the vehicle and obstacle recognition information. The vision based obstacle recognition is discussed in the former sections.

[0081] Radar based obstacle detection and tracking is an important component of the midair collision avoidance subsystem. The first step is to detect the targets in clutter. A lot of algorithms have been proposed in the literature for target detection in clutter, such as, Constant False Alarm Rate (CFAR), time-frequency analysis, wavelet based, neural network based, and fractal based methods. CFAR based method is widely used for real-time automatic radar target detection. The detection threshold is adaptively determined online. With this adaptive threshold, the influence of the environment is minimized.

[0082] Once midair obstacles are detected by the MTI radar, they are continuously tracked for possible maneuvers. Maneuver target tracking is an important problem complicated by the fact that target maneuvers are generally unknown and are not available directly through radar measurements in practice. The decision-based techniques are widely used for maneuver detection, where the state estimation is based on a hard decision on the target motion model which is made by the maneuver detector. Different techniques are implemented and evaluated, including the chi-square test such as measurement residual based and input estimate based detectors, and the likelihood ratio test such as the generalized likelihood ratio (GLR) based, the Cumulative Sum (CUSUM) based, and marginalized likelihood ratio based (MLR) detectors. Sequential detection of target maneuvers is also investigated. The update of state estimation is triggered by the detection of target maneuvers through model switching, either to a model with higher process noise, a maneuver model or state augmentation.

3 Implementation of the Integrated System: Integrated Positioning/Ground Proximity Warning System Processor

[0083] A vision based Integrated Positioning/GPWS method is designed. Besides the positioning and terrain database, the vision based terrain recognition and stereo terrain ranging are also used for ground proximity warning decision. Different terrain, such as, mountains and trees are recognized by the image recognition methods discussed above. The ranging information and 3D depth map are built up by the stereo cameras. Based on the information from the imaging sensors, the ground proximity warning can be implemented even if the GPS/INS positioning system does not work under certain environments.

3.1 Structure of the Integrated Positioning/Proximity Warning System

[0084] Referring to FIG. 5, the integrated positioning and ground proximity warning system comprises an imaging sensor 100 (for terrain recognition, ranging, 3D depth map), a global positioning system receiver 10, an inertial navigation system 20, a baro altimeter 30, a radio altimeter 40, a terrain database 50, and a positioning/ground proximity warning system processor 60.

3.2 Integrated Positioning/Ground Proximity Warning System Processor

[0085] The integrated positioning/ground proximity warning system processor 60 provides not only precision position and dynamic data of the host vehicle, but also ground warning message to the object tracking and collision avoidance processor. The position and dynamic data of the host vehicle and the absolute position or relative position of near objects obtained from the object detection system are used to detect potential collisions between the host vehicle and near objects. The ground proximity warning message is incorporated in the collision avoidance processing. For example, when a ground proximity warning is detected and the host vehicle can not avoid a collision with the other vehicle by performing a dive because it is too close to the ground.

3.3 Positioning Sensors

[0086] The positioning sensors connected with the Integrated Positioning/Ground Proximity Warning Processor include: imaging sensor 100, GPS, INS, baro altimeter 30, radio altimeter 40, terrain database 50, vision, etc.

[0087] The imaging sensor 100 may be a monocular camera, stereo camera, infrared imager, etc. The imaging sensor 100 measures the target position and altitude, for target tracking. It is also used to recognize different terrain, such as, mountains and trees, which provides useful information for ground proximity warning decision 633.

[0088] The global positioning system receiver 10 receives global positioning system signals and deriving position, velocity and time information or pseudorange and delta range measurements of the global positioning system. The global position system receiver may be selected as a differential global position system receiver or as a multi-antenna global position system receiver or as a global position system receiver with wide area augmentation.

[0089] The inertial navigation system 20 solves navigation equations with angular rate and specific force information from an inertial measurement unit and obtaining an inertial navigation solution. The inertial navigation system 20, which is connected with the integrated positioning/ground proximity warning system processor 60, and receives optimal estimates of the inertial navigation solution errors and inertial sensor errors from the integrated positioning/ground proximity warning system processor 60, and outputs the inertial navigation solution, such as position, velocity and altitude to the integrated positioning/ground proximity warning system processor 60.

[0090] The baro altimeter 30 provides baro altitude measurements. The radio altimeter 40 provides radio altitude measurement from the terrain surface. The terrain database 50 provides global terrain data and obtains the terrain height of the current vehicle position and surrounding terrain height data.

[0091] A synthetic vision system 70 can be connected with the positioning/ground proximity warning system processor 60 to provide the flight crew with an enhanced vision field display for the ground proximity warning. A voice device 80 can be connected with the positioning/ground proximity warning system processor 60 to provide the flight crew with audible ground proximity warning messages.

[0092] The terrain database 50, which is connected with the integrated positioning/ground proximity warning system processor 60, receives the query parameters of the database from the integrated positioning/ground proximity warning system processor 60, and outputs the terrain height data of the current vehicle position and surrounding area to the integrated positioning/ground proximity warning system processor 60. The terrain database 50 may be selected as:

[0093] An onboard terrain database 50.

[0094] A ground-based terrain database 50, which is accessed through a data link by the integrated positioning and ground proximity warning system.

[0095] The connection between the global position system, the inertial navigation system 20, the baro altimeter 30, the radio altimeter 40, the terrain database 50, the integrated positioning/ground proximity warning system processor 60, the synthetic vision system 70, the voice device 80, and the onboard Flight Control and management system 90 can be one of the following means:

[0096] (1) Bus-based structure, such as inter bus, MIL-1553, ARIC 429.

[0097] (2) Communication port-based structure, including synchronous communication interface and synchronous communication interface, such as RS-232, RS-422, RS-485, etc.

[0098] (3) Network-based structure, such as NE2000.

[0099] Referring to FIG. 6, an input/output module 61 manages the input and output of data from other devices and the onboard Flight Control and management system 90. A navigation solution module 62 is connected with the input/output module 61 and a Ground Proximity warning Solution module 63. The navigation solution module 62 fuses the information from all sensors of the Integrated System, such as, imaging sensor 100, GPS, INS, altimeter, to obtain an optimal navigation solution, and outputs the obtained optimal navigation solution to the onboard Flight Control and management system 90 and the Ground Proximity warning Solution module 63, and outputs optimal position and velocity to the global positioning system receiver 10, and outputs optimal estimates of inertial navigation solution error and inertial sensor errors to the inertial navigation system 20.

[0100] Referring to FIG. 7, a correlation/matching 6201 accepts individual imaging sensor 100 based target position/altitude measurements, radio altimeter 40 measurements from the radio altimeter 40, barometric altimeter measurements from the baro altimeter 30, and terrain data from the terrain database 50 through the input/output module 61. The individual imaging sensor 100 measurements, radio altimeter 40 measurements, and barometric altimeter measurements are collected to construct a measured profile of terrain in an assigned time window. The terrain data is collected to construct a set of prestored reference terrain profiles which correspond to the area over which the vehicle is flying during the assigned window time. The measured terrain profile is then compared with the set of prestored reference terrain profiles in the correlation/matching 6201. Once a match is found, the correlation/matching 6201 outputs the geographic coordinates of the best matching reference profile to a filter/estimator 6202.

[0101] In addition, the correlation/matching 6201 differences the result of adding radio altimeter 40 measurements with terrain height of the current vehicle position with the inertial altitude solution (or hybrid baro/inertial altitude measurement) to form altitude measurement residues to the filter/estimator.

[0102] The filter/estimator 6202 filters the measurements for the global positioning system receiver 10, the inertial navigation solution from the inertial navigation system 20, and the geographic coordinates of the best matching reference profile and altitude measurement residues from the correlation/matching 6201 to obtain optimal estimates of the inertial navigation solution errors, errors of the global position system receiver, and errors of inertial sensors in a centralized filtering fashion.

[0103] In addition, the correlation/matching 6201 differences the result of adding radio altimeter 40 measurements with terrain height of the current vehicle position with the inertial altitude solution (or hybrid baro/inertial altitude measurement) to form altitude measurement residues to the TAINS local filter.

[0104] The TAINS local filter models the inertial navigation solution errors and inertial sensor errors, and filters the geographic coordinates of the best matching reference profile and altitude measurement residues to obtain the local optimal estimates of inertial navigation solution errors and inertial sensor errors.

[0105] An INSGPS local filter inputs the measurements from the global positioning system receiver 10 and the inertial navigation solution from the inertial navigation system 20 to obtain the local optimal estimates of inertial navigation solution errors, global position system receiver errors, and inertial sensor errors.

[0106] A master filter receives the local optimal estimates and covariance matrix of inertial navigation solution errors, errors of the global position system receiver, and errors of the inertial sensor from the INSGPS local filter and the local optimal estimates and covariance matrix of the inertial navigation solution errors and errors of the inertial sensor from the TAINS local filter, and filters these data, and provides the global optimal estimates of inertial navigation solution errors, the global position system receiver errors, and the inertial sensor errors.

[0107] The INSGPS local filter and the TAINS local filter accepts the fed back global optimal estimates of inertial navigation solution errors, the global position system receiver errors, and the inertial sensor errors and the covariance matrix obtained by the master filter to reset the INSGPS local filter and the TAINS local filter and performs information-sharing among the master filter, the INSGPS local filter, and the TAINS local filter.

[0108] To obtain different system performances, the communication and estimation between the master filter, the INSGPS local filter, and the TAINS local filter may have different approaches.

[0109] The master filter can also perform the consistency test among the state estimates of the master filter, the INSGPS local filter and the TAINS local filter, to detect and isolate the malfunction of a satellite of the global positioning system and to reconfigure the structure and process of the navigation solution module 62.

[0110] The individual baro altitude measurement (or hybrid baro/inertial altitude measurement) is subtracted by the height of terrain at the current position to form the referenced individual terrain-clearance measurement. The radio altitude measurement is the individual measured terrain-clearance measurement. The individual terrain clearance computation outputs the deference between the measured terrain-clearance measurement and a referenced terrain-clearance measurement to a filter/estimator 6202.

[0111] Due to the undulating nature of terrain, the terrain-clearance measurement is a nonlinear function of vehicle position. Furthermore, the difference between the measured terrain-clearance measurement and the referenced terrain-clearance measurement is a function of the inertial navigation solution errors.

[0112] The terrain clearance measurement equation is theoretically a nonlinear function of the antenna pattern of the radio altimeter 40, the cross-range/downrange position, altitude, and altitude of the vehicle. Generally, a reasonable approximation to the k th referenced terrain clearance measurement is
TC_k=h_INS−h(X_k,Y_k)+ν_k

[0113] where TC_k is the kth referenced terrain clearance measurement; h_INS is the height of the vehicle indicated by the inertial navigation system 20 or hybrid baro/inertial altitude measurement; h(X_k,Y_k) is the height of the terrain at the position (X,Y) indicated by the inertial navigation system 20; ν_k is the error of the terrain database 50.

[0114] The master filter can also perform the consistency test among the state estimates of the master filter, the INSGPS local filter, and the TAINS local filter to detect and isolate the malfunction of the satellite of the global positioning system and to reconfigure the structure and process of the navigation solution module 62.

[0115] Referring to FIG. 8, the Ground Proximity warning Solution module 63 further comprises a query parameter computation 631, a flight path prediction 632, a warning decision 633, a flight path plan aiding 634, a synthetic vision processing 635, and a voice processing 636.

[0116] The query parameter computation 631 receives the optimal navigation solution from the navigation solution module 62, computes the query parameters, and outputs the query parameters to the terrain database 50 to access the terrain data of the current vehicle position and surrounding terrain.

[0117] The flight path prediction 632 receives and collects the optimal navigation solution from the navigation solution module 62, the vehicle performance and configuration data from the onboard Flight Control and management system 90, and the terrain information from the imaging sensor module, and predicts the projected flight path to a warning decision 633. In addition, the flight path prediction 632 passes the current vehicle position, velocity, and altitude to the warning decision 633.

[0118] The warning decision 633 receives the projected flight path from the flight path prediction 632, the terrain information from the imaging sensor module, the vehicle performance and configuration data from the onboard Flight Control and management system 90, such as glide slope, landing gear, etc., and the surrounding terrain data from the terrain database 50. If the projected flight path is too close to the terrain, the warning decision 633 makes a warning decision message which is sent to a flight path plan aiding 634, a synthetic vision processing 635 and a voice processing 636. In addition, the warning decision 633 passes the current vehicle position, velocity, and altitude to the flight path plan aiding 634.

[0119] The flight path plan aiding 634 receives the warning decision message and the current vehicle position, velocity, and altitude information from the warning decision 633, the surrounding terrain data from the terrain database 50, and the vehicle performance and configuration data from the onboard Flight Control and management system 90, and processes these data, and provides an optimal evasion flight path to aid the flight crew.

[0120] The synthetic vision processing 635 receives the projected flight path and warning decision message from the warning decision 633, the surrounding terrain data from the terrain database 50, and the suggested optimal evasion flight path from the flight path plan aiding 634, and provides the driving data to the synthetic vision system 70.

[0121] The voice processing 636 receives the projected flight path and warning decision message from the warning decision 633, the surrounding terrain data from the terrain database 50, and the suggested optimal evasion flight path from the flight path plan aiding 634, and provides the driving data to the voice device 80.

4 Real-Time Implementation of the Integrated System

[0122] The hardware/software system design for the real-time implementation of the Integrated System is presented here. As shown in FIG. 9, the main components of hardware and software include:

• • [0123] Hardware: Sensors 1010 (the AGNC coremicro AHRS/INS/GPS Integration Unit 1011, altimeter 1012, stereo camera 1013), Fame grabber 1020, FPGA 1030, High-end DSP board 1040.
  • [0124] Software: Terrain and obstacle recognition 2010, Stereo camera based ranging/tracking 2020, 3D terrain and 2D image feature matching 2030, positioning 2040, Midair obstacle tracking 2050, terrain collision warning 2060, midair collision warning 2070.

[0125] For the vision based Integrated System, part of components have very high computation load, such as, matching of 3D terrain features and 2D image features, image preprocessing. We divide the software components into two parts: low-computation-load and high-computation-load:

• • [0126] High-Computation-Load: terrain and obstacle recognition, stereo camera based ranging/tracking, 3D terrain and 2D image feature matching (including image preprocessing, edge detection, feature matching, etc)
  • [0127] Low-Computation-Load: positioning, midair obstacle tracking, terrain collision warning, midair obstacle collision warning.

[0128] One skilled in the art will understand that the embodiment of the present invention as shown in the drawings and described above is exemplary only and not intended to be limiting.

[0129] It will thus be seen that the objects of the present invention have been fully and effectively accomplished. The embodiments have been shown and described for the purposes of illustrating the functional and structural principles of the present invention and is subject to change without departure from such principles. Therefore, this invention includes all modifications encompassed within the spirit and scope of the following claims.