Adaptive Image Segmentation System

The present invention pertains to image processing and particularly to image segmentation. More particularly, the invention pertains to adaptive image segmentation.

Image segmentation is typically the first, and most difficult, task of any automated image understanding process. Segmentation refers to grouping of parts of an image that have "similar" image characteristics. All subsequent interpretation tasks, such as feature extraction, object detection, and object recognition, rely heavily on the quality of the segmentation process. Despite the large number of segmentation methods presently available, no general-purpose methods have been found which perform adequately across a diverse set of imagery. To date, the only effective method of using a given segmentation process is to manually modify the algorithm's control parameters until adequate results are achieved. Only after numerous modifications to an algorithm's control parameter set can any current segmentation technique be used to process the wide diversity of images encountered in real world applications such as the operation of an autonomous robotic land vehicle or aircraft, automatic target recognizer, or a photointerpretation task.

When presented with an image from one of these application domains, selecting the appropriate set of algorithm parameters is the key to effectively segmenting the image. The image segmentation problem can be characterized by several factors which make parameter selection process very difficult. First, most of the powerful segmentation methods available today contain numerous control parameters which must be adjusted to obtain optimal or peak performance. As an example, the Phoenix segmentation algorithm contains 14 separate control parameters that directly affect the segmentation results. (The Phoenix algorithm is described in "The Phoenix Image Segmentation System: Description and Evaluation," SRI International Technical Note No. 289, December, 1982.) The size of the parameter search space in these systems can be prohibitively large unless it is traversed in a highly efficient manner. Second, the parameters within most segmentation algorithms typically interact in a complex, non-linear fashion, which makes it difficult or impossible to model the parameters' behavior in an algorithmic or rule-based fashion. Thus, the multidimensional objective function which results from and defined by various parameter combinations generally cannot be modeled in a mathematical way. Third, since variations between images cause changes in the segmentation results, the objective function varies from image to image. The search method used to optimize the objective function must be able to adapt to these variations between images. Finally, the definition of the objective function itself can be subject to debate because there are no single, universally accepted measures of segmentation performance available with which to efficiently define the quality of the segmented image.

Known search and optimization techniques or methodologies which attempt to modify segmentation parameters have various drawbacks. For instance, exhaustive techniques (e.g., random walk, depth first, breadth first, enumerative) are able to locate global maximum but are computationally prohibitive because of the size of the search space.

Calculus-based techniques (e.g., gradient methods, solving systems of equations) have no closed form mathematical representation of the objective function available. Discontinuities and multimodal complexities are present in the objective function.

Partial knowledge techniques (e.g., hill climbing, beam search, best first, branch and bound, dynamic programming, A.sup.*) are inadequate. Hill climbing is plagued by the foothill, plateau, and ridge problems. Beam, best first, and A.sup.* search techniques have no available measure of goal distance. Branch and bound requires too many search points while dynamic programming suffers from the curse of dimensionality.

Knowledge-based techniques (e.g., production rule systems, heuristic methods) have a limited domain of rule applicability, tend to be brittle, and are usually difficult to formulate. Further, the visual knowledge required by these techniques may not be representable in knowledge-based formats.

Hence, a need exists to apply a technique that can efficiently search the complex space of plausible parameter combinations and locate the values which yield optimal results. The approach should not be dependent on the particular application domain nor should it have to rely on detailed knowledge pertinent to the selected segmentation algorithm. Genetic algorithms, which are designed to efficiently locate an approximate global maximum in a search space, have the attributes described above and show great promise in solving the parameter selection problem encountered in the image segmentation task.

Genetic algorithms are able to overcome many of the problems mentioned in the above optimization techniques. They search from a population of individuals (search points), which make them ideal candidates for parallel architecture implementation, and are far more efficient than exhaustive techniques. Since they use simple recombinations of existing high quality individuals and a method of measuring current performance, they do not require complex surface descriptions, domain specific knowledge, or measures of goal distance. Moreover, due to the generality of the genetic process, they are independent of the segmentation technique used, requiring only a measure of performance for any given parameter combination. Genetic algorithms are also related to simulated annealing where, although random processes are also applied, the search method should not be considered directionless. In the image processing domain, simulated annealing has been used to perform image restoration and segmentation. Simulated annealing and other hybrid techniques have the potential for improved performance over earlier optimization techniques.

This invention makes use of a machine learning method known as a genetic algorithm to facilitate adaptation in a image segmentation process. Specifically, the genetic algorithm allows the segmentation algorithm to overcome changes in image characteristics caused by variable environmental conditions such as time of day, time of year, clouds, rain, haze, and shadows.

Genetic algorithms have demonstrated their capability in efficiently locating an approximate global maxima in applications such as function optimization, and thus can be used to search the complex space of plausible control parameter settings and locate the parameter combination which yields optimal results. The present invention is the first known application of a genetic algorithm in the domain of image segmentation. The invention differs from other proposed adaptive segmentation systems since it incorporates a closed-loop feedback mechanism in the segmentation/learning cycle. This cycle contains the genetic learning system, the image segmentation algorithm, and the segmentation evaluation process.

The genetic process traverses the complex parameter search space by maintaining a population of high-quality parameter sets. During each iteration of the segmentation/learning cycle, known as a generation, individual parameter sets are combined on the basis of their strength or fitness. The combination of existing parameter sets, referred to as the reproduction operation, is accomplished using one or more genetic operators such as crossover or mutation. By combining the properties of highly fit parameter sets, the genetic algorithm functions as a "global force" that shifts attention to productive, high quality regions in the search space.

The properties of the input image are characterized using several different methods of analysis. External image characteristics (time of day, cloud cover, and other environmental factors) as well as internal image statistics (first order properties, second order properties, and histogram data) are utilized to characterize the image. This information is used by the genetic learning system to select a group of parameter sets from previous image processing sessions that most closely align with the current image properties.

The adaptive process is extensible to any segmentation algorithm whose results can be modified through the use of control parameters. The invention uses, for example, the Phoenix segmentation algorithm since it works well with general-purpose color imagery.

In order to close the feedback loop, a segmentation evaluation component is used to supply the necessary "reward" to the genetic learning system. Just as there is no single segmentation algorithm which works in all cases, there exist no measure of segmentation quality that yields valid results in all cases. Thus, the present invention includes a series of segmentation quality measures that use both object-specific and object-independent evaluation methods. This information is combined using a weighted sum method to determine the overall segmentation quality.

The adaptive image segmentation system can dynamically adapt to the images as they are acquired. For the best combination of control parameters, object regions are analyzed to determine properties such as contrast and edge/border coincidence. The extracted properties of the current image regions are compared with the properties obtained so far by the image segmentation system. The reward in the feedback loop is proportional to the similarity between the characteristics of the extracted targets and the characteristics of the past-observed images.

FIG. 1a is an image submitted for segmentation.

FIGS. 1b-1f are images at various segmentation levels.

FIG. 2 is a representation of an objective function which is to be optimized in adaptive image segmentation.

FIG. 3 is a graph of the quality surface of the image in FIG. 1a.

FIG. 4 is a block diagram of the adaptive image segmentation system.

FIG. 5 shows the format for stating the image statistics and external variables extracted from the image in FIG. 1a.

FIG. 6 represents a classifier of the invention.

FIG. 7 is a flow chart of a genetic adaption cycle.

FIG. 8 illustrates a cycle through the adaptive image segmentation system.

FIG. 9 reveals a crossover operation of the segmentation system.

FIG. 10 reveals a mutation operation of the segmentation system.

FIGS. 11a-11c are diagrams for quality measures for edge-border and boundary consistency, pixel classification and object overlap, and object contrast.

FIG. 12 shows an implementation of the adaptive segmentation system.

FIGS. 1a-1f illustrate an adaptive image segmentation task. FIG. 1a shows an original image 8 that must be segmented by the system. Gates 12 adjacent to road 14 need to be segmented. FIG. 1b indicates the "ideal" segmentation of image 8 in which gates 12 on either side of road 14 are properly segmented, along with the other main regions of image 8 (e.g., road 14, sky 16, fields 18, mountains 20, etc.). The segmentation in FIG. 1b was obtained manually. The genetic process converges on this ideal segmentation result through successive iterations. FIG. 1c shows an initial image 8 that is grossly undersegmented. FIG. 1d identifies the main regions in image 8, although gates 12 are not evident. This figure lacks sufficient details. FIG. 1e oversegments image 8 by identifying too many irrelevant small regions. However, gate 12 regions have been obtained. The final result, shown in FIG. 1f, eliminates most of the small regions while leaving gate regions 12 intact. In other words, gates 12 are properly extracted and other main image regions are correctly segmented. FIGS. 1a-1f illustrate the iterative, convergent nature of the genetic process towards ideal segmentation results.

FIG. 2 provides a generalized representation 22 of an objective function that is typical of and needs to be optimized for the image segmentation process. FIG. 3 shows a quality surface 24 for the image shown in FIG. 1a. The contours of surface 24 indicate the complex interactions between the control parameters. "Maxmin" and "hsmooth" are Phoenix algorithm parameters. An analysis of surfaces 22 and 24 in FIGS. 2 and 3 indicates a need to utilize a highly effective search strategy which can withstand the breadth of performance requirements necessary for the image segmentation task.

Genetic algorithms are known in art. The term genetic algorithm is derived from the fact that its operations are loosely based on the mechanics of genetic adaptation in biological systems. Genetic algorithms can be briefly characterized by three main concepts: a Darwinian notion of fitness or strength which determines an individual's likelihood of affecting future generations through reproduction; a reproduction operation which produces new individuals by combining selected members of the existing population; and genetic operators which create new offspring based on the structure of their parents.

A genetic algorithm maintains a constant-sized population of candidate solutions, known as individuals. The initial seed population from which the genetic process begins can be chosen randomly or on the basis of heuristics, if available for a given application. At each iteration, known as a generation, each individual is evaluated and recombined with others on the basis of its overall quality or fitness. The expected number of times an individual is selected for recombination is proportional to its fitness relative to the rest of the population. Intuitively, the high strength individuals selected for reproduction can be viewed as providers of "building blocks" from which new, higher strength offspring can be constructed.

The inherent power of a genetic algorithm lies in its ability to exploit, in a highly efficient manner, information about a large number of individuals. By allocating more reproductive occurrences to above average individuals, the overall net affect is an upward shift in the population's average fitness. Since the overall average moves upward over time, the genetic algorithm is a "global force" which shifts attention to productive regions (groups of highly fit individuals) in the search space. However, since the population is distributed throughout the search space, genetic algorithms effectively minimize the problem of converging to local maxima.

New individuals are created using two main genetic recombination operators known as crossover and mutation. Crossover operates by selecting a random location in the genetic string of the parents (crossover point) and concatenating the initial segment of one parent with the final segment of the second parent to create a new child. A second child is simultaneously generated using the remaining segments of the two parents. The string segments provided by each parent are the building blocks of the genetic algorithm. Mutation provides for occasional disturbances in the crossover operation by inverting one or more genetic elements during reproduction. This operation insures diversity in the genetic strings over long periods of time and prevents stagnation in the convergence of the optimization technique.

The individuals in the population are typically represented using a binary notation to promote efficiency and application independence in the genetic operations. Other characteristics of the genetic operators remain implementation dependent, such as whether both of the new structures obtained from crossover are retained, whether the parents themselves survive, and which other structures are replaced if the population size is to remain constant. In addition, issues such as the size of the population, crossover rate, mutation rate, generation gap, and selection strategy have been shown to affect the efficiency with which a genetic algorithm operates.

Since genetic algorithms rely on the accumulation of evidence rather than on domain dependent knowledge, they are ideal for optimization in applications where domain theories or other applicable knowledge is difficult or impossible to formulate. A simple statistical approximation to a complex evaluation process can allow genetic systems to effectively adapt in these situations and converge to an approximate global maxima. Genetic algorithms can be used to provide an adaptive behavior within a computer system. The present approach is to allow the genetic system to modify a set of control parameters that affect the output of an existing computer system. By monitoring the quality of the resulting system output, the genetic system can dynamically change the parameters to achieve the best performance.

In the present invention, adaptive image segmentation requires the ability to modify control parameters in order to respond to changes that occur in the image as a result of varying environmental conditions. The block diagram of adaptive image segmentation system 10 is shown in FIG. 4. The key features of adaptive image segmentation system 10 include a closed-loop feedback control technique which provides an adaptive capability. The feedback loop consists of genetic learning component 26, image segmentation component 28, and segmented image evaluation component 30. At the input of system 10 is image analysis component 32.

Genetic learning component 26 optimizes segmentation performance on each individual image and accumulates segmentation experience over time to reduce the effort needed to optimize succeeding images.

Image characteristics and external image variables are represented and manipulated using both numeric and symbolic forms within the genetic knowledge structure. Segmentation control parameters are represented and processed using a binary string notation. Image segmentation performance is evaluated using multiple measures of segmentation quality. These quality measures include global characteristics of the entire image as well as local features of individual target regions in the image. The global and local quality measures can be used separately or in combination.

Adaptive segmentation system 10 is very fundamental in nature and is not dependent on any specific segmentation algorithm or sensor data (visible, infrared, laser, etc.).

After acquiring an input image, image analysis component 32 analyzes the image characteristics and passes this information to genetic learning component 26. Using this data, in conjunction with the observed external variables, genetic learning component 26 selects an appropriate parameter combination, which is passed to image segmentation component 28. After the image has been segmented, the results are evaluated by component 30 and an appropriate "reward" is generated and passed back to genetic learning component 26. This process continues on the input images until a segmentation result of acceptable quality is produced.

The input image is analyzed so that a set of features can be extracted to aid in the parameter selection process performed by genetic component 26. A set of characteristics of the image is obtained by computing specific properties of the digital image itself as well as by observing the environmental conditions under which the image was acquired. Each type of information encapsulates knowledge that can be used to determine a set of appropriate starting points for the parameter adaptation process.

Image analysis component 32 produces a set of image statistics that measure various properties of the digital image. Image statistics include, for instance, first order properties which involve measurements of the shape of the first-order image histogram. These measurements include mean, variance, skewness, kurtosis, energy, entropy, and x and y intensity centroids. Also, image statistics include second order properties which are measurements of the histogram features based on joint probability distributions between pairs of pixels. These measurements include autocorrelation, covariance, inertia, co-occurrence matrices, and other derived properties. Histogram peak/valley properties are image statistics that involve the measurements of the values of the peaks and valleys in the image histogram. These measurements include maximum peak height divided by minimum valley height, total number of histogram peaks, maximum peak location, minimum valley location, distance between maximum peak and minimum valley, maximum peak-to-valley ratio, interval set score, and interval set size.

External variables are also used to characterize an input image. These factors specify the conditions under which the image was acquired. These conditions include the time of day, time of year, cloud cover, temperature, humidity, and other environmental factors such as the presence of rain, snow, haze, fog, etc. These conditions affect the quality of the image, which in turn necessitates changes in control parameters.

FIG. 5 illustrates the structure of the image statistics 34 and external variables 36 extracted from the image 8 in FIG. 1a. An image characteristics list which includes the statistics and external variables is more complex for color than for monochrome images. For instance, twelve first order properties and histogram properties for each color component (red, green, and blue) of the image include mean, variance, skewness, kurtosis, energy, entropy, x intensity centroid, y intensity centroid, maximum peak height, maximum peak location, interval set score, and interval set size. The last two features measure histogram properties used directly by the Phoenix segmentation algorithm and provide useful image similarity information. Since a gray scale version of the image is used to compute edge information and target contrast during the evaluation process, the twelve features for the Y (luminance component) image are computed as well. Combining the image characteristic data from the four color and luminance components yields a list of 48 elements. In addition, two external variables, time of day and weather conditions, are utilized in the outdoor applications to characterize each image. The external variables are represented symbolically in the list structure (e.g., time=9am, 10am, etc. and weather conditions=sunny, cloudy, hazy, etc.). The distances between these values are computed symbolically when measuring image similarity. The two external variables are added to the list to create an image characteristic list totaling 50 elements for the outdoor experiments.

Once the image statistics 34 and external variables 36 have been obtained, genetic learning component 26 uses this information to select an initial set of segmentation algorithm parameters. A classifier system is used to represent the image characteristics and the associated segmentation parameters. FIG. 6 shows an example of classifier 40 used by genetic learning component 26 (in FIG. 4). Classifier 40 stores the current quality or fitness 42 of parameter settings 38, image statistics 34 and external variables 36 of the image, and segmentation parameter set 38 used to process images having these characteristics. Image statistics 34 and external variables 36 form the condition portion of the classifier 40, C.sub.1 through C.sub.I+J, while segmentation parameters 38 indicate the actions, A.sub.1 through A.sub.N, of classifier 40. Fitness 42, W, which ranges in value from 0.0 to 1.0, measures the quality of the segmentation parameter set 38. Note that only fitness value 42 and the action portion of classifier 40 are subject to genetic adaptation the conditions remain fixed for the life of classifier 40.

When a new image is provided to genetic learning component 26 (in FIG. 4), the process begins by comparing the image characteristics of the new image (FIG. 5) with classifiers 40 in global population 44 (FIG. 6). Global population 44 represents the accumulated knowledge of the adaptive system obtained through previous segmentation experience. The algorithm computes a ranked list of individuals in population 44 that have characteristics similar to the new image. Ranking is based on the normalized Euclidean distance between the image characteristic values as well as the fitness of classifier 40. The normalized distance between images A and B is computed using ##EQU1## where C.sub.iMIN is the minimum value of the ith numeric or symbolic feature in global population 44, C.sub.iMAX is the maximum value of the ith feature in global population 44, and W.sub.i is the weight attached to the ith feature. The ranges are normalized and the W.sub.i values have been set to 1 so that each feature contributes equally to the distance calculation.

When the distances between an image and several members 46 of global population 44 are the same (e.g., if a previous image contributed multiple individuals 46 to global population 44), fitness values 42 are used to select the best individuals 46 from population 44. Temporary copies of the highest ranked individuals 46 are used to create the initial or seed population for the new image.

Once the seed population is available, genetic adaptation cycle 50 begins. Cycle 50 is shown in FIG. 7. Segmentation parameter set 38 (of FIG. 6) in each member of seed or initial population 48 is used by image segmentation unit 52 to process the image. The quality of segmented results 54 for each parameter set is then evaluated by segmentation evaluation unit 56. If the maximum segmentation quality for the current population 64 is above a predefined threshold 58 of acceptance, the cycle terminates and the high quality members of current image population 64 are used to update 60 global population 44. Less fit members 46 of global population 44 are discarded in favor of higher strength individuals 46 obtained from processing the current image. In this manner, system 50 is able to extend the knowledge of the adaptive segmentation system by incorporating new experience into classifier 40 database.

Alternatively, if after segmenting and evaluating the performance of current population 64, system 50 has not achieved acceptable segmentation quality, genetic recombination operators 62 are applied to the members of current population 64. Crossover and mutation operators 62 are applied to the high strength individuals 44 in population 64, creating a new set of offspring which will theoretically yield better performance. New population 66 is supplied back to image segmentation process unit 52, where the cycle begins again. Each pass through the loop (segmentation, evaluation and recombination) is known as a generation. Cycle 50, shown in FIG. 7, continues until the maximum fitness achieved at the end of a generation exceeds some threshold, as described earlier. Global population 44 is modified in order to retain the information "learned" during the genetic process. Global population 4 is updated and system 10 is then ready to process a new image.

FIG. 8 provides an example of adaptive segmentation system 10 (in FIG. 4). Image characteristics 34 and 36 (of FIG. 6) extracted from image 8 in FIG. 1a are used in this example as the new image data. A subset of the complete image characteristics is used here for the sake of simplicity. The new image characteristics are compared with individuals 46 in global population 44 to obtain initial seed population 48. The normalized Euclidean distance is computed from the new image to every member 48 of global population 44 and this distance is used in conjunction with fitness 42 of each individual 46 in population 44. In this example, seed population 48 is limited to 3 individuals. In actual situations, seed population 48 for each image consists of 10 classifiers while global population 44 holds 100 classifiers in order to maintain a diverse collection of segmentation experience. Matching process 68 identifies three members 46 of global population 44 with similar image characteristics and a high degree of fitness. A copy of these individuals 46 is then extracted to create seed population 48 shown in FIG. 8.

After seed population 48 is obtained, fitness 42 of each individual 46 is established by processing the new image with associated segmentation parameters 38. Thus, image segmentation and evaluation steps 52 and 56, shown in FIG. 7, are performed to derive new fitness values 42 shown in population 72 of FIG. 8. Assuming a threshold fitness value of 0.70, none of the fitness values obtained from the initial population (0.48, 0.54, and 0.30) is acceptable. Current population 64 is now passed through genetic recombination step 62 in FIG. 7 to generate a set of new individuals 46 for new population 66. In this example, the first two members 46 of current population 64 are combined using the crossover operator while third member 46 is modified using the mutation operator. FIG. 9 illustrates a crossover operation 78. There is a conversion of segmentation parameters 38 of population 72 from decimal to binary representation, the crossover point 74 in each segmentation parameter 38 is randomly selected and second halves 76 of the binary strings are exchanged. The resulting strings are converted to a decimal format of population 66 to complete crossover operation 78. Similarly, FIG. 10 provides the details of mutation operator 80. The decimal values of parameters 38 of population 72 are converted from decimal to binary representations. One of the bit locations of the binary representation is selected as a mutation bit 82. Chosen mutation bit 82 is inverted and then parameters 38 are converted to decimal format of population 66 to complete mutation operation 80.

In FIGS. 7 and 8, newly created population 66 must be passed through the segmentation and evaluation stages 52 and 56 once more to determine fitness 42 of each individual 46. Fitness values 42 of new population 66 are shown in FIG. 8. Since fitness 42 of the first individual 46 (0.79) exceeds the threshold value 58 (0.70), adaptive cycle 50 for this image is terminated. A new classifier 40 is created for the current image by inserting the appropriate image characteristics 34 and 36 and storing new parameter settings 38 and their associated fitnesses 42. This classifier 40 is then inserted into global population 44, replacing the least fit member 46. Had any other members 46 of the new image's population 66 been greater than threshold 58, they too would have been placed into the global population 44.

Since color imagery is utilized with the present description, the Phoenix segmentation algorithm is selected. The present system, with the Phoenix algorithm, employs a region-splitting technique that uses information from the histograms of the red, green, and blue image components simultaneously. The algorithm recursively splits regions in the image into smaller subregions on the basis of a peak/valley analysis of the various color histograms. Fourteen different parameters are used to control the thresholds and termination conditions used within the algorithm. Of these fourteen values, two are selected as the most critical parameters that affect the overall results of the segmentation process, maxmin and hsmooth. Maxmin specifies the lowest acceptable peak-to-valley-height ratio used when deciding whether or not to split a large region into two or more smaller parts. Hsmooth controls the width of the window used to smooth the histogram of each image region during segmentation. Smoothing helps to remove small histogram peaks corresponding to small irrelevant regions or noise in the image. The number of selected parameters used may be increased for adaptation in order to investigate more difficult segmentation tasks.

However, incorrect values in the two main parameters of the Phoenix algorithm lead to results in which, at one extreme, the desired object is not extracted from the background, and at the other extreme the object is broken up into many small regions that are of little use to higher level processes. By measuring segmentation performance using appropriate segmentation quality criteria, the genetic process attempts to identify a parameter set that yields results between these two extremes.

After the image segmentation process has been completed by the Phoenix algorithm, the overall quality of the segmented image is measured. There are a large number of segmentation quality measures that have been suggested, although none have achieved widespread acceptance as a universal measure of segmentation quality. In order to overcome the drawbacks of using only a single quality measure, an evaluation technique has been incorporated that uses five different quality measures to determine the overall fitness for a particular parameter set. The selected measures of segmentation quality include edge-border coincidence, boundary consistency, pixel classification, object overlap and object contrast. Edge-border coincidence measures the overlap of the region borders in the image acquired from the segmentation algorithm relative to the edges found using an edge operator. In this quality measure, the Sobel operator is used to compute the necessary edge information. The original, unthinned Sobel edge image is used to maximize overlap between the segmented image and the edge image. Edge-border coincidence (see FIG. 11a) is defined by letting E be the set of pixels extracted by the edge operator after thresholding and S be the set of pixels found on the region boundaries obtained from the segmentation algorithm. E and S are determined by the following equations.

Edge-border coincidence is computed with following equation. ##EQU2## where n(A)=the number of elements in set A and

Boundary consistency is similar to edge-border coincidence, except that region borders which do not exactly overlap edges can be matched with each other. In addition, region borders which do not match with any edges are used to penalize the segmentation quality. The Roberts edge operator is used to obtain the required edge information. As with the edge-border coincidence measure, the Roberts edge image is not thinned to maximize the overlap between images. Boundary consistency is computed in the following manner (see FIG. 11a). The first step is to find compatible pixel pairs in the region boundary and edge results. Each pixel in the segmented image region boundary results, S, is compared with neighboring pixels in the edge image, E. A "reward" for matching the ith pixel is computed with the following equation, ##EQU3## Thus, if the pixels overlap, R.sub.i =0.1 .sup.* (10-0)=1. Pixels that do not directly overlap contribute a reward value that is inversely proportional to their distance from each other. As matching pairs of pixels are identified, they are removed from the region boundary and edge images (S and E). The total reward for all matching pixel pairs is obtained below. ##EQU4## Once all compatible pixel pairs have been removed from E and S, the remaining pixels correspond to the difference between the two images. These pixels are used to compute a penalty as indicated here. ##EQU5## Finally, since the value of boundary consistency must be positive, an intermediate value, M, is defined by the following equation ##EQU6## Then, boundary consistency is computed using M. ##EQU7##

Pixel classification is based on the number of object pixels classified as background pixels and the number of background pixels classified as object pixels. A is designated as the set of object pixels in the groundtruth image and B as the set of object pixels in the segmented image (see FIG. 11b) in the following equations.

Since pixel classification must be positive, the intermediate value N is defined by the following expression. ##EQU8##

Pixel classification is computed using N ##EQU9##

Object overlap measures the area of intersection between the object region in the ground truth image and the segmented image, divided by the object region. As defined in the pixel classification quality measure, A is the set of object pixels in the groundtruth image and B is the set of object pixels in the segmented image (note FIG. 11b). Object overlap can be computed as ##EQU10##

Object contrast measures the contrast between the object and the background in the segmented image, relative to the object contrast in the ground truth image. A is the set of object pixels in the groundtruth image and B is the set of object pixels in the segmented image, as shown in FIG. 11c and defined in the pixel classification quality measure. In addition, a bounding box (X and Y) is defined for each object region in these images. These boxes are obtained by enlarging the size of the minimum bounding rectangle for each object (A and B) by 10 pixels on each side. The pixels in regions X and Y include all pixels inside these enlarged boxes with the exception of the pixels inside the A and B object regions. The average intensity is computed for each of the four regions (A, B, X, and Y) using ##EQU11## where I(j) is the intensity of the jth pixel in some region R and R.sub.max is the total number of pixels in region R. The contrast of the object in the groundtruth image, C.sub.GT, and the contrast of the object in the segmented image, C.sub.SI, can be computed by using ##EQU12## The object contrast quality measure is then computed by the following equations. ##EQU13##

The maximum and minimum values for each of the five segmentation quality measures are 1.0 and 0.0, respectively. The first two quality measures are global measures since they evaluate the segmentation quality of the whole image with respect to edge information. Conversely, the last three quality measures are local measures since they only evaluate the segmentation quality for the object regions of interest in the image. When an object is broken up into smaller parts during the segmentation process, only the largest region which overlaps the actual object in the image is used in computing the local quality measures.

The three local measures require the availability of object ground truth information in order to correctly evaluate segmentation quality. Since object groundtruth data may not always be available, we have designed the adaptive segmentation system to use three separate methods of evaluating segmentation quality. First, quality can be measured by using global evaluation methods alone. Second, if groundtruth data is available and there is only interest in correctly segmenting the object regions in the image, local evaluation methods can be used alone. Finally, if good object regions are desired as well as high quality overall segmentation results, global and local quality measures can be combined to obtain a segmentation quality measure that maximizes overall performance of the system.

The five quality measures can actually be combined into a single, scalar measure of segmentation quality using a weighted sum approach. Each of the five measures is given equal weighting in the weighted sum. In addition to the weighted sum technique currently in use, a more complex vector evaluation approach may be used to provide multidimensional feedback on segmentation quality.

FIG. 12 reveals implementation 90 of the present invention. Images are picked up by sensor 92 which is a JVC GX-S700U video camera having a 10.5 to 105 millimeter (mm) zoom lens. Sensor 92 is connected to digitizing hardware 94 which is a Data Cube Max video digitizing chassis. Hardware 94 has device 96 connected to sensor 92. Device 96 is a Max Scan 089-0250. Device 98, an ROI Store 010-0273, is connected to device 96. Device 100, a Max Graph 080-0225-02, is connected to 98 and to display monitor 104. Device 102, a 68020 central processing unit (CPU), is connected to devices 96, 98 and 100. Digitized image database 108 of image file storage unit 106 is connected to device 98 of digitizing hardware 94. Sun 3/280 file server 110 works in conjunction with database 108. Storage 106 is connected to adaptive segmentation work station 112. Work station 112 has Sun 3/60 work station and processing unit 114. Unit 114 has a 68020 CPU, a 68881 coprocessor, a 16 megabyte memory and a color monitor. Unit operates with 4.2 BSD Unix. Devices 94, 106 and 112 are interconnected using Ethernet. ##SPC1##