Method and Apparatus for Selecting Higher Order Terms for a Holographic Neural Network

The present invention relates to neural networks and more specifically to term selection in holographic and related neural networks.

In an effort to overcome the shortcomings of conventional neural networks, holographic neural networks have been developed. Holographic neural networks include two vectors and one matrix, all of complex numbers. The two vectors include the S or stimulus vector (length n) and the R or response vector (length m). The matrix is the X matrix (size n.times.m) that describes a mapping from S to R. These networks offer several advantages over conventional neural networks including that they are simpler to train and discern. In addition, while conventional neural networks are usually structured to approximate digital quantities, holographic neural networks are well suited to representing continuously valued mappings.

Notwithstanding these comparative benefits, first order holographic neural networks are limited in the complexity of the mapping that can be encoded. This is because (1) the X matrix has a finite capacity to contain encodings and (2) the encoding is unable to represent complex interrelationships between discrete components in the stimulus vector.

These disadvantageous aspects are addressed, however, through the provision of higher order terms in the stimulus vector (where a higher order term is defined as a product of two or more initial input first-order terms). The generation of higher order terms allows a large increase in the size and therefore encoding capacity of the network matrix. For example, from 10 distinct first-order terms, 55 distinct second-order terms and 220 distinct third-order terms can be generated. Higher order terms also represent interactions between discrete first-order components of the stimulus vector, allowing the holographic neural network to represent such interactions in a manner similar to that of conventional neural networks. Finally, higher order terms tend to be more evenly distributed through the phase space than first-order terms and this also increases encoding capacity.

With the availability of higher order terms, the design of a holographic neural network is directed toward the selection of which higher order terms to include in the stimulus vector. This choice can be extremely difficult and is a primary design challenge in the creation of useful holographic neural networks.

Accordingly, it is an object of the present invention to provide a method of selecting optimal terms for inclusion in the stimulus vector of a holographic neural network.

It is another object of the present invention to provide such a method that incorporates an evolutionary type algorithm.

And it is yet another object of the present invention to provide an electronic apparatus for executing such a method.

These and related objects of the present invention are achieved by the method and apparatus of selecting terms for use in a holographic neural network disclosed herein.

In one embodiment of the present invention a method is disclosed that includes the steps of creating a plurality of stimulus vectors, each stimulus vector having at least one higher order term; evaluating said plurality of stimulus vectors to determine a relative fitness thereof; selecting a subset of the plurality of stimulus vectors based on their relative fitness; modifying the content of at least one of said stimulus vectors in said subset; and selecting a final stimulus vector from said modified subset. This method may also include the step of modelling each stimulus vector as a chromosome containing a plurality of genes which individually point to a term in a corresponding stimulus vector. The subset selecting step may include the steps of deleting a least fit one of the plurality of stimulus vectors and duplicating a best fit one of the plurality of stimulus vectors. The modifying step may include the steps of swapping genes between two chromosomes or replacing genes in a chromosome with new genes.

In another embodiment of the present invention a method is disclosed for forming a holographic neural network. This method comprises some or all of the above mentioned steps and further includes the provision of first order input data and the formation of an X matrix which the stimulus vector is multiplied by to produce a response output vector.

In yet another embodiment of the present invention an apparatus is disclosed that executes a stimulus vector term selection algorithm. This apparatus may include a processor, memory, means for receiving stimulus vector input data and means for generation an output of a response vector output signal.

The attainment of the foregoing and related advantages, features and configurations of the present invention should be more readily apparent to those skilled in the art, after review of the following more detailed description of the invention taken together with the drawings.

FIG. 1 is a diagram of a holographic neural network incorporating term selection in accordance with the present invention.

FIG. 2 is a block diagram of the relationship between a chromosome and a stimulus vector in accordance with the present invention.

FIG. 3 is a block diagram of an electronic device for executing the subject method in accordance with the present invention.

FIG. 4 is a flowchart of overall holographic neural network formation in accordance with the present invention.

Referring to FIG. 1, a block diagram of a holographic neural network is shown. The first order input terms are provided in a one dimensional array 10 (hereinafter referred to as first order array 10) and each term is specified by a sequential index, 1-j, where j=number of first order terms. The first order input terms are pre-processed data points from which the holographic network attempts to recognize a pattern. The pattern may relate to any of a number of fields including weather forecasting, stock and interest rate forecasting, face recognition, voice recognition, x-ray analysis, and computer-user interfaces, etc. A method of the present invention, represented by reference numeral 20, acts on array 10 to create higher order terms for a stimulus vector 60 of a holographic neural network 70. The stimulus vector 60 is multiplied by the X matrix 80 as is known to produce a response vector 90. The stimulus vector, X matrix, and response vector make up the holographic neural network.

The present invention utilizes an evolutionary type algorithm for selection of higher order terms from first order array 10. Potential solutions (potential higher order term combinations) are preferably modeled as chromosomes that consist of genes (selected higher order terms). describing pieces of the solution. A correspondence is established between the chromosomes (which may be referred to as the genotype) and the solution represented by the chromosomes (which may be referred to as the phenotype). A fitness function to evaluate the phenotype and a method for selecting parents based on fitness are provided. In addition, techniques are defined for chromosome mating, mutation and propagation.

A high-level description of a preferred embodiment of the evolutionary optimization process of the present invention is as follows:

The three user-specified parameters in the above description are population size, N, number of chromosome pairs to be selected for crossover, P, and the percentage of chromosomes and genes therein to mutate. The value of N will vary widely depending on application and is preferably large enough to foster maintenance of diversity. A designer/user can select N by trial and error based on performance measurements. P and mutation percentage will vary in a manner similar to N. The above process is preferably configured to continually seek the best solution to date and to terminate on some user-specified condition (such as exceeding a threshold quality-of-solution or time-of-computation value). Aspects of the stimulus vector optimization process are now discussed in more detail.

Genetic Representation

Referring to FIG. 2, a block diagram of the relationship between a chromosome and a stimulus vector in accordance with the present invention is shown.

The first order array 10 is operated upon by a function 30 (discussed below) that generates higher order term indexes. This function 30 combines indexes of first order terms in array 10 in a predefined manner to create indexes of terms on a user specified order, k, represented by block 25. For example, to form a second order term index, function 30 may combine the indexes of first order terms 2 and 17. For a third order term, the indexes of first order terms 2, 17 and 32 may be combined by function 30. Thus, function 30 effectively forms a virtual pool of k order term indexes which may be dipped into when a new term index is needed as in chromosome formation, the duplicate replacement step of the crossover operator, or during mutation (discussed below).

A chromosome formation algorithm (discussed below) forms chromosome 40 which is representative of any one of the N chromosomes described herein. Chromosome 40 includes a plurality of L genes 45.0 to 45.L-1 and each gene contains an integer uniquely indexing a higher order term created by function 30. The higher order term to which a gene in chromosome 40 points is preferably provided in a corresponding location in stimulus vector 70. The terms in stimulus vector 70 are designated 70.0 to 70.L-1.

Assuming that there are j first-order stimulus terms, the number of possible unique k.sup.th order terms (hereinafter referred to as kterms) is provided by the equation: kterms=(j+k-1)|/k|(j-1)| where k=term order. By enumerating the possible unique products, a gene can represent a k.sup.th order term as an integer in the range 0 to kterms. The following C++ code fragment presents an iterative method of function 30 for generating the k.sup.th order term for gene number q (where q.epsilon.›0,kterms-1!) from the j-length complex array 10 containing first-order terms. This iterative method is called for each gene 45 in chromosome 40.

Chromosome 40 is preferably configured as a one dimensional array (thus modeling its biological analogue), though other arrangements such as a two dimensional array are contemplated. The length of chromosome 40 is specified by a user and is dependent on the desired capacity of the holographic network. This dependency comes about because the chromosome's length determines the stimulus vector's length which in turn determines the capacity of the network. General criteria for chromosome length determination includes that the resultant stimulus vector should not be short enough for the network to saturate and should not be long enough that the resultant network does not generalize well. Techniques for stimulus vector length selection are known in the art.

It should also be noted that the length of chromosome 40 can be of uniform, non-uniform, fixed or varied lengths. It should also be noted that the chromosomes may contain terms of varied order (including first order terms) or of identical order.

A high level description of the generation of an initial chromosome is generally as follows.

Using function 30, randomly generate L terms and assign each to one of L gene locations in chromosome 40.

This step produces a chromosome which can encode indexes to any combination of terms (any possible solution).

To facilitate a better understanding of chromosome composition, the following example is provided. Given j=200 first order inputs and a chromosome indexing wholly second order terms (k=2), the number of unique higher order terms that function 30 can create indexes to, termed kterms, is (j+k-1)|/k|(j-1)|=(200+2-1)|/2|(200-1)|=201|/2|199|=200.times.201/2=20100. If the desired chromosome length is L=50, then the foregoing step randomly selects 50 of the 20100 kterms indexes to form an initial configuration of a chromosome.

This formation process is repeated N times to form the initial N chromosomes.

Measuring Fitness of a Chromosome

Once a chromosome 40 has been constructed, it must be ascertained whether the stimulus vector components encoded therein are well-suited to building a holographic neural network that accurately encodes the stimulus-response mapping. Further, it must be determined which of the pool of N chromosomes performs best. This determination is preferably made using two sets of stimulus and response data in which an expected response vector is known for its corresponding stimulus input data. Each stimulus input data and corresponding response vector is referred to as a stimulus-response vector pair in the discussion which follows.

The two sets of data are the training set and the optimization set. A user encodes the network with the plurality of stimulus-response vector pairs in the training set and then examines how well the resulting network generalizes with the optimization set. The observed generalization ability is the measure of fitness.

Encoding is achieved as follows. Stimulus input data is processed to form the terms specified by the chromosome under test. The corresponding stimulus vector is multiplied by the X matrix to produce a response vector and the value of that response vector is subtracted from its expected value to produce a difference value. The X matrix, which has an initial value of zero, is then adjusted based on the difference value such that subsequent multiplications of the subject stimulus vector by the X matrix produces approximately the expected response vector or a value associated therewith as is known. Encoding is known in the art and the encoding process is carried out for each stimulus-response vector pair in the training set.

The optimization set is similar to the training set in that it includes a plurality of known stimulus-response vector pairs. The optimization set is not used, however, to train the network but to test how well the matrix generalizes. The generated response vector (termed the "recall response") is compared to its expected value and the magnitude of the difference is saved. The difference of subsequent optimization set stimulus-response vector pairs is added to the previously stored value to achieve a final difference or "SUM.sub.-- ERROR" signal. This signal may be compared to like signals from other chromosome arrangements to determine relative fitness.

The SUM.sub.-- ERROR characterizes the ability of the network to generalize to the optimization set. The method for computing the error measures that make up the SUM.sub.-- ERROR is at the discretion of the user but is generally within known criteria. For example, the measure might simply be the magnitude of the difference between two complex vectors (the recall response and the expected response), though this measure carries risk in that error in one term of the complex response vector may not be nearly as significant as a large error in another term. The user may obtain a more meaningful error measure by post-processing the two outputs and computing an error term based on the accuracy of the post-processed response.

The SUM.sub.-- ERROR indicates the fitness of a network incorporating a subject stimulus vector. After a SUM.sub.-- ERROR has been generated for each of the N chromosomes, the collective SUM.sub.-- ERRORs are ranked. One or more chromosomes with the least favorable SUM.sub.-- ERROR values are removed, while one or more chromosomes with the most favorable SUM.sub.-- ERROR values are duplicated. Chromosomes with highly favorable SUM.sub.-- ERROR values may be duplicated more than once. Chromosomes are removed and duplicated such that a new, enriched group of N chromosomes is achieved.

Once the enriched group of N chromosomes is established, it is desirable to introduce new combinations or new material to these chromosomes to see whether performance can be improved. Thus, the composition of the N chromosomes is modified before subsequent evaluation. In a referred embodiment, this modification is provided by a crossover operator and a mutation operator as described below. The best .fit chromosome before modification is preferably saved and compared to the after modification chromosomes to make sure that the modifications have not decreased performance.

Crossover and Mutation

The present invention provides a crossover operator and a mutation operator. Though these operators apply to fixed-length chromosomes containing terms of the same order in the presently discussed embodiment, it should be recognized that the operators could be configured to apply to variable-length chromosomes or mixed-order terms by one skilled in the art given the teachings herein.

The crossover operator preferably functions as follows. P pairs of the N chromosomes, termed parents, are selected for propagation. For each pair of parents: (1) a crossover point is randomly selected, (2) genes after the crossover point are swapped, and (3) duplicate genes are replaced with genes randomly generated by function 30.

With respect to the mutation operator, a user specifies a percentage of the remaining chromosomes to modify. General criteria for selection of this percentage include that the resultant mutation should be sufficient enough to achieve a measurable change, while not being sufficiently great to cause instability. Empirical evidence has indicated that selecting five percent of the N size chromosome pool for mutation and mutating five percent of the genes within the selected chromosome pool achieves this goal, though the five percent standard may vary depending on application. It should also be recognized that to enhance diversity of solutions, the product of crossover operations can be mutated.

A high order description of the mutation process includes the following steps.

Select a predefined percentage of the N-2P chromosomes for mutation.

Randomly remove a predefined percentage of genes in the selected chromosomes.

Replace each removed gene with a randomly generated gene (from function 30).

While the present invention preferably includes both crossover and mutation operators, the invention may be practiced with only one of these operator with or without additional diversity generator(s). Though a crossover and a mutation operator are described it should be recognized that other methods of generating diversity and propagating same are within the present invention.

The evaluation steps discussed above, (i.e., training set and optimization set) are repeated for this modified group of chromosomes (including the previous best chromosome) to determine relative fitness. If an event has occurred for termination of the stimulus vector development process (the genetic algorithm), the most fit chromosome is selected for defining the stimulus vector of the final holographic neural network. If termination is not called for, the sequential select, modify, and evaluate steps are continued.

Generation of the Final Network

The genetic algorithm is terminated based on some user-specified criteria. Possible criteria include computation time, best solution exceeds a certain fitness level, and convergence (e.g., the best solution is not improved on for some number of generations), amongst other criteria. The result of the algorithm is a chromosome representing the optimal higher order stimulus vector for construction of a usable holographic neural network.

To create the final holographic network, the user invokes an encoding and optimization procedure that terminates when the measure of generalization begins to deteriorate. This encoding and optimization procedure preferably functions as follows. A holographic neural network is established that includes the selected stimulus vector and a null X matrix. This network is preferably trained and tested with the same sets of stimulus-response vector pairs and optimization stimulus-response vector pairs utilized in the fitness measurement procedure described above. A SUM.sub.-- ERROR is accordingly produced. The network is then retrained and retested with the same vector-response pairs to generate a new SUM.sub.-- ERROR value. Empirical evidence indicates that iteratively retraining the network with the same training set improves generalization up to a certain number of retrainings after which generalization degrades. Hence, the processes of sequentially retraining and retesting are preferably continued until the succeeding SUM.sub.-- ERROR is greater than that of the preceding network. At this point, the preceding network (that which had the better SUM.sub.-- ERROR) is recovered and implemented as the final holographic neural network.

Machine Implementation

Referring to FIG. 3, a block diagram of an electronic device 110 for executing the method described above is shown. The device may take many forms including a computer, a personal communicator, an electronic control for use in home, car or elsewhere, etc. The device preferably includes a processing unit 120 and memory 130 for executing the above described steps in a machine executable form. Given the teachings herein, the writing of a program to accomplish these steps, for example in C++, would be apparent to one skilled in the art.

Input data for array 10 is preferably provided via data input 140 which may include a sensor, scanner, keyboard, modem/adapter for downloading data, or other implement for receiving holographic neural network input data. Output device 150 preferably provides response vector data in a user recognizable format. Output device 150 may include a monitor, manufacturing device, loud speaker, signal generator, or other device for communicating the output of a holographic neural network.

Referring to FIG. 4, a flowchart 200 of overall holographic neural network formation in accordance with the present invention is shown. Step 202 indicates the creation of a plurality of stimulus vectors or chromosomes as discussed above. Step 204 indicates the evaluation of these stimulus vectors. Step 206 indicates selection of a subset of the stimulus vectors based on their measured fitness. Step 208 indicates modification of one or more of the stimulus vectors. This modification may include the exchange of terms between vectors (cross-over), modification of selected terms (mutation) or other modification. Step 210 indicates selection of a final stimulus vector which may include the steps of training and retraining a holographic neural network, amongst other steps.

While the invention has been described in connection with specific embodiments thereof, it will be understood that it is capable of further modification, and this application is intended to cover any variations, uses, or adaptations of the invention following, in general, the principles of the invention and including such departures from the present disclosure as come within known or customary practice in the art to which the invention pertains and as may be applied to the essential features hereinbefore set forth, and as fall within the scope of the invention and the limits of the appended claims.