Method and Apparatus for Displaying Speech Information

The present invention relates to a device for producing an image of speech information, particularly a symbol indicating one of a plurality of groups of detected sounds, and producing that image in the field of view of a hearing impaired wearer of the device.

Innumerable situations exist in which it is desirable to supply information to an individual by superimposing an image onto his normal field of vision. One example where such a display is needed is for the production of symbols indicating one of a plurality of detected groups of sounds onto the field of vision of a deaf or hearing impaired person.

Communication in any spoken language is made up of sequences of sounds which are called phonemes. By observation of the movements of the lips of a speaking person, a hearing impaired of deaf person can discern that each sound is one of a limited number of possible phonemes. Unfortunately, however, the ambiguities for a totally deaf person are too great for effective communication to take place using only lipreading.

If a person has some aid in resolving ambiguities, for example, understanding of an additional 10-20% of phonemes in addition to those understood by lipreading alone, then enough of the information in the speech can be understood by a trained lipreader for effective transfer of information. Often a lipreader will have limited hearing sufficient for this purpose. Alternatively, manual curing, a technique developed by Orin Cornett of Gallaudet College, utilizes hand cues to remove sufficient ambiguities to make lipreading practical. The difficulty with manually cued speech, of course, is that it can be used only with those individuals who have been trained to use it, thus limiting severely the number of people whom a deaf person can understand.

The different sounds of any language have different waveform characteristics which permit limited differentiation into different groups of sounds. These basic analyzing techniques are old and are described, for example, in pages 139-148, J. L. Flanagan, Speech Analysis, Synthesis and Perception, Academic Press, 1965. Using these analytic techniques, signals can be produced from detected spoken sounds, each signal indicating one of a plurality of different sound groups. The sounds in each group are differentiable on the lip so that, if this information can be effectively communicated to the lipreader, sufficient ambiguities can be removed to permit effective lipreading.

One way to communicate sufficient information to a lipreader to make lipreading truly effective is to superimpose a symbol identifying a sound group upon the viewer's field of vision which he can see as he watches a speaker's lips. This basic technique is described in two patents to Upton U.S. Pat. Nos. 3,463,885 and 3,936,605. In both of these patents a display is disclosed which is mounted upon a pair of spectacles intended to be worn by the hearing impaired or deaf person. In the system described in Upton U.S. Pat. No. 3,463,885, three types of sounds are detected--fricative, plosive and voiced. A number of bulbs are mounted on a lens of the spectacles, and each associated with one of these types of sounds. The associated bulb is activated when that type of sound is detected. In one embodiment, sounds which are a combination of these different types of sounds activate more than one bulb. In another embodiment, separate bulbs are utilized to denote combinations.

In Upton U.S. Pat. No. 3,936,605, a device is described in which the source of the image is mounted on the eyeglass frame at the side of the eye of the wearer and outside his field of vision. The eyeglass lens supports a generally spherical mirror which is positioned with respect to the eye so that the information source at the side of the eye is reflected by the mirror to project an image into the field of view.

The present invention is a unique display system which finds particular utility in projecting symbols into the field of view of a person for use in resolving the ambiguities of lipreading. According to the present invention, the display including a lamp array is mounted paraxially with the viewer's line of sight and directly in the field of view on a transparent plate which subtends and mounts an eyeglass lens. Light from the array is projected forwardly of the eyeglass wearer and reflected back to the wearer by either a semi-transparent mirror coating on the lens, or by magnifying dimple mirrors in the lens. The mirror coating can be formed by a coating of a reflecting material such as silver or aluminum, or multiple layers of dielectric material. For some people, the internal surface of their prescription lenses can be ground to a sufficiently short radius to project a suitably magnified image. Alternatively, short radius, spherical dimples facing the display can be ground into the internal surface before coating. After coating to form the mirror, the dimples can then be filled with material having the same optical characteristics as the lens thus removing the refractive power of the dimples. A high-magnification mirror operating as a beam-splitter permits mounting the display itself in the field of view since the amount of light that the display blocks from the eye is too small to substantially interfere with vision.

Preferably, the display is an array which can provide any of a number of symbols. One of the difficulties with the system of Upton is that each of its indications is that of a single phoneme, and, at normal rates of speaking, the sounds occur so quickly that it is doubtful that they can be effectively used at that rate by the brain. According to the present invention, this problem is reduced by displaying information as syllables, i.e., normally a combination of a consonant sound and a vowel sound, although occasionally a single phoneme can be a syllable. This concept is described in detail and claimed in a co-pending application Ser. No. 198,211 filed even date herewith, and entitled "Method and Apparatus For Automatic Curing", the disclosure of which is hereby incorporated by reference. One way that syllable information can be displayed is with a symbol indicating one of a plurality of consonant groups in a mode indicating an associated vowel group. For example, a symbol indicating one of nine consonant groups can be produced in one of four spatial locations, i.e., quadrants, the spatial location indicating the associated vowel group. Another approach is to produce the symbol in one of a number of colors, for example, four, each color indicating an associated vowel group. The present device facilitates display of syllable symbols and the effective communication of information to the lipreader.

Other objects and purposes of the invention will be clear from the following detailed description of the drawings.

FIG. 1 shows a perspective view of the device of the present invention mounted on an eyeglass frame;

FIG. 2 shows a view of one of the symbols of the present invention superimposed upon an image in the field of view;

FIG. 3 shows a diagram of the consonant and vowel groups;

FIGS. 4A and 4B show a schematic view of one embodiment of the display of the present invention mounted on a spectacle lens;

FIG. 5 shows a second embodiment of the present invention;

FIG. 6 shows a schematic view of the present invention projecting an image onto the field of view of a lipreader; and

FIG. 7 shows a block diagram of an apparatus for producing signals to actuate the display.

Reference is now made to FIGS. 1 and 2 which show a schematic view of the display 20 of the present invention mounted on a conventional prescription lens 22 which is in turn mounted within a conventional eyeglass frame 24 and producing a symbol superimposed on the field of view. As discussed in detail below, display 20 produces an image of one of a number of different symbols, each symbol being associated with one of a number of groups of consonants as shown in FIG. 3, and produced in a mode identifying a group to which an associated vowel is assigned. For example, each symbol can be produced in one of four quadrants, each quadrant being associated with a different group of vowels, the combination of the consonant symbol and its location defining a syllable. Alternatively, the color of the symbol can indicate the vowel group by activating one or more displays of different colors or controlling a conventional multi-color display. Techniques for producing single color symbols are described in the above-mentioned patent application and book and in detail in the reports entitled "A Feasibility Study for the Development of a Speech Autocuer" (NASA Contract Number NAS5-2554).

FIG. 3 illustrates one possible set of groupings of consonants and vowels and the associated consonant symbols and vowel locations or other modes of presentation. With the exception of the group of consonants containing "n" and "ng", the consonants of each group can be readily differentiated by a lip-reader from each of the other consonants of that group from observation of the speaker's mouth alone. The single exception causes minimal linguistic confusion in English because discrimination between "n" and "ng" is rarely required for understanding. Some syllables have no vowels and rapidly spoken syllables often have a vowel which may not be detected. The detected absence of a vowel is treated as a component of one vowel group. Absence of a consonant is treated similarly. Diphthongs are handled by movement of the symbol from one quadrant position to another, based upon the vowel components of the diphthong as schematically shown in FIG. 3.

Reference is now made to FIGS. 4A and 4B which illustrate a first embodiment of the present invention. In this embodiment, display 20 is made up of two light emitting arrays, 30 and 32, and structure defining a beamsplitter associated with each of the respective arrays. In the embodiment of FIG. 4, the beamsplitter is formed by a coating 34 of silver, aluminum or other reflective material coated onto the interior surface of a conventional prescription lens 36 of glass or suitable plastic. Arrays 30 and 32 are mounted upon a transparent substrate 28 by any suitable means, and the substrate 28 in turn fixed at its extremities to prescription lens 36 by any suitable means.

Lens 36 is selected for optical correction where necessary. Next, spherical dimples 38 and 40 are ground on the inner surface of lens 36 to a radius suitable to provide the desired magnification. The interior surface of lens 36 is next optically coated with coating 34, for example, by deposition in a vacuum. The dimples are then back filled with material having the same, or virtually the same index of refraction as the material of lens 38. The material filling dimples 38 and 40 may be the same material as the lens 36.

FIG. 5 illustrates a second embodiment in which the interior surface of lens 40 is curved at a short enough radius to project a suitably magnified symbol image. A thin coating 42 serves as the beamsplitting mirror and can be formed as described above. Displays 44 and 46 are mounted on transparent substrate 48 which is fixed to lens 40 in any suitable fashion.

Either the top portion of the lower portion of one display, i.e., a light emitting diode (LED) array 30 and 32 is actuated in the embodiments of both FIGS. 4 and 5 to produce a symbol by the associated mirror to one of the four quadrants as shown schematically in FIG. 6. In other words, each symbol is produced in an upper or lower spatial location depending on whether the upper or lower portion of the display is activated.

The embodiment of FIGS. 4 and 5 produces a symbol indicating a group of consonants, for example, as shown in FIG. 2 to a quadrant indicating the group to which an associated vowel belongs. Alternatively, the color of the symbol can indicate the vowel group.

The contrast between the magnified image and the normal visual field may be increased by any one or a combination of methods. A neutral-density material may be used for the mirror substrate to reduce normal visual field brightness. Tinted material can alternatively be used for the substrate to provide an absorption maximum near the wavelength of display emission or illumination. A wavelength-selective mirror coating such as a multi-layer dielectric having a reflectivity peak near the wavelength of the display emission or illumination can be employed.

FIG. 7 shows schematically the apparatus for detecting the speech sounds and producing the signals for actuating display 20. A conventional detector 50 receives the speech sounds and produces an electrical signal having characteristics uniquely associated with each sound. These signals are applied to the microcomputer 52 which produces the signals to activate display 20. These elements are described in detail in the above-mentioned co-pending application and may be mounted on frame 24 adjacent the lens.

Many changes and modifications in the above-described embodiment of the invention can, of course, be made without departing from the scope of that invention. Accordingly, the scope is intended to be limited only by the appended claims.