System for conveying musical beat information to the hearing impaired

[0002] Not Applicable.

[0003] Not Applicable.

[0004] The present invention relates generally to hearing prostheses and, more particularly, to systems for providing musical beat information to the hearing impaired.

[0005] In general, people with a hearing impairment like to participate in activities and social occasions in a manner similar to people without hearing loss. As is known in the art, there are a variety of conventional prostheses to assist people with varying degrees of hearing loss. Most hearing aids are useful to people with a slight to moderate hearing loss. More aggressive approaches are also known in the art, such as cochlear inner ear implants, for helping those with more severe hearing. Cochlear implants can help a person with damaged cochleae but with viable nerves running from the cochleae to the brain. Even more aggressive approaches are known, such as brain stem implants, that include systems for transmitting sound information directly to the brain stem, bypassing damaged cochlear nerves.

[0006] Typical hearing aids do not help those with a severe hearing impairment, and implants require expensive equipment, as well as delicate surgery or series of surgeries, which may not provide satisfactory results. Many people with uncorrected severe hearing loss would enjoy the ability to appreciate musical beat information. In particular, those with severe or total hearing loss would appreciate the ability to sense the beat of music, thereby enabling them to dance at social functions and the like.

[0007] Those of ordinary skill in the art understand the beat of music to be the repetitive change in signal intensity or amplitude within some part of the total acoustic frequency spectrum of the music. The beat information may be located in different parts of the spectrum for different music. For example, the beat may be a repetitive beat of a bass drum, with spectral characteristics dominating the lower end of a total acoustic music spectrum. Alternatively, the beat of the music may be dominated by a repetitive change in amplitude at a higher frequency part of the spectrum. For example, an orchestral piece may contain a repetitive sound from a violin, with spectral characteristics dominating a higher frequency part of the total music acoustic spectrum.

[0008] U.S. Pat. No. 3,958,486 to Sears, entitled Keyboard Attachments, discloses a hearing prosthesis having a direct electrical connection between a keyboard and indicator lights. The keyboard device provides a visual indication of the music played on the keyboard.

[0009] U.S. Pat. No. 4,908,869 to Ledermen, entitled Induction-Based Assistive Listening System, discloses a hearing prosthesis for inducing electromagnetic signals into existing hearing aids. The Lederman device can communicate to hearing aids without sound, thus providing non-acoustic communication.

[0010] U.S. Pat. No. 4,354,064 to Scott, entitled Vibratory Aid for Presbycusis, discloses a hearing prosthesis-that provides a tactile output in response to sound energy. The Scott device provides a tactile representation of the high frequency components of either speech or music.

[0011] While these devices can overcome hearing loss, such devices do not enable a person with a severe hearing loss to easily enjoy musical beat information. It would, therefore, be desirable to provide a system for providing musical beat information to those having severe hearing impairment.

[0012] The present invention provides a system for conveying musical beat information via tactile representation. With this arrangement, a person, particularly a person with severe or total hearing impairment, can perceive musical beat information so as to enable that person to dance. While the invention is primarily shown and described in conjunction with conveying musical beat information, it is understood that the invention is applicable to other systems in which it is desirable to provide a tactile representation of an audio signal.

[0013] In one aspect of the invention, a system for conveying musical beat information includes a sound detector for receiving music and providing a signal corresponding to the music to an audio processing unit. The audio processing unit generates one or more electrical outputs to tactile actuators for conveying a tactile representation of the beat information to a user. With this arrangement, a tactile output is provided in response to the musical beat so that a person with severe or total hearing impairment can sense a musical beat.

[0014] In a further aspect of the invention, the audio processing unit contains one or more electronic filters for selecting corresponding audio frequency bands. The audio processing unit provides one or more band-selected electrical outputs to tactile actuators, where the band-selected outputs correspond to the filter frequency bands. In one embodiment, each tactile actuator is responsive to the beat within a particular audio frequency band.

[0015] The invention will be more fully understood from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0016] FIG. 1 is a top level block diagram of a system for conveying musical beat information in accordance with the present invention;

[0017] FIG. 2 is a block diagram of an audio processing unit that can form a part of the system of FIG. 1;

[0018] FIG. 3 is a block diagram showing an exemplary implementation of the system of FIG. 1; and

[0019] FIG. 4 is a block diagram showing an exemplary implementation of the phase locked loop that forms part of the system of FIGS. 2 and 3.

[0020] FIG. 1 shows an exemplary system 2 for conveying musical beat information to a user in accordance with the present invention. The system 2 includes a sound detector, such as a microphone 4, an audio processing unit (APU) 100, and a series of tactile actuators 300a-N, 302. The microphone 4, which is an audio transducer, is used to convert sound energy into electrical energy. It will be readily apparent to one of ordinary skill in the art that a variety of microphones can be used to provide electrical signals to the APU 100. Exemplary sound detectors include diaphragm/coil and piezoelectric type microphones.

[0021] The microphone 4 provides an electrical signal 200 to the APU 100. The APU 100 processes the electrical signal 200 from the microphone 4 and provides one or more output signals 204a-N, 206 to respective tactile actuators 300a-N, 302. In general, the tactile actuators 300a-N, 302 impart tactile sensation to the skin of a person using the invention.

[0022] It is understood that the tactile actuators 300a-N, 302 can be of various constructions and techniques. Exemplary tactile actuators include vibratory and solenoid type devices. One of ordinary skill in the art will recognize that a wide range of tactile actuators can be used to provide tactile sensation to the user, including vibrating devices, tapping devices, devices that move over the skin, and devices that vary pressure against the skin slowly. It will also be recognized by one skilled in the art that visible lights may be used in place of the tactile actuators to convey the musical beat information to a user.

[0023] The illustrative embodiment of FIG. 1 describes an embodiment having five tactile actuators 300a-N, 302, each of which provides a corresponding tactile sensation to the user. It will be appreciated that fewer than five or more than five tactile actuators can be used without departing from the present invention. In the illustrated embodiment, the five tactile actuators allow the user to distinguish five different tactile sensations, each representing different beat information.

[0024] FIG. 2 shows an exemplary embodiment of the APU 100 of FIG. 1, in which like reference numbers indicate like elements. The APU 100 has multiple parallel processing channels 102a-N. Each processing channel 102a-N operates upon a selected band of the total bandwidth of the electrical signal 202 from the microphone preamplifier 104. It is understood that each of the processing channels 102a-N can be of similar construction, differing only in the electrical bandwidth which each processes.

[0025] In general, each of the plurality of processing channels cover respective frequency bands since it is not known apriori in which acoustic band or bands the musical beat information will be found. For example, the beat information may dominate at relatively low frequencies, which can correspond to the beat of a drum for example. Alternatively, the beat information may dominate at relatively high frequencies, which can correspond to the repetitive tones of a clarinet, for example. Thus, the system 2 employs multiple acoustic processing channels 102a-N in parallel. Each such processing channel 102a-N employs a unique band pass filter 106a-N for processing the beat information of the spectral band that relates to its particular band pass filter 106a-N.

[0026] In one embodiment, the APU 100 includes an optional phase locked loop (PLL) 104 in communication with one of the processing channels 102N. In response to beat information, the PLL 104 sends a repetitive signal to the fifth tactile actuator 302. The repetitive PLL signal 104 has different characteristics than acoustic processing channel output signals 102a-N in that it greatly reduces any fluctuations in beat-to-beat timing, yielding a more periodic and likely truer representation of the beat itself. It should be recognized by one of ordinary skill in the art that other embodiments can have additional PLLs connected to any or all of the processing channels 102a-N.

[0027] FIG. 3 shows an exemplary implementation of the APU 100 of FIG. 2. The acoustic processing channels 102a-N of the APU 100 each receive an input signal 202 from the microphone preamplifier 104 corresponding to the music. It is understood that the channel A acoustic processing channel 102ais representative of the other channels 102b-N of the unit. The signal 202 from the preamplifier 104 is provided to the channel A band pass filter 106a.

[0028] Exemplary frequency bands of band pass filters 106a-N for the acoustic processing channels 102a-N are in Table 1.

[0029] It will be recognized by one skilled in the art, that other bands of interest may be used without departing from the invention. It will also be recognized by one skilled in the art that the band pass filter bands can overlap.

[0030] The output of the channel A band pass filter 106ais provided to the channel A variable gain amplifier 108awhich provides automatic gain control (AGC) in conjunction with signal magnitude feed back 210afrom the channel A second low pass filter 116a.It should be noted that the AGC feedback via the channel A second low pass filter 116ais a slowly varying control signal. Depending upon the filter characteristics of the channel A second low pass filter 116a,the gain of the channel A AGC amplifier 106acan vary rapidly or slowly in response to varying signal amplitudes. In one embodiment, the channel A second low pass filter 116ahas a low cutoff frequency corresponding to a time constant of about one second, thus creating an AGC with a relatively slowly varying gain.

[0031] The signal from the channel A AGC amplifier 106ais provided to the channel A detector/filter pair, which includes a detector 110aand a first low pass filter 112a.Detector/filter circuits for detecting lower frequency amplitude fluctuations of a higher frequency carrier signal are well known to one of ordinary skill in the art. Such a detector is commonly used as the audio signal detector in an amplitude modulated (AM) radio receiver, for example.

[0032] Those of ordinary skill in the art will appreciate that the detector/filter function can be achieved using a variety of techniques. For example, the detector can include a full-wave or half-wave rectifier, with or without additional active devices, absolute value circuit, square-law device, or other memory-less non-linearity, each of which has many known configurations. Furthermore, the low pass filter of the pair can be achieved using many known techniques, such as resistor and capacitor networks, alone or in conjunction with active devices.

[0033] In an exemplary embodiment, the detector 110aand the filter 112aemploy diodes, resistors, and operational amplifiers for the detector and resistors, capacitors, and operational amplifiers for the filter.

[0034] The output signal 212afrom the channel A first low pass filter 112ais representative of the amplitude fluctuations of the original signal 202. It is understood that the original signal 202 is a composite signal, representative of the original acoustic energy, with complex frequency and time characteristics. In contrast the channel A signal 212ais a reduced bandwidth signal, band-limited by the channel A band pass filter 106a.If the signal 212acontains signal characteristics where the fluctuations are rhythmic and repetitive, the fluctuations represent the beat of the music. If the signal 212adoes not contain repetitive characteristics, another of the filter output signals 212a-N, each representing a different acoustic frequency band, will likely contain repetitive characteristics. Thus, beat information is obtained at one or more of the outputs 204a-N from the APU.

[0035] In one embodiment, the channel A first low pass filter 112ahas a time constant, τ, of about 40 milliseconds, corresponding to maximum practical beat frequency of about 1 /(2πτ) or 4 Hz. However, those familiar with the art will recognize that other time constants would be possible without departing from the spirit of this invention. Time constants representing beat frequencies in the range of 0.1 to 10 Hz are easily realizable. Frequencies in this range are readily recognizable as beat information through tactile means, although beat frequencies range from perhaps 0.25 to 4 Hz in the majority of cases.

[0036] The beat information signal 212ais then passed through the channel A driver 114awhich converts the low power (low voltage and low current) signal 212ato a higher power signal 204a.The higher power signal 204ais sent out of the APU 100 to a channel A tactile actuator 300a.Each output signal 204a-N and corresponding tactile actuator 300a-N indicates beat information from different regions of the overall complex acoustic frequency spectrum.

[0037] As described above, the APU can include one or more phase locked loops for providing more precise beat information. Although the illustrative embodiment of FIG. 3 shows a single PLL 104, it is understood that more and fewer PLLs can be used.

[0038] The PLL 104, which receives the output 212N of the channel N first low pass filter 112N with a Schmidt trigger 117, can be used to detect low frequency repetitive signal content of the band pass input signal 106N. Alternatively, the phase locked loop can receive the output of the preamplifier 202 with the Schmidt trigger 117 if a separate detector and low-pass filter are provided between them.

[0039] Phase-locked loops are well known to those skilled in the art. In general, the PLL operates with a long time constant to keep the phase and frequency of the PLL output signal 214 equal to the time-averaged input phase and frequency of the input signal 212N.

[0040] In one particular embodiment, the PLL 104 includes a Type 2 phase detector 118, a loop filter 120, and a variable frequency oscillator 122. Input signal 212N is phase-compared to PLL output signal 214. The loop characteristics of the PLL are designed to keep the PLL output signal 214 in average phase relationship with the input signal 212N. When this condition is achieved, the PLL is said to be locked.

[0041] In operation, phase and frequency fluctuations of the input signal 212N are effectively removed to create an average phase and frequency signal 214 with slowly varying characteristics in response to the average phase and frequency of the input 212N. Thus, the PLL output signal 214 is a repetitive signal, relatively constant in frequency and amplitude. This is in contrast to the other output signals 212a-N that vary in frequency and amplitude. The PLL circuit is thus used to give a further indication of the musical beat. The PLL output signal 214 is a constant amplitude repetitive AC signal that corresponds to the musical beat.

[0042] It is understood that the PLL dynamic characteristics are determined by the characteristics of the phase detector 118, loop filter 120, and the variable frequency oscillator (VFO) 122. One skilled in the art will recognize that a variety of phase detectors, loop filters and variable frequency oscillators suitable for this application can be used.

[0043] While the exemplary embodiment shown has one PLL, it should be recognized that other numbers of PLLs and associated tactile actuators, in communication with other of the acoustic processing channels 102a-N, may be employed without departing from the invention. Each may have different time constants or topologies.

[0044] FIG. 4 shows a schematic of an exemplary implemention of a phase locked loop 104 that can be used as part of this invention. The phase locked loop design 104 is based upon a conventional 4046 integrated phase locked loop device 118, 122a.As known to one skilled in the art, 4046 refers to a commonly available generic phase locked loop integrated circuit. The 4046 device contains the type 2 phase detector 118 and a voltage controlled oscillator 122a.

[0045] The input signal 212N from the channel N first low pass filter 112N is presented to a Schmidt trigger device 117. A Schmidt trigger device is known to one skilled in the art to provide an amplified square wave output 213 from an analog input 212N. Here, the input 212N is an analog waveform representing the beat of the music. The Schmidt trigger output 213 is an amplified signal that will take two states, either high or low, depending upon whether the input 212N is above a first voltage threshold or below a second voltage threshold. The use of the Schmidt trigger 117 is particular to the phase locked loop implementation shown herein, where the Type 2 phase detector 118 of the 4046 device 118, 122arequires a square wave input for proper operation.

[0046] The phase detector output 150 is presented to a loop filter 126 that provides a filtered signal 152 to the input of a variable frequency oscillator 122. Loop filter 126 characteristics are determined by the value of associated components 122a-c.The variable frequency oscillator 122 is composed of the voltage controlled oscillator (VCO) 122aand a divider 122b,a divide by 1024 device. The indicated 4040 device 122brepresents the generic part. number of one type of digital divider among many types of commonly available suitable digital dividers. Together, the VCO 122aand divider 122bprovide a PLL output signal 214 in response to the filtered signal 152. The relationship between the voltage 152 and the frequency of the PLL output signal 214 is related to the value of capacitor 136 and resistors 132, 134.

[0047] The type 2 phase detector 118 produces a voltage output that corresponds to the phase difference between the input signal 212N and the PLL output signal 214. Feedback from the PLL output signal 214 to the phase detector 118 operates to keep the input signal 212N and the PLL output signal 214 substantially in an average phase relationship as mentioned earlier.

[0048] One skilled in the art will realize that there are many design parameters associated with a phase locked loop. Two PLL parameters include the lock range and the hold range. The lock range represents the range of input frequencies over which the PLL can acquire the lock condition. Once locked, the hold range represents the range of input frequencies over which the loop can remain locked. Type 1 phase detectors are generally limited to capture ranges of perhaps +−30%, far too narrow for the 0.1 to 10 Hz range desired. Therefore a Type 2 detector is preferred, which has can easily function over a 1:1000 range of frequencies. A type 2 phase detector also has the advantage that in the locked state the PLL output 214 will be precisely in-phase with input 212N that represents the beat, while Type 1 detectors will yield outputs that typically lead or lag the beat. However, for situations when the beat is known to be in a relatively narrow range (such as in some dance music), a Type 1 detector can be employed. This has the advantage of greater noise immunity, and can be connected to the output 208N of the detector 110N directly without filtering.

[0049] It will be recognized to one skilled in the art that other PLL topologies and schematic designs with additional time constants and/or nonlinearities can be employed in this invention.

[0050] One skilled in the art will appreciate that various elements of the APU can achieved by different techniques, such as digital techniques, without departing from the present invention. For example, the band pass filters 106a-N, the AGC amplifiers 108a-N, the detectors 110a-N, and the first low pass filters 112a-N, can be digitally implemented. One of ordinary skill in the art will recognize a digital band pass filter could be implemented by using a fast Fourier transform (FFT) mathematical technique. Once converted to a digital signal with a standard analog to digital converter (A/D), also commonly known in the art, the entire composite time domain signal can be converted to a frequency domain spectral representation by performing an FFT. Selected bands of this frequency spectrum representation can be separated from other digital data to yield a band pass function. An inverse FFT process can then regenerate a time domain representation of the band passed signal. One skilled in the art will recognize other digital techniques for implementation of a digital band pass filter, including FIR and IIR filter designs and/or wavelet decomposition. One skilled in the art will also recognize that the digital band pass filter can be replaced with a digital Kalman filter.

[0051] The AGC 108a-N can also be digitally implemented. The magnitude can be digitally detected by processing the digital values from the A/D converter. A desired scaling factor can be determined and used to scale the digital values. Techniques known in the art can multiply or divide the digital signal data values by the scaling factor. Thus, an AGC amplifier can be implemented digitally.

[0052] The detector 110a-N and first low pass filter 112a-N functions can also be digitally implemented. Once the signal is converted from an analog signal to a digital signal with an analog to digital converter, (A/D), digitally band pass filtered and amplified as above, the detector can be performed with a simple mathematical absolute value function upon the sampled digital values. The low pass filter 112a-N can also be achieved digitally with well known techniques including finite impulse response (FIR) and infinite impulse response (IIR) filters.

[0053] One of ordinary skill in the art will also recognize that the PLL 104 can also be digitally implemented whereby the loop filter 126 and VCO 122aare provided using digital techniques.

[0054] One skilled in the art will recognize that digital implementations allow more sophisticated processing techniques to be employed. Such techniques include neural networks that can allow a hearing person to first ‘teach’ the system of this invention. Such techniques also include pre-selection by the user of the type of beat to expect, for example, a waltz.

[0055] One skilled in the art will recognize that any or all of the aforementioned electronic circuitry can be integrated into custom integrated circuits.

[0056] In an exemplary embodiment, the beat detection system is portably mounted to the user. In alternative embodiments, one or more of the elements of the invention are stationarily mounted. In further embodiments, the system includes a central processing station, for example a stationary APU connected to a stationary radio transmitter that broadcasts beat information to a radio receiver and tactile actuator sets mounted to the user. In further embodiments, the system includes a stationary radio transmitter that broadcasts music to a radio receiver, APU, and tactile actuator sets mounted to the user. In other embodiments, the APU can be directly connected to a stereo or other receiver. In still other embodiments, visible lights can provide the beat information.

[0057] One of ordinary skill in the arts will appreciate further features and advantages of the invention based on the above-described embodiments. Accordingly, the invention is not to be limited by what has been particularly shown and described, except as indicated by the appended claims. All publications and references cited herein are expressly incorporated herein by reference in their entirety.