US 9,401,158 B1Grant
Microphone signal fusion
Issue Date:2016-07-26
•28 Claims
•9 Drawing Sheets
Abstract
Provided are systems and methods for microphone signal fusion. An example method commences with receiving a first and second signal representing sounds captured, respectively, by internal and external microphones. The second signal includes at least a voice component. The first signal and the voice component are modified by at least human tissue. The first and second signals are processed to obtain noise estimates. The first signal is aligned with the second signal. The second signal and the aligned first signal are blended based on the noise estimates to generate an enhanced voice signal. The internal microphone is located inside an ear canal and sealed for isolation from acoustic signals outside the ear canal. The external microphone is located outside the ear canal. All of parts of the processing, blending and aligning of the systems and method may be performed on a subband basis in the frequency domain.
Metadata
Assignee
- Knowles Electronics, LLC
Inventors
- Kuan-Chieh Yen
- Thomas Miller
- Mushtaq Syed
Application Information
Application Number:US 14/853,947
Filing Date:2015-09-14
Priority Date:2015-09-14
Art Unit:2654
Classifications
IPC:
H04R3/00G10L21/02G10L21/0232G10L21/0208
Field of Search:
H04R 2420/00H04R 2420/01H04R 2420/07H04R 2460/00H04R 2460/01H04R 2460/03H04R 2460/05H04R 2460/09H04R 2460/11H04R 2460/15H04R 2201/00H04R 2201/003H04R 2201/02H04R 2201/10H04R 2201/107H04R 2201/109H04R 2201/34H04R 2201/40H04R 2201/405H04R 2225/00H04R 2225/023H04R 2225/025H04R 2225/39H04R 2225/41H04R 2225/43H04R 2225/53H04R 2225/55H04R 2225/61H04R 3/005H04R 2410/05G10L 21/0205G10L 2021/02087
Patent Drawings (9 sheets)
Description
FIELD
[0001] The present application relates generally to audio processing and, more specifically, to systems and methods for fusion of microphone signals.
BACKGROUND
[0002] The proliferation of smart phones, tablets, and other mobile devices has fundamentally changed the way people access information and communicate. People now make phone calls in diverse places such as crowded bars, busy city streets, and windy outdoors, where adverse acoustic conditions pose severe challenges to the quality of voice communication. Additionally, voice commands have become an important method for interaction with electronic devices in applications where users have to keep their eyes and hands on the primary task, such as, for example, driving. As electronic devices become increasingly compact, voice command may become the preferred method of interaction with electronic devices. However, despite recent advances in speech technology, recognizing voice in noisy conditions remains difficult. Therefore, mitigating the impact of noise is important to both the quality of voice communication and performance of voice recognition.
[0003] Headsets have been a natural extension of telephony terminals and music players as they provide hands-free convenience and privacy when used. Compared to other hands-free options, a headset represents an option in which microphones can be placed at locations near the user's mouth, with constrained geometry among user's mouth and microphones. This results in microphone signals that have better signal-to-noise ratios (SNRs) and are simpler to control when applying multi-microphone based noise reduction. However, when compared to traditional handset usage, headset microphones are relatively remote from the user's mouth. As a result, the headset does not provide the noise shielding effect provided by the user's hand and the bulk of the handset. As headsets have become smaller and lighter in recent years due to the demand for headsets to be subtle and out-of-way, this problem becomes even more challenging.
[0004] When a user wears a headset, the user's ear canals are naturally shielded from outside acoustic environment. If a headset provides tight acoustic sealing to the ear canal, a microphone placed inside the ear canal (the internal microphone) would be acoustically isolated from outside environment such that environmental noise would be significantly attenuated. Additionally, a microphone inside a sealed ear canal is free of wind-buffeting effect. On the other hand, a user's voice can be conducted through various tissues in user's head to reach the ear canal, because it is trapped inside of the ear canal. A signal picked up by the internal microphone should thus have much higher SNR compared to the microphone outside of the user's ear canal (the external microphone).
[0005] Internal microphone signals are not free of issues, however. First of all, the body-conducted voice tends to have its high-frequency content severely attenuated and thus has much narrower effective bandwidth compared to voice conducted through air. Furthermore, when the body-conducted voice is sealed inside an ear canal, it forms standing waves inside the ear canal. As a result, the voice picked up by the internal microphone often sounds muffled and reverberant while lacking the natural timbre of the voice picked up by the external microphones. Moreover, effective bandwidth and standing-wave patterns vary significantly across different users and headset fitting conditions. Finally, if a loudspeaker is also located in the same ear canal, sounds made by the loudspeaker would also be picked by the internal microphone. Even with acoustic echo cancellation (AEC), the close coupling between the loudspeaker and internal microphone often leads to severe voice distortion after AEC.
[0006] Other efforts have been attempted in the past to take advantage of the unique characteristics of the internal microphone signal for superior noise reduction performance. However, attaining consistent performance across different users and different usage conditions has remained challenging.
SUMMARY
[0007] This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
[0008] According to one aspect of the described technology, an example method for fusion of microphone signals is provided. In various embodiments, the method includes receiving a first signal and a second signal. The first signal includes at least a voice component. The second signal includes the voice component modified by at least a human tissue. The method also includes processing the first signal to obtain first noise estimates. The method further includes aligning the second signal with the first signal. Blending, based at least on the first noise estimates, the first signal and the aligned second signal to generate an enhanced voice signal is also included in the method. In some embodiments, the method includes processing the second signal to obtain second noise estimates and the blending is based at least on the first noise estimates and the second noise estimates.
[0009] In some embodiments, the second signal represents at least one sound captured by an internal microphone located inside an ear canal. In certain embodiments, the internal microphone may be sealed during use for providing isolation from acoustic signals coming outside the ear canal, or it may be partially sealed depending on the user and the user's placement of the internal microphone in the ear canal.
[0010] In some embodiments, the first signal represents at least one sound captured by an external microphone located outside an ear canal.
[0011] In some embodiments, the method further includes performing noise reduction of the first signal based on the first noise estimates before aligning the signals. In other embodiments, the method further includes performing noise reduction of the first signal based on the first noise estimates and noise reduction of the second signal based on the second noise estimates before aligning the signals.
[0012] According to another aspect of the present disclosure, a system for fusion of microphone signals is provided. The example system includes a digital signal processor configured to receive a first signal and a second signal. The first signal includes at least a voice component. The second signal includes at least the voice component modified by at least a human tissue. The digital signal processor is operable to process the first signal to obtain first noise estimates and in some embodiments, to process the second signal to obtain second noise estimates. In the example system, the digital signal processor aligns the second signal with the first signal and blends, based at least on the first noise estimates, the first signal and the aligned second signal to generate an enhanced voice signal. In some embodiments, the digital signal processor aligns the second signal with the first signal and blends, based at least on the first noise estimates and the second noise estimates, the first signal and the aligned second signal to generate an enhanced voice signal.
[0013] In some embodiments, the system includes an internal microphone and an external microphone. In certain embodiments, the internal microphone may be sealed during use for providing isolation from acoustic signals coming outside the ear canal, or it may be partially sealed depending on the user and the user's placement of the internal microphone in the ear canal. The second signal may represent at least one sound captured by the internal microphone. The external microphone is located outside the ear canal. The first signal may represent at least one sound captured by the external microphone.
[0014] According to another example, embodiments of the present disclosure, the steps of the method for fusion of microphone signals are stored on a non-transitory machine-readable medium comprising instructions, which when implemented by one or more processors perform the recited steps.
[0015] Other example embodiments of the disclosure and aspects will become apparent from the following description taken in conjunction with the following drawings.
BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Embodiments are illustrated by way of example and not limitation in the figures of the accompanying drawings, in which like references indicate similar elements.
[0017] FIG. 1 is a block diagram of a system and an environment in which the system is used, according to an example embodiment.
[0018] FIG. 2 is a block diagram of a headset suitable for implementing the present technology, according to an example embodiment.
[0019] FIGS. 3-5 are examples of waveforms and spectral distributions of signals captured by an external microphone and an internal microphone.
[0020] FIG. 6 is a block diagram illustrating details of a digital processing unit for fusion of microphone signals, according to an example embodiment.
[0021] FIG. 7 is a flow chart showing a method for microphone signal fusion, according to an example embodiment.
[0022] FIG. 8 is a computer system which can be used to implement methods for the present technology, according to an example embodiment.
DETAILED DESCRIPTION
[0023] The technology disclosed herein relates to systems and methods for fusion of microphone signals. Various embodiments of the present technology may be practiced with mobile devices configured to receive and/or provide audio to other devices such as, for example, cellular phones, phone handsets, headsets, wearables, and conferencing systems.
[0024] Various embodiments of the present disclosure provide seamless fusion of at least one internal microphone signal and at least one external microphone signal utilizing the contrasting characteristics of the two signals for achieving an optimal balance between noise reduction and voice quality.
[0025] According to an example embodiment, a method for fusion of microphone signals may commence with receiving a first signal and a second signal. The first signal includes at least a voice component. The second signal includes the voice component modified by at least a human tissue. The example method provides for processing the first signal to obtain first noise estimates and in some embodiments, processing the second signal to obtain second noise estimates. The method may include aligning the second signal with the first signal. The method can provide blending, based at least on the first noise estimates (and in some embodiments, also based on the second noise estimates), the first signal and the aligned second signal to generate an enhanced voice signal.
[0026] Referring now to FIG. 1 , a block diagram of an example system 100 for fusion of microphone signals and environment thereof is shown. The example system 100 includes at least an internal microphone 106, an external microphone 108, a digital signal processor (DSP) 112, and a radio or wired interface 114. The internal microphone 106 is located inside a user's ear canal 104 and is relatively shielded from the outside acoustic environment 102. The external microphone 108 is located outside of the user's ear canal 104 and is exposed to the outside acoustic environment 102.
[0027] In various embodiments, the microphones 106 and 108 are either analog or digital. In either case, the outputs from the microphones are converted into synchronized pulse coded modulation (PCM) format at a suitable sampling frequency and connected to the input port of the DSP 112. The signals xin and xex denote signals representing sounds captured by the internal microphone 106 and external microphone 108, respectively.
[0028] The DSP 112 performs appropriate signal processing tasks to improve the quality of microphone signals xin and xex. The output of DSP 112, referred to as the send-out signal (sout), is transmitted to the desired destination, for example, to a network or host device 116 (see signal identified as sout uplink), through a radio or wired interface 114.
[0029] If a two-way voice communication is needed, a signal is received by the network or host device 116 from a suitable source (e.g., via the radio or wired interface 114). This is referred to as the receive-in signal (rin) (identified as rin downlink at the network or host device 116). The receive-in signal can be coupled via the radio or wired interface 114 to the DSP 112 for necessary processing. The resulting signal, referred to as the receive-out signal (rout), is converted into an analog signal through a digital-to-analog convertor (DAC) 110 and then connected to a loudspeaker 118 in order to be presented to the user. In some embodiments, the loudspeaker 118 is located in the same ear canal 104 as the internal microphone 106. In other embodiments, the loudspeaker 118 is located in the ear canal opposite to the ear canal 104. In example of FIG. 1 , the loudspeaker 118 is found in the same ear canal as the internal microphone 106, therefore, an acoustic echo canceller (AEC) can be needed to prevent the feedback of the received signal to the other end. Optionally, in some embodiments, if no further processing on the received signal is necessary, the receive-in signal (rin) can be coupled to the loudspeaker without going through the DSP 112.
[0030] FIG. 2 shows an example headset 200 suitable for implementing methods of the present disclosure. The headset 200 includes example inside-the-ear (ITE) module(s) 202 and behind-the-ear (BTE) modules 204 and 206 for each ear of a user. The ITE module(s) 202 are configured to be inserted into the user's ear canals. The BTE modules 204 and 206 are configured to be placed behind the user's ears. In some embodiments, the headset 200 communicates with host devices through a Bluetooth radio link. The Bluetooth radio link may conform to a Bluetooth Low Energy (BLE) or other Bluetooth standard and may be variously encrypted for privacy.
[0031] In various embodiments, ITE module(s) 202 includes internal microphone 106 and the loudspeaker 118, both facing inward with respect to the ear canal. The ITE module(s) 202 can provide acoustic isolation between the ear canal(s) 104 and the outside acoustic environment 102.
[0032] In some embodiments, each of the BTE modules 204 and 206 includes at least one external microphone. The BTE module 204 may include a DSP, control button(s), and Bluetooth radio link to host devices. The BTE module 206 can include a suitable battery with charging circuitry.
Characteristics of Microphone Signals
[0033] The external microphone 108 is exposed to the outside acoustic environment. The user's voice is transmitted to the external microphone 108 through the air. When the external microphone 108 is placed reasonably close to the user's mouth and free of obstruction, the voice picked up by the external microphone 108 sounds natural. However, in various embodiments, the external microphone 108 is exposed to environmental noises such as noise generated by wind, cars, and babble background speech. When present, environmental noise reduces the quality of the external microphone signal and can make voice communication and recognition difficult.
[0034] The internal microphone 106 is located inside the user's ear canal. When the ITE module(s) 202 provides good acoustic isolation from outside environment (e.g., providing a good seal), the user's voice is transmitted to the internal microphone 106 mainly through body conduction. Due to the anatomy of human body, the high-frequency content of the body-conducted voice is severely attenuated compared to the low-frequency content and often falls below a predetermined noise floor. Therefore, the voice picked up by the internal microphone 106 can sound muffled. The degree of muffling and frequency response perceived by a user can depend on the particular user's bone structure, particular configuration of the user's Eustachian tube (that connects the middle ear to the upper throat) and other related user anatomy. On the other hand, the internal microphone 106 is relatively free of the impact from environment noise due to the acoustic isolation.
[0035] FIG. 3 shows an example of waveforms and spectral distributions of signals 302 and 304 captured by the external microphone 108 and the internal microphone 106, respectively. The signals 302 and 304 include the user's voice. As illustrated in this example, the voice picked up by the internal microphone 106 has a much stronger spectral tilt toward the lower frequency. The higher-frequency content of signal 304 in the example waveforms is severely attenuated and thus results in a much narrower effective bandwidth compared to signal 302 picked up by the external microphone.
[0036] FIG. 4 shows another example of the waveforms and spectral distributions of signals 402 and 404 captured by external microphone 108 and internal microphone 106, respectively. The signals 402 and 404 include only wind noise in this example. The substantial difference in the signals 402 and 404 indicate that wind noise is evidently present at the external microphone 108 but is largely shielded from the internal microphone 106 in this example.
[0037] The effective bandwidth and spectral balance of the voice picked by the internal microphone 106 may vary significantly, depending on factors such as the anatomy of user's head, user's voice characteristics, and acoustic isolation provided by the ITE module(s) 202. Even with exactly the same user and headset, the condition can change significantly between wears. One of the most significant variables is the acoustic isolation provided by the ITE module(s) 202. When the sealing of the ITE module(s) 202 is tight, user's voice reaches internal microphone mainly through body conduction and its energy is well retained inside the ear canal. Since due to the tight sealing the environment noise is largely blocked from entering the ear canal, the signal at the internal microphone has very high signal-to-noise ratio (SNR) but often with very limited effective bandwidth. When the acoustic leakage between outside environment and ear canal becomes significant (e.g., due to partial sealing of the ITE module(s) 202), the user's voice can reach the internal microphone also through air conduction, thus the effective bandwidth improves. However, as the environment noise enters the ear canal and body-conducted voice escapes out of ear canal, the SNR at the internal microphone 106 can also decrease.
[0038] FIG. 5 shows yet another example of the waveforms and spectral distributions of signals 502 and 504 captured by external microphone 108 and internal microphone 106, respectively. The signals 502 and 504 include the user's voice. The internal microphone signal 504 in FIG. 5 has stronger lower-frequency content than the internal microphone signal 304 of FIG. 3 , but has a very strong roll-off after 2.0-2.5 kHz. In contrast, the internal microphone signal 304 in FIG. 3 has a lower level, but has significant voice content up to 4.0-4.5 kHz in this example.
[0039] FIG. 6 illustrates a block diagram of DSP 112 suitable for fusion of microphone signals, according to various embodiments of the present disclosure. The signals xin and xex are signals representing sounds captured from, respectively, the internal microphone 106 and external microphone 108. The signals xin and xex need not be the signals directly from the respective microphones; they may represent the signals that are directly from the respective microphones. For example, the direct signal outputs from the microphones may be preprocessed in some way, for example, conversion into synchronized pulse coded modulation (PCM) format at a suitable sampling frequency, with the converted signal being the signals processed by the method.
[0040] In the example in FIG. 6 , the signals xin and xex are first processed by a noise tracking/noise reduction (NT/NR) modules 602 and 604 to obtain running estimate of the noise level picked up at each microphone. Optionally, noise reduction (NR) can be performed by NT/NR modules 602 and 604 by utilizing the estimated noise level. In various embodiments, the microphone signals xin and xex, with or without NR, and noise estimates (e.g., “external noise and SNR estimates” output from NT/NR 602 and/or “internal noise and SNR estimates” output from NT/NR 604) from the NT/NR modules 602 and 604 are sent to a microphone spectral alignment (MSA) module 606, where a spectral alignment filter is adaptively estimated and applied to the internal microphone signal xin. A primary purpose of MSA is to spectrally align the voice picked up at the internal microphone 106 to the voice picked up at the external microphone 108 within the effective bandwidth of the in-canal voice signal.
[0041] The external microphone signal xex, the spectrally-aligned internal microphone signal xin,align, and the estimated noise levels at both microphones 106 and 108 are then sent to a microphone signal blending (MSB) module 608, where the two microphone signals are intelligently combined based on the current signal and noise conditions to form a single output with optimal voice quality.
[0042] Further details regarding the modules in FIG. 6 are set forth variously below.
[0043] In various embodiments, the modules 602-608 (NT/NR, MSA, and MSB) operate in a fullband domain (a time domain) or a certain subband domain (frequency domain). For embodiments having a module operating in a subband domain, a suitable analysis filterbank (AFB) is applied, for the input to the module, to convert each time-domain input signal into the subband domain. A matching synthesis filterbank (SFB) is provided in some embodiments, to convert each subband output signal back to the time domain as needed depending on the domain of the receiving module.
[0044] Examples of the filterbanks include Digital Fourier Transform (DFT) filterbank, Modified Digital Cosine Transform (MDCT) filterbank, 1/3-Octave filterbank, Wavelet filterbank, or other suitable perceptually inspired filterbanks. If consecutive modules 602-608 operate in the same subband domain, the intermediate AFBs and SFBs may be removed for maximum efficiency and minimum system latency. Even if two consecutive modules 602-608 operate in different subband domains in some embodiments, their synergy can be utilized by combining the SFB of the earlier module and the AFB of the later module for minimized latency and computation. In various embodiments, all processing modules 602-608 operate in the same subband domain.
[0045] Before the microphone signals reach any of the modules 602-608, they may be processed by suitable pre-processing modules such as direct current (DC)-blocking filters, wind buffeting mitigation (WBM), AEC, and the like. Similarly, the output from the MSB module 608 can be further processed by suitable post-processing modules such as static or dynamic equalization (EQ) and automatic gain control (AGC). Furthermore, other processing modules can be inserted into the processing flow shown in FIG. 6 , as long as the inserted modules do not interfere with the operation of various embodiments of the present technology.
Further Details of the Processing Modules
Noise Tracking/Noise Reduction (NT/NR) Module
[0046] The primary purpose of the NT/NR modules 602 and 604 is to obtain running noise estimates (noise level and SNR) in the microphone signals. These running estimates are further provided to subsequent modules to facilitate their operations. Normally, noise tracking is more effective when it is performed in a subband domain with sufficient frequency resolution. For example, when a DFT filterbank is used, the DFT sizes of 128 and 256 are preferred for sampling rates of 8 and 16 kHz, respectively. This results in 62.5 Hz/band, which satisfies the requirement for lower frequency bands (<750 Hz). Frequency resolution can be reduced for frequency bands above 1 kHz. For these higher frequency bands, the required frequency resolution may be substantially proportional to the center frequency of the band.
[0047] In various embodiments, a subband noise level with sufficient frequency resolution provides richer information with regards to noise. Because different types of noise may have very different spectral distribution, noise with the same fullband level can have very different perceptual impact. Subband SNR is also more resilient to equalization performed on the signal, so subband SNR of an internal microphone signal estimated, in accordance with the present technology, remains valid after the spectral alignment performed by the subsequent MSA module.
[0048] Many noise reduction methods are based on effective tracking of noise level and thus may be leveraged for the NT/NR module. Noise reduction performed at this stage can improve the quality of microphone signals going into subsequent modules. In some embodiments, the estimates obtained at the NT/NR modules are combined with information obtained in other modules to perform noise reduction at a later stage. By way of example and not limitation, suitable noise reduction methods is described by Ephraim and Malah, “Speech Enhancement Using a Minimum Mean-Square Error Short-Time Spectral Amplitude Estimator,” IEEE Transactions on Acoustics, Speech, and Signal Processing, December 1984., which is incorporated herein by reference in its entirety for the above purposes.
Microphone Spectral Alignment (MSA) Module
[0049] In various embodiments, the primary purpose of the MSA module 606 is to spectrally align voice signals picked up by the internal and external microphones in order to provide signals for the seamlessly blending of the two voice signals at the subsequent MSB module 608. As discussed above, the voice picked up by the external microphone 108 is typically more spectrally balanced and thus more naturally-sounding. On the other hand, the voice picked up by the internal microphone 106 can tend to lose high-frequency content. Therefore, the MSA module 606, in the example in FIG. 6 , functions to spectrally align the voice at internal microphone 106 to the voice at external microphone 108 within the effective bandwidth of the internal microphone voice. Although the alignment of spectral amplitude is the primary concern in various embodiments, the alignment of spectral phase is also a concern to achieve optimal results. Conceptually, microphone spectral alignment (MSA) can be achieved by applying a spectral alignment filter (HSA) to the internal microphone signal:
Xin,align(f)=HSA(f)Xin(f) (1)
where Xin(f) and Xin,align(f) are the frequency responses of the original and spectrally-aligned internal microphone signals, respectively. The spectral alignment filter, in this example, needs to satisfy the following criterion:
Xin,align(f)=HSA(f)Xin(f) (1)
where Xin(f) and Xin,align(f) are the frequency responses of the original and spectrally-aligned internal microphone signals, respectively. The spectral alignment filter, in this example, needs to satisfy the following criterion:
[0050]
where Ωin,voice is the effective bandwidth of the voice in the ear canal, Xex,voice(f) and Xin,voice(f) are the frequency responses of the voice signals picked up by the external and internal microphones, respectively. In various embodiments, the exact value of δ is equation (2) is not critical, however, it should be a relatively small number to avoid amplifying the noise in the ear canal. The spectral alignment filter can be implemented in either the time domain or any subband domain. Depending on the physical location of the external microphone, addition of a suitable delay to the external microphone signal might be necessary to guarantee the causality of the required spectral alignment filter.
where Ωin,voice is the effective bandwidth of the voice in the ear canal, Xex,voice(f) and Xin,voice(f) are the frequency responses of the voice signals picked up by the external and internal microphones, respectively. In various embodiments, the exact value of δ is equation (2) is not critical, however, it should be a relatively small number to avoid amplifying the noise in the ear canal. The spectral alignment filter can be implemented in either the time domain or any subband domain. Depending on the physical location of the external microphone, addition of a suitable delay to the external microphone signal might be necessary to guarantee the causality of the required spectral alignment filter.
[0051] An intuitive method of obtaining a spectral alignment filter is to measure the spectral distributions of voice at external microphone and internal microphone and to construct a filter based on these measurements. This intuitive method could work fine in well-controlled scenarios. However, as discussed above, the spectral distribution of voice and noise in the ear canal is highly variable and dependent on factors specific to users, devices, and how well the device fits into the user's ear on a particular occasion (e.g., the sealing). Designing the alignment filter based on the average of all conditions would only work well under certain conditions. On the other hand, designing the filter based on a specific condition risks overfitting, which might leads to excessive distortion and noise artifacts. Thus, different design approaches are needed to achieve the desired balance.
Clustering Method
[0052] In various embodiments, voice signals picked up by external and internal microphones are collected to cover a diverse set of users, devices, and fitting conditions. An empirical spectral alignment filter can be estimated from each of these voice signal pairs. Heuristic or data-driven approaches may then be used to assign these empirical filters into clusters and to train a representative filter for each cluster. Collectively, the representative filters from all clusters form a set of candidate filters, in various embodiments. During the run-time operation, a rough estimate on the desired spectral alignment filter response can be obtained and used to select the most suitable candidate filter to be applied to the internal microphone signal.
[0053] Alternatively, in other embodiments, a set of features is extracted from the collected voice signal pairs along with the empirical filters. These features should be more observable and correlate to variability of the ideal response of spectral alignment filter, such as the fundamental frequency of the voice, spectral slope of the internal microphone voice, volume of the voice, and SNR inside of ear canal. In some embodiments, these features are added into the clustering process such that a representative filter and a representative feature vector is trained for each cluster. During the run-time operation, the same feature set may be extracted and compared to these representative feature vectors to find the closest match. In various embodiments, the candidate filter that is from the same cluster as the closest-matched feature vector is then applied to the internal microphone signal.
[0054] By way of example and not limitation, an example cluster tracker method is described in U.S. patent application Ser. No. 13/492,780, entitled “Noise Reduction Using Multi-Feature Cluster Tracker,” (issued Apr. 14, 2015 as U.S. Pat. No. 9,008,329), which is incorporated herein by reference in its entirety for the above purposes.
Adaptive Method
[0055] Other than selecting from a set of pre-trained candidates, adaptive filtering approach can be applied to estimate the spectral alignment filter from the external and internal microphone signals. Because the voice components at the microphones are not directly observable and the effective bandwidth of the voice in the ear canal is uncertain, the criterion stated in Eq. (2) is modified for practical purpose as:
[0056]
[0057] where superscript * represents complex conjugate and E{•} represents a statistical expectation. If the ear canal is effectively shielded from outside acoustic environment, the voice signal would be the only contributor to the cross-correlation term at the numerator in Eq. (3) and the auto-correlation term at the denominator in Eq. (3) would be the power of voice at the internal microphone within its effective bandwidth. Outside of its effective bandwidth, the denominator term would be the power of noise floor at the internal microphone and the numerator term would approach 0. It can be shown that the filter estimated based on Eq. (3) is the minimum mean-squared error (MMSE) estimator of the criterion stated in Eq. (2).
[0058] When the acoustic leakage between the outside environment and the ear canal becomes significant, the filter estimated based on Eq. (3) is no longer an MMSE estimator of Eq. (2) because the noise leaked into the ear canal also contributes to the cross-correlation between the microphone signals. As a result, the estimator in Eq. (3) would have bi-modal distribution, with the mode associated with voice representing the unbiased estimator and the mode associated with noise contributing to the bias. Minimizing the impact of acoustic leakage can require proper adaptation control.
Example embodiments for providing this proper adaptation control are described in further detail below.
Time-Domain Implementations
[0059] In some embodiments, the spectral alignment filter defined in Eq. (3) can be converted into time-domain representation as follows:
hSA=E{xin*(n)xinT(n)}−1E(xin*(n)xex(n)) (4)
where hSA is a vector consisting of the coefficients of a length-N finite impulse response (FIR) filter:
hSA=[hSA(0)hSA(1) . . . hSA(N−1)]T (5)
and xex (n) and xin (n) are signal vectors consisting of the latest N samples of the corresponding signals at time n:
x(n)=[x(n)×(n−1) . . . x(n−N+1)]T (6)
where the superscript T represents a vector or matrix transpose, and the superscript H represents a Hermitian transpose. The spectrally-aligned internal microphone signal can be obtained by applying the spectral alignment filter to the internal microphone signal:
xin,align(n)=xinT(n)hSA (7)
hSA=E{xin*(n)xinT(n)}−1E(xin*(n)xex(n)) (4)
where hSA is a vector consisting of the coefficients of a length-N finite impulse response (FIR) filter:
hSA=[hSA(0)hSA(1) . . . hSA(N−1)]T (5)
and xex (n) and xin (n) are signal vectors consisting of the latest N samples of the corresponding signals at time n:
x(n)=[x(n)×(n−1) . . . x(n−N+1)]T (6)
where the superscript T represents a vector or matrix transpose, and the superscript H represents a Hermitian transpose. The spectrally-aligned internal microphone signal can be obtained by applying the spectral alignment filter to the internal microphone signal:
xin,align(n)=xinT(n)hSA (7)
[0060] In various embodiments, many adaptive filtering approaches can be adopted to implement the filter defined in Eq. (4). One such approach is:
ĥSA(n)=Rin,in−1(n)ex,in(n) (8)
where ĥSA (n) is the filter estimate at time n. Rin,in(n) and rex,in(n) are the running estimates of E{xin*(n)xinT(n)} and E{xin*(n)xex(n)}, respectively. These running estimates can be computed as:
Rin,in(n)=Rin,in(n−1)+αSA(n)(xin*(n)xinT(n)−Rin,in(n−1)) (9)
rex,in(n)=rex,in(n−1)+αSA(n)(xin*(n)xex(n)−rex,in(n−1)) (10)
where αSA(n) is an adaptive smoothing factor defined as:
αSA(n)=αSA0ΓSA(n) (11).
ĥSA(n)=Rin,in−1(n)ex,in(n) (8)
where ĥSA (n) is the filter estimate at time n. Rin,in(n) and rex,in(n) are the running estimates of E{xin*(n)xinT(n)} and E{xin*(n)xex(n)}, respectively. These running estimates can be computed as:
Rin,in(n)=Rin,in(n−1)+αSA(n)(xin*(n)xinT(n)−Rin,in(n−1)) (9)
rex,in(n)=rex,in(n−1)+αSA(n)(xin*(n)xex(n)−rex,in(n−1)) (10)
where αSA(n) is an adaptive smoothing factor defined as:
αSA(n)=αSA0ΓSA(n) (11).
[0061] The base smoothing constant αSA0 determines how fast the running estimates are updated. It takes a value between 0 and 1, with the larger value corresponding to shorter base smoothing time window. The speech likelihood estimate ΓSA (n) also takes Values between 0 and 1, with 1 indicating certainty of speech dominance and 0 indicating certainty of speech absence. This approach provides the adaptation control needed to minimize the impact of acoustic leakage and maintain the estimated spectral alignment filter unbiased. Details about ΓSA (n) will be further discussed below.
[0062] The filter adaptation shown in Eq. (8) can require matrix inversion. As the filter length N increases, this becomes both computationally complex and numerically challenging. In some embodiments, a least mean-square (LMS) adaptive filter implementation is adopted for the filter defined in Eq. (4):
[0063]
where μSA is a constant adaptation step size between 0 and 1, ∥xin(n)∥ is the norm of vector xin (n), and eSA (n) is the spectral alignment error defined as:
eSA(n)=xex(n)−xinT(n)ĥSA(n) (13)
where μSA is a constant adaptation step size between 0 and 1, ∥xin(n)∥ is the norm of vector xin (n), and eSA (n) is the spectral alignment error defined as:
eSA(n)=xex(n)−xinT(n)ĥSA(n) (13)
[0064] Similar to the direct approach shown in Eqs. (8)-(11), the speech likelihood estimate δSA (n) can be used to control the filter adaptation in order to minimize the impact of acoustic leakage on filter adaptation.
[0065] Comparing the two approaches, the LMS converges slower, but is more computationally efficient and numerically stable. This trade-off is more significant as the filter length increases. Other types of adaptive filtering techniques, such as fast affine projection (FAP) or lattice-ladder structure, can also be applied to achieve different trade-offs. The key is to design an effective adaptation control mechanism for these other techniques. In various embodiments, implementation in a suitable subband domain can result in a better trade-off on convergence, computational efficiency, and numerical stability. Subband-domain implementations are described in further detail below.
Subband-Domain Implementations
[0066] When converting time-domain signals into a subband domain, the effective bandwidth of each subband is only a fraction of the fullband bandwidth. Therefore, down-sampling is usually performed to remove redundancy and the down-sampling factor D typically increases with the frequency resolution. After converting the microphone signals xex (n) and xin(n) into a subband domain, the signals in the k-th are denoted as xex,k (m) and xin,k(m), respectively, where m is sample index (or frame index) in the down-sampled discrete time scale and is typically defined as m=n/D.
[0067] The spectral alignment filter defined in Eq. (3) can be converted into a subband-domain representation as:
hSA,k=E{xin,k*(m)xin,kT(m)}−1E{xin,k*(m)xex,k(m)} (14)
which is implemented in parallel in each of the subbands (k=0, 1, . . . , K). Vector hSA,k consists of the coefficients of a length-M FIR filter for subband k:
hSA,k=[hSA,k(0)hSA,k(1)] . . . hSA,k(M−1)T (15)
and xex,k (m) and xim,k (m) are signal vectors consisting of the latest M samples of the corresponding subband signals at time m:
xk(m)=[xk(m)xk(m−1) . . . xk(m−M+1)]T (16)
hSA,k=E{xin,k*(m)xin,kT(m)}−1E{xin,k*(m)xex,k(m)} (14)
which is implemented in parallel in each of the subbands (k=0, 1, . . . , K). Vector hSA,k consists of the coefficients of a length-M FIR filter for subband k:
hSA,k=[hSA,k(0)hSA,k(1)] . . . hSA,k(M−1)T (15)
and xex,k (m) and xim,k (m) are signal vectors consisting of the latest M samples of the corresponding subband signals at time m:
xk(m)=[xk(m)xk(m−1) . . . xk(m−M+1)]T (16)
[0068] In various embodiments, due to down-sampling, the filter length required in the subband domain to cover similar time span is much shorter than that in the time domain. Typically, the relationship between M and N is M=┌N/D┐. If the subband sample rate (frame rate) is at or slower than 8 mini-second (ms) per frame, as typically is the case for speech signal processing, M is often down to 1 for headset applications due to the proximity of all microphones. In that case, Eq. (14) can be simplified to:
hSA,k=E{xex,k(m)xin,k*(m)}/E{∥xin,k(m)|2} (17)
where hSA,k is a complex single-tap filter. The subband spectrally-aligned internal microphone signal can be obtained by applying the subband spectral alignment filter to the subband internal microphone signal:
xin,align,k(m)=hSA,kxin,k(m). (18)
hSA,k=E{xex,k(m)xin,k*(m)}/E{∥xin,k(m)|2} (17)
where hSA,k is a complex single-tap filter. The subband spectrally-aligned internal microphone signal can be obtained by applying the subband spectral alignment filter to the subband internal microphone signal:
xin,align,k(m)=hSA,kxin,k(m). (18)
[0069] The direct adaptive filter implementation of the subband filter defined in Eq. (17) can be formulated as:
ĥSA,k(m)=rex,in,k(m)/rin,in,k(m) (19)
where ĥSA,k(m) is the filter estimate at frame m, and rin,in,k(m) and rex,in,k (m) are the running estimates of E{|xin,k(m)|2} and E{xex,k(m)xin,k*(m)}, respectively. These running estimates can be computed as:
rin,in,k(m)=rin,in,k(m−1)+αSA,k(m)(|xin,k(m)|2−rin,in,k(m−1)) (20)
rex,in,k(m)=rex,in,k(m−1)+αSA,k(m)(xex,k(m)xin,k*(m)−rex,in,k(m−1)) (21)
where αSA,k(m) is a subband adaptive smoothing factor defined as
αSA,k(m)=αSA0,kΓSA,k(m) (22)
ĥSA,k(m)=rex,in,k(m)/rin,in,k(m) (19)
where ĥSA,k(m) is the filter estimate at frame m, and rin,in,k(m) and rex,in,k (m) are the running estimates of E{|xin,k(m)|2} and E{xex,k(m)xin,k*(m)}, respectively. These running estimates can be computed as:
rin,in,k(m)=rin,in,k(m−1)+αSA,k(m)(|xin,k(m)|2−rin,in,k(m−1)) (20)
rex,in,k(m)=rex,in,k(m−1)+αSA,k(m)(xex,k(m)xin,k*(m)−rex,in,k(m−1)) (21)
where αSA,k(m) is a subband adaptive smoothing factor defined as
αSA,k(m)=αSA0,kΓSA,k(m) (22)
[0070] The subband base smoothing constant αsA0,k determines how fast the running estimates are updated in each subband. It takes a value between 0 and 1, with larger value corresponding to shorter base smoothing time window. The subband speech likelihood estimate ΓSA,k (m) also takes values between 0 and 1, with 1 indicating certainty of speech dominance and 0 indicating certainty of speech absence in this subband. Similar to the case in the time-domain, this provides the adaptation control needed to minimize the impact of acoustic leakage and maintain the estimated spectral alignment filter unbiased. However, because speech signals often are distributed unevenly across frequency, being able to separately control the adaptation in each subband provides the flexibility of a more refined control and thus better performance potential. In addition, the matrix inversion in Eq. (8) is reduced to a simple division operation in Eq. (19), such that computational and numerical issues are greatly reduced. The details about ΓSA,k (m) will be further discussed below.
[0071] Similar to the time-domain case, an LMS adaptive filter implementation can be adopted for the filter defined in Eq. (17):
[0072]
where μSA is a constant adaptation step size between 0 and 1, ∥xin,k(M)∥ is the norm of xin,k(m) and eSA,k(m) is the subband spectral alignment error defined as:
eSA,k(m)=xex,k(m)−ĥSA,k(m)xin,k(m). (24)
where μSA is a constant adaptation step size between 0 and 1, ∥xin,k(M)∥ is the norm of xin,k(m) and eSA,k(m) is the subband spectral alignment error defined as:
eSA,k(m)=xex,k(m)−ĥSA,k(m)xin,k(m). (24)
[0073] Similar to the direct approach shown in Eqs. (19)-(22), the subband speech likelihood estimate ΓSA,k (m) can be used to control the filter adaptation in order to minimize the impact of acoustic leakage on filter adaptation. Furthermore, because this is a single-tap LMS filter, the convergence is significantly faster than its time-domain counterpart shown in Eq. (12)-(13).
Speech Likelihood Estimate
[0074] The speech likelihood estimate ΓSA(n) in Eqs. (11) and (12) and the subband speech likelihood estimate ΓSA,k(m) in Eqs. (22) and (23) can provide adaptation control for the corresponding adaptive filters. There are many possibilities in formulating the subband likelihood estimate. One such example is:
[0075]
where ξex,k(m) and ξin,k(m) are the signal ratios in subband signals xex,k(m) and xin,k(m), respectively. They can be computed using the running noise power estimates (PNZ,ex,k (m) PNZ,in,k (m)) or SNR estimates (SNRex,k(m), SNRex,k(m)) provided by the NT/NR modules 602, such as:
where ξex,k(m) and ξin,k(m) are the signal ratios in subband signals xex,k(m) and xin,k(m), respectively. They can be computed using the running noise power estimates (PNZ,ex,k (m) PNZ,in,k (m)) or SNR estimates (SNRex,k(m), SNRex,k(m)) provided by the NT/NR modules 602, such as:
[0076]
[0077] As discussed above, the estimator of spectral alignment filter in Eq. (3) exhibits bi-modal distribution when there is significant acoustic leakage. Because the mode associated with voice generally has a smaller conditional mean than the mode associated with noise, the third term in Eq. (25) helps exclude the influence of the noise mode.
[0078] For the speech likelihood estimate ΓSA(n), one option is to simply substitute the components in Eq. (25) with their fullband counterpart. However, because the power of acoustic signals tends to concentrate in the lower frequency range, applying such a decision for time-domain adaptation control tends to not work well in the higher frequency range. Considering the limited bandwidth of voice at the internal microphone 106, this often leads to volatility in high frequency response of the estimated spectral alignment filter. Therefore, using perceptual-based frequency weighting, in various embodiments, to emphasize high-frequency power in computing the fullband SNR will lead to more balanced performance across frequency. Alternatively, using a weighted average of the subband speech likelihood estimates as the speech likelihood estimate also achieves a similar effect.
Microphone Signal Blending (MSB) Module
[0079] The primary purpose of the MSB module 608 is to combine the external microphone signal xex(n) and the spectrally-aligned internal microphone signal xin,align (n) to generate an output signal with the optimal trade-off between noise reduction and voice quality. This process can be implemented in either the time domain or subband domain. While the time-domain blending provides a simple and intuitive way of mixing the two signals, the subband-domain blending offers more control flexibility and thus a better potential of achieving a better trade-off between noise reduction and voice quality.
Time-Domain Blending
[0080] The time-domain blending can be formulated as follows:
Sout(n)=gSBxin,align(n+(1−gSB)xex(n) (27)
where gSB is the signal blending weight for the spectrally-aligned internal microphone signal which takes value between 0 and 1. It can be observed that the weights for xex(n) and xin,align (n) always sum up to 1. Because the two signals are spectrally aligned within the effective bandwidth of the voice in ear canal, the voice in the blended signal should stay consistent within this effective bandwidth as the weight changes. This is the primary benefit of performing amplitude and phase alignment in the MSA module 606.
Sout(n)=gSBxin,align(n+(1−gSB)xex(n) (27)
where gSB is the signal blending weight for the spectrally-aligned internal microphone signal which takes value between 0 and 1. It can be observed that the weights for xex(n) and xin,align (n) always sum up to 1. Because the two signals are spectrally aligned within the effective bandwidth of the voice in ear canal, the voice in the blended signal should stay consistent within this effective bandwidth as the weight changes. This is the primary benefit of performing amplitude and phase alignment in the MSA module 606.
[0081] Ideally, gSB should be 0 in quiet environments so the external microphone signal should then be used as the output in order to have a natural voice quality. On the other hand, gSB should be 1 in very noisy environment so the spectrally-aligned internal microphone signal should then be used as the output in order to take advantage of its reduced noise due to acoustic isolation from the outside environment. As the environment transits from quiet to noisy, the value of gSB increases and the blended output shifts from an external microphone toward an internal microphone. This also results in gradual loss of higher frequency voice content and, thus, the voice can become muffle sounding.
[0082] The transition process for the value of gSB can be discrete and driven by the estimate of the noise level at the external microphone (PNZ,ex) provided by the NT/NR module 602. For example, the range of noise level may be divided into (L+1) zones, with zone 0 covering quietest conditions and zone L covering noisiest conditions. The upper and lower thresholds for these zones should satisfy:
TSB,Hi,0<TSB,Hi,1< . . . <TSB,Hi,L−1
TSB,Lo,1<TSB,Lo,2< . . . <TSB,Lo,L (28)
where TSB,Hi,l and TSB,Lo,t are the upper and lower thresholds of zone l, l=0, 1, . . . , L. It should be noted that there is no lower bound for zone 0 and no upper bound for zone L. These thresholds should also satisfy:
TSB,Lo,l+1≦TSB,Hi,l≦TSB,Lo,l+2 (29)
such that there are overlaps between adjacent zones but not between non-adjacent zones. These overlaps serve as hysteresis that reduces signal distortion due to excessive back-and-forth switching between zones. For each of these zones, a candidate gSB value can be set. These candidates should satisfy:
gSB,0=0≦gSB,1≦gSB,2≦ . . . ≦gSB,L−1<gSB,L=1. (30)
TSB,Hi,0<TSB,Hi,1< . . . <TSB,Hi,L−1
TSB,Lo,1<TSB,Lo,2< . . . <TSB,Lo,L (28)
where TSB,Hi,l and TSB,Lo,t are the upper and lower thresholds of zone l, l=0, 1, . . . , L. It should be noted that there is no lower bound for zone 0 and no upper bound for zone L. These thresholds should also satisfy:
TSB,Lo,l+1≦TSB,Hi,l≦TSB,Lo,l+2 (29)
such that there are overlaps between adjacent zones but not between non-adjacent zones. These overlaps serve as hysteresis that reduces signal distortion due to excessive back-and-forth switching between zones. For each of these zones, a candidate gSB value can be set. These candidates should satisfy:
gSB,0=0≦gSB,1≦gSB,2≦ . . . ≦gSB,L−1<gSB,L=1. (30)
[0083] Because the noise condition changes at a much slower pace than the sampling frequency, the microphone signals can be divided into consecutive frames of samples and a running estimate of noise level at an external microphone can be tracked for each frame, denoted as PNZ,ex(m), where m is the frame index. Ideally, perceptual-based frequency weighting should be applied when aggregating the estimated noise spectral power into the fullband noise level estimate. This would make PNZ,ex (m) better correlate to the perceptual impact of current environment noise. By further denoting the noise zone at frame m as ΛSB (m), a state-machine based algorithm for the MSB module 608 can be defined as:
-
- [0084] 1. Initialize frame 0 as being in noise zone 0, i.e., ΛSB (0)=0.
- [0085] 2. If frame (m−1) is in noise zone 1, i.e., As (m−1)=1, the noise zone for frame m, ΛSB (m), is determined by comparing the noise level estimate PNZ,x (m) to the thresholds of noise zone l:
[0086]
-
- [0087] 3. Set the blending weight for xin,align (n) in frame m as a candidate in zone ΛSB(m):
gSB(m)=gSB,ΛSB (m) (32) - [0088] and use it to compute the blended output for frame m based on Eq. (27).
- [0089] 4. Return to step 2 for the next frame.
- [0087] 3. Set the blending weight for xin,align (n) in frame m as a candidate in zone ΛSB(m):
[0090] Alternatively, the transition process for the value of gSB can be continuous. Instead of dividing the range of a noise floor estimate into zones and assigning a blending weight in each of these zones, the relation between the noise level estimate and the blending weight can be defined as a continuous function:
gSB(m)=fSB(PNZ,ex(m)) (33)
where fSB (●) is a non-decreasing function of PNZ,ex(m) that has a range between 0 and 1. In some embodiments, other information such as noise level estimates from previous frames and SNR estimates can also be included in the process of determining the value of gSB (m). This can be achieved based on data-driven (machine learning) approaches or heuristic rules. By way of example and not limitation, examples of various machine learning and heuristic rules approaches are described in U.S. patent application Ser. No. 14/046,551, entitled “Noise Suppression for Speech Processing Based on Machine-Learning Mask Estimation”, filed Oct. 4, 2013.
gSB(m)=fSB(PNZ,ex(m)) (33)
where fSB (●) is a non-decreasing function of PNZ,ex(m) that has a range between 0 and 1. In some embodiments, other information such as noise level estimates from previous frames and SNR estimates can also be included in the process of determining the value of gSB (m). This can be achieved based on data-driven (machine learning) approaches or heuristic rules. By way of example and not limitation, examples of various machine learning and heuristic rules approaches are described in U.S. patent application Ser. No. 14/046,551, entitled “Noise Suppression for Speech Processing Based on Machine-Learning Mask Estimation”, filed Oct. 4, 2013.
Subband-Domain Blending
[0091] The time-domain blending provides a simple and intuitive mechanism for combining the internal and external microphone signals based on the environmental noise condition. However, in high noise conditions, a selection would result between having higher-frequency voice content with noise and having reduced noise with muffled voice quality. If the voice inside the ear canal has very limited effective bandwidth, its intelligibility can be very low. This severely limits the effectiveness of either voice communication or voice recognition. In addition, due to the lack of frequency resolution in the time-domain blending, a balance is performed between the switching artifact due to less frequent but more significant changes in blending weight and the distortion due to finer but more constant changes. In addition, the effectiveness of controlling the blending weights, for the time domain blending, based on estimated noise level is highly dependent on factors such as the tuning and gain settings in the audio chain, the locations of microphones, and the loudness of user's voice. On the other hand, using SNR as a control mechanism can be less effective in the time domain due to the lack of frequency resolution. In light of the limitation of the time-domain blending, subband-domain blending, according to various embodiments, may provide the flexibility and potential for improved robustness and performance for the MSB module.
[0092] In subband-domain blending, the signal blending process defined in Eq. (27) is applied to the subband external microphone signal xex,k(m) and the subband spectrally-aligned internal microphone signal xin,align,k(m) as:
Sout,k(m)=gSB,kxin,align,k(m)+(1−gB,k)xex,k(m) (34)
where k is the subband index and m is the frame index. The subband blended output sout,k(m) can be converted back to the time domain to form the blended output sout(n) or stay in the subband domain to be processed by subband processing modules downstream.
Sout,k(m)=gSB,kxin,align,k(m)+(1−gB,k)xex,k(m) (34)
where k is the subband index and m is the frame index. The subband blended output sout,k(m) can be converted back to the time domain to form the blended output sout(n) or stay in the subband domain to be processed by subband processing modules downstream.
[0093] In various embodiments, the subband-domain blending provides the flexibility of setting the signal blending weight (gSB,k) for each subband separately, thus the method can better handling the variabilities in factors such as the effective bandwidth of in-canal voice and the spectral power distributions of voice and noise. Due to the refined frequency resolution, SNR-based control mechanism can be effective in the subband domain and provides the desired robustness against variabilities in diverse factors such as gain settings in audio chain, locations of microphones, and loudness of user's voice.
[0094] The subband signal blending weights can be adjusted based on the differential between the SNRs in internal and external microphones as:
[0095]
where SNRex,k(m) and SNRin,k(m) are the running subband SNRs of the external microphone signal and internal microphone signals, respectively, and are provided from the NT/NR modules 602. βSB is the bias constant that takes positive values and is normally set to 1.0. ρSB is the transition control constant that also takes positive values and is normally set to a value between 0.5 and 4.0. When β=1.0, the subband signal blending weight computed from Eq. (35) would favor the signal with higher SNR in the corresponding subband. Because the two signals are spectrally aligned, this decision would allow selecting the microphone with lower noise floor within the effective bandwidth of in-canal voice. Outside this bandwidth, it would bias toward external microphone signal within the natural voice bandwidth or split between the two when there is no voice in the subband. Setting βSB to a number larger or smaller than 1.0 would bias the decision toward an external or an internal microphone, respectively. The impact of βSB is proportional to its logarithmic scale. ρSB controls the transition between the microphones. Larger ρSB leads to a sharper transition while smaller ρSB leads to a softer transition.
where SNRex,k(m) and SNRin,k(m) are the running subband SNRs of the external microphone signal and internal microphone signals, respectively, and are provided from the NT/NR modules 602. βSB is the bias constant that takes positive values and is normally set to 1.0. ρSB is the transition control constant that also takes positive values and is normally set to a value between 0.5 and 4.0. When β=1.0, the subband signal blending weight computed from Eq. (35) would favor the signal with higher SNR in the corresponding subband. Because the two signals are spectrally aligned, this decision would allow selecting the microphone with lower noise floor within the effective bandwidth of in-canal voice. Outside this bandwidth, it would bias toward external microphone signal within the natural voice bandwidth or split between the two when there is no voice in the subband. Setting βSB to a number larger or smaller than 1.0 would bias the decision toward an external or an internal microphone, respectively. The impact of βSB is proportional to its logarithmic scale. ρSB controls the transition between the microphones. Larger ρSB leads to a sharper transition while smaller ρSB leads to a softer transition.
[0096] The decision in Eq. (35) can be temporally smoothed for better voice quality. Alternatively, the subband SNRs used in Eq. (35) can be temporally smoothed to achieve similar effect. When the subband SNRs for both internal and external microphones signals are low, the smoothing process should slow down for more consistent noise floor.
[0097] The decision in Eq. (35) is made in each subband independently. Cross-band decision can be added for better robustness. For example, the subbands with relatively lower SNR than other subbands can be biased toward the subband signal with lower power for better noise reduction.
[0098] The SNR-based decision for gSB,k(m) is largely independent of the gain settings in the audio chain. Although it is possible to directly or indirectly incorporate the noise level estimates into the decision process for enhanced robustness against the volatility in SNR estimates, the robustness against other types of variabilities can be reduced as a result.
Example Alternative Usages
[0099] Embodiments of the present technology are not limited to devices having a single internal microphone and a single external microphone. For example, when there are multiple external microphones, spatial filtering algorithms can be applied to the external microphone signals first to generate a single external microphone signal with lower noise level while aligning its voice quality to the external microphone with the best voice quality. The resulting external microphone signal may then be processed by the proposed approach to fuse with the internal microphone signal.
[0100] Similarly, if there are two internal microphones, one in each of the user's ear canals, coherence processing may be first applied to the two internal microphone signals to generate a single internal microphone signal with better acoustic isolation, wider effective voice bandwidth, or both. In various embodiments, this single internal signal is then processed using various embodiments of the method and system of the present technology to fuse with the external microphone signal.
[0101] Alternatively, the present technology can be applied to the internal-external microphone pairs at the user's left and right ears separately, for example. Because the outputs would preserve the spectral amplitudes and phases of the voice at the corresponding external microphones, they can be processed by suitable processing modules downstream to further improve the voice quality. The present technology may also be used for other internal-external microphone configurations.
[0102] FIG. 7 is flow chart diagram showing a method 700 for fusion of microphone signals, according to an example embodiment. The method 700 may be implemented using DSP 112. The example method 700 commences in block 702 with receiving a first signal and a second signal. The first signal represents at least one sound captured by an external microphone and includes at least a voice component. The second signal represents at least one sound captured by an internal microphone located inside an ear canal of a user, and includes at least the voice component modified by at least a human tissue. In place, the internal microphone may be sealed for providing isolation from acoustic signals coming outside the ear canal, or it may be partially sealed depending on the user and the user's placement of the internal microphone in the ear canal.
[0103] In block 704, the method 700 allows processing the first signal to obtain first noise estimates. In block 706 (shown dashed as being optional for some embodiments), the method 700 processes the second signal to obtain second noise estimates. In block 708, the method 700 aligns the second signal to the first signal. In block 710, the method 700 includes blending, based at least on the first noise estimates (and optionally also based on the second noise estimates), the first signal and the aligned second signal to generate an enhanced voice signal.
[0104] FIG. 8 illustrates an exemplary computer system 800 that may be used to implement some embodiments of the present invention. The computer system 800 of FIG. 8 may be implemented in the contexts of the likes of computing systems, networks, servers, or combinations thereof. The computer system 800 of FIG. 8 includes one or more processor units 810 and main memory 820. Main memory 820 stores, in part, instructions and data for execution by processor units 810. Main memory 820 stores the executable code when in operation, in this example. The computer system 800 of FIG. 8 further includes a mass data storage 830, portable storage device 840, output devices 850, user input devices 860, a graphics display system 870, and peripheral devices 880.
[0105] The components shown in FIG. 8 are depicted as being connected via a single bus 890. The components may be connected through one or more data transport means.
[0106] Processor unit 810 and main memory 820 is connected via a local microprocessor bus, and the mass data storage 830, peripheral device(s) 880, portable storage device 840, and graphics display system 870 are connected via one or more input/output (I/O) buses.
[0107] Mass data storage 830, which can be implemented with a magnetic disk drive, solid state drive, or an optical disk drive, is a non-volatile storage device for storing data and instructions for use by processor unit 810. Mass data storage 830 stores the system software for implementing embodiments of the present disclosure for purposes of loading that software into main memory 820.
[0108] Portable storage device 840 operates in conjunction with a portable non-volatile storage medium, such as a flash drive, floppy disk, compact disk, digital video disc, or Universal Serial Bus (USB) storage device, to input and output data and code to and from the computer system 800 of FIG. 8 . The system software for implementing embodiments of the present disclosure is stored on such a portable medium and input to the computer system 800 via the portable storage device 840.
[0109] User input devices 860 can provide a portion of a user interface. User input devices 860 may include one or more microphones, an alphanumeric keypad, such as a keyboard, for inputting alphanumeric and other information, or a pointing device, such as a mouse, a trackball, stylus, or cursor direction keys. User input devices 860 can also include a touchscreen. Additionally, the computer system 800 as shown in FIG. 8 includes output devices 850. Suitable output devices 850 include loudspeakers, printers, network interfaces, and monitors.
[0110] Graphics display system 870 include a liquid crystal display (LCD) or other suitable display device. Graphics display system 870 is configurable to receive textual and graphical information and processes the information for output to the display device.
[0111] Peripheral devices 880 may include any type of computer support device to add additional functionality to the computer system.
[0112] The components provided in the computer system 800 of FIG. 8 are those typically found in computer systems that may be suitable for use with embodiments of the present disclosure and are intended to represent a broad category of such computer components that are well known in the art. Thus, the computer system 800 of FIG. 8 can be a personal computer (PC), hand held computer system, telephone, mobile computer system, workstation, tablet, phablet, mobile phone, server, minicomputer, mainframe computer, wearable, or any other computer system. The computer may also include different bus configurations, networked platforms, multi-processor platforms, and the like. Various operating systems may be used including UNIX, LINUX, WINDOWS, MAC OS, PALM OS, QNX ANDROID, IOS, CHROME, TIZEN and other suitable operating systems.
[0113] The processing for various embodiments may be implemented in software that is cloud-based. In some embodiments, the computer system 800 is implemented as a cloud-based computing environment, such as a virtual machine operating within a computing cloud. In other embodiments, the computer system 800 may itself include a cloud-based computing environment, where the functionalities of the computer system 800 are executed in a distributed fashion. Thus, the computer system 800, when configured as a computing cloud, may include pluralities of computing devices in various forms, as will be described in greater detail below.
[0114] In general, a cloud-based computing environment is a resource that typically combines the computational power of a large grouping of processors (such as within web servers) and/or that combines the storage capacity of a large grouping of computer memories or storage devices. Systems that provide cloud-based resources may be utilized exclusively by their owners or such systems may be accessible to outside users who deploy applications within the computing infrastructure to obtain the benefit of large computational or storage resources.
[0115] The cloud may be formed, for example, by a network of web servers that comprise a plurality of computing devices, such as the computer system 800, with each server (or at least a plurality thereof) providing processor and/or storage resources. These servers may manage workloads provided by multiple users (e.g., cloud resource customers or other users). Typically, each user places workload demands upon the cloud that vary in real-time, sometimes dramatically. The nature and extent of these variations typically depends on the type of business associated with the user.
[0116] The present technology is described above with reference to example embodiments. Therefore, other variations upon the example embodiments are intended to be covered by the present disclosure.
Claims
What is claimed is:
1. A method for fusion of microphone signals, the method comprising:
receiving a first signal including at least a voice component and a second signal including at least the voice component modified by at least a human tissue;
processing the first signal to obtain first noise estimates;
aligning the voice component in the second signal spectrally with the voice component in the first signal; and
blending, based at least on the first noise estimates, the first signal and the aligned voice component in the second signal to generate an enhanced voice signal, the blending including:
assigning, based at least on the first noise estimates, a first weight to the first signal and a second weight to the second signal; and
mixing the first signal and the second signal according to the first weight and the second weight.
2. The method of claim 1, wherein the second signal represents at least one sound captured by an internal microphone located inside an ear canal.
3. The method of claim 2, wherein the internal microphone is at least partially sealed for isolation from acoustic signals external to the ear canal.
4. The method of claim 1, wherein the first signal represents at least one sound captured by an external microphone located outside an ear canal.
5. The method of claim 1, further comprising processing the second signal to obtain second noise estimates.
6. The method of claim 5, wherein the assigning, of the first weight to the first signal and the second weight to the second signal, is based at least on the first noise estimates and the second noise estimates.
7. The method of claim 6, wherein the first weight receives a larger value than the second weight when a signal-to-noise ratio (SNR) of the first signal is larger than a SNR of the second signal, and wherein the second weight receives a larger value than the first weight when the SNR of the first signal is smaller than the SNR of the second signal, the difference between the first weight and the second weight corresponding to the difference between the SNR of the first signal and the SNR of the second signal.
8. The method of claim 5, further comprising:
prior to the aligning, performing, based on the first noise estimates, noise reduction of the first signal; and
prior to the aligning, performing, based on the second noise estimates, noise reduction of the second signal.
9. The method of claim 5, further comprising:
after the aligning, performing noise reduction of the first signal based on the first noise estimates; and
after the aligning, performing noise reduction of the second signal based on the second noise estimates.
10. The method of claim 1, wherein at least one of the aligning and blending is performed for subbands in the frequency domain.
11. The method of claim 1, wherein the processing, aligning, and blending are performed for subbands in the frequency domain.
12. The method of claim 1, further comprising performing noise reduction of the first signal.
13. The method of claim 1, further comprising performing noise reduction of the second signal.
14. The method of claim 1, wherein the aligning includes applying a spectral alignment filter to the second signal.
15. The method of claim 14, wherein the spectral alignment filter includes an empirically derived filter.
16. The method of claim 14, wherein the spectral alignment filter includes an adaptive filter calculated based on cross-correlation of the first signal and the second signal and auto-correlation of the second signal.
17. A system for fusion of microphone signals, the system comprising:
a digital signal processor, configured to:
receive a first signal including at least a voice component and a second signal including at least the voice component modified by at least a human tissue;
process the first signal to obtain first noise estimates;
align the voice component in the second signal spectrally with the voice component in the first signal; and
blend, based at least on the first noise estimates, the first signal and the aligned voice component in the second signal to generate an enhanced voice signal, including:
assigning, based at least on the first noise estimates, a first weight to the first signal and a second weight to the second signal; and
mixing the first signal and the second signal according to the first weight and the second weight.
18. The system of claim 17, further comprising:
an internal microphone located inside an ear canal and sealed to be isolated from acoustic signals external to the ear canal, the second signal representing at least one sound captured by the internal microphone; and
an external microphone located outside the ear canal, the first signal representing at least one sound captured by the external microphone.
19. The system of claim 17, wherein the digital signal processor is further configured to process the second signal to obtain second noise estimates.
20. The system of claim 19, wherein the assigning, of the first weight to the first signal and the second weight to the second signal, is based at least on the first noise estimates and the second noise estimates.
21. The system of claim 20, wherein the first weight receives a larger value than the second weight when a signal-to-noise ratio (SNR) of the first signal is larger than a SNR of the second signal, and wherein the second weight receives a larger value than the first weight when the SNR of the first signal is smaller than the SNR of the second signal, the difference between the first weight and second weight corresponding to the difference between the SNR of the first signal and the SNR of the second signal.
22. The system of claim 19, wherein the digital signal processor is further configured to:
perform, prior to the aligning and based on the first noise estimates, noise reduction of the first signal; and
perform, prior to the aligning and based on the second noise estimates, noise reduction of the second signal.
23. The system of claim 19, wherein the digital signal processor is further configured to:
perform, after the aligning and based on the first noise estimates, noise reduction of the first signal; and
perform, after the aligning and based on the second noise estimates, noise reduction of the second signal.
24. The system of claim 17, wherein the processing, aligning, and blending are performed for subbands in the frequency domain.
25. The system of claim 17, wherein the digital signal processor is further configured to perform noise reduction of the first signal and the second signal.
26. The system of claim 17, wherein the aligning includes applying a spectral alignment filter to the second signal.
27. The system of claim 26, wherein the spectral alignment filter includes one of an empirically derived filter and an adaptive filter, the adaptive filter being calculated based on cross-correlation of the first signal and the second signal and auto-correlation of the second signal.
28. A non-transitory computer-readable storage medium having embodied thereon instructions, which when executed by at least one processor, perform steps of a method, the method comprising:
receiving a first signal including at least a voice component and a second signal including at least the voice component modified by at least a human tissue;
processing the first signal to obtain first noise estimates;
aligning the voice component in the second signal spectrally to the voice component in the first signal; and
blending, based at least on the first noise estimates, the first signal and the aligned voice component in the second signal to generate an enhanced voice signal, the blending including:
assigning, based at least on the first noise estimates, a first weight to the first signal and a second weight to the second signal; and
mixing the first signal and the second signal according to the first weight and the second weight.
Patent Citations (258)
| Patent | Date | Inventor | Cited By |
|---|---|---|---|
| US2535063(A) | 1950-12-01 | Halstead | Applicant |
| US3995113(A) | 1976-11-01 | Tani | Applicant |
| US4150262(A) | 1979-04-01 | Ono | Applicant |
| US4455675(A) | 1984-06-01 | Bose et al. | Applicant |
| US4516428(A) | 1985-05-01 | Konomi | Applicant |
| US4520238(A) | 1985-05-01 | Ikeda | Applicant |
| US4588867(A) | 1986-05-01 | Konomi | Applicant |
| US4596903(A) | 1986-06-01 | Yoshizawa | Applicant |
| US4644581(A) | 1987-02-01 | Sapiejewski | Applicant |
| US4652702(A) | 1987-03-01 | Yoshii | Applicant |
| US4696045(A) | 1987-09-01 | Rosenthal | Applicant |
| US4975967(A) | 1990-12-01 | Rasmussen | Applicant |
| US5208867(A) | 1993-05-01 | Stites, III | Applicant |
| US5251263(A) | 1993-10-01 | Andrea et al. | Applicant |
| US5282253(A) | 1994-01-01 | Konomi | Applicant |
| US5295193(A) | 1994-03-01 | Ono | Applicant |
| US5305387(A) | 1994-04-01 | Sapiejewski | Applicant |
| US5319717(A) | 1994-06-01 | Holesha | Applicant |
| US5327506(A) | 1994-07-01 | Stites, III | Applicant |
| USD360691(S) | 1995-07-01 | Mostardo | Applicant |
| USD360948(S) | 1995-08-01 | Mostardo | Applicant |
| USD360949(S) | 1995-08-01 | Mostardo | Applicant |
| US5734621(A) | 1998-03-01 | Ito | Applicant |
| USD414493(S) | 1999-09-01 | Jiann-Yeong | Applicant |
| US5960093(A) | 1999-09-01 | Miller | Applicant |
| US5983073(A) | 1999-11-01 | Ditzik | Applicant |
| US6044279(A) | 2000-03-01 | Hokao et al. | Applicant |
| US6094492(A) | 2000-07-01 | Boesen | Applicant |
| US6118878(A) | 2000-09-01 | Jones | Applicant |
| US6122388(A) | 2000-09-01 | Feldman | Applicant |
| US6130953(A) | 2000-10-01 | Wilton et al. | Applicant |
| US6184652(B1) | 2001-02-01 | Yang | Applicant |
| US6211649(B1) | 2001-04-01 | Matsuda | Applicant |
| US6255800(B1) | 2001-07-01 | Bork | Applicant |
| US6362610(B1) | 2002-03-01 | Yang | Applicant |
| US6373942(B1) | 2002-04-01 | Braund | Applicant |
| US6408081(B1) | 2002-06-01 | Boesen | Applicant |
| US6462668(B1) | 2002-10-01 | Foseide | Applicant |
| US6567524(B1) | 2003-05-01 | Svean et al. | Applicant |
| US6661901(B1) | 2003-12-01 | Svean et al. | Applicant |
| US6683965(B1) | 2004-01-01 | Sapiejewski | Applicant |
| US6694180(B1) | 2004-02-01 | Boesen | Applicant |
| US6717537(B1) | 2004-04-01 | Fang et al. | Applicant |
| US6738485(B1) | 2004-05-01 | Boesen | Applicant |
| US6751326(B2) | 2004-06-01 | Nepomuceno | Applicant |
| US6754358(B1) | 2004-06-01 | Boesen et al. | Applicant |
| US6754359(B1) | 2004-06-01 | Svean et al. | Applicant |
| US6757395(B1) | 2004-06-01 | Fang et al. | Applicant |
| US6801632(B2) | 2004-10-01 | Olson | Applicant |
| US6879698(B2) | 2005-04-01 | Boesen | Applicant |
| US6920229(B2) | 2005-07-01 | Boesen | Applicant |
| US6931292(B1) | 2005-08-01 | Brumitt et al. | Applicant |
| US6937738(B2) | 2005-08-01 | Armstrong et al. | Applicant |
| US7024010(B2) | 2006-04-01 | Saunders et al. | Applicant |
| US7039195(B1) | 2006-05-01 | Svean et al. | Applicant |
| US7103188(B1) | 2006-09-01 | Jones | Applicant |
| US7136500(B2) | 2006-11-01 | Collins | Applicant |
| US7203331(B2) | 2007-04-01 | Boesen | Applicant |
| US7209569(B2) | 2007-04-01 | Boesen | Applicant |
| US7215790(B2) | 2007-05-01 | Boesen et al. | Applicant |
| US7289636(B2) | 2007-10-01 | Saunders et al. | Applicant |
| US7302074(B2) | 2007-11-01 | Wagner et al. | Applicant |
| USD573588(S) | 2008-07-01 | Warren et al. | Applicant |
| US7406179(B2) | 2008-07-01 | Ryan | Applicant |
| US7433481(B2) | 2008-10-01 | Armstrong et al. | Applicant |
| US7477754(B2) | 2009-01-01 | Rasmussen et al. | Applicant |
| US7477756(B2) | 2009-01-01 | Wickstrom et al. | Applicant |
| US7502484(B2) | 2009-03-01 | Ngia et al. | Applicant |
| US7590254(B2) | 2009-09-01 | Olsen | Applicant |
| US7680292(B2) | 2010-03-01 | Warren et al. | Applicant |
| US7747032(B2) | 2010-06-01 | Zei et al. | Applicant |
| US7773759(B2) | 2010-08-01 | Alves et al. | Applicant |
| US7869610(B2) | 2011-01-01 | Jayanth et al. | Applicant |
| US7889881(B2) | 2011-02-01 | Ostrowski | Applicant |
| US7899194(B2) | 2011-03-01 | Boesen | Applicant |
| US7965834(B2) | 2011-06-01 | Alves et al. | Applicant |
| US7983433(B2) | 2011-07-01 | Nemirovski | Applicant |
| US8005249(B2) | 2011-08-01 | Wirola et al. | Applicant |
| US8019107(B2) | 2011-09-01 | Ngia et al. | Applicant |
| US8027481(B2) | 2011-09-01 | Beard | Applicant |
| US8045724(B2) | 2011-10-01 | Sibbald | Applicant |
| US8077873(B2) | 2011-12-01 | Shridhar et al. | Applicant |
| US8081780(B2) | 2011-12-01 | Goldstein et al. | Applicant |
| US8103029(B2) | 2012-01-01 | Ngia et al. | Applicant |
| US8111853(B2) | 2012-02-01 | Isvan | Applicant |
| US8116489(B2) | 2012-02-01 | Mejia et al. | Applicant |
| US8116502(B2) | 2012-02-01 | Saggio, Jr. et al. | Applicant |
| US8135140(B2) | 2012-03-01 | Shridhar et al. | Applicant |
| US8180067(B2) | 2012-05-01 | Soulodre | Applicant |
| US8189799(B2) | 2012-05-01 | Shridhar et al. | Applicant |
| US8199924(B2) | 2012-06-01 | Wertz et al. | Applicant |
| US8213643(B2) | 2012-07-01 | Hemer | Applicant |
| US8213645(B2) | 2012-07-01 | Rye et al. | Applicant |
| US8229125(B2) | 2012-07-01 | Short | Applicant |
| US8229740(B2) | 2012-07-01 | Nordholm et al. | Applicant |
| US8238567(B2) | 2012-08-01 | Burge et al. | Applicant |
| US8249287(B2) | 2012-08-01 | Silvestri et al. | Applicant |
| US8254591(B2) | 2012-08-01 | Goldstein et al. | Applicant |
| US8270626(B2) | 2012-09-01 | Shridhar et al. | Applicant |
| US8285344(B2) | 2012-10-01 | Kahn et al. | Applicant |
| US8295503(B2) | 2012-10-01 | Sung et al. | Applicant |
| US8311253(B2) | 2012-11-01 | Silvestri et al. | Applicant |
| US8315404(B2) | 2012-11-01 | Shridhar et al. | Applicant |
| US8325963(B2) | 2012-12-01 | Kimura | Applicant |
| US8331604(B2) | 2012-12-01 | Saito et al. | Applicant |
| US8363823(B1) | 2013-01-01 | Santos | Applicant |
| US8376967(B2) | 2013-02-01 | Mersky | Applicant |
| US8385560(B2) | 2013-02-01 | Solbeck et al. | Applicant |
| US8401200(B2) | 2013-03-01 | Tiscareno et al. | Applicant |
| US8401215(B2) | 2013-03-01 | Warren et al. | Applicant |
| US8416979(B2) | 2013-04-01 | Takai | Applicant |
| US8462956(B2) | 2013-06-01 | Goldstein et al. | Applicant |
| US8483418(B2) | 2013-07-01 | Platz et al. | Applicant |
| US8488831(B2) | 2013-07-01 | Saggio, Jr. et al. | Applicant |
| US8494201(B2) | 2013-07-01 | Anderson | Applicant |
| US8498428(B2) | 2013-07-01 | Schreuder et al. | Applicant |
| US8503689(B2) | 2013-08-01 | Schreuder et al. | Applicant |
| US8503704(B2) | 2013-08-01 | Francart et al. | Applicant |
| US8509465(B2) | 2013-08-01 | Theverapperuma | Applicant |
| US8526646(B2) | 2013-09-01 | Boesen | Applicant |
| US8553899(B2) | 2013-10-01 | Salvetti et al. | Applicant |
| US8553923(B2) | 2013-10-01 | Tiscareno et al. | Applicant |
| US8571227(B2) | 2013-10-01 | Donaldson et al. | Applicant |
| US8594353(B2) | 2013-11-01 | Anderson | Applicant |
| US8620650(B2) | 2013-12-01 | Walters et al. | Applicant |
| US8634576(B2) | 2014-01-01 | Salvetti et al. | Applicant |
| US8655003(B2) | 2014-02-01 | Duisters et al. | Applicant |
| US8666102(B2) | 2014-03-01 | Bruckhoff et al. | Applicant |
| US8681999(B2) | 2014-03-01 | Theverapperuma et al. | Applicant |
| US8682001(B2) | 2014-03-01 | Annunziato et al. | Applicant |
| US8705787(B2) | 2014-04-01 | Larsen et al. | Applicant |
| US8837746(B2) | 2014-09-01 | Burnett | Applicant |
| US8942976(B2) | 2015-01-01 | Li et al. | Applicant |
| US8983083(B2) | 2015-03-01 | Tiscareno et al. | Applicant |
| US9014382(B2) | 2015-04-01 | Van De Par et al. | Applicant |
| US9025415(B2) | 2015-05-01 | Derkx | Applicant |
| US9042588(B2) | 2015-05-01 | Aase | Applicant |
| US9047855(B2) | 2015-06-01 | Bakalos | Applicant |
| US9100756(B2) | 2015-08-01 | Dusan et al. | Applicant |
| US9123320(B2) | 2015-09-01 | Carreras et al. | Applicant |
| US9154868(B2) | 2015-10-01 | Narayan et al. | Applicant |
| US9167337(B2) | 2015-10-01 | Shin | Applicant |
| US9208769(B2) | 2015-12-01 | Azmi | Applicant |
| US9226068(B2) | 2015-12-01 | Hendrix et al. | Applicant |
| US9264823(B2) | 2016-02-01 | Bajic et al. | Applicant |
| US2001/0011026(A1) | 2001-08-01 | Nishijima | Applicant |
| US2001/0021659(A1) | 2001-09-01 | Okamura | Applicant |
| US2001/0049262(A1) | 2001-12-01 | Lehtonen | Applicant |
| US2002/0016188(A1) | 2002-02-01 | Kashiwamura | Applicant |
| US2002/0021800(A1) | 2002-02-01 | Bodley et al. | Applicant |
| US2002/0038394(A1) | 2002-03-01 | Liang et al. | Applicant |
| US2002/0054684(A1) | 2002-05-01 | Menzl | Applicant |
| US2002/0056114(A1) | 2002-05-01 | Fillebrown et al. | Applicant |
| US2002/0067825(A1) | 2002-06-01 | Baranowski et al. | Applicant |
| US2002/0098877(A1) | 2002-07-01 | Glezerman | Applicant |
| US2002/0159023(A1) | 2002-10-01 | Swab | Applicant |
| US2002/0176330(A1) | 2002-11-01 | Ramonowski et al. | Applicant |
| US2002/0183089(A1) | 2002-12-01 | Heller et al. | Applicant |
| US2003/0002704(A1) | 2003-01-01 | Pronk | Applicant |
| US2003/0013411(A1) | 2003-01-01 | Uchiyama | Applicant |
| US2003/0017805(A1) | 2003-01-01 | Yeung et al. | Applicant |
| US2003/0058808(A1) | 2003-03-01 | Eaton et al. | Applicant |
| US2003/0207703(A1) | 2003-11-01 | Liou et al. | Applicant |
| US2003/0223592(A1) | 2003-12-01 | Deruginsky et al. | Applicant |
| US2005/0027522(A1) | 2005-02-01 | Yamamoto et al. | Applicant |
| US2006/0034472(A1) | 2006-02-01 | Bazarjani et al. | Applicant |
| US2006/0153155(A1) | 2006-07-01 | Jacobsen et al. | Applicant |
| US2006/0227990(A1) | 2006-10-01 | Kirchhoefer | Applicant |
| US2007/0104340(A1) | 2007-05-01 | Miller et al. | Applicant |
| US2007/0147635(A1) | 2007-06-01 | Dijkstra et al. | Applicant |
| US2008/0101640(A1) | 2008-05-01 | Ballad et al. | Applicant |
| US2008/0181419(A1) | 2008-07-01 | Goldstein et al. | Applicant |
| US2009/0041269(A1) | 2009-02-01 | Hemer | Applicant |
| US2009/0182913(A1) | 2009-07-01 | Rosenblatt et al. | Applicant |
| US2009/0207703(A1) | 2009-08-01 | Matsumoto et al. | Applicant |
| US2009/0214068(A1) | 2009-08-01 | Wickstrom | Applicant |
| US2010/0022280(A1) | 2010-01-01 | Schrage | Examiner |
| US2010/0081487(A1) | 2010-04-01 | Chen et al. | Applicant |
| US2010/0183167(A1) | 2010-07-01 | Phelps et al. | Applicant |
| US2012/0197638(A1) | 2012-08-01 | Li et al. | Applicant |
| US2012/0321103(A1) | 2012-12-01 | Smailagic et al. | Applicant |
| US2013/0024194(A1) | 2013-01-01 | Zhao et al. | Applicant |
| US2013/0051580(A1) | 2013-02-01 | Miller | Applicant |
| US2013/0058495(A1) | 2013-03-01 | Furst et al. | Applicant |
| US2013/0070935(A1) | 2013-03-01 | Hui et al. | Applicant |
| US2013/0272564(A1) | 2013-10-01 | Miller | Applicant |
| US2013/0287219(A1) | 2013-10-01 | Hendrix et al. | Applicant |
| US2013/0315415(A1) | 2013-11-01 | Shin | Applicant |
| US2014/0044275(A1) | 2014-02-01 | Goldstein et al. | Applicant |
| US2014/0086425(A1) | 2014-03-01 | Jensen et al. | Applicant |
| US2014/0169579(A1) | 2014-06-01 | Azmi | Applicant |
| US2014/0270231(A1) | 2014-09-01 | Dusan et al. | Applicant |
| US2014/0273851(A1) | 2014-09-01 | Donaldson et al. | Applicant |
| US2014/0355787(A1) | 2014-12-01 | Jiles et al. | Applicant |
| US2015/0043741(A1) | 2015-02-01 | Shin | Applicant |
| US2015/0055810(A1) | 2015-02-01 | Shin | Applicant |
| US2015/0078574(A1) | 2015-03-01 | Shin | Applicant |
| US2015/0110280(A1) | 2015-04-01 | Wardle | Applicant |
| US2015/0161981(A1) | 2015-06-01 | Kwatra | Applicant |
| US2015/0172814(A1) | 2015-06-01 | Usher | Examiner |
| US2015/0243271(A1) | 2015-08-01 | Goldstein | Applicant |
| US2015/0245129(A1) | 2015-08-01 | Dusan et al. | Applicant |
| US2015/0264472(A1) | 2015-09-01 | Aase | Applicant |
| US2015/0304770(A1) | 2015-10-01 | Watson et al. | Applicant |
| US2015/0310846(A1) | 2015-10-01 | Andersen et al. | Applicant |
| US2015/0325229(A1) | 2015-11-01 | Carreras et al. | Applicant |
| US2015/0325251(A1) | 2015-11-01 | Dusan et al. | Applicant |
| US2015/0382094(A1) | 2015-12-01 | Grinker et al. | Applicant |
| US2016/0021480(A1) | 2016-01-01 | Johnson et al. | Applicant |
| US2016/0029345(A1) | 2016-01-01 | Sebeni et al. | Applicant |
| US2016/0042666(A1) | 2016-02-01 | Hughes | Applicant |
| US2016/0044151(A1) | 2016-02-01 | Shoemaker et al. | Applicant |
| US2016/0044398(A1) | 2016-02-01 | Siahaan et al. | Applicant |
| US2016/0044424(A1) | 2016-02-01 | Dave et al. | Applicant |
| CN204119490(U) | 2015-01-01 | Applicant | |
| CN204145685(U) | 2015-02-01 | Applicant | |
| CN204168483(U) | 2015-02-01 | Applicant | |
| CN204669605(U) | 2015-09-01 | Applicant | |
| CN204681587(U) | 2015-09-01 | Applicant | |
| CN204681593(U) | 2015-09-01 | Applicant | |
| CNZL2015203769650 | 2015-09-01 | Applicant | |
| CNZL2015204747042 | 2015-09-01 | Applicant | |
| CNZL2015204903074 | 2015-09-01 | Applicant | |
| DE915826 | 1949-07-01 | Applicant | |
| DE3723275 | 1988-03-01 | Applicant | |
| DE102009051713 | 2011-05-01 | Applicant | |
| DE102011003470 | 2012-08-01 | Applicant | |
| EP124870 | 1984-11-01 | Applicant | |
| EP500985 | 1992-09-01 | Applicant | |
| EP684750 | 1995-11-01 | Applicant | |
| EP806909 | 1997-11-01 | Applicant | |
| EP1299988 | 2003-04-01 | Applicant | |
| EP1509065 | 2006-04-01 | Applicant | |
| EP2434780 | 2012-03-01 | Applicant | |
| JPS5888996 | 1983-05-01 | Applicant | |
| JPS60103798 | 1985-06-01 | Applicant | |
| JP2007150743(A) | 2007-06-01 | Applicant | |
| KR20110058769(A) | 2011-06-01 | Applicant | |
| WO8303733 | 1983-10-01 | Applicant | |
| WO9407342 | 1994-03-01 | Applicant | |
| WO9623443 | 1996-08-01 | Applicant | |
| WO25551 | 2000-05-01 | Applicant | |
| WO217835 | 2002-03-01 | Applicant | |
| WO217836 | 2002-03-01 | Applicant | |
| WO217837 | 2002-03-01 | Applicant | |
| WO217838 | 2002-03-01 | Applicant | |
| WO217839 | 2002-03-01 | Applicant | |
| WO3073790 | 2003-09-01 | Applicant | |
| WO2006114767(A1) | 2006-11-01 | Applicant | |
| WO2007073818 | 2007-07-01 | Applicant | |
| WO2007082579 | 2007-07-01 | Applicant | |
| WO2007147416 | 2007-12-01 | Applicant | |
| WO2008128173 | 2008-10-01 | Applicant | |
| WO2009012491 | 2009-01-01 | Applicant | |
| WO2009023784 | 2009-02-01 | Applicant | |
| WO2011051469 | 2011-05-01 | Applicant | |
| WO2011061483 | 2011-05-01 | Applicant | |
| WO2013033001(A1) | 2013-03-01 | Applicant |
Non-Patent Literature (18)
- Ephraim, Y. et al., “Speech enhancement using a minimum mean-square error short-time spectral amplitude estimator,” IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. ASSP-32, No. 6, Dec. 1984, pp. 1109-1121.Applicant
- Sun et al., “Robust Noise Estimation Using Minimum Correction with Harmonicity Control.” Conference: Interspeech 2010, 11th Annual Conference of the International Speech Communication Association, Makuhari, Chiba, Japan, Sep. 26-30, 2010. page 1085-1088.Applicant
- Lomas, “Apple Patents Earbuds With Noise-Canceling Sensor Smarts,” Aug. 27, 2015. [retrieved on Sep. 16, 2015]. TechCrunch. Retrieved from the Internet: <URL: http://techcrunch.com/2015/08/27/apple-wireless-earbuds-at-last/>. 2 pages.Applicant
- Smith, Gina, “New Apple Patent Applications: The Sound of Hearables to Come,” aNewDomain, Feb. 12, 2016, accessed Mar. 2, 2016 at URL: <http://anewdomain.net/2016/02/12/new-apple-patent-applications-glimpse-hearables-come/.Applicant
- Qutub, Sarmad et al., “Acoustic Apparatus with Dual MEMS Devices,” U.S. Appl. No. 14/872,887, filed Oct. 1, 2015.Applicant
- Office Action dated Feb. 4, 2016 in U.S. Appl. No. 14/318,436, filed Jun. 27, 2014.Applicant
- Office Action dated Jan. 22, 2016 in U.S. Appl. No. 14/774,666, filed Sep. 10, 2015.Applicant
- Non-Final Office Action, Sep. 23, 2015, U.S. Appl. No. 13/224,068, filed Sep. 1, 2011.Applicant
- Non-Final Office Action, Mar. 10, 2004, U.S. Appl. No. 10/138,929, filed May 3, 2002.Applicant
- Final Office Action, Jan. 12, 2005, U.S. Appl. No. 10/138,929, filed May 3, 2002.Applicant
- Non-Final Office Action, Jan. 12, 2006, U.S. Appl. No. 10/138,929, filed May 3, 2002.Applicant
- Notice of Allowance, Sep. 27, 2012, U.S. Appl. No. 13/568,989, filed Aug. 7, 2012.Applicant
- Hegde, Nagaraj, “Seamlessly Interfacing MEMS Microphones with BlackfinTM Processors”, EE350 Analog Devices, Rev. 1, Aug. 2010, pp. 1-10.Applicant
- Korean Office Action regarding Application No. 10-2014-7008553, dated May 21, 2015.Applicant
- Written Opinion of the International Searching Authority and International Search Report mailed Jan. 21, 2013 in Patent Cooperation Treaty Application No. PCT/US2012/052478, filed Aug. 27, 2012.Applicant
- Langberg, Mike, “Bluelooth Sharpens Its Connections,” Chicago Tribune, Apr. 29, 2002, Business Section, p. 3, accessed Mar. 11, 2016 at URL: <http://articles.chicagotribune.com/2002-04-29/business/0204290116—1—bluetooth-enabled-bluetooth-headset-bluetooth-devices>.Applicant
- Duplan Corporaton vs. Deering Milliken decision, 197 USPQ 342.Applicant
- Combined Bluetooth Headset and USB Dongle, Advance Information, RTX Telecom A/S, vol. 1, Apr. 6, 2002.Applicant