Earphones Having Configurable Microphone Arrays

[0001] This application claims the benefit of the filing date of U.S. provisional application no. 61/708,826, filed on Oct. 02, 2012, the teachings of which are incorporated herein by reference in their entirety.

[0002] 1. Field

[0003] The present disclosure relates to audio products and, more specifically but not exclusively, to earphones and headphones having built-in microphones, such as those used for telephony applications.

[0004] 2. Description of the Related Art

[0005] This section introduces aspects that may help facilitate a better understanding of the disclosure. Accordingly, the statements of this section are to be read in this light and are not to be understood as admissions about what is prior art or what is not prior art.

[0006] Cellular smartphones provide a wide variety of audio functionality to users, including telephony, voice-memo recording, audio-video recording, and a voice interface to automated speech-recognition engines used in dictation and automated voice assistants. Almost all smartphones can be used with earphones, also known as in-ear headphones, earbuds, or headsets.

[0007] Many earphone designs to date contain a single omnidirectional microphone embedded in one of the earphone wires. This microphone is usually located in one of the earphone wires that connects to the left or right earphone transducer. For example, in the earphone designs by Apple, Beats, and Bose, to name only three, the microphone is located in the control capsule that provides the user with a set of audio controls. The control capsule on most earphones is located on either the left or right earphone wire and is placed along the wire so that, when the earphones are worn normally, the microphone is in line with the user's mouth. Such configurations limit the transmission quality of the user's speech as well as the accuracy of speech-recognition engines when used with automated voice assistants such as Apple's Siri and Google's Google Now.

[0008] In one embodiment, the present invention is an accessory system for a telephone. The accessory system comprises (a) at least one earphone configured to receive from the telephone incoming audio signals for rendering by the at least one earphone and (b) at least one microphone array comprising a plurality of microphones used to generate outgoing audio signals for (i) processing by a signal processor and (ii) transmission by the telephone.

[0014] Detailed illustrative embodiments of the present invention are disclosed herein. However, specific structural and functional details disclosed herein are merely representative for purposes of describing example embodiments of the present invention. Embodiments of the present invention may be embodied in many alternative forms and should not be construed as limited to only the embodiments set forth herein. Further, the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments of the invention.

[0015] As used herein, the singular forms “a,” “an,” and “the,” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It further will be understood that the terms “comprises,” “comprising,” “has,” “having,” “includes,” and/or “including” specify the presence of stated features, steps, or components, but do not preclude the presence or addition of one or more other features, steps, or components.

[0016] This disclosure is related to an earphone design that incorporates microphone arrays in the earphone's wires or in the earphones themselves or both to enhance the transmission quality of the user's speech for the purpose of improving the quality of telephone calls and for improving the accuracy of speech-recognition engines when used with automated voice assistants such as Apple's Siri. The earphone design described here incorporates a microphone array in one or both of the left and right wires leading to the earphone's loudspeaker transducers. When two microphone arrays are deployed, each array can be used separately, or the two arrays can be mechanically configured to result in a longer and more-effective array for enhanced transmit-speech quality, particularly in noisy environments. The two arrays can also be arranged for use in stereo recording, such as might be used in making a video recording of a musical performance.

[0017] Compared to a single omnidirectional microphone, a microphone array has two or more omnidirectional (or directional) microphone elements arranged in a line, a two-dimensional grid, or a three-dimensional frame or solid. The nominal inter-microphone spacing may be chosen based on the physics of sound propagation such that the mathematical combination of the signals from all microphones produces an acoustic sensitivity that is greater than, and more directional than, the signal from a single omnidirectional element or single directional element. The sensitivity beampattern resulting from the microphone array can be designed to have maximum sensitivity in the direction of the user's mouth while minimizing sound contributions from all other directions, and in this way a microphone array can greatly enhance the quality of a user's speech by increasing the speech-signal-to-noise ratio beyond that provided by a single omnidirectional or directional microphone element.

[0018] The mathematics and engineering of microphone arrays is a mature science. See, e.g., S. L. Gay and J. Benesty, eds., Acoustic Signal Processing for Telecommunications, Kluwer Academic Publishers, Boston, 2000; M. Brandstein and D. Ward, eds., Microphone Arrays—Signal Processing Techniques and Applications, Springer-Verlag, Berlin, 2001; and J. Benesty, J. Chen, Y. Huang, eds., Microphone Array Signal Processing, Springer-Verlag, Berlin, 2008, the teachings of all of which are incorporated by reference herein in their entirety. This disclosure focuses on the mechanical features of an earphone accessory containing one or more microphone arrays that can be configured for different speech transmission and sound recording use scenarios.

[0019] FIGS. 1(A)-1(E) illustrate five different modes of using an exemplary earphone/microphone accessory system 100 according to one embodiment of the disclosure. In this embodiment, both the left and right earphone wires 102l and 102r have embedded along their lengths a capsule containing a line array 104l/r of two or more omnidirectional or directional microphones. In the illustrated implementation, each array contains five omnidirectional microphones. These microphones could be traditional electret capsules, MEMS microphones, or another suitable sound-transducing capsule design. The capsules are generally located in the vicinity of the user's mouth, but need not be.

[0020] For all use modes, the mathematical combination of microphone signals in one or both arrays for the purpose of generating a directional beampattern could take place centrally where one processing unit has access to all microphone signals. Typically, but not necessarily, this central unit would be the smartphone (not shown). Alternatively, the beamforming could be implemented local to the arrays where special-purpose processors (e.g., digital signal processors) are (i) embedded with the array capsules themselves in shared housing or (ii) separate from the array capsules in their own housing and, in either case, not shown in the figures.

[0021] FIGS. 2(A)-2(E) graphically illustrate prototypical sensitivity patterns associated with the five modes of FIGS. 1(A)-1(E), respectively. The sensitivity is greatest in the zero-degree bearing direction, which is the direction towards the user's mouth as shown in FIG. 1. Narrower patterns indicate an acoustic sensitivity that is less sensitive in all directions other than the direction of maximum sensitivity.

[0022] Depending on the particular implementation, the arrays 104l/r and the earbuds 106l/r could similarly or differently communicate with the smartphone via either a wired link or a wireless link If the arrays communicate wirelessly, they may be located anywhere including remotely from the earbuds and any earbud wires, such as, for example, clipped to the user's clothing.

[0023] FIG. 3(A) illustrates the exemplary earphone/microphone accessory system 100 of FIG. 1 configured to a cellphone 300, and FIG. 3(B) shows a simplified block diagram of the configuration of FIG. 3(A). As shown in FIG. 3(B), voice reception layer 302 of cellphone 300 receives and processes an incoming telephone signal from a far-end telephone and sends the resulting incoming audio signals for rendering by the loudspeakers 304 of earbuds 106l/r into audible acoustic sounds for the user to hear. In addition, microphones 306 of microphone arrays 104l/r convert audible acoustic signals into outgoing audio signals for appropriate signal processing by array beamformer/mode detector 308 and then by the voice transmission layer 310 of cellphone 300 for transmission towards the far-end telephone. Depending on the implementation, array beamformer/mode detector 308 might or might not be implemented within cellphone 300. As indicated in both FIGS. 3(A) and 3(B), depending on the particular situation, acoustic signals generated by the earbud loudspeakers can get picked up by the array microphones.

[0024] In the single-sided mode of FIGS. 1(A) and 2(A), only one of the earphones (e.g., 106l) is inserted in the user's ear. The microphone array 104l associated with that earphone 106l is activated, and beamforming is used to maximize the sensitivity pattern of the array in the direction of the user's mouth. The other array 104r is not used in this mode.

[0025] Any of a number of methods could be used to automatically determine which of the two earphones 106l and 106r is inserted in the user's ear, and therefore which of the two arrays to use for speech transmission. For example, the earphones could contain a microphone-like transducer that detects the acoustic coupling from the earphone's loudspeaker. This coupling is greater when the earphone is inserted in the user's ear than when it is not, because the ear canal serves as a closed acoustic resonator. Alternatively, any number of voice-activity detection methods could be employed to determine which of the two arrays, left or right, provides the highest speech activity measure or rating when the user is talking, and that array is then selected for speech transmission. In yet another alternative, accelerometers could be embedded in the earphones and/or earphone wires and/or earphone control box. The direction of gravity as reported by the accelerometers could be used to determine the orientation of the earbuds relative to the wires and arrays, and from that, which earbud and corresponding array are predominately parallel to a normal to Earth's surface (indicating a higher likelihood of being worn compared to the other earbud). Another possibility could be to use the phase delay (or equivalently group delay) from the acoustic and vibration sources (talker's mouth, vibration sensor signal, and/or earbud loudspeaker signal) to triangulate the position of the microphones that make up the array in order to compute the array geometry and orientation. Other techniques are also possible.

[0026] In the two-sided mode of FIGS. 1(B) and 2(B), both earphones 106l and 106r are inserted in the user's ears, and either or both arrays 104l and/or 104r could be selected for the transmission of speech. For example, if one of the arrays is more dominated by environmental noise, such as wind noise, the less noisy array could be automatically selected, with the more noisy array de-activated.

[0027] To determine whether the earphones are in two-sided mode or one-sided mode, the acoustic coupling described in the previous section could be thresholded for both arrays. If the acoustic coupling for both arrays is greater than the specified threshold level, then the earphones can be assumed to be in two-sided mode. If the acoustic coupling for only one array is greater than the specified threshold level, then the earphones can be assumed to be in one-sided mode, with the earphone having the greater acoustic coupling being the inserted earphone and the other earphone being disabled.

[0028] In the enhanced directivity mode of FIGS. 1(C) and 2(C), the two arrays 1041 and 104r are mechanically configured to result in a longer continuous line array of microphones. (For two- or three-dimensional microphone array geometries, a larger grid array or larger array surface results from connecting the two arrays.) It is well known that a larger array can be used to result in a directivity gain towards the user's mouth that exceeds the directivity gain of each single array. This use mode would be particularly beneficial in noisy environments, where the user wishes to benefit from improved transmission speech quality. For example, this mode of the array could be used while driving in a car or when the user is at a noisy restaurant.

[0029] The two arrays could be connected by embedding magnets in the capsule ends, or a physical connector of any of various types could be used. In addition to magnets or physical connectors, the ends of the capsules could contain electrical connections to provide a means to combine the electrical signals from the two arrays. For example, this might be necessary if the processing is implemented local to the arrays. The electrical connections would typically consist of power, ground, and a microphone-signal bus.

[0030] For example, using the implementation of FIG. 1(C), to utilize two arrays 104l and 104r of five microphones each, a signal bus that can combine ten microphone signals would be used. For microphone elements providing an output signal in analog format, this can be accomplished using ten wires, with at least seven connections established between the two arrays (power, ground, and five microphone signals). Alternatively, for digital-output microphones, one wire serving as a time-division multiplexed bus could be used to multiplex the five microphone signals, with only three connections established between the arrays (power, ground, and one digital bus).

[0031] The user could use an application on the smartphone to select Enhanced Directivity mode, or the act of connecting the two arrays together could automatically signal to the smartphone, or processors embedded in the array capsules, to configure the beamforming algorithm for enhanced directivity.

[0032] In the stereo recording mode of FIGS. 1(D) and 2(D), the two arrays 104l and 104r are used in an audio-video or audio-only recording configuration to capture a two-channel, stereo recording of a performance or other event. The two earphones may be worn by the user or held or otherwise arranged to point towards the performance in a left-of-stage and right-of-stage orientation. In this way, the sensitivity patterns of the two arrays are oriented as in FIG. 2(D), and the resulting stereo recording captures a more-realistic presentation of the event than achievable using a single omnidirectional microphone.

[0033] Alternative embodiments may also be able to support one or more surround or other multichannel sound (e.g., 5.1) recording modes. For instance, with two arrays arranged in a “V”, “T”, or “+” configuration, it is possible to form multiple beams pointing in directions that would be appropriate for surround sound recording and playback. 5.1 surround playback systems have defined preferred loudspeaker locations. It would therefore be desirable to point multiple beams in the preferred 5.1 directions to achieve good spatial playback on the 5.1 system. In certain embodiments, the audio signals generated by all of the different microphones in the arrays could be individually recorded in memory (e.g., in the smartphone) and used for later postprocessing to give the user control over the desired beampatterns that suit desired spatial playback characteristics.

[0034] For best results, it would be desirable to know what the actual user-selected microphone array geometry was. One could achieve this by detecting the magnetic field strength at various points along the arrays and using this information to determine the overall array geometry between two or more arrays. It is conceivable that one could place magnets in a way that constrains the microphone array geometry to a few available geometries. From one or more magnetic sensors configured to detect the residual magnetic fields or one or more other strategically placed pressure sensors, one could therefore detect which discrete geometry was selected.

[0035] Another possibility to detect the array geometry and/or orientation is to use an earbud's loudspeaker to output a known acoustic test signal and use some or all of the resulting microphone signals to determine the array geometry/orientation. Processing could compute the phase delay and/or amplitude differences in the acoustic test signal traveling from the loudspeaker to the different microphones and derive the array geometry/orientation from these phase delay and/or amplitude differences. The known acoustic test signal could be purposefully designed to be inaudible to humans. In some implementations, the inaudible audio signals could be dynamically designed to be masked by ambient sound. Note that, as used in this specification, the term “inaudible” covers sounds that are outside the human listening range in terms of signal frequencies as well as sounds that are otherwise undetectable such as sounds that would be masked to humans by background noise in the listening environment.

[0036] In the conference mode of FIGS. 1(E) and 2(E), the two microphone arrays 104l and 104r are configured and used as a transmitting, beamformed microphone array in a two-way speakerphone call. The smartphone's loudspeaker (not shown), or the earphone loudspeakers 106l and 106r if capable, are used as the loudspeaker that renders audio received by the phone from the other party in the two-way call. FIG. 1(E) shows two possible arrangements of the two arrays. The arrays are either overlapped or adjoined at (generally) right angles to each other to result in a two-dimensional microphone array structure. Such an arrangement facilitates beamforming and beamsteering at arbitrary angles in the entire 360-degree space of the plane containing the two arrays. FIG. 2(E) depicts the rotatable beampattern possible with such an arrangement.

[0037] Geometries other than the X- or “+”-shaped and the L- or V-shaped geometries of FIG. 1(E) are possible, such as a T-shaped geometry in which the end of one array is connected to the middle of the other array.

[0038] The enhanced directivity mode of FIGS. 1(C) and 2(C) places the two arrays end-to-end to form a linear array of twice the length of each array. As discussed, the longer baseline array can be exploited to improve overall beam directivity (narrow beam) in the direction of the user's mouth. Instead of an end-to-end configuration, the two microphone arrays could alternatively be adjoined side-by-side to result in a two-dimensional array having two microphones in one dimension and five in the other (for the five-microphone arrays shown in FIG. 1). Those skilled in the art understand that such two-dimensional arrays can be exploited, mathematically, to produce beampattern and beam directivity characteristics that differ from a one-dimensional linear array, providing advantages in certain use scenarios. For example, a differential microphone beamformer designed for a linear array of microphones, while benefitting from high directivity towards the user's mouth, is not steerable off the 0-degree axis. A two-dimensional differential array, in contrast, offers a tradeoff between beampattern directivity and beam direction.

[0039] A special case of the single-sided mode is when the user holds the array corresponding to the earphone that is not inserted into or otherwise connected to the ear, close to his/her mouth. A speech-activity-based metric would then select the better of the two arrays, and not necessarily the one side whose earphone is on the ear. This mode is essentially a hybrid of single-sided mode and the two-sided mode.

[0040] Although the disclosure has been described in the context of smartphone accessory systems having two earphones, in alternative embodiments, an accessory system might have only one earphone.