Teleconferencing system with visual feedback

[0002] The invention relates generally to the reception, mixing, analysis, and selection of acoustic signals in a noisy environment, particularly in the context of speakerphone and telephone conferencing systems.

[0003] Although telephone technology has been with us for some time and, through a steady flow of innovations over the past century, has matured into a relatively effective, reliable means of communication, the technology is not flawless. Great strides have been made in signal processing and transmission of telephone signals and in digital networks and data transmission. Nevertheless, the basic telephone remains largely unchanged, with a user employing a handset that includes a microphone located near and directed towards the user's mouth and an acoustic transducer positioned near and directed towards the user's ear. This arrangement can be rather awkward and inconvenient. In spite of the inconvenience associated with holding a handset, this arrangement has survived for many years: for good reason. The now familiar, and inconvenient, telephone handset provides a means of limiting the inclusion of unwanted acoustic signals that might otherwise be directed toward a receiver at the “other end” of the telephone line. With the telephone's microphone held close to and directed toward a speaker's mouth other acoustic signals in the speaker's immediate vicinity are overpowered by the desired speech signal.

[0004] However, there are many situations in which the use of a telephone handset is simply impractical, whether because the telephone user's hands must be free for activities other than holding a handset or because several speakers have gathered for a telephone conference. “Hands free” telephone sets of various designs, including various speaker-phones and telephone conferencing systems, have been developed for just such applications. Unfortunately, speaker-phones and telephone conferencing systems in general tend to exhibit annoying artifacts of their acoustic environments. In addition to the desired acoustic signal from a speaker, echos, reverberations, and background noise are often combined in a telephone transmission signal.

[0005] In audio telephony systems it is important to accurately reproduce the desired sound in the local environment, i.e., the space in the immediate vicinity of a speaker, while minimizing background noise and reverberance. This selective reproduction of sound from the local environment and exclusion of sound outside the local environment is the function at which a handset is particularly adept. The handset's particular facility for this function is the primary reason that, in spite of their inconvenience, handsets nevertheless remain in widespread use. For teleconferencing applications handsets are impractical, yet it is particularly advantageous to capture the desired acoustic signals with a minimum of background noise and reverberation in order to provide clear and understandable audio at the receiving end of telephone line.

[0006] A number of technologies have been developed to acquire sound in the local environment. Some teleconferencing systems employ directional microphones, i.e., microphones having a fixed directional pickup pattern most responsive to sounds along the microphone's direct axis, in an attempt to reproduce the selectivity of a telephone handset. If speakers are arranged within a room at predetermined locations which locations are advantageously chosen based upon the responsivity of microphones situated about the room, acceptable speech reproduction may be achieved. The directional selectivity of the directional microphones accents speech that is directed toward a microphone and suppresses other acoustic signals such as echo, reverberations, and other off-axis room sounds. Of course, if these undesirable acoustic signals are directed on-axis toward one of the microphones, they too will be selected for reproduction. In order to accommodate various speakers within a room, such systems typically gate signals from the corresponding microphones on or off, depending upon who happens to be actively speaking. It is generally assumed that the microphone receiving the loudest acoustic signal is the microphone corresponding to the active speaker. However, this assumption can lead to undesirable results, such as acoustic interference, which is discussed in greater detail below.

[0007] Moreover, it is unnatural and uncomfortable to force a speaker to constantly “speak into the microphone” in order to be heard. More recently, attempts have been made to accommodate speakers as the change positions in their seats, as they move about a conference room, and as various participants in a conference become active speakers. One approach to accommodating a multiplicity of active speakers within a conference room involves combining signals from two directional microphones to develop additional sensitivity patterns, or “virtual microphones”, associated with the combined microphone signals. To track an active speaker as the speaker moves around the conference room, the signal from the directional microphone or virtual directional microphone having the greatest response is chosen as the system's output signal. In this manner, the system acts, to some extent, as directional microphone that is rotated around a room to follow an active speaker.

[0008] However, such systems only provide a limited number of directions of peak sensitivity and the beamwidth is typically identical for all combinations. Some systems employ microphone arrangements which produce only dipole reception patterns. Although useful in some contexts, dipole patterns tend to pick up noise and unwanted reverberations. For example, if two speakers are seated across a table from one another, a dipole reception pattern could be employed to receive speech from either speaker, without switching back and forth between the speakers. This provides a significant advantage, in that the switching of microphones can sometimes be distracting, either because the speech signal changes too abruptly or because the background noise level shifts too dramatically. On the other hand, if a speaker has no counterpart directly across the table, a dipole pattern will, unfortunately, pick up the background noise across the table from the speaker, as well as that in the immediate vicinity of the speaker. Additionally, with their relatively narrow reception patterns, or beams, dipole arrangements are not particularly suite for wide area reception, as may be useful when two speakers, although seated on the same side of a conference table, are separated by some distance. Consequently, systems which employ dipole arrangements tend to switch between microphones with annoying frequency in such a situation. This is also true when speakers are widely scattered about the microphone array.

[0009] One particularly annoying form of acoustic interference that crops up in the context of a telephone conference, particularly in those systems which select signals from among a plurality of microphones, is a result of the fact that the energy of an acoustic signal declines rapidly with distance. A relatively small acoustic signal originating close to a microphone may provide a much more energetic signal to a microphone than a large signal that originates far away from a microphone. For example, rustling papers or drumming fingers on a conference table could easily dominate the signal from an active speaker pacing back and forth at some distance from the conference table. As a result, the receiving party may hear the drumbeat of “Sing, Sing, Sing” pounded out by fingertips on the conference table, rather than the considered opinion of a chief executive officer in the throes of a takeover battle. Oftentimes people engage in such otherwise innocuous activities without even knowing they are doing so. Without being told by an irritated conferee that they are disrupting the meeting, there is no way for them to know that they have done so, and they continue to “drown out” the desired speech. At the same time, the active speaker has no way of knowing that their speech has been suppressed by this noise unless a party on the receiving end of the conversation asks them to repeat a statement.

[0010] A telephone system in accordance with the principles of the present invention includes two or more cardioid microphones held together and directed outwardly from a central point. Mixing circuitry and control circuitry combines and analyzes signals from the microphones and selects the signal from one of the microphones or from one of one or more predetermined combinations of microphone signals in order to track a speaker as the speaker moves about a room or as various speakers situated about the room speak then fall silent.

[0011] In an illustrative embodiment, an array of three cardioid directional microphones, A, B, and C, are held together directed outward from a central point and separated by 120 degrees. Visual indicators, in the form of light emitting diodes (LEDs) are evenly spaced around the perimeter of a circle concentric with the microphone array. Mixing circuitry produces ten combination signals, A+B, A+C, B+C, A+B+C, A−B, B−C, A−C, A−0.5(B+C), B−0.5(A+C), and C−0.5(B+A), with the “listening beam” formed by combinations, such as A−0.5(B+C), that involve the subtraction of signals, generally being more narrowly directed than beams formed by combinations, such as A+B, that involve only the addition of signals. An omnidirectional combination A+B+C is employed when active speakers are widely scattered throughout the room. Weighting factors are employed in a known manner to provide unity gain output. That is, the combination signals are weighted so that they produce a response that is normalized to that of a single microphone, with the maximum output signal from a combination equal to the maximum output signal from a single microphone.

[0012] Control circuitry selects the signal from the microphone or from one of these predetermined microphone combinations, based generally on the energy level of the signal, and employs the selected signal as the output signal. The control circuitry also operates to limit dithering between microphones and, by analyzing the beam selection pattern may switch to the omnidirectional reception pattern afforded by the A+B+C combination. Similarly, the control system analyzes the beam selection pattern to select a broader beam that encompasses two active speakers, rather than switching between two narrower beams that each covers one of the speakers. Through the addition and subtraction of the basic cardioid reception patterns, the control circuitry may be employed to form a wide variety of combination reception patterns. In the illustrative embodiment, the output microphone signal is chosen from one of a plurality of predetermined patterns though. That is, although a plurality of combinations are employed, reception patterns typically are not eliminated, although patterns may be added, in the process of selecting and adjusting reception patterns.

[0013] The control circuitry also operates the visual feedback indicator, i.e., a concentric ring of LEDs in the illustrative embodiment, to indicate the direction and width of the listening beam, thereby providing visual feedback to users of the system and allowing speakers to know when the microphone system is directed at them.

[0014] The above and further advantages of the invention may be better understood by referring to the following description in conjunction with the accompanying drawings in which:

[0015] FIG. 1 is a top plan view of the possible pickup response for a 3-microphone system.

[0016] FIG. 2 is a top plan view of the pickup response provided when only one of the three microphone elements is used.

[0017] FIG. 3 is a top plan view of the pickup response provided when two of the microphone elements responses are summed together equally.

[0018] FIG. 4 is a top plan view of the possible pickup response provided when one microphone signal is subtracted from the signal of another.

[0019] FIG. 5 is a top plan view of the possible pickup response provided when all three microphone signals are added equally.

[0020] FIG. 6 is a top plan view of the possible pickup response when the signals of two microphones are added, scaled and subtracted from the signal of a third microphone.

[0021] FIG. 7 is a top plan view of a LED microphone layout and LED pattern in accordance with the principles of the invention.

[0022] FIGS. 8athrough 8dare top plan views, respectively, of the LED illumination patterns when one microphone signal is being used, the signals of two microphones are summed equally, the signals of all three microphones are added equally, and the signals of two microphones are added, scaled and subtracted from the signal of a third microphone.

[0023] FIG. 9 is a functional block diagram showing the steps involved in beam selection and visual feedback for the microphone system.

[0024] FIG. 10 is a conceptual block diagram of cascaded microphone arrays in accordance with the principles of the present invention.

[0025] A telephone system in accordance with the principles of the present invention includes two or more cardioid microphones held together and directed outwardly from a central point. Mixing circuitry and control circuitry combines and analyzes signals from the microphones and selects the signal from one of the microphones or from one of one or more predetermined combinations of microphones in order to track a speaker as the speaker moves about a room or as various speakers situated about the room talk then fall silent. The system may include, for example, an array of three cardioid directional microphones, A, B, and C, held together, directed outwardly from a central point, and separated by 120 degrees. Directional indicators, in the form of light emitting diodes (LEDs) are evenly spaced around the perimeter of a circle concentric with the microphone array each microphone generates an output signal designated as A, B, C, respecitvely. Mixing circuitry produces combination signals, such as A+B, A+C, B+C, A+B+C, A−B, B−C, A−C, A−0.5(B+C), B−0.5(A+C), and C−.05(A+B), with the “listening beam” formed by higher order combinations that include subtraction of signals, such as the A−0.5(B+C) combination, being more narrowly directed than that do not involve the subtraction of signals. Control circuitry selects the signal from the microphone or from one of the predetermined microphone combinations, based generally on the energy level of the signal, and employs the selected signal as the output signal. Additionally, the control circuitry lights selected LEDs to indicate the direction and width of the listening beam. This automatic visual feedback mechanism thereby provides a speaker with a near-end indication of whether he is being heard and also provides others within the room an indication that they may be interrupting the conversation.

[0026] Referring to the illustrative embodiment of FIG. 1, a microphone system 100 assembled in accordance with the principles of the invention includes three cardioid microphones, A, B, and C, mounted 120 degrees apart, as close to each other and a central origin as possible. Each of the microphones has associated with it a cardioid response lobe, La, Lb, and Lc, respectively. Microphones having cardioid response lobes are known. Various directional microphone response patterns are discussed in U.S. Pat. No. 5,121,426, to Baumhauer, Jr. et al., which is hereby incorporated by reference. The microphones, A, B, and C, are oriented outwardly from an origin 102 so that the null of each microphone's response lobe is directed at the origin. By combining the microphones' electrical response signals in various proportions, different system response lobes may be produced, as discussed in greater detail in the discussion related to FIG. 14.

[0027] As seen is FIG. 1, each cardioid microphone has a response that varies with the off-axis angle fq according to the following equation:

[0028] ½+½ cos φ  (1)

[0029] The response pattern described by this equation is the pear-shaped response shown by lobes La, Lb, and Lc for the microphones A, B, and C. Response lobe La is centered about 0 degrees, Lb about 120 degrees, and Lc 240 degrees. As illustrated by equation (1), each microphone has a normalized pickup value of unity along its main axis of orientation pointing outwardly from the origin 102, and a value of zero pointing in the opposite direction, i.e., towards the origin 102.

[0030] The pear-shaped response pattern of a single microphone, microphone A, is more clearly illustrated the response chart of FIG. 2, where like components to those shown in FIG. 1 are assigned like descriptors. Note that the response pattern of microphone A falls off dramatically outside the range of +−60 degrees. Consequently noise and reverberance outside that range, particularly to the rear of the microphone would have little effect on the signal produced by microphone A. Consequently, this arrangement could be used advantageously to reproduce sound from a speaker in that +−60 degree range.

[0031] By combining signals from various microphones a number of response patterns may be obtained. The response lobe L(a+b) of FIG. 3 illustrates that a much broader response pattern may be obtained from a combination of cardioid microphones arranged as illustrated. With the inputs from microphones A and B each given equal weight then added, the response pattern L(a+b) is described by the following equation:

[0032] (½+½ cos φ)+(½+½ cos(φ−120))=1+½(cos φ+cos(φ−120))  (2)

[0033] A multiplicative gain would be applied to this signal to normalize to unity gain. That is, the response of each of the microphones combined in a simple addition would be multiplied by ⅔. This response pattern provides a wider acceptance angle than that of a single cardioid microphone, yet, unlike a combination of dipole, or polar, microphones, still significantly reduces the contribution of noise and reverberation from the “rear” of the response pattern, i.e., from the direction of the axis of microphone C. This response pattern would be particularly useful in accepting sounds within the range of −60 and 180. A broader acceptance angle such as this is particularly advantageous for a situation where two speakers are located somewhere between the axes of microphones A and B. A wider acceptance angle such as this permits a system to select a signal corresponding to this broader acceptance angle, rather than dithering between signals from microphones A and B as a system might, should dipole response patterns be all that were available to it. Such dithering is known in the art to be a distraction and an annoyance to a listener at the far end of a telephone conference. Being able to avoid dithering in this fashion provides a significant performance advantage to the inventive system.

[0034] That is not to say that a dipole response pattern is never desirable. As illustrated in the response pattern of FIG. 4, a dipole response pattern may be obtained, for example, by subtracting the response of microphone B from that of microphone A. In FIG. 4 a dipole response lobe L(a−b) is produced by subtracting the response of microphone B from that of microphone A according to the following equation:

[0035] (½+½ cos φ)−(½+½ cos(φ−120))=½ cos φ−(½ cos(φ−120))=½(cos φ−cos(φ−120))=0.866(cos(φ+30)  (3)

[0036] A multiplicative gain would be applied to this signal to normalize to unity gain. By subtracting the signal of B from that of A, a narrower double sided pickup pattern is produced. In this example, the pattern effectively picks up sound between −75 and 15 degrees, and 105 and 195 degrees. This is especially well-suited for scenarios where audio sources are located to either side of the microphone, especially along broken line 104, and noise must be reduced from other directions.

[0037] Additional response patterns may be produced by using all three microphones. For example, FIG. 5 illustrates a response pattern that results from the addition of equally weighted signals from microphones A, B and C, which produces an omni-directional response pattern according to the following equation:

[0038] (½+½ cos φ)+(½+½ cos(φ−120))+(½+½ cos(φ+120))=1.5+½(cos φ+cos(φ−120)+cos (φ+120))=1.5  (4)

[0039] A multiplicative gain would be applied to this signal to normalize to unity gain. This angle-independent response allows for sounds from sources anywhere about the microphone array to be picked up. However, no noise or reverberance reduction is achieved.

[0040] As illustrated by the response pattern of FIG. 6, signals from all three microphones may be combined in other ways to produce, for example, the narrow dipole response pattern L(a−0.5(b+c)). The resulting narrow dipole pattern is directed toward 0 and 180 as described by the following equation:

[0041] (½+½ cos φ)−0.5((½+½ cos(φ−120))+(½+½ cos(φ+120)))=

[0042] (½+½ cos φ)−0.25(1+cos φ−120)+cos(φ+120))=

[0043] ½ cos φ−0.25(cos(φ−120)+cos(φ+120))=

[0044] 0.75 cos φf

[0045] A multiplicative gain would be applied to this signal to normalize to unity gain. With this combination, the pattern effectively picks up sound between −45 and 45 degrees, and between 135 and 225 degrees. This response pattern is especially well-suited for scenarios where audio sources are located to either side of the microphone, and noise must be reduced from other directions.

[0046] In the illustrative embodiment, responses from predetermined microphones and microphone combinations, such as that provided by microphones A, B, and C, and by microphone combinations A+C, A+B, B+C, A+B+C, A−B, B−C, A−C, A−0.5(B+C), B−0.5(A+C), and C−0.5(A+B) are analyzed and one of the predetermined combinations is employed as the output signal, as described in greater detail in the discussion related to FIG. 14.

[0047] In the illustrative embodiment, the microphone system includes six LEDs arranged in a concentric circle around the perimeter of the microphone array 100, with LEDs 106, 108, 110, 112, 114, and 116 situated at 0, 60, 120, 180, 240, and 300 degrees, respectively. As the LEDs are used for visual feedback, more or fewer LEDs could be employed, and any of a number of other visual indicators, such as an LCD display that displays a pivoting virtual microphone, could be substituted for the LEDs. The number and direction of LEDs lit indicates the width and direction of the reception pattern that has been selected to produce the telephone output signal. FIGS. 8athrough 8billustrate the LED lighting patterns corresponding to various reception pattern selections. In FIG. 8a, for example, LED 106 is lit to indicate that reception pattern La has been selected. Similarly, in FIG. 8b, LEDs 106, 108, and 110 are lit to indicate that the lobe, or reception pattern, L(a+b). In FIG. 8call the LEDs are lit to indicate that the omnidirectional pattern L(a+b+c) has been selected. And, in FIG. 8d, LEDs 106 and 112 are lit to indicate that the L(a−0.5(b+c)) pattern has been selected. The LED lighting pattern will typically be updated at the same time the response pattern selection decision is made.

[0048] Signal mixing, selection of reception patterns, control of the audio output signal and control of the visual indicators may be accomplished by an apparatus 900 which, in the illustrative embodiment, is implemented by a digital signal processor according to the functional block diagram of FIG. 9. Each microphone A, B, C, produces an electrical signal M_A, M_B, M_C, respectively, in response to an acoustic input signal. The analog response signals, M_A, M_B, and M_Cfor each microphone are sampled at 8,000 samples per second. Digitized signals from each of the three microphones A,B, and C are combined with one another to produce a total of thirteen microphone signals M_A, M_B, M_C, M_(A+B), etc., which provide maximum signal response for each of six radial directions spaced 60° apart and other combinations as discussed above. Response signals M_(A+B), M_(A+C), M_(B+C), etc., are formed by weighting, adding and subtracting the individual sampled response signals, thereby producing a total of thirteen response signals as previously described. For example, wM_A+(1−w)M_B=M_(A+B), where w is a weighing factor less than one, chosen to produce a response corresponding to a microphone situated between microphones A and B.

[0049] Because each of the thirteen signals is operated upon in the following manner before being operated upon in the beam selection functional block 910, only the operation upon signal M_A, will be described in detail, the same process applies to all thirteen signals. The digital signals are decimated by four in the decimator 902 to reduce signal processing requirements. Signal energies P_i(k) are continuously computed in functional block 904 for 16 ms signal blocks (32 samples) related to each of the thirteen response signals, by summing the absolute values of the thirty-two signal samples within each 16 ms block; i.e., totaling the thirty-two absolute values of signal samples within each block:

[0050] P_i(k)=Σ|m_ij(k)|

[0051] where:

[0052] i is an index ranging from 1 to 13, corresponding to the thirteen response signals and 1≧j≧32

[0053] P_i(k) is the signal energy associated with the ith response signal

[0054] |m_ij(k)| is the absolute value of the jth sample of the ith signal

[0055] The signal energies thus-computed are continuously low-pass filtered by adding a weighted filtered energy value from the previous block to a weighted energy value from the current block:

[0056] F_i(k)=aP_i(k)+(1−a)F_i(k−1)

[0057] Where:

[0058] F_iis the ith microphone's filtered energy value for the kth sample block

[0059] P_iis the ith microphone's signal energy value for the kth sample block

[0060] i is an index which varies from 1 to 13

[0061] 0_<a_<1, typically a=0.9

[0062] The minimum of all block energy values computed for a given microphone over the previous 1.6 seconds (100 sample blocks) is used in functional block 906 as a noise estimate for the associated microphone, or virtual microphone, i.e.,

[0063] N_i(k)=min {P_i(k) over 1.6 seconds}The current filtered energy values F_i(k) are summed to yield a total filtered energy value F_T(k).

[0064] F_T(k)=ΣF_i(k)

[0065] Similarly, the respective noise values, N_i(k), are summed to yield a total noise energy value.

[0066] The microphone signal associated with the highest current filtered energy value F_i(k) is selected in functional block 910 as a candidate for the microphone array's output signal. Smoothing is performed in functional block 912 as follows. If the total filtered energy value F_T(k) is greater than 1.414 times the previous total filtered energy value, and is greater than twice the total noise energy value, the selected output signal is used as the array output signal. Otherwise, the current signal from the previously-used microphone is used as the array output signal. This smoothing process significantly reduces whatever residual dithering may remain in the beam selection process. That is, although the broader beam patterns afforded by combinations such as the A+B, A+C, etc. combinations reduce dithering, when compared to conventional systems, the smoothing process provides additional margin, particularly when selecting among narrower beam patterns. The thus-selected output array signal is coupled for transmission on telephone lines in functional block 916. The selected signal is also employed, in functional block 914, to control the visual indicators, as previously described.

[0067] A plurality of the microphone arrays just described may be cascaded, as illustrated in FIG. 10. In such as cascaded arrangement, the output audio signal from one microphone system 1000 is input into a second similar system 1002. The second system 1002 uses its two directional microphones in addition to the first system's output to produce its composite output signal. Thus, the third microphone signal in the second unit is being replaced by the composite signal of the first unit. Similarly, a third microphone systems 1004 may be linked to the others. Such a cascading of microphone systems may employ two or more microphone systems. Alternatively, the microphone units may act independently, with an external controller determining the amount of mixing and switching among the systems' outputs. The composite outputs from each system would be fed into this controller.

[0068] The forgoing description of specific embodiments of the invention has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise forms disclosed, and many modifications and variations are possible in the light of the above teachings. The embodiments were chosen and described in order to best explain the principles of the invention and its practical application and to thereby enable others skilled in the art to best utilize the invention. It is intended that the scope of the invention be limited only by the claims appended hereto.