US 12,587,787 B2Grant
System for dynamically adjusting the gain structure of sound sources contained within one or more inclusion and exclusion zones
Issue Date:2026-03-24
•27 Claims
•42 Drawing Sheets
Abstract
A system is provided for intelligent and optimized zone gain management of sound sources within priority (inclusion) zones and adjacent to the priority (inclusion) zone boundaries of the 3D space by using sound source location and signal level information of sound sources from both inside the inclusion zone and outside the inclusion zone in the exclusion zone for the purpose of optimizing the audio gain structure of desired sound sources located in priority (inclusion) zones and minimizing the gain structure of undesired sound sources in low priority (exclusion) zones. The system utilizes all virtual microphones in the 3D space by preferably assigning all available virtual microphones to either an inclusion zone or exclusion zone configuration for the purpose of tracking and monitoring all sound sources in the space regardless of their position in the 3D space.
Metadata
Assignee
- NUREVA, INC.
Inventor
- Kael Blais
Application Information
Application Number:US 18/644,745
Filing Date:2024-04-24
Priority Date:2023-05-09
Art Unit:2695
Classifications
IPC:
H04R5/00H04R3/00H04R3/04
Field of Search:
H04R 3/04H04R 3/005
Patent Drawings (42 sheets)
Description
CROSS REFERENCE TO RELATED APPLICATIONS
[0001] This application claims priority to U.S. Provisional Patent Application No. 63/465,087, filed May 9, 2023, the entire contents of which are incorporated herein by reference.
BACKGROUND OF THE INVENTION
1. Field of the Invention
[0002] The present invention generally relates to audio capture systems, and more particularly, the defining and configuration of one or more combinations of inclusion and exclusion zones to intelligently prioritize areas of the 3D space for audio sound source pick up while dynamically optimizing the gain structure of sound sources in and transitioning location relative to the borders of the prioritized zones/areas by taking into account the location and signal level for all sound sources in the 3D space for multi-user conference systems to optimize audio signal and noise level performance in and around the prioritized areas of the shared space.
2. Description of Related Art
[0003] Obtaining high quality audio at both ends of a conference call is difficult to manage due to, but not limited to, variable room dimensions, dynamic seating plans, roaming participants, unknown number of microphones and locations, unknown speaker system locations, known steady state and unknown dynamic noise, variable desired sound source levels, and unknown room characteristics. This may result in conference call audio having a combination of desired sound sources (participants) and undesired sound sources (return speaker echo signals, HVAC ingress, feedback issues and varied gain levels across all sound sources, etc.).
[0004] To provide an audio conference system that addresses dynamic room usage scenarios and the audio performance variables discussed above, microphone systems need to be thoughtfully designed, installed, configured, and calibrated to perform satisfactorily in the environment. The process starts by placing an audio conference system in the room utilizing one or more microphones. The placement of microphone(s) is critical for obtaining adequate room coverage which must then be balanced with proximity of the microphone(s) to the participants to maximize desired vocal audio pickup while reducing the pickup of speakers and undesired sound sources. In a small space where participants are collocated around a table, simple audio conference systems can be placed on the table to provide adequate performance and participant audio room coverage. Larger spaces require multiple microphones of various form factors which may be mounted in any combination of, but not limited to, the ceiling, tables, walls, etc., making for increasingly complex and difficult installations. To optimize performance of the audio capture system even further with usage of the room in mind, the microphone system will typically be configured to provide zone-based coverage areas. The idea is to create areas in the room of higher priority for sound source pickup than other areas of the room. Examples of this would be but not limited to the front of a classroom where a teacher has priority over the students, or presentation rooms where the presenter has priority over the attendees, or a boardroom where the seats at the table have priority over areas outside the table boundaries. If more than one priority zone is desired microphone systems of sufficient complexity can be configured to provide more than one priority area/zone. The idea is to minimize unwanted sound source contributions that are not located within high priority areas of the room while maximizing the audio pickup of sound sources in the priority areas/zone.
[0005] Zoning implementations in the current art have typically been limited to certain approaches. One approach is to use wireless and/or a combination of wired discrete microphones to limit the sound source audio pickup to a specific microphone location which is typically collocated in very close proximity to a person. The very nature of this type of microphone will create a small zone/area of audio pickup which does isolate the desired talker (person) but at the expense of system installation complexity, limited room coverage, requiring a physical microphone for each presenter and system setup and maintenance complexities especially if the system needs to be expanded. For small and simple tabletop installations this may be an acceptable approach.
[0006] Another approach in the current art has been to use the performance properties of a beamformer microphone array. The beamformer array has a polar plot on the surface that seems to support a zoning implementation. The typical polar plot contains an area of on-axis gain which is designed to maximize gain in this region and an area of off-axis rejection which is designed to eliminate sounds from this area of the coverage pattern. With a sufficiently complex beamformer array it is possible to define one or more zones in the space by aiming and shaping the on-axis beams to point at the desired coverage area providing specific coverage for regions in the room. Sound sources outside of the on-axis region will be ignored. Placement of the beamformer will be critical to the positioning and shaping of the priority regions/zones that can be configured and placed in the room, as the regions/zones are constrained to the placement of the aperture of the array in the room. The shapes of the zones will be further limited to the available lobing patterns or simple geometric layouts of aggregating lobes/beam patterns which can be limiting and lack flexibility especially in a 3D spatial context. Complex geometric coverage zones with specific dimensions in the x, y, z axis is typically not feasible.
[0007] In addition to the coverage region shaping and positioning issues the performance of the transition area between on-axis and off-axis regions can cause the array audio response to be very rigid and abrupt as sound sources approach or cross this region of the polar plot. A sound source straddling the zone boundary or put another way moving between the on-axis and off axis region of the coverage pattern may be heard at the far end of the call in a very uneven way or drop in and out of the conference call all together. Since the lobe shape properties are directly tied to the creation of and configuration of the in-room zone configurations the performance properties of the beamformer array make managing the gain structure of sound sources on the edge of the on-axis region and in the off-axis regions difficult and unpredictable.
[0008] The optimum solution would be a conference system that is able to implement independent of the array or physical discrete microphones, one or more zone coverage configurations with intelligent gain structure management for desired sound sources based on their location in and around the priority zones in such a manner that it is not limited to or constrained by the position of, geometry and implementation of the array. However, fully realizing independent of the physical array, priority coverage zones with both inclusion and exclusion zone properties while setting intelligent gain structures for the desired sound sources based on knowing the location and signal level of all sound sources in the room relative to inclusion and exclusion zones has proven difficult and insufficient within the current art.
[0009] Being able to optimize the desired sound source audio gain when they are in, between and transitioning to and from priority zones requires the monitoring and tracking of all sound sources independent of the location of the one or more priority zones is preferably required, and where the one or more priority zones can be placed, sized and shaped to very precise x, y, z coordinates in the 3D space independent of the array which further improves the system's ability to manage the desired sound source's audio signal gain while minimizing the contribution of unwanted sound sources, reduction of ingress from other non-priority areas, and sound source bleed-through from coverage grids that extend beyond wall boundaries and wide-open spaces.
[0010] Systems in the current art do not continually monitor and track all sound sources in the 3D space irrespective of the configured priority zones and thus are not able to intelligently manage the gain structure of all sound sources whether they are in a priority zone, outside the priority zone or transitioning between zones and instead rely on standard polar plot on-axis and off-axis region to form priority coverage zone areas and gain management of sound sources.
[0011] Therefore, the current art is not able to provide intelligent gain management for the target sounds sources located within and in close proximity to priority zones boundaries, nor is the current art able to provide priority zones disassociated from the location of the physical array with complex zone shapes, sizes and positioning in the 3D space.
SUMMARY OF THE INVENTION
[0012] An object of the present embodiments is to, in real-time, provide intelligent and optimized zone gain management of sound sources within priority (inclusion) zones and adjacent to the priority (inclusion) zone boundaries of the 3D space by using sound source location and signal level information of sound sources from both inside the inclusion zone and outside the inclusion zone in the exclusion zone for the purpose of optimizing the audio gain structure of desired sound sources located in priority (inclusion) zones and minimizing the gain structure of undesired sound sources in low priority (exclusion) zones.
[0013] More specifically, it is an object of the present invention to preferably utilize all virtual microphones in the 3D space by preferably assigning all available virtual microphones to either an inclusion zone or exclusion zone configuration for the purpose of tracking and monitoring all sound sources in the space regardless of their position in the 3D space.
[0014] And even more specifically, it is an object of the present invention to identify the virtual microphone with the largest processing gain value in each inclusion and exclusion zone for the purpose of maximizing the gain of the target virtual microphone in the inclusion zone with the highest priority which is correlated to the active desired sound source and to conversely minimize the gain of the highest processing gain virtual microphone in the exclusion zone to significantly reduce the contribution of undesired sound sources in the output signal at the remote end of the conference call.
[0015] The present invention provides a real-time adaptable solution to undertake automatic zone gain control to optimize the gain of the selected targeted virtual microphone in the inclusion zone and to manage sound source targets at the edge of and outside the edge of the inclusion zone for the best listening experience at the remote end of the conference call.
[0016] The preferred embodiments comprise both algorithms and hardware accelerators to implement the structures and functions described herein.
[0017] These advantages and others are achieved, for example, by a system for dynamically adjusting gain structures of sound sources in a shared 3D space including one or more inclusion zones and one or more exclusion zones. The system includes a combined microphone array including one or more of individual microphones and/or microphone arrays each including a plurality of microphones. The microphones in each microphone array are arranged along a microphone axis. The system further includes one or more system processors communicating with the combined microphone array. The one or more system processors include one or more audio channel profiles (ACPs) and are configured to perform operations. The operations includes steps of (i) obtaining predetermined coverage zone dimensions based on the locations of the microphones of the combined microphone array, (ii) populating the coverage zone dimensions with one or more virtual microphones, (iii) obtaining a combined microphone signal, for each audio channel profile (ACP), by combining microphone signals into desired channel audio signals by applying positional based gain control (PBGC) parameters to adjust microphones to control positional based microphone gains based on location information of the sound sources, (iv) performing processes to obtain a zoning gain for each ACP, and (v) generating an output channel for each ACP by multiplying the zoning gain with the combined microphone signal. The performing processes to obtain a zoning gain for each ACP includes steps of receiving a list of sound sources obtained by utilizing the virtual microphones, receiving zone parameters for one or more inclusion zones (IZ) and one or more exclusion zones (EZ), identifying a gain source (GS) and a list of one or more attenuation sources (AS), determining a zoning ratio based on the gain source, the list of the one or more attenuation sources and active zone configuration parameters, and calculating zoning gain based on the zoning ratio, maximum gain of the one or more inclusion zones and minimum gain of the one or more exclusion zones.
[0018] These advantages and others are achieved, for example, by a method for dynamically adjusting gain structures of sound sources in a shared 3D space including one or more inclusion zones and one or more exclusion zones. The method includes steps (i)-(v) described above.
[0019] These advantages and others are achieved, for example, by one or more non-transitory computer-readable media for dynamically adjusting gain structures of sound sources in a shared 3D space including one or more inclusion zones and one or more exclusion zones. The computer-readable media includes instructions configured to cause a system processor to perform the steps (i)-(v) described above.
BRIEF DESCRIPTION OF THE DRAWINGS
[0020] FIGS. 1a, 1b and 1c are diagrammatic examples of a typical audio conference setups across multiple device types.
[0021] FIGS. 2a and 2b are graphical structural examples of microphone array layouts supported in the embodiment of the present invention.
[0022] FIGS. 3a and 3b are examples of microphone coverage pattern arrangements.
[0023] FIGS. 4a, 4b and 4c are prior art examples of coverage pattern arrangements as it relates to on-axis and off-axis beamformer array performance.
[0024] FIGS. 5a and 5b are examples of 2D and 3D virtual microphone arrangements supported in the embodiment of the present invention.
[0025] FIG. 6 is diagrammatic example of virtual microphone zone shape configurations supported in the embodiment of the present invention.
[0026] FIGS. 7a and 7b are exemplary illustrative diagrams of the present invention mapping specific virtual microphones in a 3D space to identified target sources in the defined Inclusion zone and Exclusion zone areas.
[0027] FIG. 8 is an illustration of a typical virtual microphone coverage map with no zone configurations applied.
[0028] FIGS. 9a, 9b, 9c, 9d, 9e, 9f, 9g, and 9h are exemplary top-down illustrative diagrams of the present invention mapping specific virtual microphones in a 3D space to identified target sources in the defined inclusion and exclusion zone areas.
[0029] FIG. 10 is an illustration of two separate sound source targets moving between virtual microphones in the inclusion and exclusion zones in a 3D space supported in the embodiment of the present invention.
[0030] FIGS. 11a, 11b, 11c, 11d and 11e are functional and structural diagrams of an exemplary embodiment of the present invention for automatically identifying targets within defined inclusion and exclusion zones for the purpose of dynamically adjusting the gains of the targets in the inclusion zones and exclusion zones relative to each other in a 3D space.
[0031] FIGS. 12a, 12b, 12c and 12d are exemplary embodiments of the logic flowcharts of the Automatic Zoning Gain Control processor process of the present invention.
[0032] FIGS. 13a, 13b, 13c, 13d, 13e and 13f are exemplary illustrative concepts of the present invention outlining the tracking of both a gain source and an attenuation source in a 3D space based on identified sound sources in and around the inclusion and exclusion zones.
[0033] FIGS. 14a, 14b, 14c, 14d, 14e and 14f are exemplary illustrative concepts of the present invention mapping different inclusion and exclusion zone configurations to different ACP.
[0034] FIGS. 15a, 15b, 15c and 15d are exemplary illustrative drawings of the present invention for outlining the gain and attenuation sources relative to a sound source target moving between inclusion, exclusion, and undefined zones in the 3D space.
DETAILED DESCRIPTION OF THE PRESENT INVENTION
[0035] The present invention is directed to apparatus and methods that enable groups of people (and other sound sources, for example, recordings, broadcast music, Internet sound, etc.), known as “participants”, to join together over a network, such as the Internet or similar electronic channel(s), in a remotely-distributed real-time fashion employing personal computers, network workstations, and/or other similarly connected appliances, often without face-to-face contact, to engage in effective audio conference meetings that utilize large multi-user rooms (spaces) with distributed participants that require specific zone coverage configurations.
[0036] Advantageously, embodiments of the present apparatus and methods afford an ability to provide a microphone array system that establishes a virtual microphone array coverage grid that is adapted to each unique installation, room and situation by allowing the user to configure the microphone array for any number of gain zones and/or attenuation zones with dynamic gain structures based on the sound sources' locations relative to any one zone and/or within a zone including sound sources that transition from one zone to another in real-time irrespective of array geometry and configuration to maximize desired sound source audio quality and performance for all participants at the far end of the conference call.
[0037] A notable challenge to creating a microphone array that can instantiate and manage the tracking and monitoring of a plurality of sound sources in a 3D space for the purpose of intelligently adjusting the gain structure of the desired sound source in the gain zone is being able to monitor and track the level and location of sound sources that are not in a gain zone without adding additional arrays or hardware to track and measure these sound sources. And preferably utilize a microphone array system that can completely cover the room with a coverage grid that is capable of creating any number of gain and attenuation zones that are able to be monitored for the purpose of tracking and measuring all sound sources in the complete space to allow for the intelligent optimization of the gain structure of sound sources in a gain zone and sound sources entering and leaving the gain zones while minimizing the contribution of undesired sound sources so the participants at the remote end of the call get the best experience possible.
[0038] A “microphone” in this specification may include, but is not limited to, one or more of, any combination of transducer device(s) such as, microphone element, condenser mics, dynamic mics, ribbon mics, USB mics, stereo mics, mono mics, shotgun mics, boundary mic, small diaphragm mics, large diaphragm mics, multi-pattern mics, strip microphones, digital microphones, fixed microphone arrays, dynamic microphone arrays, beam forming microphone arrays, and/or any transducer device capable of receiving acoustic signals and converting to electrical signals, and/or digital signals.
[0039] A “microphone point source” is defined for the purpose of this specification as the center of the aperture of each physical microphone. The microphones are considered to be omni-directional as defined by their polar plot and essentially can be considered an isotropic point source. This is required for determining the geometric arrangement of the physical microphones relative to each other. The microphones are considered to be a microphone point source in 3D space.
[0040] A “microphone arrangement” may be defined in this specification as a geometric arrangement of all the microphones contained in the microphone system. Microphone arrangements are required to determine the virtual microphone distribution pattern. The microphones can be mounted at any point in the 3D space, which may be a room boundary, such as a wall, ceiling, or floor. Alternatively, the microphones may be offset from the room boundaries by mounting on stands, tables or structures that provide offset from the room boundaries. The microphone arrangements are used to describe all the possible geometric layouts of the physical microphones.
[0041] An “inclusion zone” (IZ) may be defined in this specification as a defined area that encompasses a group of virtual microphones. This can be a 2-dimensional area in the case of a 2-dimensional arrangement of virtual microphones or a 3-dimensional volume in the case of a 3-dimensional arrangement of virtual microphones. The inclusion zone represents a physical space in which sounds are considered to be desirable. A zoning configuration will prioritize sound sources in inclusion zones when creating an output signal. In the context of Zoning Automatic Gain Control (AGC), an inclusion zone represents a region from which sound sources will have a positive gain applied.
[0042] An “exclusion zone” (EZ) may be defined in this specification as a defined area that encompasses a group of virtual microphones. This can be a 2-dimensional area in the case of a 2-dimensional arrangement of virtual microphones or a 3-dimensional volume in the case of a 3-dimensional arrangement of virtual microphones. The exclusion zone represents a physical space in which sounds are considered to be undesirable. In the context of Zoning AGC, an exclusion zone represents a region from which sound sources will have a negative gain applied.
[0043] An “undefined zone” (UZ) may be defined in this specification as representing any virtual microphones that are not part of an inclusion or exclusion zone. Sounds coming from an undefined zone are considered neither desirable nor undesirable. The virtual microphones in an undefined zone are simply ignored. An undefined zone represents a region from which no gain is specified, and the resulting level is only dependent on the IZ and EZ of the configuration.
[0044] An “audio channel profile” (ACP) may be defined in this specification to represent a configuration that is applied to an output audio channel. In the case of a system with multiple audio output channels, each channel has its own ACP. This allows each output channel to be configured independently for different needs. For example, a user might want two channels to focus on different areas of the room. This could be configured in the ACP of each channel. An ACP will contain the Zoning Parameters of an output channel such as the location and gains of inclusion and exclusion zones for that channel.
[0045] A “gain source” (GS) may be defined in this specification as representing a virtual microphone that tracks a sound source in an inclusion zone. Gain sources are bound to inclusion zones and remain inside of them at all times. The location of a gain source represents the physical location for which the individual microphone signals of the system will be aligned to produce the output signal of an ACP. Therefore, each ACP has one gain source. An ACP can have multiple inclusion zones but will always have one gain source. In the case of multiple inclusion zones, the gain source can move between inclusion zones but will always be inside one of them. The power of the gain source is used to measure the sound level inside of the inclusion zones.
[0046] An “attenuation source” (AS) may be defined in this specification as representing a virtual microphone that tracks a sound source in an exclusion zone. Attenuation sources are bound to exclusion zones and remain inside of them at all times. Attenuation sources are only used to measure the power of sound sources in exclusion zones so an ACP can be configured to support multiple attenuation sources. The power of the attenuation sources is used to measure the sound level inside of the exclusion zones. Like gain sources, attenuation sources can move between any of the exclusion zones in an ACP. Unlike with gain sources, an ACP does not align an output signal to any AS location so an ACP can support multiple simultaneous AS's.
[0047] A “microphone axis” may be defined in this specification as an arrangement of microphones that forms and is constrained to a single 1D line. Two or more microphone axis arrangements can be combined to form an overall microphone aperture arrangement. For example, two microphone axes arranged perpendicular to each other will form a microphone plane and two microphone planes arranged perpendicular to each other will form a microphone hyperplane.
[0048] A “virtual microphone” in this specification represents a point in space that has been focused on by the combined microphone array by time-aligning and combining a set of physical microphone signals according to the time delays based on the speed of sound and the time to propagate from the sound source each to physical microphone. A virtual microphone emulates the performance of a single, physical, omnidirectional microphone at that point in space.
[0049] A “coverage zone” in the specification may include physical boundaries such as wall, ceiling and floors that contain a space with regards to the establishment of installing and configuring a microphone system coverage patterns and dimensions. The coverage zone dimension can be known ahead of time or derived with a number of sufficiently placed microphone arrays also known as boundary devices placed on or offset from physical room boundaries.
[0050] A “combined array” in this specification can be defined as the combining of two more individual microphone elements, groups of microphone elements and other combined microphone elements into a single combined microphone array system that is aware of the relative distance between each microphone element to a reference microphone element, determined in configuration, and is aware of the relative orientation of the microphone elements such as a m-axis, m-plane and m-hyperplane sub arrangements of the combined array. A combined array will integrate all microphone elements into a single array and will be able to form coverage pattern configurations as a combined array.
[0051] A “conference enabled system” in this specification may include, but is not limited to, one or more of, any combination of device(s) such as, unified communications (UC) compliant devices and software, computers, dedicated software, audio devices, cell phones, a laptop, tablets, smart watches, a cloud-access device, and/or any device capable of sending and receiving audio signals to/from a local area network or a wide area network (e.g. the Internet), containing integrated or attached microphones, amplifiers, speakers and network adapters. PSTN, Phone networks etc.
[0052] A “communication connection” in this specification may include, but is not limited to, one or more of or any combination of network interface(s) and devices(s) such as, Wi-Fi modems and cards, internet routers, internet switches, LAN cards, local area network devices, wide area network devices, PSTN, Phone networks, etc.
[0053] A “device” in this specification may include, but is not limited to, one or more of, or any combination of processing device(s) such as, a cell phone, a Personal Digital Assistant, a smart watch or other body-borne device (e.g., glasses, pendants, rings, etc.), a personal computer, a laptop, a pad, a cloud-access device, a white board, and/or any device capable of sending/receiving messages to/from a local area network or a wide area network (e.g., the Internet), such as devices embedded in cars, trucks, aircraft, household appliances (refrigerators, stoves, thermostats, lights, electrical control circuits, the Internet of Things, etc.).
[0054] A “participant” in this specification may include, but is not limited to, one or more of, any combination of persons such as students, employees, users, attendees, or any other general groups of people that can be interchanged throughout the specification and construed to mean the same thing. Who gathering into a room or space for the purpose of listening to and or being a part of a classroom, conference, presentation, panel discussion or any event that requires a public address system and a UCC connection for remote participants to join and be a part of the session taking place. Throughout this specification a participant is a desired sound source, and the two words can be construed to mean the same thing.
[0055] A “desired sound source” in this specification may include, but is not limited to, one or more of a combination of audio source signals of interest such as: sound sources that have frequency and time domain attributes, specific spectral signatures, and/or any audio sounds that have amplitude, power, phase, frequency and time, and/or voice characteristics that can be measured and/or identified such that a microphone can be focused on the desired sound source and said signals processed to optimize audio quality before delivery to an audio conferencing system. Examples include one or more speaking persons, one or more audio speakers providing input from a remote location, combined video/audio sources, multiple persons, or a combination of these. A desired sound source can radiate sound in an omni-polar pattern and/or in any one or combination of directions from the center of origin of the sound source.
[0056] An “undesired sound source” in this specification may include, but is not limited to, one or more of a combination of persistent or semi-persistent audio sources such as: sound sources that may be measured to be constant over a configurable specified period of time, have a predetermined amplitude response, have configurable frequency and time domain attributes, specific spectral signatures, and/or any audio sounds that have amplitude, power, phase, frequency and time characteristics that can be measured and/or identified such that a microphone might be erroneously focused on the undesired sound source. These undesired sources encompass, but are not limited to, Heating, Ventilation, Air Conditioning (HVAC) fans and vents; projector and display fans and electronic components; white noise generators; any other types of persistent or semi-persistent electronic or mechanical sound sources; external sound source such as traffic, trains, trucks, etc.; and any combination of these. An undesired sound source can radiate sound in an omni-polar pattern and/or in any one or combination of directions from the center of origin of the sound source.
[0057] A “system processor” is preferably a computing platform composed of standard or proprietary hardware and associated software or firmware processing audio and control signals. An example of a standard hardware/software system processor would be a Windows-based computer. An example of a proprietary hardware/software/firmware system processor would be a Digital Signal Processor (DSP).
[0058] A “communication connection interface” is preferably a standard networking hardware and software processing stack for providing connectivity between physically separated audio-conferencing systems. A primary example would be a physical Ethernet connection providing TCP/IP network protocol connections.
[0059] A “Unified Communication Client (UCC)” is preferably a program that performs the functions of but not limited to messaging, voice and video calling, team collaboration, video conferencing and file sharing between teams and or individuals using devices deployed at each remote end to support the session. Sessions can be in the same building and/or they can be located anywhere in the world that a connection can be establish through a communications framework such but not limited to Wi-Fi, LAN, Intranet, telephony, wireless or other standard forms of communication protocols. The term “Unified Communications” may refer to systems that allow companies to access the tools they need for communication through a single application or service (e.g., a single user interface). Increasingly, Unified Communications have been offered as a service, which is a category of “as a service” or “cloud” delivery mechanisms for enterprise communications (“UCaaS”). Examples of prominent UCaaS providers include Dialpad, Cisco, Mitel, RingCentral, Twilio, Voxbone, 8×8, and Zoom Video Communications.
[0060] An “engine” is preferably a program that performs a core function for other programs. An engine can be a central or focal program in an operating system, subsystem, or application program that coordinates the overall operation of other programs. It is also used to describe a special-purpose program containing an algorithm that can sometimes be changed. The best-known usage is the term search engine which uses an algorithm to search an index of topics given a search argument. An engine is preferably designed so that its approach to searching an index, for example, can be changed to reflect new rules for finding and prioritizing matches in the index. In artificial intelligence, for another example, the program that uses rules of logic to derive output from a knowledge base is called an inference engine.
[0061] As used herein, a “server” may comprise one or more processors, one or more Random Access Memories (RAM), one or more Read Only Memories (ROM), one or more user interfaces, such as display(s), keyboard(s), mouse/mice, etc. A server is preferably apparatus that provides functionality for other computer programs or devices, called “clients.” This architecture is called the client-server model, and a single overall computation is typically distributed across multiple processes or devices. Servers can provide various functionalities, often called “services”, such as sharing data or resources among multiple clients, or performing computation for a client. A single server can serve multiple clients, and a single client can use multiple servers. A client process may run on the same device or may connect over a network to a server on a different device. Typical servers are database servers, file servers, mail servers, print servers, web servers, game servers, application servers, and chat servers. The servers discussed in this specification may include one or more of the above, sharing functionality as appropriate. Client-server systems are most frequently implemented by (and often identified with) the request-response model: a client sends a request to the server, which performs some action and sends a response back to the client, typically with a result or acknowledgement. Designating a computer as “server-class hardware” implies that it is specialized for running servers on it. This often implies that it is more powerful and reliable than standard personal computers, but alternatively, large computing clusters may be composed of many relatively simple, replaceable server components.
[0062] The servers and devices in this specification typically use the one or more processors to run one or more stored “computer programs” and/or non-transitory “computer-readable media” to cause the device and/or server(s) to perform the functions recited herein. The media may include Compact Discs, DVDs, ROM, RAM, solid-state memory, or any other storage device capable of storing the one or more computer programs.
[0063] With reference to FIG. 1a , shown is illustrative of a typical audio conference scenario in the current art, where a remote user 101 is communicating with a shared space conference room 112, for example, via headphone (or speaker and microphone) 102 and computer 104. Room, shared space, environment, free space, conference room and 3D space can be construed to mean the same thing and will be used interchangeably throughout the specification. The purpose of this illustration is to portray a typical audio conference system 110 in the current art in which there is sufficient system complexity due to either room size and/or multiple installed microphones 106 and speakers 105 that the microphone 106 and speaker 105 system may require custom room coverage patterns, configuration setup and zoning configurations. Zoning in this specification is defined as a microphone array's 124 ability to configure a room 112 into defined and discrete areas known as gain and attenuation zones and/or regions for the purpose of prioritizing important areas of the room 112 for sound source 107 pickup and zones that are not prioritized for sound source 107 pickup. Important areas can be for example defined as but not limited to boardroom tables 108, interactive display areas 122, presentation locations (not shown), teacher front of class areas (not shown) and any space 112 where desired sound sources 107 have priority over other sound sources 107 and areas of the room 112. The goal is to have the microphone and speaker bar combination unit 114 only target and focus on desired sound sources 107 in specific areas of the room 112 for optimal benefit of the remote users 101. How zoning is accomplished is very important to the result that the remote user 101 experiences in audio quality and performance at the far end of the conference call. Microphone 106 coverage pattern setup is typically required to support zoning capabilities in all but the simplest audio conference system 110 installations where the microphones 106 are static in location and their coverage patterns limited, well understood and fixed in design such as a simple table-top 108 units and/or as illustrated in FIG. 1b simple wall mounted microphone and speaker bar combination unit 114.
[0064] For clarity purposes, a single remote user 101 is illustrated. However, it should be noted that there may be a plurality of remote users 101 connected to the conference system 110 which can be located anywhere a communication connection 123 is available. The number of remote users is not specifically germane to the preferred embodiment of the invention and is included for the purpose of illustrating the context of how the audio conference system 110 is intended to be used once it has been installed and calibrated. Individual remote users 101 may be on separate streaming channels that would allow for separate in-room 112 ACP zoning profile configurations and would be within scope of the invention as outlined in the structural diagram (FIG. 11d ) and the logic diagrams outlined in FIGS. 12a, 12b and 12c respectively. The room 112 is configured with examples of, but not limited to, ceiling, wall, and desk mounted microphones 106 and examples of, but not limited to, ceiling and wall mounted speakers 105 which are connected to the audio conference system 110 via audio interface connections. In-room participants 107 may be located around a table 108 or moving about the room 112 to interact with various devices such as the touch screen monitor 122. A touch screen/flat screen monitor 122 is located on the long wall. A microphone 106 enabled webcam 109 is located on the wall beside the touch screen 122 aiming towards the in-room participants 107. The microphone 106 enabled web cam 109 is connected to the audio conference system 110 through common industry standard audio/video interfaces. The complete audio conference system 110 as shown is sufficiently complex that a manual setup for the microphone system is most likely required, for example by using computer 103, for the purpose of establishing coverage zone areas between microphones, gain structure and microphone gating levels of the microphones 106, including feedback and echo calibration of the system 110 before it can be used by the participants 107 in the room 112. As the participants 107 move around the room 112, the audio conference system 110 will need to determine the microphone 106 with the best audio pickup performance in real-time and adjust or switch to that microphone 106. Problems can occur when microphone coverage zones overlap between the physically spaced microphones 106. This can create microphone 106 selection confusion especially in systems relying on gain detection and level gate thresholding to determine the most appropriate microphone 106 to activate for the talking participant at any one time during the conference call. Some systems in the current art will try to blend individual microphones through post processing means, which is also a compromise trying to balance the signal levels appropriately across separate microphone elements 106 and can create a comb filtering effect if the microphones 106 are not properly aligned and summed in the time domain. Conference systems 110 that do not have a properly configured and cohesive coverage area including the ability to configure for zone specific prioritizations within the coverage area can never really be optimized for all dynamic situations in the room 112.
[0065] The size, shape, construction materials and the usage scenario of the room 112 dictates situations in which equipment can or cannot be installed in the room 112. In many situations the installer is not able to install the microphone system 106 in optimal locations in the room 112 and compromises must be made. To further complicate the system 110 installation as the room 112 increases in size, an increase in the number of speakers 105 and microphones 106 is typically required to ensure adequate audio pickup and sound coverage throughout the room 112 and thus increases the complexity of the installation, setup, and calibration of the audio conference system 110.
[0066] The speaker system 105 and the microphone system 106 may be installed in any number of locations and anywhere in the room 112. The number of devices 105, 106 required is typically dictated by the size of the room and the specific layout and intended usages. Trying to optimize all devices 105, 106 and specifically the microphones 106 for all potential room scenarios can be problematic.
[0067] It should be noted that microphone 106 and speaker 105 systems can be integrated in the same device such as tabletop devices and/or wall mounted integrated enclosures or any combination thereof and is within the scope of this disclosure as illustrated in FIG. 1 b.
[0068] With reference to FIG. 1b , shown is illustrative of a microphone 106 and speaker 105 bar combination unit 114. It is common for these combination units 114 to contain multiple microphone 106 elements in what is known as a microphone array 124. A microphone array 124 is a method of organizing more than one microphone 106 into a common microphone array 124 of microphones 106 which consists of two or more and most likely five (5) or more physical microphones 106 ganged together to form a microphone array 124 element in the same enclosure 114. The microphone array 124 acts like a single microphone 106 but typically has more gain, wider coverage, fixed or configurable directional coverage patterns to try and optimize microphone 106 pickup in the room 112. It should be noted that a microphone array 124 is not limited to a single enclosure and can be formed out of separately located microphones 106 if the microphone 106 geometry and locations are known, designed for and configured appropriately during the manual installation and calibration process.
[0069] With reference to FIG. 1c , shown is illustrative of the use of microphones 106 and speakers 105 bar combination units (bar units) 114 mounted on separate walls. The location of the bar units 114 for example may be mounted on the same wall, opposite walls or ninety degrees to each other as illustrated. Both bar units 114 contain microphone arrays 124 with their own unique and independent coverage patterns. If the room 112 requirements are sufficiently large, any number of microphone 106 and speaker 105 bar units 114 can be mounted to meet the room 112 coverage needs and is only limited by the specific audio conference system 113 limitations for scalability. This is a typical deployment strategy in the industry and coordination and hand off between the separate microphone array 124 coverage patterns need to be managed and calibrated for, and/or dealt with in firmware to allow the bar units 114 to determine which unit 114 is utilized based on the active speaking participant 107 location in the room, and to automatically switch to the correct bar unit 114. Mounting multiple bar units 114 to increase microphone 106 coverage in larger rooms 112 is common. It should be noted that each microphone array 124 operates independently of each other, as each microphone array 124 is not aware of the other microphone array 124 in any way plus each microphone array 124 has its own specific microphone coverage configuration patterns. The management of multiple microphone arrays 124 is typically performed by a separate system processor 117 and/or DSP module. Because the microphone arrays 124 operate independently the advantage of combining the arrays and creating a single intelligent coverage zoning strategy is not possible.
[0070] For the purpose of this invention, it is assumed that a microphone array 124 is required. A microphone array 124 is defined as a microphone array that provides coverage of a room 112 through the use of virtual microphones. If more than one microphone arrays 124 are installed in the room 112, the microphone arrays 124 can be configured to form a physical combined array as described in U.S. patent application Ser. No. 18/116,632 filed Mar. 2, 2023, and a unified coverage map as described in U.S. patent application Ser. No. 18/124,344 filed Mar. 21, 2023, entire content of which are incorporated herein by reference. It should be noted that multiple microphone arrays 124 are not required to form the one or multiple of coverage zones outlined in the preferred embodiment of the invention and as long as the microphone array 124 is able to instantiate and distribute virtual microphones 304 throughout the room 112 it is considered within scope of supporting the invention.
[0071] With reference to FIG. 2a , shown are representative examples, but not an exhaustive list, of microphone array and microphone speaker bar layouts 114a, 114b, 114c, 114d, 114e, 114f, 114g, 114h, 114i, 114j to demonstrate the types of microphone arrays 124 and speaker 105 arrangements that are supported within the context of the invention. The microphone array 124 and speaker 105 layout configurations are not critical and can be laid out in a linear, offset or any geometric pattern that can be described to a reference set of coordinates within the microphone and speaker bar layouts 114a, 114b, 114c, 114d, 114e, 114f, 114g, 114h, 114i, 114j. It should be noted that certain configurations where microphone elements are closely spaced relative to each other (for example, layouts 114a, 114c, 114e) may require higher sampling rates to provide required accuracy. At low frequencies, the wavelengths of audio signals become much larger. To differentiate between two points of a wavelength, a larger distance is required. Therefore, if a low-frequency wavelength hits two microphones that are very close to each other, they will both show the same data. At higher frequencies, the wavelengths become shorter, and the two near microphones can properly differentiate between two signals. Therefore, in order to get the benefit of having multiple microphones that are very close to each other, higher frequencies must be supported. To support higher frequencies, a higher sampling rate must be used. FIG. 2a also illustrates the different microphone arrangements that are supported within the context of the invention. The microphones may be distributed in a single linear axis 201 or a single plane 202 or any combinations thereof and can be construed to be within scope of the invention.
[0072] FIG. 2b extends the support for the microphone array 124 to various wall mounting scenarios. The microphones 106 can share the same mounting plane and/or be distributed across multiple walls (planes) A, B, C, D or E and be within scope of the invention. This also shows that the microphone plane 202 can be rotated in any axis 203.
[0073] With reference to FIGS. 3a and 3b , shown are descriptive illustrations outlining the distinctive difference in approaches between the directional coverage pattern performance characteristics of a basic beamforming array 308, as outlined in FIG. 3a , to a non-directional full room coverage approach of a microphone array 124, as outlined in FIG. 3b . The microphone array 124 does not utilize or use beamforming algorithms, microphones 106 placement or design principals to create on-axis 302 gain or off-axis 303 rejection polar plot patterns and instead focuses the microphone array 124 to single point in space referred to as a virtual microphone 304, allowing the microphone array 124 to receive sound sources 107 in an omni-directional pattern from that focus point in the 3d space 112. Refer to U.S. Pat. No. 10,063,987.
[0074] The beamformer 308 array in FIG. 3a is representative of the typical on-axis 302 and off-axis 303 coverage pattern characteristics by using a combination of microphone 106 selection, number of microphones 106, placement, and specific spacings in combination with complex post signal processing methods to create a directional coverage pattern with a specific polar response.
[0075] The coverage pattern (polar plot) contains a directional region of maximized sound source 107 pickup referred to as the on-axis 302 gain region and a region of active sound source 107 signal cancellation referred to as the off-axis 303 rejection (attenuation) region. This means that if a sound source 107 is located anywhere in the on-axis region 302, the beamforming array 308 maximizes the gain of the signal. If the sound source 107 is not located in the on-axis region 302 of the array 308, it is by default in the off-axis region 303 of the beamformer array 308. The beamformer array 308 actively cancels the off-axis 303 signal. The beamforming array 308 by design utilizes nulls and aliasing frequencies in combination with signal processing algorithms to obtain the desired polar response with upwards of for example 40 dB or more attenuation in the off-axis 303 region and is well understood in the current art. In practical terms this means that the beamformer 308 is not actively tracking or aware of sound sources 107 in the off-axis 303 region by design. In addition, the beamforming array 308 will typically be subject to lobing regions 307 of unwanted frequency specific gain in the polar plot which can be due to for example, but not limited to frequency specific wavelength issues in combination with microphone 106 spacing, number and placement considerations, which is a by-product of design choices in the beamforming array 308. This is an unwanted gain artifact of beamforming arrays 308 which can create non-linear gain and frequency response issues in the beamforming array 308 for sound sources 107 in the off-axis 303 region of the polar response which is also relative to room 112 placement of beamforming array 308. Lobing artifacts can impact the zoning capabilities of the beamforming array 308 and the ability to create clear and defined regions in the room 112 of desired sound source 107 pick-up verses regions in the room 112 of undesired sound source 107 pick-up.
[0076] The goal is to know when a sound source 107 is in an undesired region of the room 112 and to deal with the undesired sound source 107 in an appropriate manner to maintain the proper gain structure of desired sound sources 107 in the desired region of the room 112 without being influenced or impacted by undesired sound sources 107 in an undefined and unknown manner. A by-product of the typical polar plot of a beamformer array 308 is that it is not designed to look at the whole room 112 equally from a 3D spatial (x, y, z) perspective as any space in the rejection region 303 is simply ignored and effectively cancelled as undesired signals that are outside of the on-axis 302 gain region. The limitations of beamformer arrays 308 become readily apparent when there is a need or requirement to dynamically and intelligently adjust the gain of and track the location (x, y, z) of sound sources 107 that are outside of the on-axis region 302. By design beamformers 308 are implemented to maximize the gain of a sound source 107 in the beam (on-axis 302 region) and reject reflections and other sound sources 107 and noises outside of the beam. In effect the beamforming array 308 is designed to maximally reject off axis signals 303 and maximize on-axis 302 signals right at the beamformer array 308 thus eliminating the possibility of a beamformer array 308 to have awareness of sound sources 107 in the off-axis 303 region of the polar plot. Adding additional beamformer arrays 308 to create full room 112 coverage or adding additional on-axis 302 lobes is expensive and complex and does not change the impact of off-axis 303 lobing issues. Sound sources 107 located in each zone or transitioning between undesired and desired zone areas of the room 112 will still be impacted by the characteristics of the off-axis 303 rejection of the beamformer 308. Creating additional on-axis 302 regions/zones to obtain awareness of and gain information about sound sources 107 in undesired zones would not be an effective solution. The gain structure in each on-axis 302 lobe/region is independent of sound sources 107 outside of the on-axis regions 302. So sound sources 107 that are not static in location and move around the room 112 cannot be managed effectively with respect to leaving and entering the on-axis 302 regions creating abrupt audio transitions that are unpleasant to listen to at the far end of call for the remote users 101.
[0077] FIG. 3b illustrates an approach to room 112 coverage that does not have the same limitations of the current art of beamforming, by continuously monitoring the whole room 112 without the constraints of on-axis 302 and off-axis 303 polar responses, and utilizes the preferred embodiment outlined in this specification referred to as automatic zoning gain control. Automatic zoning gain control works by preferably defining inclusion zones 305 and exclusion zones 306 independent of the microphone array 124 where the sound source 107 gain structures can be managed across multiple zones of any type and multiple sound sources 107 throughout the whole room 112 resulting in intelligent and predictable zone-specific gain control. Smoother transitions for sound sources 107 moving between desired (inclusion zone 305) to undesired (exclusion zone) 306, and optimum gain management of sound sources 107 within each inclusion zone (IZ) 305 and exclusion zone (EZ) 306 is obtained because their specific location (x, y, z) and signal levels are known and continuously tracked.
[0078] The microphone array 124 as installed into a typical room 112 preferably covers the whole room 112 with 1000's of virtual microphones 304 evenly distributed throughout the room 112. The virtual microphones 304 in this example completely fill the room 112 in all three dimensions (x, y, z). However, the size and the shape of the overall virtual microphone 304 grid is a configurable set of parameters that can be preferably defined in the x, y, z coordinate space 112 allowing for partial to preferably complete room 112 coverage. Once a virtual microphone 304 is focused on by the microphone array 124, the frequency response and gain of the array is linear and consistent. This applies to all virtual microphones 304 positions in the defined coverage map across the full room 112. Because the virtual microphones 304 are distributed through the room 112 and always available, each virtual microphone 304 can be monitored continuously to provide defined parameters such as for example but not limited to (on/off and signal power). In this preferred example of the invention, an inclusion zone 305 and an exclusion zone 306 have been configured within the virtual microphone 304 grid by grouping all the available virtual microphones 304 based on their location into the room 112 into either an inclusion zone 305 or an exclusion zone 306. The inclusion zone 305 is a zone where positive gain structure is applied to targeted sound sources 107 while in the exclusion zone 306 a negative gain structure is applied to targeted sound sources 107 identified in this zone. The automatic zone gain control processor 1150 as defined in FIG. 11e uses the signal power level of each sound source 107 target in each zone and derives a specific gain structure, for the targeted inclusion zone 305 gain source as outlined in FIG. 11d . The shapes of the IZ 305 and EZ 306 are independent of the microphone array 124 structure and can be disassociated, shaped, and configured in any manner or size that contains a full or subset of virtual microphone 304 distributions. The benefit of this preferred embodiment of the invention that defines an inclusion zones 305 and exclusion zones 306 within the full virtual microphone grid is that the microphone array 124 can monitor all sound sources 107 in the 3D space 112 and make the appropriate targeted sound source 107 gain structure (automatic zone gain control) decisions based on the sound source 107 location relative to the zone type they are in or moving between in the room 112 and overcoming the limitations present in beamformer array 308 implementations.
[0079] With reference to FIGS. 4a, 4b and 4c , shown are illustrative current art examples outlining beamformer 308 coverage patterns in a typical room 112 where there may be more than one region/lobe configured to provide coverage and on-axis gain 302 to certain areas of the room 112. FIG. 4a illustrates a room 112 configured with a boardroom table 108 and a beamformer array 308 suspended from the ceiling. To provide independent coverage areas four on-axis 302 regions BF GZ1, BF GZ2, BF GZ3, and BF GZ4 have been configured. The gain structure in each on-axis 302 region can be configured manually and automatic gain control (AGC) is typically applied to sound sources 107 within the on-axis 302 regions. What is not illustrated is the off-axis 303 lobing 307 that overlap each adjacent on-axis 302 region. The combined off-axis 303 region has been configured to be any space outside of the boardroom table 108. The beamformer array 308 will actively reject any sound sources 403 in the off-axis 303 region by design at the beamformer array 308. The nature of off-axis 303 rejection means the beamformer array 308 is typically unaware of, or able to do any dynamic gain adjustments to the sound sources 107a and 107b in the on-axis 302 regions based on off-axis 303 sound source 107c levels. The implications are that if sound source 107c is in close proximity to the on-axis 302 region it may be rejected entirely which may or may not be desirable or it may cause ingress issues causing the gain to fluctuate in the on-axis 302 zones in an unpredictable manner. If the sound source 107c located in the off-axis 303 region is loud enough the beamformer array 308 will not be able to adjust the adjacent on-axis 302 regions gain structure in an intelligent manner because it is not able to locate and determine if the sound source 107c is from a valid location to apply gain to. This could cause the gain in the on-axis 302 region to track and fluctuate based on sound source 107c which would be undesirable behavior resulting in poor performance at the remote end 101 of the call. Specifically, the beamformer array 308 is not able to locate and manage sound source 107c in the off-axis 303 region because it has no active coverage in that part of the room 112. The beamformer 308 derives its gain structure determinations based on on-axis 302 sound sources 107a, 107b only. Additional beamformers 308 can be added in an attempt to overcome this limitation by adding more regions/lobes 302 to create specific zones with configurable parameters, however the same limitations apply as there will typically be off-axis 303 regions where the beamformer 308 is not able to track and monitor sound sources 107 that could impact the on-axis 302 regions in unpredictable ways. Therefore, it is preferable to have a microphone array that can continually monitor the full room utilizing zone-based coverage configurations to intelligently optimize the audio quality for desired sound sources 107a, 107b in one section of the room 112 and to intelligently ignore undesired 107c sound sources based on their location in the room 112.
[0080] FIG. 4b illustrates the same room configured with a beamformer 308 that uses a beam-tracking approach. The on-axis 302 region can appear to be square shaped compared to the typical lobing patterns as seen in FIG. 3a . The same limitations apply though in that if a sound source 107c is not located in the specific on-axis 302 coverage area it is by default in the off-axis 303 region and is hidden from the beamformer 308. So, gain structure determinations in the on-axis 302 region are made based solely on sound sources 107a, 107b within the on-axis 302 region. If a sound source 107a, 107b, 107c traverses between on-axis 302 and off-axis 303 regions the transition can be abrupt and unpleasant with sudden gain shifts in levels as the AGC tries to compensate at the remote 101 end of the call. If a sound source walks 107a, 107b, 107c just outside the edge of the on-axis 302 region their sound level can be unpredictable and potentially abruptly attenuated due to off-axis 303 rejection causing undesired effects for the remote user 101 of the conference call.
[0081] FIG. 4c illustrates the same beam tracking beamformer 308 configured to supply two independent zones with on-axis 302 regions BF Z1, BF Z2 of coverage that are dissociated from each other. It should be noted that this represents a floor plan of the coverage. With a beamformer microphone array 308, all beams originate from the center of the aperture formed by the physical microphones in the array. In this case, the beamformer 308 is mounted on the ceiling of the room and so the beams covering BF Z2 must be connected to 308 at a point on the ceiling. Therefore, although BF Z2 is shown on the right side of the room and dissociated from beamformer 308 at the floor level, the beam is connected at the ceiling level. This means the coverage range of BF Z2 changes based on the height. This limitation of beamformer arrays means that coverage zones such as BF Z2 that are further away from the beamformer 308 can result in undesired behavior as the height of the participant 107d changes. This arrangement shares the same limitations as previous examples in the desired on-axis 302 regions for static non-moving sound sources 107a, 107b, 107d however when the sound sources 107c traverses between on-axis 302 regions BF Z1 and BF Z2 into the off-axis 303 rejection region there will not be a smooth transition between the on-axis 302 regions with potentially abrupt signal degradation and possible total loss of the sound source 107c which would be undesirable. The sound source 107c in the off-axis 303 region will be actively rejected by the beamformer array 308 creating abrupt loss of sound at the remote end of the call. It should be noted that to create a zone with a beamformer array 308 either an individual on-axis region 302 is configured or groups of on-axis regions 302 are grouped together to form an area of desired on-axis 302 gain in the room 112. Multiple on-axis 302 regions may be possible within the limits of available microphones 106 and processing capability. In either scenario there will be areas of off-axis 303 response within the system and room 112 unless an on-axis 302 region is configured to cover the whole space 112 at which point there is no longer separate zones (regions) for the purpose of creating desired and undesired areas of pickup in the room 112 and the beamformer array 308 operates per normal behavior.
[0082] A preferable approach would be to establish desired and undesired zones that remain active where sound sources 107 can be tracked throughout the complete room 112 for the purpose of managing the gain of the sound sources 403 at the edges of the on-axis 302 zones BF Z1 and BF Z2 based on their position within the off-axis 303 regions in an intelligent manner creating the best experience for the remote users 101.
[0083] With reference to FIGS. 5a and 5b , shown are illustrative examples of a 2D and 3D microphone 106 arrangements illustrating the effective impact on virtual microphone 304 shape, size and coverage pattern dispersion of the virtual microphones 304 and mirrored virtual microphones 501 (2D array) in a space 112. For details of how virtual microphones are formed and positioned in the 3D space 112, refer to U.S. Pat. No. 10,063,987. And for forming a combined array from ad-hoc arrays and discrete microphones, refer to U.S. patent application Ser. No. 18/116,632 filed Mar. 2, 2023.
[0084] FIG. 5a is an illustrative diagram of the virtual microphone 304 shape that is formed from a microphone array 124 of microphones 106 and the distribution of the virtual microphones 304 along the mounting axis of the microphone array 124. Each virtual microphone 304 is drawn as a circle (bubble) to illustrate its relative position to the microphone array 124. The number of virtual microphones 304 that can be created is a direct function of the setup and hardware limitations of the system processor 117. In the case of a microphone array 124 arrangement the virtual microphone 304 cannot be resolved specifically to a point in space and instead is represented as a toroid in the 3D space. The toroid 502 is centered on the microphone axis 201 which is the same as the X axis in this configuration as illustrated in the side view illustration. The effect of this virtual microphone 304 toroid shape 502 is that there are always many points within the toroid 502 geometry and will be seen as equal and cannot be differentiated. The impact of this is a real virtual microphone 304 and a mirrored virtual microphone 501 on the same plane. Due to this toroid geometry, the virtual microphones 304 cannot differentiate between spots in the z-axis. Therefore, the virtual microphones 304 are aligned in a single x-y plane. Allocating individual virtual microphones 304 in the z-dimension is not possible due to symmetry imposed by the microphone array 124 configuration. Note that each toroid will intersect with the x-y plane in two different spots. One of these is the true desired virtual microphone 304 location and the other is a mirrored location 501 at the same distance on the opposite side of the microphone array 124. The microphone array 124 cannot distinguish between the two virtual microphone 304, 501 positions (or any along the path of the toroid). As a result of this, it is a recommended constraint that a microphone array 124 arrangement be positioned on a solid boundary layer such as wall or ceiling so the mirrored virtual microphone 501 can be ignored as sound behind the boundary (wall). Using this mounting constraint, any sound source 107 found by the microphone array 124 will be considered to be in the room 112 in front of the front wall.
[0085] The geometric layout of the virtual microphones 304 will be equally represented in the mirrored virtual microphone plane 501 behind the wall. The virtual microphone distribution geometries are symmetrical as represented by front of wall and behind the wall. The number of virtual microphones 304 can be configured to the y-axis dimensions, front of wall depth and the horizontal-axis, width across the front of wall. As stated previously, the same dimensions will be mirrored 501 behind the wall. For example, the y-axis coverage pattern configuration limit will be equally mirrored behind the wall in the y-axis in the opposite direction. The z-axis cannot be configured due to the toroid 502 shape of the virtual microphone geometry. Put another way the number of virtual microphones 304 can be configured in the y-axis and x-axis but not in the z-axis for the microphone array 124 arrangement. As mentioned previously the microphone array 124 arrangement is well suited to a boundary mounting scenario where the mirrored virtual microphones 501 can be ignored and the z-axis is not critical for the function of the microphone array 124 in the room 112. The preferred embodiment of the invention can position the virtual microphone 304 map in relative position to the microphone array 124 orientation and can be configured to constrain the width (x-axis) and depth (y-axis) of the virtual microphone 304 map if the room boundary dimensions are known relative to the microphone array 124 position in the room 112.
[0086] FIG. 5b is an illustrative example of two microphone arrays 124 arranged to form a multiplane arrangement of microphones 106 resulting in a virtual microphone 304 distribution that is not mirrored on either side of the microphone arrays 124 nor is it rotated around the microphone array 124 forming a toroid 502 shape. The multiplane 203 arrangement is the most preferable microphone 106 arrangement as it affords the most configuration flexibility in the x-axis, y-axis and z-axis and eliminates the mirrored virtual microphone 501 geometry. This means that although the microphones 106 are illustrated as being shown as mounted to a boundary they are not constrained to a boundary mounting location and can be offset, suspended and/or even table mounted, and optimal performance is maintained as there is no mirrored virtual microphones 501 to be accounted for. As per the microphone array 124 arrangement all virtual microphones 304 are considered to be a point source in space.
[0087] For simplicity the illustration of the multiplane arrangement is shown as cubic however it is not constrained to a cubic geometry for virtual microphone 304 coverage map form factor and instead is meant to represent that the virtual microphones 304 are not distributed on an axis or a plane and thus incurring the limitations of those geometries. The virtual microphones 304 can be distributed in any geometry and pattern supported by the hardware and mounting locations of the individual microphone arrays 124 or within the combined array and be considered within the scope of the invention.
[0088] With reference to FIG. 6 , shown is a diagrammatic illustration of zoning shapes that are considered within the scope of the invention. Various zone shapes, whether they be IZ 305 or EZ 306 defined can be configured by combining any number of virtual microphones 304 in an ACP configuration. Geometric shapes such as but not limited to triangular 603, spherical 606, elliptical 601, cubic 604, rectangular 602 or 605 and point 609 are all readily possible including non-geometric shapes. Planar 2D (x, y) and 3D (x, y, z) zones are configurable by combining the appropriate location and number of specific virtual microphones 304 distributed throughout the room 112. A single virtual microphone 304 can be configured as desired to IZ 305 or EZ 306 of any type for the maximum spatial granularity. Any number of inclusion 305 or exclusion zones 306 can be created and configured and is only limited by physical system resources and the hardware allocated to the implementation.
[0089] With reference to FIGS. 7a and 7b , illustrated are examples of a preferred embodiment of the invention as it pertains to the demarcation of zones in 2D and 3D layouts. FIG. 7a is a top-down view of the room 112. Illustrated are examples of but not limited to an inclusion zone 305, exclusion zone 306 and undefined zone 710 ACP configuration. The preferred embodiment of the invention is to not assign any undefined zones 710 and instead use exclusion zones 306 to define low priority areas in the room. Any number of each zone type is supported and to be considered in scope of the invention, only limited by the availability of unassigned/unallocated virtual microphones 304 within the space 112. FIG. 7b is an ACP configuration where the inclusion zone 305 in defined as a cubic rectangle shape disassociated from the microphone array 124 (not shown) in the geometric middle space of the room 112. An undefined zone 710 is defined at the very top of the room 112. The exclusion zone 306 is defined to include all other virtual microphones 304 not contained in the inclusion zone 305 and undefined zone 710.
[0090] With reference to FIG. 8 , shown is a current art illustration of a typical virtual microphone speaker array 114 installation in a room 112 without zoning enabled, meaning all virtual microphones 304 are without a zone configuration. The virtual microphone map 801 has been configured to fill approximately 80% of the space 112. Individual virtual microphones 304 are not shown for clarity purposes however it would be considered typical and preferable that 1000's of virtual microphones 304 are distributed evenly throughout the coverage grid 801. In this configuration scenario, if a sound source is within the virtual microphone 304 coverage grid 801 it will be treated equally by the virtual microphone array 114 which will locate the sound source and focus on it applying the same gain rules for each virtual microphone 304 regardless of its location in the room 112 within the coverage grid 801.
[0091] With reference to FIGS. 9a, 9b, 9c, 9d, 9e, 9f and 9h , shown are illustrations of exemplary embodiments outlining a top-down perspective of the room 112 with various ACP configurations of IZ 305 and EZ 306. The illustrations are depicted as top down for clarity purposes, and the zones can be distributed and shaped in 2D and 3D axes geometries based on the microphone array 124 geometry as shown in FIGS. 5a and 5b , and are considered to be within scope of the invention.
[0092] In FIG. 9a the microphone array 124 distributes the virtual microphones 304 throughout the whole room 112 to the wall boundaries of the room 112. The virtual microphone 304 grid has been configured for one inclusion zone 305 and a surrounding exclusion 306 zone. As stated previously in the specification the virtual microphones 304 are distributed throughout both zones IZ 305 and EZ 306 maintaining full room coverage for sound source monitoring and targeting. A single sound source target 902 is located within the inclusion zone 305 and will be defined as a Gain Source (GS) 1139. The other active sound source 901 is located in the exclusion zone 306 and will be defined as an Attenuation Source (AS) 1201. It should be noted that each sound source 901, 902 are directly tied to the closest virtual microphone 304 to their location within the coverage grid of preferably 1000's of distributed virtual microphones 304 in the 3D space 112. Since all the virtual microphones 304 are available in this coverage grid, the monitoring, targeting and intelligent gain management of any virtual microphone 304 can occur based on the specific algorithms described in the zoning processor 1150 based on the GS 1139 and AS 1201 virtual microphones 304 location in the inclusion zone 305 and exclusion zone 306 as per an exemplary embodiment of the invention. Unlike beamformer arrays 308 with on-axis 302 gain and off-axis 303 rejection which treats sound sources based solely on their position in the polar response of the beamforming array 308, the microphone array 124 is able to advantageously track and adjust the gain for the GS 1137 and AS 1201 targeted sound sources within the room 112.
[0093] FIG. 9b further illustrates another preferred embodiment of the invention by showing the ability to create an inclusion zone 305 that is dissociated from the front plane of the microphone arrays 124. Multiple sound source targets 901a-901b, 902a-902c are tracked and processed by the zoning processor 1150. The zoning processor 1150 continually looks at all active sound sources 901a-901b, 902a-902c and makes the appropriate gain-processing decisions based on its specific targeting configuration to maximize audio pickup and performance of the selected GS 1137 target. The inclusions zone 305 is rectangular in shape and contains three active targets 902a-902c within the IZ 305. As per FIG. 9b each potential sound source (GS) target 902a-c is associated directly with a virtual microphone 304 located at that specific 3D location in the room 112. Because the three potential GS 1137 targets 902a-902c are within the inclusion zone 305 they could be selected as a GS 1137 by the zoning processor 1150. The selected GS 1137 will be output via the audio processor 1103. The two sound sources 901a-901b located in the exclusion zone 306 will be added to the AS list 1139 by the zoning processor 1150 and be used in the zoning gain calculations for the selected GS 1137 target either 902a, or 902b or 902c.
[0094] FIG. 9c further extends the embodiment to illustrate the configuration with two active inclusion zones 305 IZ1 and IZ2 and one overall exclusion zone 306 EZ1. Each inclusion zone 305 contains one potential active sound source target 902a and 902b. The exclusion zone 306 contains one active sound source target 901. The zoning processor 1150 will prioritize the appropriate sound source 902a or 902b based on the configuration parameters loaded into the zoning processor 1150. Whichever target 902a or 902b gets selected in the inclusion zone 305 will be gain structured in conjunction with the sound source target 901 in the exclusion zone 306. Either sound source 902a or 902b can be selected and made the GS 1137 based on the targeting parameters of the zoning processor 1150. There is no requirement for the sound sources 901, 902a and 902b to be static in position as the zoning processor 1150 will dynamically track and adapt to any new positional coordinates of all sound source targets 901, 902a and 902b in real-time and adjust the desired active sound source gain of the GS 1137 at either 902a or 902b accordingly. Any number of inclusion 305 and exclusion 306 zones can be configured for an ACP and be considered within scope of the invention.
[0095] FIG. 9d further illustrates an exemplary embodiment of the invention demonstrating two inclusion zones 305 IZ1 and IZ2 configured with one exclusion zone 306 allocated to the remainder of the virtual microphone 304 grid distributed throughout the whole room 112. The inclusion zone 305 IZ1 in this example illustrates an example of creating a spherical zone 606 shape (see FIG. 6 ) that would be considered not feasible in the current art. To further clarify the zone inclusion 305 zone shape in this example is not meant to illustrate a 2D planform view as seen top down but is instead meant to represent a 3D spherical zone 606 shape placed in the middle of the room 112. In practice, as stated in FIG. 6 of the specification any shape 2D or 3D is supported that can contain any number of virtual microphones 304 and be considered within scope of the invention. The inclusion zone 305 IZ1 contains two sound source targets 902a and 902b. Inclusion zone 305 IZ2 contains one sound source target 902c. The exclusion zone 306 contains three sounds source targets 901a-901c which will be added to the AS list 1139. The sound sources in the AS list 1139 will be used in the processing calculations by the zoning processor 1150 to set the gain structure of the selected active sound source target either 902a or 902b or 902c which is set as the GS 1137. This is a dynamic situation accounting for changes in AS list 1139 parameters and GS 1137 parameters in real-time allowing for the audio processor 1105 to take into account changes for all sound source targets 901a-901c, 902a-902c within the room 112. Although six sound source 901a-901c, and 902a-902c targets are illustrated any number of sound source targets can be tracked in real-time by the zoning processor 1150 because of the distribution of virtual microphones 304 throughout the room 112.
[0096] FIG. 9e is yet another example of a zone shape supported by the invention. A hexagonal inclusion zone 305 IZ1 shape which includes two potential gain source targets 902a-902b and an inclusion zone 305 IZ2 which contains one potential GS 1137 target 902c have been configured. The exclusion zone 306 contains a total of four sound source targets 901a-d to be tracked and added to the AS list 1139.
[0097] FIG. 9f illustrates a more complex zone configuration supported by the invention. Two inclusion zones 305 IZ1 and IZ2 each containing one potential GS target 1137 902a and 902b. Two exclusion zones 306 EZ1 and EZ2 are configured in the room 112 containing when active sound source targets 901a-901d to be tracked and added to the AS list 1139. Exclusion zone 306 EZ1 is contained within the boundaries of an inclusion zone 305 IZ1 demonstrating the ability to support zone-in-zone capabilities. The zone-in-zone capability is possible due to the distribution of virtual microphones 304 throughout the coverage grid. Any virtual microphone 304 is available to be assigned to an inclusion zone 305, or an exclusion zone 306. The ability to configure any number of zones and zone types allows the audio conference system to handle complex environments where inclusion zones 305 can be placed in optimal high priority areas for desired sound source pickup while still being able to configure preferably one or more exclusion zones 306 to deprioritize undesired sound sources 901a-901d throughout the room 112. An example of this is the configuration of exclusion zone 306 EZ2 in the middle of the table 108. The microphone array 124 can be prevented from assigning sound sources 901a and 901b in the EZ1 region as a GS 1137 while calculating the best gain structure for the sound sources 902a and 902b in the inclusion zones 305 IZ1 or IZ2.
[0098] FIG. 9g illustrates the installation of a third microphone array 124 into the room 112. The invention is not limited to the number of or require a certain number of microphone arrays 124 to support the invention. A single microphone array or a plurality of microphone arrays 124 is supported and considered within scope of the invention. The number of microphone arrays 124 installed is determined by the room 112 dimensions and coverage requirements. The number of potential sound source targets 901a-901e, 902a-902d has been increased illustrating the ability of the target processor 1102 to monitor and track any number of sound sources in the room 112 in real-time.
[0099] FIG. 9h illustrates an example of a small exclusion zone 306 EZ1 and a much larger inclusion zone 305 IZ1. It may be desirable to create an exclusion zone 306 around a known static undesired sound source such as but not limited to an HVAC, fans or other intrusive sound sources. This prevents the targeting processor 1102 from targeting sound sources in the EZ1 zone 306 and assigning them as a GS 1137. The microphone array 124 will still be aware of and monitor sound sources that enter the EZ1 exclusion zone 306 and be able to manage the gain of the currently active sound source target 902a-b accordingly.
[0100] With reference to FIG. 10 , shown is a illustrative of a preferred embodiment of the invention that demonstrates the behavior of the zoning processor 1150 when sound sources 107a and 107b move between the configured inclusion 305 and exclusion 306 zones. The zoning processor 1150 initially will be able to select between the following potential sound source 107c and 107a targets 902 and 1001a to assign as a GS 1137 and will have sound source 107b target 1002a in the AS source list 1139. This is because initially two sound sources 107c mapped to target 902 and 107a mapped to target 1001a are located in the inclusion zone 305 and sound source 107b is mapped to 1002a and added to the AS List 1139. Assuming sound source 107a is active it will be the selected target by the targeting processor 1102 and the microphone array 124 will continue to focus on sound source 107a until they leave the inclusion zone 305. At which point the sound source 107c target 902 will be the only available sound source 107c target 902 for the microphone array 124 to focus on and assign as a GS 1137. At any point in the following sequence, sound source 107c target 902 can be selected by the zoning processor 1150 as a potential GS 1137 target because it is always within the inclusion zone 305. At the start of the sequence only actively transmitting/speaking sound sources 107c and 107a can be a selected as a GS 1137, unless there is no one speaking at the time at which point the zoning processor 1150 will default to the last virtual microphone 304 or the virtual microphone 304 with the largest ambient gain value as the GS 1137. The exact same logic is applied to the exclusion zone 306 in that the AS list 1139 target will be defaulted to the last known virtual microphone 304 in the exclusion zone 306 or to the virtual microphone 304 with the largest ambient noise gain value and defined as the AS 1201. If both sound sources 107c and 107a are emitting sound the zoning processor 1150 will select the appropriate sound source target 902 or 1001a in real-time based on the selection logic.
[0101] Sound source 107a will be tracked by the targeting processor 1102 as long as the sound source 107a is emitting sound. If sound source 107c is not emitting sound and the sound source 107a is emitting sound while moving to target location 1001b the targeting processor 1102 will be bounded at target location 1001a by the edge boundary of the inclusion zone 305 at which point the virtual microphone 304 location will be locked at sound source target 1001a until sound source 107a stops talking, or if sound source 107c starts actively talking taking the focus away from sound source 107a. If sound source 107c does not actively talk the gain structure of sound source 107a will be attenuated, according to the algorithms outlined in FIGS. 12a-12d descriptions. This ensures a smooth transition between inclusion 305 and exclusion zones 306 unlike the typical abrupt and sharp attenuation caused by beamformer arrays 308 off-axis rejection 303 resulting in unpleasant artifacts at the far end of conference call such as but not limited to varying audio levels, abrupt loss of talkers and potential noise floor pumping as the AGC adjusts to the sudden change in the audio signal level and quality. By tracking all sound sources 107a, 107b and 107c in the room 112 regardless of the zone 305, 306 they are in a logical, smooth and planned transition between IZ 305 and EZ 306 zones for sound sources 107a, 107b, or 107c can be managed by focusing the microphone array 124 on the appropriate active sound source 107a, 107b, or 107c resulting in a predictable, smooth and stable audio transition between inclusion 305 and exclusion 306 zones.
[0102] The same logic of the preferred embodiment is applied to sound sources such as 107b that starts off located in an exclusion zone 306 and moves into an inclusion zone 305. If no other sound sources 107c and 107a are actively talking in the inclusion zone 305 and the sound source 107b at sound source target location 1002a starts to actively talk in the exclusion zone 306. The targeting processor 1150 will prioritize the virtual microphone 304 at target location 1002b which is the virtual microphone 304 now assigned as a GS 1137 with the best signal performance at the edge of the inclusion zone 305. The microphone array 124 will be focused on the virtual microphone 304 at target location 1002b and the gain structure will be set by the zoning processor 1150. As long as no other sound sources 107c or 107a become active in the inclusion zone 305 while the sound source 107b is actively talking in the exclusion zone 306 the zoning processor maintains the sound source 107b in the AS list 1139 and will adapt the gain structure of the virtual microphone 304 at target location 1002b accordingly. Once the active sound source 107b enters the inclusion zone 305 it will be managed as an inclusion zone 305 GS 1137 by the zoning processor 1150. Sound sources entering or leaving the inclusion 305 zone while actively talking are tracked and can be assigned as the active GS 1137, edge boundary target, are added to the appropriate AS list 1139 if they enter the exclusion zone 306 and managed by the zoning processor 1150 to ensure smooth audio transition performance between zones 305, 306. Zone based gain control effectively overcomes the limitation in the current art by eliminating the hash and potentially abrupt transition caused between on-axis 302 and off-axis 303 performance typical of a beamformer array 308. At any point, any actively talking sound source in the inclusion zone 305 such as sound source 107c will have priority over any sound source in the exclusion 306 zone.
[0103] With reference to FIG. 11a , shown is a block diagram showing a subset of high-level system components related to a preferred embodiment of the invention. The three major processing blocks are the Array Configuration and Calibration 1101, the Targeting Processor 1102, and Audio Processor 1103. The Array Configuration and Calibration 1101 uses configuration constraints 1120 to find the location of all physical microphones 106 in the system by injecting a known signal 1119 to the speakers 105 and measuring the delays to each microphone 106. This process is described in more detail in U.S. patent application Ser. No. 18/116,632 filed Mar. 2, 2023. Once the location of all physical microphones 106 has been determined, the next step is to create coverage zone dimensions and populate the coverage zone dimensions with virtual microphones 304. Herein, populating the coverage zone dimensions with the virtual microphones includes densely or non-densely (or sparsely) filling the coverage zone dimensions with the virtual microphones and uniformly or non-uniformly placing the virtual microphones in the coverage zone dimensions. Any number of virtual microphones can be contained in the coverage zone dimensions. This process is described in more detail in U.S. patent application Ser. No. 18/124,344 filed Mar. 21, 2023. The results of the Array Configuration and Calibration 1101 are the physical locations of the physical microphones 106 and virtual microphones 304 and the corresponding weights and delays of the physical microphones 106 associated with all virtual microphones 304 in the system. These results are then passed to the Targeting Processor 1102 and the Audio Processor 1103 through 1122 and 1116 respectively. The Targeting Processor block 1102 uses the delays 1122 to identify sound source attributes 1111 from the virtual microphones 304 as described in FIG. 11b . The resulting real-time location results from the Targeting Processor 1102 are sent to the Audio Processor 1103. The invention described herein involves the Audio Processor block 1103 which uses the sound source attributes 1111 discovered by the Targeting Processor 1102 to time-align the microphones signals and combine them in the correct way in generate the intended audio signals 1144 to be sent out from the audio interface 1145 as described in FIG. 11 d.
[0104] FIG. 11b describes the target processor 1102. A sound source is picked up by a microphone array 124 of many (M) physical microphones 106. The microphone signals 1118 are inputs to the mic element processors 1101 as described in FIG. 11c . This returns an N*M*Time 3D array of each 2D mic element processor output 1120 that then sums all (M) microphones 106 for each virtual microphone 304 n=1 . . . N in 1104. This is a sum of sound pressure that is then converted to power in 1105 by squaring each sample. The power signals are then preferably summed over a given time window such as 50-100 ms by the N accumulators at node 1107. The sum represents the signal energy over that given time period. The processing gain for each virtual microphone 304 is preferably calculated at node 1108 by dividing the energy of each virtual microphone 304 by the energy of an ideal unfocused signal 1122. The unfocused signal energy is preferably calculated by summing in 1119 the energies of each microphone signal 1118 over the given time window, weighted by the maximum ratio combining weight squared. This is the energy that we would expect if all the signals were uncorrelated. The processing gain 1108 is then preferably calculated for each virtual microphone 304 by dividing the microphone array 124 signal energy by the unfocused signal energy 1122. Node 1106 as described in FIG. 12a searches through the 1D array of processing gain 1121 to find all current sound sources 1140. This will contain a number of sound sources from 1 to S, with S corresponding to the maximum number of sound sources that can be tracked by the system.
[0105] FIG. 11c shows the Mic Element Processor 1101. Individual microphone signals 1118 are passed through a precondition process 1117 that can filter off undesired frequencies such as frequencies below 100 Hz that are not found in typical voicebands from the signal before being stored in a delay line 1111. The Mic Element Processor 1101 uses the delay 1112 and weight 1114 from each virtual microphone 304 (n) to create the N*Time 2D output array 1120. Each entry is created by multiplying the delayed microphone by the weight in 1123. The weight and delay of each entry are based on the bubble position 1115 and the delay 1116 from the microphone 106 to that virtual microphone 304. The position of all N virtual microphones 304 gets filled by the Bubble Map Positioner Processor 1121 based on the location of the available physical microphones 106 as described in U.S. patent application Ser. No. 18/124,344 filed Mar. 21, 2023.
[0106] One embodiment may comprise the processor described and depicted in U.S. Pat. No. 10,063,987, the entire contents of which are incorporated herein by reference.
[0107] FIG. 11d shows an example configuration of the Audio Processor 1103 as described in FIG. 11a . Here, microphone array devices 124a, 124b, and 124c (comprising a plurality of microphones 106) and microphone 106a represent the combined microphone array found by the Array Configuration and Calibration block 1101. The signals 1118 from this combined mic array are used by both the Target Processor 1102 and the Audio Processor 1103. For the Audio Processor 1103, the individual raw mic signals 1118 are first preferably processed for example but not limited to remove noise, reverberation, and echo in block 1142. This creates the processed audio streams 1138 that are used by the multipliers 1125. Note that some or all of this processing in the Audio Processor 1103 may also be optionally applied to the audio streams 1141 that are used by the Target Processor 1102. Doing so can help the Target Processor 1102 to focus on desired sound sources such as Participants 107 instead of undesired sources such as coherent noise sources or residual echo signals. Alternatively, the raw microphone signals 1118 could be used by both the multipliers 1125 and the Target Processor 1102 and the resulting combined microphone stream could later be subjected to the processing described in 1142.
[0108] The Target Processor 1102 utilizing the Microphone Array signals 1141 preferably determines the substantially exact positional location (X, Y, Z) coordinates of the sound sources 1140 with the highest processing gain. This is passed in as input to the Zoning Processor 1150 described in FIG. 11e . Each Audio Channel Profile 1126 has its own Zoning Processor 1150 which determines the location of the gain source 1137 that affects the weights 1124 and delays 1128 that the Gain Weight Processor 1149 and Delay Processor 1123 use to align the microphone signals 1138 to a constant time 1132 and sum them in 1133 to produce the combined mic signal 1143 aligned at time 1132 and the resulting ACP audio signal 1144. Note that the Gain Weight Processor 1149 also has access to the physical location of the microphones 1116 determined by the Array Configuration and Calibration block 1101. The process of aligning microphones with the correct weights 1124 and delays 1128 based on a physical location 1137 is described in more details in U.S. patent application Ser. No. 18/126,739, filed Mar. 27, 2023. The Audio Processor 1103 contains at least one but potentially multiple ACPs 1126, each of which is used to produce an output signal 1144. The signal or signals 1144 produced by the Audio Processor 1103 will all get sent out of the system through the Audio Interface 1145 described in FIG. 11a . The zoning gain 1147 found by the Zoning Processor 1150 is multiplied by the combined mic signal 1143 in an element 1148 to create the output channel 1144 for the current ACP. This process is repeated for all ACPs to generate all outputs 1144 of the Audio Processor 1103.
[0109] FIG. 11e represents the Zoning Processor 1150. This takes in as input the list of sound sources 1140 preferably discovered by the Target Processor 1102. The Zoning Processor contains an Active Zone Configuration 1127 for the ACP 1126 that the Zoning Processor 1150 belongs to. The sound source list 1140 is passed in as input to the Sound Source Allocation Block 1136 along with the inclusion 305 and exclusion 306 zone parameters 1129 that represent the physical boundaries of the Active Zone Configuration Parameters, the weight of the inclusion zones 305 WZ and the maximum number of attenuation sources that can be allocated for the current ACP 1126. The Sound Source Allocation block 1136 finds the Gain Source 1137 and the Attenuation Source List 1139 as described in FIG. 12a . The Calculate Zoning Ratio Block 1135 as described in FIG. 12b uses the GS 1137 and AS list 1139, along with the Pmin, YGZ and YAZ Active Zone Configuration parameters 1130 for the current ACP to determine the zoning ratio r 1146. The zoning ratio r, along with the GmaxZ and GminZ parameters 1131 of the Active Zone Configuration 1127 are used by the Calculate Zoning Gain block 1134 to find the zoning gain GAZGC 1147 as described in FIG. 12c . The Gain Source 1137 and Zoning Gain GAZGC 1147 are the resulting outputs of the Zoning Processor 1150.
[0110] FIG. 12a is a preferred embodiment of the logic flow for the procedure 1106 for finding the sound source attributes 1140. This process begins at step S1200 with an array of processing gains for all virtual microphones 1121 as calculated in the target processor 1102 described in FIG. 11b . This array of virtual microphones 304 is rearranged from the virtual microphone 304 with the highest processing gain to the virtual microphone 304 with the lowest processing gain in step S1210. Step S1220 initializes the sound source list with the first virtual microphone 304 of the rearranged array which corresponds to the highest processing gain in the array. Then, S1230 begins the process of analyzing the rest of the array one virtual microphone 304 at a time. First, S1240 checks if the sound source is full, meaning all S sound sources that can be allocated have been allocated. If the list is full, the process can exit in S12100 to the Sound Source Allocation process 1136. If the list is not full, the processing gain of the current virtual microphone 304 is checked in S1250 to see if it is above some minimum threshold p. If the gain is below this threshold, this virtual microphone 304 is not desired. Since all virtual microphones 304 after this will have lower processing gain, the sound list can be considered done and this process can exit by moving to the Sound Source Allocation logic at step S12100. If the virtual microphone 304 processing gain is above p, this virtual microphone 304 is then checked in S1260 to see if it is within some minimum distance d of any sound source on the list. If it is, then this is considered a part of the same sound source and this virtual microphone 304 can be ignored in S1270. If the virtual microphone 304 is not within d of any other sound source, this is added to the sound source list in S1280. The process then checks in S1290 if the current virtual microphone 304 is the last virtual microphone 304 in the array. If so, the process can exit by moving to the Sound Source Allocation logic in S12100. If this is not the last mic, the next virtual microphone 304 is loaded and the loop starts over from S1240. The output of this process is the sound source list 1140.
[0111] FIG. 12b is a preferred embodiment of the logic flow for the procedure for allocating sound sources as Gain Sources 1137 or Attenuation sources 1201. This process begins with a list of sound source attributes 1140 preferably discovered by the Target Processor 1102 as described in FIG. 11b . These attributes include the location of the sound sources along with the power of the virtual microphone 304 at that location. The process also requires an initialization step S12110 which initializes the Gain Source 1137 to have a power of zero (0) and the same location as the last valid GS 1137 found. If there is no last valid GS 1137 available, this can be initialized to be at the center of any inclusion zone 305. S12110 also initializes a blank list of Attenuation Sources 1139. Once S12110 is complete, the source processing loop can start at step S12120. This takes in the next available source in the list of 1140 and checks where it is located in step S12130. If the source is located in an exclusion zone 306, it gets passed on to S12140 which checks if the attenuation list 1139 is full. Note that the attenuation list 1139 has a maximum number of AS 1201 allowed which is a parameter of the ACP. If the AS list 1139 is not full, the new source is added in S12170. If the AS list 1139 is full, the power of the new source is checked in S12150 to see if it is greater than the smallest AS 1201 power in the AS list 1139. If the new power is greater, the AS 1201 with the smallest power in the AS list 1139 is overwritten with the new source in S12160. If not, the new source is simply ignored in S12180. If S12130 finds that the source is located in an undefined zone 710, it simply gets ignored in S12180. If the source is located in an inclusion zone 305, its power is multiplied by the weight of the inclusion zone 305 WZ in S12190. This is a parameter of the ACP that represents a way to assign different priorities to different inclusion zones 305. A greater WZ means that the virtual microphone 304 power is greater which increases the likelihood of replacing the Gain Source 1137. Therefore, a higher WZ assigns a higher priority to any zone. WZ is a weight value between 0 and 1.0. The weighted virtual microphone 304 power is checked to see if its power is greater than the current gain source 1137 power in S12200. If it is, this source becomes the new gain source 1137 in S12210. If not, this source is ignored in S12180. After each source has been processed, the loop will check if there are any other sound sources to process in S12220. If there are, the next sound source is checked in S12120. If not, the process exits to the next stage in S12230 by passing the GS 1137 and AS list 1139 to the Calculate Zoning Ratio process 1135 described in FIG. 12 c.
[0112] FIG. 12c is a preferred embodiment of the logic flow for the procedure for finding the Zoning Ratio r. This ratio is representative of the sound power inside of the inclusion zone 305 compared to the sound power inside of the exclusion zone 306. The ratio will range from −1 to 1. An r value of −1 means that the sound in the room 112 is coming primarily from the exclusion zones 306 and sounds in the inclusion zones 305 can be deemed negligible. An r value of 1 means that the sound in the room 112 is coming primarily from the inclusion zones 305 and sounds in the exclusion zones 306 can be deemed negligible. A value of 0 means that there is equal sound coming from the inclusion 305 and exclusion 306 zones. This process 1135 starts at step S12240 and takes in as input the GS 1137 power 1137 and AS List 1139 preferably calculated by the Sound Source Allocation process 1136 as described in FIG. 12b . Step S12250 will find the power of the AS 1201 with the maximum virtual microphone 304 power in the AS list 1139. This gets stored as PAS. Note that this is just one option of finding PAS. Another option is to take the average of all the powers in the AS list 1139. If there are no AS 1201 in the list 1139, PAS can get set to some very small value ∈ to prevent division by zero. The power of the gain source gets stored as PGS. Step S12260 first checks both PGS and PAS to see if both values are below the minimum threshold Pmin. If they are, it is deemed that there is no appreciable sound source in the room 112 and so the zoning ratio r is set to 0 in S12300. An r value of 0 corresponds to a gain of 1 which means no gain or attenuation will be applied in this case. If either or both of PGS and PAS are above Pmin, it is deemed that there is an appreciable sound source in the room 112. Next, S12270 checks which of PAS and PGS is greater. If PGS is greater, it is deemed that there is a higher signal in the inclusion zones 305 than in the exclusion zones 306. From there, r is determined to be positive. In S12280, the ratio of PGS/PAS is checked against the first threshold YGZ. If the ratio is greater than this threshold, it is assumed that the sound signal in the inclusion zones 305 is much louder than that of the exclusion zones 306 and so r is set to the maximum possible value of 1 in S12300. If PGS/PAS is less than YGZ, it is assumed that there are signals in the inclusion 305 and exclusion 306 zones and so r is set to
[0113]
in step S12300. Since it is already a condition that PGS is greater than PAS and PGS/PAS is less than or equal to YGZ, this means that r will take on some value between 1/YGZ and 1. If it is found that PAS is greater than or equal to PGS in S12270, S12290 checks the ratio of PGS/PAS against the second threshold YAZ. If the ratio is greater than the threshold, it is assumed that the sound signal in the exclusion zones 306 is much louder than that of the inclusion zone 305 and so r is set to the minimum possible value of −1. If PGS/PAS is less than YAZ, it is assumed that there are signals in the inclusion 305 and exclusion zones 306 and so r is set to
in step S12300. Since it is already a condition that PGS is greater than PAS and PGS/PAS is less than or equal to YGZ, this means that r will take on some value between 1/YGZ and 1. If it is found that PAS is greater than or equal to PGS in S12270, S12290 checks the ratio of PGS/PAS against the second threshold YAZ. If the ratio is greater than the threshold, it is assumed that the sound signal in the exclusion zones 306 is much louder than that of the inclusion zone 305 and so r is set to the minimum possible value of −1. If PGS/PAS is less than YAZ, it is assumed that there are signals in the inclusion 305 and exclusion zones 306 and so r is set to
[0114]
in step S12300. Since it is already a condition that PAS is greater than PGS and PGS/PAS is less than or equal to YAZ, this means that r will take on some value between −1/YAZ and −1. Note that YGZ and YAZ are ACP parameters configurable to each inclusion 305 and exclusion 306 zone respectively per ACP. YGZ corresponds to the parameters of the inclusion 305 zone to which the GS 1137 belongs while YAZ corresponds to the parameters of the exclusion 306 zone to which the AS 1201 from which PAS was derived belongs. Typical values of YAZ and YGZ can preferably range anywhere from but not limited to 2 to 8. Pmin is another ACP parameter. Pmin values are tied to typical virtual microphone 304 powers and should be experimentally determined based on the number of microphones 106 and type of individual microphone processing 1142 of the system. The output of process 1135 is the zoning ratio r 1146 which is then sent in S12310 to the Calculate Zoning Gain block 1134 as described inFIG. 12 d.
in step S12300. Since it is already a condition that PAS is greater than PGS and PGS/PAS is less than or equal to YAZ, this means that r will take on some value between −1/YAZ and −1. Note that YGZ and YAZ are ACP parameters configurable to each inclusion 305 and exclusion 306 zone respectively per ACP. YGZ corresponds to the parameters of the inclusion 305 zone to which the GS 1137 belongs while YAZ corresponds to the parameters of the exclusion 306 zone to which the AS 1201 from which PAS was derived belongs. Typical values of YAZ and YGZ can preferably range anywhere from but not limited to 2 to 8. Pmin is another ACP parameter. Pmin values are tied to typical virtual microphone 304 powers and should be experimentally determined based on the number of microphones 106 and type of individual microphone processing 1142 of the system. The output of process 1135 is the zoning ratio r 1146 which is then sent in S12310 to the Calculate Zoning Gain block 1134 as described in
[0115] FIG. 12d is a preferred embodiment of the logic flow for the procedure 1134 for finding the Zoning Gain GAZGC. The input S12320 to this process is the zoning ratio r 1146 preferably calculated by the Finding Zoning Ratio process 1135 as described in FIG. 12c . If r is found positive in S12230, the zoning gain is set to GAZGC=1+r*(GmaxZ−1) in S12340. This will preferably give a value between 1 as r approaches 0 and GmaxZ if r is 1. If r is negative, the zoning gain is set to GAZGC=1+r*(1-GminZ) in S12340. This will give a value between GminZ in if r is −1 and 1 if r is 0. Note that GmaxZ represents the maximum possible gain of the inclusion zone 305 that the GS 1137 is in. GminZ represents the minimum possible gain of the exclusion zone 306 that the AS with the strongest power is in. Both of these values are configurable parameters that are defined as part of the Active Zone Configuration 1127 per ACP 1126. Typical values of GmaxZ range from 2 to 6 while typical values of GminZ range from ⅙ to ½. The resulting gain GAZGC gets sent to the multiplier 1137 that applies the zoning gain to the combined microphone signal 1143 to produce the ACP output signal 1144 as shown in FIG. 11 d.
[0116] With reference to FIGS. 13a-13f , shown are examples of the behavior of the sound source tracking in different scenarios. Target in the examples is meant to show the location of the virtual microphone 304 location that can be defined as a AS 1201 and/or GS 1137. In FIG. 13a , there is a person 107 talking in the exclusion zone 306. Since the person 107 is talking inside of the exclusion zone 306, there is an AS 1201 target 901 directly tracking the person 107. In this case, the GS 1137 target 902 must remain in the inclusion zone 305 and so it picks the location in 305 with the strongest virtual microphone 304 power. This corresponds to the border of the inclusion zone 305 at the virtual microphone 304 that is closest to the person 107 while still being in the inclusion zone 305. In FIG. 13b , the person 107 is now in the inclusion zone 305 and so the GS 1137 target 902 is tracking them directly. The AS 1201 target 901 must remain in the exclusion zone 306 and so it ends up on the border of the exclusion zone 306 where the virtual microphone 304 power is loudest. In FIG. 13c , the person 107 is in the undefined zone 710 and so the GS 1137 target 902 is on the border of the IZ 305 and the AS 1201 target 901a is on the border of the EZ 306. In FIG. 13d , the person 107 is in UZ1 710 and the GS 1137 target 902 and AS 1201 target 901a are tracking the person 107 to the closest spots in IZ1 305 and EZ1 306 respectively. In this configuration, EZ1 supports at least 2 AS's 1201. There is another sound source 1313 in EZ1 that represents the noise from an HVAC. In this case, the second AS 1201 target 901b tracks this noise as well. FIG. 13e shows a scenario with two people 107a in IZ1 305 and 107b in UZ1 710 are talking at the same time at similar levels. In this case, the GS 1137 target 902 tracks the person 107a in IZ1 while the AS 1201 target 901 tracks the closest sound source to EZ1 which in this case is the person 107b in UZ1 710. FIG. 13f shows another scenario with two people 107a and 107b talking at the same time at similar levels in IZ1 305 and EZ1 306 respectively. In this case, the GS 1137 target 902 tracks the person 107a in the IZ while the AS 1201 target 901 tracks the person 107b in the EZ.
[0117] With reference to FIG. 14a , shown is an example of a room 112 with two (2) microphone arrays 124. In this case, the room 112 is configured with one ACP. The ACP has 2 different inclusion zones IZ1 305a and IZ2 305b and two exclusion zones EZ1 306a and EZ2 306b. In this case, the ACP channel has a Pmin of −30 dB and supports 2 attenuation sources 1201 as defined in 1401. IZ1 is configured with a GmaxZ of 2, a YGZ of 3 and a WZ of 1 while IZ2 is configured with a GmaxZ of 4, a YGZ of 4 and a WZ of 0.7. EZ1 is configured with a GminZ of ¼ and a YAZ of 2. EZ2 is configured with a GminZ of 1 and a YAZ of 1, meaning that sources in this region will be measured but not attenuated nor boosted. Two possible locations that the GS 1137 targets 902a and 902b can occupy are illustrated.
[0118] FIG. 14b and FIG. 14c represent a system configured with 2 different ACP configurations. In this example, the microphone arrays 124 are not shown but assumed to be present somewhere in the room 112. FIG. 14b represents the first ACP ACP1 while FIG. 14c represents the second ACP ACP2. ACP1 supports 2 AS 1201 targets and has a Pmin of −30 dB as defined in 1421. ACP1 has one inclusion zone IZ1 305a with a GmaxZ of 4, a YGZ of 8 and a WZ of 1.0 and two exclusion zones EZ1 306a with a GminZ of ½ and a YAZ of 2 and EZ2 306c with a GminZ of 1 and a YAZ of 1. ACP2 supports 1 AS 1201 target and has a Pmin of −30 dB as defined in 1422. ACP2 has one inclusion zone IZ1 305b with a GmaxZ of 2, a YGZ of 4 and a WZ of 1.0 and two exclusion zones EZ1 306b with a GminZ of ½ and a YAZ of 2 and EZ2 306d with a GminZ of 1 and a YAZ of 1. For both ACPs, ACP1 and ACP2, EZ2 is configured with a GminZ of 1 and a YAZ of 1, meaning that sources in this region will be measured but not attenuated nor boosted.
[0119] FIGS. 14d, 14e and 14f represent a system configured with 3 different ACP ACP1, ACP2 and ACP3 configurations respectively. FIG. 14d represents ACP1 while FIG. 14e represents ACP2 and FIG. 14f represents ACP3. In this example, the microphone arrays 124 are not shown but assumed to be present somewhere in the room 112. ACP1 supports 3 AS 1201 targets and has a Pmin of −40 dB as defined in 1410. ACP1 has two inclusion zones IZ1 305a with a GmaxZ of 4, a YGZ of 8 and a WZ of 1.0 and IZ2 305b with a GmaxZ of 3, a YGZ of 4 and a WZ of 0.5 along with two exclusion zones EZ1 306a with a GminZ of ½ and a YAZ of 2 and EZ2 306f with a GminZ of 1 and a YAZ of 1. ACP2 supports 3 AS 1201 targets and has a Pmin of −40 dB as defined in 1411. ACP2 has one inclusion zone IZ1 305c with a GmaxZ of 4, a YGZ of 8 and a WZ of 1.0 along with three exclusion zone EZ1 306b with a GminZ of ½ and a YAZ of 2, EZ2 306c with a GminZ of ½ and a YAZ of 2 and EZ3 306g with a GminZ of 1 and a YAZ of 1. ACP3 supports 5 AS 1201 targets and has a Pmin of −40 dB as defined in 1412. ACP3 has one inclusion zone IZ1 305d with a GmaxZ of 3, a YGZ of 4 and a WZ of 1.0 along with three exclusion zones EZ1 306d with a GminZ of ½ and a YAZ of 2, EZ2 306e with a GminZ of ½ and a YAZ of 2 and EZ3 306h with a GminZ of 1 and a YAZ of 1. In this configuration, all 3 ACPs ACP1, ACP2 and ACP3 have the same exclusion zone 306a, 306b and 306d on the left side of the room. This might correspond to a spot with an undesirable sound such as an HVAC 1413 in the room since it is always part of an exclusion zone 306. All ACPs ACP1, ACP2 and ACP3 also contain an exclusion zone 306f, 306g and 306h with a GminZ of 1 and a YAZ of 1 in the background of the room. These represent an area from where signals are neither attenuated not boosted. On the right side of the room 112, each ACP ACP1, ACP2 and ACP3 is configured differently. ACP1 has 2 inclusion zones 305a and 305b in the top and bottom of the right side of the room. This could for example correspond to a presenter 107 with a podium in IZ1 and audience seating 1414 in IZ2 305. This means a remote participant 101 listening to ACP1 would get good signal level and coverage from both of those locations. In this case, IZ1 has a weight of 1.0 while IZ2 has a weight of 0.5 so if the presenter 107 and the audience 1414 are talking at the same time, the GS 1137 will only focus on the audience 1414 if they are talking at more than double the volume of the presenter 107. Otherwise, the GS 1137 will target the presenter. ACP2 has an inclusion zone IZ1 305c in the same location as 305a from ACP1 and an exclusion zone EZ2 306c at the same spot as 305b from ACP1. ACP2 configuration is effectively taking ACP1 and converting 305b to an exclusion zone 306c. A remote participant 101 listening to ACP2 would only get sound source targets and signal gain from the top right corner of the room 112 and would correspond to the presenter 107 at the podium. Similarly, ACP3 has an inclusion zone IZ1 305d in the same location as 305b from ACP1 and an exclusion zone EZ2 306e at the same location as 305a from ACP1. This channel is effectively taking ACP1 and converting 305a to an inclusion zone 306e. A remote participant 101 listening to ACP3 would only get sound source targets from the bottom right corner of the room 112 corresponding to the audience seating 1414. With the configuration presented in these 3 ACPs, ACP1, ACP2 and ACP3 one or more remote participants 101 could choose to place attention on the presenter 107 using ACP2, the audience 1414 using ACP3 or both using ACP1. Automatic zoning gain control not only allows for proper and intelligent gain source structure mapping at all locations, virtual microphone 304 x, y, z positions, in the room 112 according to the sound source location relative to the IZ 305 and EZ 306 location and boundaries but also allows for one or more audio streams with custom ACP configurations to be sent to one or more remote participants 101 at the far end of the conference call.
[0120] FIG. 15a shows an example of a woman 107 moving in a room 112 with two microphone arrays 124 from an inclusion zone IZ1 305 to an exclusion zone EZ1 306a while speaking at a constant volume. Target in the following examples is meant to show the location of the virtual microphone 304 location that can be defined as a AS 1201 and/or GS 1137. In this case, the inclusion zone IZ1 305 has a GmaxZ of 2, a YGZ of 3 and a WZ of 1.0 while the exclusion zone EZ1 306a has a GminZ of ¼ and a YAZ of 2. This configuration supports 1 AS 1201 target. The woman 107 begins speaking at point A. At this point, the GS 1137 target 902a will be focused on her since point A is inside of the inclusion zone 305. At this point, the AS 1201 target will pick the loudest virtual microphone 304 in the exclusion zone EZ1 306a. It is assumed that there are no other sound sources in the exclusion zone 305 EZ1 so the AS 1201 target 901a is the closest to the person 107 at position A since that point will have the most energy out of all virtual microphones 304 in EZ1 306a. At position A, the GS 1137 target 902a will have a much louder power than the AS 1201 target 901a. Following the logic defined in FIG. 12b , this means that PGS>PAS and in this case
[0121]
for this position. This means that the zoning ratio r becomes 1 for this position. The zoning gain as calculated inFIG. 12d for this position A is then GAZGC=1+r*(GmaxZ−1)=GmaxZ=2. As the woman 107 keeps moving into the exclusion zone 306a EZ1, she eventually reaches position B on the border of the inclusion zone IZ1 305. At this point, the GS 1137 target 902b is still tracking the woman's position. Since the woman 107 is still in the inclusion zone IZ1 305, the AS 1201 target 901a remains at the closest point of EZ1 306a. The GS 1137 target is still closer to the woman 107 than the AS 1201 is so PGS>PAS is still true. Now however, the GS 1137 at target 902b and AS 1201 at target 901a are fairly close to each other. In this position,
for this position. This means that the zoning ratio r becomes 1 for this position. The zoning gain as calculated in
[0122]
Following the logic inFIG. 12c , that means that
Following the logic in
[0123]
Applying the logic inFIG. 12d , the zoning gain for position B is then calculated as GAZGC 1+r*(GmaxZ−1)=1+0.35*(2−1)=1.35. As the woman 107 crosses the border into the exclusion zone EZ1 306a, she reaches position C. At this point, she is now within EZ1 306a so the AS 1201 starts tracking her at 901a. The GS 1137 target no longer has a valid source to track in the inclusion zone IZ1 305 so the GS 1137 target will remain on the border at position 902b where the virtual microphone 304 power is loudest. Now, the AS 1201 target 901a is closer to the sound source 107 than the GS 1137 target 902b so PGS<PAS. The AS 1201 target 901a and GS 1137 target 902b are still very close to each other and
Applying the logic in
[0124]
Following the logic inFIG. 12b , that means to
Following the logic in
[0125]
Applying the logic inFIG. 12c , the zoning gain for position C is then calculated as
Applying the logic in
[0126]
The woman 107 keeps walking and eventually reaches position D. At this point, she is in the middle of the exclusion zone EZ1 306a and the AS 1201 is tracking her at target 901b. The GS 1137 target 902b is still at the border of the inclusion zone IZ1 305 so PGS<PAS and now
The woman 107 keeps walking and eventually reaches position D. At this point, she is in the middle of the exclusion zone EZ1 306a and the AS 1201 is tracking her at target 901b. The GS 1137 target 902b is still at the border of the inclusion zone IZ1 305 so PGS<PAS and now
[0127]
which means that r is set to −1 and the zoning gain is set to GAZGC=1+r*(1−GminZ)=GminZ=0.25. EZ2 306b fills the rest of the room 112 with an exclusion zone 306 with a GminZ of 1 and a YAZ of 1 which means that sources picked up in this space should have no boost or attenuation. For this example, it is considered that target 901a is closer to points A and B than any point in EZ2 306b. In IZ1 305, the minimum value r can take is
which means that r is set to −1 and the zoning gain is set to GAZGC=1+r*(1−GminZ)=GminZ=0.25. EZ2 306b fills the rest of the room 112 with an exclusion zone 306 with a GminZ of 1 and a YAZ of 1 which means that sources picked up in this space should have no boost or attenuation. For this example, it is considered that target 901a is closer to points A and B than any point in EZ2 306b. In IZ1 305, the minimum value r can take is
[0128]
since PGS is always greater than PAS. With this r, the resulting gain is
since PGS is always greater than PAS. With this r, the resulting gain is
[0129]
This means r will range from ⅓ to 1 and GAZGC will range from 4/3 to 2. In EZ1, the maximum value r can take is
This means r will range from ⅓ to 1 and GAZGC will range from 4/3 to 2. In EZ1, the maximum value r can take is
[0130]
since PAS is always greater than PGS. With this r, the resulting gain is
since PAS is always greater than PGS. With this r, the resulting gain is
[0131]
This means r will range from −1 to −½ and GAZGC will range from ¼ to ⅝. A sound source such as the woman 107 will experience a maximum gain of 2 in the middle of IZ1 305. As she moves closer to the edge of IZ1 305, the gain will drop to a minimum potential value of 4/3. As she crosses from IZ1 305 to EZ1 306a, the gain will jump from at least the minimum IZ1 305 gain 4/3(2.5 dB) to at least the minimum attenuation or maximum gain of EZ1 306a ⅝ (−4 dB). This is a total jump of 6 dB. As the woman 107 keeps moving into the center of EZ1 306a, the gain will gradually lower to its minimum gain of ¼. This border effect is one that can be tuned using the YAZ and YGZ thresholds. For example, with a larger YGZ the ratio r in IZ1 305 could drop to a smaller minimum value since the minimum r in an IZ is 1/YGZ. This would result in a lower gain at position B. Likewise, a larger value of YAZ would lead to a larger maximum r of EZ1 306a since the maximum r in an EZ 306 is −1/YAZ. This would result in a higher gain at position C. Both thresholds could be tuned to have a higher or lower transition from IZ1 305 to EZ1 306a. The gain values GminZ and GmaxZ could also be tuned to change this effect but these will also change the gain values in the center of the zones so it is usually preferred to tune the gain values for the desired zone gains and the thresholds for the border transitions. Note that tuning the thresholds will also affect how far from the border of the zone a source must be before reaching GmaxZ and GminZ, For example, for a source in an IZ 305, a large threshold YGZ means that PGS needs to be higher before PGS/PAS is greater than YGZ. This means the GS 1137 target will need to be farther from the AS 1201 target before the maximum gain GmaxZ is reached. Likewise, with a higher YAZ, the AS 1201 target will need to be further from the GS 1137 target before the minimum EZ 306 gain GminZ is reached for a source in an EZ 306.
This means r will range from −1 to −½ and GAZGC will range from ¼ to ⅝. A sound source such as the woman 107 will experience a maximum gain of 2 in the middle of IZ1 305. As she moves closer to the edge of IZ1 305, the gain will drop to a minimum potential value of 4/3. As she crosses from IZ1 305 to EZ1 306a, the gain will jump from at least the minimum IZ1 305 gain 4/3(2.5 dB) to at least the minimum attenuation or maximum gain of EZ1 306a ⅝ (−4 dB). This is a total jump of 6 dB. As the woman 107 keeps moving into the center of EZ1 306a, the gain will gradually lower to its minimum gain of ¼. This border effect is one that can be tuned using the YAZ and YGZ thresholds. For example, with a larger YGZ the ratio r in IZ1 305 could drop to a smaller minimum value since the minimum r in an IZ is 1/YGZ. This would result in a lower gain at position B. Likewise, a larger value of YAZ would lead to a larger maximum r of EZ1 306a since the maximum r in an EZ 306 is −1/YAZ. This would result in a higher gain at position C. Both thresholds could be tuned to have a higher or lower transition from IZ1 305 to EZ1 306a. The gain values GminZ and GmaxZ could also be tuned to change this effect but these will also change the gain values in the center of the zones so it is usually preferred to tune the gain values for the desired zone gains and the thresholds for the border transitions. Note that tuning the thresholds will also affect how far from the border of the zone a source must be before reaching GmaxZ and GminZ, For example, for a source in an IZ 305, a large threshold YGZ means that PGS needs to be higher before PGS/PAS is greater than YGZ. This means the GS 1137 target will need to be farther from the AS 1201 target before the maximum gain GmaxZ is reached. Likewise, with a higher YAZ, the AS 1201 target will need to be further from the GS 1137 target before the minimum EZ 306 gain GminZ is reached for a source in an EZ 306.
[0132] FIG. 15b shows the same room 112 with microphone arrays 124 and ACP zoning configuration as FIG. 15a but the woman 107 is now walking from position D to position A. Since the gains are measured independently at each target position, this means positions A, B, C and D will have the same r and GAZGC values as in FIG. 15a . In FIG. 15a , the woman 107 started out in the middle of the inclusion zone IZ1 305 with the gain set to the maximum possible value of 2 (6 dB). As she walked closer to the border between IZ1 305 and EZ1 306a, the ratio of PGS/PAS dropped to a value of 1.05 that is very close to its minimum possible value of 1. This means that r went from its maximum positive value of 1 to 0.35, which is close to its minimum possible value of ⅓ for a source in IZ1 305. This point corresponded to a gain value of 1.35 (2.6 dB). As the woman 107 crossed the border into EZ1 306a, this caused the AS 1201 target to have more power than the corresponding GS 1137 target. As a result, r then became a negative value of −0.525 which is close to its maximum possible value of −0.5 for a source in EZ1 306a. This position C corresponded in a gain of 0.60625 (−4.3 dB). Therefore, as the woman 107 crossed the border from B in IZ1 305 to C in EZ1 306a, the gain applied to her voice dropped from a boost of 1.35 (2.6 dB) to an attenuation of 0.60625 (−4.3 dB). As she walked from C to D, this gain further dropped to its maximum attenuation of 0.25 (−12 dB). FIG. 15b shows the opposite scenario. The woman 107 is now walking from D to A. At position D, she is in the middle of the exclusion zone 306 so the gain is set to its minimum value of 0.25. As she approaches the border in position C, the gain increases to 0.60625. Once she crosses the border into position B in IZ1 305, the gain becomes a positive boost of 1.35. As she then proceeds into position A in the middle of IZ1 305, the gain applied to her voice reaches its maximum possible value of 2. EZ2 306b fills the rest of the room 112 with an exclusion zone 306 with a GminZ of 1 and a YAZ of 1 which means that sources picked up in this space should have no boost or attenuation. For this example, it is considered that target 901a is closer to points A and B than any point in EZ2 306b.
[0133] FIG. 15c shows the same room 112 with microphone arrays 124 and zoning configurations as FIGS. 15a and 15b . Now, the woman 107 is walking from position E at the border of IZ1 305 and EZ1 306a to position A in EZ2 306b. At position E, the woman 107 is at the border of EZ1 306a. As shown in position C of FIGS. 15a and 15b , the gain here is 0.60625. The woman 107 then walks into the inclusion zone 305 IZ1 and reaches position D in the middle of IZ1 305. As shown in position A of FIGS. 15a and 15b , the GS 1137 target 902a is her current location and the gain at this point is set to GmaxZ=2. Next, the woman 107 reaches position C on the border of IZ1 305 and EZ2 306b to GS 1137 target 902b. Now, the AS 1137 target 901b shifted to EZ2 306b which is the closest point in any exclusion zone 306 to the sound source 107 at position C. Here, PGS>PAS and is 1.05. The zoning ratio is calculated as
[0134]
This results in again of GAZGC=1+r*(GmaxZ−1)=1+0.35*(2−1)=1.35. Note that this is the exact same gain that was on the border of IZ1 305 and EZ1 306a in position B onFIGS. 15a and 15b . This is because the IZ 305 parameters are the same and the zoning ratio is the same. Once the woman 107 crosses into EZ2 306b and reaches position B, the AS 1201 target 901b is determined. The GS 1137 target 902b remains inside of IZ1 305. Now,
This results in again of GAZGC=1+r*(GmaxZ−1)=1+0.35*(2−1)=1.35. Note that this is the exact same gain that was on the border of IZ1 305 and EZ1 306a in position B on
[0135]
since YAZ for EZ2 is 1. This means that r is set to −1 and the gain is set to GAZGC=1+r*(1-GminZ)=1. As the woman 107 reaches position A, she is now much further away from IZ1 305 In this position, the AS 1201 target 901c is set and the GS 1137 target 902b is maintained. Now, PAS is much greater than PGS but this still results in an r of −1, meaning the gain also remains 1. This configuration shows the advantage of filing the room with an exclusion zone 306 configuration with a YAZ of 1 and a GminZ of 1. In this configuration, any sound source in IZ1 305 will get a positive gain applied. In IZ1 305 zone, the minimum value r can take is
since YAZ for EZ2 is 1. This means that r is set to −1 and the gain is set to GAZGC=1+r*(1-GminZ)=1. As the woman 107 reaches position A, she is now much further away from IZ1 305 In this position, the AS 1201 target 901c is set and the GS 1137 target 902b is maintained. Now, PAS is much greater than PGS but this still results in an r of −1, meaning the gain also remains 1. This configuration shows the advantage of filing the room with an exclusion zone 306 configuration with a YAZ of 1 and a GminZ of 1. In this configuration, any sound source in IZ1 305 will get a positive gain applied. In IZ1 305 zone, the minimum value r can take is
[0136]
since PGS is always greater than PAS. With this r, the resulting gain is
since PGS is always greater than PAS. With this r, the resulting gain is
[0137]
This means r will range from ⅓ to 1 and GAZGC will range from 4/3(2.5 dB) to 2(6 dB). In EZ1 306a, the maximum value r can take is
This means r will range from ⅓ to 1 and GAZGC will range from 4/3(2.5 dB) to 2(6 dB). In EZ1 306a, the maximum value r can take is
[0138]
since PAS is always greater than PGS. With this r, the resulting gain is
since PAS is always greater than PGS. With this r, the resulting gain is
[0139]
This means r will range from −1 to −½ and GAZGC will range from ⅝(−4 dB). to ¼(−12 dB). This means any sound source in EZ1 306a will always have a negative gain applied. With a GminZ and a YAZ of 1, sound sources in EZ2 306b will always have a gain of 1 (0 dB) applied. This created the effect of IZ1 305 being a positive gain region, EZ1 306a being a negative gain region and EZ2 306b being a neutral gain region.
This means r will range from −1 to −½ and GAZGC will range from ⅝(−4 dB). to ¼(−12 dB). This means any sound source in EZ1 306a will always have a negative gain applied. With a GminZ and a YAZ of 1, sound sources in EZ2 306b will always have a gain of 1 (0 dB) applied. This created the effect of IZ1 305 being a positive gain region, EZ1 306a being a negative gain region and EZ2 306b being a neutral gain region.
[0140] FIG. 15d shows the same room 112 with microphone arrays 124 as FIGS. 15a, 15b and 15c . This ACP configuration has an undefined zone UZ1 710 instead of the exclusion zone EZ2 306b used in FIGS. 15a, 15b and 15c . The woman 107 is walking from position E at the border of IZ1 305 and EZ1 306a to position A in the undefined zone UD1 710. At position E, the woman 107 is at the border of EZ1 306a. As shown in position C of FIGS. 15a and 15b , the gain here is 0.60625. The woman 107 then walks into the inclusion zone 305 IZ1 and reaches position D in the middle of IZ1 305. As shown in position A of FIGS. 15a and 15b , the GS 1137 target 902a tracks her and the gain at this point is set to GAZGC=GmaxZ=2. Next, the woman reaches position C on the edge of IZ1 305. At this point, the GS 1137 target 902b tracks her and the AS 1201 target 901 is still on the border of EZ1 306a. Just like position C, the ratio of PGS/PAS is greater than the YGZ of 3 so the gain is set to GAZGC=GmaxZ=2. Next, the woman 107 leaves the inclusion zone IZ1 305 and reaches point B in the undefined zone. At this point, 1501b represents the closest virtual microphone 304 to the woman 107. This virtual microphone 304 is ignored since it is in neither in a IZ 305 nor an EZ 306. The GS 1137 target 902b is assigned, which is still very close to position B. At this point, the GS 1137 target 902b is much louder than the AS 1201 target 901 so the gain is still set to GAZGC=GmaxZ=2. The woman 107 keeps walking into UZ1 710 until she reaches position A. Here, the virtual microphone 304 at 1501a is also ignored. Now, the woman 107 is further away from the GS 1137 target 902b and the AS 1201 target 901. Here, the GS 1137 target 902b is still stronger than the AS 1201 target 901 but the levels are closer. In this case,
[0141]
The zoning ratio is then
The zoning ratio is then
[0142]
and the zoning gain is GAZGC=1+0.4*(2−1)=1.4. The scenario here represents an alternative configuration to the one presented inFIG. 15c . Here, the virtual microphones 304 in UZ1 710 are not monitored and UZ1 710 will have some non-zero gain applied based on the position of the sound source 107 relative to its nearest IZ 305 and EZ 306. In FIG. 15c , the virtual microphones 304 in EZ2 306b are still being monitored and any sound source 107 in EZ2 306b will have a gain of 1 applied. In FIG. 15c , there is a slight border effect of transitioning from IZ1 305 to EZ2 306b where the gain jumps from 4/3 to 1 when transitioning from IZ1 305 to EZ2 306b. In FIG. 15d , there is no such border effect when transitioning from any zone to UZ1 710. Typically, FIG. 15c represents the preferred implementation. However, in cases where this border effect is considered problematic, using a UZ 710 such as shown in FIG. 15d becomes a viable option.
and the zoning gain is GAZGC=1+0.4*(2−1)=1.4. The scenario here represents an alternative configuration to the one presented in
[0143] While the present invention has been described with respect to what is presently considered to be the preferred embodiments, it is to be understood that the invention is not limited to the disclosed embodiments. To the contrary, the invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.
Claims
What is claimed is:
1. A system for dynamically adjusting gain structures of sound sources in a shared 3D space including one or more inclusion zones and one or more exclusion zones, comprising:
a combined microphone array comprising one or more of individual microphones and/or microphone arrays each including a plurality of microphones; and
one or more system processors communicating with the combined microphone array, wherein the one or more system processors comprise one or more audio channel profiles (ACPs) and are configured to perform operations comprising:
obtaining predetermined coverage zone dimensions based on the locations of the microphones of the combined microphone array;
populating the coverage zone dimensions with one or more virtual microphones;
obtaining a combined microphone signal, for each audio channel profile (ACP), by combining microphone signals into desired channel audio signals by applying positional based gain control (PBGC) parameters to adjust microphones to control positional based microphone gains based on location information of the sound sources;
performing processes to obtain a zoning gain for each ACP, comprising:
receiving a list of sound sources obtained by utilizing the virtual microphones;
receiving zone parameters for one or more inclusion zones (IZ) and one or more exclusion zones (EZ);
identifying a gain source (GS) and a list of one or more attenuation sources (AS);
determining a zoning ratio based on the gain source, the list of the one or more attenuation sources and active zone configuration parameters; and
calculating zoning gain based on the zoning ratio, maximum gain of the one or more inclusion zones and minimum gain of the one or more exclusion zones; and
generating an output channel for each ACP by multiplying the zoning gain with the combined microphone signal.
2. The system of claim 1 wherein the zone parameters for the one or more inclusion zones and the one or more exclusion zones comprise physical boundaries of the active zone configuration parameters, weights of the inclusion zones, and a maximum number of the attenuation sources that are allocated for the ACP.
3. The system of claim 1 wherein the active zone configuration parameters includes a minimum power threshold (Pmin), a first threshold for PGS/PAS, and a second threshold for PAS/PGS, where PGS is a power of the gain source and PAS is a power of the attenuation source.
4. The system of claim 1 wherein the ACP contains zoning parameters of the output channel including locations and gains of the one or more inclusion zones and the one or more exclusion zones for the output channel.
5. The system of claim 1 wherein a location of the gain source represents a physical location for which the individual microphone signals are aligned to produce the output signal of an ACP.
6. The system of claim 1 wherein each ACP is configured to track or identify a single gain source among gain sources in the one or more inclusion zones.
7. The system of claim 1 wherein each ACP is configured to support multiple attenuation sources.
8. The system of claim 1 wherein the shared 3D space is entirely filled or partially filled with the virtual microphones for monitoring and tracking the sound sources.
9. The system of claim 1 wherein each output channel is configured independently for different needs.
10. The system of claim 1 wherein the one or more inclusion zones and the one or more exclusion zones are configured by grouping all the available virtual microphones into either the one or more inclusion zones or the one or more exclusion zones based on the locations of the virtual microphones in the shared 3D space.
11. The system of claim 1 wherein the one or more system processors are configured to apply a positive gain structure to targeted sound sources in the one or more inclusion zones and to apply a negative gain structure to targeted sound sources in the one or more exclusion zones.
12. The system of claim 1 wherein the one or more inclusion zones and the one or more exclusion zones are configured by grouping at least one virtual microphone or more than one virtual microphones into either the one or more inclusion zones or the one or more exclusion zones based on the locations of the virtual microphones in the shared 3D space.
13. The system of claim 1 wherein the one or more inclusion zones and the one or more exclusion zones are configured to support any dimensioned 3D or 2D shape that contains the one or more virtual microphones in the shared 3D space.
14. A method for dynamically adjusting gain structures of sound sources in a shared 3D space including one or more inclusion zones and one or more exclusion zones, comprising:
obtaining predetermined coverage zone dimensions, via one or more system processors, based on locations of microphones of a combined microphone array, wherein the combined microphone array comprises one or more of individual microphones and/or microphone arrays each including a plurality of microphones, and the system processors communicate with the combined microphone array and comprise one or more audio channel profiles (ACPs);
populating the coverage zone dimensions with one or more virtual microphones;
obtaining a combined microphone signal, for each audio channel profile (ACP), by combining microphone signals into desired channel audio signals by applying positional based gain control (PBGC) parameters to adjust microphones to control positional based microphone gains based on location information of the sound sources;
performing processes to obtain a zoning gain for each ACP, comprising:
receiving a list of sound sources obtained by utilizing the virtual microphones;
receiving zone parameters for one or more inclusion zones (IZ) and one or more exclusion zones (EZ);
identifying a gain source (GS) and a list of one or more attenuation sources (AS);
determining a zoning ratio based on the gain source, the list of the one or more attenuation sources and active zone configuration parameters; and
calculating zoning gain based on the zoning ratio, maximum gain of the one or more inclusion zones and minimum gain of the one or more exclusion zones; and
generating an output channel for each ACP by multiplying the zoning gain with the combined microphone signal.
15. The method of claim 14 wherein the zone parameters for the one or more inclusion zones and the one or more exclusion zones comprise physical boundaries of the active zone configuration parameters, weights of the inclusion zones, and a maximum number of the attenuation sources that are allocated for the ACP.
16. The method of claim 14 wherein the active zone configuration parameters includes a minimum power threshold (Pmin), a first threshold for PGS/PAS, and a second threshold for PAS/PGS, where PGS is a power of the gain source and PAS is a power of the attenuation source.
17. The method of claim 14 wherein the ACP contains zoning parameters of the output channel including locations and gains of the inclusion zones and exclusion zones for the output channel.
18. The method of claim 14 wherein a location of the gain source represents a physical location for which the individual microphone signals are aligned to produce the output signal of an ACP.
19. The method of claim 14 wherein each ACP is configured to track or identify a single gain source among gain sources in the one or more inclusion zones.
20. The method of claim 14 wherein the ACP is configured to support multiple attenuation sources.
21. The method of claim 14 wherein the shared 3D space is entirely filled or partially filled with the virtual microphones for monitoring and tracking the sound sources.
22. The method of claim 14 wherein each output channel is configured independently for different needs.
23. The method of claim 14 wherein the one or more inclusion zones and the one or more exclusion zones are configured by grouping all the available virtual microphones into either the one or more inclusion zones or the one or more exclusion zones based on the locations of the virtual microphones in the shared 3D space.
24. The method of claim 14 wherein a positive gain structure is applied to targeted sound sources in the inclusion zone and a negative gain structure is applied to targeted sound sources in the exclusion zone.
25. The method of claim 14 wherein the one or more inclusion zones and the one or more exclusion zones are configured by grouping at least one virtual microphone or more than one virtual microphones into either the one or more inclusion zones or the one or more exclusion zones based on the locations of the virtual microphones in the shared 3D space.
26. The method of claim 14 wherein the one or more inclusion zones and the one or more exclusion zones are configured to support any dimensioned 3D or 2D shape that contains the one or more virtual microphones in the shared 3D space.
27. One or more non-transitory computer-readable media for dynamically adjusting gain structures of sound sources in a shared 3D space including one or more inclusion zones and one or more exclusion zones, the computer-readable media comprising instructions configured to cause a system processor to perform operations comprising:
obtaining predetermined coverage zone dimensions, via one or more system processors, based on locations of microphones of a combined microphone array, wherein the combined microphone array comprises one or more of individual microphones and/or microphone arrays each including a plurality of microphones, and the system processors communicate with the combined microphone array and comprise one or more audio channel profiles (ACPs);
populating the coverage zone dimensions with one or more virtual microphones;
obtaining a combined microphone signal, for each audio channel profile (ACP), by combining microphone signals into desired channel audio signals by applying positional based gain control (PBGC) parameters to adjust microphones to control positional based microphone gains based on location information of the sound sources;
performing processes to obtain a zoning gain for each ACP, comprising:
receiving a list of sound sources obtained by utilizing the virtual microphones;
receiving zone parameters for one or more inclusion zones (IZ) and one or more exclusion zones (EZ);
identifying a gain source (GS) and a list of one or more attenuation sources (AS);
determining a zoning ratio based on the gain source, the list of the one or more attenuation sources and active zone configuration parameters; and
calculating zoning gain based on the zoning ratio, maximum gain of the one or more inclusion zones and minimum gain of the one or more exclusion zones; and
generating an output channel for each ACP by multiplying the zoning gain with the combined microphone signal.
Patent Citations (76)
| Patent | Date | Inventor | Cited By |
|---|---|---|---|
| US4499578(A) | 1985-02-01 | Marouf et al. | Applicant |
| US4536887(A) | 1985-08-01 | Kaneda et al. | Applicant |
| US5477270(A) | 1995-12-01 | Park | Applicant |
| US5699437(A) | 1997-12-01 | Finn | Applicant |
| US6469732(B1) | 2002-10-01 | Chang et al. | Applicant |
| US6593956(B1) | 2003-07-01 | Potts et al. | Applicant |
| US6912178(B2) | 2005-06-01 | Chu et al. | Applicant |
| US6912718(B1) | 2005-06-01 | Chang et al. | Applicant |
| US7130705(B2) | 2006-10-01 | Amir et al. | Applicant |
| US7254241(B2) | 2007-08-01 | Rui et al. | Applicant |
| US7489788(B2) | 2009-02-01 | Leung et al. | Applicant |
| US7720232(B2) | 2010-05-01 | Oxford et al. | Applicant |
| US7848531(B1) | 2010-12-01 | Vickers et al. | Applicant |
| US7995768(B2) | 2011-08-01 | Miki et al. | Applicant |
| US8185387(B1) | 2012-05-01 | Lachapelle | Applicant |
| US8861537(B1) | 2014-10-01 | Braithwaite et al. | Applicant |
| US8953819(B2) | 2015-02-01 | Ko et al. | Applicant |
| US9706292(B2) | 2017-07-01 | Duraiswami et al. | Applicant |
| US9800964(B2) | 2017-10-01 | McIntosh et al. | Applicant |
| US10003900(B2) | 2018-06-01 | Cartwright et al. | Applicant |
| US10042038(B1) | 2018-08-01 | Lord et al. | Applicant |
| US10063987(B2) | 2018-08-01 | McGibney | Applicant |
| US10229697(B2) | 2019-03-01 | Bastyr et al. | Applicant |
| US10237639(B2) | 2019-03-01 | McIntosh et al. | Applicant |
| US10387108(B2) | 2019-08-01 | McGibney | Applicant |
| US10397726(B2) | 2019-08-01 | McGibney | Applicant |
| US10848896(B2) | 2020-11-01 | McGibney | Applicant |
| US10972835(B2) | 2021-04-01 | Rollow, IV | Applicant |
| US11127415(B2) | 2021-09-01 | Magnusson et al. | Applicant |
| US11190871(B2) | 2021-11-01 | Yorga et al. | Applicant |
| US2005/0280701(A1) | 2005-12-01 | Wardell | Applicant |
| US2006/0034469(A1) | 2006-02-01 | Tamiya et al. | Applicant |
| US2006/0165242(A1) | 2006-07-01 | Miki et al. | Applicant |
| US2008/0085014(A1) | 2008-04-01 | Chen et al. | Applicant |
| US2008/0107277(A1) | 2008-05-01 | Somasundaram et al. | Applicant |
| US2008/0285771(A1) | 2008-11-01 | Tanaka et al. | Applicant |
| US2009/0129609(A1) | 2009-05-01 | Oh et al. | Applicant |
| US2010/0034397(A1) | 2010-02-01 | Nakadai et al. | Applicant |
| US2010/0135118(A1) | 2010-06-01 | Van Leest et al. | Applicant |
| US2011/0135125(A1) | 2011-06-01 | Zhan et al. | Applicant |
| US2012/0093344(A1) | 2012-04-01 | Sun et al. | Applicant |
| US2012/0245933(A1) | 2012-09-01 | Flaks et al. | Applicant |
| US2013/0083934(A1) | 2013-04-01 | Ahgren | Applicant |
| US2013/0101134(A1) | 2013-04-01 | Betts-Lacroix | Applicant |
| US2013/0142342(A1) | 2013-06-01 | Del Galdo et al. | Applicant |
| US2013/0258813(A1) | 2013-10-01 | Herre et al. | Applicant |
| US2014/0050328(A1) | 2014-02-01 | Fischer | Applicant |
| US2014/0098964(A1) | 2014-04-01 | Rosca | Examiner |
| US2014/0119552(A1) | 2014-05-01 | Beaucoup | Applicant |
| US2014/0133666(A1) | 2014-05-01 | Tanaka et al. | Applicant |
| US2014/0185824(A1) | 2014-07-01 | Burnett | Applicant |
| US2014/0314251(A1) | 2014-10-01 | Rosca et al. | Applicant |
| US2014/0348342(A1) | 2014-11-01 | Laaksonen et al. | Applicant |
| US2015/0185312(A1) | 2015-07-01 | Gaubitch et al. | Applicant |
| US2015/0222996(A1) | 2015-08-01 | Chu et al. | Applicant |
| US2015/0230026(A1) | 2015-08-01 | Eichfeld et al. | Applicant |
| US2016/0071526(A1) | 2016-03-01 | Wingate et al. | Applicant |
| US2016/0112469(A1) | 2016-04-01 | Liu | Applicant |
| US2016/0173976(A1) | 2016-06-01 | Podhradsky | Applicant |
| US2017/0178628(A1) | 2017-06-01 | Macours et al. | Applicant |
| US2017/0347217(A1) | 2017-11-01 | McGibney | Applicant |
| US2017/0366896(A1) | 2017-12-01 | Adsumilli et al. | Applicant |
| US2017/0374454(A1) | 2017-12-01 | Bernardini et al. | Applicant |
| US2018/0074782(A1) | 2018-03-01 | McGibney | Applicant |
| US2018/0098174(A1) | 2018-04-01 | Goodwin et al. | Applicant |
| US2018/0249267(A1) | 2018-08-01 | Klingler et al. | Applicant |
| US2019/0349471(A1) | 2019-11-01 | Ferguson et al. | Applicant |
| US2021/0035563(A1) | 2021-02-01 | Cartwright et al. | Applicant |
| US2022/0004355(A1) | 2022-01-01 | Mcgibney | Examiner |
| EP903055(B1) | 2007-10-01 | Applicant | |
| EP2975609(A1) | 2016-01-01 | Applicant | |
| JP3154468(B2) | 2001-04-01 | Applicant | |
| JP2018026701(A) | 2018-02-01 | Applicant | |
| WO3010995(A2) | 2003-02-01 | Applicant | |
| WO2022/118072(A1) | 2022-06-01 | Applicant | |
| WO2023/164773(A1) | 2023-09-01 | Applicant |
Non-Patent Literature (60)
- International Search Report and Written Opinion mailed Aug. 9, 2024, for International Patent Application No. PCT/CA2024/050615, 9 sheets.Applicant
- Final Office Action dated Dec. 5, 2024, from U.S. Appl. No. 17/739,926, 29 sheets.Applicant
- International Search Report and Written Opinion mailed Sep. 15, 2017, from International Application No. PCT/CA2017/050642, 12 sheets.Applicant
- Joseph Hector Dibiase, Thesis entitled, “A High-Accuracy, Low-Latency Technique for Talker Localization in Reverberant Environments Using Microphone Arrays”, Brown University, May 2000.Applicant
- Notice of Allowance dated Apr. 30, 2018, from U.S. Appl. No. 15/597,646, 18 sheets.Applicant
- International Search Report and Written Opinion mailed Jun. 7, 2023, from International Application No. PCT/CA2023/50412, 11 sheets.Applicant
- Extended European Search Report mailed May 7, 2019, from European Patent Application No. 17805437.5, 23 sheets.Applicant
- Notice of Allowance dated May 24, 2019, from U.S. Appl. No. 16/110,393, 6 sheets.Applicant
- Non-Final Rejection dated Sep. 17, 2018, from U.S. Appl. No. 16/110,393, 14 sheets.Applicant
- International Search Report and Written Opinion mailed Oct. 12, 2017, from International Application No. PCT/CA2017/050676, 8 sheets.Applicant
- Notice of Allowance dated Apr. 2, 2019, from U.S. Appl. No. 15/603,986, 42 sheets.Applicant
- Final Rejection dated May 25, 2018, from U.S. Appl. No. 15/603,986, 13 sheets.Applicant
- Non-Final Rejection dated Jan. 24, 2018, from U.S. Appl. No. 15/603,986, 26 sheets.Applicant
- Communication pursuant to Article 94(3) EPC dated Feb. 17, 2020, from European Patent Application No. 17805437.5, 6 sheets.Applicant
- Notice of Allowance dated Jul. 16, 2020, from U.S. Appl. No. 16/518,013, 19 sheets.Applicant
- Non-Final Rejection dated Feb. 28, 2020, from U.S. Appl. No. 16/518,013, 21 sheets.Applicant
- Notice of Allowance dated Apr. 11, 2019, from U.S. Appl. No. 16/110,393, 18 sheets.Applicant
- Notice of Allowance dated Jan. 19, 2018, from U.S. Appl. No. 15/597,646, 22 sheets.Applicant
- Notice of Allowance dated Aug. 16, 2021, from U.S. Appl. No. 17/097,560, 33 sheets.Applicant
- Extended European search report from European Application No. 20194651.4 with a mailing date of Jan. 21, 2021, 11 sheets.Applicant
- Extended European search report from European Application No. 17847841.8 with a mailing date of Jun. 28, 2019, 12 sheets.Applicant
- Communication pursuant to Rules 70(2) and 70a(2) EPC from European Application No. 17847841.8 with a mailing date of Jul. 16, 2019, 1 sheet.Applicant
- U.S. Appl. No. 62/343,512, filed May 31, 2016, 41 sheets.Applicant
- U.S. Appl. No. 62/162,091, filed May 15, 2015, 52 sheets.Applicant
- U.S. Appl. No. 62/345,208, filed Jun. 3, 2016, 44 sheets.Applicant
- Non-Final Rejection dated Feb. 9, 2021, from U.S. Appl. No. 16/434,725, 72 sheets.Applicant
- Notice of Allowance dated Jan. 25, 2023, from U.S. Appl. No. 17/374,585, 11 sheets.Applicant
- Notice of Allowance dated Jan. 19, 2023, from U.S. Appl. No. 17/374,585, 41 sheets.Applicant
- Non-Final Rejection dated Aug. 8, 2022, from U.S. Appl. No. 17/374,585, 57 sheets.Applicant
- The extended European search report completed Feb. 2, 2022 (dated Feb. 10, 2022), from European Application No. 19808293.5, 8 sheets.Applicant
- Emanuel A. P. Habets and Jacob Benesty, “A Two-Stage Beamforming Approach for Noise Reduction and Dereverberation”, IEEE Transactions on Audio, Speeach, and Language Processing, vol. 21, No. 5, May 2013, pp. 945-958.Applicant
- Gerhard Doblinger, “An Adaptive Microphone Array for Optimum Beamforming and Noise Reduction”, 14th European Signal Processing Conference (EUSIPCO 2006), Florence, Italy, Sep. 4-8, 2006, 5 sheets.Applicant
- Taylor B. Spalt, Christopher R. Fuller, Thomas F. Brooks, William M. Humphreys, Jr., “A Background Noise Reduction Technique using Adaptive Noise Cancellation for Microphone Arrays”, p. 1-16, available at: https://ntrs.nasa.gov/search.jsp?R=20110012472 May 16, 2018T17:29:07+00:00Z.Applicant
- International Search Report and Written Opinion dated Oct. 3, 2019, from PCT/CA2019/050708, 9 sheets.Applicant
- Notice of Allowance dated Feb. 10, 2023, from U.S. Appl. No. 16/421,908, 24 sheets.Applicant
- Final Rejection dated Oct. 11, 2022, from U.S. Appl. No. 16/421,908, 26 sheets.Applicant
- Non-Final Rejection dated Apr. 20, 2022, from U.S. Appl. No. 16/421,908, 38 sheets.Applicant
- International Search Report and Written Opinion dated Jun. 15, 2023, from International Application No. PCT/CA2023/050371, 7 sheets.Applicant
- International Search Report and Written Opinion dated May 30, 2023, from International Application No. PCT/CA2023/050277, 7 sheets.Applicant
- The extended European search report dated Sep. 27, 2022, from European Patent Application No. 20749339.6, 15 sheets.Applicant
- International Search Report and Written Opinion mailed May 22, 2020, from International Application No. PCT/CA2020/050100, 11 sheets.Applicant
- Non-Final Rejection dated Dec. 10, 2020, from U.S. Appl. No. 16/774,258, 28 sheets.Applicant
- Notice of Allowance dated Jul. 26, 2021, from U.S. Appl. No. 16/774,258, 21 sheets.Applicant
- Lightspeed Technologies, “Audio Solutions for Classroom Reopening Challenges”, Duplicom Presentation Systems, https://www.duplicom.com/products/lightspeed-audio-solutions/, Aug. 27, 2020, 7 sheets.Applicant
- Luis Guerra, Troy Jensen, “How to Use The Shure MXA910 Ceiling Array Microphone for Voice Lift”, Shure Incorporated, USA, Created Sep. 2016, upated Jul. 2018, Shure Incorporated, 11 sheets.Applicant
- Alberta Infrastructure, “Sound-Field Systems Guide for Classrooms”, published in May 2004, 19 sheets.Applicant
- The extended European Search Report dated Feb. 11, 2022, from European Patent Application No. 21204322.8, 7 sheets.Applicant
- International Search and Written Opinion dated Jul. 18, 2022, from PCT/CA2022/050731, 8 sheets.Applicant
- Non-Final Office Action dated Apr. 25, 2024, from U.S. Appl. No. 17/739,926, 57 sheets.Applicant
- Non-Final Office Action dated Oct. 11, 2023, from U.S. Appl. No. 17/516,480, 40 sheets.Applicant
- Communication pursuant to Article 94(3) EPC dated Oct. 14, 2024, from European Patent Application No. 20749339.6-1207, 10 sheets.Applicant
- Notice of Allowance dated Feb. 7, 2024, from U.S. Appl. No. 17/516,480, 10 sheets.Applicant
- Communication pursuant to Article 94(3) EPC dated Feb. 29, 2024, from European Patent Application No. 21204322.8, 9 sheets.Applicant
- Communication pursuant to Article 94(3) EPC dated Mar. 12, 2024, from European Patent Application No. 19808293.5, 6 sheets.Applicant
- Theodoropoulos D et al.: “A reconfigurable beamformer for audio applications”, 7th Symposium on Application Specific Processors, 2009. SASP '09. IEEE,, Jul. 27, 2009, pp. 80-87, XP031522047, ISBN: 978-1-4244-4939-2.Applicant
- The extended European search report dated Dec. 17, 2025, from European Patent Application No. 23762644.5, 9 sheets.Applicant
- Kozintsev I et al.: “Position Calibration of Microphones and Loudspeakers in Distributed Computing Platforms”, IEEE Transactions On Speech and Audio Processing, IEEE Service Center, New York, NY, US, vol. 13, No. 1, Jan. 1, 2005 (Jan. 1, 2005), pp. 70-83, XP011123587, ISSN: 1063-6676, DOI: 10.1109/TSA.2004.838540.Applicant
- Pasi Pertila et al.: “Closed-form self-localization of asynchronous microphone arrays”, Hands-Free Speech Communication and Microphone Arrays (HSCMA), 2011 Joint Workshop On, IEEE, May 30, 2011 (May 30, 2011), pp. 139-144, XP031957280, DOI: 10.1109/HSCMA.2011.5942380 ISBN: 978-1-4577-0997-5.Applicant
- Kovalyov Anton et al.: “Joint Calibration and Synchronization of Two Arrays of Microphones and Loudspeakers Using Particle Swarm Optimization”, IEEE Open Journal of Signal Processing, IEEE, vol. 2, Oct. 11, 2021 (Oct. 11, 2021), pp. 535-544, XP011885550, DOI: 10.1109/OJSP.2021.3118574 [retrieved on Oct. 27, 2021].Applicant
- The extended European search report dated Feb. 10, 2026, from European Patent Application No. 23773395.1, 10 sheets.Applicant