Signal processing apparatus and signal processing method

[0001] This application is a U.S. National Phase of International Patent Application No. PCT/JP2022/000834 filed on Jan. 13, 2022, which claims priority benefit of Japanese Patent Application No. JP 2021-038488 filed in the Japan Patent Office on Mar. 10, 2021. Each of the above-referenced applications is hereby incorporated herein by reference in its entirety.

[0002] The present technology relates to a signal processing apparatus, a signal processing method, and a program and, in particular, relates to a signal processing apparatus, a signal processing method, and a program that make it possible to improve precision of target sound extraction.

[0003] There has been technologies proposed to extract a sound that is desired to be extracted (hereinafter, referred to as a target sound as appropriate) from a mixed sound signal which is a mixture of the target sound and a sound that is desired to be removed (hereinafter, referred to as an interfering sound as appropriate) (e.g., see PTL 1 to PTL 3 described below.).

[PTL 1]

[0004] Japanese Patent Laid-open No. 2006-72163

[PTL 2]

Japanese Patent No. 4449871

[PTL 3]

[0005] Japanese Patent Laid-open No. 2014-219467

[0006] It is desired in such a field to improve precision of target sound extraction.

[0007] The present technology has been made in view of such a situation, and an object thereof is to make it possible to improve precision of target sound extraction.

[0008] A signal processing apparatus according to a first aspect of the present technology includes a reference signal generating section that generates a reference signal corresponding to a target sound on the basis of a mixed sound signal which is recorded with multiple microphones arranged at different positions and is a mixture of the target sound and a non-target sound, and a sound source extracting section that extracts, from the mixed sound signal of one frame or multiple frames, a signal of one frame which is similar to the reference signal and in which the target sound is more enhanced.

[0009] A signal processing method or program according to the first aspect of the present technology includes steps of generating a reference signal corresponding to a target sound on the basis of a mixed sound signal which is recorded with multiple microphones arranged at different positions and is a mixture of the target sound and a non-target sound, and extracting, from the mixed sound signal of one frame or multiple frames, a signal of one frame which is similar to the reference signal and in which the target sound is more enhanced.

[0010] In the first aspect of the present technology, a reference signal corresponding to a target sound is generated on the basis of a mixed sound signal which is recorded with multiple microphones arranged at different positions and is a mixture of the target sound and a non-target sound, and a signal of one frame which is similar to the reference signal and in which the target sound is more enhanced is extracted from the mixed sound signal of one frame or multiple frames.

[0011] A signal processing apparatus according to a second aspect of the present technology includes a reference signal generating section that generates a reference signal corresponding to a target sound on the basis of a mixed sound signal which is recorded with multiple microphones arranged at different positions and is a mixture of the target sound and a non-target sound, and a sound source extracting section that extracts, from the mixed sound signal, a signal which is similar to the reference signal and in which the target sound is more enhanced. In a case where a process of generating the reference signal and a process of extracting the signal from the mixed sound signal are performed iteratively, the reference signal generating section generates a new reference signal on the basis of the signal extracted from the mixed sound signal, and the sound source extracting section extracts the signal from the mixed sound signal on the basis of the new reference signal.

[0012] A signal processing method or program according to the second aspect of the present technology includes performing a process of generating a reference signal corresponding to a target sound on the basis of a mixed sound signal which is recorded with multiple microphones arranged at different positions and is a mixture of the target sound and a non-target sound and a process of extracting, from the mixed sound signal, a signal which is similar to the reference signal and in which the target sound is more enhanced. In a case where the process of generating the reference signal and the process of extracting the signal from the mixed sound signal are performed iteratively, the signal processing method or program includes steps of generating a new reference signal on the basis of the signal extracted from the mixed sound signal, and extracting the signal from the mixed sound signal on the basis of the new reference signal.

[0013] In the second aspect of the present technology, a process of generating a reference signal corresponding to a target sound on the basis of a mixed sound signal which is recorded with multiple microphones arranged at different positions and is a mixture of the target sound and a non-target sound and a process of extracting, from the mixed sound signal, a signal which is similar to the reference signal and in which the target sound is more enhanced are performed. In a case where the process of generating the reference signal and the process of extracting the signal from the mixed sound signal are performed iteratively, a new reference signal is generated on the basis of the signal extracted from the mixed sound signal, and the signal is extracted from the mixed sound signal on the basis of the new reference signal.

[0014] A signal processing apparatus according to a third aspect of the present technology includes a reference signal generating section that generates a reference signal corresponding to a target sound on the basis of a mixed sound signal which is recorded with multiple microphones arranged at different positions and is a mixture of the target sound and a non-target sound, and a sound source extracting section that estimates an extraction filter as a solution that optimizes an objective function that includes an extraction result being a signal which is similar to the reference signal and in which the target sound is more enhanced by the extraction filter, and an adjustable parameter of a sound source model representing similarity between the extraction result and the reference signal, the objective function reflecting independence and the similarity between the extraction result and a separation result of another imaginary sound source, and extracts the signal from the mixed sound signal on the basis of the estimated extraction filter.

[0015] A signal processing method or program according to the third aspect of the present technology includes steps of generating a reference signal corresponding to a target sound on the basis of a mixed sound signal which is recorded with multiple microphones arranged at different positions and is a mixture of the target sound and a non-target sound, estimating an extraction filter as a solution that optimizes an objective function that includes an extraction result being a signal which is similar to the reference signal and in which the target sound is more enhanced by the extraction filter, and an adjustable parameter of a sound source model representing similarity between the extraction result and the reference signal, the objective function reflecting independence and the similarity between the extraction result and a separation result of another imaginary sound source, and extracting the signal from the mixed sound signal on the basis of the estimated extraction filter.

[0016] In the third aspect of the present technology, a reference signal corresponding to a target sound is generated on the basis of a mixed sound signal which is recorded with multiple microphones arranged at different positions and is a mixture of the target sound and a non-target sound. An extraction filter is extracted as a solution that optimizes an objective function that includes an extraction result being a signal which is similar to the reference signal and in which the target sound is more enhanced by the extraction filter, and an adjustable parameter of a sound source model representing similarity between the extraction result and the reference signal, the objective function reflecting independence and the similarity between the extraction result and a separation result of another imaginary sound source. The signal is extracted from the mixed sound signal on the basis of the estimated extraction filter.

[Notation in Present Specification]

(Notation of Formulae)

[0034] Note that formulae are explained hereinbelow in accordance with the notation described below.

• • [0035] conj(X) represents the complex conjugate of a complex number X. In formulae, the complex conjugate of X is represented by X with a line over it.
  • [0036] Assignments of values are represented by “=” or “←.” In particular, an operation in which the equality between both sides does not hold true (e.g., “x←x+1”) is always represented by “←.”
  • [0037] Matrices are represented by capital letters, and vectors and scalars are represented by lowercase letters. In addition, in formulae, matrices and vectors are represented by thick letters, and scalars are represented by italics.

[0038] In the present specification, “sounds (sound signals)” and “voices (voice signals)” are used with different meanings. “Sounds” are used as a term having a typical meaning such as sounds or audio, and “voices” are used as a term representing voices or speeches.

[0039] In addition, “separation” and “extraction” are used with different meanings as follows. “Separation” is the opposite of mixing, and is used as a term meaning dividing a signal which is a mixture of multiple raw signals into the respective raw signals (there are both multiple inputs and multiple outputs). “Extraction” is used as a term meaning taking out one raw signal from a signal which is a mixture of multiple raw signals (there are multiple inputs, but there is one output).

[0040] “Applying a filter” and “filtering” have the same meaning, and similarly, “applying a mask” and “masking” have the same meaning.

[0041] To start with, in order to facilitate understanding of the present disclosure, an overview and the background of the present disclosure and problems that should be considered in the present disclosure are explained.

[0042] The present disclosure relates to sound source extraction using reference signals (references). In addition to recording, with multiple microphones, of a signal which is a mixture of a sound that is desired to be extracted (target sound) and a sound that is desired to be eliminated (interfering sound), a signal processing apparatus generates a “rough” amplitude spectrogram corresponding to the target sound, and uses the amplitude spectrogram as a reference signal to thereby generate an extraction result which is similar to and more precise than the reference signal. That is, an embodiment of the present disclosure is a signal processing apparatus that extracts, from the mixed sound signal, a signal which is similar to the reference signal and in which the target sound is more enhanced.

[0043] In a process performed at the signal processing apparatus, an objective function reflecting both the similarity between the reference signal and the extraction result and the independence between the extraction result and another imaginary separation result is prepared, and an extraction filter is determined as a solution that optimizes the objective function. With use of a deflation method used in blind sound source separation, a signal to be output can be a signal of only one sound source corresponding to the reference signal. Since it can be regarded as a beamformer considering both the similarity and the independence, it is referred to as a Similarity-and-Independence-aware Beamformer (SIBF) as appropriate hereinbelow.

[0044] The present disclosure relates to sound source extraction using reference signals (references). In addition to recording, with multiple microphones, of a signal which is a mixture of a sound that is desired to be extracted (target sound) and a sound that is desired to be eliminated (interfering sound), a “rough” amplitude spectrogram corresponding to the target sound is acquired or generated, and the amplitude spectrogram is used as a reference signal to thereby generate an extraction result which is similar to and more precise than the reference signal.

[0045] It is assumed that assumed situations where the present disclosure is used satisfy all of conditions (1) to (3) described below, for example.

[0046] (1) An observation signal is recorded synchronously with multiple microphones.

[0047] (2) It is assumed that a zone, that is, a time range, in which a target sound is being produced is known, and the observation signal described before includes at least the zone.

[0048] (3) It is assumed that, as a reference signal, a rough amplitude spectrogram (rough target sound spectrogram) corresponding to the target sound has been acquired or can be generated from the observation signal described before.

[0049] Each of the conditions described above is explained supplementarily.

[0050] Regarding the condition (1) described above, the microphones may be or may not be fixed, and in either case, the positions of the microphones and sound sources may be unknown. Examples of fixed microphones include a microphone array, and conceivable examples of unfixed microphones include pin microphones or the like that are worn by speakers.

[0051] Regarding the condition (2) described above, the zone in which the target sound is being produced is an utterance zone in a case where, for example, a voice of a particular speaker is to be extracted. It is assumed that, while the zone is known, it is unknown whether or not the target sound is being produced outside the zone. That is, the hypothesis that there is no target sound outside the zone does not hold true in some cases.

[0052] Regarding (3) described above, the rough target sound spectrogram means a spectrogram that has deteriorated as compared with the true target sound spectrogram since the spectrogram meets one or more of the following conditions a) to f).

[0053] a) It is data of real numbers not including phase information.

[0054] b) The target sound is dominant, but an interfering sound is also included.

[0055] c) The interfering sound has mostly been removed, but the sound is distorted as a side effect of the removal.

[0056] d) In the time direction and/or frequency direction, the resolution has deteriorated as compared with the true target sound spectrogram.

[0057] e) Unlike the observation signal, magnitude comparison of the scales of amplitude of the spectrograms is meaningless. For example, even if the amplitude of the rough target sound spectrogram is half the amplitude of an observation signal spectrogram, this does not necessarily mean that the target sound and the interfering sound are included in the observation signal at equal magnitude.

[0058] f) It is an amplitude spectrogram generated from a non-sound signal.

[0059] The rough target sound spectrogram described above is acquired or generated by a method like the ones below, for example.

• • [0060] A sound is recorded with a microphone placed near the target sound (e.g., a pin microphone worn by a speaker), and an amplitude spectrogram is determined from the recorded sound (this corresponds to the example of b described above).
  • [0061] A neural network (NN) to extract a particular type of sound in the amplitude spectrogram domain is trained in advance, and the observation signal is input to the neural network (this corresponds to a, c, and e described above).
  • [0062] An amplitude spectrogram is determined from a signal acquired with a sensor such as a bone conduction microphone that is different from a typically used air conduction microphone (this corresponds to c described above).
  • [0063] A linear frequency-domain spectrogram is generated by applying a predetermined transform on data equivalent to a spectrogram calculated in a non-linear frequency domain such as the Mel frequency domain (this corresponds to a, d, and e described above).
  • [0064] Instead of a microphone, a sensor that can observe vibrations of the skin surface near the mouth or throat of a speaker is used to determine an amplitude spectrogram from a signal acquired with the sensor (this corresponds to d, e, and f described above).

[0065] One object of the present disclosure is to use, as a reference signal, a rough target sound spectrogram acquired/generated in the manner described above and generate an extraction result which is more precise than the reference signal (and is closer to a true target sound). More specifically, in a sound source extraction process in which a linear filter is applied to a multi-channel observation signal to generate an extraction result, a linear filter to generate an extraction result which is more precise than a reference signal (closer to a true target sound) is estimated.

[0066] A linear filter for the sound source extraction process is estimated in the present disclosure for enjoying the following merits that the linear filter provides.

[0067] Merit 1: It provides a less distorted extraction result as compared with a non-linear extraction process. Accordingly, in a case where it is combined with voice recognition or the like, deterioration of the recognition precision due to distortions can be avoided.

[0068] Merit 2: The phase of the extraction result can be estimated appropriately by a rescaling process described later. Accordingly, in a case where it is combined with post-processing dependent on the phase (also including cases where the extraction result is reproduced as a sound and humans listen to it), it is possible to avoid problems attributable to an inappropriate phase.

[0069] Merit 3: The extraction precision can be improved easily by increasing the number of microphones.

[0070] One of objects of the present disclosure is restated as follows.

[0071] Object: Assuming that the following conditions a) to c) are satisfied, a linear filter for generating an extraction result which is more precise than a signal of c) is estimated.

[0072] a) There is a signal recorded with multi-channel microphones. The arrangement of the microphones and the position of each sound source may be unknown.

[0073] b) A zone in which a target sound (sound that is desired to be kept) is being produced is known. Note that it is unknown whether or not there is a target sound also outside the zone.

[0074] c) A rough amplitude spectrogram of the target sound (or data similar to it) has been acquired or can be generated. The amplitude spectrogram includes real numbers, and the phase cannot be known.

[0075] However, conventionally there have been no linear filtering schemes that satisfy all of the three conditions described above. There are mainly the following three types of known typical linear filtering schemes.

• • [0076] Adaptive beamformers
  • [0077] Blind sound source separation
  • [0078] Existing linear filtering processes using reference signals

[0079] The following explains problems of the schemes.

(Problems of Adaptive Beamformers)

[0080] Adaptive beamformers described here are schemes in which a linear filter for extracting a target sound is adaptively estimated with use of a signal observed with multiple microphones and information representing which sound source is to be extracted as a target sound. Adaptive beamformers include schemes described in Japanese Patent Laid-open No. 2012-234150 and Japanese Patent Laid-open No. 2006-072163, for example.

[0081] The following explains an S/N ratio (Signal to Noise Ratio) maximizing beamformer (also called a GEV beamformer) as an adaptive beamformer that can be used even in a case where the arrangement of microphones, the direction of a target sound, and the like are unknown.

[0082] The S/N ratio maximizing beamformer (maximum SNR beamformer) is a scheme to determine a linear filter that maximizes the ratio V_s/V_n between the following a) and b).

[0083] a) A variance V_s of a processing result obtained by applying a predetermined linear filter to a zone in which only a target sound is being produced.

[0084] b) A variance V_n of a processing result obtained by applying the same linear filter to a zone in which only an interfering sound is being produced.

[0085] This scheme can estimate a linear filter if the respective zones can be detected, and does not require the arrangement of microphones or the direction of the target sound.

[0086] However, in assumed situations where the present disclosure can be applied, the only known zone is a timing at which the target sound is being produced. Since there are both the target sound and the interfering sound in the zone, the zone cannot be used as none of the zones in a) and b) described above. Regarding other adaptive beamformer schemes also, for a reason that the zone in b) described above is separately necessary, for a reason that the direction of the target sound needs to be known, or for other reasons, it is difficult to use the schemes in situations where the present disclosure can be applied.

(Problems of Blind Sound Source Separation)

[0087] Blind sound source separation is a technology to estimate each sound source from a signal which is a mixture of multiple sound sources, by using only the signal observed with multiple microphones (without using information such as the directions of sound sources or the arrangement of the microphones). Examples of such a technology include the technology of Japanese Patent No. 4449871. The technology of Japanese Patent No. 4449871 is an example of technologies called independent component analysis (Independent Component Analysis; hereinbelow, referred to as ICA as appropriate), and ICA decomposes a signal observed with N microphones into N sound sources. It is sufficient if the observation signal used at that time includes a zone in which a target sound is being produced, and information regarding a zone in which only the target sound is being produced or only an interfering sound is being produced is unnecessary.

[0088] Accordingly, ICA can be used in situations where the present disclosure can be applied, by decomposing an observation signal of a zone in which a target sound is being produced into N components by applying ICA, and thereafter selecting only one component which is the most similar to a rough target sound spectrogram which is a reference signal. As a method for assessing whether or not a component is similar to the rough target sound spectrogram, it is sufficient if each separation result is transformed into an amplitude spectrogram, then the square error (Euclidean distance) between each amplitude spectrogram and the reference signal is calculated, and a separation result corresponding to an amplitude spectrogram that gives the minimum error is adopted.

[0089] However, the method of performing the selection after the separation in such a manner has the following problems.

[0090] 1) Despite the fact that a sound source that is desired to be obtained is only one sound source, the N sound sources are generated in an intermediate step, and accordingly, this is disadvantageous in terms of calculation costs and memory usage.

[0091] 2) The rough target sound spectrogram, which is the reference signal, is used only in the step of selecting one sound source from the N sound sources, but is not used in the step separating into the N sound sources. Accordingly, the reference signal does not contribute to improvement of extraction precision.

(Problems of Existing Linear Filtering Processes Using Reference Signals)

[0092] There have conventionally also been several schemes to estimate linear filters by using reference signals. Here, the following a) and b) are described as such technologies.

• • [0093] a) Independent deeply learned matrix analysis
  • [0094] b) Sound source extraction using time envelopes as reference signals

[0095] Independent deeply learned matrix analysis (hereinbelow, referred to as IDLMA as appropriate) is a developed form of independent component analysis. For details, refer to the following Document 1.

“(Document 1)

• • [0096] N. Makishima et al.,
  • [0097] “Independent Deeply Learned Matrix Analysis for Determined Audio Source Separation,”
  • [0098] in IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 27, no. 10, pp. 1601-1615, October 2019.
  • [0099] doi: 10.1109/TASLP.2019.2925450”

[0100] A feature of IDLMA is that a neural network (NN) to generate a power spectrogram (the square of an amplitude spectrogram) of each sound source that is desired to be separated is trained in advance. For example, in a case where it is desired to separate the part of each musical instrument from music including simultaneous performance of multiple musical instruments, NNs each of which outputs a musical instrument sound by receiving an input of music is trained in advance. At a time of separation, the separation is performed by inputting an observation signal to each NN, and using, as a reference signal, a power spectrogram which is an output therefrom. Accordingly, as compared with a completely blind separation process, improvement of the separation precision by a degree corresponding to the use of reference signals can be expected. Further, it has been reported also that, by re-inputting a separation result generated once to each NN, a power spectrum which is more precise than that of the initial generation is generated, and by performing separation with use of the power spectrum as a reference signal, a separation result which is more precise than the initial separation is obtained.

[0101] However, for the following reason, it is difficult to use IDLMA in situations where the present disclosure can be applied.

[0102] In IDLMA, generation of N separation results requires N different power spectrograms as reference signals. Accordingly, even if there is only one sound source of interest and other sound sources are unnecessary, reference signals need to be prepared for all the sound sources. However, in reality, this is difficult in some cases. In addition, Document 1 described above mentions only regarding a case where the number of microphones and the number of sound sources match, and does not mention how many reference signals have to be prepared in a case where the number of microphones and the number of sound sources do not match. In addition, since IDLMA is a sound source separation method, the use of IDLMA for the purpose of sound source extraction requires a step of keeping a separation result of only one sound source after N separation results are generated once. Accordingly, the problem of sound source separation that there is waste in terms of calculation costs and memory usage still remains.

[0103] Examples of sound source extraction that uses time envelopes as reference signals include, for example, a technology described in Japanese Patent Laid-open No. 2014-219467 proposed by the present inventor or other technologies. As in the present disclosure, this scheme estimates a linear filter by using a reference signal and a multi-channel observation signal. Note that there are differences in the following respects.

• • [0104] The reference signal is not a spectrogram, but a time envelope. This is equivalent to one that is obtained by making a rough target sound spectrogram uniform by applying an operation of averaging or the like on the rough target sound spectrogram in the frequency direction. Accordingly, in a case where changes of a target sound in the time direction have a feature that they are different among different frequencies, the reference signal cannot express the feature appropriately; as a result, there is a possibility that the extraction precision deteriorates.
  • [0105] The reference signal is reflected only as an initial value in an iteration process for determining an extraction filter. Since the second and subsequent iterations are not constrained by the reference signal, there is a possibility that another sound source different from the reference signal is extracted. For example, in a case where there is a sound that is generated only momentarily in a zone, it is optimum for an objective function to extract that sound, and accordingly, there is a possibility that an undesired sound is extracted depending on the number of times of iterations.

[0106] As described above, there has been a problem that it is difficult to use the technologies described above in situations where the present disclosure can be applied or sufficiently precise extraction results cannot be obtained.

[0107] Next, the technology used in the present disclosure is explained. The sound source extraction technology that meets the object of the present disclosure can be realized by introducing the following elements together to the technique of blind sound source separation based on independent component analysis.

[0108] Element 1: In a separation procedure, an objective function that reflects not only the independence among separation results but also the similarity between one of the separation results and a reference signal is prepared, and is optimized.

[0109] Element 2: Also in the separation procedure, a technique called a deflation method of separating sound sources one at a time is introduced. Then, the separation process is exited at a time point when the first sound source has been separated.

[0110] The sound source extraction technology according to the present disclosure extracts one desired sound source by applying an extraction filter which is a linear filter from a multi-channel observation signal observed with multiple microphones. Accordingly, it can be regarded as one type of beamformer (BF). Both the similarity between a reference signal and an extraction result and the independence between the extraction result and another separation result are reflected in an extraction procedure. In view of this, the sound source extraction scheme according to the present disclosure is referred to as a Similarity-and-Independence-aware Beamformer (SIBF) as appropriate.

[0111] The separation procedure according to the present disclosure is explained with use of FIG. 1. The frame which is given (1-1) surrounds a separation procedure that is assumed to be performed in conventional time-frequency-domain independent component analysis (Japanese Patent No. 4449871, etc.), and (1-5) and (1-6) outside the frame are elements that are added according to the present disclosure. Hereinbelow, conventional time-frequency-domain blind sound source separation is explained with use of the separation procedure surrounded by the frame (1-1) first, and the separation procedure according to the present disclosure is explained next.

[0112] In FIG. 1, X₁ to X_N are observation signal spectrograms (1-2) each corresponding to one of N microphones. These are complex number data, and are generated by applying the short-time Fourier transform described later to the waveform of a sound observed with each microphone. In each spectrogram, the vertical axis represents frequency, and the horizontal axis represents time. It is assumed that the time length is the same as or longer than the length of time in which a target sound that is desired to be extracted is being produced.

[0113] In the independent component analysis, separation result spectrograms Y₁ to Y_N (1-4) are generated by multiplying the observation signal spectrograms by a predetermined square matrix called a separation matrix which is given (1-3). The number of the separation result spectrograms is N, and is the same as the number of the microphones. In the separation, values of the separation matrix are decided such that Y₁ to Y_N become statistically independent (i.e., differences among Y₁ to Y_N are maximized). Since such a matrix cannot be determined with a single operation, an objective function reflecting the independence among the separation result spectrograms is prepared, and such a separation matrix that the function is optimized (maximized or minimized depending on the property of the objective function) is determined iteratively. After the separation matrix and results of the separation result spectrograms are determined, the inverse Fourier transform is applied to each of the separation result spectrograms to generate a waveform which is a signal representing a corresponding estimated sound source before being mixed.

[0114] The separation procedure of conventional time-frequency-domain independent component analysis has been explained thus far. According to the present disclosure, the two elements described before are added to the separation process.

[0115] One of the additional element is the similarity with a reference signal. The reference signal is a rough amplitude spectrogram of a target sound, and is generated by a reference signal generating section which is given (1-5). In the separation procedure, in addition to the independence among the separation result spectrograms, the separation matrix is decided by taking into consideration also the similarity between Y₁, which is one of the separation result spectrograms, and a reference signal R. That is, the objective function is prepared to reflect both of the following, and a separation matrix that optimizes the function is determined.

[0116] a) The independence among Y₁ to Y_N (solid line L1)

[0117] b) The similarity between Y₁ and R (dotted line L2) A specific formula of the objective function is described later.

[0118] By preparing the objective function to reflect both the independence and the similarity, the following merits can be attained.

[0119] Merit 1: It is indefinite in typical time-frequency-domain independent component analysis at which position of the separation result spectrograms which raw signal appears, and this changes depending on initial values of the separation matrix, the degrees of mixture of observation signals (signals corresponding to a mixed sound signal described later), differences between algorithms to determine separation matrices, and the like. In contrast, since the similarity between the separation result Y₁ and the reference signal R also is taken into consideration in addition to the independence in the present disclosure, a spectrogram similar to R can be caused to always appear in Y₁.

[0120] Merit 2: Merely solving a problem of simply making Y₁, which is one of separation results, similar to the reference signal R can make Y₁ closer to R, but cannot make Y₁ superior to the reference signal R (make Y₁ closer to a target sound) in terms of extraction precision. In contrast, since the independence among separation results also is taken into consideration in the present disclosure, the extraction precision of the separation result Y₁ can be made superior to the reference signal.

[0121] However, even if the similarity with the reference signal is introduced in time-frequency-domain independent component analysis, the number of signals to be generated is N since it is still a separation technique. That is, even if the desired sound source is only Y₁, (N−1) signals are simultaneously generated undesirably despite the fact they are unnecessary.

[0122] In view of this, as still another additional element, a deflation method is introduced. The deflation method is a scheme to estimate raw signals one at a time, instead of simultaneous separation of all sound sources. For a general explanation regarding the deflation method, refer to Chapter 8 of the following Document 2, for example.

“(Document 2)

• • [0123] Detailed Explanation: Independent Component Analysis—New World of Signal Analysis
  • [0124] Aapo Hyv″arinen (Author), Erkki Oja (Author), Juha Karhunen (Author),
  • [0125] Aapo Hyv″arinen (Author), Erkki Oja (Author), Juha Karhunen (Author),
  • [0126] Nemoto Iku (Translator), Kawakatsu Masaki (Translator) (Original Title)
  • [0127] Independent Component Analysis
  • [0128] Aapo Hyvarinen (Author), Juha Karhunen (Author), Erkki Oja (Author)”

[0129] Since the order of separation results is typically indefinite even in the deflation method, it is indefinite at which position a desired sound source appears. However, by applying the deflation method to sound source separation using an objective function reflecting both the independence and the similarity as described above, it becomes possible to cause a separation result similar to a reference signal to appear always at the first position. That is, it is sufficient if the separation process is exited at a time point when the first one sound source has been separated (estimated), and it becomes unnecessary to generate unnecessary (N−1) separation results. In addition, it is unnecessary to estimate all elements of the separation matrix, and it is sufficient if only elements that are necessary for generating Y₁ among all the elements are estimated.

[0130] In the deflation method in which only one sound source is estimated, separation results (i.e., Y₂ to Y_N) other than Y₁ among separation results which are given (1-4) in FIG. 1 are imaginary separation results, and are not actually generated. However, for a calculation of the independence, a calculation which is equivalent to one that is performed by using all the separation results, Y₁ to Y_N, is performed. Accordingly, while a merit of sound source separation which can make Y₁ more precise than R can be attained by taking the independence into consideration, it is also possible to avoid a wasteful task of generating unnecessary separation results, Y₂ to Y_N.

[0131] The deflation method is one of schemes for separation (estimation of all sound sources before mixing), but in a case where separation is suspended at a time point when one sound source has been estimated, it can be used as a scheme for extraction (estimation of one desired sound source). In view of this, in the following explanation, an operation to estimate only the separation result Y₁ is called “extraction,” and Y₁ is referred to as a “(target sound) extraction result” as appropriate. Further, each separation result is generated from a vector included in the separation matrix which is given (1-3). This vector is referred to as an “extraction filter” as appropriate.

[0132] A sound source extraction scheme using a reference signal based on the deflation method is explained with use of FIG. 2. FIG. 2 depicts details of FIG. 1, and elements necessary for the application of the deflation method are added.

[0133] Observation signal spectrograms which are given (2-1) in FIG. 2 are identical to (1-2) in FIG. 1, and are generated by applying the short-time Fourier transform to a time-domain signal observed with N microphones. By applying a process called uncorrelation which is given (2-2) to the observation signal spectrograms, uncorrelated observation signal spectrograms which are given (2-3) are generated. Uncorrelation is also called whitening, and is a transform to make signals that are observed with the microphones uncorrelated. Specific formulae used in the process are described later. By performing uncorrelation as pre-processing of separation, it becomes possible to apply, in the separation, an efficient algorithm using the property of uncorrelated signals. The deflation method is one of such algorithms.

[0134] The number of the uncorrelated observation signal spectrograms is the same as the number of the microphones, and the uncorrelated observation signal spectrograms are denoted by U₁ to U_N. It is sufficient if the generation of the uncorrelated observation signal spectrograms is performed once as a process performed before determining an extraction filter. As explained with reference to FIG. 1, in the deflation method, filters to generate the separation results Y₁ to Y_N are estimated one at a time, instead of estimation of a matrix to generate the separation results simultaneously. Since only Y₁ is generated in the present disclosure, a filter to be estimated is only w₁ having the function of generating Y₁ by receiving an input of U₁ to U_N, and Y₂ to Y_N and w₂ to w_N are imaginary ones that are not actually generated.

[0135] A reference signal R which is given (2-8) is identical to (1-6) in FIG. 1. As described before, in the estimation of the filter w₁, both the independence among Y₁ to Y_N and the similarity between R and Y₁ are taken into consideration.

[0136] In the sound source extraction method according to the present disclosure, only one sound source is estimated (extracted) for one zone. Accordingly, in a case where there are multiple sound sources that are desired to be extracted, that is, target sounds, and moreover there are overlapping zones in which the target sounds are being produced, each of the overlapping zones is detected, a reference signal is generated for each of the zones, and sound source extraction is then performed. This is explained with use of FIG. 3.

[0137] In an example depicted in FIG. 3, target sounds are human voices, and the number of sound sources of the target sounds, that is, the number of speakers, is two. Needless to say, target sounds may be any type of voice, and the number of sound sources also is not limited to two. In addition, it is assumed that there are zero or more interfering sounds which are not treated as the subject of extraction. It is assumed that a non-voice signal is an interfering sound but a sound output from equipment such as a speaker unit is treated as an interfering sound even if it is a voice.

[0138] The two speakers are defined as a speaker 1 and a speaker 2. In addition, an utterance which is given (3-1) and an utterance which is given (3-2) in FIG. 3 are utterances by the speaker 1. In addition, an utterance which is given (3-3) and an utterance which is given (3-4) in FIG. 3 are utterances by the speaker 2. (3-5) represents an interfering sound. In FIG. 3, the vertical axis represents differences of the positions of the sound sources, and the horizontal axis represents time. The utterances (3-1) and (3-3) have partially overlapping utterance zones. For example, this corresponds to a case where the speaker 2 starts uttering immediately before the speaker 1 finishes speaking. The utterances (3-2) and (3-4) also have overlapping zones, and, for example, this corresponds to a case where the speaker 2 makes a short utterance such as a quick response while the speaker 1 is making a long utterance. Both are phenomena that occur frequently in human conversations.

[0139] First, extraction of the utterance (3-1) is considered. In a time range (3-6) in which the utterance (3-1) is being made, there are a total of three sound sources which are a part of the utterance (3-3) by the speaker 2 and a part of the interfering sound (3-5) in addition to the utterance (3-1) by the speaker 1. Extraction of the utterance (3-1) in the present disclosure is to generate (estimate) a signal which is as close as possible to a clean sound (including only the voice of the speaker 1 and not including other sound sources), by using a reference signal, that is, a rough amplitude spectrogram, corresponding to the utterance (3-1), and an observation signal in the time range (3-6) (a mixture of the three sound sources).

[0140] Similarly, extraction of the utterance (3-3) by the speaker 2 is estimation of a signal close to a clean sound of the speaker 2 by using a reference signal corresponding to (3-3) and an observation signal in a time range (3-7). In such a manner, even if utterance zones overlap, different extraction results can be generated according to the present disclosure, provided that reference signals corresponding to the respective target sounds can be prepared.

[0141] Likewise, the time range of the utterance (3-4) by the speaker 2 is covered completely by the time range of the utterance (3-2) by the speaker 1, but different extraction results can be generated by preparing different reference signals therefor. That is, in order to extract the utterance (3-2), a reference signal corresponding to the utterance (3-2) and an observation signal of the time range (3-8) are used, and in order to extract the utterance (3-4), a reference signal corresponding to the utterance (3-4) and an observation signal of the time range (3-9) are used.

[0142] Next, an objective function to be used for estimation of a filter and an algorithm that optimizes the objective function are explained with use of formulae.

[0143] An observation signal spectrogram X_k corresponding to a k-th microphone is represented by a matrix having, as its elements, X_k(f,t) as represented by the following Formula (1).

[0144] $\left[\mathrm{Math}.1\right]$ $\begin{array}{cc}{x}_k=\left[\begin{array}{ccc}{x}_k\left(1,1\right)& \dots & x_k\left(1,T\right)\\ ⋮& \ddots & ⋮\\ x_k\left(F,1\right)& \dots & x_k\left(F,T\right)\end{array}\right]& \left(1\right)\end{array}$

[0145] In Formula (1), f means a frequency bin number, t means a frame number, and both f and t are indexes that appear by the short-time Fourier transform. Hereinbelow, changing f is expressed as a “frequency direction,” and changing t is expressed as a “time direction.”

[0146] An uncorrelated observation signal spectrogram U_k and a separation result spectrogram Y_k also are similarly expressed as matrices having, as their elements, U_k(f,t) and Y_k(f,t), respectively (descriptions of the formulae are omitted).

[0147] In addition, a vector x(f,t) having, as its elements, observation signals of all microphones (all channels) with particular f and t is represented by the following Formula (2).

[0148] $\left[\mathrm{Math}.2\right]$ $\begin{array}{cc}{x}_k=\left[\begin{array}{c}{x}_1\left(f,t\right)\\ ⋮\\ x_N\left(f,t\right)\end{array}\right]& \left(2\right)\end{array}$

[0149] Regarding an uncorrelated observation signal and a separation result also, vectors, u(f,t) and y(f,t), having the same shape are prepared (descriptions of the formulae are omitted).

[0150] The following Formula (3) is a formula for determining the vector u(f,t) of the uncorrelated observation signal.
[Math. 3]
u(f,t)=P(f)x(f,t)  (3)

[0151] This vector is generated by the product of P(f) called an uncorrelation matrix and the observation signal vector x(f,t). The uncorrelation matrix P(f) is calculated by the following Formula (4) to Formula (6).

[0152] $\begin{array}{cc}\left[\mathrm{Math}.4\right]& \\ R_{\mathrm{xx}}\left(f\right)={\langle x\left(f,t\right){x\left(f,t\right)}^{{ }^{}H}\rangle }_t& \left(4\right)\end{array}$ $\begin{array}{cc}\left[\mathrm{Math}.5\right]& \\ R_{\mathrm{xx}}\left(f\right)=V\left(f\right)D\left(f\right){V\left(f\right)}^{{ }^{}H}& \left(5\right)\end{array}$ $\begin{array}{cc}\left[\mathrm{Math}.6\right]& \\ P\left(f\right)={D\left(f\right)}^{-\frac{1}{2}}{V\left(f\right)}^{{ }^{}H}& \left(6\right)\end{array}$

[0153] Formula (4) described above is a formula for determining a covariance matrix R_xx(f) of an observation signal at an f-th frequency bin. <⋅>_t of the right side represents an operation to calculate the average at t (frame number) in a predetermined range. In the present disclosure, the range of t is a time length of a spectrogram, that is, a zone in which a target sound is being produced (or a range including the zone). In addition, the superscript H represents the Hermitian transpose (complex conjugate transpose).

[0154] Eigen decomposition is applied to the covariance matrix R_xx(f), and the covariance matrix R_xx(f) is decomposed into the product of three terms like the right side of Formula (5). V(f) is a matrix including eigenvectors, and D(f) is a diagonal matrix including eigenvalues. V(f) is a Hermitian matrix, and the inverse matrix of V(f) and the Hermitian transpose of V(f) are identical.

[0155] The uncorrelation matrix P(f) is calculated in accordance with Formula (6). Since D(f) is a diagonal matrix, the −½th power of D(f) is determined by raising each diagonal element to the −½th power.

[0156] Since the thus-determined uncorrelated observation signal u(f,t) has elements that are uncorrelated with each other, values of a covariance matrix calculated in accordance with the following Formula (7) are an identity matrix I.
[Math. 7]
u(f,t)u(f,t)^Ht=I  (7)

[0157] The following Formula (8) is a formula for generating a separation result y(f,t) for all channels at f and t, and is determined as the product of a separation matrix W(f) and u(f,t). A method to determine W(f) is described later.
[Math. 8]
y(f,t)=W(f)u(f,t)=W(f)P(f)x(f,t)  (8)

[0158] Formula (9) is a formula for generating only a k-th separation result, and w_k(f) is a k-th row vector of the separation matrix W(f). Since only Y₁ is generated as an extraction result in the present disclosure, Formula (9) is basically used only with k=1.
[Math. 9]
y_k(f,t)=w_k(f)u(f,t)=w_k(f)P(f)x(f,t)  (9)

[0159] It has been proven that it is sufficient if, in a case where uncorrelation is performed as pre-processing of the separation, the separation matrix W(f) is found from a unitary matrix. In a case where the separation matrix W(f) is a unitary matrix, the following Formula (10) is satisfied, and also the row vector w_k(f) included in W(f) satisfies the following Formula (11). By using this feature, separation by the deflation method becomes possible (similarly to Formula (9), Formula (11) is basically used only with k=1).
[Math. 10]
W(f)W(f)^H=I  (10)
[Math. 11]
w_k(f)w_k(f)^H=1  (11)

[0160] The reference signal R is represented by a matrix having, as its elements, r(f,t) as in Formula (12). The shape itself is the same as the observation signal spectrogram X_k, but elements r(f,t) of R are non-negative real numbers while elements X_k(f,t) of X_k are complex number values.

[0161] $\begin{array}{cc}\left[\mathrm{Math}.12\right]& \\ R=\left[\begin{array}{ccc}r\left(1,1\right)& \dots & r\left(1,T\right)\\ ⋮& \ddots & ⋮\\ r\left(F,1\right)& \dots & r\left(F,T\right)\end{array}\right]& \left(12\right)\end{array}$

[0162] According to the present disclosure, instead of estimation of all elements of the separation matrix W(f), only w₁(f) is estimated. That is, only an element used in generation of the first separation result (target sound extraction result) is estimated. Hereinbelow, derivation of a formula for estimating w₁(f) is explained. The derivation of the formula includes the following three points, and these are explained in order.

• • [0163] (1) Objective Function
  • [0164] (2) Sound Source Model
  • [0165] (3) Update Formula
    (1) Objective Function

[0166] An objective function used in the present disclosure is a negative log likelihood, and is basically the same as the one used in Document 1 or the like. This objective function gives the minimum value when separation results are mutually independent. Note that, since the objective function is prepared to reflect also the similarity between an extraction result and a reference signal in the present disclosure, the objective function is derived as follows.

[0167] In order to prepare the objective function to reflect the similarity described above, formulae for uncorrelation and separation (extraction) are revised slightly. Formula (13) is a revised version of Formula (3), which is a formula for uncorrelation, and Formula (14) is a revised version of Formula (8), which is a formula for separation. In both of them, the reference signal r(f,t) is added to vectors on both sides, and an element 1 representing the “pass-through of the reference signal” is added to the matrix of the right side. The matrices and vectors having these additional elements are expressed with a prime symbol given to the original matrices and vectors.

[0168] $\begin{array}{cc}\left[\mathrm{Math}.13\right]& \\ \left[\begin{array}{c}r\left(f,t\right)\\ u\left(f,t\right)\end{array}\right]=\left[\begin{array}{cc}1& 0\\ 0& P\left(f\right)\end{array}\right]\left[\begin{array}{c}r\left(f,t\right)\\ x\left(f,t\right)\end{array}\right]& \left(13\right)\end{array}$ $\iff u^{{ }^{}\prime }\left(f,t\right)=P^{{ }^{}\prime }\left(f\right)x^{{ }^{}\prime }\left(f,t\right)$ $\begin{array}{cc}\left[\mathrm{Math}.14\right]& \\ \left[\begin{array}{c}r\left(f,t\right)\\ y\left(f,t\right)\end{array}\right]=\left[\begin{array}{cc}1& 0\\ 0& W\left(f\right)\end{array}\right]\left[\begin{array}{c}r\left(f,t\right)\\ u\left(f,t\right)\end{array}\right]& \left(14\right)\end{array}$ $\iff y^{{ }^{}\prime }\left(f,t\right)=W^{{ }^{}\prime }\left(f\right)u^{{ }^{}\prime }\left(f,t\right)=W^{{ }^{}\prime }\left(f\right)P^{{ }^{}\prime }\left(f\right)x^{{ }^{}\prime }\left(f,t\right)$

[0169] As the objective function, a negative log likelihood L of a reference signal and an observation signal represented by the following Formula (15) is used.
[Math. 15]
L=−log p(R,X₁, . . . ,X_N|W′)  (15)

[0170] In this Formula (15), W′ represents a set including W′(f) of all frequency bins. That is, it is a set including all parameters to be estimated. In addition, p(⋅) is a conditional probability density function (hereinbelow, referred to as a pdf as appropriate), and represents a probability that the reference signal R and the observation signal spectrograms X₁ to X_N occur simultaneously when W′ is given. In the following explanation also, in a case where multiple elements are written between the parentheses of a pdf (a case where multiple variables are written or a case where a matrix or a vector is written), it represents a probability that those elements occur simultaneously.

[0171] For optimization (minimization in this case) of the extraction filter w₁(f), the negative log likelihood L needs to be transformed such that w₁(f) is included. For this purpose, the following hypotheses are made regarding an observation signal and a separation result.

[0172] Hypothesis 1: Observation signal spectrograms have similarity in the channel direction (i.e., spectrograms corresponding to microphones resemble each other), but are independent in the time direction and frequency direction. That is, in one spectrogram, components included at points occur mutually independent, and are not influenced by other factors such as time and frequency.

[0173] Hypothesis 2: In addition to the time direction and frequency direction, separation result spectrograms are independent also in the channel direction. That is, separation-result spectrograms do not resemble each other.

[0174] Hypothesis 3: Y₁, which is a separation result spectrogram, and a reference signal have similarity. That is, both have resembling spectrograms.

[0175] A procedure of transformation of p(R, X₁, . . . , X_N|W′) is represented by Formula (16) to Formula (21).

[0176] $\begin{array}{cc}\left[\mathrm{Math}.16\right]& \\ p\left(R,X_1,\dots ,X_N❘W^{{ }^{}\prime }\right)=\prod _f\prod _{t}p\left(r\left(f,t\right),x_1\left(f,t\right),\dots ,x_N\left(f,t\right)❘W^{{ }^{}\prime }\left(f\right)\right)& \left(16\right)\end{array}$ $\begin{array}{cc}\left[\mathrm{Math}.17\right]& \\ =\prod _f\prod _{t}p\left(x^{{ }^{}\prime }\left(f,t\right)❘W^{{ }^{}\prime }\left(f\right)\right)& \left(17\right)\end{array}$ $\begin{array}{cc}\left[\mathrm{Math}.18\right]& \\ =\prod _f\prod _{t}p\left(u^{{ }^{}\prime }\left(f,t\right)❘W^{{ }^{}\prime }\left(f\right)\right)❘\det \left(P^{{ }^{}\prime }\left(f\right)\right){❘}^{{ }^{}2}& \left(18\right)\end{array}$ $\begin{array}{cc}\left[\mathrm{Math}.19\right]& \\ =\prod _f\prod _{t}p(y^{{ }^{}\prime }\left(f,t\right)❘\det \left(W^{{ }^{}\prime }\left(f\right)\right){❘}^{{ }^{}2}❘\det \left(P^{{ }^{}\prime }\left(f\right)\right){❘}^{{ }^{}2}& \left(19\right)\end{array}$ $\begin{array}{cc}\left[\mathrm{Math}.20\right]& \\ =\prod _f\prod _{t}p\left(y^{{ }^{}\prime }\left(f,t\right)\right)·\mathrm{const}.& \left(20\right)\end{array}$ $\begin{array}{cc}\left[\mathrm{Math}.21\right]& \\ =\prod _f\prod _t\left\{p\left(r\left(f,t\right),y_1\left(f,t\right)\right)\prod _{k\ge 2}p\left(y_k\left(f,t\right)\right)\right\}·\mathrm{const}.& \left(21\right)\end{array}$

[0177] In each formula described above, p(⋅) represents the probability density function of a variable between the parentheses, and, in a case where multiple elements are written, represents the joint probability of those elements. Even if the same letter p is used, different probability distributions are represented if variables between the parentheses are different, and accordingly, p(R) and p(Y₁) are different functions, for example. Since the joint probability of independent variables can be decomposed into the product of respective pdf's, the left side of Formula (16) is transformed into the right side in accordance with Hypothesis 1. The terms between the parentheses of the right side are represented by Formula (17) by using x′(f,t) introduced into Formula (13).

[0178] Formula (17) is transformed into Formula (18) and Formula (19) by using the relation in the lower line of Formula (14). In these formulae, det(⋅) represents the determinant of a matrix between the parentheses.

[0179] Formula (20) is an important transformation in the deflation method. Since the matrix W(f)′ is a unitary matrix similarly to the separation matrix W(f), its determinant is 1. In addition, since the matrix P′(f) does not change during the separation, the determinant is a constant. Accordingly, both the determinants can be written together as const (constant).

[0180] Formula (21) is a transformation unique to the present disclosure. Whereas components of y′(f,t) are r(f,t) and y₁(f,t) to y_N(f,t), in accordance with Hypothesis 2 and Hypothesis 3, the probability density function that includes these variables as arguments is decomposed into the product of p(r(f,t),y₁(f,t)), which is the joint probability of r(f,t) and y₁(f,t), and each of p(y₂(f,t)) to p(y_N(f,t)), which are the probability density functions of y₂(f,t) to y_N(f,t).

[0181] Assigning Formula (21) to Formula (15) gives Formula (22).

[0182] $\begin{array}{cc}\left[\mathrm{Math}.22\right]& \\ L=-\sum _f\sum _t\left\{\log p\left(r\left(f,t\right),y_1\left(f,t\right)\right)+\sum _{k\ge 2}\log p\left(y_k\left(f,t\right)\right)\right\}+\mathrm{const}.& \left(22\right)\end{array}$

[0183] The extraction filter w₁(f) is a subset of arguments with which Formula (22) gives the minimum value. Since the only term in the terms of Formula (22) that includes w₁(f) is y₁(f,t) at a particular f, w₁(f) is determined as the minimum solution of the following Formula (23). Note that, in order to eliminate w₁(f)=0, which is a self-evident solution, a constraint represented by Formula (11) that the norm of a vector is 1 is placed.

[0184] $\begin{array}{cc}\left[\mathrm{Math}.23\right]& \\ w_1\left(f\right)=\arg \underset{w_1\left(f\right)}{\min }L=\arg \underset{w_1\left(f\right)}{\min }\left\{-\sum _t\log p\left(r\left(f,t\right)y_1\left(f,t\right)\right)\right\}& \left(23\right)\end{array}$

[0185] In a case where the extraction filter on which the constraint that the norm is 1 is placed is applied to an uncorrelated observation signal, the scale of each frequency bin of an extraction result to be generated is different from the scale of a true target sound. Accordingly, after a filter is estimated, an extraction filter and an extraction result are corrected for each frequency bin. Such post-processing is called rescaling. Specific formulae of the rescaling are described later.

[0186] In order to solve the minimization problem of Formula (23), the following two points need to be clarified.

• • [0187] What type of formula is allocated as p(r(f,t),y₁(f,t)), which is the joint probability of r(f,t) and y₁(f,t). This probability density function is called a sound source model.
  • [0188] What type of algorithm is used to determine the minimum solution w₁(f). Basically, w₁(f) cannot be determined with a single operation, and needs to be updated iteratively. A formula for updating w₁(f) is called an update formula. Hereinbelow, each of these is explained.
    (2) Sound Source Model

[0189] The sound source model p(r(f,t),y₁(f,t)) is a pdf having, as its arguments, two variables which are the reference signal r(f,t) and the extraction result y₁(f,t), and represents the similarity of the two variables. Sound source models can be formulated on the basis of various concepts. The following three manners are used in the present disclosure.

• • [0190] a) Bivariate Spherical Distribution
  • [0191] b) Model Based on Divergence
  • [0192] c) Time-Frequency-Varying Variance Model

[0193] Hereinbelow, each of these is explained.

a) Bivariate Spherical Distribution

[0194] Spherical distributions are a type of multi-variate pdf. Multiple arguments of a pdf are regarded as a vector, and by assigning the norm of the vector (L2 norm) to a univariate (univariate) pdf, a multi-variate pdf is formed. If spherical distributions are used in independent component analysis, an advantage of making variables that are used in arguments similar to each other is attained. For example, the technology described in Japanese Patent No. 4449871 uses this property to solve a problem called the frequency permutation problem that “sound sources to appear in k-th separation results differ between different frequency bins.”

[0195] If a spherical distribution having, as its arguments, a reference signal and an extraction result is used as a sound source model according to the present disclosure, it is possible to make both of them similar to each other. The spherical distribution used here can be represented by a general form as in the following Formula (24). In this formula, a function F is any univariate pdf. In addition, c₁ and c₂ are positive constants, and by changing these values, it is possible to adjust the influence of the reference signal on an extraction result. If a Laplace distribution is used as a univariate pdf similarly to Japanese Patent No. 4449871, the following Formula (25) is obtained. Hereinafter, this formula is called a bivariate Laplace distribution.
[Math. 24]
p(r(f,t),y₁(f,t))∝F√{square root over ((c₁r(f,t)²+c₂|y₁(f,t)|²))}  (24)
[Math. 25]
p(r(f,t),y₁(f,t))∝exp(−√{square root over (c₁r(f,t)²+c₂|y₁(f,t)|²))}  (25)
b) Model Based on Divergence

[0196] Another type of sound source model is pdf's based on divergence, which is a superordinate concept of distance scale, and is represented in the form of the following Formula (26). In this formula, divergence(r(f,t), |y₁(f,t)|) represents any divergence between r(f,t), which is a reference signal, and |y₁(f,t)|, which is the amplitude of an extraction result.
[Math. 26]
p(r(f,t),y₁(f,t))∝exp(−α·divergence(r(f,t),|y₁(f,t)|))  (26)

[0197] In addition, α is a positive constant, and is a correction term for making the right side of Formula (26) satisfy conditions of a pdf, and there are no problems even if α=1 since the value of α is unrelated in the minimization problem of Formula (23). When this pdf is assigned to Formula (23), the formula becomes equivalent to the problem of minimizing the divergence between r(f,t) and |y₁(f,t)|, and accordingly, both of them are necessarily similar to each other.

[0198] In a case where the Euclidean distance is used as a divergence, the following Formula (27) is obtained. In addition, in a case where the Itakura-Saito divergence is used, the following Formula (28) is obtained. Since the Itakura-Saito divergence is the distance scale between power spectrums, squared values are used for both r(f,t) and |y₁(f,t)|. On the other hand, a distance scale similar to the Itakura-Saito divergence may be calculated for amplitude spectrums, and in that case, the following Formula (29) is obtained.

[0199] $\begin{array}{cc}\left[\mathrm{Math}.27\right]& \\ p\left(r\left(f,t\right),y_1\left(f,t\right)\right)\propto \exp \left\{-(r\left(f,t\right)-{❘y_1\left(f,t\right)❘}^{{ }^{}2}\right\}& \left(27\right)\end{array}$ $\begin{array}{cc}\left[\mathrm{Math}.28\right]& \\ p\left(r\left(f,t\right),y_1\left(f,t\right)\right)\propto \exp \left\{-\left(\frac{{❘y_1\left(f,t\right)❘}^{{ }^{}2}}{{r\left(f,t\right)}^{{ }^{}2}}-\log \frac{{❘y_1\left(f,t\right)❘}^{{ }^{}2}}{{r\left(f,t\right)}^{{ }^{}2}}-1\right)\right\}& \left(28\right)\end{array}$ $\begin{array}{cc}\left[\mathrm{Math}.29\right]& \\ p\left(r\left(f,t\right),y_1\left(f,t\right)\right)\propto \exp \left\{-\left(\frac{{❘y_1\left(f,t\right)❘}^{ }}{{{r\left(f,t\right)}^{ }}^{ }}-\log \frac{{❘y_1\left(f,t\right)❘}^{ }}{{r\left(f,t\right)}^{ }}-1\right)\right\}& \left(29\right)\end{array}$

[0200] The following Formula (30) is another pdf based on divergence. Since the ratio approaches 1 as the similarity between r(f,t) and |y₁(f,t)| increases, the square error between the ratio and 1 functions as a divergence.

[0201] $\begin{array}{cc}\left[\mathrm{Math}.30\right]& \\ p\left(r\left(f,t\right),y_1\left(f,t\right)\right)\propto \exp \left\{-{\left(\frac{❘y_1\left(f,t\right)❘}{r\left(f,t\right)}-1\right)}^{{ }^{}2}\right\}& \left(30\right)\end{array}$
C) Time-Frequency-Varying Variance Model

[0202] As another sound source model, the time-frequency-varying variance (TFVV) model also is possible. This is a model in which points included in a spectrogram have different variances or standard deviations for different times and frequencies. Then, it is interpreted that a rough amplitude spectrogram which is a reference signal represents the standard deviation of each point (or some value dependent on the standard deviation).

[0203] If a Laplace distribution having time-frequency-varying variance (hereinafter, a TFVV Laplace distribution) is hypothesized as a distribution, it can be represented by the following Formula (31). In this formula, α is a correction term for making the right side satisfy conditions of a pdf similarly to Formula (26), and there are no problems even if α=1. β is a term for adjusting the magnitude of influence of a reference signal on an extraction result. A true TFVV Laplace distribution corresponds to a case where β=1, but other values such as ½ or 2 may be used.

[0204] $\begin{array}{cc}\left[\mathrm{Math}.31\right]& \\ p\left(r\left(f,t\right),y_1\left(f,t\right)\right)\propto \frac{1}{{r\left(f,t\right)}^{{ }^{}\beta }}\exp \left(-\alpha \frac{❘y_1\left(f,t\right)❘}{{r\left(f,t\right)}^{{ }^{}\beta }}\right)& \left(31\right)\end{array}$

[0205] Similarly, when a TFVV Gaussian distribution is hypothesized, the following Formula (32) is obtained. On the other hand, when a TFVV Student-t distribution is hypothesized, a sound source model of the following Formula (33) is obtained.

[0206] $\begin{array}{cc}\left[\mathrm{Math}.32\right]& \\ p\left(r\left(f,t\right),y_1\left(f,t\right)\right)\propto \frac{1}{{r\left(f,t\right)}^{{ }^{}\beta }}\exp \left(-\alpha \frac{{❘y_1\left(f,t\right)❘}^{{ }^{}2}}{{r\left(f,t\right)}^{{ }^{}2\beta }}\right)& \left(32\right)\end{array}$ $\begin{array}{cc}\left[\mathrm{Math}.33\right]& \\ p\left(r\left(f,t\right),y_1\left(f,t\right)\right)\propto \frac{1}{{r\left(f,t\right)}^{{ }^{}2}}{\left(1+\frac{2}{v}\frac{{❘y_1\left(f,t\right)❘}^{{ }^{}2}}{{r\left(f,t\right)}^{{ }^{}2}}\right)}^{-\frac{2+v}{2}}& \left(33\right)\end{array}$

[0207] ν (nu) in Formula (33) is a parameter called the degree of freedom, and the shape of the distribution can be changed by changing its value. For example, ν=1 represents a Cauchy distribution, and ν→∞ represents a Gaussian distribution.

[0208] The sound source models of Formula (32) and Formula (33) are used also in Document 1, but the present disclosure is different in that those models are used not for separation but for extraction.

(3) Update Formula

[0209] In many cases, there is no closed form solution (a solution without iteration) regarding the solution w₁(f) of the minimization problem of Formula (23), and an iterative algorithm needs to be used (n.b. there is a closed form solution as described later in a case where the TFVV Gaussian distribution of Formula (32) is used as a sound source model).

[0210] A high-speed, stable algorithm called an auxiliary function method can be applied to Formula (25), Formula (31), and Formula (33). On the other hand, another algorithm called the fixed point method can be applied to Formula (27) to Formula (30).

[0211] Hereinbelow, an update formula in a case where Formula (32) is used is explained first, and update formulae using the auxiliary function method and the fixed point method are explained next.

[0212] Assigning the TFVV Gaussian distribution represented by Formula (32) to Formula (23) and further ignoring terms unrelated to minimization gives the following Formula (34).

[0213] $\begin{array}{cc}\left[\mathrm{Math}.34\right]& \\ w_1\left(f\right)=\arg \underset{w_1\left(f\right)}{\min }\sum _t\frac{{❘y_1\left(f,t\right)❘}^{{ }^{}2}}{{r\left(f,t\right)}^{{ }^{}2\beta }}=\arg \underset{w_1\left(f\right)}{\min }{w}_1\left(f\right)\left\{\sum _t\frac{u\left(f,t\right){u\left(f,t\right)}^{{ }^{}H}}{{r\left(f,t\right)}^{{ }^{}2\beta }}\right\}{w_1\left(f\right)}^{{ }^{}H}& \left(34\right)\end{array}$ $\mathrm{subject}\mathrm{to}{w}_1\left(f\right){w_1\left(f\right)}^{{ }^{}H}=1$

[0214] This formula can be interpreted as a minimization problem of a weighted covariance matrix of u(f,t), and can be solved by using eigen decomposition (strictly speaking, the terms between the curly brackets of the right side of Formula (34) are not the weighted covariance matrix itself, but represents the product of the weighted covariance matrix and T; however, since the difference does not influence the solution of the minimization problem of Formula (34), the sigma between the curly brackets itself also is called a weighted covariance matrix hereinafter).

[0215] A function that has a matrix A as its argument and is for determining all eigenvectors by performing eigen decomposition on the matrix is represented by eig(A). When this function is used, the eigenvector of the weighted covariance matrix of Formula (34) can be written as the following Formula (35).

[0216] $\begin{array}{cc}\left[\mathrm{Math}.35\right]& \\ \left[a_{\min }\left(f\right)\dots a_{\max }\left(f\right)\right]=\mathrm{eig}\left(\sum _t\frac{u\left(f,t\right){u\left(f,t\right)}^{{ }^{}H}}{{r\left(f,t\right)}^{{ }^{}2\beta }}\right)& \left(35\right)\end{array}$
a_min(f), . . . , a_max(f) of the left side of Formula (35) are eigenvectors. a_min(f) corresponds to the smallest eigenvalue, and a_max(f) corresponds to the largest eigenvalue. The norm of each eigenvector is 1, and also is orthogonal to each other. w₁(f) that minimizes Formula (34) is the Hermitian transpose of the eigenvector corresponding to the smallest eigenvalue as represented by the following Formula (36).
[Math. 36]
w₁(f)←a_min(f)^H  (36)

[0217] Next, a method of applying the auxiliary function method to Formula (25), Formula (31), and Formula (33) to derive update formulae is explained.

[0218] The auxiliary function method is one of methods to efficiently solve optimization problems, and details thereof are described in Japanese Patent Laid-open No. 2011-175114 and Japanese Patent Laid-open No. 2014-219467.

[0219] Assigning the TFVV Laplace distribution represented by Formula (31) to Formula (23) and ignoring terms unrelated to minimization gives the following Formula (37).

[0220] $\begin{array}{cc}\left[\mathrm{Math}.37\right]& \\ w_1\left(f\right)=\arg \underset{w_1\left(f\right)}{\min }\sum _t\frac{❘y_1\left(f,t\right)❘}{{r\left(f,t\right)}^{{ }^{}\beta }}& \left(37\right)\end{array}$ $\mathrm{subject}\mathrm{to}{w}_1\left(f\right){w_1\left(f\right)}^{{ }^{}H}=1$

[0221] A solution of this minimization problem cannot be determined in a closed form.

[0222] In view of this, an inequality like Formula (38) for “pressing from above” is prepared.

[0223] $\begin{array}{cc}\left[\mathrm{Math}.38\right]& \\ ❘y_1\left(f,t\right)❘\le \frac{1}{2}\sum _t\left(\frac{{❘y_1\left(f,t\right)❘}^{{ }^{}2}}{b\left(f,t\right)}+b\left(f,t\right)\right)& \left(38\right)\end{array}$

[0224] The right side of Formula (38) is called an auxiliary function, and b(f,t) therein is called an auxiliary variable. This inequality holds true when b(f,t)=|y₁(f,t)|. If this inequality is applied to Formula (37), the following Formula (39) is obtained. Hereinafter, the right side of this inequality is written as G.

[0225] $\begin{array}{cc}\left[\mathrm{Math}.39\right]& \\ \sum _t\frac{❘y_1\left(f,t\right)❘}{{r\left(f,t\right)}^{{ }^{}\beta }}\le \frac{1}{2}\sum _t\left(\frac{{❘y_1\left(f,t\right)❘}^{{ }^{}2}}{b\left(f,t\right){r\left(f,t\right)}^{{ }^{}\beta }}+\frac{b\left(f,t\right)}{{r\left(f,t\right)}^{{ }^{}\beta }}\right)=G& \left(39\right)\end{array}$

[0226] In the auxiliary function method, the following two steps are repeated alternately to thereby solve a minimization problem fast and stably.

[0227] 1. As represented by the following Formula (40), w₁(f) is fixed, and b(f,t) to minimize G is determined.

[0228] $\left[\mathrm{Math}.40\right]$ $\begin{array}{cc}b\left(f,t\right)\leftarrow \arg \underset{b\left(f,t\right)}{\min }G=❘y_1\left(f,t\right)❘=❘w_1\left(f\right)u\left(f,t\right)❘& \left(40\right)\end{array}$

[0229] 2. As depicted in the following Formula (41), b(f,t) is fixed, and w₁(f) to minimize G is determined.

[0230] $\left[\mathrm{Math}.41\right]$ $\begin{array}{cc}\begin{array}{c}{w}_1\left(f\right)=\arg \underset{w_1\left(f\right)}{\min }G=\arg \underset{w_1\left(f\right)}{\min }\sum _t\frac{{❘y_1\left(f,t\right)❘}^2}{b\left(f,t\right){r\left(f,t\right)}^{\beta }}\\ =\arg \underset{w_1\left(f\right)}{\min }{w}_1\left(f\right)\left(\sum _t\frac{u\left(f,t\right){u\left(f,t\right)}^H}{b\left(f,t\right){r\left(f,t\right)}^{\beta }}\right){w_1\left(f\right)}^H\end{array}& \left(41\right)\end{array}$ $\mathrm{subject}\mathrm{to}{w}_1\left(f\right){w_1\left(f\right)}^H=1$

[0231] Formula (40) is minimized when the equality in Formula (38) holds true. Since the value of y₁(f,t) also changes every time w₁(f) changes, a calculation is performed by using Formula (9). Since Formula (41) is a minimization problem of a weighted covariance matrix similarly to Formula (34), it can be solved by using eigen decomposition.

[0232] When the eigenvector of the weighted covariance matrix of Formula (41) is calculated in accordance with the following Formula (42), w₁(f), which is a solution of Formula (41), is the Hermitian transpose of the eigenvector corresponding to the minimum value (Formula (36)).

[0233] $\left[\mathrm{Math}.42\right]$ $\begin{array}{cc}\left[a_{\min }\left(f\right)\dots a_{\max }\left(f\right)\right]=\mathrm{eig}\left(\sum _t\frac{u\left(f,t\right){u\left(f,t\right)}^H}{b\left(f,t\right){r\left(f,t\right)}^{\beta }}\right)& \left(42\right)\end{array}$

[0234] Note that, since both w₁(f) and y₁(f,t) are unknown in the initial execution of iteration, Formula (40) cannot be applied. In view of this, the initial value of the auxiliary variable b(f,t) is calculated in accordance with any one of the following methods.

[0235] a) A normalized value of a reference signal is used as the auxiliary variable. That is, b(f,t)=normalize(r(f,t)).

[0236] b) A tentative value is calculated as the separation result y₁(f,t), and the auxiliary variable is calculated therefrom in accordance with Formula (40).

[0237] c) A tentative value is assigned to w₁(f), and a calculation is performed in accordance with Formula (40).

[0238] normalize( ) of a) described above is a function defined by the following Formula (43), and s(t) in this formula represents a certain time-series signal. The function of normalize( ) is to normalize the mean square of the absolute value of a signal to 1.

[0239] $\left[\mathrm{Math}.43\right]$ $\begin{array}{cc}\mathrm{normalize}\left(s\left(t\right)\right)=\frac{s\left(t\right)}{\sqrt{{\langle {❘s\left(t\right)❘}^2)}_t}}& \left(43\right)\end{array}$

[0240] Conceivable examples of operations of y₁(f,t) in b) described above include an operation of selecting components corresponding to one channel in an observation signal and an operation of averaging components corresponding to all channels in an observation signal. For example, in a case where a microphone placement mode as in FIG. 5 described later is used, since there is always a microphone allocated to a speaker who is making an utterance, it is better to use an observation signal of the microphone as a tentative extraction result. Assuming that a number of a microphone is k, y₁(f,t)=normalize (x_k(f,t)).

[0241] Regarding the tentative value in c) described above, in addition to a simple method in which a vector whose all elements have an identical value is used for example, it is also possible to store a value of an extraction filter estimated regarding the previous target-sound zone and use the value as the initial value of w₁(f) when a calculation for the next target-sound zone is performed. For example, in a case where sound source extraction is performed regarding the utterance (3-2) depicted in FIG. 3, an extraction filter estimated regarding the previous utterance (3-1) of the same speaker is used as a tentative value of w₁(f) in the current extraction.

[0242] Regarding the bivariate Laplace distribution represented by Formula (25) also, a solution can be obtained similarly by using an auxiliary function. Assigning Formula (25) to Formula (23) gives the following Formula (44).

[0243] $\left[\mathrm{Math}.44\right]$ $\begin{array}{cc}{w}_1\left(f\right)=\arg \underset{w_1\left(f\right)}{\min }\sum _t\sqrt{c_1{r\left(f,t\right)}^2+c_2{❘y_1\left(f,t\right)❘}^2}& \left(44\right)\end{array}$ $\mathrm{subject}\mathrm{to}{w}_1\left(f\right){w_1\left(f\right)}^H=1$

[0244] Here, such an auxiliary function as the following Formula (45) is prepared.

[0245] $\left[\mathrm{Math}.45\right]$ $\begin{array}{cc}\sqrt{c_1{r\left(f,t\right)}^2+c_2{❘y_1\left(f,t\right)❘}^2}\le \frac{1}{2}\left(\frac{c_1{r\left(f,t\right)}^2+c_2{❘y_1\left(f,t\right)❘}^2}{b\left(f,t\right)}+b\left(f,t\right)\right)& \left(45\right)\end{array}$

[0246] Then, the step of determining the auxiliary variable b(f,t) (corresponding to Formula (40)) can be represented as Formula (46).
[Math. 46]
b(f,t)=√{square root over (c₁r(f,t)²+c₂|y₁(f,t)|²)}=√{square root over (c₁r(f,t)²+c₂|w₁(f)u(f,t)|²)}  (46)

[0247] The step of determining the extraction filter w₁(f) (corresponding to Formula (41)) can be represented as the following Formula (47).

[0248] $\left[\mathrm{Math}.47\right]$ $$\begin{gathered} \begin{array}{cc}{w}_1\left(f\right)=\arg \underset{w_1\left(f\right)}{\min }\sum _t\frac{{❘y_1\left(f,t\right)❘}^2}{b\left(f,t\right)}=\arg \underset{w_1\left(f\right)}{\min }{w}_1\left(f\right)\left(\sum _t\frac{u\left(f,t\right){u\left(f,t\right)}^H}{b\left(f,t\right)}\right){w_1\left(f\right)}^H \\ & \left(47\right)\end{array} \end{gathered}$$ $\mathrm{subject}\mathrm{to}{w}_1\left(f\right){w_1\left(f\right)}^H=1$

[0249] The minimization problem can be solved by eigen decomposition of the following Formula (48).

[0250] $\left[\mathrm{Math}.48\right]$ $\begin{array}{cc}\left[a_{\min }\left(f\right)\dots a_{\max }\left(f\right)\right]=\mathrm{eig}\left(\sum _t\frac{u\left(f,t\right){u\left(f,t\right)}^H}{b\left(f,t\right)}\right)& \left(48\right)\end{array}$

[0251] Next, the case of the TFVV Student-t distribution represented by Formula (33) is explained. Since an example in which the auxiliary function method is applied to the TFVV Student-t distribution is described in Document 1, only an update formula is described.

[0252] The step of determining the auxiliary variable b(f,t) is represented by the following Formula (49).

[0253] $\left[\mathrm{Math}.49\right]$ $\begin{array}{cc}b\left(f,t\right)\leftarrow \frac{v}{v+2}{r\left(f,t\right)}^2+\frac{2}{v+2}{❘y_1\left(f,t\right)❘}^2=\frac{v}{v+2}{r\left(f,t\right)}^2+\frac{2}{v+2}{❘w_1\left(f\right)u\left(f,t\right)❘}^2& \left(49\right)\end{array}$

[0254] The degree of freedom ν functions as a parameter for adjusting the respective degrees of influence of r(f,t), which is a reference signal, and y₁(f,t), which is an intermediate extraction result during iterations. In a case where ν=0, the reference signal is ignored, and in a case where ν is equal to or greater than 0 and smaller than 2, the influence of the extraction result is greater than the influence of the reference signal. In a case where ν is greater than 2, the influence of the reference signal is greater, and in a case where ν→∞, which means ν is an infinite value, the extraction result is ignored, and this is equivalent to a TFVV Gaussian distribution.

[0255] The step of determining the extraction filter w₁(f) is represented by the following Formula (50).

[0256] $\left[\mathrm{Math}.50\right]$ $\begin{array}{cc}{w}_1\left(f\right)=\arg \underset{w_1\left(f\right)}{\min }\sum _r\frac{{❘y_1\left(f,t\right)❘}^2}{b\left(f,t\right)}=\arg \underset{w_1\left(f\right)}{\min }{w}_1\left(f\right)\left(\sum _t\frac{u\left(f,t\right){u\left(f,t\right)}^H}{b\left(f,t\right)}\right){w_1\left(f\right)}^H& \left(50\right)\end{array}$ $\mathrm{subject}\mathrm{to}{w}_1\left(f\right){w_1\left(f\right)}^H=1$

[0257] Since Formula (50) is identical to Formula (47) in a case of a bivariate Laplace distribution, the extraction filter can be determined similarly in accordance with Formula (48).

[0258] Next, a method of deriving update formulae from Formula (27) to Formula (30) which is a sound source model based on divergence is explained. When these pdf's are assigned to Formula (23), a formula that minimizes the sum total of divergence at the f-th frequency bin can be obtained in any of them, but an appropriate auxiliary function for each divergence has not been found. In view of this, the fixed point method, which is another optimization algorithm, is applied.

[0259] A fixed point algorithm represents, by a formula, a condition to hold true when a parameter that is desired to be optimized (in the present disclosure, w₁(f), which is an extraction filter) has converged, and an update formula is derived by transforming the formula into a fixed point format, “w₁(f)=J(w₁(f)).” In the present disclosure, as the condition to hold true at a time of convergence, a formula in which the partial derivative of a parameter is zero is used, and partial differentiation depicted in the following Formula (51) is performed to derive a specific formula.

[0260] $\left[\mathrm{Math}.51\right]$ $\begin{array}{cc}\frac{\partial L}{\partial \stackrel{\_}{w_1\left(f\right)}}=0& \left(51\right)\end{array}$

[0261] The left side of Formula (51) is a partial derivative according to conj(w₁(f)). Then, Formula (51) is transformed to obtain the format of Formula (52).
[Math. 52]
w₁(f)=J(w₁(f))  (52)

[0262] In the fixed point algorithm, the following Formula (53), in which the equal sign in Formula (52) is replaced with assignment, is executed iteratively. Note that, since the constraint of Formula (11) needs to be satisfied regarding w₁(f) in the present disclosure, norm normalization according to Formula (54) also is performed after Formula (53).

[0263] $\left[\mathrm{Math}.53\right]$ $\begin{array}{cc}{w}_1\left(f\right)\leftarrow J\left(w_1\left(f\right)\right)& \left(53\right)\end{array}$ $\left[\mathrm{Math}.54\right]$ $\begin{array}{cc}{w}_1\left(f\right)\leftarrow \frac{w_1\left(f\right)}{\sqrt{w_1\left(f\right){w_1\left(f\right)}^H}}& \left(54\right)\end{array}$

[0264] Hereinbelow, update formulae corresponding to Formula (27) to Formula (30) are explained. Only a formula corresponding to Formula (53) is described in any case, but in actual extraction processes, norm normalization of Formula (54) also is performed after assignment.

[0265] An update formula derived from Formula (27), which is a pdf corresponding to the Euclidean distance, is represented by the following Formula (55).

[0266] $\left[\mathrm{Math}.55\right]$ $\begin{array}{cc}{w}_1\left(f\right)\leftarrow \sum _t\frac{y_1\left(f,t\right)r\left(f,t\right){u\left(f,t\right)}^H}{❘y_1\left(f,t\right)❘}& \left(55\right)\end{array}$ $w_1\left(f\right)\leftarrow w_1\left(f\right)\sum _t\frac{r\left(f,t\right)}{❘w_1\left(f\right)u\left(f,t\right)❘}u\left(f,t\right){u\left(f,t\right)}^H$

[0267] Whereas Formula (55) is written in two lines, it is assumed that the upper line is used after y₁(f,t) is calculated by using Formula (9), and it is assumed that the lower line uses w₁(f) and u(f,t) directly without calculating y₁(f,t). This similarly applies also to Formula (56) to Formula (60) described later.

[0268] Since both the extraction filter w₁(f) and the extraction result y₁(f,t) are unknown only at the initial execution of iteration, w₁(f) is calculated by either of the following methods.

[0269] a) A tentative value as the separation result y₁(f,t) is calculated, and w₁(f) is calculated therefrom in accordance with the formula in the upper line in Formula (55).

[0270] b) A tentative value is assigned to w₁(f), and w₁(f) is calculated therefrom in accordance with the formula in the lower line in Formula (55).

[0271] For the tentative value of y₁(f,t) in a) described above, the method of b) in the explanation regarding Formula (40) can be used. Similarly, for the tentative value of w₁(f) in b), the method of c) in the explanation regarding Formula (40) can be used.

[0272] Update formulae derived from Formula (28), which is a pdf corresponding to the Itakura-Saito divergence (power spectrogram version), are the following Formula (56) and Formula (57).

[0273] $\left[\mathrm{Math}.56\right]$ $\begin{array}{cc}{w}_1\left(f\right)\leftarrow \left(\sum _t\frac{y_1\left(f,t\right){u\left(f,t\right)}^H}{{r\left(f,t\right)}^2}\right){\left(\sum _t\frac{u\left(f,t\right){u\left(f,t\right)}^H}{{❘y_1\left(f,t\right)❘}^2}\right)}^{-1}& \left(56\right)\end{array}$ $w_1\left(f\right)\leftarrow w_1\left(f\right)\left(\sum _t\frac{u\left(f,t\right){u\left(f,t\right)}^H}{{r\left(f,t\right)}^2}\right){\left(\sum _t\frac{u\left(f,t\right){u\left(f,t\right)}^H}{{❘w_1\left(f\right)u\left(f,t\right)❘}^2}\right)}^{-1}$

[0274] Formula (57) is described below.

[0275] $\left[\mathrm{Math}.57\right]$ $\begin{array}{cc}{w}_1\left(f\right)\leftarrow \left(\sum _t\frac{y_1\left(f,t\right){u\left(f,t\right)}^H}{{❘y_1\left(f,t\right)❘}^2}\right){\left(\sum _t\frac{u\left(f,t\right){u\left(f,t\right)}^H}{{r\left(f,t\right)}^2}\right)}^{-1}& \left(57\right)\end{array}$ $w_1\left(f\right)\leftarrow w_1\left(f\right)\left(\sum _t\frac{u\left(f,t\right){u\left(f,t\right)}^H}{{❘w_1\left(f\right)u\left(f,t\right)❘}^2}\right){\left(\sum _t\frac{u\left(f,t\right){u\left(f,t\right)}^H}{{r\left(f,t\right)}^2}\right)}^{-1}$

[0276] Since there are two possible ways of transformation into the form of Formula 52, there are also two types of update formulae.

[0277] Both the second term of the right side of the lower line in Formula (56) and the third term of the right side of the lower line in Formula (57) include only u(f,t) and r(f,t), and are constant during an iteration process. Accordingly, it is sufficient if these terms are calculated only once before iterations, and it is sufficient if its inverse matrix also is calculated once in Formula (57).

[0278] Update formulae derived from Formula (29), which is a pdf corresponding to the Itakura-Saito divergence (amplitude spectrogram version), are the following Formula (58) and Formula (59). There are also two possible ways.

[0279] $\left[\mathrm{Math}.58\right]$ $\begin{array}{cc}{w}_1\left(f\right)\leftarrow \left(\sum _t\frac{y_1\left(f,t\right){u\left(f,t\right)}^H}{r\left(f,t\right)❘y_1\left(f,t\right)❘}\right){\left(\sum _t\frac{u\left(f,t\right){u\left(f,t\right)}^H}{{❘y_1\left(f,t\right)❘}^2}\right)}^{-1}& \left(58\right)\end{array}$ $w_1\left(f\right)\leftarrow w_1\left(f\right)\left(\sum _t\frac{u\left(f,t\right){u\left(f,t\right)}^H}{r\left(f,t\right)❘w_1\left(f\right)u\left(f,t\right)❘}\right){\left(\sum _t\frac{u\left(f,t\right){u\left(f,t\right)}^H}{{❘w_1\left(f\right)u\left(f,t\right)❘}^2}\right)}^{-1}$

[0280] Formula (59) is described below.

[0281] $\left[\mathrm{Math}.59\right]$ $\begin{array}{cc}{w}_1\left(f\right)\leftarrow \left(\sum _t\frac{y_1\left(f,t\right){u\left(f,t\right)}^H}{{❘y_1\left(f,t\right)❘}^2}\right){\left(\sum _t\frac{u\left(f,t\right){u\left(f,t\right)}^H}{r\left(f,t\right)❘y_1\left(f,t\right)❘}\right)}^{-1}& \left(59\right)\end{array}$ $w_1\left(f\right)\leftarrow w_1\left(f\right)\left(\sum _t\frac{u\left(f,t\right){u\left(f,t\right)}^H}{{❘w_1\left(f\right)u\left(f,t\right)❘}^2}\right){\left(\sum _t\frac{u\left(f,t\right){u\left(f,t\right)}^H}{r\left(f,t\right)❘w_1\left(f\right)u\left(f,t\right)❘}\right)}^{-1}$

[0282] An update formula derived from Formula (30) is represented by the following Formula (60). Regarding this formula also, it is sufficient if the last term of the right side is calculated only once before iterations.

[0283] $\left[\mathrm{Math}.60\right]$ $\begin{array}{cc}{w}_1\left(f\right)\leftarrow \left(\sum _t\frac{y_1\left(f,t\right){u\left(f,t\right)}^H}{r\left(f,t\right)❘y_1\left(f,t\right)❘}\right){\left(\sum _t\frac{u\left(f,t\right){u\left(f,t\right)}^H}{{r\left(f,t\right)}^2}\right)}^{-1}& \left(60\right)\end{array}$ $w_1\left(f\right)\leftarrow w_1\left(f\right)\left(\sum _t\frac{u\left(f,t\right){u\left(f,t\right)}^H}{r\left(f,t\right)❘w_1\left(f\right)u\left(f,t\right)❘}\right){\left(\sum _t\frac{u\left(f,t\right){u\left(f,t\right)}^H}{{r\left(f,t\right)}^2}\right)}^{-1}$

[0284] The contents of processes explained thus far are applied to an embodiment of the present disclosure explained next.

[Configuration Example of Sound Source Extracting Apparatus]

[0285] FIG. 4 is a figure depicting a configuration example of a sound source extracting apparatus (sound source extracting apparatus 100) which is an example of the signal processing apparatus according to the present embodiment. For example, the sound source extracting apparatus 100 has multiple microphones 11, an AD (Analog to Digital) converting section 12, an STFT (Short-Time Fourier Transform) section 13, an observation signal buffer 14, a zone estimating section 15, a reference signal generating section 16, a sound source extracting section 17, and a control section 18. As necessary, the sound source extracting apparatus 100 has a post-processing section 19 and a zone/reference signal estimation sensor 20.

[0286] The multiple microphones 11 are placed at mutually different positions. There are several variations of placement modes of the microphones as described later. A mixed sound signal which is a mixture of a target sound and non-target sounds is input (recorded) through the microphones 11.

[0287] The AD converting section 12 converts a multi-channel signal acquired at each of the microphones 11 into a digital signal for each channel. This signal is referred to as a (time-domain) observation signal as appropriate.

[0288] The STFT section 13 applies the short-time Fourier transform to the observation signals to thereby convert the observation signals into time-frequency-domain signals. The time-frequency-domain observation signals are sent to the observation signal buffer 14 and the zone estimating section 15.

[0289] The observation signal buffer 14 accumulates observation signals of a predetermined length of time (number of frames). Observation signals are stored in the unit of frames, and when a request representing an observation signal of which time range is needed is received from another module, an observation signal corresponding to the time range is returned. The signals accumulated here are used at the reference signal generating section 16 and the sound source extracting section 17.

[0290] The zone estimating section 15 detects a zone in which a target sound is included in a mixed sound signal. Specifically, the zone estimating section 15 detects a start time of the target sound (a time at which the target sound starts being produced), an end time of the target sound (a time at which the target sound finishes being produced), and the like. What type of technology is used to perform this zone estimation depends on a use scene of the present embodiment and a placement mode of the microphones, and accordingly, details are described later.

[0291] The reference signal generating section 16 generates a reference signal corresponding to a target sound on the basis of a mixed sound signal. For example, the reference signal generating section 16 estimates a rough amplitude spectrogram of the target sound. Since processes performed by the reference signal generating section 16 depend on a use scene of the present embodiment and a placement mode of the microphones, details are described later.

[0292] The sound source extracting section 17 extracts, from a mixed sound signal, a signal which is similar to a reference signal and in which a target sound is more enhanced. Specifically, the sound source extracting section 17 estimates an estimation result of a target sound by using an observation signal and a reference signal that correspond to a zone in which the target sound is being produced. Alternatively, the sound source extracting section 17 estimates an extraction filter for generating such an estimation result from an observation signal.

[0293] An output of the sound source extracting section 17 is sent to the post-processing section 19 as necessary. Examples of post-processing performed at the post-processing section 19 include voice recognition and the like. In a case where the present embodiment is combined with voice recognition, the sound source extracting section 17 outputs a time-domain extraction result, that is, a voice waveform, and a voice recognizing section (post-processing section 19) performs a recognition process on the voice waveform.

[0294] Note that, whereas voice recognition has a voice zone detection functionality in some cases, a voice zone detection functionality on the side of voice recognition can be omitted in the present embodiment since the zone estimating section 15 equivalent to one having a voice zone detection functionality is included. In addition, whereas voice recognition often includes an STFT for extracting, from a waveform, voice features necessary for recognition processes, in a case where voice recognition is combined with the present embodiment, an STFT on the side of voice recognition may be omitted. In a case where the STFT on the side of voice recognition is omitted, the sound source extracting section 17 outputs a time-frequency-domain extraction result, that is, a spectrogram, and the spectrogram is transformed into voice features on the side of voice recognition.

[0295] The control section 18 comprehensively controls each section of the sound source extracting apparatus 100. For example, the control section 18 controls operation of each section described above. Although omitted in FIG. 4, the control section 18 and each functional block described above are interlinked.

[0296] The zone/reference signal estimation sensor 20 is a sensor that is assumed to be used in zone estimation or reference signal generation, and is different from microphones in the microphones 11. Note that the post-processing section 19 and the zone/reference signal estimation sensor 20 are sandwiched by parentheses in FIG. 4, and this represents that the post-processing section 19 and the zone/reference signal estimation sensor 20 can be omitted from the sound source extracting apparatus 100. That is, if the precision of zone estimation or reference signal generation can be improved by providing a dedicated sensor different from the microphone 11, such a sensor may be used.

[0297] For example, in a case where schemes that use lip images described in Japanese Patent Laid-open No. Hei 10-51889 and the like are used as an utterance zone detection method, an imaging element (camera) can be applied as a sensor. Alternatively, the following sensor used as an auxiliary sensor in Japanese Patent Application No. 2019-073542 proposed by the present inventor may be included, and zone estimation or reference signal generation may be performed by using signals acquired with the auxiliary sensor.

• • [0298] A microphone such as a bone conduction microphone or a pharyngeal microphone that is of a type used in a body-worn state.
  • [0299] A sensor that can observe vibrations of the skin surface near the mouth or throat of a speaker. For example, a combination of a laser pointer and an optical sensor.
    [Zone Estimation and Reference Signal Generation]

[0300] There are several conceivable variations of use scenes of the present embodiment and placement modes of the microphones 11, and what type of technology can be applied for zone estimation and reference signal generation differs for each of them. For explanation of each variation, it is necessary to clarify whether or not target-sound zones can overlap and how to cope with a case where zones of the target sounds can overlap. Hereinbelow, approximately three variations are depicted as typical use scenes and placement modes, and each of them is explained with use of a corresponding one of FIG. 5 to FIG. 7.

[0301] FIG. 5 is a figure depicting an assumed situation where there are N (two or more) speakers in an environment and a microphone is allocated to each speaker. In a situation where microphones are allocated, each speaker wears a pin microphone, a headset microphone, or the like, or a microphone is placed very close to each speaker, for example. It is assumed that the N speakers are S1, S2, . . . , and Sn and the microphones allocated to the speakers are M1, M2, . . . , and Mn. In this case, for example, the microphones M1 to Mn are used as the microphones 11. Further, there are zero or more interfering-sound sound sources Ns.

[0302] Such a situation corresponds to, for example, a scene where a conference is being held in a room and voice recognition of a voice collected with a microphone of each speaker is performed in order to automatically create minutes of the conference. In this case, there is a possibility that utterances overlap, and if utterances overlap, a signal which is a mixture of voices is observed at each microphone. In addition, there can be sounds of a fan of a projector or an air conditioner, reproduced sounds emitted from equipment including a speaker unit, or the like as an interfering-sound sound source, and these sounds also are included in an observation signal of each microphone. Any of them can be a cause of erroneous recognition, but when the sound source extraction technology according to the present embodiment is used, it is possible to keep only a voice of a speaker corresponding to each microphone and remove (suppress) other sound sources (other speakers and interfering-sound sound sources), thereby making it possible to improve the voice recognition precision.

[0303] Hereinbelow, a zone detection method and a reference signal generation method that can be used in such a situation are explained. Note that, hereinafter, a voice of a corresponding (target) speaker in sounds observed with each microphone is referred to as a main voice or a main utterance, and voices of other speakers are referred to as echoes or crosstalk, as appropriate.

[0304] As a zone detection method, main utterance detection described in Japanese Patent Application No. 2019-227192 can be used. In the application, by performing training using a neural network, a detector that reacts to a main voice while ignoring crosstalk is realized. In addition, since it also copes with a situation where utterances overlap, the detector can estimate the zone and speaker of each utterance as in FIG. 3 even if utterances overlap.

[0305] There are at least two possible reference signal generation methods. In one method, a reference signal is directly generated from a signal observed with a microphone allocated to a speaker. For example, whereas a signal observed with the microphone M1 in FIG. 5 is a mixture of all sound sources, a voice of the speaker S1, who is the closest sound source, is collected as a large voice; on the other hand, as compared with the voice, sounds from other sound sources are collected as smaller sounds. Accordingly, when an amplitude spectrogram is generated by segmenting the observation signal of the microphone M1 according to an utterance zone of the speaker S1 and obtaining the absolute value after application of the short-time Fourier transform to a segment corresponding to the utterance zone, the amplitude spectrogram is a rough amplitude spectrogram of the target sound, and can be used as a reference signal in the present embodiment.

[0306] The other method uses a crosstalk reduction technology described in Japanese Patent Application No. 2019-227192 described before. In the application described above, by training a neural network, it is made possible to realize removal (reduction) of crosstalk from a signal which is a mixture of a main voice and crosstalk, keeping the main voice. An output of the neural network is an amplitude spectrogram which is a crosstalk reduction result or a time-frequency mask, and if it is the former, the output can be used as a reference signal with no changes being made thereto. Even if the output is the latter, an amplitude spectrogram which is a crosstalk removal result can be generated by applying the time-frequency mask to the amplitude spectrogram of an observation signal, and accordingly, the amplitude spectrogram can be used as a reference signal.

[0307] Next, a reference signal generation process and the like in a use scene different from that in FIG. 5 are explained with use of FIG. 6. The example depicted in FIG. 6 represents an assumed environment where there are one or more speakers and one or more interfering-sound sound sources. Whereas FIG. 5 mainly focuses more on overlapping utterances than on the presence of the interfering-sound sound sources Ns, the example depicted in FIG. 6 mainly focuses on acquisition of a clean voice in a noisy environment where there are large interfering sounds. Note that, in a case where there are two or more speakers, overlapping utterances also are a problem.

[0308] It is assumed that there are m speakers, and the speakers are a speaker S1 to a speaker Sm. m is equal to or greater than one. Whereas FIG. 6 depicts only one interfering-sound sound source Ns, but the number of interfering-sound sound sources Ns can be any number.

[0309] There are two types of sensors to be used. One of them is sensors worn by the speakers or sensors placed very close to the speakers (sensors corresponding to the zone/reference signal estimation sensor 20), and are hereinbelow referred to as sensors SE (sensors SE1, SE2, . . . , and SEm) as appropriate. The other of them is a microphone array 11A including multiple microphones 11 whose positions are fixed.

[0310] The zone/reference signal estimation sensor 20 used may be of a type similar to the microphones in FIG. 5 (a type of microphone that is called an air conduction microphone and collects sounds propagated through the atmospheric air), but other than this, as explained with reference to FIG. 4, a type of microphone such as a bone conduction microphone or a pharyngeal microphone that is of a type used in a body-worn state or a sensor that can observe vibrations of the skin surface near the mouth or throat of a speaker may be used. In any case, since each sensor SE is nearer to the corresponding speaker than the microphone array 11A is or is worn on the body of the speaker, the sensor SE can record an utterance by the speaker corresponding to the sensor at a high S/N ratio.

[0311] Possible placement modes of the microphone array 11A include also a mode in which microphones are placed at multiple locations in a space which mode is called distributed microphones, in addition to a mode in which multiple microphones are placed in one apparatus. Conceivable examples of the mode of distributed microphones include a mode in which microphones are placed on the wall surface and ceiling surface of a room, a mode in which microphones are placed on the seat, wall surface, ceiling, dashboard, and the like in an automobile, and other modes.

[0312] In the present example, signals acquired with the sensors SE1 to SEm corresponding to the zone/reference signal estimation sensor 20 are used for zone estimation and reference signal generation, and multi-channel observation signals acquired from the microphone array 11A are used for sound source extraction. As a zone estimation method and a reference signal generation method in a case where air conduction microphones are used as the sensors SE, methods similar to the methods explained with use of FIG. 5 can be used.

[0313] On the other hand, in a case where body-worn microphones are used, in addition to methods similar to the methods depicted in FIG. 5, methods using a feature that a signal with less mixing of interfering sounds or utterances by other speakers can be acquired can also be used. For example, for zone estimation, a method of identifying a zone on the basis of a threshold regarding the power of an input signal can also be used, and an amplitude spectrogram generated from the input signal can be used as a reference signal with no changes being made thereto. Since sounds recorded with body-worn microphones have attenuated high frequencies and, moreover, include also recorded sounds such as swallowing sounds that are generated inside bodies in some cases, it is not necessarily appropriate to use the sounds as inputs for voice recognition or the like, but the sounds can be used effectively as sounds for zone estimation or reference signal generation.

[0314] In a case where sensors such as optical sensors other than microphones are used as the sensors SE, a method described in Japanese Patent Application No. 2019-227192 can be used. In this patent application, a neural network is trained in advance with the correspondence, to clean target sounds, of sounds acquired with air conduction microphones (mixtures of target sounds and interfering sounds) and signals acquired with auxiliary sensors (some signal corresponding to target sounds), and at a time of inference, signals acquired with an air conduction microphone and an auxiliary sensor are input to the neural network to thereby generate a cleaner target sound. Since an output of the neural network is an amplitude spectrogram (or a time-frequency mask), the output can be used as a reference signal (or can be used to generate a reference signal) according to the present embodiment. In addition, since a method of estimating also a zone in which a target sound is being produced simultaneously with the generation of the clean target sound is described as a modification example, it can be used also as zone detecting means.

[0315] Sound source extraction is basically performed by using observation signals acquired with the microphone array 11A. Note that, in a case where air conduction microphones are used as the sensors SE, it is also possible to add observation signals acquired with the air conduction microphones. That is, if the microphone array 11A includes N microphones, sound source extraction may be performed by using observation signals of (N+m) channels additionally including signals from the m zone/reference signal estimation sensors. In addition, since, in that case, there are multiple air conduction microphones even if N=1, a single microphone may be used instead of the microphone array 11A.

[0316] Similarly, in addition to signals from the sensors SE, signals acquired with the microphone array may be used also for zone estimation or reference signal generation. Since the microphone array 11A is apart from any speakers, utterances by the speakers are always observed as crosstalk. By comparing the signals of the microphone array 11A and signals of the microphones for zone/reference signal estimation, it can be expected that the zone estimation precision, in particular, the zone estimation precision when utterances overlap, is improved.

[0317] FIG. 7 depicts a microphone placement mode different from that in FIG. 6. The microphone placement mode is the same as that in FIG. 6 in that there are one or more speakers and one or more interfering-sound sound sources in the assumed environment, but the microphone array 11A is the only microphone to be used, and there are no sensors placed very close to speakers. Examples of the mode of the microphone array 11A that can be applied include, similarly to FIG. 6, multiple microphones placed in one apparatus, multiple microphones (distributed microphones) placed in a space, and other modes.

[0318] The problem in such a situation is how utterance zone estimation and reference signal estimation, which are a premise of sound source extraction according to the present disclosure, are performed, and technologies that can be applied differ depending on whether the frequency of occurrence of mixing of voices is low or high. Hereinbelow, each of them is explained.

[0319] Cases where the frequency of occurrence of mixing of voices is low include a case where there is only one speaker in an environment (i.e., only the speaker S1) and the interfering-sound sound sources Ns can be regarded as non-voice sound sources. In that case, as a zone estimation method, a voice zone detection technology paying attention to “voiciness” described in Japanese Patent No. 4182444 or the like can be applied. That is, in a case where it is considered that the only “voicy” signal is a signal of an utterance by the speaker S1 in the environment depicted in FIG. 7, non-voice signals are ignored, and a portion (timing) including the voicy signal is detected as a target-sound zone.

[0320] As a reference signal generation method, a technique called denoising (denoise) described in Document 3, that is, a process in which a signal which is a mixture of a voice and a non-voice sound is input, the non-voice sound is removed, and the voice is kept, can be applied. Whereas a wide variety of methods can be applied as denoising, for example, the following method uses a neural network, and its outputs can be used as reference signals with no changes being made thereto since its outputs are amplitude spectrograms.

“Document 3

• • [0321] Liu, D. & Smaragdis, P. & Kim, M. (2014).
  • [0322] “Experiments on deep learning for speech denoising,”
  • [0323] Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH. 2685-2689.”

[0324] On the other hand, cases where the frequency of occurrence of mixing of voices is high include a case where multiple speakers are having a conversation in an environment and their utterances overlap and a case where interfering-sound sound sources produce voices even if there is only one speaker, for example. Examples of the latter case include a case where voices are output from speaker units of a television, a radio, or the like and other cases. In such a case, a scheme that can be applied also to a mixture of voices needs to be used for utterance zone detection. For example, technologies like the ones below can be applied.

• • [0325] a) Voice zone detection using sound source direction estimation
  • [0326] (e.g., methods described in Japanese Patent Laid-open No. 2010-121975 and Japanese Patent Laid-open No. 2012-150237)
  • [0327] b) Voice zone detection using facial images (lip images)
  • [0328] (e.g., methods described in Japanese Patent Laid-open No. Hei 10-51889 and Japanese Patent Laid-open No. 2011-191423)

[0329] Since there is the microphone array in the microphone placement mode depicted in FIG. 7, sound source direction estimation which is a premise of a) can be applied. In addition, when an imaging element (camera) is used as the zone/reference signal estimation sensor 20 in the example depicted in FIG. 4, b) can also be applied. Since, in either scheme, the direction of an utterance also can be known at a time point when an utterance zone of the utterance is detected (in the method of b) described above, an utterance direction can be calculated from the position of a lip in an image), the value of the direction can be used for reference signal generation. Hereinbelow, a sound source direction estimated in utterance zone estimation is referred to as θ as appropriate.

[0330] A reference signal generation method also needs to cope with mixing of voices, and the following can be applied as such a technology.

a) Time Frequency Mask Using Sound Source Direction

[0331] This is a reference signal generation method used in Japanese Patent Laid-open No. 2014-219467. Calculating a steering vector corresponding to the sound source direction θ and calculating a cosine similarity between the steering vector and an observation signal vector (Formula (2) described above) give a mask to keep a sound arriving from the direction θ and attenuate sounds arriving from other directions. The mask is applied to the amplitude spectrogram of an observation signal, and a signal generated thereby is used as a reference signal.

b) Neural-Network-Based Selective Listening Technologies Such as Speaker Beam or Voice Filter

[0332] Selective listening technologies described here are technologies to extract a human voice of one specified person, from a monaural signal which is a mixture of multiple voices. Clean voices (which may be utterance contents different from mixed voices), not mixed with voices of other speakers, of a speaker whose voice is desired to be extracted are recorded in advance, and when both a mixed signal and a clean voice are input to a neural network, a voice of a specified speaker included in the mixed signal is output. To be precise, a time-frequency mask for generating such a spectrogram is output. If the thus-output mask is applied to the amplitude spectrogram of an observation signal, a signal generated thereby can be used as a reference signal according to the present embodiment. Note that details of Speaker Beam and Voice Filter are described in the following Document 4 and Document 5, respectively.

“Document 4:

• • [0333] M. Delcroix, K. Zmolikova, K. Kinoshita, A. Ogawa and T. Nakatani,

[0334] “Single channel target speaker extraction and recognition with speaker beam,” IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018.”

“Document 5:

• • [0335] Author: Quan Wang, Hannah Muckenhire, Kevin Wilson, Prashant Sridhar, Zelin Wu, John Hershey, Rif A. Saurous, Ron J. Weiss, Ye Jia, Ignacio Lopez Moreno

[0336] “VOICEFILTER: Targeted Voice Separation by Speaker-Conditioned Spectrogram Masking,” arXiv:1810.04826v3 [eess.AS]27 Oct. 2018 https://arxiv.org/abs/1810.04826”

(Details of Sound Source Extracting Section)

[0337] Next, details of the sound source extracting section 17 are explained with use of FIG. 8. The sound source extracting section 17 has a pre-processing section 17A, an extraction filter estimating section 17B, and a post-processing section 17C, for example.

[0338] The pre-processing section 17A performs the uncorrelation process represented in Formula (3) to Formula (7), that is, performs the uncorrelation process or the like on a time-frequency-domain observation signal.

[0339] The extraction filter estimating section 17B estimates a filter that extracts a signal in which a target sound is more enhanced. Specifically, the extraction filter estimating section 17B performs extraction filter estimation and extraction result generation for sound source extraction. More specifically, the extraction filter estimating section 17B estimates an extraction filter as a solution that optimizes an objective function reflecting the similarity between a reference signal and an extraction result from the extraction filter and the independence between the extraction result and a separation result of another imaginary sound source.

[0340] As described above, as a sound source model that is included in the objective function and that represents the similarity between the reference signal and the extraction result, the extraction filter estimating section 17B uses any one of:

• • [0341] a bivariate spherical distribution of the extraction result and the reference signal,
  • [0342] a time-frequency-varying variance model that regards the reference signal as a value corresponding to the variance of each time frequency, and
  • [0343] a model using a divergence between the absolute value of the extraction result and the reference signal. In addition, the bivariate Laplace distribution may be used as the bivariate spherical distribution. In addition, any one of a time-frequency-varying variance Gaussian distribution, a time-frequency-varying variance Laplace distribution, and a time-frequency-varying variance Student-t distribution may be used as the time-frequency-varying variance model. In addition, any one of the Euclidean distance or square error between the absolute value of the extraction result and the reference signal, the Itakura-Saito distance between the power spectrum of the extraction result and the power spectrum of the absolute value, the Itakura-Saito distance between the amplitude spectrum of the extraction result and the amplitude spectrum of the absolute value, the square error between the ratio between the absolute value of the extraction result and the reference signal and one may be used as the divergence of the model using divergence.

[0344] The post-processing section 17C performs at least a process of applying the extraction filter to a mixed sound signal. The post-processing section 17C may perform a process of generating an extraction-result waveform by applying the inverse Fourier transform to an extraction-result spectrogram, in addition to a rescaling process described later.

[Procedure of Processes Performed at Sound Source Extracting Apparatus]

(Overall Procedure)

[0345] Next, a procedure of processes (overall procedure) performed at the sound source extracting apparatus 100 is explained with reference to a flowchart depicted in FIG. 9. Note that the processes explained below are performed by the control section 18 unless noted otherwise.

[0346] In Step ST11, the AD converting section 12 converts an analog observation signal (mixed sound signal) input to the microphones 11 into a digital signal. The observation signal at this time point is a time-domain observation signal. Then, the process proceeds to Step ST12.

[0347] In Step ST12, the STFT section 13 applies the short-time Fourier transform (STFT) to the time-domain observation signal, and obtains a time-frequency-domain observation signal. Input may be performed from a file, a network, or the like as necessary, other than being performed through the microphones. Details of a specific process performed at the STFT section 13 are described later. Since there are multiple input channels (corresponding to the number of the microphones) in the present embodiment, the AD conversion and the STFT also are performed for all the channels. Then, the process proceeds to Step ST13.

[0348] In Step ST13, a process of accumulating (buffering) the observation signal converted into the time-frequency-domain signal by the STFT, by an amount corresponding to a predetermined length of time (a predetermined number of frames), is performed. Then, the process proceeds to Step ST14.

[0349] In Step ST14, the zone estimating section 15 estimates a start time of the target sound (a time at which the target sound starts being produced) and an end time of the target sound (a time at which the target sound finishes being produced). Further, in a case of specification in an environment where utterances can overlap, information that enables identification as to which speaker's utterance an utterance is is estimated as well. For example, in the use modes depicted in FIG. 5 and FIG. 6, the number of a microphone allocated to each speaker also is estimated, and in the use mode depicted in FIG. 7, the directions of utterances also are estimated.

[0350] Sound source extraction and processes that accompany it are performed for each target-sound zone. Accordingly, it is assessed in Step ST15 whether or not a target-sound zone is detected. Then, only in a case where a zone is detected in Step ST15, the process proceeds to Step ST16, and in a case where a zone is not detected, Steps ST16 to ST19 are skipped, and the process proceeds to Step ST20.

[0351] In a case where a zone is detected in Step ST15, in Step ST16, the reference signal generating section 16 generates, as a reference signal, a rough amplitude spectrogram of a target sound that is being produced in the zone. Schemes that can be used for reference signal generation have been explained with reference to FIG. 5 to FIG. 7. The reference signal generating section 16 generates a reference signal on the basis of an observation signal supplied from the observation signal buffer 14 and a signal supplied from the zone/reference signal estimation sensor 20, and supplies the reference signal to the sound source extracting section 17. Then, the process proceeds to Step ST17.

[0352] In Step ST17, the sound source extracting section 17 generates an extraction result of the target sound by using the reference signal determined in Step ST16 and an observation signal corresponding to a time range of the target-sound zone. That is, the sound source extracting section 17 performs a sound source extraction process. Details of the process are described later.

[0353] In Step ST18, it is determined whether to or not to iterate the processes according to Step ST16 and Step ST17 a predetermined number of times. This iteration means that, when the sound source extraction process generates an extraction result which is more precise than the observation signal or the reference signal, next, by generating a reference signal again from the extraction result and executing the sound source extraction process again by using the reference signal, an extraction result which is still more precise than the previous iteration can be obtained.

[0354] For example, in a case where a reference signal is generated by inputting an observation signal to a neural network, when the first extraction result, instead of an observation signal, is input to the neural network, the possibility that an output therefrom is more precise than the first extraction result is high. Accordingly, when the second extraction result is generated with use of the reference signal generated as described above, the possibility that the second extraction result is more precise than the first extraction result is high, and by further iterations, it is also possible to obtain more precise extraction results. A feature of the present embodiment is that not a separation process but the extraction process is performed iteratively. Note that one should be careful that this iteration is different from iteration that is used to estimate a filter in the auxiliary function method or the fixed point method inside the sound source extraction process according to Step ST17. After the process according to Step ST18, the process proceeds to Step ST19. That is, in a case where it is assessed in Step ST18 that the iteration is to be performed, the process returns to Step ST16, and the processes described above are performed repeatedly. In a case where it is assessed in Step ST18 that the iteration is not to be performed, the process proceeds to Step ST19.

[0355] In Step ST19, post-processing by the post-processing section 17C is performed by using the extraction result generated in Step ST17. Conceivable examples of the post-processing include voice recognition, response generation for a voice conversation using a recognition result of the voice recognition, and the like. Then, the process proceeds to Step ST20.

[0356] In Step ST20, it is assessed whether to or not to continue the process. In a case where the process is to be continued, the process returns to Step ST11, and in a case where the process is to be continued, the process ends.

(STFT)

[0357] Next, the short-time Fourier transform performed at the STFT section 13 is explained with reference to FIGS. 10A, 10B, and 10C. In the present embodiment, since microphone observation signals are multi-channel signals observed in multiple signals, the STFT is performed for each channel. The following is an explanation regarding the STFT for a k-th channel.

[0358] Waveforms with a predetermined length are obtained by segmentation of the waveform of a microphone recording signal obtained by the AD conversion process according to Step ST11, and a window function such as the Hann window or the Humming window is applied to the waveforms (see FIG. 10A). Each of these units obtained by segmentation is called a frame. By applying the short-time Fourier transform to data of one frame (see FIG. 10B), xk(1,t) to xk(F,t) are obtained as time-frequency-domain observation signals. Note that t represents a frame number and F represents the total number of frequency bins (see FIG. 10C).

[0359] Frames to be obtained by the segmentation may overlap, and by doing so, changes of time-frequency-domain signals between the consecutive frames become smooth. In FIGS. 10A, 10B, and 10C, xk(1,t) to xk(F,t) which are data of one frame are illustrated collectively as one vector xk(t) (FIG. 10C). xk(t) is called a spectrum, and a data structure formed by arranging multiple spectrums next to each other in the time direction is called a spectrogram.

[0360] In FIG. 10C, the horizontal axis represents frame numbers, and the vertical axis represents frequency bin numbers. From observation signals 51, 52, and 53 obtained by segmentation, three spectrums 51A, 52A, and 53A are generated, respectively.

(Sound Source Extraction Process)

[0361] Next, the sound source extraction process according to the present embodiment is explained with reference to a flowchart depicted in FIG. 11. The sound source extraction process explained with reference to FIG. 11 corresponds to the process in Step ST17 in FIG. 9.

[0362] In Step ST31, the pre-processing section 17A performs pre-processing. Examples of the pre-processing include the uncorrelation represented by Formula (3) to Formula (6). In addition, some special processes are performed only in the initial execution depending on update formulae used in filter estimation, and such processes also are performed as the pre-processing. For example, according to an estimation result of a target-sound zone supplied from the zone estimating section 15, the pre-processing section 17A reads out an observation signal (observation signal vector x(f,t)) of the target-sound zone from the observation signal buffer 14, and performs, as the pre-processing, an uncorrelation process or the like in accordance with the calculation of Formula (3) on the basis of the observation signal having been read out. Then, the pre-processing section 17A supplies a signal (uncorrelated observation signal u(f,t)) obtained by the pre-processing to the extraction filter estimating section 17B, and thereafter, the process proceeds to Step ST32.

[0363] In Step ST32, the extraction filter estimating section 17B performs a process of estimating an extraction filter. Then, the process proceeds to Step ST33. In Step ST33, the extraction filter estimating section 17B assesses whether or not the extraction filter has converged. In a case where it is assessed in Step ST33 that the extraction filter has not converged, the process returns to Step ST32, and the processes described above are performed repeatedly. Steps ST32 and ST33 represent iterations for estimating the extraction filter. Since the extraction filter is not determined in a closed form except for a case where the TFVV Gaussian distribution of Formula (32) is used as a sound source model, the process according to Step ST32 is repeated until the extraction filter and the extraction result converge or is repeated a predetermined number of times.

[0364] The extraction filter estimation process according to Step ST32 is a process of determining the extraction filter w₁(f), and specific formulae differ between different sound source models.

[0365] For example, in a case where the TFVV Gaussian distribution of Formula (32) is used as a sound source model, the weighted covariance matrix of the right side of Formula (35) is calculated by using the reference signal r(f,t) and the uncorrelated observation signal u(f,t), and next, eigenvectors are determined by using eigen decomposition. Then, as in Formula (36), the Hermitian transpose is applied to an eigenvector corresponding to the smallest eigenvalue, and a result obtained therefrom is the extraction filter w₁(f) to be determined. This process is performed for all frequency bins, that is, f=1 to F.

[0366] Similarly, in a case where the TFVV Laplace distribution of Formula (31) is used as a sound source model, first, the auxiliary variable b(f,t) is calculated by using the reference signal r(f,t) and the uncorrelated observation signal u(f,t) in accordance with Formula (40). Next, the weighted covariance matrix of the right side of Formula (42) is calculated, and eigen decomposition is applied to the weighted covariance matrix to determine eigenvectors. Last, the extraction filter w₁(f) is obtained in accordance with Formula (36). Since the extraction filter w₁(f) at this time point has not converged yet, the process returns to Formula (40), and a calculation of the auxiliary variable is performed again. These processes are executed for the number of times.

[0367] Similarly, also in a case where the bivariate Laplace distribution of Formula (25) is used as a sound source model, a calculation of the auxiliary variable b(f,t) (Formula (46)) and a calculation of the extraction filter (Formula (48) and Formula (36)) are performed alternately.

[0368] On the other hand, in a case where the model based on divergence represented by Formula (26) is used as a sound source model, calculations of the update formulae (Formula (55) to Formula (60)) corresponding to the respective models and a calculation of the formula (Formula (54)) to normalize the norm to 1 are performed alternately.

[0369] In a case where it is assessed in Step ST33 that the extraction filter has converged, that is, in a case where iterations have been performed until the extraction filter converges or are performed a predetermined number of times, the extraction filter estimating section 17B supplies the extraction filter or the extraction result to the post-processing section 17C, and the process proceeds to Step ST34.

[0370] In Step ST34, the post-processing section 17C performs post-processing. After the process in Step ST34 is performed, the sound source extraction process ends, and this means that the process in Step ST17 in FIG. 11 has ended. In the post-processing, rescaling of the extraction result is performed. Further, by performing the inverse Fourier transform as necessary, a time-domain waveform is generated. The rescaling is a process of adjusting the scale of each frequency bin of the extraction result. Although the constraint that the norm of a filter is 1 is placed in order to apply an efficient algorithm in the extraction filter estimation, the extraction result generated by applying the extraction filter on which the constraint has been placed has a scale different from the scale of an ideal target sound. In view of this, the post-processing section 17C adjusts the scale of the extraction result by using the observation signal (observation signal vector x(f,t)) which has not yet been subjected to uncorrelation and which is acquired from the observation signal buffer 14 or the like.

[0371] The rescaling process is as follows.

[0372] First, assuming that k=1 in Formula (9), y₁(f,t), which is the extraction result before the rescaling, is calculated from the converged extraction filter w₁(f). A rescaling coefficient γ(f) can be determined as a value that minimizes the following Formula (61), and the specific formula is represented by Formula (62).

[0373] $\left[\mathrm{Math}.61\right]$ $\begin{array}{cc}\gamma \left(f\right)=\arg \underset{\gamma \left(f\right)}{\min }{\langle {❘x_k\left(f,t\right)-\gamma \left(f\right)y_1\left(f,t\right)❘}^2\rangle }_t& \left(61\right)\end{array}$ $\left[\mathrm{Math}.62\right]$ $\begin{array}{cc}\gamma \left(f\right)={\langle x_i\left(f,t\right)\stackrel{\_}{y_1\left(f,t\right)}\rangle }_t& \left(62\right)\end{array}$