Sound source localization apparatus and method

The present invention relates generally to sound source localization. More specifically, embodiments of the present invention relate to apparatuses and methods for performing sound source localization through an array of microphones.

Examples of sound source localization include localizing sound sources using an array of microphones. For example, a method (SRP-PHAT algorithm) of performing sound source localization based on time difference (phase difference) between the signals of different microphones has been proposed in J. DiBiase, "A high-accuracy, low-latency technique for talker localization in reverberant environments", PhD thesis, Brown University, Providence RI, USA, May 2000.

US 2005/0276419 A1US20050276419A1 shows methods for sound source localization based on joint learning and evaluation of ITD and ILD representations that are measured in a complementary, correlation-based way using binaural time-frequency spectrums. From these measurements and learned representatives, which are created by combinations of measurements from signals belonging to the same class, i.e., the same azimuthal location, probability distributions over frequency and class are computed. These probability distributions can be combined over cue and frequency using information-theoretic approaches to get a robust classification of the location and additionally a confidence measure for the quality of the classification result.

JP 2004-324284 AJP2004324284A shows a sound source separation system having a means for inputting the acoustic signals generated from the plurality of the sound sources from sound receiving parts of both the left and right sides; a means for dividing input signals of both the left and right sides in every a frequency band; and a means for obtaining phase difference (IPD) between both ears of every frequency band from a cross spectrum of the input signals of both the left and right sides, and a level difference (ILD) between both the ears from a level difference of a power spectrum. Further, the system comprises a means for presuming candidates of the directions of the sound sources for each of the frequency bands by comparing the IPD and the ILD with a database in all frequency bands; a means for presuming the direction having a high frequency of occurrence to be the direction of the sound source from among the directions of the sound sources obtained in each of the frequency bands, and a means for separating the sound source by extracting mainly the frequency band of the specific direction of the sound source based on information on the presumed direction of the sound source.

US 2009/0296526 A1US20090296526A1 shows an acoustic treatment apparatus for obtaining a first output signal by performing filtering for forming a directivity in a first direction for received sound signals of sound receivers, for obtaining a second output signal by performing filtering for forming a directivity in a second direction different from the first direction for received sound signals of sound receivers, for obtaining a strength ratio between a strength of the first output signal and a strength of the second output signal, and for estimating a sound source direction on the basis of the strength ratio.

According to the present invention, there is provided a method of performing sound source localization according to claim 1, an apparatus of performing sound source localization according to claim 8, and a computer-readable medium according to claim 15. Dependent claims relate to preferred embodiments of the present invention.

According to an embodiment of the present invention, a method of performing sound source localization is provided. The method includes calculating a frame amplitude difference vector based on short time frame data acquired through an array of microphones, the frame amplitude difference vector reflecting differences between amplitudes captured by microphones of the array during recording the short time frame data; evaluating similarity between the frame amplitude difference vector and each of a plurality of reference frame amplitude difference vectors, each of the plurality of reference frame amplitude difference vectors reflecting differences between amplitudes captured by microphones of the array during recording sound from one of a plurality of candidate locations; and estimating a desired location of sound source based at least on the candidate locations and associated similarity.

According to another embodiment of the present invention, an apparatus for performing sound source localization is provided. The apparatus includes a vector calculator that calculates a frame amplitude difference vector based on short time frame data acquired through an array of microphones, the frame amplitude difference vector reflecting differences between amplitudes captured by microphones of the array during recording the short time frame data; a similarity evaluator which evaluates similarity between the frame amplitude difference vector and each of a plurality of reference frame amplitude difference vectors, each of the plurality of reference frame amplitude difference vectors reflecting differences between amplitudes captured by microphones of the array during recording sound from one of a plurality of candidate locations; and an estimator which estimates a desired location of sound source based at least on the candidate locations and associated similarity.

According to another embodiment of the present invention, a computer-readable medium having computer program instructions recorded thereon for enabling a processor to perform sound source localization is provided. The computer program instructions include means for calculating a frame amplitude difference vector based on short time frame data acquired through an array of microphones, the frame amplitude difference vector reflecting differences between amplitudes captured by microphones of the array during recording the short time frame data; means for evaluating similarity between the frame amplitude difference vector and each of a plurality of reference frame amplitude difference vectors, each of the plurality of reference frame amplitude difference vectors reflecting differences between amplitudes captured by microphones of the array during recording sound from one of a plurality of candidate locations; and means for estimating a desired location of sound source based at least on the candidate locations and associated similarity.

Further features and advantages of the invention, as well as the structure and operation of various embodiments of the invention, are described in detail below with reference to the accompanying drawings. It is noted that the invention is not limited to the specific embodiments described herein. Such embodiments are presented herein for illustrative purposes only. Additional embodiments will be apparent to persons skilled in the relevant art(s) based on the teachings contained herein.

The present invention is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements and in which:

Fig. 1 is a block diagram illustrating an example apparatus for performing sound source localization according to an embodiment of the present invention;

Fig. 2 depicts an example array of three cardioid microphones;

Fig. 3 depicts an example method of performing sound source localization according to an embodiment of the present invention;

Fig. 4 is a block diagram illustrating an example apparatus for performing sound source localization according to an embodiment of the present invention;

Fig. 5 depicts an example method of performing sound source localization according to an embodiment of the present invention;

Fig. 6 is a block diagram illustrating an example apparatus for performing sound source localization according to an embodiment of the present invention;

Fig. 7 depicts an example method of performing sound source localization according to an embodiment of the present invention;

Fig. 8 is a block diagram illustrating an example apparatus for performing sound source localization according to an embodiment of the present invention;

Fig. 9 depicts an example method of performing sound source localization according to an embodiment of the present invention;

Fig. 10 depicts a graph illustrating an example of a combined probability function obtained by multiplying a first probability function and a second probability function;

Fig. 11 is a block diagram illustrating an exemplary system for implementing aspects of the present invention.

The embodiments of the present invention are below described by referring to the drawings. It is to be noted that, for purpose of clarity, representations and descriptions about those components and processes known by those skilled in the art but unrelated to the present invention are omitted in the drawings and the description.

As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method or computer program product. Accordingly, aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a "circuit," "module" or "system." Furthermore, aspects of the present invention may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon.

Any combination of one or more computer readable medium(s) may be utilized. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof.

A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired line, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).

Aspects of the present invention are described below with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.

The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

Fig. 1 is a block diagram illustrating an example apparatus 100 for performing sound source localization according to an embodiment of the present invention.

Apparatus 100 may be useful in various applications. In one application, apparatus 100 may detect speakers' location information in a meeting. This location information can be used to separate the speakers in meeting recording, or it can be used for spatial audio coding of the meeting.

As illustrated in Fig. 1, apparatus 100 includes a vector calculator 101, a similarity evaluator 102 and an estimator 103.

Vector calculator 101 is configured to calculate a frame amplitude difference vector (FADV) based on short time frame data acquired through an array of microphones. The frame amplitude difference vector reflects differences between amplitudes captured by microphones of the array during recording the short time frame data.

In general, due to diversity between distances from a sound source to different microphones of the array, or diversity between sensitivity of the microphones to sound signals from the sound source, amplitudes of the sound signals captured by the microphones from the same sound source are different. For different sound source locations, distributions of amplitude differences between the microphones can be different. For example, in case of an array of unidirectional microphones, or in case that the sound source is close to an array of omni-directional microphones, the distributions of the amplitude differences between the microphones can be significantly different. Based on this observation, the distributions of the amplitude differences between the microphones may be associated with different sound locations, at least those locations exhibiting this diversity. In this regard, it is possible to estimate whether a sound source is located at one of these locations according to this association, based on amplitude differences between the microphones introduced by the sound source.

In the following, an array of unidirectional microphones will be adopted as an example to describe the embodiments of the present invention.

The unidirectional microphones may be cardioid microphones. Fig. 2 depicts an example array of three cardioid microphones 201, 202 and 203. Graphs 204, 205 and 206 illustrate directionality patterns of cardioid microphones 201, 202 and 203 respectively. Cardioid microphones 201, 202 and 203 are orientated in directions of 0 degrees, -120 degrees and -240 degrees respectively in a plane. Further, the distance between each pair of two microphones may be 1.15cm. To facilitate description, this cardioid microphones array is named as CMA.

In general, the location of a sound source may refer to the angle of direction (DOA) of arrival or position of the sound source. In some cases, the distributions of amplitude differences between microphones for different positions along the DOA are substantially similar, and therefore, the DOA may be employed to measure the location. Depending on specific applications, the DOA may be represented with the azimuth angle of the sound source in a plane (named as horizontal plane) where the CMA is located. In this case, the audio localization problem is simplified to angle detection problem. However, it is possible to detect both the azimuth angle in the horizontal plane and the elevation angle in a vertical plane of source by adding one microphone facing upwards.

In the following, the azimuth angle of the sound source in the horizontal plane will be adopted as an example of the location.

It should be noted that various microphone arrays may be applied to the embodiments of the present invention as long as the distributions of amplitude differences between microphones for different locations can exhibit significant diversity.

The FADV reflects amplitude differences between microphones. In the following, the FADV is represented as a vector (v₁, v₂, ..., v_M), where M represents the number of the microphones, and v_i represents the amplitude captured by the i-th microphone. One skilled in the art can understand that other formats may be applied to the FADV to reflect amplitude differences between microphones. For example, v_i in the vector (v₁, v₂, ..., v_M) may represent the amplitude difference relative to the amplitude captured by a certain microphone. For another example, the FADV may be represented as a vector (d_1,1, ..., d_1,M, d_2,2, ..., d_2,M, ..., d_M-1,M), where M represents the number of the microphones, and d_i,j represents the difference of amplitudes captured by the i-th microphone and the j-th microphone.

The short time frame data may be extracted from audio data stream pre-recorded through the array or recorded through the array in real time. Further, a window may be multiplied on the short time frame data. The window may be hamming window, hanning window, etc. Assuming that the short time frame contains N samples and the number of microphone is M, the short time frame data can be stored as a matrix X with M rows and N columns, where X(n,m) corresponds to the n-th sample of the m-th microphone.X=x1,1x1,2…x1,Nx2,1x2,2…x2,N⋮⋮⋮xM,1xM,2…xM,N.

The value of N may be determined based on the sampling rate and expected time length of a short time frame: N=time length×sample rate

Adjacent short time frames may or may not overlap with each other.

The value of N and whether to use overlapped short time frame is dependent on applications' requirement on time resolution and computation complexity. A larger N usually means more accurate estimation with more data, but lower time resolution if there is no overlapped data between adjacent short time frames. The time resolution can be increased by using overlapped data between adjacent short time frames. But using overlapped data between adjacent short time frames may increase the computation complexity. If Fast Fourier transform (FFT) is performed, N preferably belongs to a set expressed as {2^k, k=1, 2, ...}.

As an example, in one implementation, N=32768 is assumed for CMA when the sampling rate is 48 KHz.

There are multiple methods to calculate the FADV. The methods may be classified into energy-based and eigenvector-based. The methods may also be classified into ones based on time domain sample values and ones based on frequency domain parameters. Specifically, the FADV may be calculated based on average amplitude on respective channels in the short time frame data, or based on eigenvector analysis on a covariance matrix including time domain sample values of respective channels in the short time frame data (i.e., portions corresponding to respective microphones in the short time frame data), or based on average amplitude on frequency domain of respective channels in the short time frame data, or based on eigenvector analysis on spectral parameters of respective channels in the short time frame data.

This method calculates the FADV directly based on the short-time average amplitudes of the channels (i.e., audio data recorded through respective microphones). That is to say, the FADV is calculated based on average amplitude on respective channels in the short time frame data.

>

First, the root mean square ampf_m(θ) of each channel m of one short time frame is calculated, and then a vector (ampf₁(θ), ampf₂(θ), ..., ampf_M(θ)) is obtained, where ampfmθ=∑n=1NX⁢mn2N where θ represents a desired location of the sound source.

To facilitate comparison with reference frame amplitude difference vector (RFADV) (to be described later), preferably, a normalization of root mean square (RMS) is performed on this vector to obtain the FADV ampf(θ): ampfθ=ampf1θ,ampf2θ,…,ampfMθampf1θ×ampf1θ+ampf2θ×ampf2θ+…+ampfMθ×ampfMθ

According to this method, the FADV is calculated based on eigenvector analysis on a covariance matrix including time domain sample values of respective channels in the short time frame data. First, the covariance matrix CovX of the short time frame data X is calculated by CovX=X*Xʹ, where asterisk operator (*) means to perform a convolution operation, and apostrophe operator (') means to perform a complex conjugate transpose. For real value matrix or vector, apostrophe operator (') means to perform a transpose.

Then, the largest eigenvector of the covariance matrix is calculated as the FADV by eigendecomposition.

According to this method, the FADV is calculated as an average amplitude on frequency domain of respective channels in the short time frame data. The frequency domain of FFT can be divided into a number W of frequency bins B_i, i= 1, ..., W, where frequencies covered by frequency bin B_i are lower than that covered by frequency bin B_j if i < j. If it is known that some frequency bins may be affected by noise, it is possible to exclude the noisy frequency bins and only use a set of other frequency bins to calculate the FADV. For example, it is possible to exclude low frequency bins and high frequency bins to reduce the influence of low frequency noise and high frequency noise.

It is assumed that a set BU of L frequency bins to be used in calculating the FADV is denoted as BU= {Bin(1), Bin(2), ..., Bin(L)}, where Bin(i) ∈ {B₁, ..., B_W}. If one of the W frequency bins is completely covered by all the frequency bands which are determined or assumed as not being affected by noise, the bin may be included in the set BU. Alternatively, if one of the W frequency bins is completely or mainly covered by all the frequency bands which are determined or assumed as not being affected by noise, the bin may be included in the set BU.

In an example, it is determined that a frequency range from frequency nStartFrequency to frequency nEndFrequency are not affected by noise and is used to calculate the FADV. Let BU= {Bin(1), Bin(2), ..., Bin(L)} = {B_StartBin, B_StartBin+1, ..., B_StartBin+L-1}, then the set BU can be determined by determining StartBin=ceilnStartFrequency×W/SamplerateEndBin=floornEndFrequency×W/Samplerate
where ceil(v) is a function returning a minimum one of all the integers greater than or equal to a value v, and floor(v) is a function returning a maximum one of all the integers smaller than or equal to a value v.

In one implementation, it is possible to set nStartFrequency=300 and nEndFrequency =8000.

First, N samples of each channel m is grouped into S sub-frames, and each sub-frame contains K samples. xm,1⁢xm,2…xm,N⇒Sm=xm,1,1xm,2,1⋯xm,S,1xm,1,2xm,2,2⋯xm,S,2⋮⋮⋯⋮xm,1,Kxm,2,K⋯xm,S,K

As a special case, the number S of the sub-frames can be set to 1, which means the spectral analysis is performed directly on the all N samples of one short time frame.

Then, spectral analysis is performed on each sub-frame to obtain parameters F_m in spectral domain. For example, it is possible to apply FFT analysis on each sub-frame to obtain FFT parameters. Of course, other spectral analysis such as Mel-frequency cepstral coefficients (MFCCs) analysis, Sub-band Energy analysis in critical band or Mel band may also be employed to obtain other frequency domain parameters. It is also possible to employ complex spectral parameters such as complex FFT parameters or amplitude spectral parameters such as amplitude of FFT parameters. Fm=fftSm=fm,1,1fm,2,1⋯fm,S,1fm,2,1fm,2,2⋯fm,S,2⋮⋮⋯⋮fm,1,Kfm,2,K⋯fm,S,K
where fft() represents FFT.

Then, frequency domain parameters of all the channels in each frequency bin Bin(i) ∈ BU are calculated as P_Bin(i).PBini=f1,1,Binif1,2,Bini⋯f1,S,Binif2,1,Binif2,2,Bini⋯f2,S,Bini⋮⋮⋮⋮fM,1,BinifM,2,Bini⋯fM,S,Bini

Then, the amplitude ampf_m(θ) of each channel m of one short time frame is calculated by ampfmθ=∑i=1L∑s=1Sfm,s,Bini2
where |f_m,sBin(i)| means to calculate the absolute value of f_m,s,Bin(i).

Then a vector (ampf₁(θ), ampf₂(θ), ..., ampf_M(θ)) is obtained, where θ represents a desired location of the sound source.

Then, to facilitate comparison with RFADV (to be described later), preferably, a normalization of root mean square (RMS) is performed on this vector to obtain the FADV ampf(θ) by ampfθ=ampf1θ,ampf2θ,…,ampfMθampf1θ×ampf1θ+ampf2θ×ampf2θ+…+ampfMθ×ampfMθ.

According to this method, the FADV is calculated based on eigenvector analysis on spectral parameters of respective channels in the short time frame data. As described on the foregoing, N samples of each channel m is grouped into S sub-frames, and each sub-frame contains K samples. In case that all the samples of the channel are grouped into sub-frames, it is possible to obtain the spectral parameters of each of the channels by performing spectral analysis on the sub-frames of the channel. For example, for matrix P_Bin(i) containing frequency domain parameters of all the channels in each frequency bin Bin(i) ∈ BU, a covariance matrix is calculated by: CovBini=PBini*PBiniʹ.

In one implementation, it is possible to obtain a first covariance matrix as a sum of second covariance matrices. Each of the second covariance matrices corresponds to respective one of used frequency bins and includes spectral parameters of all the sub-frames of all the channels for the respective used frequency bin. Accordingly, it is possible to calculate the FADV based on the eigenvector analysis on the first covariance matrix.

Specifically, the FADV can be calculated by finding the largest eigenvector based on the covariance matrices for the frequency bins. For example, it is possible to calculate a sum Cov_sum of the covariance matrices of different bins by Covsum=∑i=1LCovBini
and then calculate the largest eigenvector ampf(θ) of the sum Cov_sum as the FADV.

As a special case, the number S of sub-frames for one short time frame equals to 1, that is to say, the grouping is not performed. This means that the spectral analysis is performed directly on the all N samples of one short time frame for each channel to obtain the spectral parameters of the channel. That is, frequency domain parameters of all the channels in all the frequency bin Bin(i) ∈ BU are calculated as P_f by Pf=fftX=f1,Bin1f1,Bin2⋯f1,BinLf2,Bin1f2,Bin2⋯f2,BinL⋮⋮⋮⋮fM,Bin1fM,Bin2⋯fM,BinL

In this case, the FADV is calculated as the largest eigenvector of a covariance matrix which includes spectral parameters of all the used frequency bins of all the channels. For example, for matrix P_f containing frequency domain parameters of all the channels for all the frequency bins Bin(i) ∈ BU, a covariance matrix is calculated by: Covf=Pf*Pfʹ
and then the largest eigenvector ampf(θ) of the covariance matrix Cov_f is calculated as the FADV.

In another implementation, it is possible to calculate the FADV by averaging the largest eigenvectors of covariance matrices. Each of the covariance matrices corresponds to respective one of used frequency bins and includes spectral parameters of all the sub-frames of all the channels for the respective used frequency bin. For example, it is possible to calculate the largest eigenvector ampf_Bin(i) of the covariance matrix Cov_Bin(i) for each frequency bin Bin(i) and calculate the average of the eigenvectors as the FADV. ampf=1L∑i=1LampfBini.

Further more, the FADV may be calculated in an adaptive way. That is to say, vector calculator 101 may be configured to calculate the FADV adaptively based on the short time frame data and its previous short time frame data.

In one implementation, it is possible to calculate the desired FADV adaptively by calculating a current FADV based on the short time frame data, and smoothing the current FADV and a historic FADV based on the previous short time frame data as the desired FADV. The desired FADV or the current FADV may be used as a historic FADV for the next desired FADV. For example, in case of having calculated the FADV ampf_curr (current FADV) of a current short time frame with the method as described in the above, it is possible to calculate a smoothed FADV ampf_smooth (desired FADV) by ampfsmooth=a*ampfcurr+1-a*ampfhist,
where ampf_hist is the historic FADV obtained based on the previous short time frame data. For example, it is possible to use ampf_curr or ampf_smooth for the current short time frame as ampf_hist for the next short time frame. α is a constant to adjust the importance of the FADV of the current short time frame.

In another implementation, it is possible to calculate the FADV according to an eigenvector-based method (based on time domain sample values or based on frequency domain parameters). In this case, it is possible to obtain the final covariance matrix (summed covariance matrix, covariance matrix for a used frequency bin, or covariance matrix for all the used frequency bins) for calculating an eigenvector based on the short time frame data by deriving a current covariance matrix based on the short time frame data, and smoothing the current covariance matrix and a historic covariance matrix based on the previous short time frame data as the final covariance matrix. The final covariance matrix for calculating an eigenvector based on the short time frame data or the current covariance matrix may be used as a historic covariance matrix for the next final covariance matrix. For example, it is possible to calculate the covariance matrix Cov_smooth (final covariance matrix) based on data of the current short time frame and the previous short time frames, and calculate the FADV for the current short time frame based on the covariance matrix. As one example, it is possible to calculate the covariance matrix for eigenvector analysis of one frame by: Covsmooth=a*Covcurr+1-a*Covhist,
where Con_cur (current covariance matrix) is the covariance matrix calculated based on data of the current short time frame, Cov_hist (historic covariance matrix) is the historic covariance matrix based on data of the previous short time frames. For example, it is possible to use Cov_cur or Cov_smooth as Cov_hist for the next short time frame. α is a constant to adjust the importance of the covariance matrix of the current short time frame.

Returning to Fig. 1, similarity evaluator 102 is configured to evaluate similarity between the FADV and each of a plurality of RFADVs 104. Each of the plurality of RFADVs reflects differences between amplitudes captured by the microphones of the array during recording sound from one of a plurality of candidate locations.

For comparison with the FADV, the RFADVs have the same format as the FADV. Because each RFADV reflects the amplitude differences associated with one of the candidate locations, the RFADV is associated with the candidate locations. The term "candidate location" means that the sound source may locate at the location and originate the sound for the current short time frame at the location.

It is possible to assume an even probability distribution for all locations, and thus the candidate locations may include all the locations spaced at an even interval depending on the localizing resolution. Preferably, to reduce the computation complexity, the candidate location may be a subset of all the locations. The subset may be different in different scenarios based on a prior knowledge of the source location's probability distribution.

Various methods can be adopted to calculate the similarity between the FADV ampf and the RFADV ampr(θ). For example, it is possible to directly measure the similarity through a distance ampdis(θ) between the FADV and the RFADV In this case, a larger distance means the lower similarity, and a smaller distance means the higher similarity. For another example, the similarity may be an inverse of the distance.

The distances can be implemented as Euclidean distance. ampdisθ=‖ampf-amprθ‖=∑i=1Nampfi-ampriθ2.
where θ represents a candidate location, ampf_i and ampr_i(θ) represent the i-th dimension of the FADV ampf and the RFADV ampr(θ) respectively.

The distance ampdis(θ) can also be calculated based on inner product: ampdisθ=1const+∑i=1Nampfi.ampriθ
where const is a constant to avoid division by a small number.

Estimator 103 is configured to estimate a desired location of sound source based at least on the candidate locations and associated similarity. For example, a RFADV having the highest similarity to the FADV may be found, and the candidate location associated with the RFADV may be estimated as the desired location of the sound source. Further, the desired location may be estimated with reference to an estimation result obtained through another sound source localization method, for example, a method based on time difference.

Fig. 3 depicts an example method 300 of performing sound source localization according to an embodiment of the present invention.

As illustrated in Fig. 3, method 300 starts from step 301. At step 303, a FADV is calculated based on short time frame data acquired through an array of microphones. The FADV reflects differences between amplitudes captured by microphones of the array during recording the short time frame data. As described on the foregoing, there are multiple methods to calculate the FADV. The methods may be classified into energy-based and eigenvector-based. The methods may also be classified into ones based on time domain sample values and ones based on frequency domain parameters. Specifically, the FADV may be calculated based on average amplitude on respective channels in the short time frame data, or based on eigenvector analysis on a covariance matrix including time domain sample values of respective channels in the short time frame data, or based on average amplitude on frequency domain of respective channels in the short time frame data, or based on eigenvector analysis on spectral parameters of respective channels in the short time frame data.

As an example of calculating the FADV based on eigenvector analysis on spectral parameters of respective channels in the short time frame data, as described in the foregoing, the spectral parameters of each of the channels may be obtained by performing spectral analysis on a plurality of sub-frames of the channel, wherein all the samples of the channel are grouped into the sub-frames. In this case, a first covariance matrix may be obtained as a sum of second covariance matrices. Each of the second covariance matrices corresponds to respective one of used frequency bins and includes spectral parameters of all the sub-frames of all the channels for the respective used frequency bin. The FADV may be calculated based on the eigenvector analysis on the first covariance matrix. Alternatively, the FADV may be calculated by averaging the largest eigenvectors of covariance matrices. Each of the covariance matrices corresponds to respective one of used frequency bins and includes spectral parameters of all the sub-frames of all the channels for the respective used frequency bin.

As another example of calculating the FADV based on eigenvector analysis on spectral parameters of respective channels in the short time frame data, as described in the foregoing, the spectral parameters of each of the channels may be obtained by performing spectral analysis directly on all the samples of the channel. In this case, the FADV may be calculated as the largest eigenvector of a covariance matrix. The covariance matrix includes spectral parameters of all the used frequency bins of all the channels.

Further more, the FADV may be calculated adaptively based on the short time frame data and its previous short time frame data.

As an example, the FADV may be calculated adaptively by calculating a current frame amplitude difference vector based on the short time frame data, and smoothing the current frame amplitude difference vector and a historic frame amplitude difference vector calculated adaptively based on the previous short time frame data as the frame amplitude difference vector. The frame amplitude difference vector or the current frame amplitude difference vector may be used as a historic frame amplitude difference vector for the next frame amplitude difference vector.

As another example, the FADV may be calculated according to an eigenvector-based method, and the final covariance matrix for calculating an eigenvector based on the short time frame data may be obtained by deriving a current covariance matrix based on the short time frame data, and smoothing the current covariance matrix and a historic covariance matrix for calculating an eigenvector based on the previous short time frame data as the final covariance matrix. The final covariance matrix for calculating an eigenvector based on the short time frame data or the current covariance matrix may be used as the historic covariance matrix for the next final covariance matrix.

At step 305, similarity between the FADV and each of a plurality of RFADVs is evaluated. Each of the plurality of RFADVs reflects differences between amplitudes captured by microphones of the array during recording sound from one of a plurality of candidate locations.

At step 307, a desired location of sound source is estimated based at least on the candidate locations and associated similarity.

At step 309, the method ends.

Fig. 4 is a block diagram illustrating an example apparatus 400 for performing sound source localization according to an embodiment of the present invention.

As illustrated in Fig. 4, apparatus 400 includes a vector calculator 401, a similarity evaluator 402, an estimator 403 and a reference vector calculator 405. Vector calculator 401, similarity evaluator 402 and estimator 403 have the same function as vector calculator 101, similarity evaluator 102 and estimator 103, and will not be described in detail hereafter.

Reference vector calculator 405 may be configured to calculate the RFADVs based on audio data obtained by capturing sound originated from the candidate locations respectively through the array. In this case, for each candidate location θ, a reference amplitudes r_m(θ) of each microphone m is obtained by placing a sound source with a fixed energy at the location θ and measuring the average amplitude of the microphone m. Accordingly, a vector (r₁(θ), r₂(θ), ..., r_M(θ)) is obtained. To facilitate comparison with the FADV, preferably, a normalization of root mean square (RMS) is performed on the vector. Then the RFADV ampr(θ) for the candidate location θ can be calculated by: amprθ=r1θ,r2θ,…,rMθr1θ×r1θ+r2θ×r2θ+…+rMθ×rMθ.

Alternatively, reference vector calculator 405 may also be configured to calculate the RFADVs based on sensitivity of the microphones of the array to sound originated from the candidate locations. For example, the sensitivity of a microphone to sound originated from locations can be defined through directionality pattern of the microphone. Accordingly, the RFADVs can be estimated according to the directionality pattern of the microphones.

For example, in case of the CMA, when the sound source is placed at location θ and no noise is present, the theoretical amplitude of microphone 201, 202 and 203 (see Fig. 2) can be estimated according to their cardioid directionality pattern by r1θ=0.5+0.5⁢cosθr2θ=0.5+0.5⁢cosθ+23⁢πr3θ=0.5+0.5⁢cosθ+43⁢π

Then for CMA, r₁(θ)× r₁(θ)+r₂(θ) ×r₂(θ)+r₃(θ) × r₃(θ) =1.15 holds for all the locations.

Then the RFADV for location θ is amprθ=0.5+0.5⁢cosθ,0.5+0.5⁢cosθ+23⁢π,…,0.5+0.5⁢cosθ+43⁢π1.15.

Further, by considering the influence of noise, the reference amplitudes of microphones 201, 202 and 203 may be calculated byr1θ=0.5+0.5⁢cosθ+n1θr2θ=0.5+0.5⁢cosθ+23⁢π+n2θr3θ=0.5+0.5⁢cosθ+43⁢π+n3θ,
where n₁(θ), n₂(θ) and n₃(θ) are assumed as noise's amplitudes on the microphones in case that the sound is from location θ.

Assuming that the noise is independent of the sound source's location and the noise level of three microphones is the same, n₁(θ), n₂(θ) and n₃(θ) are constant n₀. n₀ can be set based on an estimated noise level in specific application scenarios. n₁(θ), n₂(θ) and n₃(θ) can also be estimated by the signal noise ratio (SNR) of the recorded signals by the microphone SNR=PsignalPnoise=AsignalAnoise2.

Then Anoise=1SNR⁢Asignal.

Then r1θ=0.5+0.5⁢cosθ+1SNRr2θ=0.5+0.5⁢cosθ+23⁢π+1SNRr3θ=0.5+0.5⁢cosθ+43⁢π+1SNR.

Various methods can be used to estimate SNR. As one example, SNR can be estimated by using a voice activity detection algorithm to classify voice signal segment and noise segment of the recording of an omni-directional microphone, and then comparing the power of signal segment and the power of noise segment. Further, the SNR estimation can be done in runtime in an adaptive way to handle the change of SNR, and in response, the RFADVs will be updated accordingly.

Fig. 5 depicts an example method 500 of performing sound source localization according to an embodiment of the present invention.

As illustrated in Fig. 5, method 500 starts from step 501. At step 502, the RFADVs are acquired by capturing sound originated from the candidate locations respectively through the array. In this case, for each candidate location θ, a reference amplitudes rm(θ) of each microphone m is obtained by placing a sound source with a fixed energy at the location θ and measuring the average amplitude of the microphone m. Alternatively, at step 502, the RFADVs may be calculated based on sensitivity of the microphones of the array to sound originated from the candidate locations.

Steps 503, 505, 507 and 509 have the same function as step 303, 305, 307 and 309, and will not be described in detail herein.

Fig. 6 is a block diagram illustrating an example apparatus 600 for performing sound source localization according to an embodiment of the present invention.

As illustrated in Fig. 6, apparatus 600 includes a vector calculator 601, a similarity evaluator 602, an estimator 603 and a possibility evaluator 606. Vector calculator 601 and similarity evaluator 602 have the same function as vector calculator 101 and similarity evaluator 102, and will not be described in detail hereafter.

Possibility evaluator 606 is configured to evaluate possibility that each of a plurality of possible locations is the desired location according to an audio localization method based on time difference. Reference to the term "possible locations" is only for purpose of distinguishing from the candidate locations in the above embodiments based on amplitude difference. The possible locations are dependent on the method based on time difference. The term "possibility" is dependent on the measurement adopted by the method based on time difference to evaluate the closeness of possible locations to the desired location.

Estimator 603 is configured to estimate the desired location based on the candidate locations, their similarity, the possible locations and their possibility. Estimator 603 has two kinds of information to estimate the desired location. One is the candidate locations and their similarity, another is the possible locations and their possibility. Considering that one kind of information is a refinement to another, various policies may be adopted to estimate the desired location. For example, the estimation may be performed in similar to a voting problem.

Fig. 7 depicts an example method 700 of performing sound source localization according to an embodiment of the present invention.

As illustrated in Fig. 7, method 700 starts from step 701. Steps 703 and 705 have the same function as step 303 and 305, and will not be described in detail herein.

After step 705, method 700 proceeds to step 706. At step 706, possibility that each of a plurality of possible locations is the desired location is evaluated according to an audio localization method based on time difference.

At step 707, the desired location is estimated based on the candidate locations, their similarity, the possible locations and their possibility. Estimator 603 has two kinds of information to estimate the desired location.

Method 700 ends at step 709.

It should be noted that step 706 may be performed before step 705, or in parallel to step 705.

Fig. 8 is a block diagram illustrating an example apparatus 800 for performing sound source localization according to an embodiment of the present invention.

As illustrated in Fig. 8, apparatus 800 includes a vector calculator 801, a similarity evaluator 802, an estimator 803, a possibility evaluator 806, a first function generator 807, a second function generator 808 and a third function generator 809. Vector calculator 801, similarity evaluator 802 and possibility evaluator 806 have the same function as vector calculator 601, similarity evaluator 602 and possibility evaluator 606, and will not be described in detail hereafter.

First function generator 807 is configured to derive a first probability function for estimating probability that all locations are the desired location based on the possible locations and their possibility. The first probability function may estimate the probability that the possible locations are the desired location. Furthermore, the first probability function may also estimate the probability that other locations are the desired location.

Various functions can be used to derive the first probability function of different locations based on the possibility.

For example, the possibility is measured by steered response power (SRP). One method is to directly use the steered response powerP˜bPHAT (θ) corresponding to different locations θ as the first probability function prob_phase(θ) of different locations by probphaseθ=P˜bPHATθ

For another example, it is possible to derive the first probability function prob_phase(θ) of different locations θ based on the steered response powerP˜bPHAT (θ) corresponding to different locations by probphaseθ=e-1/P˜bPHATθ2CC=1L∑θ=θ1θL1/P˜bPHATθ2

Second function generator 808 is configured to derive a second probability function for estimating probability that all locations are the desired location based on the candidate locations and their similarity. The second probability function may estimate the probability that the candidate locations are the desired location. Furthermore, the second probability function may also estimate the probability that other locations are the desired location.

The second probability function can be estimated with various methods.

For example, the second probability function prob_amp(θ) of all locations θ can be derived based on the distances ampdis(θ) between the FADV and the RFADVs by probampθ=e-ampdis⁢θ2CC=1L∑θ=θ1θLampdis⁢θ2

For another example, the second probability function prob_amp(θ) of all locations θ can be also estimated by probampθ=1ampdisθ

Third function generator 809 is configured to derive a combined probability function for estimating probability that all locations are the desired location based on the first probability function and the second probability function. The combined probability function may estimate the probability that the possible locations and the candidate locations are the desired location. Further more, the combined probability function may also estimate the probability that other locations are the desired location.

Various methods can be used to derive the combined probability function based on two probability functions. For example, it is possible to derive the combined probability function by multiplying the first and the second probability functions as follows:probcombθ=probampθ*probphaseθ

Fig. 10 depicts a graph illustrating an example of a combined probability function prob_comb(θ) obtained by multiplying a first probability function prob_phase(θ) and a second probability function prob_amp(θ). In Fig. 10, the horizontal axis indicates locations, and the longitudinal axis indicates the probability that the locations are a desired location of sound source.

Estimator 803 is configured to estimate the location θ_ampsrp with the highest prob_comb(θ) as the desired location, i.e., θampsrp=argmaxθprobcombθ

The location θ_ampsrp is also called a combined estimation result.

In the example of Fig. 10, the sound source with the largest energy is located at 307 degrees. There are multiple angles with local maxima in the first probability functions, including 50 degrees, 182 degrees and 307 degrees, and the angle with the largest prob_phase(θ) is 50 degrees. The angle with the largest prob_amp(θ) is 288 degrees, which is close to the angle of the sound source but not accurate. prob_comb(θ) can be obtained by multiplying prob_amp(θ) with prob_phase(θ). The angle with the largest prob_comb(θ) is 305 degrees, which is very close to the real angle.

Preferably, estimator 803 is further configured to choose the closest one to the location having the largest combined probability from one or more peak locations in the first probability function or from one or more possible locations having the higher possibility.

For example, if the combined estimation result is close to the estimated location by the time difference based algorithm (i.e., possible locations having the higher possibility), the combined estimation result can be adjusted to the estimated location.

For example, if the combined estimation result is close to one potential location, i.e., one local maxima (peak) of SRP curve, the combined estimation result can be adjusted to that location.

As an example, it is possible to estimate a location θ_phase by θphase=argmaxθprobphaseθ.

Then θ_ampsrp is compared with θ_phase.

If diff(θ_ampsrp, θ_phase) < threshold, then θ_ampsrp = θ_phase.

As another example, it is possible to calculate all the local maxima of SRP curves as θ_{phase_1}, θ_{phase_2}, ..., θ_{phase_C}. Then the local maxima θ_{phase_c} closest to θ_ampsrp is found as the follows:

If diff(θ_ampsrp, θ_{phase_c}) < threshold, then θ_ampsrp= θ_{phase_c}.

In the example of Fig. 10, the local maxima θ_{phase_c} closest to θ_ampsrp =305 is 307 degrees. If using a threshold = 10, θ_ampsrp will be refined as 307 degree.

Alternatively, the refinement can be performed by comparing the θ_ampsrp with the angle having the largest prob_phase(θ) and the angle having the largest prob_amp(θ). If θ_ampsrp is closer to the angle having the largest prob_phase(θ) than to the angle having the largest prob_amp(θ), θ_ampsrp can be refined as the angle having the largest prob_phase(θ).

Fig. 9 depicts an example method 900 of performing sound source localization according to an embodiment of the present invention.

As illustrated in Fig. 9, method 900 starts from step 901. Steps 903, 905 and 906 have the same function as steps 703, 705 and 706, and will not be described in detail herein.

After step 906, method 900 proceeds to step 907. At step 907, a first probability function for estimating probability that all locations are the desired location is derived based on the possible locations and their possibility.

At step 908, a second probability function for estimating probability that all locations are the desired location is derived based on the candidate locations and their similarity.

At step 909, a combined probability function for estimating probability that all locations are the desired location is calculated based on the first probability function and the second probability function.

At step 910, a location having the highest combined probability is estimated as the desired location, based on the combined probability function.

Method 900 ends at step 911.

It should be noted that step 907 may be executed at any time between steps 905 and 909, and step 908 may be executed at any time between steps 906 and 909.

Further, the first probability function may be derived by incorporating a first factor, and the second probability function may be derived by incorporating a second factor. The first factor and the second factor enable the combined probability function to be more sensitive to the similarity.

For example, Equation (1) may be adjusted to probphaseθ=Rphase+P˜bPHATθ
where a smaller R_phase will make the final decision more dependent on the amplitude difference cues and vice versa. In one implementation for CMA, R_phase =0, then probphaseθ=P˜bPHATθ.

For example, Equation (3) may be adjusted to C=Ramp*1L∑θ=θ1θL1/P˜bPHATθ2
where a smaller R_ampe will make the final decision more dependent on the amplitude difference cues and vice versa. In one implementation for CMA, R_amp =1.

For example, Equation (5) may be adjusted to C=Ramp*1L∑θ=θ1θLampdis⁢θ2
where R_amp is a constant to adjust the probability function. A smaller R_amp will make the probability function more sensitive to the distances between the FADV and RFADVs. Since this probability function is used to model the angle estimation result by another method, a smaller R_amp can make the final decision more dependent on the amplitude difference cues and vice versa. In one implementation for CMA, R_amp =3.

For example, Equation (6) may be adjusted to probampθ=1Ramp+ampdisθ

Similarly, a smaller Ramp will make the final decision more dependent on the amplitude difference cues and vice versa. In one implementation for CMA, R_amp =6.

In a modification to the embodiments of Fig. 8 and Fig. 9, third function generator 809 (step 909) and one of first function generator 807 (step 907) and second function generator 808 (step 908) may be omitted. In this case, another of first function generator 807 (step 907) and second function generator 808 (step 908) may derive a probability function for estimating probability that all locations are the desired location based on the possible locations and their possibility, or based on the candidate locations and their similarity. Accordingly, estimator 803 (step 910) is further configured to choose one having the highest probability based on the probability function from one or more of the candidate locations having the higher similarity, or from one or more of the possible locations having the higher possibility.

For example, a time difference based algorithm (such as SRP) returns all the angles with a local maxima value in steered response power curve for all angles while an amplitude difference based algorithm returns probability function. Then the probability function's values on the returned angles by SRP-PHAT are compared, and the angle with the largest probability is chosen as the final estimated angle.

In the example of Fig. 10, there are multiple angles with local maxima in the SRP-PHAT including 50 degrees, 182 degrees and 307 degrees, The prob_amp(θ) on 307 degrees is larger than the prob_amp(θ) on 50 degrees, 182 degrees, thus 307 degrees is chosen as final estimated angle.

In a further example of the embodiments of Fig. 6 and Fig. 7, estimator 603 (step 707) is further configured to find a pair of the candidate location having the higher similarity and the possible location having the higher possibility, which are closest to each other, and choose one of the pair of the candidate location and the possible location as the desired location.

In the example of Fig. 10, there are multiple angles with local maxima in the SRP-PHAT including 50 degrees, 182 degrees and 307 degrees. The angle with the smallest distance between the frame amplitude difference vector and reference amplitude difference vectors is 288 degrees, which is returned by the amplitude difference based algorithm. The SRP candidate closest to the candidate provided by amplitude difference based algorithm is 307 degrees, which is chosen as the final estimated angle.

Fig. 11 is a block diagram illustrating an exemplary system for implementing the aspects of the present invention.

In Fig. 11, a central processing unit (CPU) 1101 performs various processes in accordance with a program stored in a read only memory (ROM) 1102 or a program loaded from a storage section 1108 to a random access memory (RAM) 1103. In the RAM 1103, data required when the CPU 1101 performs the various processes or the like is also stored as required.

The CPU 1101, the ROM 1102 and the RAM 1103 are connected to one another via a bus 1104. An input / output interface 1105 is also connected to the bus 1104.

The following components are connected to the input / output interface 1105: an input section 1106 including a keyboard, a mouse, or the like ; an output section 1107 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), or the like, and a loudspeaker or the like; the storage section 1108 including a hard disk or the like ; and a communication section 1109 including a network interface card such as a LAN card, a modem, or the like. The communication section 1109 performs a communication process via the network such as the internet.

A drive 1110 is also connected to the input/ output interface 1105 as required. A removable medium 1111, such as a magnetic disk, an optical disk, a magneto - optical disk, a semiconductor memory, or the like, is mounted on the drive 1110 as required, so that a computer program read therefrom is installed into the storage section 1108 as required.

In the case where the above-described steps and processes are implemented by the software, the program that constitutes the software is installed from the network such as the internet or the storage medium such as the removable medium 1111.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.