Evaluation

Measuring the results of a source separation approach is a challenging problem. Generally, there are two main categories for evaluating the outputs of a source separation approach: objective and subjective. Objective measures rate separation quality by performing a set of calculations that compare the output signals of a separation system to the ground truth isolated sources. Subjective measures involve having human raters give scores for the source separation system’s output.

Objective and subjective measures both have benefits and drawbacks. Objective measures struggle because there are many aspects of human perception that are extremely difficult capture by computational means alone. However, compared to subjective measures, they are much faster and cheaper to obtain. On the other hand, subjective measures are expensive, time-consuming, and subject to the variability of human raters, but they can be more reliable than objective measures because actual human listeners are involved in the evaluation process.

Objective measures are, by far, much more popular than subjective measures, but we feel it is worth understanding them both to some extent.

Objective Measures

SDR, SIR, and SAR

Source-to-Distortion Ratio (SDR), Source-to-Interference Ratio (SIR), and Source-to-Artifact Ratio (SAR) are, to date, the most widely used methods for evaluating a source separation system’s output.

An estimate of a Source \(\hat{s}_i\) is assumed to actually be composed of four separate components,

\[ \hat{s}_i = s_{\text{target}} + e_{\text{interf}} + e_{\text{noise}} + e_{\text{artif}}, \]

where \(s_{\text{target}}\) is the true source, and \(e_{\text{interf}}\), \(e_{\text{noise}}\), and \(e_{\text{artif}}\) are error terms for interference, noise, and added \text{artif}acts, respectively. The actual calculations of these terms is quite complex, so we refer the curious reader to the original paper for their exact calculation: [VGFevotte06].

Using these four terms, we can define our measures. All of the measures are in terms of decibels (dB), with higher values being better. To calculate they require access to the ground truth isolated sources and are usually calculated on a signal that has been divided into short windows of a few seconds long.

Source-to-Artifact Ratio (SAR)

\[ \text{SAR} := 10 \log_{10} \left( \frac{\| s_{\text{target}} + e_{\text{interf}} + e_{\text{noise}} \|^2}{ \| e_{\text{artif}} \|^2} \right) \]

This is usually interpreted as the amount of unwanted artifacts a source estimate has with relation to the true source.

Source-to-Interference Ratio (SIR)

\[ \text{SIR} := 10 \log_{10} \left( \frac{\| s_{\text{target}} \|^2}{ \| e_{\text{interf}} \|^2} \right) \]

This is usually interpreted as the amount of other sources that can be heard in a source estimate. This is most close to the concept of “bleed”, or “leakage”.

Source-to-Distortion Ratio (SDR)

\[ \text{SDR} := 10 \log_{10} \left( \frac{\| s_{\text{target}} \|^2}{ \| e_{\text{interf}} + e_{\text{noise}} + e_{\text{artif}} \|^2} \right) \]

SDR is usually considered to be an overall measure of how good a source sounds. If a paper only reports one number for estimated quality, it is usually SDR.

Note

As of this writing (October 2020), the best reported SDR for singing voice separation on MUSDB18 is \(7.24 dB\). [TM20] Recent research papers have been reporting vocal SDRs on MUSDB18 in the range of 6-7 dB. Compare the SDR of different systems at this Papers with Code link.

Signal-to-Noise Ratio (SNR)

This is not used as widely, but does appear sometimes in source separation:

\[ \text{SNR} := 10 \log_{10} \left( \frac{\| s_{\text{target}} \|^2}{ \| s_{\text{target}} - \hat{s} \|^2} \right) \]

where \(\hat{s}\) is the estimate of \(s_{\text{target}}\).

SI-SDR

Waveform shown at many different time scales from a few seconds to a few samples.

Fig. 19 The way SDR calculates the \(e_{\text{interf}}\), \(e_{\text{noise}}\), and \(e_{\text{artif}}\) terms can lead to issues where the original signal (top) can be horribly degraded (bottom) and still get a very high SDR score. Image used courtesy of Jonathan Le Roux. [LRWEH19]

A handful of issues have been brought up in the years since it was originally proposed. Although some of the issues are implementation-specific, one issue that persists is that SDR is easy to “cheat” on. The way that SDR calculates the \(e_{\text{interf}}\), \(e_{\text{noise}}\), and \(e_{\text{artif}}\) terms can cause issues where scores are artificially inflated.

Scale-Invariant Source-to-Distortion Ratio (SI-SDR) aims to remedy this by removing SDR’s dependency on the amplitude scaling of the signal. [LRWEH19] It also comes with accompanying SI-SAR, and SI-SIR, which corresponds to SAR and SIR described above, respectively. Although these measures are not sensitive to amplitude scaling, it is a quicker computation because it does not require windowing the estimated and ground truth signals like SDR.

In Fig. 19, the discrepancy between SDR and SI-SDR scores is shown. The top spectrogram shows the ground truth signal. Above it are its scores for SDR, SNR, and SI-SDR. As expected the ground truth signal gets high values for SDR, SNR, and SI-SDR (268.1 dB, infinity dB, and infinity dB, respectively. Shown above the plot). The bottom spectrogram shows a highly degraded version of the top signal. While the SI-SDR, and SNR are quite low (-4.72 dB and 1.26 dB, respectively), the SDR value is still quite high (11.56 db). For reference, this is a speech signal and this SDR value is higher than state-of-the-art speech separation systems from a few years ago.

Reported SDRs

Many times SDRs are reported in research papers. When this happens, it is common to show just one number for SDR, but usually this number represents the mean of a distribution of SDRs calculated on a dataset. This is not the best practice, unfortunately, and the authors of this tutorial are certainly guilty of it. But when reading papers, be aware that this number is a summary statistic that could obscure a whole distribution.

Does Higher SDR Imply Better Quality?

SDR doesn’t tell you the whole story. Consider the following four audio examples. First is the input mixture, then the ground truth reference source, followed by two outputs from two different source separation models named ConvTasnet and Open-Unmix, respectively. Have a listen:

%%capture
!pip install nussl
Mixture
Reference Source