%matplotlib inline
import numpy, scipy, matplotlib.pyplot as plt, IPython.display as ipd
import librosa, librosa.display
import stanford_mir; stanford_mir.init()
Novelty Functions¶
To detect note onsets, we want to locate sudden changes in the audio signal that mark the beginning of transient regions. Often, an increase in the signal's amplitude envelope will denote an onset candidate. However, that is not always the case, for notes can change from one pitch to another without changing amplitude, e.g. a violin playing slurred notes.
Novelty functions are functions which denote local changes in signal properties such as energy or spectral content. We will look at two novelty functions:
- Energy-based novelty functions (FMP, p. 306)
- Spectral-based novelty functions (FMP, p. 309)
Energy-based Novelty Functions¶
Playing a note often coincides with a sudden increase in signal energy. To detect this sudden increase, we will compute an energy novelty function (FMP, p. 307):
- Compute the short-time energy in the signal.
- Compute the first-order difference in the energy.
- Half-wave rectify the first-order difference.
First, load an audio file into the NumPy array x and sampling rate sr.
x, sr = librosa.load('audio/simple_loop.wav')
print(x.shape, sr)
Plot the signal:
plt.figure(figsize=(14, 5))
librosa.display.waveplot(x, sr)
Listen:
ipd.Audio(x, rate=sr)
RMS Energy¶
librosa.feature.rmse returns the root-mean-square (RMS) energy for each frame of audio. We will compute the RMS energy as well as its first-order difference.
hop_length = 512
frame_length = 1024
rmse = librosa.feature.rmse(x, frame_length=frame_length, hop_length=hop_length).flatten()
rmse_diff = numpy.zeros_like(rmse)
rmse_diff[1:] = numpy.diff(rmse)
print(rmse.shape)
print(rmse_diff.shape)
To obtain an energy novelty function, we perform half-wave rectification (FMP, p. 307) on rmse_diff, i.e. any negative values are set to zero. Equivalently, we can apply the function $\max(0, x)$:
energy_novelty = numpy.max([numpy.zeros_like(rmse_diff), rmse_diff], axis=0)
Plot all three functions together:
frames = numpy.arange(len(rmse))
t = librosa.frames_to_time(frames, sr=sr)
plt.figure(figsize=(15, 6))
plt.plot(t, rmse, 'b--', t, rmse_diff, 'g--^', t, energy_novelty, 'r-')
plt.xlim(0, t.max())
plt.xlabel('Time (sec)')
plt.legend(('RMSE', 'delta RMSE', 'energy novelty'))
Log Energy¶
The human perception of sound intensity is logarithmic in nature. To account for this property, we can apply a logarithm function to the energy before taking the first-order difference.
Because $\log(x)$ diverges as $x$ approaches zero, a common alternative is to use $\log(1 + \lambda x)$. This function equals zero when $x$ is zero, but it behaves like $\log(\lambda x)$ when $\lambda x$ is large. This operation is sometimes called logarithmic compression (FMP, p. 310).
log_rmse = numpy.log1p(10*rmse)
log_rmse_diff = numpy.zeros_like(log_rmse)
log_rmse_diff[1:] = numpy.diff(log_rmse)
log_energy_novelty = numpy.max([numpy.zeros_like(log_rmse_diff), log_rmse_diff], axis=0)
plt.figure(figsize=(15, 6))
plt.plot(t, log_rmse, 'b--', t, log_rmse_diff, 'g--^', t, log_energy_novelty, 'r-')
plt.xlim(0, t.max())
plt.xlabel('Time (sec)')
plt.legend(('log RMSE', 'delta log RMSE', 'log energy novelty'))
Spectral-based Novelty Functions¶
There are two problems with the energy novelty function:
- It is sensitive to energy fluctuations belonging to the same note.
- It is not sensitive to spectral fluctuations between notes where amplitude remains the same.
For example, consider the following audio signal composed of pure tones of equal magnitude:
sr = 22050
def generate_tone(midi):
T = 0.5
t = numpy.linspace(0, T, int(T*sr), endpoint=False)
f = librosa.midi_to_hz(midi)
return numpy.sin(2*numpy.pi*f*t)
x = numpy.concatenate([generate_tone(midi) for midi in [48, 52, 55, 60, 64, 67, 72, 76, 79, 84]])
Listen:
ipd.Audio(x, rate=sr)
The energy novelty function remains roughly constant:
hop_length = 512
frame_length = 1024
rmse = librosa.feature.rmse(x, frame_length=frame_length, hop_length=hop_length).flatten()
rmse_diff = numpy.zeros_like(rmse)
rmse_diff[1:] = numpy.diff(rmse)
energy_novelty = numpy.max([numpy.zeros_like(rmse_diff), rmse_diff], axis=0)
frames = numpy.arange(len(rmse))
t = librosa.frames_to_time(frames, sr=sr)
plt.figure(figsize=(15, 4))
plt.plot(t, rmse, 'b--', t, rmse_diff, 'g--^', t, energy_novelty, 'r-')
plt.xlim(0, t.max())
plt.xlabel('Time (sec)')
plt.legend(('RMSE', 'delta RMSE', 'energy novelty'))
Instead, we will compute a spectral novelty function (FMP, p. 309):
- Compute the log-amplitude spectrogram.
- Within each frequency bin, $k$, compute the energy novelty function as shown earlier, i.e. (a) first-order difference, and (b) half-wave rectification.
- Sum across all frequency bins, $k$.
Luckily, librosa has librosa.onset.onset_strength which computes a novelty function using spectral flux.
spectral_novelty = librosa.onset.onset_strength(x, sr=sr)
frames = numpy.arange(len(spectral_novelty))
t = librosa.frames_to_time(frames, sr=sr)
plt.figure(figsize=(15, 4))
plt.plot(t, spectral_novelty, 'r-')
plt.xlim(0, t.max())
plt.xlabel('Time (sec)')
plt.legend(('Spectral Novelty',))
Questions¶
Novelty functions are dependent on frame_length and hop_length. Adjust these two parameters. How do they affect the novelty function?
Try with other audio files. How do the novelty functions compare?
ls audio