Paper reading - joint beat and downbeat tracking with recurrent neural networks

1 summary

Recently, I have been working on projects related to music cards , Need to understand the basic characteristics of music , such as beats and downbeats Is the most basic feature .madmom It's a pair I found beats and downbeats Third party libraries are available for the detection of , So I studied hard , Write down the methods used and your understanding .

madmom Medium beats and downbeats The test is a recurrence Joint Beat and Downbeat Tracking with Recurrent Neural Networks This paper , The core idea is HMM, If the HMM Without a solid understanding , I suggest you read another article I wrote first understand HMM. This article will not HMM The basic concept of .

Before speaking about the thesis , Let me first explain some special terms in music , Align some concepts , It is convenient for the following narration . Some terms may not reappear in this article , But it helps to understand music , It was written together .

Noun	A term is used to explain
time (beat)	In music , Time is divided into equal basic units , Each unit is called a “ time ” Or one shot .
Strong shot (downbeat)	A strong beat in music . yes beats Subset . It is usually the beginning of each section beat
Section (measure/bar)	Music is alternating between strong and weak beats , It is to organize a section according to a certain law to form the smallest beat , And then cycle back and forth on this basis . A section usually has 2 pat ,3 pat ,4 Shoot or 6 Shoot and wait
Rhythm (meter)	beats per measure, The beat of a song , It is generally expressed as 1/4 Shoot or 2/4 Shoot or 3/8 Shoot and wait . such as 3/4 Beat representation 4 It's divided into 1 pat , Each section has 3 pat , The beat intensity is strong 、 weak 、 weak .
Global rhythm (tempo)	It's usually used bpm(beats per minute) To make the unit , Indicates how many... Are there per minute beats, To measure the speed of music , A piece of music can be composed of different tempo play .
Local rhythm (tempi)	Every two adjacent beats There can be differences between 1/16 note , or 1/8 note , or 1/4 And so on , This is the local rhythm
onset	The moment when a note is sounded by an instrument or a person .
Peak value (peak)	onset The peak of the envelope , concrete peak_pick.

Back to this paper . In a nutshell, this paper is to cut the signal into several frames, Every frame Will be in RNN After the network corresponds to a probability output , It means that we should frame yes beat still downbeat Still not .HMM We will use this to calculate the launch probability RNN result , hold frame stay measure The relative position and tempi As an implicit variable , Get the best hidden variable path , And decode the path into beats and downbeats.

Each step will be explained step by step below , The point is HMM And the prediction part . These two parts are a bit confusing .

2 Signal preprocessing

This part is actually a similar to MFCC The process of extracting speech signal features , As RNN The input of , Here is a brief introduction . First, the Hanning window is used to segment the signals with overlap , Divided into 100fps(frame per second), Based on this, the short-time Fourier transform (STFT). The phase feature is discarded , Only amplitude features are preserved , Because the human ear cannot recognize the phase . In order to ensure the accuracy of time domain and frequency domain , do STFT when , Used separately 1024,2048 and 4096 Three window sizes . Limit the frequency band to [30, 17000]Hz The scope of , And give each Octave. Divide up 3,6 and 12 Frequency bands , Corresponding to 1024,2048 and 4096 Three window sizes . At the same time, the first-order difference of the spectrum is also carried out , Also as a feature concat go in . Final , The characteristic dimension of the input network is 314.

3 Classification neural network

This part uses LSTM A network built by , The aim is to give each frame To classify , Output two probability values ,( yes beat Probability , yes downbeat Probability ), Take the larger as the frame The classification of . To avoid confusion ,downbeat Do not belong to beat.

meanwhile , Also set up $\theta =0.05$ The threshold of , Only greater than this threshold , Will be judged as beat perhaps downbeat. This is to reduce the interference of blank space at the beginning and end of the music .

Since each frame yes beat still downbeat We know the probability of , Then I'll take each one directly frame Category of maximum probability , If both probabilities are relatively small, it is considered that it is neither beat Neither downbeat, This is what happened ？

The ideal is very good , This is not the case , Don't be misled by the pictures in the paper , Let's see what the actual output of the model looks like .

chart 3-1 RNN Network output diagram

The output of the model is shown in the figure above 3-1 Shown , This is a 30s Left and right actual examples . The upper half of the graph is for each frame by beat Probability , The lower half of the graph is for each frame by downbeat Probability . Just look at downbeat In this part , It is not difficult to see that the probability is high frame The distance between them is long and short , And in a section (bar) among , Strong shot (downbeat) It should only appear once , The appearance of the second strong beat will interfere with it .RNN I don't know that the emergence of strong beat is periodic , So we need to HMM To determine which position is a strong shot , Which positions are weak shots , Where is nothing .

Think about it from another angle ,RNN I don't know music here meter How many shots , Without this information , want RNN It is very difficult to judge the strong beat and weak beat directly . Next section HMM Is the basis RNN Result , To choose the best beats and downbeats.

4 Dynamic Bayesian network （HMM）

This is the key part , Let's go into detail .Joint Beat and Downbeat Tracking with Recurrent Neural Networks The explanation of this part is not clear , Let's look directly at what it uses An Efficient State-Space Model for Joint Tempo and Meter Tracking As described in .

4.1 The original bar pointer model

As early as 2006 In the year ,Bayesian Modelling of Temporal Structure in Musical Audio I have proposed to use HMM solve beat tracking The way of the problem . Now the way , It is optimized on the basis of it , Let's take a look at the earliest version HMM How it was designed .

Let's make No $k$ individual frame The implicit variable of is ${\mathbf{x}}_k=[{\Phi }_k,{\dot{\Phi }}_k]$.${\Phi }_k\in \{1,2,...,M\}$ It means the first one $k$ individual frame In a certain section (bar) Relative position in ,1 Indicates the starting position ,$M$ Indicates the end position , This bar Divided into M A relative position , It is generally divided equally .${\dot{\Phi }}_k\in \{{\dot{\Phi }}_{min},{\dot{\Phi }}_{min}+1,...,{\dot{\Phi }}_{max}\}$ It means the first one $k$ individual frame Of tempi,${\dot{\Phi }}_{min}$ and ${\dot{\Phi }}_{max}$ It is a man-made upper and lower bounds . Put it in a more general way , hold ${\Phi }_k$ As displacement ,${\dot{\Phi }}_k$ It's speed ,$k+1$ individual frame The location of ${\Phi }_{k+1}$ Namely ${\Phi }_k+{\dot{\Phi }}_k$. Say a little music , Namely ${\dot{\Phi }}_k$ Represents the second $k$ individual frame It's a fraction of a note , such as 1/8 note , If you know this song is 4/4 If you shoot it , The first $k$ individual frame And went away $(1/8)/(4\ast 1/4)=1/8$ A section (bar), Here we use discrete $M$ A number to represent .

The observed variables are our frames Characteristic sequence of , Write it down as $\{{\mathbf{y}}_1,{\mathbf{y}}_2,...,{\mathbf{y}}_K\}$. We want to find a sequence of hidden variables ${\mathbf{x}}_{1:K}^{\ast }=\{{\mathbf{x}}_1^{\ast },{\mathbf{x}}_2^{\ast },...,{\mathbf{x}}_K^{\ast }\}$ bring

$${\mathbf{x}}_{1:K}^{\ast }=arg\underset{{\mathbf{x}}_{1:K}}{\max }P({\mathbf{x}}_{1:K}|{\mathbf{y}}_{1:K})\text{\hspace{1em}(4-1)}$$

type $(4-1)$ It can be used viterbi Algorithm to solve , For those who are not clear, please refer to my understand HMM. Solution $(4-1)$ You need to know three models , One is the initial probability model $P({\mathbf{x}}_1)$, The second is the state transition model $P({\mathbf{x}}_k|{\mathbf{x}}_{k-1})$, The third is the launch probability $P({\mathbf{y}}_k|{\mathbf{x}}_k)$.

（1） Initial probability
Initial probability $P({\mathbf{x}}_1)$, The author uses uniform distribution initialization , Just use data science later , Nothing to say. .

（2） Transfer probability
Transition probability is a key probability , Let's take a closer look at

$$\begin{array}{rl}P({\mathbf{x}}_k|{\mathbf{x}}_{k-1})& =P({\Phi }_k,{\dot{\Phi }}_k|{\Phi }_{k-1},{\dot{\Phi }}_{k-1})\\ & =P({\Phi }_k|{\dot{\Phi }}_k,{\Phi }_{k-1},{\dot{\Phi }}_{k-1})P({\dot{\Phi }}_k|{\Phi }_{k-1},{\dot{\Phi }}_{k-1})\end{array}$$

We put ${\Phi }_k$ Understood as displacement ,${\dot{\Phi }}_k$ Understood as speed , This should also be the reason why the sign of these two variables is only one sign of first derivative .

${\Phi }_k$ yes $k$ The location of the moment , It is determined by the position of the last moment ${\Phi }_{k-1}$ And the speed of the last moment ${\dot{\Phi }}_{k-1}$ decision , And $k$ Speed independent of time , There are

$$P({\Phi }_k|{\dot{\Phi }}_k,{\Phi }_{k-1},{\dot{\Phi }}_{k-1})=P({\Phi }_k|{\Phi }_{k-1},{\dot{\Phi }}_{k-1})$$

${\dot{\Phi }}_k$ yes $k$ The speed of the moment , It only works with $k-1$ The speed of the moment , Speed doesn't change much , It's not about location , There are

$$P({\dot{\Phi }}_k|{\Phi }_{k-1},{\dot{\Phi }}_{k-1})=P({\dot{\Phi }}_k|{\dot{\Phi }}_{k-1})$$

So there is

$$P({\mathbf{x}}_k|{\mathbf{x}}_{k-1})=P({\Phi }_k|{\Phi }_{k-1},{\dot{\Phi }}_{k-1})P({\dot{\Phi }}_k|{\dot{\Phi }}_{k-1})\text{\hspace{1em}(4-2)}$$

$P({\Phi }_k|{\Phi }_{k-1},{\dot{\Phi }}_{k-1})$ It is a calculation of displacement , Can be defined as

$$P({\Phi }_k|{\Phi }_{k-1},{\dot{\Phi }}_{k-1})=\{\begin{array}{ll}1,& if({\Phi }_{k-1}+{\dot{\Phi }}_{k-1}-1)\%M+1\\ 0,& otherwise\end{array}\text{\hspace{1em}(4-3)}$$

That is to say ${\Phi }_k=({\Phi }_{k-1}+{\dot{\Phi }}_{k-1}-1)\%M+1$ It means . The remainder here is for the next bar in , Position recount . Why should we reduce it first 1 Take the rest and add 1？ My guess is to avoid ${\Phi }_k=0$ The situation of , bring ${\Phi }_k$ The value is $\{1,2,...,M\}$, Just for the unity of symbols .

$P({\dot{\Phi }}_k|{\dot{\Phi }}_{k-1})$ Is the probability of a change in speed , Defined as

$$P({\dot{\Phi }}_k|{\dot{\Phi }}_{k-1})=\{\begin{array}{ll}1-p,& {\dot{\Phi }}_k={\dot{\Phi }}_{k-1}\\ \frac{p}{2},& {\dot{\Phi }}_k={\dot{\Phi }}_{k-1}+1\\ \frac{p}{2},& {\dot{\Phi }}_k={\dot{\Phi }}_{k-1}-1\end{array}\text{\hspace{1em}(4-4)}$$

$p$ Is a probability that the speed will change , We have artificially limited the speed to a distance of 1 Shift or remain constant in speed . At the border ${\dot{\Phi }}_{min}$ and ${\dot{\Phi }}_{max}$ On , It's a little different , Keep the speed within the limit .$p$ You need to learn .

chart 4-1 Schematic diagram of original state transition

The original state transition diagram is shown in Figure 4-1 Shown , It is obvious that the relationship between position and velocity .

（3） Emission probability
Here is the union RNN The result of . Let's assume that No $k$ individual frame by beat Is the probability that $b_k$, by downbeat Is the probability that $d_k$, neither beat Neither downbeat The probability of is $n_k$. Then the launch probability is

$$P({\mathbf{y}}_k|{\mathbf{x}}_k)=\{\begin{array}{ll}{b}_k,& s_k\in B\\ d_k,& b_k\in D\\ \frac{n_k}{{\lambda }_0-1},& otherwise\end{array}\text{\hspace{1em}(4-5)}$$

$B$ Express $x_k$ Is considered to be beat Set ,$D$ Express $x_k$ Is considered to be downbeat Set . How about $x_k$ Would be considered to be beat or downbeat Well ？ This point is not discussed in detail in the paper , I look at the A Multi-model Approach to Beat Tracking Considering Heterogeneous Music Styles In the practice , I guess , Is to place the position near the section bisection point states As beat perhaps downbeat, Near the first bisection point of each section is downbeat（ The first shot of each measure is a strong shot ）, The other equidistant points are beat. How close is this place , In fact, it is a range that can be set artificially . In equal parts , Is a parameter that the user needs to input in advance , For example, a song is 3/4 It was taken , So every section has 3 pat , Just three equal parts , Another example is that a song is 6/8 It was taken , So every section has 6 pat , Just six equal portions . If you don't know how many beats of music it is , Just count them one by one , Take the most probable , This will be discussed in the next section .

${\lambda }_0$ It's a super parameter , Take... From the paper ${\lambda }_0=16$ The best experimental results were obtained .$B$ and $D$ The scope and ${\lambda }_0$ relevant .

4.2 The original bar pointer model The shortcomings of

（1） Location （ Time ） The resolution of the
In the original text, it is time resolution , I'm talking about the position resolution , This is more convenient to understand . The author thinks that at different speeds , The required position resolution is different . Fast , Take big strides every time , The required position resolution is very low ; Slow , Each stride is small , The required position resolution is high . In a word , At different speeds , Different position resolution .

（2） Speed resolution
In the original tempo It's my speed here . The author thinks that the human ear perceives the change of speed , It's different at different speeds . For example, when the speed is 1 When , Speed up 1, become 2, It sounds like the change is obvious . But at a speed of 10 When , Speed and price 1, become 11, It sounds like nothing has changed . That is to say, the human hearing is not linear Of , It is log linear Of .

（3） Stability of speed
Use HMM when , There is a homogeneous Markov hypothesis , Think ${\Phi }_k$ Depend only on ${\dot{\Phi }}_k$, This may lead to the same beat in , The speed passes through several frames After that, great changes have taken place , That is, the stability of speed cannot be guaranteed .

4.3 The improved model

The paper aims at 4.2 The model is improved by the three points proposed in , The improvements are all aimed at the calculation of state transition . Improved state transition diagram , Here's the picture 4-2 Shown .

chart 4-2 Improved state transition diagram

（1） Improvement of position resolution
To put it bluntly, it is to adjust one according to the speed bar To be divided into several parts . From the picture 4-2 You can see from the horizontal of ,tempo The bigger ,bar position The more sparsely they are divided , On the contrary, the more dense . The division method is from the specific madmom From the realization of , Don't read what the paper says , What is said in the paper is a little vague .

""" max_bpm： User input , Represents the largest beats per minute, The default is 215 min_bpm： User input , The smallest beats per minute, The default is 55 fps： User input ,frame per second min_interval： To calculate the , The smallest frame per beat max_interval： To calculate the , maximal frame per beat """
# convert timing information to construct a beat state space
min_interval = 60. * fps / max_bpm # second per beat * frame per second = frame per beat
max_interval = 60. * fps / min_bpm

In the code slice above min_interval and max_interval By the largest bpm And the smallest bpm Identify each beat How many are divided into frames. One bar How many beats It is also input by the user , So a bar Of bar positions It is determined according to different speeds .

（2） Improvement of speed resolution
Speed at min_bpm and max_bpm Within the scope of , Press log linear Is divided into . See this paragraph for specific implementation https://github.com/CPJKU/madmom/blob/master/madmom/features/beats_hmm.py#L66. It should be said that it is being implemented , There is no such thing as speed , All changed into positions . Different bpm There are different location points under ,log linear The object of is location .

（3） Speed stability
In order to ensure the stability of speed , The speed can only be in each beat The position of , Change depends on a distribution , This distribution is obtained from several experiments .

When ${\Phi }_k\in B$ when

$$P({\dot{\Phi }}_k|{\dot{\Phi }}_{k-1})=f({\dot{\Phi }}_k,{\dot{\Phi }}_{k-1})\text{\hspace{1em}(4-6)}$$

among $f({\dot{\Phi }}_k,{\dot{\Phi }}_{k-1})$ It can be a variety of functions , The better effect is

$$f({\dot{\Phi }}_k,{\dot{\Phi }}_{k-1})=exp(-\lambda \times |\frac{{\dot{\Phi }}_k}{{\dot{\Phi }}_{k-1}}-1|)\text{\hspace{1em}(4-7)}$$

It's not hard to see. , When ${\dot{\Phi }}_{k-1}={\dot{\Phi }}_k$ When the probability is maximum .$\lambda \in [1,300]$ It's a super parameter , Different $\lambda$ Next $f({\dot{\Phi }}_k,{\dot{\Phi }}_{k-1})$ The results are as follows 4-3 Shown .

chart 4-3 Under different parameters f Results of

When ${\Phi }_k\notin B$ when
$$P({\dot{\Phi }}_k|{\dot{\Phi }}_{k-1})=\{\begin{array}{ll}1,& {\dot{\Phi }}_k={\dot{\Phi }}_{k-1}\\ 0,& otherwise\end{array}\text{\hspace{1em}(4-8)}$$

It means that the speed does not change .

5 forecast

madmom The whole model is clearly divided into two parts ,DBNDownBeatTrackingProcessor and RNNDownBeatProcessor.RNNDownBeatProcessor It's ours RNN The Internet ,DBNDownBeatTrackingProcessor yes HMM Part of . On the macro level ,beats and downbeats The position of is by RNN Roughly determine , Then from HMM Determining the final position according to the condition of periodicity .RNN It can be understood as a partial understanding ,HMM It is a decision for the overall situation .

At the time of prediction , We will tell the model this song every bar Of beat There are several , If you're not sure , Just put beats_per_bar=[2,3,4,6] Fill in all , Let each model run to one side , Then take the one with the greatest probability .

At every beats_per_bar Next , We calculate an optimal hidden variable path

$${\mathbf{x}}_{1:K}^{\ast }=arg\underset{x_{1:K}}{\max }({\mathbf{x}}_{1:K}|{\mathbf{y}}_{1:K})\text{\hspace{1em}(5-1)}$$

Solve this with viterbi Algorithm will do .

The end result is

$$B^{\ast }=\{k:{\mathbf{x}}_k^{\ast }\in B\}\text{\hspace{1em}(5-2)}$$

$$D^{\ast }=\{k:{\mathbf{x}}_k^{\ast }\in D\}\text{\hspace{1em}(5-2)}$$

To determine the $B^{\ast }$ and $D^{\ast }$ after , And according to RNN Result , Correct the point to the nearby probability peak point .

When validating the model , Keep the error within 70ms All points within are considered correct .madmom The effect is still very good .

Reference material

[1] madmom implementation
[2] Joint Beat and Downbeat Tracking with Recurrent Neural Networks
[3] An Efficient State-Space Model for Joint Tempo and Meter Tracking
[4] Baidu Encyclopedia - Beats
[5] Bayesian Modelling of Temporal Structure in Musical Audio
[6] A Multi-model Approach to Beat Tracking Considering Heterogeneous Music Styles