2D human pose estimation for pose estimation - simdr: is 2D Heatmap representation even necessity for human pose estimation?

from

[ Paper reading ： Attitude recognition ] 2107 SimDR: Is 2D Heatmap Representation Even Necessary for Human Pose Estimation?_cheerful090 The blog of -CSDN Blog

a farewell Heatmap, A new representation method for human posture estimation SimDR - You know

From Tsinghua University 、 Open vision 、 Southeast . This one is the same as the last one TokenPose From the same research team .

paper https://arxiv.org/pdf/2107.03332.pdf

github https://github.com/leeyegy/SimDR

SimDR Mainly studied Key coordinates represent . The coordinates of key points are represented by decoupling coordinates , Position the joint x、y The coordinates are decomposed into two independent one-dimensional vectors , The key point positioning task is regarded as a classification subtask in the horizontal and vertical directions . And common based on Thermogram Or the coordinate representation based on regression is different . Put forward Ordinary SimDR and Spatially aware SimDR Two versions .

• Top-down Pattern No complex post-processing , For example, there is no need for deconvolution , It is conducive to the lightweight realization of attitude recognition network .
• Good versatility . It can be with the general CNN perhaps Transformer Framework in combination with , Better than similar algorithms based on heat map ; At low resolution , The effect is very good .

Abstract ：

Two dimensional heat map representation due to its high performance , It has dominated the estimation of human posture for many years . However , be based on Thermogram There are also some ways Insufficient :1) For those often encountered in real scenes Low resolution image , Sharp performance degradation .2) in order to Improve positioning accuracy , Multiple sample layers may be required to recover from low to high feature map The resolution of the , The cost of calculation is high .3) Usually Additional coordinate refinement , To reduce the quantization error of the reduced scale heat map . To solve these problems , We propose a simple and promising Key point coordinates decoupling Express (a Simple yet promising DisentangledRepresentation for keypoint coordinate,SimDR), Human key point localization is redefined as a classification task . To be specific , We suggest Decompose the horizontal and vertical coordinate representations of key positions , So as to obtain a more effective scheme , Without additional upsampling and refinement . stay COCO Comprehensive experiments on data sets show that , What this article puts forward Heatmap method On the input resolution of all tests Better than the method based on heat map , Especially at lower resolution , Its performance is obviously better than the method based on heat map .

1.Introduction

Two dimensional human pose estimation (Human Pose Estimation,HPE) It aims to locate human joints from a single image . At present, the commonly used method is to use Encoder - decoder Pipeline to estimate the position of key points . because Deep convolution neural network (CNN) It has good performance , Most methods use it as Feature coder . In the decoder part , The existing methods are mainly divided into two categories : be based on heatmap Based on Return to Methods . In most cases, the former .

Directly regress the numerical coordinates of the joint Is the most direct way to locate key points . Numerical regression tends to be simple and computationally friendly . However , It lacks spatial generalization , Resulting in most difficult situations , The prediction results are poor .

The other way is Encode the joint coordinates into 2D Thermogram . Due to its remarkable performance without constraints , be based on heatmap The method of has naturally become HPE In fact, the standard label indicates . Heat map by Ground-truth A two-dimensional Gaussian distribution centered on joint points is generated . The coordinate representation based on heat map suppresses false positive examples , And by assigning a probability to each location ( uncertainty ) value , So as to make the training process more smooth , Thus, it achieves significant improvement over the regression based method .

However , be based on Thermogram There are several ways to do it shortcoming . First , To export 2D Thermogram , It always needs to be done Expensive upsampling operation ( Such as SimpleBaselines Deconvolution in ). secondly , In order to reduce the heat map to GT Projection error , It is inevitable that Additional post-processing To further refine the results . Last , Performance of heat map based method It usually drops sharply with the decrease of input resolution . for example , We observed that , When the input resolution is reduced , Such as input from 256x192 Reduced to 64x64 when ,HRNet-W48 Performance from 75.1 AP Down to 48.5 AP. At low input resolution , The advantages of the method based on heat map are often masked by its quantization error , Resulting in poor performance .

therefore , We came up with one problem : Joint coding of horizontal and vertical key coordinates 2D Is heat map representation necessary to maintain excellent performance ?（is the 2D heatmap representation that jointly encodes horizontal and vertical keypoint coordinates necessary for sustaining superior performance?）

We have been recently based on transformer Inspired by human posture estimation method [TokenPose 17, Transpose 38], These methods also use 2D heatmap As the output . However , With the typical full convolution network (Fully Convolutional Network, FCN) The difference in architecture is , They do not always maintain the characteristic diagram throughout the pipeline 2D structure . especially ,TokenPose[17] Use a shared MLP To predict the 2D Thermogram , Each type of key point heat map is from the key point tokens (1D vector ) Transformed from . Besides ,[7] A compressed volume heat map is also proposed , It can encode multiple key points in compressed code , And recover their position through the decoder . These results suggest that , Heat map representation with explicit spatial structure may not be a necessary condition for encoding location information .

** surface 1:heatmap And SimDR Comparison .**H and W Represents the height and width of the input image respectively .λ Is the down sampling ratio , Usually set to 4.K(≥1) Is a fission factor (splitting factor).

In order to further study the effectiveness of key point representation , We propose a method for ** The estimation of human posture is simple decoupling Coordinate representation (SimDR)** Method .SimDR Put the key point (x, y) The coordinates are encoded into two independent 1D vector , And the quantization level is the same as or higher than the input image . The comparison between different coordinate representation schemes is shown in Figure 1 Shown .

chart 1： Comparison of different coordinate representation schemes .H,W The height and width of the original input image .λ(∈{1,2,4,…}) Is the down sampling ratio of two-dimensional heat map , stay SimpleBaseline、Hourglass or HRNet And other common methods , Usually set to 4.k(≥1) by SimDR The splitting factor of .

We will SimDR Apply to typical based on CNN Or based on Transformer Human posture estimation model , Under various input resolution conditions , Especially under low resolution input , Better results are obtained than two-dimensional heat map representation . We hope that this simple baseline can inspire us to rethink the coordinate representation design of two-dimensional human pose estimation . Our contributions are summarized as follows :

Contributions

• A new representation method of key point position is proposed , This method will be the key point of x、y The coordinate representation is decomposed into two independent one-dimensional vectors . It regards the key point positioning task as two subtasks classified in horizontal and vertical directions . Compared with the method based on heat map , The advantages of our method are shown in the table 1 Shown .

• Proposed SimDR It allows people to remove time-consuming upsampling modules in some methods . application SimDR Deconvolution module and remove ,SimBa-Res50[37] Of GFLOPs Greatly reduced 55% above , And higher model performance ( See table 4).

• stay COCO Key detection data set [18]、CrowdPose[14] and MPII[1] Comprehensive experiments were carried out on three data sets . Proposed SimDR For the first time, the non heat map method has reached the competitive performance level with the heat map method , It is much better than the latter in the case of low input resolution .

2. Related work

Regression based 2D Attitude recognition The method based on regression is explored in the early stage of two-dimensional human pose estimation . Different from relying on two-dimensional grid heat map , This work line Directly regress key point coordinates in a computationally friendly framework . However , A regression based approach Lack of spatial generalization ability [23]. therefore , There is a difference between the regression based method and the heat map based method There is a huge ( precision ) disparity , This limits its practical application .

Based on the heat map 2D Attitude recognition The other working line adopts Two dimensional Gaussian distribution ( namely heatmap) Represents the joint coordinates . Each position on the heat map has a probability assigned to Ground Truth. As heatmap One of the earliest applications ,thompson et al.[33] A hybrid architecture composed of deep convolution network and Markov random field is proposed .Stacked Hourglass Introduce the hourglass structure into HPE.Papandreou et al.[26] It is proposed to aggregate the heat map and offset prediction , In order to improve the positioning accuracy .SimpleBaselines A simple baseline is proposed , utilize ResNet Backbone Feature extraction , Then, three deconvolution layers are used for up sampling to obtain the final predicted heat map .HRNet A novel network , Maintain a high-resolution representation throughout the process , Significant improvements have been made . Besides ,DARK The coordinate representation of distribution perception is introduced to deal with the quantization error of reduced scale heat map . Due to the intervention of spatial uncertainty , This learning mode has tolerance for jitter errors . When the post-processing is optimized for coordinate migration , False positive cases are reduced . therefore , The heat map based method has maintained stable and state-of-the-art performance for many years . However , Quantization error is still an important problem in the method based on heat map , Especially in the case of low input resolution . Besides , In the actual deployment scenario , Additional post-processing It's complex and expensive . by comparison , What this article puts forward SimDR These problems have been solved very well , It has made remarkable improvement in various input resolutions .

3. Framework Overview

In this section , We first review the coordinate representation based on heat map . then , We illustrate the proposed ** Simple disentangled coordinate representation (SimDR)** Coordinates of key points of people . This paper mainly studies the top-down algorithm of multi person attitude estimation (Top-down) normal form .

chart 2. take SimDR Schematic diagram combined with a given neural network . When used like SimpleBaseline or HRNet When such a neural network is used as an encoder , Key point embedding (Embedding),, By way of feature map Rearrange the shapes of into , among n Is the number of key types . then ,SimDR head As a shared linear projection (linear projection), Embed each (Embedding) Convert to two with a length of W·k and H·k One dimensional vector of ( ​).

3.1. Coordinate representation based on heat map

As a matter of fact in human posture estimation Standard coordinates represent ,heatmap The spatial confidence distribution is used to represent the coordinates of key points . The resulting heat map follows Two dimensional Gaussian distribution Design :

among m For position Heat map pixels ,µ Is the target joint position .Σ Is a predefined diagonal covariance matrix . Please note that , Each output heat map represents the spatial distribution of a specific key point . The final coordinates are obtained from the maximum index of the predicted heat map .

3.2. SimDR： Re locate the joint points from the perspective of classification

stay SimDR in ,x Coordinates and y The coordinates are decoupled into one-dimensional vectors , Not joint coding .

Coordinate coding Given an input image ( Size is HxWx3), We will be the first to p Class ground truth The joint point coordinates are expressed as . In order to enhance the positioning accuracy , We introduced a splitting factor The zoom factor k(>=1), And rewind Groung truth The coordinate is a new coordinate ：

The key point is x and y Coordinates are represented by two independent one-dimensional vectors , By a scaling factor k(>=1), The length of the obtained one-dimensional vector will also be greater than or equal to the edge length of the picture . For the first p A key point , The encoded coordinates will be expressed as ：

among round(.) Is the rounding function . The zoom factor k Can enhance sub-pixel (sub-pixel) Horizontal positioning accuracy . Besides , Monitoring information Is defined as

among , Denotes an indicator function （ by 0 Or for 1）, It's all one-dimensional vectors , Its dimensions are , among W,H Is the length and width of the input image ,k It's the zoom factor splitting factor. Zoom factor k Its function is to enhance the positioning accuracy to a level smaller than that of a single pixel

Coordinate decoding For a key point , Model output is 2 One dimensional vector , Final forecast coordinates The calculation is as follows ：

namely , The position of the maximum point on the one-dimensional vector is divided by the scaling factor to restore to the image scale .

Experienced lambda Gaussian heat map of sub sampling , The quantization error level is , The method level of this article is .
among Is the sampling rate of the heat map

Network architecture Pictured 2 Shown ,SimDR Indicates that the output of neural network architecture is required n A key point is embedded (n Is the number of key types ), And add a linear layer , Embed each key into two fixed length projections 1D Vector . therefore , This method can be used with any common CNN Or based on Transformer Combined with neural network , Learn powerful feature representations .

After knowing the principle ,SimDR The head structure is very intuitive ,n Key points correspond to n individual embedding, That is, network output n One dimensional vector , Then through linear projection （MLP etc. ） by n individual SimDR characterization .

Specifically speaking , about CNN-based Model , The output feature graph can be straightened to d One dimensional vector of dimension , And then through the linear projection d Dimension up to W*k Peace-keeping H*k dimension . And for Transformer-based Model , The output is already a one-dimensional vector , Just do the same projection .

Training and loss function because SimDR Locate key points in the task As A kind of Classification task , therefore The general classification loss function can be used to replace the mean square error in the representation of two-dimensional heat map (MSE) Loss . We use Cross entropy loss To train the model .( The smoothing of labels is adopted to help model training )

Training target And the objective function

It is natural to find , The method in this paper transforms the key point location problem into the classification problem , Therefore, the objective function can be used in comparison with L2(MSE) Loss Better classification loss, For simplicity, this paper uses cross entropy .

A little booty ： In fact, a lot of work has been done before turning the positioning problem into the classification problem , such as Generalized Focal Loss Proposed in the work Distribution Focal Loss, Is to use the vector distribution to represent bbox The position of the coordinate point , It has made outstanding achievements in light weight and precision , It has also spawned famous open source projects Nanodet.

3.3. Advanced space aware SimDR

As mentioned above SimDR There is a problem , That is, as a classification problem, the label is one-hot Of , Except for the correct point, the other wrong coordinates are equal , Will be equally punished , But in fact, the closer the position predicted by the model is to the correct coordinates , The lower the punishment, the more reasonable . therefore , This article further proposes an upgraded version SimDR, adopt 1D Gaussian distribution to generate a surveillance signal , Use KL Divergence as a loss function , Calculate the target vector and prediction vector KL Spread out for training ：

As shown above ,SimDR The treatment of false labels is equal , The spatial correlation of adjacent labels in key point positioning task is ignored . To solve this problem , We proposed SimDR Advanced variants of , be called SimDR∗, it Generate surveillance signals in a spatially aware manner : Gaussian function is also used

  among σ Is the standard deviation . We use KL The divergence (Kullback–Leibler divergence) Model training .

4. experimental result

In the following chapters , We will empirically study the proposed SimDR For the effectiveness of two-dimensional human pose estimation . We conducted experiments on three benchmark data sets :COCO [18], CrowdPose[14] and MPII[1]. stay CrowdPose and MPII The results on are in the appendix .

4.1. COCO Key point detection

Use COCO Data sets ,Object Keypoint Similarity(OKS) As an evaluation indicator .

 Baselines There are many based on cnn And recently based on Transformer Of HPE Method . To show the proposed SimDR The advantages of , We chose two most advanced methods from the former ( namely SimpleBaseline[37] and HRNet[29]), Choose one of the latter ( namely TokenPose[17]) As our baseline

Implementation details For the selected Baselines, We follow the original setup in their paper . say concretely , about SimpleBaseline[37], Set the basic learning rate to 1e−3, stay 90 and 120 They are reduced to 1e−4 and 1e−5. about HRNet[29], Set the basic learning rate to 1e−3, In the 170 and 200 individual epoch Reduced to 1e−4 and 1e−5. about SimpleBaseline[37] and HRNet[29], The total training process is 140 and 210 individual epoch Terminate within . Be careful ,TokenPose-S The training process follows [29].

In this paper , We use a two-stage top-down human posture estimation : First, detect human examples , Then estimate the key points . about COCO Verification set , We use [37] Provided AP Rate is 56.4% Human body detector . The experiment is in 8 individual NVIDIA Tesla V100 gpu on .

surface 2. stay COCO Compare on validation set heatmap And proposed SimDR, Use the same body detector .Extra post. = Additional post-processing , To refine the predicted key point coordinates . （ So what does this extra post-processing mean ？

4.1.1 2D heatmap vs. 1D SimDR

In this part , We are right. Use SimDR As a coordinate representation scheme And The heat map shows The superiority of the scheme is comprehensively studied . These comparisons are from complexity 、 performance and Speed From the angle of .

surface 3. stay COCO Verify the results with higher input resolution on the set . Different from the method based on heat map , be based on SimDR The method does not require additional post-processing to refine the prediction coordinates .SimDR∗ yes SimDR Advanced spatial awareness variants of .

surface 4. Delay comparison . It turns out that COCO Implement... On the validation set .“Deconv.' Represents the deconvolution module , Use SimDR Can be retained or deleted directly after .

After calculation, the amount of calculation has indeed decreased so much , but （ doubt ： Why? GFLOPs It can drop so much ？ Isn't it just the removal of 3 Layer deconvolution layer , Ahead ResNet50 Backbone It doesn't change ？

Coordinates represent complexity Given a size of H ×W× 3 Image , be based on heatmap The method for obtaining the size of ​ Two dimensional heat map of , among λ Is the lower sampling rate , It's a constant . therefore ,heatmap It means The scale complexity is O(H×W). contrary , be based on SimDR Methods Designed to generate two sizes H·k and W·k One dimensional vector of . in consideration of k It's a constant ,SimDR It means The complexity is O(H + W), Than heatmap Is much more efficient . especially ,SimDR Some methods are allowed to directly remove additional independent deconvolution modules , Thus, the model parameters and GFLOPs.

Versatility and multi-scale robustness We put forward through experimental research SimDR stay COCO Generality and robustness on verification set ( Various models and input resolutions ). We choose some of the best performance based on CNN And based on Transformer As our baseline . surface 2 The two-dimensional heat map and one-dimensional heat map are given SimDR Comparison , It shows that the proposed method Always provide significant performance gains , Especially in the case of low resolution input .

It should be noted that , And based on heatmap Different methods , be based on SimDR The method does not require additional post-processing ( Such as empirical second highest value removal strategy [Stacked hourglass 22]) To improve the accuracy of predicting joint position . This paper is based on the most advanced HRNet-W48[29] For example , The advantages of this method are illustrated . stay 64×64 Under the input size of ,SimDR Bi Ji Yu heatmap Having additional post-processing SimDR Improved respectively 11.2 AP and 22.8 AP.128×128 and 256×192 The input sizes of are improved to 3.1/8.7 AP and 0.8/2.8 AP.

Reasoning delay analysis We discussed our proposal SimDR Yes **SimpleBaseline[37]、TokenPose-S[17] and HRNet-W48[29]** The effect of reasoning delay . there “ Reasoning delay ” It refers to the average time consumption of model feedforward ( We calculated 300 individual batchsize=1 The sample of ). We use FPS To quantitatively explain the reasoning delay .CPU The results are achieved on the same machine (Intel Xeon Gold 6130 CPU @ 2.10GHz) Present on .

SimpleBaseline[37] use On the sampling The resolution obtained by the module is 1/4 Of 2D Thermogram , Contains three time-consuming deconvolution layers ( The deconvolution part takes up so much computation ). because Use SimDR As coordinate representation instead of heat map representation , The upper sampling module can be deleted . surface 4 Shows SimpleBaseline stay COCO Verify the results on the set . We can see , use SimDR Can be removed SimpleBaseline Expensive deconvolution in . In this way ,SimDR contribution 4.9 AP gain , The model parameters and GFLOPs (27.4%;57.1%), The input size is 64 x 64 when , Increased speed 32%. surface 4 Illustrates the SimDR The computational cost of different input resolutions can be reduced consistently .

because SimpleBaseline[37] Use encoder - Decoder architecture , We can use SimDR The linear projection head replaces its decoder part ( deconvolution ). But for the HRNet[29] and TokenPose[17], They have no additional independent modules as decoders . In order to apply SimDR, We Directly in the original HRNet Add an additional linear layer to , And replace... With a linear layer TokenPose Of MLP Head . These are minor changes to the original architecture , therefore Only to HRNet[29] Brings a small amount of computational overhead , Even reduced TokenPose[17] The cost of Computing ( See model parameters ). therefore ,SimDR Yes HRNet or TokenPose The reasoning delay has only a slight effect . for example , Use heatmap or SimDR Of HRNet-W48 The input size is 256×192 At the time of the FPS Almost the same (4.5/4.8).

4.1.2 And SOTA Compare

COCO Verify the results on the set . We are COCO A large number of experiments have been carried out on the verification set to compare the results based on heatmap And proposed SimDR, Such as surface 2 Shown . In various state-of-the-art models and input resolutions ,SimDR It shows the consistent performance advantages compared with the heat map based model , No additional post-processing is required to refine the predicted key point coordinates . You can 4.1.1 See more details and discussion in this section .

SimDR Where are the performance boundaries of ? To explore the proposed SimDR And its advanced variant, spatial perception SimDR* Performance boundaries of , We compared them in Higher input resolution Result , Such as surface 3 Shown . about SimpleBaseline-Res50 [37], SimDR Than the representation based on heat map 0.8 Fractional gain ( See Appendix for more results A). The input size is 384 x 288 Of HRNet-W48[29], simple SimDR Can cause performance degradation . It turns out that , In simplicity SimDR In the model , The equivalent treatment of false labels is suboptimal , This can result in very large input sizes , The model with large capacity has been fitted . As shown in the table 3 Shown , We can see that this question has been raised Space perception SimDR It's solved , The SimDR The spatial correlation of adjacent labels is considered . say concretely , The input size is 384 x 288 when ,SimDR Bi Ji Yu heatmap Of HRNet-W48[29] Improved 0.6 spot . From simplicity SimDR Space awareness SimDR The shift may indicate , By using a more carefully designed loss function or supervisory signal ,SimDR There is still room for further improvement in the performance boundary of .

COCO test-dev set The result on . We are surface 5 The results of our method and other state-of-the-art methods are reported in . According to heatmap The degree of dependence , We further divide the existing methods into be based on heatmap Methods 、 Mixed representation and nothing heatmap Methods . Use both heat map representation and coordinates ( Or offset, etc ) As a supervisory signal Methods [25,24,32,36,DERK 9,TFPose 20] Is considered to be “ It means mixing ” Method .

surface 5 The results show that , What this article puts forward SimDR For the first time, it is based on heatmap And none heatmap The gap between methods . say concretely , As a method of non heat map ,SimDR∗ Enter the dimension in the 384×288 Got it. 76.0 AP, Much more than PRTR[15] (↑3.9). Besides , Even with the Based on the corresponding version of the heat map ,SimDR∗ Still in 384×288 The input size of shows 0.5 Two improvements .

surface 5. COCO test-dev set Result .’‘Trans.’’ representative Transformer Abbreviation .“Hybrid representation” Indicates that two-dimensional heat map and absolute coordinates are used at the same time ( Or offset, etc ) As a method of monitoring signal .

4.2. Ablation Experiment

Analyze the cleavage factor k. review 3.2 section , Optimize SimDR There is only one Hyperparameters , Fission factor k. We point out that ,k Controls the SimDR The sub-pixel accuracy level of the joint position in the . among ,k The bigger it is ,SimDR The smaller the quantization error . However , When k increases , Model training becomes more difficult . therefore , There is a trade-off between quantization error and model performance .

We are based on SimpleBaseline[37] and HRNet[29] Test at various input resolutions k∈{1,2,3,4}. Such as chart 3 Shown , With k The increase of , The performance of the model increases first and then decreases . about HRNet-W32 [29], 128×128 and 256×192 The recommended setting for the input size is k = 2. about SimBa-Res50 [37], 128×128 and 256×192 The recommended settings for the input size are k = 3 and k = 2.

5. Discuss

SimDR Allow people to directly delete time-consuming upsampling modules in some methods , This may lead to HPE Lightweight architecture . Our large number of experiments on various neural networks also reflect that the attitude estimation model can be regarded as two parts : An encoder learns good embedding , A header converts the embedding into key point coordinate coding . This may encourage future research to explore more effective design of neural networks as encoders and more potential coordinate coding schemes .

The other direction is to SimDR Apply to bottom-up (bottom-up) Multiplayer pose estimation , Due to the existence of many people , When decoding joint candidate positions from two de entanglement vectors, recognition ambiguity is generated . Future work is likely to introduce x and y A new dimension outside the dimension to solve this problem .

6. Conclusion

In this paper , We explore a simple and promising coordinate representation ( namely SimDR). The method Position the joint x、y The coordinates are decomposed into two independent one-dimensional vectors , The key point positioning task is regarded as a classification subtask in the horizontal and vertical directions . Experimental results show that , Two dimensional structure may not be a key factor in coordinate representation , To maintain superior performance . Proposed SimDR It has advantages over the representation based on heat map in terms of model performance and simplicity of post-processing steps . meanwhile , also by HPE Lightweight model design provides a new idea . We proved that SimDR It can be easily associated with anything based on CNN Or based on Transformer General neural network integration . Comprehensive experiments show that , The proposed SimDR yes Universal Of , In all cases, it is superior to similar algorithms based on heat map , Especially in the case of low input resolution .

appendix

See the original text for details https://arxiv.org/pdf/2107.03332.pdf

A Results on Higher Input Resolution

B Results on CrowdPose

C Results on MPII Human Pose Estimation

D Analysis on the Quantisation Error

• D.1 SimDR-based Representation
• D.2 Heatmap-based Representation