Overview of head pose estimation

Head posture estimation （Head Pose Estimation ）： Get the posture angle of the head through a facial image . stay 3D In the space , Express The rotation of an object Can be Three Euler angles (Euler Angle) To express ： Separate calculation pitch( around X Shaft rotation ),yaw( around Y Shaft rotation ) and roll( around Z Shaft rotation ) , Separate scientific names Pitch angle 、 Yaw angle and roll angle , Generally speaking, it is Look up 、 Shake your head and turn your head . Seeing is believing , Upper schematic diagram ：

3D 2D Head estimation of mapping （ Conventional ）

If you are familiar with camera calibration , It's easier to understand , because Head Pose Estimation The more difficult part has been solved by the Bulls , A classic Head Pose Estimation The steps of the algorithm are generally ：

（1）2D Face key point detection ;（2）3D Face model matching ;（3） solve 3D Point and corresponding 2D Transformation relationship of points ;（4） The Euler angle is solved according to the rotation matrix .

As we all know, the attitude of an object relative to the camera can be expressed by rotation matrix and translation matrix ：

（1） Translation matrix ： The spatial position relation matrix of the object relative to the camera , use T Express ;

（2） Rotation matrix ： The spatial attitude relation matrix of the object relative to the camera , use R Express .

It seems that there must be coordinate system transformation , Namely ： World coordinate system (UVW)、 Camera coordinate system (XYZ)、 Image center coordinate system (uv) And pixel coordinate system (xy), Here's the picture ：

See blog ： Head pose estimation principle and visualization , It is more effective to face or face measurement ,yaw horn 150 The accuracy above degree is not enough , And affected by key points

Research progress of head pose estimation

Head pose estimation (HPE) The traditional head pose calculation method is to estimate some key points of the face , Use the average head model to solve the corresponding problem from two dimensions to three dimensions , We think this is a fragile approach , Because it completely depends on the performance of face key points , Headform and 3D To 2D The projection of . We propose a robust attitude determination method , stay 300W-LP( A large integrated extended data set ) Train a convolutional neural network with multiple losses through joint attitude classification and regression , Predict the inherent Euler angle directly from the image ( Yaw 、 Pitch and roll ).

Fine-Grained Head Pose Estimation Without Keypoints
We prove , Compared with the key based method , Use convolution neural network to estimate directly from the image 3D Head posture shows higher accuracy . Due to deep learning , The key point detector has been significantly improved recently . But the recovery of head posture is essentially a two-step process , There are many opportunities for mistakes . There are two main problems ：

• There may be not enough key points detected
• secondly , The accuracy of attitude estimation depends on 3D The mass of the headform .
  General head model , It may be biased by any individual , Adjusting the head model to suit everyone needs a lot of data , And it will produce a huge amount of computation .

in application , need High precision head pose estimation , A common solution is to use RGBD( depth ) The camera . These can be very accurate , But there are also many constraints :

• Vulnerable to external light , And unconstrained posture changes .
• Depth camera draw ratio RGB More power , It is easy to have battery life problems
• RGBD The camera has a larger amount of data , Increased data storage and conversion time .
  
  WHENet: Real-time Fine-Grained Estimation for Wide Range Head Pose

Although many postures are expected to perform well , However, the network structure is relatively large, which is not suitable for running on mobile and embedded platforms .

Model

Calculate by multiple losses .


By using softmax Predict every bin Probability （ Classified loss ）, To calculate yaw,pitch,roll The expectations of the , Expected and true angle Calculation MSE Loss .

In order to make the network lighter , Used EfficientNet-B0, It's not Alexnet,ResNet50 etc. ,EfficientNet-B0 yes EfficientNet The basic model of the family , Use the residual module (Inverted Residual Blocks) originate MobileNet V2, Thus, the parameters of the network are reduced . stay CPU Up to 60FPS （ Or is it 8155 platform ）

Data sets

You can prune the model , Is that the parameters are further reduced .