Principle of line of sight tracking and explanation of the paper

L2CS-Net: Fine-Grained Gaze Estimation in Unconstrained Environments


at present , Line of sight tracking technology has the following application platforms ：

• The computer ： Mainly used for human-computer interaction —— Computer communication and text input （ More efficient than a mouse , And it is more suitable for the disabled ）
• TV ： Select and navigate menus and switch channels
• Head gear ： Apply to user attention 、 Cognitive research 、 psychoanalysis ; Or is it VR Partial rendering of , If - - It can estimate the direction of people's line of sight through the built-in camera in the helmet , You can render the scene locally , That is, only the scene within the scope of human line of sight is finely rendered , Thus, the hardware cost is greatly reduced .
• Car equipment ： Check whether the driver is tired and focused .
• Handheld devices ： brightness 、 Volume adjustment and other human-computer interaction functions .
  
  Reference blog ：

Eye tracking is used in various applications, such as human-computer interaction and virtual reality . lately , Convolutional neural networks (CNN) The method has made significant progress in predicting the direction of line of sight . However , Outdoor line of sight tracking is still a challenging problem , Due to the unique eye appearance , Light conditions , And the diversity of head posture and gaze direction .

In this project , We propose a method based on cnn To predict the direction of gaze
We suggest returning to each gaze angle separately , To improve the prediction accuracy of each angle , This will improve the overall gazing ability . Besides , We use two identical losses , One for each angle , To improve and increase the generalization of e-learning . We evaluated our model using two popular data sets , These data sets are collected with unconstrained settings . Our proposed model realizes advanced 3.92◦ Accuracy and 10.41◦ Yes MPIIGaze and Gaze360 Data sets .

Originally, it was a multi task way , Insufficient accuracy , Various losses are combined , It is difficult to make all parties' training satisfactory ., Improved use of multiple loss estimates 3D Line of sight tracking , Use two parallel full connection layers to predict yaw horn and pitch horn , And independent losses are used for both angles . Each loss includes bin Classification and regression , Use softmax And cross entropy estimation gaze angle （L2+ Cross entropy ).

There are mainly two ways to realize line of sight tracking ：1. The conventional and CNN based Method :

• routine ： Use regression , Build a specific mapping relationship with line of sight estimation , such as adaptive linear regression and gaussian process regression For the sight effect with little change , But the range of sight change is relatively large , The effect is relatively poor
• CNN: CNN Build a nonlinear mapping relationship between line of sight and image

Loss function

Most of them use L2 Loss of estimated line of sight direction yaw and pitch horn , We're dealing with two gaze Two independent loss functions are proposed from the angle of , Each loss function includes Cross entropy loss and Mean square error loss , According to the estimate softmax classification bin Probability , To calculate gaze bin The expectation of , Using this method, fine-grained optimization . Then use with the real ground truth The mean square error of improves the prediction accuracy of the output .

Network architecture

According to the proposed classification and regression loss , We built a simple network (L2CS-Net), Will recognize the face image feed To resnet50 backbone in , Preliminary extraction of network features . Compared with the previous return in a network gaze Of yaw and pich angle , We propose that each corner uses a fully connected network independently . These two full connection layers share one backbone Extracted features . At the same time, we define the loss function for each branch of the full connection layer .

Data sets

Conclusion

Source code ：https://github.com/ahmednull/l2cs-net