Smart fitness gesture recognition: PP - TinyPose build AI virtual trainer!

In the artificial intelligence technology change and under the background of the outbreak era,With the change of living patterns of residents,智能运动健身行业得到了极大的发展.brought by Liu Genghong“本草纲目”Fitness boom also officially opened the home fitness era,Then comes the need for personalization、Intelligent fitness action recognition、矫正、Strong need for counting,How to make it convenient anytime, anywhere、The standard fitness gradually become a hot topic.

in fitness,There is an inseparable relationship between the action type and the positional changes of the human skeleton points,Therefore, it is more appropriate to adopt human key point detection technology to realize intelligent fitness action recognition and counting..

                          图1：基于PP-TinyPosefitness action recognition

So how can we use human body key point detection to realize intelligent fitness action recognition and counting?？Today is handy with you build a fly OARSAIvirtual fitness trainer！

不慌,Code the complete tutorial first：

https://github.com/PaddlePaddle/PaddleDetection

* 记得Star收藏 *

防止走丢又实时关注更新

项目难点

Unconventional posture

在健身、in dance scenes,The human body posture is obviously different from the common standing posture,There are more complex gestures.This type of data is often missing from common public datasets,Therefore, when dealing with such complex gestures,It's easy to have a test box position or point position is not accurate,False detection and missed detection in action recognition,Unable to achieve satisfactory results.

Model volume and speed requirements are high

The duration of physical fitness movements is usually within seconds,Therefore, it requires extremely high detection response speed.,At the same time as the edge of the popularity of the hardware in the fitness industry,The volume requirements of the model are becoming more and more stringent.

Video jitter severe

When processing video data using the keypoint algorithm,Since the prediction is done for a single frame of image,There is often jitter in the video results.For application scenarios that rely on refined coordinates（Such as fitness counting、Keypoint-based virtual rendering, etc.）,It is very easy to cause false detection or unsmooth visualization..

The key points needs changing

For example, in ballet scenarios,垫脚、Actions such as hooking the foot only rely on the key points of the ankle and cannot be distinguished,Often more skeletal point information is needed to achieve refined action recognition.

方案设计

图2：方案设计

模型选型

PP-TinyPoseThe birth of the machine completely broke the problem that precision and speed could not be achieved at the same time.,Be the best choice for lightweight keypoint detection.PP-TinyPose采用Top-Down的方式,先应用3.3M、150FPS的超轻量检测网络PP-PicoDet检测出人体,再用基于Lite-HRNetThe mobile-optimized model ensures that keypoint detection is高精度,同时扩大数据集,减小输入尺寸,预处理与后处理加入AID、UDP和DARK等策略,保证模型的高性能.实现速度在FP16下达到122FPS的情况下,精度也可达到COCO数据集51.8%AP,不仅比其他类似实现速度更快,精度更是提升了130%！

数据准备

Data collection includes two parts: human detection and human key point detection,均使用COCO train2017和AI Challenger trainset作为训练集.Unify the meanings of the labels of the two datasets,并转换为COCOPerform training after data format.

图3：数据准备

模型训练优化

更令人惊喜的是,在原版的基础上,Difficulties in sports and fitness scenarios,PP-TinyPose分别从Scene data diversity、Detection Model Iteration、key point stabilityComprehensive upgrade in three directions,Released an enhanced version for sports action recognition,Finally compared with the original on the fitness scene data提升9.1%AP,实现速度85FPS,精度77.1%AP.同时,新增7Accurate recognition of complex movements,Including sideways、lie down、跳跃、Unusual action, high leg lifts.具体来看：

Scene data diversity

在COCO数据集和AI Challenger数据集的基础上,Going further with the acquisition of fitness、Dance more difficult scene data such as for iteration model,提升模型预测效果.

图4：PP-TinyPoseEnhanced version effect comparison chart

Detection Model Iteration

使用PP-PicoDetEnhanced version of the model,On the basis of the original introductionTAL及ETA Head,优化PAN等结构,COCODataset Accuracy Improvement2个点以上.Based on rich fitness scene data,人体检测模型精度提升6.8% mAP****.

key point stability

加入OneEuro滤波器和EMATwo key point stabilization methods,Combine the current key point coordinate results with the historical key point coordinate results to calculate,Output the coordinates of the more stable smooth.

图5：Stability upgrade comparison chart

If the model still does not meet the requirements,Developers can also develop kits based on flying paddle target detectionPaddleDetectionProvided secondary development documents,Model training and optimization、key points added、Custom cut and category,5Personalized development of key point detection can be realized in minutes.

图6：PP-TinyPoseEnhanced version supports fingers、脚趾等29个关键点

• 完整链接：

https://github.com/PaddlePaddle/PaddleDetection/blob/develop/docs/advanced_tutorials/customization/keypoint_detection.md

后处理逻辑

After obtaining the detection results of human key points,Can combine the relative relationship of different key points in the same frame,Or the position change of the same key point in different frames for action recognition and counting.

Take the lateral raise as an example,When the arm is raised at an angle close to horizontal,the arm is considered to be in a raised motion;When the arm is lowered at an angle close to vertical,Argues that the arm is to lay down their actions;After the two movements are completed alternately, the count of lateral raises is considered to be increased by one..

图7：Actions count logic

除此之外,PP-TinyPoseThe enhanced version is also equipped with action recognition counts for squatting and standing lifts.

• 详细代码链接：

https://github.com/zhiboniu/pose_demo_android/blob/release/1.0/app/src/main/cpp/pose_action.cc

端侧部署

图8：Effect of end side deployment

Through the above methods, an efficient fitness action recognition counting model can be obtained,Relying on the propeller end-to-side reasoning enginePaddle Lite,Can be flexibly and quickly deployed to various end-side devices.

blog.csdnimg.cn/img_convert/7479679e3abfb3507e9b0dddb05c8b65.png)

If you want to update model and realize the recognition of the more abundant the fitness movement,可参考以下流程：

1. The trained model is exported as.nb后缀的Paddle Lite模型;

2. 替换pose_demo_android中models下的nb模型文件;

3. 通过Android Studio编译运行代码,重新生成APP.

• Detailed steps reference link：
  https://github.com/zhiboniu/pose_demo_android

Open source as lightweight skeleton point detectionSOTA算法, PP-TinyPose以High precision and speed、Small target recognition effect is good、Multi-person detection is fast、Device-side deployment is easy to use、Physical fitness action detection著称,On the side of the weak computing power,使用飞桨轻量化推理引擎Paddle Lite,Can rapid implementation of fitness action identification and counting！

还在等什么,快来搭建属于你的AIfitness trainer！

• Tutorial document and model download link：

https://github.com/PaddlePaddle/PaddleDetection/blob/develop/industrial_tutorial/README.md

最后,欢迎大家扫码进群,与PP-TinyPoseThe core r&d engineers discussion together,同时GetThe latest product news of flying paddle！

关注【飞桨PaddlePaddle】公众号

获取更多技术内容~