[happy Spring Festival] if you feel happy, dance

Project introduction

This project builds a simple and practical dance machine based on the pose estimation model .

Project use PyQT5 Designed a simple interactive interface , The original dance video is displayed on the left , The camera is displayed on the right , Here I use a built-in camera in my notebook , Friends can change other cameras or projectors , May experience better .

Be careful : All materials are from the Internet , If there is infringement , Please contact to delete

B Station video experience is as follows ：

b Station video link ：https://www.bilibili.com/video/BV1am4y1f7c2/

Interface design

The interface layout is mainly divided into 4 part , The upper middle shows the number of correct posture matching :

• On the top left is the drum beat, which always refers to the posture of the dance
• Below the left is the reference dance video source
• On the top right is the posture of the real-time video when the drum is beating
• On the lower right is the real-time video source of the camera

Menu management

Right click in the reference dance video source area to display the menu , It mainly includes

• Camera settings
• Song management
• Start the game ( Before the game begins )
• Stop swimming at the beginning ( When the game is running )
• Full screen display
• sign out

Click song management to add or delete dance videos , You can also switch the currently selected song in the song list .

Preload dance videos

After adding a dance video , The background of the program will automatically analyze the drum time of the dance video and the corresponding posture description .

Use librosa Ku analyzes the drums of dance songs , Finally, it is converted into the frame number and sequence number corresponding to the video .

        y, sr = librosa.load(video_path)
        _, beats = librosa.beat.beat_track(y=y, sr=sr)
        beat_times = list(librosa.frames_to_time(beats, sr=sr))
        beat_times.append(beat_times[-1] + 1)
        frameIndex = [round(bt * 25) for bt in beat_times]

After preloading the dance video, it will generate PKL file , Stored in projects/songs in , The dictionary format is as follows :

 	   data = {
            "path": video_path,
            "fps": capture.get(cv2.CAP_PROP_FPS),
            "width": int(capture.get(cv2.CAP_PROP_FRAME_WIDTH)),
            "height": int(capture.get(cv2.CAP_PROP_FRAME_HEIGHT)),
            "beats": frameIndex,
            "poses": [],
            "pose_imgs": [],
            "start_ends": start_ends,
            "count": 0
        }

Attitude estimation

Due to the high real-time requirements of human-computer interaction for model reasoning , After investigating many models , The final model selection is PaddleDetection Open source PicoDet-S-Pedestrian as well as PP-TinyPose, Model reasoning time single frame 20ms about , The speed and effect can meet the requirements .

PP-TinyPose yes PaddleDetecion Real time attitude detection model optimized for mobile devices , It can smoothly perform multi person pose estimation tasks on mobile devices . With the help of PaddleDetecion Self developed excellent lightweight detection model PicoDet, We also provide a characteristic lightweight pedestrian detection model .

PP-TinyPose link : https://github.com/PaddlePaddle/PaddleDetection/tree/release/2.3/configs/keypoint/tiny_pose

Attitude matching algorithm

After the estimation of human posture , Compare the pose of the real-time camera with the reference dance video . In practice, it is difficult to synchronize the action time of the two , Here is a simple treatment , It is acceptable not to exceed 10 In the frame range .

The actual comparison is the relevant bone connecting lines on the left and right sides of the human body , In particular, the included angle of the connecting line is compared , stay 15 Within the range of degrees, it is considered to be similar .

The current matching algorithm is very rough , Many further optimizations are needed , Especially in similarity judgment , Key points are adopted in the later stage Graph The way may work better .

The calculation logic of angle matching is as follows :

   def angle_match(self, pose_ref, pose_cur, points_index=()):
        if len(points_index) != 3:
            return False
            
        for i in range(3):
            idx = points_index[i]
            if pose_ref[idx][2] < 0.3:
                print("pose ref true")
                return True

        def get_angle(pose):
            point_1 = pose[points_index[0]]
            point_2 = pose[points_index[1]]
            point_3 = pose[points_index[2]]
            a = math.sqrt((point_2[0] - point_3[0]) * (point_2[0] - point_3[0]) + (point_2[1] - point_3[1]) * (
                    point_2[1] - point_3[1]))
            b = math.sqrt((point_1[0] - point_3[0]) * (point_1[0] - point_3[0]) + (point_1[1] - point_3[1]) * (
                    point_1[1] - point_3[1]))
            c = math.sqrt((point_1[0] - point_2[0]) * (point_1[0] - point_2[0]) + (point_1[1] - point_2[1]) * (
                    point_1[1] - point_2[1]))
            return math.degrees(math.acos((b * b - a * a - c * c) / (-2 * a * c)))

        return abs(get_angle(pose_ref) - get_angle(pose_cur)) < 15

    def pose_match(self, pose_ref, pose_cur):
        canditate_list = [
            [7, 5, 11], # elbow, shoulder, hip
            [8, 6, 12],
            [9, 7, 5], # wirst, elbow, shoulder
            [10, 8, 6],
            [5, 11, 13], # shoulder, hip, knee
            [6, 12, 14],
            [11, 13, 15], # hip, knee, angle
            [12, 14, 16]
        ]

        err_list = []
        for p_idx in canditate_list:
            if not self.angle_match(pose_ref, pose_cur, p_idx):
                err_list.append(p_idx)
        return err_list

At the end

This project requires local operation , Basically operational , The stability and matching effect of the program need to be further optimized .

More function upgrades will be made later , For example, try 3D Posture , Multiplayer mode ,Unity wait .

About author

• Head of Chengdu oar pilot group
• PPDE
• AICA Third stage students

I am here AI Studio Get diamond grade on , Lighten up 10 A badge , Let's talk to each other ~
https://aistudio.baidu.com/aistudio/personalcenter/thirdview/89442