Openpose2d transform 3D pose recognition

Recently, the author encountered a problem is openpose When it comes to attitude recognition , The recognized human posture is a 2d Plane posture . about AR For interactive applications ,2d Posture is not as good as 3d Posture . Of course you can kinect Depth camera or open source project vnect Realization 3d Real time attitude recognition . But if we can openpose Make some changes , Can achieve an approximate effect .

First openpose Of body_25 The model output is shown in the figure :

In order to make an approximate effect , We can first assume that
1. The torso is facing the camera and will not deviate （ Spinning ）【 Joint point 5 2 1 8】.
2. The inclination of the limbs is only left Right deviation and Lean forward , No leaning back .
Then first of all, our goal is to do a stretching posture as shown in the figure ( Limb in 2d In the plane ), Measure the length of each limb .

Next , We use 1 2 and 2 3 Take the joint point as an example , Make one 3d Attitude recognition .
First we assume 5 2 1 8 Always keep parallel with the camera , So we can take this as a reference .

about 2d In terms of plane We know this angle , It can be obtained by calculating the included angle between vectors .
So when 3 When leaning forward , Got 2d The output is actually 2 3＇. We know 2 3＇ Length and 2 3 The length of .
2 3＇ = 2 3 * cos Θ. So you can calculate Caster angle θ The angle of . Got it The angle in the two pictures , You can get joint points 3 The spatial location of

about OpenPose Of unity In terms of plug-ins , Is to calibrate the obtained attitude information on the screen , And then use LineRenderer Drawing . On this basis, the author adds a limb model


Script in lsfather lsfather Empty object lshoulder For the actual model , This is convenient for the adjustment of angle and model .

using System.Collections;
using System.Collections.Generic;
using UnityEngine;

public class To3DBlog : MonoBehaviour
{
    
    [SerializeField] RectTransform Joint0;
    [SerializeField] RectTransform Joint1;
    Vector3 direction;
    bool enabledstate;

    GameObject Renderobj;
    [SerializeField]  float MaxSize = 0;
    [SerializeField]  float curLength;
    float curHeight;
    [SerializeField] GameObject childobj;

    // Start is called before the first frame update
    void Start()
    {
    
        Renderobj = this.gameObject;
        childobj = this.gameObject.transform.GetChild(0).gameObject;
    }

    // Update is called once per frame
    void Update()
    {
    
        if (Joint0 && Joint1)
        {
    
            bool enabled = Joint0.gameObject.activeInHierarchy && Joint1.gameObject.activeInHierarchy;
            enabledstate = enabled;
            childobj.SetActive(enabled);
            if (enabled)
            {
    
                // Conduct 
                direction = (Joint0.transform.localPosition - Joint1.transform.localPosition);
                curLength = direction.magnitude;

                // Modules of vectors 
                if (curLength > MaxSize)
                {
    
                    MaxSize = direction.magnitude;
                }
                // seek Joint1  Of  z  value 

                //curHeight = MaxSize * Mathf.Sin(Mathf.Acos(curLength/MaxSize));
                curHeight = Mathf.Sqrt(Mathf.Pow(MaxSize,2)- Mathf.Pow(curLength,2));
                // The direction ahead is  -z  Axis direction   however   Consider mirroring  
                direction = (Joint1.transform.localPosition + new Vector3(0, 0, curHeight) - Joint0.transform.localPosition);
                Renderobj.transform.localPosition = (Joint1.transform.localPosition + Joint0.transform.localPosition + new Vector3(0,0,-curHeight)) / 2;
                Renderobj.transform.up = direction;
                
                Renderobj.transform.localScale = new Vector3(MaxSize, MaxSize, MaxSize);
                //scale The problem of 
            }

        }
    }
}

The algorithm is explained in the figure : among curLength yes openpose Identify the distance between two joint points （ This body is also in z The projection of the axis ）,MaxSize Is the actual length of the limb , In this way, we can find the current Joint1 Height .

For the right shoulder, apply the results

For the right arm, the results apply Left 2d Right 3d The effect is as shown in the picture