当前位置：网站首页>Threedposetracker project resolution

The code of algorithm model prediction data in the project is mainly VNectBarracudaRunner.cs. function PredictPose() The updated node data in is JointPoint Three variables in （ That is, the algorithm data to be used when driving ）：

 public class JointPoint
{
    public Vector3 Pos3D = new Vector3();
    public float score3D;
    public bool Visibled;
}

JointPoint There are also several variables in the class that are used for data optimization in the prediction process （ Anti jitter, etc ） Of ：

public class JointPoint
{
    public Vector3 Now3D = new Vector3();
    public Vector3[] PrevPos3D = new Vector3[6];
    public Vector3 P = new Vector3();
    public Vector3 X = new Vector3();
    public Vector3 K = new Vector3();
}

JointPoint Other data in the class need to be calculated by yourself when driven by limbs ：

public class JointPoint
{
        public Transform Transform = null;
        public Quaternion InitRotation;
        public Quaternion Inverse;
        public Quaternion InverseRotation;

        public JointPoint Child = null;
        public JointPoint Parent = null;
}

（2） Theoretical analysis of algorithm

Resolve jitter ： Kalman filtering + Low pass filtering ： Key smoothing scheme

Algorithm model analysis ： Analytical model 、 heatmap&offset

There are three input, It's all the same , All input (448,448,3) Pictures of the ：

    private void UpdateVNectAsync()
    {
        input = new Tensor(videoCapture.MainTexture, 3);
        if (inputs[inputName_1] == null)
        {
            inputs[inputName_1] = input;
            inputs[inputName_2] = new Tensor(videoCapture.MainTexture, 3);
            inputs[inputName_3] = new Tensor(videoCapture.MainTexture, 3);
        }
        else
        {
             inputs[inputName_3].Dispose();
             inputs[inputName_3] = inputs[inputName_2];
              inputs[inputName_2] = inputs[inputName_1];
              inputs[inputName_1] = input;
        }

        if (!Lock && videoCapture.IsPlay())
        {
            StartCoroutine(ExecuteModelAsync());
        }
    }

four output, But we only use the last two ：

for (var i = 2; i < _model.outputs.Count; i++)
{
     b_outputs[i] = _worker.PeekOutput(_model.outputs[i]);
}
offset3D = b_outputs[2].data.Download(b_outputs[2].shape);
heatMap3D = b_outputs[3].data.Download(b_outputs[3].shape);

use heatmap Roughly position the joint , And then use offset stay heatmap As a result, the joint position is adjusted accurately .

heatmap Of ( 672 , 28 , 28 ) representative 24 With two joints 28 Size is (28,28) Characteristic graph . and offset Than heatmap Three times more characteristic graphs , Obviously, it is the precise positioning just mentioned , It just needs to be offset To x,y,z Three coordinates , So it's a triple relationship .

heatmap The order of is 1 The first joint 1 A feature map 、 The first 1 The first joint 2 A feature map 、…、 The first 2 The first joint 1 A feature map 、 The second joint 2 A feature map 、…、 The first 24 The first joint 28 A feature map
offsetmap The order of is 1 The first joint 1 Corresponding to each characteristic graph x Coordinate offset 、 The first 1 The first joint 2 Corresponding to each characteristic graph x Coordinate offset 、 The first 1 The first joint 3 Corresponding to each characteristic graph x Coordinate offset 、…、 The first 1 The first joint 28 Corresponding to each characteristic graph x Coordinate offset 、…、 The first 2 The first joint 1 Corresponding to each characteristic graph x Coordinate offset 、…、 The first 24 The first joint 28 Corresponding to each characteristic graph x Coordinate offset 、 The first 1 The first joint 1 Corresponding to each characteristic graph y Coordinate offset 、 The first 1 The first joint 2 Corresponding to each characteristic graph y Coordinate offset 、…、 The first 24 The first joint 28 Corresponding to each characteristic graph y Coordinate offset 、 The first 1 The first joint 1 Corresponding to each characteristic graph z Coordinate offset 、 The first 1 The first joint 2 Corresponding to each characteristic graph z Coordinate offset 、…、 The first 24 The first joint 28 Corresponding to each characteristic graph z Coordinate offset .

2.Unity Body drive

The body driven code in the project is mainly VNectModel.cs.

（1） Related knowledge

The inverse of four elements ,⽐ Such as ⼀ Quaternions are (1,1,1,1) Its inverse is (-1,-1,-1,1).

Quaternion Inverse(Quaternion rotation);

return ⼀ An opposite rotation .

Watch the rotation

public static Quaternion LookRotation(Vector3 forward, [DefaultValue("Vector3.up")] Vector3 upwards);

Create a specific forward and upward Direction of rotation , Returns a computed quaternion . If used to determine a transform： If the vector is orthogonal , z The axis will be and forward alignment ,y The axis will be and upward alignment . If forward The direction is 0 Will be an error .

（2） Use in projects

from Of the current joint lookrotation= Initial rotation InitRotation× Align matrix And $AA^{-1}=1$ It can be concluded that ：

a. Quaternion.Inverse( Align matrix )= Current rotation Rotation*Quaternion.Inverse( Of the current joint lookrotation)

hip.Inverse = Quaternion.Inverse(Quaternion.LookRotation(forward));
hip.InverseRotation = hip.Inverse * hip.InitRotation;

function Init() There are a lot of similar code in , The purpose is Find the intermediate matrix , According to the intermediate matrix , We can get the current object The rotation of the .VNectModel.cs Medium hip.InverseRotation Namely Quaternion.Inverse( Align matrix ).

b. Current rotation Rotation= Of the current joint lookrotation×Quaternion.Inverse( Align matrix )

// Calculate the orientation of the human body based on the coordinates of the root joint and the left and right crotch joints , Then use this as all joints LookRotation Of y Direction 
var forward = TriangleNormal(jointPoints[PositionIndex.hip.Int()].Pos3D, jointPoints[PositionIndex.lThighBend.Int()].Pos3D, jointPoints[PositionIndex.rThighBend.Int()].Pos3D);

jointPoints[***].Transform.rotation = Quaternion.LookRotation(Vector3.up, forward) * jointPoint[***].InverseRotation;

function PoseUpdate() For limb drive , The above line of code is used to calculate the current rotation of the joint .

c. Align matrix = Of the current joint lookrotation*Quaternion.Inverse( Current rotation Rotation);

root = animator.GetBoneTransform(HumanBodyBones.Hips);
midRoot = Quaternion.Inverse(root.rotation) * Quaternion.LookRotation(forward);//midRoot  To align the matrix

These codes do not exist in the project , Just to better explain the formula .