Ue5 how to convert screen coordinates to world coordinates and World Directions

hello , Hello everyone , Let's share something about UE5 How to convert screen coordinates to world coordinates .

Screen coordinates to world coordinates

stay PlayerControoler Find this function under : DeprojectScreenPositionToWorld

bool APlayerController::DeprojectScreenPositionToWorld(float ScreenX, float ScreenY, FVector& WorldLocation, FVector& WorldDirection) const;

As long as the screen position is passed in, the world position and direction based on the screen position can be obtained .

So how did it come true ？

bool UGameplayStatics::DeprojectScreenToWorld(APlayerController const* Player, const FVector2D& ScreenPosition, FVector& WorldPosition, FVector& WorldDirection)
{
	ULocalPlayer* const LP = Player ? Player->GetLocalPlayer() : nullptr;
	if (LP && LP->ViewportClient)
	{
		// get the projection data
		FSceneViewProjectionData ProjectionData;
		if (LP->GetProjectionData(LP->ViewportClient->Viewport, eSSP_FULL, /*out*/ ProjectionData))
		{
                        // Start the inverse  
			FMatrix const InvViewProjMatrix = ProjectionData.ComputeViewProjectionMatrix().InverseFast();
			FSceneView::DeprojectScreenToWorld(ScreenPosition, ProjectionData.GetConstrainedViewRect(), InvViewProjMatrix, /*out*/ WorldPosition, /*out*/ WorldDirection);
			return true;
		}
	}

	// something went wrong, zero things and return false
	WorldPosition = FVector::ZeroVector;
	WorldDirection = FVector::ZeroVector;
	return false;
}

Among them FSceneViewProjectionData Is a data description of the scene viewport

// Projection data for a FSceneView
struct FSceneViewProjectionData
{
	/** The view origin. */
	FVector ViewOrigin;

	/** Rotation matrix transforming from world space to view space. */
	FMatrix ViewRotationMatrix;

	/** UE4 projection matrix projects such that clip space Z=1 is the near plane, and Z=0 is the infinite far plane. */
	FMatrix ProjectionMatrix;

protected:
	//The unconstrained (no aspect ratio bars applied) view rectangle (also unscaled)
	FIntRect ViewRect;

	// The constrained view rectangle (identical to UnconstrainedUnscaledViewRect if aspect ratio is not constrained)
	FIntRect ConstrainedViewRect;
};

ViewOrigin： Is the origin of the viewport

ViewRotationMatrix: It's a rotation matrix It can be understood as ViewMatrix It just doesn't contain the translation matrix

ProjectionMatrix： Projection matrix

What needs to be distinguished is ViewMatrix Is to convert the object of the world point into the local coordinates in the viewport , and WorldMatrix Is to convert local coordinates into world coordinates .

Among them the VP Perform inversion I hope that the point under the viewport can be converted into world coordinates through this inversion

FMatrix ComputeViewProjectionMatrix() const
{
	return FTranslationMatrix(-ViewOrigin) * ViewRotationMatrix * ProjectionMatrix;
}

FMatrix const InvViewProjMatrix = ProjectionData.ComputeViewProjectionMatrix().InverseFast();

When you get FSceneViewProjectionData Structured data , Now let's see DeprojectScreenToWorld

We know that the formula for rays is

Let's take a look at DeprojectScreenToWorld This function What's going on inside .

We know that if we change a local model point from local space to screen space , It has gone through these changes

a. adopt worldmatrix Turn the local points of the model into world points
b. adopt viewmatrix The matrix changes the point from the world position to the viewport position
c. adopt projematrix The projection matrix converts the viewport point to a point near the clipping surface
d. Map to by projective division NDC Space [-1.1]
e. Use half Lambert technique to [-1.1] Mapping to [0,1]
f. Then multiply the width and height of the screen to get the position of the screen

If you want to implement pickup , Then it will be an inverse process , Already know the screen position , How to find the corresponding screen position World position and Direction ？

Here we review the knowledge of a matrix .

If one worldmatrix You can turn a point from local space to world space , So I want to turn this point from world space to local space , Just for this worldmatrix Just find the inverse , It can be understood that inverse operation is the division of matrix .

such as It is known that   , Now I hope to pass 500 seek 100, It only needs    You can get 100. among    Namely 5 The inverse of , The matrix is the same .

 

 

Let's see UE5 How it's done ：

---

1. Get the screen coordinates first

float PixelX = FMath::TruncToFloat(ScreenPos.X);
float PixelY = FMath::TruncToFloat(ScreenPos.Y);

2. Get the coordinates and map them to 0-1 Range

const float NormalizedX = (PixelX - ViewRect.Min.X) / ((float)ViewRect.Width());
const float NormalizedY = (PixelY - ViewRect.Min.Y) / ((float)ViewRect.Height());

3. Mapping coordinates to NDC Space [-1,1]

const float ScreenSpaceX = (NormalizedX - 0.5f) * 2.0f;
const float ScreenSpaceY = ((1.0f - NormalizedY) - 0.5f) * 2.0f;

4. Ray at the beginning and end of projection space , among 0.01 It is near the clipping plane ,1 Is the far clipping plane

const FVector4 RayStartProjectionSpace = FVector4(ScreenSpaceX, ScreenSpaceY, 1.0f, 1.0f);
const FVector4 RayEndProjectionSpace = FVector4(ScreenSpaceX, ScreenSpaceY, 0.01f, 1.0f);

5. Through the VP The inverse operation of Turn the point of crop space into world space

const FVector4 HGRayStartWorldSpace = InvViewProjMatrix.TransformFVector4(RayStartProjectionSpace);
const FVector4 HGRayEndWorldSpace = InvViewProjMatrix.TransformFVector4(RayEndProjectionSpace);
FVector RayStartWorldSpace(HGRayStartWorldSpace.X, HGRayStartWorldSpace.Y, HGRayStartWorldSpace.Z);
FVector RayEndWorldSpace(HGRayEndWorldSpace.X, HGRayEndWorldSpace.Y, HGRayEndWorldSpace.Z);

6. Divide the vector by W To undo any projection and get 3-space coordinate ( You can understand the near clipping plane of the world )

if (HGRayStartWorldSpace.W != 0.0f)
{
	RayStartWorldSpace /= HGRayStartWorldSpace.W;
}
if (HGRayEndWorldSpace.W != 0.0f)
{
	RayEndWorldSpace /= HGRayEndWorldSpace.W;
}

7. Get the direction

const FVector RayDirWorldSpace = (RayEndWorldSpace - RayStartWorldSpace).GetSafeNormal();

8. Finally get the required value

out_WorldOrigin = RayStartWorldSpace;
out_WorldDirection = RayDirWorldSpace;

---

Let's talk about rays again sometime

Now let's apply this idea to   Self research small engine ：