Vulkan pre rotation processing equipment direction

Use Vulkan Orientation of pre rotating processing equipment

 

This paper introduces how to effectively deal with Vulkan Device rotation in the application .

Vulkan Allows you to specify a ratio OpenGL More information about render States . With this power comes new responsibilities ; You should explicitly implement the OpenGL The driver in . One of them is Device direction And with   Render surface orientation The relationship between . at present ,Android Can pass 3 There are three ways to coordinate the render surface of the device with the device orientation ：

1. The device has a display processing unit (DPU), It can effectively deal with surface rotation in hardware .（ Only on supported devices ）
2. Android The operating system can handle surface rotation by adding synthesizer channels , This incurs performance costs , It depends on how the compositor must handle the rotated output image .
3. The application itself can handle surface rotation by rendering the rotated image to a rendered surface that matches the current orientation of the display .

What does this mean for your application ？

There is currently no way for an application to know whether surface rotation handled outside the application is free . Even if there is one DPU Can solve this problem for you , You may still have to pay for measurable performance losses . If your application is affected by CPU Limit , because Android Synthesizer （ It usually operates at a higher frequency ） Added GPU Usage rate , This will become a power problem . If your application is affected by GPU Limit , that Android Compositor can also preempt your application GPU Work , This results in additional performance loss .

stay Pixel 4XL When running a delivery game on , We see SurfaceFlinger（ drive Android Higher priority tasks for compositor ） Will periodically preempt the work of the application , Which leads to 1-3 Millisecond frame time hit , And increase GPU The pressure at the top /texture Memory , Because the synthesizer must read the entire frame buffer to complete the synthesis work .

The correct processing direction almost completely stopped SurfaceFlinger Of GPU preemption , and GPU The frequency has dropped 40%, Because there is no need for Android The lifting frequency used by the synthesizer .

To ensure that the surface rotation is handled correctly with as little overhead as possible （ As shown in the example above ）, We suggest the implementation method 3, This is called Pre rotation . This tells Android operating system Your application   Treatment surface rotation . You can do this by passing the surface transform flag in the specified direction during the creation of the swap chain . at present prevent Android Synthesizer own Rotate .

Know how to set the surface transform flag for each Vulkan Applications are important , Because applications tend to support multiple directions or a single direction , Where the direction of the rendered surface is different from that of the identity considered by the device . for example , Horizontal only apps on vertical identity phones , Or portrait only apps on a landscape identity tablet .

In this paper , We will describe in detail how to Vulkan Pre rotation and processing device rotation are implemented in the application .

modify AndroidManifest .xml

To handle device rotation in your application , First, change the application's  AndroidManifest.xml file , tell Android Your application will handle changes in orientation and screen size . This prevents Android Destroy and recreate when a direction change occurs Android And on the surface of the existing window Activity Call the function .onDestroy() This is through orientation（ To support the API Level <13） and screenSize Attribute is added to the activity  configChanges Part of it ：

<span style="color:var(--devsite-code-color)"><code><span style="color:var(--devsite-code-keywords-color)"><activity</span> <span style="color:var(--devsite-code-types-color)">android:name</span>=<span style="color:var(--devsite-code-strings-color)">"android.app.NativeActivity"</span>
          <span style="color:var(--devsite-code-types-color)">android:configChanges</span>=<span style="color:var(--devsite-code-strings-color)">"orientation|screenSize"</span><span style="color:var(--devsite-code-keywords-color)">></span>
</code></span>

If your application uses this property to fix its screen orientation ,screenOrientation  There is no need to do this . Besides , If your application uses fixed orientation , Then it just needs to start the application / Set the primary exchange chain during recovery .

Get identity screen resolution and camera parameters

The next thing you need to do is to test the relationship with the VK_SURFACE_TRANSFORM_IDENTITY_BIT_KHR Value the screen resolution of the associated device . This resolution is associated with the identity direction of the device , Therefore, it is always necessary to set up the exchange chain . The most reliable method is to call... When the application starts  vkGetPhysicalDeviceSurfaceCapabilitiesKHR() And store the returned range . You need to according to  currentTransform Back to Swap width and height , To ensure the storage identity screen resolution ：

<span style="color:var(--devsite-code-color)"><code><span style="color:var(--devsite-code-types-color)">VkSurfaceCapabilitiesKHR</span> capabilities;
vkGetPhysicalDeviceSurfaceCapabilitiesKHR(physDevice, surface, &capabilities);

uint32_t width = capabilities.currentExtent.width;
uint32_t height = capabilities.currentExtent.height;
<span style="color:var(--devsite-code-keywords-color)">if</span> (capabilities.currentTransform & VK_SURFACE_TRANSFORM_ROTATE_90_BIT_KHR ||
    capabilities.currentTransform & VK_SURFACE_TRANSFORM_ROTATE_270_BIT_KHR) {
  <span style="color:var(--devsite-code-comments-color)">// Swap to get identity width and height</span>
  capabilities.currentExtent.height = width;
  capabilities.currentExtent.width = height;
}

displaySizeIdentity = capabilities.currentExtent;
</code></span>

displaySizeIdentity yes VkExtent2D The structure we use to store the identification resolution of the application window surface in the natural direction of the display .

Detect the change of equipment direction (Android 10+)

The most reliable way to detect changes in direction in an application is to verify vkQueuePresentKHR() Whether the function returns  VK_SUBOPTIMAL_KHR. for example ：

<span style="color:var(--devsite-code-color)"><code><span style="color:var(--devsite-code-keywords-color)">auto</span> res = vkQueuePresentKHR(queue_, &present_info);
<span style="color:var(--devsite-code-keywords-color)">if</span> (res == VK_SUBOPTIMAL_KHR){
  orientationChanged = <span style="color:var(--devsite-code-keywords-color)">true</span>;
}
</code></span>

Be careful ： This scheme is only applicable to the operation Android 10 And later ; That is Android Start  VK_SUBOPTIMAL_KHR from vkQueuePresentKHR().  We store the results of this check-in ,orientationChanged Sure boolean Access... From the application's main rendering loop .

Detect the change of equipment direction （Android 10 Previous version ）

For operation below 10 The old version of Android The equipment , Different implementations are required , because VK_SUBOPTIMAL_KHR I won't support it .

Use polling

stay Android 10 On the previous device , You can every  pollingInterval Frame polling current device transform , among pollingInterval The granularity of is determined by the programmer . The way to do this is to call  vkGetPhysicalDeviceSurfaceCapabilitiesKHR() Fields returned , And then take it.  currentTransform Compare with the field of the currently stored surface transformation （ In this code example, it is stored in in pretransformFlag）.

<span style="color:var(--devsite-code-color)"><code>currFrameCount++;
<span style="color:var(--devsite-code-keywords-color)">if</span> (currFrameCount >= pollInterval){
  <span style="color:var(--devsite-code-types-color)">VkSurfaceCapabilitiesKHR</span> capabilities;
  vkGetPhysicalDeviceSurfaceCapabilitiesKHR(physDevice, surface, &capabilities);

  <span style="color:var(--devsite-code-keywords-color)">if</span> (pretransformFlag != capabilities.currentTransform) {
    window_resized = <span style="color:var(--devsite-code-keywords-color)">true</span>;
  }
  currFrameCount = <span style="color:var(--devsite-code-numbers-color)">0</span>;
}
</code></span>

Running Android 10 Of Pixel 4 On , polling  vkGetPhysicalDeviceSurfaceCapabilitiesKHR() Time consuming 0.120-.250 millisecond , And running Android 8 Of Pixel 1XL On , Polling time 0.110-.350 millisecond .

Use callback

Running on the Android 10 The second option for the following devices is to register  onNativeWindowResized() Callback to invoke settings  orientationChanged Flag function , Signal the application that a direction change has occurred ：

<span style="color:var(--devsite-code-color)"><code><span style="color:var(--devsite-code-keywords-color)">void</span> android_main(<span style="color:var(--devsite-code-keywords-color)">struct</span> android_app *app) {
  ...
  app->activity->callbacks->onNativeWindowResized = <span style="color:var(--devsite-code-types-color)">ResizeCallback</span>;
}
</code></span>

among ResizeCallback Defined as ：

<span style="color:var(--devsite-code-color)"><code><span style="color:var(--devsite-code-keywords-color)">void</span> <span style="color:var(--devsite-code-types-color)">ResizeCallback</span>(<span style="color:var(--devsite-code-types-color)">ANativeActivity</span> *activity, <span style="color:var(--devsite-code-types-color)">ANativeWindow</span> *window){
  orientationChanged = <span style="color:var(--devsite-code-keywords-color)">true</span>;
}
</code></span>

The disadvantage of this solution is onNativeWindowResized() Only in 90 Called when the degree direction changes （ From horizontal to vertical , vice versa ）, So, for example, the change of direction from horizontal to reverse horizontal will not trigger the reconstruction of the exchange chain , need Android Compositor flips your application .

Processing direction changes

orientationChanged  To handle direction changes , Please set the variable to true Call the direction change routine at the top of the main render loop . for example ：

<span style="color:var(--devsite-code-color)"><code><span style="color:var(--devsite-code-keywords-color)">bool</span> <span style="color:var(--devsite-code-types-color)">VulkanDrawFrame</span>() {
 <span style="color:var(--devsite-code-keywords-color)">if</span> (orientationChanged) {
   <span style="color:var(--devsite-code-types-color)">OnOrientationChange</span>();
}
</code></span>

In the OnOrientationChange() Function , You will do all the work required to recreate the exchange chain . This involves destroying  Framebuffer And any existing instances of ImageView; Re create the exchange chain while destroying the old exchange chain （ It will be discussed below ）; Then use the new exchange chain DisplayImages Recreate the framebuffer . Please note that , Attachment image （ For example, depth / Template image ） There is usually no need to recreate , Because they are based on the identity resolution of the pre rotated image .

<span style="color:var(--devsite-code-color)"><code><span style="color:var(--devsite-code-keywords-color)">void</span> <span style="color:var(--devsite-code-types-color)">OnOrientationChange</span>() {
 vkDeviceWaitIdle(getDevice());

 <span style="color:var(--devsite-code-keywords-color)">for</span> (<span style="color:var(--devsite-code-keywords-color)">int</span> i = <span style="color:var(--devsite-code-numbers-color)">0</span>; i < getSwapchainLength(); ++i) {
   vkDestroyImageView(getDevice(), displayViews_[i], <span style="color:var(--devsite-code-keywords-color)">nullptr</span>);
   vkDestroyFramebuffer(getDevice(), framebuffers_[i], <span style="color:var(--devsite-code-keywords-color)">nullptr</span>);
 }

 createSwapChain(getSwapchain());
 createFrameBuffers(render_pass, depthBuffer.image_view);
 orientationChanged = <span style="color:var(--devsite-code-keywords-color)">false</span>;
}
</code></span>

At the end of the function , You will orientationChanged Flag reset to false To indicate that you have processed the direction change .

Exchange chain entertainment

In the last section , We mentioned that the exchange chain must be recreated . The first step in doing this is to capture the new features of the rendered surface ：

<span style="color:var(--devsite-code-color)"><code><span style="color:var(--devsite-code-keywords-color)">void</span> createSwapChain(<span style="color:var(--devsite-code-types-color)">VkSwapchainKHR</span> oldSwapchain) {
   <span style="color:var(--devsite-code-types-color)">VkSurfaceCapabilitiesKHR</span> capabilities;
   vkGetPhysicalDeviceSurfaceCapabilitiesKHR(physDevice, surface, &capabilities);
   pretransformFlag = capabilities.currentTransform;
</code></span>

After filling the structure with new information , You can now check the fields VkSurfaceCapabilities To check if the direction has changed .currentTransform You store it in pretransformFlag  On site for future use , Because you will be right later MVP The matrix will need to be adjusted .

So , Please be there. VkSwapchainCreateInfo Structure to specify the following properties ：

<span style="color:var(--devsite-code-color)"><code><span style="color:var(--devsite-code-types-color)">VkSwapchainCreateInfoKHR</span> swapchainCreateInfo{
  ...
  .sType = VK_STRUCTURE_TYPE_SWAPCHAIN_CREATE_INFO_KHR,
  .imageExtent = displaySizeIdentity,
  .preTransform = pretransformFlag,
  .oldSwapchain = oldSwapchain,
};

vkCreateSwapchainKHR(device_, &swapchainCreateInfo, <span style="color:var(--devsite-code-keywords-color)">nullptr</span>, &swapchain_));

<span style="color:var(--devsite-code-keywords-color)">if</span> (oldSwapchain != VK_NULL_HANDLE) {
  vkDestroySwapchainKHR(device_, oldSwapchain, <span style="color:var(--devsite-code-keywords-color)">nullptr</span>);
}
</code></span>

The imageExtent The field will be populated with displaySizeIdentity The range you store at application startup . The preTransform The field will be populated with pretransformFlag Variable （ Set to Of currentTransform Field surfaceCapabilities）. You will also oldSwapchain Field is set to the exchange chain to be destroyed .

Be careful ： Fields and   Field matching very important ,  Because it makes Android Know that we are dealing with the change of direction ourselves , Thus avoiding Android Synthesizer .surfaceCapabilities.currentTransformswapchainCreateInfo.preTransform

MVP Matrix adjustment

The last thing you need to do is by applying the rotation matrix to MVP Matrix to apply the pre transformation . This essentially applies rotation in clip space , To rotate the generated image to the current device direction . then , You can simply update this MVP The matrix is passed into your vertex shader , And use it as usual , Without modifying your shaders .

<span style="color:var(--devsite-code-color)"><code>glm::mat4 pre_rotate_mat = glm::mat4(<span style="color:var(--devsite-code-numbers-color)">1.0f</span>);
glm::vec3 rotation_axis = glm::vec3(<span style="color:var(--devsite-code-numbers-color)">0.0f</span>, <span style="color:var(--devsite-code-numbers-color)">0.0f</span>, <span style="color:var(--devsite-code-numbers-color)">1.0f</span>);

<span style="color:var(--devsite-code-keywords-color)">if</span> (pretransformFlag & VK_SURFACE_TRANSFORM_ROTATE_90_BIT_KHR) {
  pre_rotate_mat = glm::rotate(pre_rotate_mat, glm::radians(<span style="color:var(--devsite-code-numbers-color)">90.0f</span>), rotation_axis);
}

<span style="color:var(--devsite-code-keywords-color)">else</span> <span style="color:var(--devsite-code-keywords-color)">if</span> (pretransformFlag & VK_SURFACE_TRANSFORM_ROTATE_270_BIT_KHR) {
  pre_rotate_mat = glm::rotate(pre_rotate_mat, glm::radians(<span style="color:var(--devsite-code-numbers-color)">270.0f</span>), rotation_axis);
}

<span style="color:var(--devsite-code-keywords-color)">else</span> <span style="color:var(--devsite-code-keywords-color)">if</span> (pretransformFlag & VK_SURFACE_TRANSFORM_ROTATE_180_BIT_KHR) {
  pre_rotate_mat = glm::rotate(pre_rotate_mat, glm::radians(<span style="color:var(--devsite-code-numbers-color)">180.0f</span>), rotation_axis);
}

MVP = pre_rotate_mat * MVP;
</code></span>

matters needing attention - Non full screen viewport and scissors

If your application is using a non full screen viewport / Scissors area , You need to update the device according to its orientation . This requires you to Vulkan Enable dynamic during pipe creation Viewport and Scissor Options ：

<span style="color:var(--devsite-code-color)"><code><span style="color:var(--devsite-code-types-color)">VkDynamicState</span> dynamicStates[<span style="color:var(--devsite-code-numbers-color)">2</span>] = {
  VK_DYNAMIC_STATE_VIEWPORT,
  VK_DYNAMIC_STATE_SCISSOR,
};

<span style="color:var(--devsite-code-types-color)">VkPipelineDynamicStateCreateInfo</span> dynamicInfo = {
  .sType = VK_STRUCTURE_TYPE_PIPELINE_DYNAMIC_STATE_CREATE_INFO,
  .pNext = <span style="color:var(--devsite-code-keywords-color)">nullptr</span>,
  .flags = <span style="color:var(--devsite-code-numbers-color)">0</span>,
  .dynamicStateCount = <span style="color:var(--devsite-code-numbers-color)">2</span>,
  .pDynamicStates = dynamicStates,
};

<span style="color:var(--devsite-code-types-color)">VkGraphicsPipelineCreateInfo</span> pipelineCreateInfo = {
  .sType = VK_STRUCTURE_TYPE_GRAPHICS_PIPELINE_CREATE_INFO,
  ...
  .pDynamicState = &dynamicInfo,
  ...
};

<span style="color:var(--devsite-code-types-color)">VkCreateGraphicsPipelines</span>(device, VK_NULL_HANDLE, <span style="color:var(--devsite-code-numbers-color)">1</span>, &pipelineCreateInfo, <span style="color:var(--devsite-code-keywords-color)">nullptr</span>, &mPipeline);
</code></span>

The actual calculation of the viewport range during command buffer recording is as follows ：

<span style="color:var(--devsite-code-color)"><code><span style="color:var(--devsite-code-keywords-color)">int</span> x = <span style="color:var(--devsite-code-numbers-color)">0</span>, y = <span style="color:var(--devsite-code-numbers-color)">0</span>, w = <span style="color:var(--devsite-code-numbers-color)">500</span>, h = <span style="color:var(--devsite-code-numbers-color)">400</span>;

glm::vec4 viewportData;

<span style="color:var(--devsite-code-keywords-color)">switch</span> (device-><span style="color:var(--devsite-code-types-color)">GetPretransformFlag</span>()) {
  <span style="color:var(--devsite-code-keywords-color)">case</span> VK_SURFACE_TRANSFORM_ROTATE_90_BIT_KHR:
    viewportData = {bufferWidth - h - y, x, h, w};
    <span style="color:var(--devsite-code-keywords-color)">break</span>;
  <span style="color:var(--devsite-code-keywords-color)">case</span> VK_SURFACE_TRANSFORM_ROTATE_180_BIT_KHR:
    viewportData = {bufferWidth - w - x, bufferHeight - h - y, w, h};
    <span style="color:var(--devsite-code-keywords-color)">break</span>;
  <span style="color:var(--devsite-code-keywords-color)">case</span> VK_SURFACE_TRANSFORM_ROTATE_270_BIT_KHR:
    viewportData = {y, bufferHeight - w - x, h, w};
    <span style="color:var(--devsite-code-keywords-color)">break</span>;
  <span style="color:var(--devsite-code-keywords-color)">default</span>:
    viewportData = {x, y, w, h};
    <span style="color:var(--devsite-code-keywords-color)">break</span>;
}

<span style="color:var(--devsite-code-keywords-color)">const</span> <span style="color:var(--devsite-code-types-color)">VkViewport</span> viewport = {
    .x = viewportData.x,
    .y = viewportData.y,
    .width = viewportData.z,
    .height = viewportData.w,
    .minDepth = <span style="color:var(--devsite-code-numbers-color)">0.0F</span>,
    .maxDepth = <span style="color:var(--devsite-code-numbers-color)">1.0F</span>,
};

vkCmdSetViewport(renderer-><span style="color:var(--devsite-code-types-color)">GetCurrentCommandBuffer</span>(), <span style="color:var(--devsite-code-numbers-color)">0</span>, <span style="color:var(--devsite-code-numbers-color)">1</span>, &viewport);
</code></span>

x and y Variable defines the coordinates of the upper left corner of the viewport , And he w , respectively, h Define the width and height of the viewport . The same calculation can also be used to set the scissors test , For completeness , Include it below ：

<span style="color:var(--devsite-code-color)"><code><span style="color:var(--devsite-code-keywords-color)">int</span> x = <span style="color:var(--devsite-code-numbers-color)">0</span>, y = <span style="color:var(--devsite-code-numbers-color)">0</span>, w = <span style="color:var(--devsite-code-numbers-color)">500</span>, h = <span style="color:var(--devsite-code-numbers-color)">400</span>;
glm::vec4 scissorData;

<span style="color:var(--devsite-code-keywords-color)">switch</span> (device-><span style="color:var(--devsite-code-types-color)">GetPretransformFlag</span>()) {
  <span style="color:var(--devsite-code-keywords-color)">case</span> VK_SURFACE_TRANSFORM_ROTATE_90_BIT_KHR:
    scissorData = {bufferWidth - h - y, x, h, w};
    <span style="color:var(--devsite-code-keywords-color)">break</span>;
  <span style="color:var(--devsite-code-keywords-color)">case</span> VK_SURFACE_TRANSFORM_ROTATE_180_BIT_KHR:
    scissorData = {bufferWidth - w - x, bufferHeight - h - y, w, h};
    <span style="color:var(--devsite-code-keywords-color)">break</span>;
  <span style="color:var(--devsite-code-keywords-color)">case</span> VK_SURFACE_TRANSFORM_ROTATE_270_BIT_KHR:
    scissorData = {y, bufferHeight - w - x, h, w};
    <span style="color:var(--devsite-code-keywords-color)">break</span>;
  <span style="color:var(--devsite-code-keywords-color)">default</span>:
    scissorData = {x, y, w, h};
    <span style="color:var(--devsite-code-keywords-color)">break</span>;
}

<span style="color:var(--devsite-code-keywords-color)">const</span> <span style="color:var(--devsite-code-types-color)">VkRect2D</span> scissor = {
    .offset =
        {
            .x = (int32_t)viewportData.x,
            .y = (int32_t)viewportData.y,
        },
    .extent =
        {
            .width = (uint32_t)viewportData.z,
            .height = (uint32_t)viewportData.w,
        },
};

vkCmdSetScissor(renderer-><span style="color:var(--devsite-code-types-color)">GetCurrentCommandBuffer</span>(), <span style="color:var(--devsite-code-numbers-color)">0</span>, <span style="color:var(--devsite-code-numbers-color)">1</span>, &scissor);
</code></span>

consider - Fragment shader derivatives

dFdx If your application is using derivative calculations such as and dFdy, Additional transformations may be required to interpret the rotation coordinate system , Because these calculations are performed in pixel space . This requires the application to put some preTransform Indicates to pass to the clip shader （ For example, an integer representing the current device direction ） And use it to map derivative calculations correctly ：

• about 90 degree Pre rotating frame
  • dFdx Need to map to dFdy
  • dFdy Need to map to -dFdx
• about 270 degree Pre rotating frame
  • dFdx Need to map to -dFdy
  • dFdy Need to map to dFdx
• about 180 degree Pre rotating frame ,
  • dFdx Need to map to -dFdx
  • dFdy Need to map to -dFdy

Conclusion

In order for your application to Android Make full use of Vulkan, Pre rotation must be achieved . The most important harvest of this article is ：

• Ensure that during the creation or re creation of the exchange chain , Set up pretransform sign , Make it relate to Android The flags returned by the operating system match . This will avoid synthesizer overhead .
• Fix the swap chain size to the identification resolution of the application window surface in the natural direction of the display .
• Rotate... In clip space MVP Matrix to consider the equipment direction , Because the exchange chain resolution / The range no longer updates with the display orientation .
• Update viewports and scissor rectangles as needed by the application

Sample application ： The smallest Android Pre rotation

//