Wwdc22 multimedia feature summary

M2 chip

• M2 The chip adopts enhanced second generation 5 Nano process technology and encapsulated more than 200 100 million transistors , comparison M1 many 25%;

• Support 100GB/s Unified memory bandwidth , comparison M1 promote  50%, The highest support 24GB Of LPDD5 Memory ;

• CPU Architecture continuation 8 Core architecture （4 Personality can be the core + 4 Energy efficiency core ）, comparison M1 Performance improvement  18%;

• GPU Support 10 The core , Overall graphics performance ratio at the same power consumption M1 promote  25%, At the maximum power consumption level M1 Performance improvements up to  35%;

• Equipped with a new generation of neural network engine , It can be processed per second 15.8 Trillion operations , comparison M1 promote  40%;

• Equipped with a new generation of media processing engine , Support 8K H.264、HEVC codec ; Equipped with ProRes Video processing engine ; Multi channel support 4K and 8K Video decoding and playing ;

Macbook Air 

Newly released Macbook Air With the latest M2 chip , Support 1080p Camera acquisition . So far 7 paragraph Mac Device support 1080p Video capture , Except for this release MBA, rest 6 They are 2021 paragraph 14 Inch MBP、2021 paragraph 16 Inch MBP、2021 paragraph 24 Inch 2 port iMac、2021 paragraph 24 Inch 4 port iMac、27 Inch iMac、iMac Pro.

Performance aspect , stay Photoshop When using filters and image effects in , Compared with the previous generation M1 Of Macbook Air  promote 20%.

  Use final cut pro When editing video, its performance is better than that of the previous generation M1 Of Macbook Air  promote 38%.

 

Macbook Pro 13 Inch

Newly released 13 Inch Macbook Pro With the latest M2 chip , Support 720p Camera acquisition .

Performance aspect ,ProRes Video transcoding speed compared with previous generation promote  3 times ; Use affinity photo When processing images , Compared with the performance of the previous generation promote 39%.

 

Baldur's Gate III The game performance is better than that of the previous generation promote 39%.

 

macOS

Continuity Camera 

Continuity camera The feature supports the wireless connection of iPhone As Mac Video capture camera of the device , In this way, it can be used in Mac Upper use iPhone Camera features such as portrait mode 、 Background blur 、 Multi camera capture 、 Portrait center, etc ; except Facetime, Third party software can also be used continuity camera, Such as zoom、teams、Webex. This feature also supports iPhone As Mac Audio input device .

With continuity camera, In the future Mac The external acquisition will be more flexible and easy to use , to Mac The multi-channel acquisition of audio and video provides more possibilities .

 

ScreenCaptureKit 

ScreenCaptureKit Is in macOS 12.3 To launch the Mac End screen acquisition frame , Can be carried out High performance 、 Fine control Screen capture of . In this issue WWDC It focuses on ScreenCaptureKit Functional characteristics of ：

• Support customizable screen content collection , Support full screen acquisition , Support to add or delete one or more windows specified in the screen content , Support the collection of single window content .

• Support the simultaneous acquisition of video and audio of applications .

• Support the setting of acquisition parameters , Including output resolution 、 Acquisition area 、 Color format 、 Pixel format 、 The cursor displays 、 Frame rate 、 Cache queue size , Collection parameters support dynamic setting .

• Acquisition frame buffer exists GPU In order to reduce memory copies .

• Acquisition based on hardware acceleration 、 The zoom 、 Pixel and color format conversion , Effectively reduce screen acquisition CPU Consume , Excellent overall performance .

• stay OBS Upper use CGWindowListCeateImage API Collection and use ScreenCaptureKit Collect and compare ： In the same scenario CGWindowListCeateImage API The acquisition frame rate is 7fps, and ScreenCaptureKit Accessible 60fps; Use ScreenCaptureKit Acquisition memory drops 15%,CPU Consumption decrease 50%.

iOS

Space audio

iOS 14 Start to support spatial audio technology , This technology makes fine adjustment to the sound frequency received by both ears of the user through directional audio filtering , stay AirPods Pro It simulates the traditional surround sound system . This kind of simulation doesn't just stop at realizing surround sound , It's about taking the iOS The equipment is simulated as a set of loudspeaker equipment in a fixed position in the space .

Now? iOS 16 in , You can create personalized spatial audio files by using the original depth camera of your mobile phone , So as to achieve a more accurate and immersive personalized auditory experience .

Metal 3

Metal 3 Many performance optimizations have been made on the basis of the previous version , Introduced many important new features and API, Let's introduce the main features one by one .

New characteristics

• MetalFX Upscaling

  GPU Render a lower resolution frame first , And then use it MetalFX The frame performs anti aliasing and upsampling to the target resolution , Compared with directly rendering frames of the same scene at the target resolution, the overall rendering time can be saved .MetalFX There are two kinds of upscale The way ：Temporal antialiased upscaling and Spatial upscaling.

   

• Fast Resource Loading

  In order to reduce the GPU Resource loading time ,Metal 3 Added fast resource loading API, It provides from storage devices to GPU A more direct path to , Minimize the waiting time for resource loading , such GPU You can access textures and buffers more quickly .

  Usually to optimize the resource loading time , First load a low-quality resource material , Until the high-quality resources are loaded .

Metal 3 New quick resources API Provide faster and more consistent performance , Improved resource loading speed , Thus, more time is reserved for the drawing of high-quality resources .

 

• Offline Shader Compilation

shader Compilation often needs to be done at run time , Runtime compilation can have a performance impact , This causes the frame rate to drop 、 Loading time is extended , Impact on user experience ;Metal 3 Support offline shader compile , Can be generated during project construction shader Binary file , This reduces loading time .

• Mesh Shaders

  Render pipeline with new object and mesh shaders Replace the original vertex shaders, This enables more flexible occlusion culling and LOD choice .

• Ray tracing optimization

  Metal 3 Optimized ray tracing , Significant savings CPU and GPU Time consuming ：

  1. Promoted acceleration structure Build speed

  2. To convert a part of an operation from CPU Move to GPU This reduces CPU expenses

  3. Optimize intersection and shading operations by directly accessing entity data

 

 

• Machine learning hardware acceleration

Metal 3 A lot of optimization has been done to support the hardware acceleration of machine learning

Supported models

 

HLS

HLS Content Steering

HLS Content Steering It is proposed by apple to improve the availability of global streaming media services HLS The content turns to （Content Steering） Mechanism . With the help of HLS Content steering mechanism , Content providers can deploy content to servers With all HLS The client establishes a side channel . During the use of streaming media services , The client will send to the content steering server on a regular basis Steering Manifest request , Turning to the server will be specific to the current client situation , Send a turn list to the client, that is CDN Priority of service , In this way, the latest CDN The policy is applied to the client .

Now ,HLS Content Steering Support path cloning .

• This feature is compatible with Content Stering 1.2.

• With this function, you can use the existing CDN Add a new to the list CDN route .

• Add a new... To the configuration file CDN You do not need to complete the path URI, Just fill in the server and parameter fields to support flexible URI Replacement rules .

 

HLS Interstitials 

HLS interstitials It 's apple on 2021 Launched in to make the deployment of advertising content more convenient HLS standard , Whether on the server side or the client side , It no longer needs to rely on SSAI Special labels in .

In this issue HLS interstitials Add the following features ：

• Support CUE Attribute configuration , This attribute can be used to configure the advertisement before video content 、 Post video advertising 、 One time insert advertisement .

• Support X-SNAP attribute , This attribute can be used to calibrate the time offset of advertisement insertion in live broadcast scenarios .

• Add request parameters ：

  - HLS_start_offset： Get the broadcast duration of the advertisement .

  - HLS_primary_id： Identify play clips and ads to avoid repeating the same ads .

• AVFoundation API Provide AVPlayerInterstitialController and AVPlayerInterstitialEvent, It is used to support the plug-in advertisement playing on the client .

AVQT

2021 In, Apple introduced advanced video quality tools (AVQT) , utilize AVFoundation frame ,AVQT Support SDR and HDR Various video formats in the domain 、 codecs 、 Resolution and frame rate , So as to achieve a simple and efficient workflow —— for example , No need to decode to original pixel format, etc .AVQT Use Metal , Sink the heavy pixel level computation to GPU To achieve high processing speed , It is usually used to analyze video that exceeds the real-time video frame rate . With excellent ease of use and computational efficiency ,AVQT You can delete low quality video from the video directory , So that they don't otherwise affect the users in the application .

This year, AVQT Bring the following updates ：

• Support generation based on HTML Visual reports for , Easily mark questions , Share Report .

• Support to analyze the video quality of video clips within the start and end time

• Expanded right YUV Supported types of formats , Can support 20 format , Include 444、422、420、411、410, Also support 8-bit、10-bit、12-bit、16-bit Format ; Support analysis of uncompressed native video ; Support the analysis of compressed and decoded videos outside the apple ecosystem .

• Support Linux System , Thus, server-side deployment can be supported .

 

DriverKit

DriiverKit Is a framework for developing device drivers . Currently, the driver development of the following modules is supported ：Networking、Block Storage、Serial、Audio、USB、PCI、HID、SCSI Controllers、SCSI Periphersals.

This time, DriverKit The updates brought mainly include ：

• AudioDriverKit Support the registration of real-time callbacks ; But every time IO Get callback during operation ; Callbacks can be used to process threads in real time, such as signal processing .

• Enable new permission setting parameters .

• DriverKit Support iPad,USBDriverKit、PCIDriverKit and AudioDriverKit Can be used for iPadOS 16, carrying M1 Chip iPad You can use .

 

EDR

EDR（Extended Dynamic Range） It is a kind of product launched by apple Rendering technology , It can support the device to display correctly on the screen at the same time SDR and HDR Content ,EDR Not directly HDR The area becomes brighter , But to recognize HDR While improving the overall screen brightness after content , Reduce non HDR The white dot value of the area , Make it look less bright .

This time WWDC, Apple once again introduced... Through several keynote speeches EDR principle ; How to use CoreImage Library display EDR Video images ; How to use AVFoundation Library decoding HDR Video and make it support EDR Show play .

It introduces EDR API The following new features ：

• iOS and iPadOS Support EDR API.

• 12.9 Inch iPad Pro Two new features are added ：Reference Patterns and Sidecar Support for EDR Rendering （Sidecar It's a technology of apple , Support will iPad As Mac Extended screen ）.

summary

Netease cloud audio and video call 2.0 SDK Has supported 4K & 8K Ultra high resolution and use on specific conference systems , Newly released Macbook Air and Macbook Pro 13 If used together 4K & 8K Camera or other HD video source , Get a great experience with ultra-high resolution .

Netease cloud audio and video call 2.0 SDK Support based on GPU Video pre-processing framework , Can maximize GPU Computing power and save CPU Consume , Excellent video pre-processing capability with excellent performance ; In the future Metal 3 New features will give further play to Netease cloud audio and video calls 2.0 SDK Performance advantages .

Netease cloud audio and video call 2.0 SDK PC and Mac The terminal now supports video dual capture , Fully meet the video needs of users in different scenarios ;Continuity Camera The expansibility and flexibility of will give Netease cloud audio and video calls 2.0 SDK The dual camera function is Mac The end brings more application scenarios and play space .

Netease cloud audio and video call 2.0 SDK High performance screen capture is now supported , Only the required windows and contents can be collected .ScreenCaptureKit by Mac The screen acquisition of the new terminal system brings more optional technical solutions .

Overall speaking , This time WWDC New features in audio and video multimedia bring more imagination to developers , In the future, Netease cloud audio and video calls 2.0 SDK With the support of these new features, it will be even more powerful .