Overview of webrtc's audio network Countermeasures

WebRTC Audio data processing , It is expected that audio data processing and transmission can be realized , Low delay , Good interaction , The sound is stable without jitter , Low bit rate and low bandwidth consumption . In data transmission ,WebRTC The adoption is based on UDP Of RTP/RTCP agreement ,RTP/RTCP It does not provide reliable data transmission and quality assurance . The public Internet is a packet switched network , Naturally, there is a loss of packet transmission 、 repeat 、 Disordered sequence and delay .WebRTC It is difficult to achieve these goals of audio data processing at the same time ,WebRTC In the realization of audio network confrontation, these targets are balanced according to different situations .

Take a closer look here WebRTC Audio data processing pipeline , And pay special attention to the logic related to audio network confrontation .

WebRTC Audio data receiving and decoding playback control pipeline

In the front WebRTC Audio data coding and transmission control pipeline This paper analyzes WebRTC Audio data coding and transmission control related logic , Here's another look WebRTC Audio data receiving, decoding and playing process .

WebRTC The complete process of the conceptual abstraction level of audio data receiving and processing is as follows ：

-----------------------------     --------------------------     ---------------------------
|                           |     |                        |     |                         |
| webrtc::AudioDeviceModule | <== | webrtc::AudioTransport | <== | webrtc::AudioProcessing |
|                           |     |                        |     |                         |
-----------------------------     --------------------------     ---------------------------
                                                                            / \
                                                                            ｜｜
                                          +=+===============================+=+
                                          | |
                    -------------------------------------------- 
                    |                                          |
                    |            webrtc::AudioMixer            | 
                    |                                          |
                    --------------------------------------------
                                          / \
                                          | |
-------------------------     ---------------------------------------------------------
|                       |     |                                                       |
| cricket::MediaChannel | ==> | webrtc::AudioMixer::Source/webrtc::AudioReceiveStream |
|                       |     |                                                       |
-------------------------     ---------------------------------------------------------
                                                      ｜｜
                                                      \ /
-------------------------------------------     ---------------------
|                                         |     |                   |
| cricket::MediaChannel::NetworkInterface | <== | webrtc::Transport |
|                                         |     |                   |
-------------------------------------------     ---------------------

about WebRTC Audio data receiving and processing process ,webrtc::AudioDeviceModule Responsible for the sound PCM The data is sent to the device through the system interface and played out .webrtc::AudioDeviceModule There is usually a special playback thread inside , The whole decoding and playing process is driven by the playing thread .webrtc::AudioTransport As an adapter and glue module , It plays audio data and webrtc::AudioProcessing Audio data processing and mixing , It passes through webrtc::AudioMixer Get and mix each remote audio stream synchronously , These audio data after mixing are returned to webrtc::AudioDeviceModule Used to play outside , Will be sent to webrtc::AudioProcessing, As a reference signal for echo cancellation .webrtc::AudioMixer::Source / webrtc::AudioReceiveStream Provide decoded data for the playback process .RTCP Feedback in webrtc::AudioMixer::Source / webrtc::AudioReceiveStream China will pass webrtc::Transport Send out .webrtc::Transport It is also an adapter and glue module , It passes through cricket::MediaChannel::NetworkInterface Actually send packets to the network .cricket::MediaChannel Receive audio packets from the network and send them to webrtc::AudioMixer::Source / webrtc::AudioReceiveStream.

If the adapter and glue module on the audio data receiving and processing pipeline are omitted , The audio data receiving and processing pipeline can be simplified as follows ：

-----------------------------     ---------------------------
|                           |     |                         |
| webrtc::AudioDeviceModule | <== | webrtc::AudioProcessing |
|                           |     |                         |
-----------------------------     ---------------------------
                                             / \
                                             ｜｜
                    -------------------------------------------- 
                    |                                          |
                    |            webrtc::AudioMixer            | 
                    |                                          |
                    --------------------------------------------
                                          / \
                                          | |
-------------------------     ---------------------------------------------------------
|                       |     |                                                       |
| cricket::MediaChannel | ==> | webrtc::AudioMixer::Source/webrtc::AudioReceiveStream |
|                       |     |                                                       |
-------------------------     ---------------------------------------------------------
                                                      ｜｜
                                                      \ /
------------------------------------------------------------------------
|                                                                      |
|                 cricket::MediaChannel::NetworkInterface              |
|                                                                      |
------------------------------------------------------------------------

webrtc::AudioMixer::Source / webrtc::AudioReceiveStream Is the center of the whole process , Its implementation lies in webrtc/audio/audio_receive_stream.h / webrtc/audio/audio_receive_stream.cc, The related class hierarchy is shown in the following figure ：

stay RTC in , To achieve interaction and low latency , The audio data receiving and processing cannot only do packet reordering and decoding , It should also fully consider network confrontation , Such as PLC And send RTCP Feedback, etc , This is also a very complicated process .WebRTC The idea of separating control flow from data flow is widely used in the design of , This is in webrtc::AudioReceiveStream It is also reflected in the design and implementation of . analysis webrtc::AudioReceiveStream The design and implementation of , It can also be configured and controlled , And data flow .

It can be done to webrtc::AudioReceiveStream The configuration and control performed mainly include the following ：

• NACK,jitter buffer Maximum size ,payload type And codec Mapping, etc ;
• Configuration is used to put RTCP Packets sent to the network webrtc::Transport 、 Decryption parameters, etc ;
• webrtc::AudioReceiveStream Life cycle control of , Such as starting and stopping ;

For data flow , First, packets received from the network are sent to webrtc::AudioReceiveStream; Second, when playing ,webrtc::AudioDeviceModule from webrtc::AudioReceiveStream Get the decoded data , And send it to the playback device to play it out ; The third is webrtc::AudioReceiveStream send out RTCP Feedback packets are sent to the sender to assist in congestion control , Impact on coding transmission process .

webrtc::AudioReceiveStream In the implementation of , The main data processing flow —— Audio data reception 、 Decoding and playing process , And related modules are shown in the figure below ：

The arrows in this diagram indicate the direction of data flow , The sequence of data processing in each module is from left to right . The red box at the bottom of the figure shows the logic closely related to network confrontation .

webrtc::AudioReceiveStream In the data processing flow of the implementation of , The input data is the audio network packet and the data sent from the opposite end RTCP package , From cricket::MediaChannel, The output data is decoded PCM data , Given to webrtc::AudioTransport, And constructed RTCP Feedback package , Such as TransportCC、RTCP NACK package , Given to webrtc::Transport Send it out .

webrtc::AudioReceiveStream Inside the implementation of , Audio network packets are eventually sent to NetEQ The buffer webrtc::PacketBuffer in , When the play NetEQ Do decoding 、PLC etc. , The decoded data is provided to webrtc::AudioDeviceModule.

WebRTC Construction process of audio data receiving and processing pipeline

Here's a look at ,webrtc::AudioReceiveStream Implementation of the data processing pipeline construction process .

webrtc::AudioReceiveStream The data processing pipeline is built step by step . We surround the above webrtc::AudioReceiveStream Data processing flow chart Look at this process .

stay webrtc::AudioReceiveStream objects creating , That is to say webrtc::voe::(anonymous namespace)::ChannelReceive When an object is created , Will create some key objects , And establish the connection between some objects , The procedure is as follows ：

#0  webrtc::voe::(anonymous namespace)::ChannelReceive::ChannelReceive(webrtc::Clock*, webrtc::NetEqFactory*, webrtc::AudioDeviceModule*, webrtc::Transport*, webrtc::RtcEventLog*, unsigned int, unsigned int, unsigned long, bool, int, bool, bool, rtc::scoped_refptr<webrtc::AudioDecoderFactory>, absl::optional<webrtc::AudioCodecPairId>, rtc::scoped_refptr<webrtc::FrameDecryptorInterface>, webrtc::CryptoOptions const&, rtc::scoped_refptr<webrtc::FrameTransformerInterface>)
    (this=0x61b000008c80, clock=0x602000003bb0, neteq_factory=0x0, audio_device_module=0x614000010040, rtcp_send_transport=0x619000017cb8, rtc_event_log=0x613000011f40, local_ssrc=4195875351, remote_ssrc=1443723799, jitter_buffer_max_packets=200, jitter_buffer_fast_playout=false, jitter_buffer_min_delay_ms=0, jitter_buffer_enable_rtx_handling=false, enable_non_sender_rtt=false, decoder_factory=..., codec_pair_id=..., frame_decryptor=..., crypto_options=..., frame_transformer=...) at webrtc/audio/channel_receive.cc:517
#2  webrtc::voe::CreateChannelReceive(webrtc::Clock*, webrtc::NetEqFactory*, webrtc::AudioDeviceModule*, webrtc::Transport*, webrtc::RtcEventLog*, unsigned int, unsigned int, unsigned long, bool, int, bool, bool, rtc::scoped_refptr<webrtc::AudioDecoderFactory>, absl::optional<webrtc::AudioCodecPairId>, rtc::scoped_refptr<webrtc::FrameDecryptorInterface>, webrtc::CryptoOptions const&, rtc::scoped_refptr<webrtc::FrameTransformerInterface>)
    (clock=0x602000003bb0, neteq_factory=0x0, audio_device_module=0x614000010040, rtcp_send_transport=0x619000017cb8, rtc_event_log=0x613000011f40, local_ssrc=4195875351, remote_ssrc=1443723799, jitter_buffer_max_packets=200, jitter_buffer_fast_playout=false, jitter_buffer_min_delay_ms=0, jitter_buffer_enable_rtx_handling=false, enable_non_sender_rtt=false, decoder_factory=..., codec_pair_id=..., frame_decryptor=..., crypto_options=..., frame_transformer=...) at webrtc/audio/channel_receive.cc:1137
#3  webrtc::internal::(anonymous namespace)::CreateChannelReceive(webrtc::Clock*, webrtc::AudioState*, webrtc::NetEqFactory*, webrtc::AudioReceiveStream::Config const&, webrtc::RtcEventLog*) (clock=0x602000003bb0, audio_state=
    0x628000004100, neteq_factory=0x0, config=..., event_log=0x613000011f40) at webrtc/audio/audio_receive_stream.cc:79
#4  webrtc::internal::AudioReceiveStream::AudioReceiveStream(webrtc::Clock*, webrtc::PacketRouter*, webrtc::NetEqFactory*, webrtc::AudioReceiveStream::Config const&, rtc::scoped_refptr<webrtc::AudioState> const&, webrtc::RtcEventLog*) (this=
    0x61600005be80, clock=0x602000003bb0, packet_router=
    0x61c000060908, neteq_factory=0x0, config=..., audio_state=..., event_log=0x613000011f40)
    at webrtc/audio/audio_receive_stream.cc:103
#5  webrtc::internal::Call::CreateAudioReceiveStream(webrtc::AudioReceiveStream::Config const&) (this=
    0x620000001080, config=...) at webrtc/call/call.cc:954
#6  cricket::WebRtcVoiceMediaChannel::WebRtcAudioReceiveStream::WebRtcAudioReceiveStream(webrtc::AudioReceiveStream::Config, webrtc::Call*) (this=0x60b000010fd0, config=..., call=0x620000001080) at webrtc/media/engine/webrtc_voice_engine.cc:1220
#7  cricket::WebRtcVoiceMediaChannel::AddRecvStream(cricket::StreamParams const&) (this=0x619000017c80, sp=...)
    at webrtc/media/engine/webrtc_voice_engine.cc:2025
#8  cricket::BaseChannel::AddRecvStream_w(cricket::StreamParams const&) (this=0x619000018180, sp=...)
   ebrtc/pc/channel.cc:567
#9  cricket::BaseChannel::UpdateRemoteStreams_w(std::vector<cricket::StreamParams, std::allocator<cricket::StreamParams> > const&, webrtc::SdpType, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)
    (this=0x619000018180, streams=std::vector of length 1, capacity 1 = {...}, type=webrtc::SdpType::kOffer, error_desc=0x7ffff2387e00)
    at webrtc/pc/channel.cc:725
#10 cricket::VoiceChannel::SetRemoteContent_w(cricket::MediaContentDescription const*, webrtc::SdpType, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) (this=0x619000018180, content=0x6130000003c0, type=webrtc::SdpType::kOffer, error_desc=0x7ffff2387e00)
    at webrtc/pc/channel.cc:926
#11 cricket::BaseChannel::SetRemoteContent(cricket::MediaContentDescription const*, webrtc::SdpType, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) (this=0x619000018180, content=0x6130000003c0, type=webrtc::SdpType::kOffer, error_desc=0x7ffff2387e00)
    at webrtc/pc/channel.cc:292

webrtc::AudioReceiveStream adopt webrtc::Call establish , Pass in webrtc::AudioReceiveStream::Config, It includes and NACK、jitter buffer Maximum size 、payload type And codec Mapping related to , And webrtc::Transport And other configurations .

webrtc::voe::(anonymous namespace)::ChannelReceive The constructor of the object is as follows ：

ChannelReceive::ChannelReceive(
    Clock* clock,
    NetEqFactory* neteq_factory,
    AudioDeviceModule* audio_device_module,
    Transport* rtcp_send_transport,
    RtcEventLog* rtc_event_log,
    uint32_t local_ssrc,
    uint32_t remote_ssrc,
    size_t jitter_buffer_max_packets,
    bool jitter_buffer_fast_playout,
    int jitter_buffer_min_delay_ms,
    bool jitter_buffer_enable_rtx_handling,
    bool enable_non_sender_rtt,
    rtc::scoped_refptr<AudioDecoderFactory> decoder_factory,
    absl::optional<AudioCodecPairId> codec_pair_id,
    rtc::scoped_refptr<FrameDecryptorInterface> frame_decryptor,
    const webrtc::CryptoOptions& crypto_options,
    rtc::scoped_refptr<FrameTransformerInterface> frame_transformer)
    : worker_thread_(TaskQueueBase::Current()),
      event_log_(rtc_event_log),
      rtp_receive_statistics_(ReceiveStatistics::Create(clock)),
      remote_ssrc_(remote_ssrc),
      acm_receiver_(AcmConfig(neteq_factory,
                              decoder_factory,
                              codec_pair_id,
                              jitter_buffer_max_packets,
                              jitter_buffer_fast_playout)),
      _outputAudioLevel(),
      clock_(clock),
      ntp_estimator_(clock),
      playout_timestamp_rtp_(0),
      playout_delay_ms_(0),
      rtp_ts_wraparound_handler_(new rtc::TimestampWrapAroundHandler()),
      capture_start_rtp_time_stamp_(-1),
      capture_start_ntp_time_ms_(-1),
      _audioDeviceModulePtr(audio_device_module),
      _outputGain(1.0f),
      associated_send_channel_(nullptr),
      frame_decryptor_(frame_decryptor),
      crypto_options_(crypto_options),
      absolute_capture_time_interpolator_(clock) {
  RTC_DCHECK(audio_device_module);

  network_thread_checker_.Detach();

  acm_receiver_.ResetInitialDelay();
  acm_receiver_.SetMinimumDelay(0);
  acm_receiver_.SetMaximumDelay(0);
  acm_receiver_.FlushBuffers();

  _outputAudioLevel.ResetLevelFullRange();

  rtp_receive_statistics_->EnableRetransmitDetection(remote_ssrc_, true);
  RtpRtcpInterface::Configuration configuration;
  configuration.clock = clock;
  configuration.audio = true;
  configuration.receiver_only = true;
  configuration.outgoing_transport = rtcp_send_transport;
  configuration.receive_statistics = rtp_receive_statistics_.get();
  configuration.event_log = event_log_;
  configuration.local_media_ssrc = local_ssrc;
  configuration.rtcp_packet_type_counter_observer = this;
  configuration.non_sender_rtt_measurement = enable_non_sender_rtt;

  if (frame_transformer)
    InitFrameTransformerDelegate(std::move(frame_transformer));

  rtp_rtcp_ = ModuleRtpRtcpImpl2::Create(configuration);
  rtp_rtcp_->SetSendingMediaStatus(false);
  rtp_rtcp_->SetRemoteSSRC(remote_ssrc_);

  // Ensure that RTCP is enabled for the created channel.
  rtp_rtcp_->SetRTCPStatus(RtcpMode::kCompound);
}

webrtc::voe::(anonymous namespace)::ChannelReceive The execution process of the constructor of the object is as follows ：

• Created a webrtc::acm2::AcmReceiver object , It is established that the label in the following figure is 1 and 2 These two connections of ;
• Created a webrtc::ModuleRtpRtcpImpl2 object , Passed in when this object was created configuration Parametric outgoing_transport The configuration item points to the incoming webrtc::Transport, It is established that the label in the following figure is 3 and 4 These two connections of ;

The green module in the figure indicates that it has been connected at this stage webrtc::voe::(anonymous namespace)::ChannelReceive Module , Those marked in yellow are those modules that have not been connected ; Solid arrows indicate the connections that have been established at this stage , Dashed arrows indicate connections that have not been established .

stay ChannelReceive Of RegisterReceiverCongestionControlObjects() Function ,webrtc::PacketRouter Be received ：

#0  webrtc::voe::(anonymous namespace)::ChannelReceive::RegisterReceiverCongestionControlObjects(webrtc::PacketRouter*)
    (this=0x61b000008c80, packet_router=0x61c000060908) at webrtc/audio/channel_receive.cc:786
#1  webrtc::internal::AudioReceiveStream::AudioReceiveStream(webrtc::Clock*, webrtc::PacketRouter*, webrtc::AudioReceiveStream::Config const&, rtc::scoped_refptr<webrtc::AudioState> const&, webrtc::RtcEventLog*, std::unique_ptr<webrtc::voe::ChannelReceiveInterface, std::default_delete<webrtc::voe::ChannelReceiveInterface> >)
    (this=0x61600005be80, clock=0x602000003bb0, packet_router=0x61c000060908, config=..., audio_state=..., event_log=0x613000011f40, channel_receive=std::unique_ptr<webrtc::voe::ChannelReceiveInterface> = {...}) at webrtc/audio/audio_receive_stream.cc:130
#2  webrtc::internal::AudioReceiveStream::AudioReceiveStream(webrtc::Clock*, webrtc::PacketRouter*, webrtc::NetEqFactory*, webrtc::AudioReceiveStream::Config const&, rtc::scoped_refptr<webrtc::AudioState> const&, webrtc::RtcEventLog*)
    (this=0x61600005be80, clock=0x602000003bb0, packet_router=0x61c000060908, neteq_factory=0x0, config=..., audio_state=..., event_log=0x613000011f40)
    at webrtc/audio/audio_receive_stream.cc:98
#3  webrtc::internal::Call::CreateAudioReceiveStream(webrtc::AudioReceiveStream::Config const&) (this=0x620000001080, config=...)
    at webrtc/call/call.cc:954

This operation also occurs in webrtc::AudioReceiveStream During object creation .ChannelReceive Of RegisterReceiverCongestionControlObjects() The implementation of the function is as follows ：

void ChannelReceive::RegisterReceiverCongestionControlObjects(
    PacketRouter* packet_router) {
  RTC_DCHECK_RUN_ON(&worker_thread_checker_);
  RTC_DCHECK(packet_router);
  RTC_DCHECK(!packet_router_);
  constexpr bool remb_candidate = false;
  packet_router->AddReceiveRtpModule(rtp_rtcp_.get(), remb_candidate);
  packet_router_ = packet_router;
}

here webrtc::PacketRouter and webrtc::ModuleRtpRtcpImpl2 Connected , The label in the previous figure is 5 This connection has also been established .NetEQ Create an audio decoder when you need an audio decoder , This process will not be repeated here .

such webrtc::AudioReceiveStream The status of the internal data processing pipeline changes to the following figure ：

webrtc::AudioReceiveStream Life cycle function of Start() When called ,webrtc::AudioReceiveStream Be added to webrtc::AudioMixer：

#0  webrtc::internal::AudioState::AddReceivingStream(webrtc::AudioReceiveStream*) (this=0x628000004100, stream=0x61600005be80)
    at webrtc/audio/audio_state.cc:59
#1  webrtc::internal::AudioReceiveStream::Start() (this=0x61600005be80) at webrtc/audio/audio_receive_stream.cc:201
#2  cricket::WebRtcVoiceMediaChannel::WebRtcAudioReceiveStream::SetPlayout(bool) (this=0x60b000010fd0, playout=true)
    at webrtc/media/engine/webrtc_voice_engine.cc:1289
#3  cricket::WebRtcVoiceMediaChannel::SetPlayout(bool) (this=0x619000017c80, playout=true)
    at webrtc/media/engine/webrtc_voice_engine.cc:1865
#4  cricket::VoiceChannel::UpdateMediaSendRecvState_w() (this=0x619000018180) at webrtc/pc/channel.cc:811

such webrtc::AudioReceiveStream The data processing pipeline has been built . The status of the entire audio data processing pipeline changes to the following figure ：

WebRTC The main process of audio data receiving and processing

WebRTC The realization of audio data receiving and processing , The buffer for saving audio packets received from the network is NetEQ Of webrtc::PacketBuffer, Receive the audio packet and save it in NetEQ Of webrtc::PacketBuffer The process is as follows ：

#0  webrtc::PacketBuffer::InsertPacketList(std::__cxx11::list<webrtc::Packet, std::allocator<webrtc::Packet> >*, webrtc::DecoderDatabase const&, absl::optional<unsigned char>*, absl::optional<unsigned char>*, webrtc::StatisticsCalculator*, unsigned long, unsigned long, int)
    (this=0x606000030e60, packet_list=0x7ffff2629810, decoder_database=..., current_rtp_payload_type=0x61600005c5c5, current_cng_rtp_payload_type=0x61600005c5c7, stats=0x61600005c180, last_decoded_length=480, sample_rate=16000, target_level_ms=80)
    at webrtc/modules/audio_coding/neteq/packet_buffer.cc:216
#1  webrtc::NetEqImpl::InsertPacketInternal(webrtc::RTPHeader const&, rtc::ArrayView<unsigned char const, -4711l>)
    (this=0x61600005c480, rtp_header=..., payload=...) at webrtc/modules/audio_coding/neteq/neteq_impl.cc:690
#2  webrtc::NetEqImpl::InsertPacket(webrtc::RTPHeader const&, rtc::ArrayView<unsigned char const, -4711l>)
    (this=0x61600005c480, rtp_header=..., payload=...) at webrtc/modules/audio_coding/neteq/neteq_impl.cc:170
#3  webrtc::acm2::AcmReceiver::InsertPacket(webrtc::RTPHeader const&, rtc::ArrayView<unsigned char const, -4711l>)
    (this=0x61b000008e48, rtp_header=..., incoming_payload=...) at webrtc/modules/audio_coding/acm2/acm_receiver.cc:136
#4  webrtc::voe::(anonymous namespace)::ChannelReceive::OnReceivedPayloadData(rtc::ArrayView<unsigned char const, -4711l>, webrtc::RTPHeader const&) (this=0x61b000008c80, payload=..., rtpHeader=...) at webrtc/audio/channel_receive.cc:340
#5  webrtc::voe::(anonymous namespace)::ChannelReceive::ReceivePacket(unsigned char const*, unsigned long, webrtc::RTPHeader const&)
    (this=0x61b000008c80, packet=0x60700002b670 "\220\357\037\261\377\364ف\a\350\224\177\276", <incomplete sequence \336>, packet_length=67, header=...) at webrtc/audio/channel_receive.cc:719
#6  webrtc::voe::(anonymous namespace)::ChannelReceive::OnRtpPacket(webrtc::RtpPacketReceived const&)
    (this=0x61b000008c80, packet=...) at webrtc/audio/channel_receive.cc:669
#7  webrtc::RtpDemuxer::OnRtpPacket(webrtc::RtpPacketReceived const&) (this=0x620000001330, packet=...)
    at webrtc/call/rtp_demuxer.cc:249
#8  webrtc::RtpStreamReceiverController::OnRtpPacket(webrtc::RtpPacketReceived const&)
    (this=0x6200000012d0, packet=...) at webrtc/call/rtp_stream_receiver_controller.cc:52
#9  webrtc::internal::Call::DeliverRtp(webrtc::MediaType, rtc::CopyOnWriteBuffer, long) (this=
    0x620000001080, media_type=webrtc::MediaType::AUDIO, packet=..., packet_time_us=1654829839622021)
    at webrtc/call/call.cc:1606
#10 webrtc::internal::Call::DeliverPacket(webrtc::MediaType, rtc::CopyOnWriteBuffer, long)
    (this=0x620000001080, media_type=webrtc::MediaType::AUDIO, packet=..., packet_time_us=1654829839622021)
    at webrtc/call/call.cc:1637
#11 cricket::WebRtcVoiceMediaChannel::OnPacketReceived(rtc::CopyOnWriteBuffer, long)::$_2::operator()() const
    (this=0x606000074c68) at webrtc/media/engine/webrtc_voice_engine.cc:2229

When the play ,webrtc::AudioDeviceModule Eventually to NetEQ request PCM data , here NetEQ From webrtc::PacketBuffer And decode the packet . The audio sampling points and... Contained in the audio data packets transmitted in the network webrtc::AudioDeviceModule The audio sampling points requested each time are not necessarily the same , For example, the sampling rate is 48kHz The audio of ,webrtc::AudioDeviceModule Each request 10ms The data of , That is to say 480 A sampling point , and OPUS Each coding frame of the audio codec contains 20ms The data of , That is to say 960 A sampling point , such NetEQ return webrtc::AudioDeviceModule After each requested sampling point , There may be a residue of decoded audio data , It needs a special PCM Data buffer . The data buffer is NetEQ Of webrtc::SyncBuffer.

webrtc::AudioDeviceModule The general process of requesting to play data is as follows ：

#0  webrtc::SyncBuffer::GetNextAudioInterleaved (this=0x606000062a80, requested_len=480, output=0x628000010110)
    at webrtc/modules/audio_coding/neteq/sync_buffer.cc:86
#1  webrtc::NetEqImpl::GetAudioInternal (this=0x61600005c480, audio_frame=0x628000010110, muted=0x7fffdc92a990, action_override=...)
    at webrtc/modules/audio_coding/neteq/neteq_impl.cc:939
#2  webrtc::NetEqImpl::GetAudio (this=0x61600005c480, audio_frame=0x628000010110, muted=0x7fffdc92a990, current_sample_rate_hz=0x7fffdcc933b0, 
    action_override=...) at webrtc/modules/audio_coding/neteq/neteq_impl.cc:239
#3  webrtc::acm2::AcmReceiver::GetAudio (this=0x61b000008e48, desired_freq_hz=48000, audio_frame=0x628000010110, muted=0x7fffdc92a990)
    at webrtc/modules/audio_coding/acm2/acm_receiver.cc:151
#4  webrtc::voe::(anonymous namespace)::ChannelReceive::GetAudioFrameWithInfo (this=0x61b000008c80, sample_rate_hz=48000, 
    audio_frame=0x628000010110) at webrtc/audio/channel_receive.cc:388
#5  webrtc::internal::AudioReceiveStream::GetAudioFrameWithInfo (this=0x61600005be80, sample_rate_hz=48000, audio_frame=0x628000010110)
    at webrtc/audio/audio_receive_stream.cc:393
#6  webrtc::AudioMixerImpl::GetAudioFromSources (this=0x61d000021280, output_frequency=48000)
    at webrtc/modules/audio_mixer/audio_mixer_impl.cc:205
#7  webrtc::AudioMixerImpl::Mix (this=0x61d000021280, number_of_channels=2, audio_frame_for_mixing=0x6280000042e8)
    at webrtc/modules/audio_mixer/audio_mixer_impl.cc:175
#8  webrtc::AudioTransportImpl::NeedMorePlayData (this=0x6280000041e0, nSamples=441, nBytesPerSample=4, nChannels=2, samplesPerSec=44100, 
    audioSamples=0x61c000080080, [email protected]: 0, elapsed_time_ms=0x7fffdc929cc0, ntp_time_ms=0x7fffdc929ce0)
    at webrtc/audio/audio_transport_impl.cc:215
#9  webrtc::AudioDeviceBuffer::RequestPlayoutData (this=0x614000010058, samples_per_channel=441)
    at webrtc/modules/audio_device/audio_device_buffer.cc:303
#10 webrtc::AudioDeviceLinuxPulse::PlayThreadProcess (this=0x61900000ff80)
    at webrtc/modules/audio_device/linux/audio_device_pulse_linux.cc:2106

Look again. WebRTC Audio data processing 、 Encoding and sending process

Look more closely at WebRTC Audio data processing 、 Encoding and sending process , Take network confrontation into account more completely , WebRTC Audio data processing 、 Encoding and sending process , And related modules are shown in the figure below ：

stay WebRTC Audio data processing 、 During encoding and sending , Encoder plays a great role in network confrontation .WebRTC Through a named audio network adapter (ANA) Module , According to the network condition , Adjust the coding process .

pacing The module smoothly sends media data to the network , Congestion control congestion control The module influences pacing Module to affect the process of sending media data , In order to control congestion .

WebRTC An overview of audio network countermeasures

from WebRTC Audio acquisition 、 Handle 、 Encoding and sending process , And audio reception 、 decode 、 Processing and playback process , You can roughly sort out WebRTC The complex mechanism of audio network confrontation ：

• audio network adapter (ANA),ANA According to the network conditions , Affect the coding process to make network confrontation , Mainly used in OPUS Encoding .ANA That can affect the encoding process 5 Parameters ：
  • Intraband FEC,OPUS The encoder can generate in band FEC, When there is packet loss , Can pass FEC Information partial recovery of lost information , Even though FEC The quality of information may not be very high ; Used to resist packet loss ;
  • DTX, When the data to be encoded is empty for a long time , Can generate DTX Packets to reduce the bit rate , This mechanism may cause the delay to increase ;
  • Bit rate ;
  • The length of the frame ,OPUS Support from the 10ms To 120 ms The length of the encoded frame ;
  • The channel number .
• pacing, Smooth sending of packets .
• congestion_controller/goog_cc, Congestion control detects network conditions , And by influencing pacing To affect the sending rhythm .
• NACK, When the packet is lost , The receiving end requests the sending end to retransmit some packets ;NACK List by NetEQ maintain .
• Jitter buffer, Reorder packets , Anti network jitter .NetEQ Where the received audio network packets are stored .
• PLC, When the packet is lost , Generate missing data . from NetEQ perform .

I can't see WebRTC There is audio out of band FEC Implementation of mechanism .

Reference article

dried food | How to ensure high definition audio for Tencent conference in complex network

Done.