Data types and usage of mediacodec

Three data types

The codec processes three kinds of data ： compressed data 、 Raw audio data and Raw video data .

All three types of data can be used ByteBuffer To deal with , But you should use the original video data surface To improve the performance of codec .Surface Use native video buffer, Without mapping or copying them to ByteBuffers; therefore , It's more efficient .

In the use of Surface when , The original video data is usually inaccessible , But you can use ImageReader Such access raw Video frame . This may still be better than using ByteBuffers More effective , Because some native buffer May map to ByteBuffer#isDirect ByteBuffers. Use ByteBuffer Mode time , have access to Image Classes and getInput/OutputImage（int） Access the original video frame .

Now basically api Versions are greater than 21 Of , So the latter two types are used more ：

• Synchronous API using buffers
• Asynchronous API using buffers

Processing Mode	API version <= 20	API version >= 21
Synchronous API using buffer arrays	Supported	Deprecated
Synchronous API using buffers	Not Available	Supported
Asynchronous API using buffers	Not Available	Supported

Raw Video Buffers

stay ByteBuffer In mode ,video buffer from MediaFormat::KEY_COLOR_FORMAT To make sure .video codec Three color formats are supported ：

• native raw video format: This is from COLOR_FormatSurface Appoint , Can be associated with input or output surface Use it together .
• flexible YUV buffers: COLOR_FormatYUV420Flexible, These can be used to enter / Output surface, It can also be in ByteBuffer Use... In mode getInput/OutputImage.
• other, specific formats: Usually only in ByteBuffer These formats are supported in mode . Some color formats are vendor specific , have access to getInput/OutputImage.

MediaCodec The synchronization of

java Realization

MediaCodec codec = MediaCodec.createByCodecName(name);
 codec.configure(format, …);
 MediaFormat outputFormat = codec.getOutputFormat(); // option B
 codec.start();
 for (;;) {
    
  int inputBufferId = codec.dequeueInputBuffer(timeoutUs);
  if (inputBufferId >= 0) {
    
    ByteBuffer inputBuffer = codec.getInputBuffer(…);
    // fill inputBuffer with valid data
    /// …
    codec.queueInputBuffer(inputBufferId, …);
  }
  int outputBufferId = codec.dequeueOutputBuffer(…);
  if (outputBufferId >= 0) {
    
    ByteBuffer outputBuffer = codec.getOutputBuffer(outputBufferId);
    MediaFormat bufferFormat = codec.getOutputFormat(outputBufferId); // option A
    // bufferFormat is identical to outputFormat
    // outputBuffer is ready to be processed or rendered.
    /// …
    codec.releaseOutputBuffer(outputBufferId, …);
  } else if (outputBufferId == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
    
    // Subsequent data will conform to new format.
    // Can ignore if using getOutputFormat(outputBufferId)
    outputFormat = codec.getOutputFormat(); // option B
  }
 }
 codec.stop();
 codec.release();

MediaCodec The asynchronous implementation of

java Realization

MediaCodec codec = MediaCodec.createByCodecName(name);
 MediaFormat mOutputFormat; // member variable
 codec.setCallback(new MediaCodec.Callback() {
    
  @Override
  void onInputBufferAvailable(MediaCodec mc, int inputBufferId) {
    
    ByteBuffer inputBuffer = codec.getInputBuffer(inputBufferId);
    // fill inputBuffer with valid data
    /// …
    codec.queueInputBuffer(inputBufferId, …);
  }
 
  @Override
  void onOutputBufferAvailable(MediaCodec mc, int outputBufferId, …) {
    
    ByteBuffer outputBuffer = codec.getOutputBuffer(outputBufferId);
    MediaFormat bufferFormat = codec.getOutputFormat(outputBufferId); // option A
    // bufferFormat is equivalent to mOutputFormat
    // outputBuffer is ready to be processed or rendered.
    /// …
    codec.releaseOutputBuffer(outputBufferId, …);
  }
 
  @Override
  void onOutputFormatChanged(MediaCodec mc, MediaFormat format) {
    
    // Subsequent data will conform to new format.
    // Can ignore if using getOutputFormat(outputBufferId)
    mOutputFormat = format; // option B
  }
 
  @Override
  void onError(…) {
    
    /// …
  }
 });
 codec.configure(format, …);
 mOutputFormat = codec.getOutputFormat(); // option B
 codec.start();
 // wait for processing to complete
 codec.stop();
 codec.release();

MediaCodec EOS

When the need ends , Must send... To codec EOS The signal , Sure adopt queueInputBuffer call , Appoint buffer Of flag by BUFFER_FLAG_END_OF_STREAM, Can be in the last valid input / output buffer On the implementation , You can also specify an additional Null I / O buffer Set up EOS flag To perform this operation , If you use empty buffer, The timestamp is ignored .

The codec will continue to return output buffer, Until it passes in dequeueOutputBuffer Medium BufferInfo Get the same EOS Mark or pass Callback#onOutputBufferAvailable return , Finally issued EOS The signal . This can be done in the last valid output buffer Set up , It can also be in the last valid output buffer After that, set... On the empty buffer , empty buffer The timestamp of should be ignored .

After sending out the input end-of-stream After the signal , Do not commit additional input buffers , Unless the codec has been flushed, or perhaps stopped and restarted.

Codec-specific Data

Some formats , In especial AAC Audio and MPEG4、H.264 and H.265 Video format , You need to have a before the actual data Codec-specific Data. When dealing with this compressed format , These data must be in start() Later and actual frame Data is submitted to the codec before . In the face of queueInputBuffer In call to , Signs must be used BUFFER_FLAG_CODEC_CONFIG To mark the data .

Codec-specific data It can also be included in key“csd-0”、“csd-1” Wait in ByteBuffer In the format configured in the entry . these key Always included from MediaExtractor#getTrackFormat To obtain the track MediaFormat in . In the format Codec-specific data stay start() Automatically submit to the codec , You can't explicitly Submit this data . If the format does not contain Codec-specific data , You can choose to use a specified number of... According to the format requirements buffer Submit... In the correct order . about H.264 AVC, You can also connect all Codec-specific data, And treat it as a single codec-config buffer Submit to codec .

MediaCodec Surface

Use input Surface

Use Input Surface when , No accessible input buffer, because buffer Will automatically input from to Surface Pass to codec . call dequeueInputBuffer Will lead to IllegalStateException,getInputBuffers() Returns a dummy that cannot be written ByteBuffer[] Array .

call signalEndOfInputStream() To signal the end of the flow . After this call , Input Surface Will immediately stop sending to the codec submit data .

Use the output Surface

When using output Surface, Data processing and ByteBuffer The pattern is almost the same ; however , Output buffer Will not be accessible , And expressed as a null value .getOutputBuffer/Image Will return null,getOutputBuffers() Will return only null-s Array of .

Use the output surface, You can choose whether to surface Render each output on buffer, You have three choices ：

• Do not render the buffer: Call releaseOutputBuffer(bufferId, false)
• Render the buffer with the default timestamp: Call releaseOutputBuffer(bufferId, true)
• Render the buffer with a specific timestamp: Call releaseOutputBuffer(bufferId, timestamp)