Reconfigure the ffmpeg plugin in chrome

FFmpegDemuxer

我们从demux开始说起,Chrome中非MSEThe source is goneffmpeg_demuxer,具体实现是在src/media/filters/ffmpeg_demuxer.cc里面进行的.

ffmpeg_demuxer先借助bufferdata to initialize oneformat_context,Records video format information,然后调avformat_find_stream_info得到所有的streams,一个stream包含一个track,循环streams,根据codec_id区分audio、video、text三种track,记录每种track的数量,设置播放时长duration,用fist_ptsInitialize the playback start timestart_time.

同时实例化一个DemuxerStream对象,This object will record the video width and height、Is there a rotation angle, etc,初始化audio_config和video_config,Used when decoding.

Almost every step here is a passPostTask进行的,That is, throw the function as a taskmedia线程处理,At the same time pass a callback function that the processing is completed.If one of them fails, the next step will not be performed,For example, an unsupported container format was encountered,Initialization fails at the first step,It will not call the callback function and go down.

FFmpegVideoDecoder

FFmpegVideoDecoderAfter configuring the following parameters at compile time,编译出来的chrome就支持ffmpeg解码,The default is soft solution.

如图,FFmpegVideoDecoder的结构很简单,在Chrome的框架下,实现了Decode和onNewFrame函数,Decode是将DecodeBuffertype of data sentFFmpegDecodingLoop解码,FFmpegDecodingLoop中会创建AVPacket,获取解码后的AVFrame并且完成AVFrame到chrome的VideoFrame的数据转移,最后通过frame_ready_cb（onnewFrame）回调,Send the decoded datarender.

下面是编译ffmpeg的参数：

is_component_ffmpeg = true
media_use_ffmpeg = true
ffmpeg_branding = "Chrome"

ffmpeg v4l2m2m

The background to start this exploration is,在我们的平台上ffmpeg v4l2m2m是可以直接使用的,所以就想在Chrome中也能用v4l2m2mPlugins go hard decoding,Ignore the hard display problem here as wellrender和GPU线程在使用hardware decoder的限制,Discuss the feasibility from a purely technical point of view.

add configure

通过阅读代码,发现ffmpeg中V4l2m2m默认是没有启用的,在Chrome中ffmpeg的源码是在third_party/ffmpeg/目录下,但是ffmpeg的编译是由third_party/ffmpeg/BUILD.gn组织的,at the beginning of this file,首先import了一个ffmpeg_generated.gni,So is this fileffmpegCompile the real code organization file.

在third_party/ffmpeg/ffmpeg_generated.gni中,首先会看到NOTE: this file is autogenerated by ffmpeg/chromium/scripts/generate_gn.py这一行注释,It appears that this file is automatically generated.From the generated code, it is found that the files referenced by different platforms are not exactly the same.

这个以codec_list.cIt is clear from the documentation.

在third_party/ffmpeg/libavcodec/allcodecs.c中codec_list.c是被include进来的,You can see more than one in the code tree at the same timecodec_list.c文件：

#if CONFIG_OSSFUZZ
const FFCodec * codec_list[] = {
    
    NULL,
    NULL,
    NULL
};
#else
#include "libavcodec/codec_list.c"
#endif

static AVOnce av_codec_static_init = AV_ONCE_INIT;
static void av_codec_init_static(void)
{
    
    for (int i = 0; codec_list[i]; i++) {
    
        if (codec_list[i]->init_static_data)
            codec_list[i]->init_static_data((FFCodec*)codec_list[i]);
    }
}

这里列出来linux平台不同ABI对应的文件：

third_party/ffmpeg/chromium/config/Chrome/linux/x64/libavcodec/codec_list.c
third_party/ffmpeg/chromium/config/Chrome/linux/arm/libavcodec/codec_list.c
third_party/ffmpeg/chromium/config/Chrome/linux/ia32/libavcodec/codec_list.c
third_party/ffmpeg/chromium/config/Chrome/linux/arm64/libavcodec/codec_list.c

实际上还有很多,如下,以arm/libavcodec/codec_list.c为例,There are only a few by defaultcodec列表：

static const FFCodec * const codec_list[] = {
    
    &ff_h264_decoder,
    &ff_theora_decoder,
    &ff_vp3_decoder,
    &ff_vp8_decoder,
    &ff_aac_decoder,
    &ff_flac_decoder,
    &ff_mp3_decoder,
    &ff_vorbis_decoder,
    &ff_pcm_alaw_decoder,
    &ff_pcm_f32le_decoder,
    &ff_pcm_mulaw_decoder,
    &ff_pcm_s16be_decoder,
    &ff_pcm_s16le_decoder,
    &ff_pcm_s24be_decoder,
    &ff_pcm_s24le_decoder,
    &ff_pcm_s32le_decoder,
    &ff_pcm_u8_decoder,
    &ff_libopus_decoder,
    NULL };

If only for simple debugging,Just find the right platform andABI对应的文件,It is also possible to modify this directly,But if it is compiled as a whole,这样改肯定不行,Cross-compilation requiredarm和ia32The corresponding code is the same,This is how to automatically generate this file later.

代码生成

回到third_party/ffmpeg/chromium/scripts/build_ffmpeg.py,如果要重新配置ffmpeg的组件,直接修改build_ffmpeg.py是可以的,Of course not,直接传递参数给build_ffmpeg.py是可以的.

Write the parameters that need to be passed first：

--enable-v4l2-m2m --enable-swscale \
--enable-decoder=h263_v4l2m2m,h264_v4l2m2m,hevc_v4l2m2m,mpeg1_v4l2m2m,mpeg2_v4l2m2m,vp8_v4l2m2m,vp9_v4l2m2m \
--enable-filter=h264_mp4toannexb,hevc_mp4toannexb \
--enable-demuxer=mpegts,flv,matroska

下面是使用build_ffmpeg.py脚本的过程：

调用ffmpegThe code if used to parts that are not enabled,就需要重新生成ffmpeg_generated.gni文件,才能完成编译,Finally to producearm和

86的libffmpeg.so

更新、生成arm-neon的相关的文件

export PATH=`pwd`/third_party/llvm-build/Release+Asserts/bin:$PATH


./build/linux/sysroot_scripts/install-sysroot.py --arch=arm


cd ./third_party/ffmpeg/

./chromium/scripts/build_ffmpeg.py linux arm-neon --branding Chrome \
-- --enable-v4l2-m2m --enable-swscale \
--enable-decoder=h263_v4l2m2m,h264_v4l2m2m,hevc_v4l2m2m,mpeg1_v4l2m2m,mpeg2_v4l2m2m,vp8_v4l2m2m,vp9_v4l2m2m \
--enable-filter=h264_mp4toannexb,hevc_mp4toannexb \
--enable-demuxer=mpegts,flv,matroska

./chromium/scripts/copy_config.sh
./chromium/scripts/generate_gn.py
cd ../../

更新、生成x86的相关的文件

export PATH=`pwd`/third_party/llvm-build/Release+Asserts/bin:$PATH

./build/linux/sysroot_scripts/install-sysroot.py --arch=i386

cd ./third_party/ffmpeg/

./chromium/scripts/build_ffmpeg.py linux ia32 --branding Chrome \
-- --enable-v4l2-m2m --enable-swscale \
--enable-decoder=h263_v4l2m2m,h264_v4l2m2m,hevc_v4l2m2m,mpeg1_v4l2m2m,mpeg2_v4l2m2m,vp8_v4l2m2m,vp9_v4l2m2m \
--enable-filter=h264_mp4toannexb,hevc_mp4toannexb \
--enable-demuxer=mpegts,flv,matroska

./chromium/scripts/copy_config.sh
./chromium/scripts/generate_gn.py
cd ../../

编译

export PATH="$PATH:/home/hui/chrome/depot_tools"

gn gen out/default/
autoninja -C out/default/ chrome -v

编译错误记录

lld: error: undefined symbol问题处理

ld.lld: error: undefined symbol: avcodec_free_context
>>> referenced by ffmpeg_audio_decoder.cc
>>>               clang_x86_v8_arm/thinlto-cache/llvmcache-49CB027A656D6EFA991374D541CC87AA4FC1D312:(media::FFmpegAudioDecoder::~FFmpegAudioDecoder())
>>> referenced by ffmpeg_audio_decoder.cc
>>>               clang_x86_v8_arm/thinlto-cache/llvmcache-49CB027A656D6EFA991374D541CC87AA4FC1D312:(media::FFmpegAudioDecoder::~FFmpegAudioDecoder())
>>> referenced by ffmpeg_audio_decoder.cc
>>>               clang_x86_v8_arm/thinlto-cache/llvmcache-49CB027A656D6EFA991374D541CC87AA4FC1D312:(media::FFmpegAudioDecoder::ConfigureDecoder(media::AudioDecoderConfig const&))
>>> referenced 11 more times

先检查ffmpeg.sigsIs there a definition in the file,如果有,删除out/default/clang_x86_v8_arm/obj/third_party/ffmpeg/ffmpeg_internal/目录下文件,重新编译（autoninja -C out/default/ chrome -v）.

rm out/default/clang_x86_v8_arm/obj/third_party/ffmpeg/ffmpeg_internal/*

比如添加了swscale库之后：

ld.lld: error: undefined symbol: sws_getContext
>>> referenced by ffmpeg_decoding_loop.cc
>>>               clang_x86_v8_arm/thinlto-cache/llvmcache-601B86BFA7603203EE89754C3E35196FBCC0AE0E:(media::FFmpegDecodingLoop::ConvertToI420(AVFrame*, AVFrame*))

ld.lld: error: undefined symbol: sws_scale
>>> referenced by ffmpeg_decoding_loop.cc
>>>               clang_x86_v8_arm/thinlto-cache/llvmcache-601B86BFA7603203EE89754C3E35196FBCC0AE0E:(media::FFmpegDecodingLoop::ConvertToI420(AVFrame*, AVFrame*))

ld.lld: error: undefined symbol: sws_freeContext
>>> referenced by ffmpeg_decoding_loop.cc
>>>               clang_x86_v8_arm/thinlto-cache/llvmcache-601B86BFA7603203EE89754C3E35196FBCC0AE0E:(media::FFmpegDecodingLoop::ConvertToI420(AVFrame*, AVFrame*))
clang++: error: linker command failed with exit code 1 (use -v to see invocation)

需要更新./third_party/ffmpeg/chromium/ffmpeg.sigs文件,添加：

SwsContext *sws_getContext(int srcW, int srcH, enum AVPixelFormat srcFormat,int dstW, int dstH, enum AVPixelFormat dstFormat,int flags, SwsFilter *srcFilter,SwsFilter *dstFilter, const double *param);
int attribute_align_arg sws_scale(struct SwsContext *c,const uint8_t * const srcSlice[],const int srcStride[], int srcSliceY,int srcSliceH, uint8_t *const dst[],const int dstStride[]);
void sws_freeContext(SwsContext *c);

recompile with -fPIC

ld.lld: error: relocation R_386_32 cannot be used against local symbol; recompile with -fPIC
>>> defined in clang_x86_v8_arm/thinlto-cache/llvmcache-FE528CDEC76E847EC1A72B262C147253194FC80C
>>> referenced by hscale_fast_bilinear_simd.c
>>>               clang_x86_v8_arm/thinlto-cache/llvmcache-FE528CDEC76E847EC1A72B262C147253194FC80C:(ff_init_hscaler_mmxext)

增加–enable-pic重新生成ffmpeg_generated.gni：

# --enable-pic

./build/linux/sysroot_scripts/install-sysroot.py --arch=i386

cd ./third_party/ffmpeg/

##--disable-x86asm

./chromium/scripts/build_ffmpeg.py linux ia32 --branding Chrome \
-- --enable-v4l2-m2m --enable-swscale --disable-x86asm --enable-pic \
--enable-decoder=h263_v4l2m2m,h264_v4l2m2m,hevc_v4l2m2m,mpeg1_v4l2m2m,mpeg2_v4l2m2m,vp8_v4l2m2m,vp9_v4l2m2m \
--enable-filter=h264_mp4toannexb,hevc_mp4toannexb \
--enable-demuxer=mpegts,flv,matroska

./chromium/scripts/copy_config.sh
./chromium/scripts/generate_gn.py

再次编译,Validation still doesn't resolve the error.

在./chromium/scripts/build_ffmpeg.py中增加参数--disable-x86asm解决不了output.asm找不到的问题,看了下output.asmThere is a sentence in itinclude,所以拷贝third_party/ffmpeg/libswscale/x86/autorename_libswscale_x86_output.asm到output.asm编译通过.

# ../../third_party/ffmpeg/libswscale/x86/autorename_libswscale_x86_output.asm:2: \
error: unable to open include file `output.asm': No such file or directory

看了下output.asm文件存在,autorename_libswscale_x86_output.asm直接include了output.asm,不知道为什么会报错,But copy directly as autorename_libswscale_x86_output.asm,这个文件就解决了：

cp third_party/ffmpeg/libswscale/x86/output.asm \
third_party/ffmpeg/libswscale/x86/autorename_libswscale_x86_output.asm

–disable-x86asm是不行的,前面的fpicMaybe this is what caused it,最后总结下,The previous part was compiled after numerous attempts,Just to guarantee thisbuild_ffmpeg.pyThe following parameters are differentarch一定要一样,Otherwise, the problems encountered are endless.

---

参考链接

从Chrome源码看audio/videoStreaming media implementation II

Chromium FFmpeg Roll Instructions

Chromium Customized FFmpeg 裁剪