当前位置：网站首页>Multimedia框架解析之MediaExtractor源码分析（一）
Multimedia框架解析之MediaExtractor源码分析（一）
2022-06-11 06:20:00 【Tinghua_M】
Extractor
Extractor在multimedia框架扮演着解析器的角色，用于解析文件的封装。extractor会把视频文件解析成音频流和视频流，把音频文件解析成音频流。这里借用一个大神的multimedia框架图展示一下Extractor所处的位置。从图中我们可以清楚的看到，NuPlayer会为每个播放的文件，创建一个MediaExtractor，这个类的作用就是解析，概念上等同于demuxer或者Parser。MediaExtractor负责从文件中分离音视频数据，并抽象成MediaSource。MediaSource生产数据，送往MediaCodec。MediaCodec又将数据通过ACodec和OMX的接口送往Decoder。Decoder解码后的数据会返回给MediaCodec，之后再由MediaCodec送往Renderer模块。
重要类说明：
- FileSource：根据构造时传递的文件描述符或者文件路径对多媒体文件进行读取等操作。
- RemoteDataSource：binder server。在加载解析器的过程中，FileSource被保存位RemoteDataSource的本地对象，供代理进行文件读取的操作。所以，通RemoteDataSource这个binder服务，extractor可以跨进程访问文件数据。
- CallbackDataSource：顾名思义就是用于回调DataSource，在加载extractor过程中，它保存了RemoteDataSource的远程对象在自己的私有变量sp mIDataSource中，通过mIDataSource，CallbackDataSource跨进程操作FileSource，从而访问多媒体文件。
- TinyCacheSource：从字面上理解就是：小型缓冲数据源。它保存了CallbackDataSource的对象，通过CallbackDataSource访问多媒体文件。
- MediaExtractorFactory：工具类，里面都是static函数。用于加载extractor组件，创建idatasource等。
- MediaExtractor：extractor组件需继承它。在8.0版本中，它是一个binder server；而在9.0的版本中，它只是一个类而已，通过其他binder服务来调用它，间接binder化。通过它可以获得解析器的音频轨或者视频轨的信息。
- RemoteMediaExtractor：binder server。RemoteMediaExtractor跟RemoteDataSource的作用是相似的，它也是将MediaExtractor保存为私有变量中，它的binder client通过它可以访问到MediaExtractor，间接解析多媒体文件。
- ExtractorPlugin：解析器组件。每一个extractor so都对应一个组件，这个组件会保存so库的句柄，路径及库中定义的解析器信息。
MediaExtractor注册
NuPlayer在Prepare阶段会去调用mediaExService->makeIDataSource(mFd, mOffset, mLength)过，该方法通过Binder机制调用Bn实现端BnMediaExtractorService子类实现者MediaExtractorService的该方法（frameworks/av/services/mediaextractor/MediaExtractorService.cpp）：
52 sp<IDataSource> MediaExtractorService::makeIDataSource(int fd, int64_t offset, int64_t length)
53 {
54 sp<DataSource> source = DataSourceFactory::CreateFromFd(fd, offset, length);
55 return CreateIDataSourceFromDataSource(source);
56 }
57 首先追踪一下MediaExtractorService类的声明（frameworks/av/services/mediaextractor/MediaExtractorService.h）：
26 class MediaExtractorService : public BinderService<MediaExtractorService>, public BnMediaExtractorService
27 {
28 friend class BinderService<MediaExtractorService>; // for MediaExtractorService()
29 public:
30 MediaExtractorService() : BnMediaExtractorService() { }
31 virtual ~MediaExtractorService() { }
32 virtual void onFirstRef() { }
33
34 static const char* getServiceName() { return "media.extractor"; }
35
36 virtual sp<IMediaExtractor> makeExtractor(const sp<IDataSource> &source, const char *mime);
37
38 virtual sp<IDataSource> makeIDataSource(int fd, int64_t offset, int64_t length);
39
40 virtual status_t dump(int fd, const Vector<String16>& args);
41
42 virtual status_t onTransact(uint32_t code, const Parcel& data, Parcel* reply,
43 uint32_t flags);
44
45 private:
46 Mutex mLock;
47 };
48
49 } // namespace android
50
51 #endif // ANDROID_MEDIA_EXTRACTOR_SERVICE_H
52 其实现的是一个Binder服务端即Bn实现端，BinderService将向ServiceManager服务管理中心添加MediaExtractorService自身服务，然后用于Binder客户端去使用即Bp代理端。
MediaExtractorService初始化：
在GenericSource::initFromDataSource()函数中，我们可以看到：
163 status_t NuPlayer::GenericSource::initFromDataSource() {
164 sp<IMediaExtractor> extractor;
165 CHECK(mDataSource != NULL);
166 sp<DataSource> dataSource = mDataSource;
167
168 mLock.unlock();
169 // This might take long time if data source is not reliable.
170 extractor = MediaExtractorFactory::Create(dataSource, NULL);
171
172 if (extractor == NULL) {
173 ALOGE("initFromDataSource, cannot create extractor!");
174 return UNKNOWN_ERROR;
175 }
176 ..................170会去调用 MediaExtractorFactory::Create函数去创建extractor。而在MediaExtractorFactory::Create函数中
40 sp<IMediaExtractor> MediaExtractorFactory::Create(
41 const sp<DataSource> &source, const char *mime) {
42 ALOGV("MediaExtractorFactory::Create %s", mime);
43
44 if (!property_get_bool("media.stagefright.extractremote", true)) {
45 // local extractor
46 ALOGW("creating media extractor in calling process");
47 return CreateFromService(source, mime);
48 } else {
49 // remote extractor
50 ALOGV("get service manager");
51 sp<IBinder> binder = defaultServiceManager()->getService(String16("media.extractor"));
52
53 if (binder != 0) {
54 sp<IMediaExtractorService> mediaExService(interface_cast<IMediaExtractorService>(binder));
55 sp<IMediaExtractor> ex = mediaExService->makeExtractor(
56 CreateIDataSourceFromDataSource(source), mime);
57 return ex;
58 } else {
59 ALOGE("extractor service not running");
60 return NULL;
61 }
62 }
63 return NULL;
64 }由于当前系统中并没有"media.stagefright.extractremote"这个property值，所以直接执行else。在else中通过 mediaExService->makeExtractor调用到MediaExtractorService::makeExtractor
33 sp<IMediaExtractor> MediaExtractorService::makeExtractor(
34 const sp<IDataSource> &remoteSource, const char *mime) {
35 ALOGV("@@@ MediaExtractorService::makeExtractor for %s", mime);
36
37 sp<DataSource> localSource = CreateDataSourceFromIDataSource(remoteSource);
38
39 sp<IMediaExtractor> extractor = MediaExtractorFactory::CreateFromService(localSource, mime);
40
41 ALOGV("extractor service created %p (%s)",
42 extractor.get(),
43 extractor == nullptr ? "" : extractor->name());
44
45 if (extractor != nullptr) {
46 registerMediaExtractor(extractor, localSource, mime);
47 return extractor;
48 }
49 return nullptr;
50 }此函数中因为目前还没有DataSource，所以37行的localSource为NULL，且这时mine也是NULL。继续执行就到了39行的MediaExtractorFactory::CreateFromService函数，两个参数都是NULL。
66 sp<IMediaExtractor> MediaExtractorFactory::CreateFromService(
67 const sp<DataSource> &source, const char *mime) {
68
69 ALOGV("MediaExtractorFactory::CreateFromService %s", mime);
70
71 UpdateExtractors(nullptr);
72
73 // initialize source decryption if needed
74 source->DrmInitialization(nullptr /* mime */);
75
76 void *meta = nullptr;
77 MediaExtractor::CreatorFunc creator = NULL;
78 MediaExtractor::FreeMetaFunc freeMeta = nullptr;
79 float confidence;
80 sp<ExtractorPlugin> plugin;
81 creator = sniff(source.get(), &confidence, &meta, &freeMeta, plugin);
82 if (!creator) {
83 ALOGV("FAILED to autodetect media content.");
84 return NULL;
85 }
86
87 MediaExtractor *ret = creator(source.get(), meta);
88 if (meta != nullptr && freeMeta != nullptr) {
89 freeMeta(meta);
90 }
91
92 ALOGV("Created an extractor '%s' with confidence %.2f",
93 ret != nullptr ? ret->name() : "<null>", confidence);
94
95 return CreateIMediaExtractorFromMediaExtractor(ret, source, plugin);
96 }UpdateExtractors
这个函数作用就是加载（注册）平台支持的媒体提取器插件so库。
294 // static
295 void MediaExtractorFactory::UpdateExtractors(const char *newUpdateApkPath) {
296 Mutex::Autolock autoLock(gPluginMutex);
297 if (newUpdateApkPath != nullptr) {
298 gPluginsRegistered = false;
299 }
300 if (gPluginsRegistered) {
301 return;
302 }
303 // 媒体提取器插件列表，初始化为空列表
304 std::shared_ptr<List<sp<ExtractorPlugin>>> newList(new List<sp<ExtractorPlugin>>());
305 // 加载该命名空间下生成so库（有多个不同文件格式提取器so库插件），so库文件夹地址为：
// /system/lib/extractors/ 或 /system/lib64/extractors/
// 现在的机器默认都加载64位库
// 再次加载system目录下的so库
306 RegisterExtractorsInSystem("/system/lib"
307 #ifdef __LP64__
308 "64"
309 #endif
310 "/extractors", *newList);
311
312 RegisterExtractorsInSystem("/vendor/lib"
313 #ifdef __LP64__
314 "64"
315 #endif
316 "/extractors", *newList);
317
318 if (newUpdateApkPath != nullptr) {
319 RegisterExtractorsInApk(newUpdateApkPath, *newList);
320 }
321
322 gPlugins = newList;
323 gPluginsRegistered = true;
324 }这个函数中最主要的就是调用了RegisterExtractorsInSystem去加载相应的so。
RegisterExtractorsInSystem
搜索加载指定目录中的so库（不同文件格式对应不同的提取器so库），现在的机器默认都加载64位库。
262 //static
263 void MediaExtractorFactory::RegisterExtractorsInSystem(
264 const char *libDirPath, List<sp<ExtractorPlugin>> &pluginList) {
265 ALOGV("search for plugins at %s", libDirPath);
// 尝试打开库文件夹
266 DIR *libDir = opendir(libDirPath);
267 if (libDir) {
268 struct dirent* libEntry;
// 循环读取库文件实例
269 while ((libEntry = readdir(libDir))) {
// 拼接成某个so库文件的全路径名
270 String8 libPath = String8(libDirPath) + "/" + libEntry->d_name;
// 打开该so库，返回该so库访问句柄无类型指针，若返回空指针则表示打开失败
271 void *libHandle = dlopen(libPath.string(), RTLD_NOW | RTLD_LOCAL);
272 if (libHandle) {
// 获取名为"GETEXTRACTORDEF"的搜索媒体提取器的方法实现，若返回空指针则表示该so库中没有该方法实现
// [dlsym]功能根据动态链接库操作句柄与符号，返回符号对应的地址，不但可以获取函数地址，也可以获取变量地址。
// 即每个媒体提取器so库都应该要实现该方法，然后才能被加载使用。
// 该方法指针声明，见下面的分析
273 MediaExtractor::GetExtractorDef getDef =
274 (MediaExtractor::GetExtractorDef) dlsym(libHandle, "GETEXTRACTORDEF");
275 if (getDef) {
276 ALOGV("registering sniffer for %s", libPath.string());
// 处理单个so库提取器信息
277 RegisterExtractor(
278 new ExtractorPlugin(getDef(), libHandle, libPath), pluginList);
279 } else {
280 ALOGW("%s does not contain sniffer", libPath.string());
281 dlclose(libHandle);
282 }
283 } else {
284 ALOGW("couldn't dlopen(%s) %s", libPath.string(), strerror(errno));
285 }
286 }
287
288 closedir(libDir);
289 } else {
290 ALOGE("couldn't opendir(%s)", libDirPath);
291 }
292 }从以下log中我们也可以看到， RegisterExtractors() 函数首先是加载了/system/lib64目录下不同格式媒体文件的extractor的so库
01-01 00:08:37.089 V 1043 1778 MediaExtractorFactory: MediaExtractorFactory::Create (null)
01-01 00:08:37.089 V 1043 1778 MediaExtractorFactory: get service manager
01-01 00:08:37.091 V 846 943 MediaExtractorService: @@@ MediaExtractorService::makeExtractor for (null)
01-01 00:08:37.091 V 846 943 MediaExtractorFactory: MediaExtractorFactory::CreateFromService (null)
01-01 00:08:37.092 V 846 943 MediaExtractorFactory: search for plugins at /system/lib64/extractors
01-01 00:08:37.093 W 846 943 MediaExtractorFactory: couldnt dlopen(/system/lib64/extractors/.) Is a directory
01-01 00:08:37.106 V 846 943 MediaExtractorFactory: registering sniffer for /system/lib64/extractors/liboggextractor.so
01-01 00:08:37.110 V 846 943 MediaExtractorFactory: registering extractor for Ogg Extractor
01-01 00:08:37.112 V 846 943 MediaExtractorFactory: registering sniffer for /system/lib64/extractors/libwavextractor.so
01-01 00:08:37.112 V 846 943 MediaExtractorFactory: registering extractor for WAV Extractor
01-01 00:08:37.112 W 846 943 MediaExtractorFactory: couldnt dlopen(/system/lib64/extractors/..) Is a directory
01-01 00:08:37.115 V 846 943 MediaExtractorFactory: registering sniffer for /system/lib64/extractors/libflacextractor.so
01-01 00:08:37.116 V 846 943 MediaExtractorFactory: registering extractor for FLAC Extractor
01-01 00:08:37.118 V 846 943 MediaExtractorFactory: registering sniffer for /system/lib64/extractors/libmp3extractor.so
01-01 00:08:37.118 V 846 943 MediaExtractorFactory: registering extractor for MP3 Extractor
01-01 00:08:37.122 V 846 943 MediaExtractorFactory: registering sniffer for /system/lib64/extractors/libmpeg2extractor.so
01-01 00:08:37.122 V 846 943 MediaExtractorFactory: registering extractor for MPEG2-PS/TS Extractor
01-01 00:08:37.125 V 846 943 MediaExtractorFactory: registering sniffer for /system/lib64/extractors/libmkvextractor.so
01-01 00:08:37.125 V 846 943 MediaExtractorFactory: registering extractor for Matroska Extractor
01-01 00:08:37.135 V 846 943 MediaExtractorFactory: registering sniffer for /system/lib64/extractors/libaacextractor.so
01-01 00:08:37.135 V 846 943 MediaExtractorFactory: registering extractor for AAC Extractor
01-01 00:08:37.139 V 846 943 MediaExtractorFactory: registering sniffer for /system/lib64/extractors/libmidiextractor.so
01-01 00:08:37.140 V 846 943 MediaExtractorFactory: registering extractor for MIDI Extractor
01-01 00:08:37.143 V 846 943 MediaExtractorFactory: registering sniffer for /system/lib64/extractors/libamrextractor.so
01-01 00:08:37.143 V 846 943 MediaExtractorFactory: registering extractor for AMR Extractor
01-01 00:08:37.154 V 846 943 MediaExtractorFactory: registering sniffer for /system/lib64/extractors/libmmparser.so
01-01 00:08:37.154 V 846 943 MediaExtractorFactory: registering extractor for QCOM Extractor
01-01 00:08:37.157 V 846 943 MediaExtractorFactory: registering sniffer for /system/lib64/extractors/libmp4extractor.so
01-01 00:08:37.157 V 846 943 MediaExtractorFactory: registering extractor for MP4 Extractor
这里分别有ogg、WAV、flac、MP3、MPEG2-PS/TS、Matroska、AAC、 MIDI、AMR、QCOM、MP4。
而在我当前的设备中，在/system/lib64下除了上述格式的extractor，还有一个不是特定格式的提取器实现模块libmmparser.so，这是高通平台自己自定义实现的媒体提取器解复用模块【mm-parser】，属于高通私有库实现。
RegisterExtractor(new ExtractorPlugin(getDef(), libHandle, libPath), pluginList) 实现分析：
此函数最里面是调用了getDef()方法即执行了每个提取器so库中对应于GetExtractorDef方法指针的实现，返回了so库提取器定义信息结构对象ExtractorDef。
ExtractorDef是so库提取器定义信息，结构体声明如下：
// [frameworks/av/include/media/MediaExtractorPluginApi.h]
struct ExtractorDef {
// 当前结构声明版本号
// version number of this structure
const uint32_t def_version;
// 当前名称提取器唯一标识符
// A unique identifier for this extractor.
// See below for a convenience macro to create this from a string.
media_uuid_t extractor_uuid;
// 当前名称提取器版本号，即有可能有两个so库提取器拥有相同唯一标识符（通常是名称相同但路径不同），
// 此时版本号大的将被使用。
// Version number of this extractor. When two extractors with the same
// uuid are encountered, the one with the largest version number will
// be used.
const uint32_t extractor_version;
// 可读的提取器名称
// a human readable name
const char *extractor_name;
union {
struct {
// 一个方法指针
SnifferFunc sniff;
} v2;
struct {
SnifferFunc sniff;
// 提取器支持的文件扩展类型（可以多种类型）
// a NULL terminated list of container mime types and/or file extensions
// that this extractor supports
const char **supported_types;
} v3;
} u;
};
再分析new ExtractorPlugin(getDef(), libHandle, libPath)，ExtractorPlugin媒体提取器插件类声明和定义：
该类是声明和定义一起实现的。该类其实就相同于一个自动管理提取器定义结构对象信息的内部指针释放问题。相当于一个智能指针类似功能。
105 struct ExtractorPlugin : public RefBase {
106 MediaExtractor::ExtractorDef def;
107 void *libHandle;
108 String8 libPath;
109 String8 uuidString;
110
111 ExtractorPlugin(MediaExtractor::ExtractorDef definition, void *handle, String8 &path)
112 : def(definition), libHandle(handle), libPath(path) {
113 for (size_t i = 0; i < sizeof MediaExtractor::ExtractorDef::extractor_uuid; i++) {
114 uuidString.appendFormat("%02x", def.extractor_uuid.b[i]);
115 }
116 }
117 ~ExtractorPlugin() {
118 if (libHandle != nullptr) {
119 ALOGV("closing handle for %s %d", libPath.c_str(), def.extractor_version);
120 dlclose(libHandle);
121 }
122 }
123 };
124 RegisterExtractor() 方法实现
173 void MediaExtractorFactory::RegisterExtractor(const sp<ExtractorPlugin> &plugin,
174 List<sp<ExtractorPlugin>> &pluginList) {
175 // sanity check check struct version, uuid, name
176 if (plugin->def.def_version == 0
177 || plugin->def.def_version > MediaExtractor::EXTRACTORDEF_VERSION) {
178 ALOGE("don't understand extractor format %u, ignoring.", plugin->def.def_version);
179 return;
180 }
181 if (memcmp(&plugin->def.extractor_uuid, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 16) == 0) {
182 ALOGE("invalid UUID, ignoring");
183 return;
184 }
185 if (plugin->def.extractor_name == NULL || strlen(plugin->def.extractor_name) == 0) {
186 ALOGE("extractors should have a name, ignoring");
187 return;
188 }
189 //for循环处理已缓存的so库插件信息列表【从头遍历】
190 for (auto it = pluginList.begin(); it != pluginList.end(); ++it) {
191 if (memcmp(&((*it)->def.extractor_uuid), &plugin->def.extractor_uuid, 16) == 0) {
192 // there's already an extractor with the same uuid
193 if ((*it)->def.extractor_version < plugin->def.extractor_version) {
194 // this one is newer, replace the old one
195 ALOGW("replacing extractor '%s' version %u with version %u",
196 plugin->def.extractor_name,
197 (*it)->def.extractor_version,
198 plugin->def.extractor_version);
199 pluginList.erase(it);
200 break;
201 } else {
202 ALOGW("ignoring extractor '%s' version %u in favor of version %u",
203 plugin->def.extractor_name,
204 plugin->def.extractor_version,
205 (*it)->def.extractor_version);
206 return;
207 }
208 }
209 }
往插件列表缓存中添加新的插件信息对象
210 ALOGV("registering extractor for %s", plugin->def.extractor_name);
211 pluginList.push_back(plugin);
212 }关于extractor的注册的一部分分析就先到这里，其实上述部分是通用的代码，而在我们的平台上，所有格式的解析都是使用高通的so。所以后续章节我们重点分析一下高通的媒体提取器解复用模块【mm-parser】。另外，关于每个so库提取器具体如何实现GetExtractorDef方法指针，我们也只是分析一下高通该功能的实现，其他so库实现方式都是类似的。
边栏推荐
- FPGA设计中提高工作频率及降低功耗题目合集
- Chapter 6 of machine learning [series] random forest model
- jenkins-凭证管理
- Learn C language well from keywords
- Stock K-line drawing
- C language war "minesweeping"
- Growth Diary 01
- Use meshlab to sample the point cloud of CAD model and display it in PCL
- What do you need to know about Amazon evaluation?
- Topic collection of FIFO minimum depth calculation
猜你喜欢
Don't be afraid of xxE vulnerabilities: understand their ferocity and detection methods
FPGA interview notes (IV) -- sequence detector, gray code in cross clock domain, ping-pong operation, static and dynamic loss reduction, fixed-point lossless error, recovery time and removal time
Build the first power cloud platform
Using Internet of things technology to accelerate digital transformation
Analyze the principle of configuration center from the perspective of Nacos client
![[must see for game development] 3-step configuration p4ignore + wonderful Q & A analysis (reprinted from user articles)](/img/4c/42933ac0fde18798ed74a23279c826.jpg)
[must see for game development] 3-step configuration p4ignore + wonderful Q & A analysis (reprinted from user articles)
FPGA interview notes (III) -- implementation of handshake signal synchronization in cross clock domain, arbitrary frequency division, binary conversion, RAM memory, original code inversion and complem
![Chapter 2 of machine learning [series] logistic regression model](/img/8f/b4c302c0309f5c91c7a40e682f9269.jpg)
Chapter 2 of machine learning [series] logistic regression model
Sentinel annotation support - @sentinelresource usage details
Transfer Learning
随机推荐
学好C语言从关键字开始
Global case | how an airline with a history of 100 years can expand and transform to promote innovation in the aviation industry
Convert text label of dataset to digital label
End of 2021 graphics of Shandong University
Database basic instruction set
CCF 2013 12-4 interesting numbers
SQLI_ LIBS range construction and 1-10get injection practice
Pycharm usage experience
What is sentinel produced by Ali?
Wechat applet (authorized login of TP5)
Learn C language well from keywords
Quantitative understanding (Quantitative deep revolutionary networks for effective information: a whitepaper)
Sqli-libs post injection question 11-17 actual combat
Do you know the functions of getbit and setbit in redis?
Observer mode (listener mode) + thread pool to realize asynchronous message sending
Servlet
021-MongoDB数据库从入门到放弃
FPGA面试题目笔记（四）—— 序列检测器、跨时钟域中的格雷码、乒乓操作、降低静动态损耗、定点化无损误差、恢复时间和移除时间
关于SIoU的原理和代码实现(回顾IoU、GIoU、DIoU、CIoU)
Shandong University machine learning final 2021