当前位置：网站首页>Multimedia框架解析之MediaExtractor源码分析（一）
Multimedia框架解析之MediaExtractor源码分析（一）
2022-06-11 06:20:00 【Tinghua_M】
Extractor
Extractor在multimedia框架扮演着解析器的角色，用于解析文件的封装。extractor会把视频文件解析成音频流和视频流，把音频文件解析成音频流。这里借用一个大神的multimedia框架图展示一下Extractor所处的位置。从图中我们可以清楚的看到，NuPlayer会为每个播放的文件，创建一个MediaExtractor，这个类的作用就是解析，概念上等同于demuxer或者Parser。MediaExtractor负责从文件中分离音视频数据，并抽象成MediaSource。MediaSource生产数据，送往MediaCodec。MediaCodec又将数据通过ACodec和OMX的接口送往Decoder。Decoder解码后的数据会返回给MediaCodec，之后再由MediaCodec送往Renderer模块。
重要类说明：
- FileSource：根据构造时传递的文件描述符或者文件路径对多媒体文件进行读取等操作。
- RemoteDataSource：binder server。在加载解析器的过程中，FileSource被保存位RemoteDataSource的本地对象，供代理进行文件读取的操作。所以，通RemoteDataSource这个binder服务，extractor可以跨进程访问文件数据。
- CallbackDataSource：顾名思义就是用于回调DataSource，在加载extractor过程中，它保存了RemoteDataSource的远程对象在自己的私有变量sp mIDataSource中，通过mIDataSource，CallbackDataSource跨进程操作FileSource，从而访问多媒体文件。
- TinyCacheSource：从字面上理解就是：小型缓冲数据源。它保存了CallbackDataSource的对象，通过CallbackDataSource访问多媒体文件。
- MediaExtractorFactory：工具类，里面都是static函数。用于加载extractor组件，创建idatasource等。
- MediaExtractor：extractor组件需继承它。在8.0版本中，它是一个binder server；而在9.0的版本中，它只是一个类而已，通过其他binder服务来调用它，间接binder化。通过它可以获得解析器的音频轨或者视频轨的信息。
- RemoteMediaExtractor：binder server。RemoteMediaExtractor跟RemoteDataSource的作用是相似的，它也是将MediaExtractor保存为私有变量中，它的binder client通过它可以访问到MediaExtractor，间接解析多媒体文件。
- ExtractorPlugin：解析器组件。每一个extractor so都对应一个组件，这个组件会保存so库的句柄，路径及库中定义的解析器信息。
MediaExtractor注册
NuPlayer在Prepare阶段会去调用mediaExService->makeIDataSource(mFd, mOffset, mLength)过，该方法通过Binder机制调用Bn实现端BnMediaExtractorService子类实现者MediaExtractorService的该方法（frameworks/av/services/mediaextractor/MediaExtractorService.cpp）：
52 sp<IDataSource> MediaExtractorService::makeIDataSource(int fd, int64_t offset, int64_t length)
53 {
54 sp<DataSource> source = DataSourceFactory::CreateFromFd(fd, offset, length);
55 return CreateIDataSourceFromDataSource(source);
56 }
57 首先追踪一下MediaExtractorService类的声明（frameworks/av/services/mediaextractor/MediaExtractorService.h）：
26 class MediaExtractorService : public BinderService<MediaExtractorService>, public BnMediaExtractorService
27 {
28 friend class BinderService<MediaExtractorService>; // for MediaExtractorService()
29 public:
30 MediaExtractorService() : BnMediaExtractorService() { }
31 virtual ~MediaExtractorService() { }
32 virtual void onFirstRef() { }
33
34 static const char* getServiceName() { return "media.extractor"; }
35
36 virtual sp<IMediaExtractor> makeExtractor(const sp<IDataSource> &source, const char *mime);
37
38 virtual sp<IDataSource> makeIDataSource(int fd, int64_t offset, int64_t length);
39
40 virtual status_t dump(int fd, const Vector<String16>& args);
41
42 virtual status_t onTransact(uint32_t code, const Parcel& data, Parcel* reply,
43 uint32_t flags);
44
45 private:
46 Mutex mLock;
47 };
48
49 } // namespace android
50
51 #endif // ANDROID_MEDIA_EXTRACTOR_SERVICE_H
52 其实现的是一个Binder服务端即Bn实现端，BinderService将向ServiceManager服务管理中心添加MediaExtractorService自身服务，然后用于Binder客户端去使用即Bp代理端。
MediaExtractorService初始化：
在GenericSource::initFromDataSource()函数中，我们可以看到：
163 status_t NuPlayer::GenericSource::initFromDataSource() {
164 sp<IMediaExtractor> extractor;
165 CHECK(mDataSource != NULL);
166 sp<DataSource> dataSource = mDataSource;
167
168 mLock.unlock();
169 // This might take long time if data source is not reliable.
170 extractor = MediaExtractorFactory::Create(dataSource, NULL);
171
172 if (extractor == NULL) {
173 ALOGE("initFromDataSource, cannot create extractor!");
174 return UNKNOWN_ERROR;
175 }
176 ..................170会去调用 MediaExtractorFactory::Create函数去创建extractor。而在MediaExtractorFactory::Create函数中
40 sp<IMediaExtractor> MediaExtractorFactory::Create(
41 const sp<DataSource> &source, const char *mime) {
42 ALOGV("MediaExtractorFactory::Create %s", mime);
43
44 if (!property_get_bool("media.stagefright.extractremote", true)) {
45 // local extractor
46 ALOGW("creating media extractor in calling process");
47 return CreateFromService(source, mime);
48 } else {
49 // remote extractor
50 ALOGV("get service manager");
51 sp<IBinder> binder = defaultServiceManager()->getService(String16("media.extractor"));
52
53 if (binder != 0) {
54 sp<IMediaExtractorService> mediaExService(interface_cast<IMediaExtractorService>(binder));
55 sp<IMediaExtractor> ex = mediaExService->makeExtractor(
56 CreateIDataSourceFromDataSource(source), mime);
57 return ex;
58 } else {
59 ALOGE("extractor service not running");
60 return NULL;
61 }
62 }
63 return NULL;
64 }由于当前系统中并没有"media.stagefright.extractremote"这个property值，所以直接执行else。在else中通过 mediaExService->makeExtractor调用到MediaExtractorService::makeExtractor
33 sp<IMediaExtractor> MediaExtractorService::makeExtractor(
34 const sp<IDataSource> &remoteSource, const char *mime) {
35 ALOGV("@@@ MediaExtractorService::makeExtractor for %s", mime);
36
37 sp<DataSource> localSource = CreateDataSourceFromIDataSource(remoteSource);
38
39 sp<IMediaExtractor> extractor = MediaExtractorFactory::CreateFromService(localSource, mime);
40
41 ALOGV("extractor service created %p (%s)",
42 extractor.get(),
43 extractor == nullptr ? "" : extractor->name());
44
45 if (extractor != nullptr) {
46 registerMediaExtractor(extractor, localSource, mime);
47 return extractor;
48 }
49 return nullptr;
50 }此函数中因为目前还没有DataSource，所以37行的localSource为NULL，且这时mine也是NULL。继续执行就到了39行的MediaExtractorFactory::CreateFromService函数，两个参数都是NULL。
66 sp<IMediaExtractor> MediaExtractorFactory::CreateFromService(
67 const sp<DataSource> &source, const char *mime) {
68
69 ALOGV("MediaExtractorFactory::CreateFromService %s", mime);
70
71 UpdateExtractors(nullptr);
72
73 // initialize source decryption if needed
74 source->DrmInitialization(nullptr /* mime */);
75
76 void *meta = nullptr;
77 MediaExtractor::CreatorFunc creator = NULL;
78 MediaExtractor::FreeMetaFunc freeMeta = nullptr;
79 float confidence;
80 sp<ExtractorPlugin> plugin;
81 creator = sniff(source.get(), &confidence, &meta, &freeMeta, plugin);
82 if (!creator) {
83 ALOGV("FAILED to autodetect media content.");
84 return NULL;
85 }
86
87 MediaExtractor *ret = creator(source.get(), meta);
88 if (meta != nullptr && freeMeta != nullptr) {
89 freeMeta(meta);
90 }
91
92 ALOGV("Created an extractor '%s' with confidence %.2f",
93 ret != nullptr ? ret->name() : "<null>", confidence);
94
95 return CreateIMediaExtractorFromMediaExtractor(ret, source, plugin);
96 }UpdateExtractors
这个函数作用就是加载（注册）平台支持的媒体提取器插件so库。
294 // static
295 void MediaExtractorFactory::UpdateExtractors(const char *newUpdateApkPath) {
296 Mutex::Autolock autoLock(gPluginMutex);
297 if (newUpdateApkPath != nullptr) {
298 gPluginsRegistered = false;
299 }
300 if (gPluginsRegistered) {
301 return;
302 }
303 // 媒体提取器插件列表，初始化为空列表
304 std::shared_ptr<List<sp<ExtractorPlugin>>> newList(new List<sp<ExtractorPlugin>>());
305 // 加载该命名空间下生成so库（有多个不同文件格式提取器so库插件），so库文件夹地址为：
// /system/lib/extractors/ 或 /system/lib64/extractors/
// 现在的机器默认都加载64位库
// 再次加载system目录下的so库
306 RegisterExtractorsInSystem("/system/lib"
307 #ifdef __LP64__
308 "64"
309 #endif
310 "/extractors", *newList);
311
312 RegisterExtractorsInSystem("/vendor/lib"
313 #ifdef __LP64__
314 "64"
315 #endif
316 "/extractors", *newList);
317
318 if (newUpdateApkPath != nullptr) {
319 RegisterExtractorsInApk(newUpdateApkPath, *newList);
320 }
321
322 gPlugins = newList;
323 gPluginsRegistered = true;
324 }这个函数中最主要的就是调用了RegisterExtractorsInSystem去加载相应的so。
RegisterExtractorsInSystem
搜索加载指定目录中的so库（不同文件格式对应不同的提取器so库），现在的机器默认都加载64位库。
262 //static
263 void MediaExtractorFactory::RegisterExtractorsInSystem(
264 const char *libDirPath, List<sp<ExtractorPlugin>> &pluginList) {
265 ALOGV("search for plugins at %s", libDirPath);
// 尝试打开库文件夹
266 DIR *libDir = opendir(libDirPath);
267 if (libDir) {
268 struct dirent* libEntry;
// 循环读取库文件实例
269 while ((libEntry = readdir(libDir))) {
// 拼接成某个so库文件的全路径名
270 String8 libPath = String8(libDirPath) + "/" + libEntry->d_name;
// 打开该so库，返回该so库访问句柄无类型指针，若返回空指针则表示打开失败
271 void *libHandle = dlopen(libPath.string(), RTLD_NOW | RTLD_LOCAL);
272 if (libHandle) {
// 获取名为"GETEXTRACTORDEF"的搜索媒体提取器的方法实现，若返回空指针则表示该so库中没有该方法实现
// [dlsym]功能根据动态链接库操作句柄与符号，返回符号对应的地址，不但可以获取函数地址，也可以获取变量地址。
// 即每个媒体提取器so库都应该要实现该方法，然后才能被加载使用。
// 该方法指针声明，见下面的分析
273 MediaExtractor::GetExtractorDef getDef =
274 (MediaExtractor::GetExtractorDef) dlsym(libHandle, "GETEXTRACTORDEF");
275 if (getDef) {
276 ALOGV("registering sniffer for %s", libPath.string());
// 处理单个so库提取器信息
277 RegisterExtractor(
278 new ExtractorPlugin(getDef(), libHandle, libPath), pluginList);
279 } else {
280 ALOGW("%s does not contain sniffer", libPath.string());
281 dlclose(libHandle);
282 }
283 } else {
284 ALOGW("couldn't dlopen(%s) %s", libPath.string(), strerror(errno));
285 }
286 }
287
288 closedir(libDir);
289 } else {
290 ALOGE("couldn't opendir(%s)", libDirPath);
291 }
292 }从以下log中我们也可以看到， RegisterExtractors() 函数首先是加载了/system/lib64目录下不同格式媒体文件的extractor的so库
01-01 00:08:37.089 V 1043 1778 MediaExtractorFactory: MediaExtractorFactory::Create (null)
01-01 00:08:37.089 V 1043 1778 MediaExtractorFactory: get service manager
01-01 00:08:37.091 V 846 943 MediaExtractorService: @@@ MediaExtractorService::makeExtractor for (null)
01-01 00:08:37.091 V 846 943 MediaExtractorFactory: MediaExtractorFactory::CreateFromService (null)
01-01 00:08:37.092 V 846 943 MediaExtractorFactory: search for plugins at /system/lib64/extractors
01-01 00:08:37.093 W 846 943 MediaExtractorFactory: couldnt dlopen(/system/lib64/extractors/.) Is a directory
01-01 00:08:37.106 V 846 943 MediaExtractorFactory: registering sniffer for /system/lib64/extractors/liboggextractor.so
01-01 00:08:37.110 V 846 943 MediaExtractorFactory: registering extractor for Ogg Extractor
01-01 00:08:37.112 V 846 943 MediaExtractorFactory: registering sniffer for /system/lib64/extractors/libwavextractor.so
01-01 00:08:37.112 V 846 943 MediaExtractorFactory: registering extractor for WAV Extractor
01-01 00:08:37.112 W 846 943 MediaExtractorFactory: couldnt dlopen(/system/lib64/extractors/..) Is a directory
01-01 00:08:37.115 V 846 943 MediaExtractorFactory: registering sniffer for /system/lib64/extractors/libflacextractor.so
01-01 00:08:37.116 V 846 943 MediaExtractorFactory: registering extractor for FLAC Extractor
01-01 00:08:37.118 V 846 943 MediaExtractorFactory: registering sniffer for /system/lib64/extractors/libmp3extractor.so
01-01 00:08:37.118 V 846 943 MediaExtractorFactory: registering extractor for MP3 Extractor
01-01 00:08:37.122 V 846 943 MediaExtractorFactory: registering sniffer for /system/lib64/extractors/libmpeg2extractor.so
01-01 00:08:37.122 V 846 943 MediaExtractorFactory: registering extractor for MPEG2-PS/TS Extractor
01-01 00:08:37.125 V 846 943 MediaExtractorFactory: registering sniffer for /system/lib64/extractors/libmkvextractor.so
01-01 00:08:37.125 V 846 943 MediaExtractorFactory: registering extractor for Matroska Extractor
01-01 00:08:37.135 V 846 943 MediaExtractorFactory: registering sniffer for /system/lib64/extractors/libaacextractor.so
01-01 00:08:37.135 V 846 943 MediaExtractorFactory: registering extractor for AAC Extractor
01-01 00:08:37.139 V 846 943 MediaExtractorFactory: registering sniffer for /system/lib64/extractors/libmidiextractor.so
01-01 00:08:37.140 V 846 943 MediaExtractorFactory: registering extractor for MIDI Extractor
01-01 00:08:37.143 V 846 943 MediaExtractorFactory: registering sniffer for /system/lib64/extractors/libamrextractor.so
01-01 00:08:37.143 V 846 943 MediaExtractorFactory: registering extractor for AMR Extractor
01-01 00:08:37.154 V 846 943 MediaExtractorFactory: registering sniffer for /system/lib64/extractors/libmmparser.so
01-01 00:08:37.154 V 846 943 MediaExtractorFactory: registering extractor for QCOM Extractor
01-01 00:08:37.157 V 846 943 MediaExtractorFactory: registering sniffer for /system/lib64/extractors/libmp4extractor.so
01-01 00:08:37.157 V 846 943 MediaExtractorFactory: registering extractor for MP4 Extractor
这里分别有ogg、WAV、flac、MP3、MPEG2-PS/TS、Matroska、AAC、 MIDI、AMR、QCOM、MP4。
而在我当前的设备中，在/system/lib64下除了上述格式的extractor，还有一个不是特定格式的提取器实现模块libmmparser.so，这是高通平台自己自定义实现的媒体提取器解复用模块【mm-parser】，属于高通私有库实现。
RegisterExtractor(new ExtractorPlugin(getDef(), libHandle, libPath), pluginList) 实现分析：
此函数最里面是调用了getDef()方法即执行了每个提取器so库中对应于GetExtractorDef方法指针的实现，返回了so库提取器定义信息结构对象ExtractorDef。
ExtractorDef是so库提取器定义信息，结构体声明如下：
// [frameworks/av/include/media/MediaExtractorPluginApi.h]
struct ExtractorDef {
// 当前结构声明版本号
// version number of this structure
const uint32_t def_version;
// 当前名称提取器唯一标识符
// A unique identifier for this extractor.
// See below for a convenience macro to create this from a string.
media_uuid_t extractor_uuid;
// 当前名称提取器版本号，即有可能有两个so库提取器拥有相同唯一标识符（通常是名称相同但路径不同），
// 此时版本号大的将被使用。
// Version number of this extractor. When two extractors with the same
// uuid are encountered, the one with the largest version number will
// be used.
const uint32_t extractor_version;
// 可读的提取器名称
// a human readable name
const char *extractor_name;
union {
struct {
// 一个方法指针
SnifferFunc sniff;
} v2;
struct {
SnifferFunc sniff;
// 提取器支持的文件扩展类型（可以多种类型）
// a NULL terminated list of container mime types and/or file extensions
// that this extractor supports
const char **supported_types;
} v3;
} u;
};
再分析new ExtractorPlugin(getDef(), libHandle, libPath)，ExtractorPlugin媒体提取器插件类声明和定义：
该类是声明和定义一起实现的。该类其实就相同于一个自动管理提取器定义结构对象信息的内部指针释放问题。相当于一个智能指针类似功能。
105 struct ExtractorPlugin : public RefBase {
106 MediaExtractor::ExtractorDef def;
107 void *libHandle;
108 String8 libPath;
109 String8 uuidString;
110
111 ExtractorPlugin(MediaExtractor::ExtractorDef definition, void *handle, String8 &path)
112 : def(definition), libHandle(handle), libPath(path) {
113 for (size_t i = 0; i < sizeof MediaExtractor::ExtractorDef::extractor_uuid; i++) {
114 uuidString.appendFormat("%02x", def.extractor_uuid.b[i]);
115 }
116 }
117 ~ExtractorPlugin() {
118 if (libHandle != nullptr) {
119 ALOGV("closing handle for %s %d", libPath.c_str(), def.extractor_version);
120 dlclose(libHandle);
121 }
122 }
123 };
124 RegisterExtractor() 方法实现
173 void MediaExtractorFactory::RegisterExtractor(const sp<ExtractorPlugin> &plugin,
174 List<sp<ExtractorPlugin>> &pluginList) {
175 // sanity check check struct version, uuid, name
176 if (plugin->def.def_version == 0
177 || plugin->def.def_version > MediaExtractor::EXTRACTORDEF_VERSION) {
178 ALOGE("don't understand extractor format %u, ignoring.", plugin->def.def_version);
179 return;
180 }
181 if (memcmp(&plugin->def.extractor_uuid, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 16) == 0) {
182 ALOGE("invalid UUID, ignoring");
183 return;
184 }
185 if (plugin->def.extractor_name == NULL || strlen(plugin->def.extractor_name) == 0) {
186 ALOGE("extractors should have a name, ignoring");
187 return;
188 }
189 //for循环处理已缓存的so库插件信息列表【从头遍历】
190 for (auto it = pluginList.begin(); it != pluginList.end(); ++it) {
191 if (memcmp(&((*it)->def.extractor_uuid), &plugin->def.extractor_uuid, 16) == 0) {
192 // there's already an extractor with the same uuid
193 if ((*it)->def.extractor_version < plugin->def.extractor_version) {
194 // this one is newer, replace the old one
195 ALOGW("replacing extractor '%s' version %u with version %u",
196 plugin->def.extractor_name,
197 (*it)->def.extractor_version,
198 plugin->def.extractor_version);
199 pluginList.erase(it);
200 break;
201 } else {
202 ALOGW("ignoring extractor '%s' version %u in favor of version %u",
203 plugin->def.extractor_name,
204 plugin->def.extractor_version,
205 (*it)->def.extractor_version);
206 return;
207 }
208 }
209 }
往插件列表缓存中添加新的插件信息对象
210 ALOGV("registering extractor for %s", plugin->def.extractor_name);
211 pluginList.push_back(plugin);
212 }关于extractor的注册的一部分分析就先到这里，其实上述部分是通用的代码，而在我们的平台上，所有格式的解析都是使用高通的so。所以后续章节我们重点分析一下高通的媒体提取器解复用模块【mm-parser】。另外，关于每个so库提取器具体如何实现GetExtractorDef方法指针，我们也只是分析一下高通该功能的实现，其他so库实现方式都是类似的。
边栏推荐
- [must see for game development] 3-step configuration p4ignore + wonderful Q & A analysis (reprinted from user articles)
- Graphsage paper reading
- [reading this article is enough!!! Easy to understand] confidence level understanding (95% confidence level and confidence interval)
- FIFO最小深度计算的题目合集
- Topic collection of FIFO minimum depth calculation
- 山东大学项目实训之examineListActivity
- CCF 2013 12-4 interesting numbers
- Don't be afraid of xxE vulnerabilities: understand their ferocity and detection methods
- Box model
- call和apply和bind的区别
猜你喜欢
FPGA interview notes (IV) -- sequence detector, gray code in cross clock domain, ping-pong operation, static and dynamic loss reduction, fixed-point lossless error, recovery time and removal time
PHP laravel8 send email
Cenos7 builds redis-3.2.9 and integrates jedis
Matlab实现均值滤波与FPGA进行对比，并采用modelsim波形仿真
Who is stronger, zip or 7-Zip, and how to choose?
Moteur de modèle de moteur thymeleaf
Notes sur les questions d'entrevue de la FPGA (IV) - - détecteur de séquence, Code gris dans le domaine de l'horloge croisée, opération de ping - pong, réduction de la perte statique et dynamique, err
Record the first data preprocessing process
MMEditing中超分模型训练与测试
jenkins-用户权限管理
随机推荐
[IOS development interview] operating system learning notes
FPGA面試題目筆記（四）—— 序列檢測器、跨時鐘域中的格雷碼、乒乓操作、降低靜動態損耗、定點化無損誤差、恢複時間和移除時間
Simple understanding of pseudo elements before and after
End of 2021 graphics of Shandong University
Metasploitabile2 target learning
How to use the markdown editor
通过两种方式手写一个消息队列
Shandong University machine learning experiment 7 pca+ SVM face recognition
PHP laravel8 send email
MATLAB realizes mean filtering and FPGA for comparison, and uses Modelsim waveform simulation
Super (subclass)__ init__ And parent class__ init__ ()
URL in flask_ for
Analyze the capacity expansion mechanism of ArrayList
使用Meshlab对CAD模型采样点云,并在PCL中显示
FMT package usage of go and string formatting
Vulhub 8.1-backdoor vulnerability recurrence
This point of arrow function
Goodbye 2021 Hello 2022
What do you need to know about Amazon evaluation?
Print sparse arrays and restore