Monday, January 4, 2010

understanding the android media framework

android media framework is built on top of a set of media libraries, including OpenCORE, vorbis and sonivox. So one of goal of android media framework is to provide a consistent interface for all services provided by underlying libraries and make them transparent to users.

The figure below shows the dependency relationships between libraries of the media framework.

android_media_lib_dep

In this figure, green components are media libraries, yellow components are android internal libraries, grey components are external libraries, and the light blue class is the java consumer of the media framework. Except for android.media.MediaPlayer class, all components are implemented in c or c++.

The core of the media framework is composed of libmedia, libmediaplayerservice and libmedia_jni. Their codes reside in frameworks/base/media folder.

libmedia defines the inheritance hierarchy and base interfaces. It’s the base library.

libmedia_jni is the shim between java application and native library. First, it implements the JNI specification so that it can be used by java application. Second, it implements the facade pattern for the convenience of caller.

libmediaplayerservice implements some of concrete players and the media service which will manage player instances.


The figure below shows the class hierarchy.android_media_classes

This is a simplified version of the class hierarchy. Only some core classes are included. Classes in blue are defined in libmedia, classes in green are defined in libmediaplayerservice, classes in light yellow are defined in binder, which implements the IPC on android. And classes in grey are defined in other libs.

Note the BpInterface and BnInterface are template classes. Any instantiation of them also inherit the template argument INTERFACE as well.

In the class hierarchy diagram, though listed as a separate module, binder is actually implemented inside libutils component whose source code locate at /frameworks/base/libs/utils folder.


An interesting thing to note is in android, the application that intends to show the media content and the player that actually renders the media content run in different process. The red line in the sequence diagram below shows the boundary of two processes.

android_media_sequence

The figure shows three most common operations, creating a new player, setting datasource and playing. The last MediaPlayerBase object is the interface that MediaPlayerService::Client object uses to refer to the concrete player instance. The concrete player can be VorbisPlayer, PVPlayer, or any other player, depending on the type of the media to be played.

When an application creates a android.media.MediaPlayer object, it’s actually holding a proxy which can be used to manipulate the concrete player resides in the mediaserver process. During the whole procedure, two process communicates with Binder IPC mechanism.


Having knowledge above, it’s not difficult to understand why MediaPlayer doesn’t provide an API to use memory stream as source. Because the memory manipulated by the stream is in the address space of the application, and it’s not directly accessible by the mediaserver process.

References:

Google I/O, Mastering the Android Media Framework

android media framework uml diagram

25 comments:

Anonymous said...

thanks,
but pictures does not appear

rehab said...

thanks for your good post
i have a question if you can help me
i want to know which library does the class Canvas.cpp belong to?
does it belong to the same library (media framework)?
thanks for your help

rxwen said...

to Anonymous
Can you see pictures on other posts?
I can view them on different browsers.

rxwen said...

to rehab
Canvas belongs to the libandroid_runtime lib. It's part of graphic framework.
I think this topic deserves a separate post. :)

rehab said...

thanks alot for your quick reply
i just want to know the library because i made some changes in the Canvas class but can not update the phone with this changes
thanks again for your help
Rehab

Anonymous said...

i saw it now, it was browser problem
great post, thanks again

Rehab said...

Hi rxwen
i have another question

Android's structure:
Applications.
Application Framework.
Libraries.
Android Runtime.
Linux Kernel.
is this library(libandroid_runtime)
belongs to Libraries or Android Runtime ?

Thanks in advance,
Rehab

rxwen said...

to rehab
It corresponds to the Android runtime in this diagram:
http://developer.android.com/guide/basics/what-is-android.html

rehab said...

is the core library in the runtime written in java?
because it is in runtime, and in the diagram it has blue color
but the Canvas.cpp is a cpp class
do you have any reason for that?
thanks
Rehab

rxwen said...

Not all of them are implemented in java.
The blue block in the diagrm includes a set of core libs that comprises the foundation of java core library.

rehab said...

Thanks, i tried it and it worked :)
Rehab

Anonymous said...

nice. found the details that i was looking for. long live rxwen!

Dreamer said...

can you give more information about stream as source?
if you show the structure for the reception and reproduction of the stream, i appreciate.

Regards.

rxwen said...

dreamer,

I'm so busy with windows ce platform recently. So I'd like to write more about android streaming a bit latter.

Dreamer said...

ok, good work rxwen.

Regards.

Anonymous said...

Great diagram and valuable information! Could it be possible to wider the diagram to cover also the linux kernel part?

rxwen said...

Google has changed the media framework a lot. I'd like to create a new post to cover new media framework.

Raghu said...

Hi,
How difficult is it to compile the media player and the codecs in the android media framework as a separate media player.

rxwen said...

Can you be more specific what you want to achieve?
The mediaplayer itself is a standalone player that consumes the media framework

Anonymous said...

Hi,

Your article is very helpful. when are your posting new one based on recent framework? eager to have a look

Thanks

sangram said...

Hi rxwen,
Thanks for sharing such a valuable info.
I had one doubt.
Is it possible to integrate any external media player in the media framework like VLC player? If yes, how that can be done?

Thanks,
Sangram

Android app development said...

This is one of the forward post.I like your blog explanation.Thanks for your support.This is one of the special post.
Android app developers

Android developers said...

I have to i appreciate you for the efforts you get in writing this article. so you to promote your blog by using some internet marketing strategy, because it can easily to reach the peoples

yoyo said...

Can we have ActivityManager and WindowManager Class Hierarachy

acolin said...

Time for an update:

"...libmedia, libmediaplayerservice and libmedia_jni. Their codes reside in frameworks/base/media folder."

This has been moved to frameworks/av. See here for more details.