Adding meta-data to video in iOS

Adding metadata to video delivered using the HTTP Live Stream protocol

Apple has proposed a standard for delivering both live and pre-recorded video over the web: HTTP Live Streaming. Whilst the name might suggest that this protocol is only relevant to the delivery of live video streams, it is equally relevant to delivering pre-recorded video on-demand. This post isn’t meant to be an introduction to HTTP Live Streaming, if you’re looking for an introduction, you might like to read the documentation that Apple makes available on their website.

One feature of the HTTP Live Streaming protocol that hasn’t received much attention yet is the support for including timed meta-data within the video stream. Perhaps it hasn’t received much attention due to the fact that Apple’s documentation doesn’t make it simple to figure out how to inject metadata in to a video stream. If you’re interested in the idea of incorporating meta-data in your video streams, this post will show you how to inject meta-data in to a stream and make use of it within an iOS application that receives the stream.

What do I mean by ‘injecting meta-data’?

Before I move on to ‘how’, let me briefly expand on what I mean by ‘injecting meta-data’ in to a HTTP Live Stream. A stream consists of a series of video segments. A video player (on a web page or in a native app) will download each of the segments as needed and combine them to create a seamless stream of video. Wouldn’t it be nice if we could annotate this stream of video with appropriate meta-data that the application playing the video could make use of during playback? For example, imagine a weekly TV show where a couple of reviewers discuss films they’ve seen that week. Wouldn’t it be nice if as you watched them discuss a film, you were offered some extra information about each of the actors/writers/directors they mentioned when they mentioned them. To do this, there would need to be some way of associating some meta-data (eg the names of actors and the URL of their profiles in the Internet Movie Database) with given points in the video as shown in ‘figure 1′.

Figure 1: Metadata in a HTTP Live Stream

As it turns out, there is support for just this sort of method of adding meta-data to Apple’s HTTP Live Streams. Whilst there are already numerous ways to achieve this sort of thing, the benefit of using the approach described in this post is that the meta-data will be embedded within the stream. The application playing the video doesn’t need to load data from two separate sources (video from one, meta-data from another) and doesn’t need to do anything to keep the two data streams in sync as the same data stream that contains the video also contains the meta-data about the video.

Creating some meta-data

The first step is to create some meta-data that you want to inject in to a video stream. For the purpose of this post, I’m going to add some meta-data to a short clip from the ABC’s (Australian Broadcasting Corporation’s) ‘At the Movies‘ programme that I alluded to earlier. ‘At the Movies‘ is a weekly programme where the hosts Margaret and David review movies that are currently screening in Australian cinemas. It might be nice to add meta-data to the video stream so that when Margaret or David mentions an actor or writer or director (which they do a lot), a viewer could tap their screen to look at the IMDB profile of that actor (for example). So, the first step is to identify what meta-data you want to add and when in the video stream you’d like to add it. In my case, I want to add the following actors’ names and links to their IMDB profiles when they’re first mentioned in the review.

Emma Stone, 68 seconds
Patricia Clarkson, 78 seconds
Stanley Tucci, 80 seconds
Alyson Michalka, 86 seconds
Thomas Haden Church, 149 seconds
Lisa Kudrow, 154 seconds
Will Gluck, 192 seconds

In order to inject this metadata in to the video stream, you first have to create files containing some ID3 tags with the metadata you want to inject at each point in time. You’ll need one file of ID3 tags for each separate moment you want to add metadata to. So in this case I’ll need 7 files. You can create these metadata files using the ‘id3taggenerator’ command-line tool included in the ‘HTTP Live Streaming tools‘ (download link) available from Apple’s website.

The syntax is:

id3taggenerator -o <output file> -artist <some artist> -url <some url>

So, to create the ID3 tags I want to inject in to my stream, I’d use the following commands:

id3taggenerator -o 1.id3 -artist "Emma Stone" -url "http://www.imdb.com/name/nm1297015/"
id3taggenerator -o 2.id3 -artist "Patricia Clarkson" -url "http://www.imdb.com/name/nm0165101/"
id3taggenerator -o 3.id3 -artist "Stanley Tucci" -url "http://www.imdb.com/name/nm0001804/"
id3taggenerator -o 4.id3 -artist "Alyson Michalka" -url "http://www.imdb.com/name/nm1425528/"
id3taggenerator -o 5.id3 -artist "Thomas Haden Church" -url "http://www.imdb.com/name/nm0002006/"
id3taggenerator -o 6.id3 -artist "Lisa Kudrow" -url "http://www.imdb.com/name/nm0001435/"
id3taggenerator -o 7.id3 -artist "Will Gluck" -url "http://www.imdb.com/name/nm0323239/"

Once you’ve created a bunch of files containing the meta-data you want to inject at different points in your video stream the next step is to create a ‘meta macro-file’ to specify which meta-data tags to inject when. This is just a plain-text file with the following format:

68 id3 /path/to/file/1.id3
78 id3 /path/to/file/2.id3
80 id3 /path/to/file/3.id3
86 id3 /path/to/file/4.id3
149 id3 /path/to/file/5.id3
154 id3 /path/to/file/6.id3
192 id3 /path/to/file/7.id3

Important points to note about this file:

• the numbers indicate the point in time you want to inject meta-data as a number of seconds from the start of the video,
• the fields are separated by spaces not tabs,
• the final line must be terminated by a newline character.

Injecting the meta-data in to a video stream

Now you’ve got a bunch of files containing the meta-data you want to inject at different points in your video and a file that indicates which metadata should be injected when, all that is left to do is to actually inject the meta-data in to the video stream. Oh, and to do something in your application if you encounter meta-data when playing the video. But I’ll get to that. First, let’s inject the meta-data in to the stream.

If you’re at all familiar with HTTP Live Streaming you’ll know that you can create a stream from a pre-recorded video file using the ‘mediafilesegmenter’ command-line tool. If you’re not familiar with HTTP Live Streaming, go and read the instructions then come back and continue reading this post.

If you’ve used ‘mediafilesegmenter’ before then adding meta-data to a stream is easy. You just have to add a single parameter to the command you use to segment your video.

Eg.

mediafilesegmenter
-base-url http://www.somedomain.com/path/to/video
-file-base /local/path/to/video
-index-file at_the_movies.m3u8
-meta-macro-file /path/to/macrofile.txt
/path/to/source/video.mp4

That’s it, your stream now contains meta-data.

Responding to meta-data during playback

The final step is to do something when you encounter meta-data in the video stream during playback. Now, according to Apple’s documentation you can access this meta-data in both native iOS apps and web apps as there is both an Objective-C and JavaScript API. However, I can’t find any documentation about the JavaScript API so I’m going to focus on native apps.

You can access the meta-data contained in a video stream by registering to receive notifications that are posted by an MPMoviePlayerController when it encounters meta-data in a video stream it is playing. More specifically, you should register for the MPMoviePlayerController’s MPMoviePlayerTimedMetadataUpdatedNotification.

Eg.

// Register for meta-data notifications
NSNotificationCenter *center = [NSNotificationCenter defaultCenter];
[center addObserver:self
           selector:@selector(metadataUpdate:)
               name:MPMoviePlayerTimedMetadataUpdatedNotification
             object:nil];

Then, in your ‘metadataUpdate:’ (or whatever you’ve called it) method, you ask the MPMoviePlayerController for its ‘timedMetadata’ and do something interesting with it. For example:

Actor *actor = [[Actor alloc] init];

if ([player timedMetadata]!=nil && [[player timedMetadata] count] > 0) {
	for (MPTimedMetadata *metadata in [player timedMetadata]) {
		if ([[metadata.allMetadata valueForKey:@"key"] isEqualToString:@"TPE1"]) {
			[actor setName:[metadata.allMetadata objectForKey:@"value"]];
		}
		if ([[metadata.allMetadata valueForKey:@"key"] isEqualToString:@"WXXX"]) {
			NSURL *url = [NSURL URLWithString:[metadata.allMetadata objectForKey:@"value"]];
			[actor setProfileURL:url];
		}
	}
}

// display some UI element for the user to interact with

I’m not entirely sure if the ‘foreach’ loop in the above example is needed as it isn’t clear to me what exactly is returned by the ‘timedMetadata’ method of MPMoviePlayerController. The documentation simply states that it returns ‘An array of the most recent MPTimedMetadata objects’ but who knows what ‘most recent’ means. Will it return all of the meta-data encountered so far? Just the very latest meta-data? or meta-data encountered in the last X seconds prior?

Likewise, the specific keys you look for in the MPTimedMetadata object that is returned will vary depending on the type of ID3 tags you injected in to the stream.

Demo

Finally, here’s a screen-cast showing an example application responding to some timed meta-data (you’ll have to wait until/skip to 68 seconds in to see the app respond to the meta-data).

About these ads