Understanding Timed Metadata in HLS Streams

Introduction

Timed metadata in HTTP Live Streaming (HLS) allows broadcasters to synchronize additional data with their audio and video streams. This data can include anything from ad markers and song information to subtitles and interactive content. In this tutorial, we'll explore what timed metadata is, how it works in HLS streams, and how to extract it using players on Android, iOS, and web environments.

What is Timed Metadata?

Timed metadata refers to data that is synchronized with specific points in time within a media stream. In the context of HLS, this metadata is embedded in the stream and can be used to trigger events or display information at precise moments during playback.

How Timed Metadata Works in HLS

In HLS, metadata is carried within a stream using ID3 tags or custom tags defined in the HLS specification. These tags are inserted into the stream at the encoder level and are then parsed by the player during playback.

ID3 Tags

ID3 tags are commonly used to carry metadata in MP3 files, but they can also be used in HLS streams. They are embedded into the transport stream (TS) files that make up the HLS stream.

Structure of ID3 Tags

An ID3 tag consists of a header followed by a series of frames, each frame holding a specific type of metadata. Here's a high-level overview of the structure:

• Header: The header is at the beginning of the ID3 tag and is used to identify the tag and provide information about it. The header includes the tag version (e.g., ID3v2.3 or ID3v2.4), the size of the tag, and flags that provide additional information about the tag's properties.
• Frames: Following the header, the tag contains one or more frames. Each frame has its own header that includes a frame ID, size, and flags. The frame ID is a four-character code that identifies the type of content the frame contains (e.g., TIT2 for the title, TPE1 for the artist). The size indicates the length of the frame's content, and the flags can signal special attributes or instructions for the frame.

Supported ID3 Tags

When working with HLS streams provided by Stingray, it's important to understand the specific ID3 tags that are supported and what information they convey. Here are the supported ID3 tags and their descriptions:

• TIT2 (Title): This tag holds the title of the audio or video track. It is a descriptive name given to the piece of content.
• TPE1 (Artist): This tag contains the name of the artist or performer who is responsible for the creation of the track.
• TALB (Album): The TALB tag is used to specify the album name from which the track is taken.
• TDRC (Recording Date): This tag denotes the year the track or album was officially recorded/released.
• TLEN (Length): This tag contains the total length of the track in milliseconds.
• TXXX (User Defined Text Information): This tag contains custom text information. In Stingray's streams, it holds comma-separated values including the track type (SONG, ADVERTISEMENT) and position information.
• LINK (Linked Information): The LINK tag is a container frame that can hold various types of linked information, including URL frames. In Stingray's streams, the WXXX frame nested inside the LINK tag is used to store the track unique ID.

Structure of LINK Tag and WXXX Frame

The WXXX ID3 tag is a user-defined URL link frame that can be used to store additional information about a track. The tag is nested within a LINK tag, thus the LINK tag acts as a container for the WXXX frame, providing a standardized method to encapsulate information. When parsing the metadata from an HLS stream, you'll need to look for the LINK tag and then extract the WXXX frame from it

Parsing ID3 Tags

Parsing ID3 tags involves reading the header to determine the tag version and size, then iterating over the frames to extract the metadata. Each frame must be read according to its specified size, and the content must be interpreted based on the frame ID. For developers working with ID3 tags in HLS streams, it's recommended to use a library or tool that can handle the parsing of these tags, as the binary format and potential for multiple versions and flags can make manual parsing complex.

For the LINK tag containing WXXX frames, additional parsing is required to extract the nested frame content. The WXXX frame within the LINK tag contains the asset ID that can be used for additional track information retrieval.

Extracting Timed Metadata

Most players have embedded functionalities to extract timed metadata. Refer to their documentations on how to do so

That being said, here's some quick example on different platform:

Android

On Android, you can use the ExoPlayer library to handle HLS streams and extract timed metadata.

1. Set Up ExoPlayer: Include ExoPlayer in your project by adding the necessary dependencies to your build.gradle file.
2. Create a Player Instance: Instantiate the player and prepare it with a MediaSource that represents your HLS stream.
3. Register a Metadata Output: Implement the MetadataOutput interface and register it with the player to receive metadata events.
4. Handle Metadata: In your MetadataOutput implementation, override the onMetadata method to handle the metadata as it's received.

player.addMetadataOutput(metadata -> {
    for (int i = 0; i < metadata.length(); i++) {
        Metadata.Entry entry = metadata.get(i);
        if (entry instanceof Id3Frame) {
            Id3Frame id3Frame = (Id3Frame) entry;
            
            // Handle standard ID3 frames
            if (id3Frame.id.equals("TIT2")) {
                String title = parseTextFrame(id3Frame.bytes);
                // Handle title
            } else if (id3Frame.id.equals("TPE1")) {
                String artist = parseTextFrame(id3Frame.bytes);
                // Handle artist
            } else if (id3Frame.id.equals("TALB")) {
                String album = parseTextFrame(id3Frame.bytes);
                // Handle album
            } else if (id3Frame.id.equals("TDRC")) {
                String year = parseTextFrame(id3Frame.bytes);
                // Handle recording date/year
            } else if (id3Frame.id.equals("TLEN")) {
                String totalLength = parseTextFrame(id3Frame.bytes);
                // Handle total length in milliseconds
            } else if (id3Frame.id.equals("TXXX")) {
                String customText = parseTextFrame(id3Frame.bytes);
                // Parse comma-separated values for type and position
                String[] values = customText.split(",");
                for (String value : values) {
                    if (value.contains("type")) {
                        String trackType = value.replace("type", "").trim();
                        // Handle track type (SONG, ADVERTISEMENT, MESSAGE)
                    } else if (value.contains("position")) {
                        String position = value.replace("position", "").trim();
                        // Handle position information
                    }
                }
            } else if (id3Frame.id.equals("LINK")) {
                // Parse LINK tag to extract WXXX frame
                String assetId = parseWxxxFromLinkFrame(id3Frame.bytes);
            }
        }
    }
});

iOS

On iOS, you can use AVPlayer to play HLS streams and AVPlayerItemMetadataOutput to extract timed metadata.

1. Create an AVPlayer Instance: Initialize an AVPlayer with the URL of the HLS stream.
2. Set Up Metadata Output: Create an instance of AVPlayerItemMetadataOutput and add it to the player's current item.
3. Handle Metadata: Implement the AVPlayerItemMetadataOutputPushDelegate protocol to receive metadata.

let metadataOutput = AVPlayerItemMetadataOutput(identifiers: nil)
metadataOutput.setDelegate(self, queue: DispatchQueue.main)
player.currentItem?.add(metadataOutput)

func metadataOutput(_ output: AVPlayerItemMetadataOutput, didOutputTimedMetadataGroups groups: [AVTimedMetadataGroup], from track: AVPlayerItemTrack?) {
    for group in groups {
        for item in group.items {
            // Handle metadata item
        }
    }
}

Web

On the web, you can use the HLS.js library to play HLS streams and listen for metadata events.

1. Include HLS.js: Add the HLS.js library to your web page.
2. Create an HLS Instance: Initialize HLS.js and attach it to a video element.
3. Listen for Metadata: Add event listeners for the Hls.Events.FRAG_PARSING_METADATA event to handle metadata.

if (Hls.isSupported()) {
    var video = document.getElementById('video');
    var hls = new Hls();
    hls.loadSource('your-stream.m3u8');
    hls.attachMedia(video);
    hls.on(Hls.Events.FRAG_PARSING_METADATA, function(event, data) {
        data.samples.forEach(function(sample) {
            // Parse the ID3 tags from the sample data similar to how it'S done on Android above.
            var metadata = parseId3Tags(sample.data);

            // Use the parsed metadata
            if (metadata.TIT2) {
                console.log("Title: " + metadata.TIT2);
            }
            if (metadata.TPE1) {
                console.log("Artist: " + metadata.TPE1);
            }
            if (metadata.TALB) {
                console.log("Album: " + metadata.TALB);
            }
            if (metadata.TRDC) {
                console.log("Release Year: " + metadata.TRDC);
            }
            if (metadata.TLEN) {
                console.log("Total Length: " + metadata.TLEN);
            }
            if (metadata.WXXX) {
                console.log("Asset ID: " + metadata.WXXX);
            }
            if (metadata.TXXX) {
                console.log("Type: " + parseTypeFromWxxx(metadata.WXXX));
                console.log("Position: " + parsePositionFromWxxx(metadata.WXXX));
            }
        });
    });
}

Conclusion

Timed metadata in HLS streams provides a powerful way to enhance the viewing experience with synchronized data. By using the appropriate tools and libraries for each platform, developers can extract and utilize this metadata to create interactive and informative applications. Remember to consult the player documentation for more detailed information and advanced usage.