How tsDemux Works — Demux MPEG-TS Streams Step by StepMPEG-TS (MPEG Transport Stream) is a container format widely used for broadcast systems, streaming, and recording of audio, video, and data. tsDemux is a tool or library for demultiplexing (demuxing) MPEG-TS streams — extracting individual elementary streams (video, audio, subtitles, metadata) from a multiplexed transport stream so they can be decoded, processed, or repackaged. This article explains how tsDemux works, the internal concepts behind demultiplexing MPEG-TS, and walks through a step-by-step process you can follow to demux streams reliably in both single-file and live (packetized) scenarios.
Overview: What is demultiplexing and why it matters
Demultiplexing is the process of separating combined streams into their original component streams. In MPEG-TS, multiple programs (each containing audio, video, and data) are interleaved into fixed-size packets (188 bytes typically). A demuxer must parse these packets, follow indices and tables (like PAT and PMT), and extract payloads for each elementary stream while preserving timing (PTS/DTS), continuity, and synchronization.
Key reasons to demux:
- Feed decoders with raw elementary streams (H.264/H.265 video, AAC/AC3 audio).
- Repackage streams into other containers (MP4, MKV, HLS).
- Analyze or edit specific streams (replace audio, add subtitles).
- Filter programs or tracks in multi-program transport streams (MPTS).
tsDemux performs these tasks while handling common challenges: packet loss, stream discontinuities, scrambled streams, and timing reconstruction.
MPEG-TS core concepts tsDemux relies on
Before diving into tsDemux internals, understand these MPEG-TS building blocks:
- TS packet: Fixed 188-byte packet. Begins with 0x47 sync byte. Contains header fields: PID (packet identifier), continuity counter, adaptation field flags, payload unit start indicator (PUSI), etc.
- PID: 13-bit identifier that labels packet stream type (e.g., video PID, audio PID, PAT PID = 0x0000).
- PAT (Program Association Table): Maps program numbers to PMT PIDs. Found on PID 0.
- PMT (Program Map Table): Lists PIDs for program’s elementary streams and their stream types (e.g., 0x1B = H.264).
- PES (Packetized Elementary Stream): Carries PES headers with PTS/DTS timing and the elementary stream payload (frames, access units).
- Continuity counter: 4-bit counter to detect lost or reordered packets for a PID.
- PCR (Program Clock Reference): Timing reference for clock synchronization (specific PID, present in adaptation field).
tsDemux parses these structures to route payload bytes into per-stream buffers and reconstruct PES packets.
Architecture of tsDemux
A typical tsDemux implementation contains these components:
- Input reader: Accepts TS packets from a file, network stream, or pipe. Validates sync and packet size.
- PID dispatcher: Routes packets to per-PID handlers based on header PIDs.
- Table parser: Parses PAT and PMT tables to discover program composition and stream types.
- PID handlers: For each active PID, maintain state (continuity counter, adaptation field info), assemble payloads, and detect PES packet boundaries.
- PES assembler: Reconstructs PES packets from payload fragments, extracts PTS/DTS, and forwards complete elementary stream packets to decoders or output sinks.
- Timing manager: Tracks PCR and PTS/DTS to present wallclock timestamps and correct jitter or discontinuities.
- Error handler/recovery: Detects lost packets (via continuity counter), resynchronizes on next PUSI or PAT/PMT, and optionally reports diagnostics.
Step-by-step demuxing flow
-
Input acquisition and sync
- Read 188-byte aligned blocks. Verify the first byte is 0x47. If not aligned, search forward for the next 0x47 and re-sync. In live streams, tolerate occasional misaligned packets but log and attempt resync quickly.
-
TS packet header parsing
- Parse header fields: payload_unit_start_indicator (PUSI), PID, adaptation_field_control, continuity_counter, transport_scrambling_control. If scrambling bits indicate encrypted content, handle accordingly or skip.
-
PID dispatching
- Use PID to look up an associated handler. If none exists:
- If PID == 0x0000, route to PAT parser.
- If PID known from PMT, create handler for that stream type (video/audio) with metadata like stream_type.
- Otherwise, create an “unknown PID” handler until PAT/PMT reveal its role.
- Use PID to look up an associated handler. If none exists:
-
Table parsing (PAT/PMT)
- When a packet with PID 0 or a PMT PID arrives with PUSI set, parse pointer_field and parse the PSI section.
- Validate CRC32 on PSI sections. Extract program->PMT PID mapping (from PAT) and stream_type->elementary_PID mapping (from PMT).
- Update PID handlers and inform downstream sinks of new streams or removed streams (useful for dynamic streams like DVB).
-
Adaptation field and PCR extraction
- If adaptation_field_control indicates presence, parse adaptation field. If PCR flag set, read PCR (base and extension) and update the timing manager. Use PCR to map PTS/DTS to wallclock and detect clock jumps or drift.
-
Continuity counter and error checking
- Verify continuity_counter for each PID increments modulo 16 for packets with payload. If mismatch:
- Mark potential packet loss.
- If loss occurs inside an active PES, consider flushing or rebuilding state based on stream type and error resilience.
- Wait for next PUSI to resynchronize PES assembly.
- Verify continuity_counter for each PID increments modulo 16 for packets with payload. If mismatch:
-
PES assembly and payload handling
- When payload contains PES data:
- If PUSI is set, and payload starts with PES start code (0x000001), start a new PES packet: parse PES header, read PES_packet_length, and extract PTS/DTS if present.
- Accumulate payload across multiple TS packets until PES packet is complete (based on PES_packet_length or next PES start).
- For video streams, collect access units (e.g., NAL units for H.264/H.265) while preserving boundaries and timestamps.
- Forward completed PES payloads with PTS/DTS to decoders or file writers.
- When payload contains PES data:
-
Timestamp handling and synchronization
- Use PCR as the master clock and PTS/DTS for frame timing. Map PTS to PCR-derived wallclock using: PTS_time = (PTS / 90000) seconds (for 90 kHz clock).
- Handle wraparound of 33-bit PTS/DTS and 42-bit PCR. Implement logic to detect and adjust for wrap events.
- For live streaming, smooth jitter using small buffer and clock recovery algorithms.
-
Output and repackaging
- Deliver elementary stream packets to the chosen output:
- Write raw elementary stream files (.h264, .aac).
- Feed decoder pipeline.
- Re-mux into MP4/Matroska/HLS segments — using timing and keyframe info to create segments aligned to access units.
- Deliver elementary stream packets to the chosen output:
-
Dynamic program changes and teardown
- Handle updated PAT/PMT tables during the stream (program additions/removals). Recreate or close PID handlers accordingly.
- On stream end, flush partial PES packets and write final metadata (e.g., segment durations, indexes).
Handling common practical challenges
- Packet loss and reordering: Use continuity counters and PUSI as recovery points. For moderate loss, drop incomplete PES and wait for next PES start. For heavy loss, trigger full rescan or request retransmission.
- Late or missing PAT/PMT: Some streams delay sending PMT. tsDemux should keep unknown PID handlers that buffer limited data until PMT arrives, then assign buffered data appropriately.
- Scrambled/encrypted streams: Detect transport_scrambling_control; if encrypted, either pass through to a descrambler or mark streams as unusable unless keys are available.
- Variable packet sizes (204 bytes in some systems): Detect and support nonstandard packet lengths by scanning for periodic sync bytes.
- Multiple programs (MPTS): Treat each program independently; allow selective demuxing of only requested program numbers to save CPU.
Example: demuxing an H.264 video + AAC audio program
- PAT identifies program 1 -> PMT PID 0x0100.
- PMT (on PID 0x0100) lists:
- Video stream_type 0x1B -> PID 0x0101 (H.264)
- Audio stream_type 0x0F -> PID 0x0102 (AAC)
- tsDemux creates handlers for PID 0x0101 and 0x0102.
- On PID 0x0101 packets, tsDemux assembles PES, extracts PTS, parses NAL units (using start codes or length-prefixed format depending on container), and writes .h264 elementary stream.
- On PID 0x0102 packets, assemble PES, extract ADTS or raw AAC frames and write .aac.
- PCRs extracted from adaptation fields are used to align PTS to real-time when repackaging into MP4 or creating HLS segments with correct timestamps.
Performance and optimization tips
- Buffering: Use ring buffers per PID sized for typical packet bursts; avoid unbounded buffering to prevent memory bloat.
- Zero-copy: Avoid copying TS payloads when routing between components; use references or memory slices.
- Multi-threading: Parse and dispatch packets in a producer-consumer model: one thread reads and validates TS packets, worker threads handle PID-specific assembly.
- Fast PAT/PMT parsing: Cache PMT parsing results and only re-parse sections when version_number changes.
- SIMD/optimized CRC: Offload CRC32 checks to optimized libraries or hardware where available.
Testing and validation
- Use sample MPTS and SPTS test streams covering:
- Different codec combinations (H.264, H.265, AAC, AC3, MPEG audio).
- Program changes and PMT updates.
- Scrambled streams (if supported).
- Packet loss and continuity counter errors.
- Validate output elementary streams by decoding them with ffmpeg/ffplay or using analyzers (e.g., tsduck, mediainfo).
- Verify timestamps: ensure monotonic PTS/DTS per stream and correct PCR-based mapping.
Conclusion
tsDemux demultiplexes MPEG-TS by parsing TS packet headers, using PAT/PMT to identify PIDs, assembling PES packets, and managing timing via PCR and PTS/DTS. Robust demuxing requires careful handling of continuity counters, adaptation fields, and dynamic program tables plus strategies for buffer management and error recovery. With proper design, tsDemux can reliably extract video, audio, and data streams for decoding, repackaging, or analysis in both stored and live-streamed environments.