Troubleshooting · LL-HLS
Streamwake reliability agents

Troubleshoot low-latency HLS
without losing the live edge.

A working guide to the low-latency HLS failure patterns Streamwake catches on real delta playlists — chunked-transfer ingest stalls, partial-segment prefetch terminations, GOP timing drift, manufacturing-server cadence drift, playlist-delta size regressions, and blocking-playlist-request storms — written so an on-call engineer can read it next to a recent chunklist capture and act on it.

Protocol: HLS+LL
Format: delta + partial segments + chunked transfer
Streamwake probes: ingest-chunked · part-prefetch · preload hint · GOP alignment · manufacturing-server cadence · delta size · blocking storm ceiling.
Probes

What Streamwake checks

Five families of probes, each with a deterministic pass / fail / warn verdict that feeds the timeline. Every check has a name — that's the name you'll see on the agents feed.

Chunked-transfer ingest
ingest.chunked_transfer_stall
ingest · chunked
  • First-chunk TTFB p95 against the phased baseline — flags origin read-through stalls on Transfer-Encoding: chunked partials.
  • Chunk-completion p95 — surfaces stalled mid-PART writes that never finish a chunk before the next PART lands.
  • Pairs with manufacturing_server.cadence_drift so the stalled socket's downstream effect on the partial-segment emitter is visible.
Partial-segment prefetch
part.prefetch_termination · preload.can_preload_hint
partial · prefetch
  • Counts INDEPENDENT PARTs the player requested versus PARTs the player actually consumed — terminations mid-stream mark a stale byte stream.
  • Reports EXT-X-PRELOAD-HINT alignment with CAN-PRELOAD-BEFORE.
  • Flags PARTs marked INDEPENDENT=NO that the player attempted to prefetch anyway.
GOP timing
gop.boundary_aligned · encoder.cadence_part_compatible
gop · cadence
  • Every keyframe sits at the start of an EXT-X-PART — mid-PART IDR is the partial-segment equivalent of a re-buffer event.
  • Encoder cadence drift > ±0.05s fails the probe — the packager cannot honor PART-TARGET.
  • Replays trailing PARTs once cadence crosses the threshold.
Manufacturing-server cadence
manufacturing_server.cadence_drift
ll-hls server
  • Per-second count of how many PARTs the partial-segment emitter actually shipped, against the expected rate driven by PART-TARGET.
  • Hides behind a green playlist.delta_updates — the playlist is produced, the BYTES trail.
  • Diff between manufactured and expected rate at p95 — > 15% drift is a fail.
Delta vs blocking
playlist.delta_updates · playlist.delta_size · blocking_playlist.storm_ceiling
delta · blocking
  • _HLS_msn + _HLS_part match the next part on every probe cycle.
  • Delta bytes p95 within a threshold; > 1KB on a delta cursor is a regression that warns before a full BLOCKING request lands.
  • BLOCKING playlist requests per minute vs ceiling — flips to fail the moment the cohort loses confidence in the cursor.
Anatomy

An LL-HLS variant + delta window

A LL-HLS variant with PART-INF, SERVER-CONTROL declaring CAN-BLOCK-RELOAD + PART-HOLD-BACK + CAN-PRELOAD-BEFORE, and a segment of 16 PARTs (one of them an off-cycle 0.660s part, one a 0.990s part — encoder cadence slipping). The blocking-replay trace on the right shows what the server returns when delta should have fired. The probes→fields map links each verdict to a line in the playlist.

LL-HLS variant (PART-INF + SERVER-CONTROL + 16 PARTs)
#EXTM3U
#EXT-X-VERSION:9
#EXT-X-TARGETDURATION:6
#EXT-X-PART-INF:PART-TARGET=0.33
#EXT-X-SERVER-CONTROL:CAN-BLOCK-RELOAD=YES,PART-HOLD-BACK=1.0,HOLD-BACK=4.0,CAN-PRELOAD-BEFORE=YES
#EXT-X-MEDIA-SEQUENCE:1470
#EXT-X-MAP:URI="init.mp4"
#EXTINF:6.000,
seg1470.m4s
#EXT-X-PROGRAM-DATE-TIME:2026-08-16T11:02:10.000Z
#EXT-X-PART:DURATION=0.330,URI="seg1470.0.m4s",INDEPENDENT=YES
#EXT-X-PART:DURATION=0.330,URI="seg1470.1.m4s"
#EXT-X-PART:DURATION=0.330,URI="seg1470.2.m4s"
#EXT-X-PART:DURATION=0.330,URI="seg1470.3.m4s"
#EXT-X-PART:DURATION=0.330,URI="seg1470.4.m4s"
#EXT-X-PART:DURATION=0.330,URI="seg1470.5.m4s"
#EXT-X-PART:DURATION=0.660,URI="seg1470.6.m4s"
#EXT-X-PART:DURATION=0.330,URI="seg1470.7.m4s"
#EXT-X-PART:DURATION=0.330,URI="seg1470.8.m4s"
#EXT-X-PART:DURATION=0.330,URI="seg1470.9.m4s"
#EXT-X-PART:DURATION=0.330,URI="seg1470.10.m4s"
#EXT-X-PART:DURATION=0.330,URI="seg1470.11.m4s"
#EXT-X-PART:DURATION=0.330,URI="seg1470.12.m4s"
#EXT-X-PART:DURATION=0.330,URI="seg1470.13.m4s"
#EXT-X-PART:DURATION=0.330,URI="seg1470.14.m4s"
#EXT-X-PART:DURATION=0.990,URI="seg1470.15.m4s"
#EXTINF:6.000,
seg1471.m4s
#EXT-X-PROGRAM-DATE-TIME:2026-08-16T11:02:16.000Z
#EXT-X-PRELOAD-HINT:TYPE=PART,URI="seg1472.0.m4s"
Blocking-replay trace (delta should have fired)
GET /live/event/llhls-variant.m3u8?_HLS_msn=1471&_HLS_part=4
HTTP/2 200
content-type: application/vnd.apple.mpegurl
date: Sat, 16 Aug 2026 11:02:17.412Z
cache-control: max-age=0
transfer-encoding: chunked
server-timing: _HLS_msn=1471; _HLS_part=4; part_drift_ms=212
x-cache: MISS

# delta window should have returned under 200B; instead the server
# shipped 4,920 B — a full BLOCKING reload triggered by a chunked-transfer
# chunk-drop on seg1470.6.m4s. The 212ms part_drift_ms is the cue.
Probes → playlist fields
Quick map from probe verdict to the line you should pull.
  • ingest.chunked_transfer_stall → first chunk TTFB + chunk-completion p95
  • part.prefetch_termination → ratio of INDEPENDENT=YES PARTs the player consumed vs requested
  • gop.boundary_aligned EXT-X-PART start times vs GOP keyframe timestamps
  • manufacturing_server.cadence_drift → shipped PARTs/sec vs EXT-X-PART-INF PART-TARGET implied rate
  • playlist.delta_size → response body bytes with _HLS_msn + _HLS_part cursor
  • blocking_playlist.storm_ceiling → full BLOCKING GETs per minute vs ceiling

The EXT-X-PART-INF:PART-TARGET=0.33 line tells the player where parts will land, EXT-X-SERVER-CONTROL opens the BLOCKING channel and gates PART-HOLD-BACK + CAN-PRELOAD-BEFORE, and every EXT-X-PART entry is a chunked partial the player can prefetch out of order — flagged with INDEPENDENT=YES when it can stand on its own.

The blocking-replay trace shows the regression in flight: the client had a delta cursor in _HLS_msn=1471, _HLS_part=4, and asked for a delta. The server returned 4,920 bytes — a full BLOCKING window — because the partial-segment emitter stalled on seg1470.6.m4s (the off-cycle 0.660s part). playlist.delta_size flips to warn, and once the cohort sees enough of those warnings, blocking_playlist.storm_ceiling flies.

The agent never reads the bytes itself — it probes what the playlist tells it the bytes would be, and then samples the cadence to look for the regression. The probes→fields map on the left pairs each verdict with one specific element in the playlist or in the delta response, so the line to grep for in a chunklist capture is right there.

Why delta stays < 1KB
LL-HLS deltas are supposed to be tiny. A regression is loud.

A delta response is the new + last-played MEDIA-SEQUENCE worth of PARTs — typically a single new EXT-X-PART line plus the trailing EXT-X-PRELOAD-HINT. A BLOCKING response is the entire served window back to the playback head. The probe counts bytes: > 1KB on a delta cursor with part_drift_ms> 0 means the server failed to honor CAN-BLOCK-RELOAD.

Failure modes

Six ways LL-HLS falls off the live edge

Each row: symptom the agent reports → underlying cause → a fix that holds under the next probe cycle. The probe names are what you'd grep for in the agents feed.

01Chunked-transfer ingest stall on partial segments
failure mode
Symptom

First chunk TTFB hovers at 600ms+ versus a 280ms baseline; chunk-completion p95 climbs into the 1.5–2.0s range. Viewers see a working player that suddenly pauses mid-PART, then jumps forward when the chunk drops land. Live edge creeps from the target 2–3s glass into an 8–10s glass.

Root cause

The ingest / packaging pipeline publishes the partial segments with Transfer-Encoding: chunked but the first chunk stalls behind a CDN edge read-through or an origin hop. The player can fetch the next PART URI but the body bytes arrive late; manufacturing_server.cadence_drift compounds because the chained partial-segment emitter is waiting on the same stalled socket.

Fix

Pin the ingest path to a hold-connection transport, push the manufacturing server onto a dedicated egress so a CDN edge cold-pull cannot stall partial chains, and turn on preemptive PART replay when ingest.chunked_transfer_stall trips a chunk-completion p95 > 1.2s for more than one cadence. Streamwake auto-rebalances the egress preference away from the saturated edge within ~30s of the verdict.

02Partial-segment prefetch termination — the client drops mid-PART
failure mode
Symptom

Across the run the agent reports 2 of 8 INDEPENDENT PARTs terminated mid-stream. The client decided the prefetch window was unhealthy, dropped the prefetch, and the player visibly downshifted to a safer buffering posture. Several cohort members see a small rebuffer right at the prefetch drop.

Root cause

preload.can_preload_hint is still passing at the playlist level, but the BYTE STREAM of the PART never completes. Either ingest.chunked_transfer_stall has a dropped chunk that the client cannot recover from, or the manufacturing server never finished publishing the late-stage parts in the prefetch window and the server returns stale bytes the player will not trust.

Fix

Emit the full PART only once manufacturing has acknowledged publish; mark any PART that is mid-publish with the INDEPENDENT=NO attribute and have the player skip the prefetch on those parts. part.prefetch_termination should fall to zero once the publish path is consistently committing ahead of the prefetch window.

03GOP timing drift — mid-PART keyframes re-buffer the player
failure mode
Symptom

PART-DURATION hits 0.660s or 0.990s against a declared PART-TARGET=0.330s. Every off-cycle PART needs a partial append mid-GOP, and the player rolls forward to the next IDR — burning ~250ms of buffer each cycle. Buffer math goes back-of-envelope wrong; ABR descends the ladder.

Root cause

The encoder is emitting 2.5s GOPs under load instead of the 2.0s GOP locked against PART-TARGET=0.33. The packager splits on inference and the parts occasionally start mid-frame, which is exactly the signature encoder.cadence_part_compatible flags. The packager cannot honor PART-TARGET without a stable GOP underneath.

Fix

Lock the encoder cadence so GOP duration divides cleanly into TARGETDURATION (PART-TARGET=0.33 with GOP=2.00 → 6 parts per GOP). The agent replays the trailing partials the moment cadence crosses the ±0.05s threshold; gop.boundary_aligned flips to pass when every PART starts with an IDR.

04Manufacturing-server cadence drift — publish path falls behind the encoder
failure mode
Symptom

agent timeline shows manufacturing_server.cadence_drift fail with manufactured_parts_per_sec_p95 = 1.7 against an expected 3.03 (one part every PART-TARGET). The playlist still lines up; the player has not panicked yet — but the publish path is consistently ~430ms behind the encoder.

Root cause

The LL-HLS packaging server (the manufacturing server) is responsible for chopping the encoded chunks into PARTs and emitting them to the playlist + CDN. Under load the packaging worker pool saturates, the partial-segment publisher stops honoring its SLA, and the playlist cursor stays ahead of the published media. The playlist.delta_updates probe stays green because the playlist is being produced — but the BYTES it references trail by hundreds of milliseconds.

Fix

Scale the manufacturing worker pool to keep manufactured_parts_per_sec_p95 inside ±15% of expected_parts_per_sec, and prefetch the next GOP from the encoder into packaging memory any time cadence drift exceeds ±10%. manufacturing_server.cadence_drift should flip back to pass on the next cadence window once the pool recovers.

05Playlist-delta size regression — a delta that should be 80 bytes returns 4,920
failure mode
Symptom

playlist.delta_updates stays green (deltas honored) but playlist.delta_size flips to warn. The agent timeline p95 for delta bytes lingers at 4,920 on a single probe cycle — most deltas are still ~80 bytes but the over-fetch ratio is climbing. Bandwidth on the live edge roughly doubles inside that window.

Root cause

A BLOCKING playlist reply slipped into the deltas — usually because ingest.chunked_transfer_stall dropped a partial chunk, the server failed to honor the cursor, or the manufacturing-server stalled and the playlist writer returned the full window the player did not need. The player treats the over-fetched delta as a full reload; CDN cache fill rules no longer apply.

Fix

Throttle any delta reply that returns over the delta_size_threshold_b to a single orchestrating client and emit a small delta only, and pair playlist.delta_size with manufacturing_server.cadence_drift so the over-fetch root cause is visible. Streamwake auto-throttles when delta bytes p95 exceeds the threshold for more than two cadences.

06Blocking-playlist-request storm — the cohort flips to a full reload
failure mode
Symptom

blocking_playlist.storm_ceiling fails. Over a 60s window the player cohort issued 18 full BLOCKING playlist GETs against a 4-per-minute ceiling. The CDN edge misses stack up; egress saturates; the cohort is now paying 4–5KB per playlist pull when ~80B was sufficient.

Root cause

Once playlist.delta_size has been failing for several cycles the cohort loses confidence in the delta cursor — every client correlates the bad delta with a refresh and issues a full BLOCKING reload. The failures feed on each other: more BLOCKING reloads produce more cache misses, cache misses inflate the BLOCKING budget, and the storm takes off.

Fix

Inject a server-side cap on BLOCKING reloads per cohort (force re-establish delta cursor on a 410 Gone-style stronger signal), and pair the cap with playlist.delta_size so a delta regression becomes a single coordinated rebalance instead of a storm. Streamwake routes the cohort to a healthier egress and forces a delta cursor replay within a single cadence.

Agentic-ops layer

Heuristics: how the agent loop classifies the incident

The probe families above produce verdicts. Three rules in the agentic-ops layer turn a stream of verdicts into an incident classification — and pair the verdict with the smallest-safe auto-rebalance — without a human reading the timeline.

Chunked-transfer TTFB drift
ingest.chunked_transfer_stall → ingest + manufacturing-server chain
heuristic
  • Rule: first-chunk TTFB p95 across the last N probes vs the phased baseline for the same cohort + export region; chunk-completion p95 above 1.2s for more than one cadence.
  • What it surfaces: an ingest / CDN edge read-through stall that drags the partial-segment emitter down with it. Latency creeps from 2–3s to 8–10s before any of the byte probes notice.
  • Agent does next: auto-rebalance the egress preference away from the saturated edge within ~30s of the verdict, raise probe cadence for this stream, prepend to the alert feed, and tag an incident "ingest_chain_pressure" — actionable before any bitrate ladder starts slipping.
Partial-segment prefetch termination
part.prefetch_termination → manufacturing-server + cursor signal
heuristic
  • Rule: ratio of consumed vs requested INDEPENDENT PARTs over the last N probes; any mid-stream termination that exceeds 5% of the prefetch window.
  • What it surfaces: a manufacturing-server stall the playlist itself never flagged, because the playlist was still produced. The client is the only observability that catches it before the ear.
  • Agent does next: meta-classify as "byte_stream_pressure", replay the trailing PARTs that were terminated, and notify the publish path to throttle — the player will converge on a healthy prefetch window within one cadence.
Blocking-playlist-storm ceiling
blocking_playlist.storm_ceiling → cohort-loss-of-confidence signal
heuristic
  • Rule: blocking playlist requests per minute vs ceiling; double-or-more spikes after one delta_size warn cycle flip the verdict.
  • What it surfaces: a cohort that has lost confidence in the delta cursor — the moment a few bad deltas are correlated with refresh bumps, every client issues a full reload.
  • Agent does next: route the cohort to a healthier egress, force a delta cursor replay, and inject a single coordinated remediation that quenches the storm rather than absorbing more bandwidth — closing the loop that a passive monitoring pipeline cannot.
Why this reduces MTTR
Active classify → rebalance → verify vs humans-on-pager

The failure-mode rows above trace back to the three heuristics: chunked-transfer ingest stalls and manufacturing-server cadence drift both surface as elevated first-chunk TTFB + PART termination ratios; partial-segment prefetch drops map onto the manufacturing-server chain dragging the prefetch window along with it; and playlist-delta size regressions funnel into a blocking-playlist storm the moment the cohort loses confidence in the cursor. A passive-monitor logs the same verdicts and waits for a rebuffer report to fire — by which point you're already paying the cohort-trust cost and writing the postmortem. The agent loop flips the polarity: classify the verdict under one of the three heuristics, rebalance by re-routing the egress preference away from the saturated edge or coordinating a replay of the trailing PARTs, then verify that the next probe cycle clears the cohort off the BLOCKING storm. Each heuristic closes a loop that a passive monitoring pipeline cannot.

Diagnose

Diagnose with Streamwake

Register the LL-HLS source against POST /api/v1/streams with protocol: HLS+LL, then read the agent timeline back through GET /api/v1/agents. The probe verdicts in the timeline are exactly the rows above.

The curl below registers an LL-HLS source URL and asks for a 30-second probe cadence. The HLS+LL protocol routing is what makes the agent run the ingest-chunked, partial-prefetch, preload-hint, GOP-alignment, manufacturing-server, delta-size, and blocking-storm probes on every rebuild.

The cookie is the same better-auth.session_token that gates every /api/v1/* call — see the auth guide for how to mint one.

Once the stream is registered, the agents endpoint returns the per-probe verdicts below. The order of checks mirrors the probe families in the section above — ingest.chunked_transfer_stall first, then prefetch / preload / GOP / manufacturing-server, with the delta + blocking probes last because those are the ones that close the loop with the cohort.

POST /api/v1/streams
curl -X POST https://streamwake.polsia.io/api/v1/streams \
  -H "content-type: application/json" \
  -b "better-auth.session_token=<your-session-cookie>" \
  -d '{
    "sourceUrl": "https://cdn.example.com/live/event/llhls-variant.m3u8",
    "protocol": "HLS+LL",
    "probeIntervalSeconds": 30,
    "agents": [
      "ingest.chunked_transfer_stall",
      "part.prefetch_termination",
      "preload.can_preload_hint",
      "gop.boundary_aligned",
      "encoder.cadence_part_compatible",
      "manufacturing_server.cadence_drift",
      "playlist.delta_updates",
      "playlist.delta_size",
      "blocking_playlist.storm_ceiling"
    ]
  }'
GET /api/v1/agents?stream_id=…
curl https://streamwake.polsia.io/api/v1/agents?stream_id=<id> \
  -b "better-auth.session_token=<your-session-cookie>"
Agent timeline response (trimmed)
{
  "stream_id": "cklivellhlsstorm712",
  "source": "https://cdn.example.com/live/event/llhls-variant.m3u8",
  "protocol": "HLS+LL",
  "playlist_mode": "delta",
  "checks": [
    {
      "probe": "ingest.chunked_transfer_stall",
      "result": "warn",
      "ttfb_ms_p95": 612,
      "chunk_completion_ms_p95": 1820,
      "detail": "Transfer-Encoding: chunked; first chunk TTFB p95 612ms vs 280ms baseline; manufacturing server stalled on seg1470.6.m4s (chunk drop)"
    },
    {
      "probe": "part.prefetch_termination",
      "result": "fail",
      "terminated_parts": 2,
      "prefetched_parts": 6,
      "detail": "2 of 8 INDEPENDENT PARTs terminated mid-stream — client dropped the prefetch window after seg1470.6.m4s"
    },
    {
      "probe": "preload.can_preload_hint",
      "result": "pass",
      "detail": "CAN-PRELOAD-BEFORE=YES in SERVER-CONTROL; EXT-X-PRELOAD-HINT TYPE=PART present for seg1472.0.m4s"
    },
    {
      "probe": "gop.boundary_aligned",
      "result": "warn",
      "gop_period_s": 2.0,
      "part_count_per_gop": "5–7",
      "detail": "2 of 16 PARTs start mid-GOP — encoder cadence slipped to ~2.5s for two cycles; seg1470.6.m4s and seg1470.15.m4s are the off-cycle parts"
    },
    {
      "probe": "encoder.cadence_part_compatible",
      "result": "fail",
      "detail": "encoder GOP slipped to ~2.5s for two cycles; PART-DURATION jitter to 0.660 and 0.990 — PART-INF PART-TARGET=0.330 cannot be honored"
    },
    {
      "probe": "manufacturing_server.cadence_drift",
      "result": "fail",
      "manufactured_parts_per_sec_p50": 2.4,
      "manufactured_parts_per_sec_p95": 1.7,
      "expected_parts_per_sec": 3.03,
      "detail": "partial-segment manufacturing cadence dropped to 1.7 parts/sec at p95; the publish path fell behind the encoder by ~430ms on the captured window"
    },
    {
      "probe": "playlist.delta_updates",
      "result": "pass",
      "delta_responses": 31,
      "delta_bytes_p50": 78,
      "delta_bytes_p95": 212,
      "detail": "32/32 deltas honored _HLS_msn + _HLS_part cursor"
    },
    {
      "probe": "playlist.delta_size",
      "result": "warn",
      "delta_bytes_p95": 4920,
      "delta_size_threshold_b": 1024,
      "detail": "delta returned 4920B on 1/32 probes — chunked-transfer drop + manufacturing-server stall pushed the server into a BLOCKING-window delta reply"
    },
    {
      "probe": "blocking_playlist.storm_ceiling",
      "result": "fail",
      "blocking_requests_per_min": 18,
      "blocking_ceiling_per_min": 4,
      "detail": "blocking playlist requests spiked to 18/min after the chunked-transfer stall; clients lost the delta cursor and fell back to full BLOCKING reloads"
    },
    {
      "probe": "llhls.part_holdback_drift",
      "result": "pass",
      "detail": "PART-HOLD-BACK matches delta cursor; CAN-PRELOAD-BEFORE aligned"
    }
  ]
}
Pair

Read the HLS / MPEG-DASH pair

The low-latency arm of the protocol-by-protocol guide pair. The HLS guide covers the shared master / variant / tag-continuity probes; the LL-HLS optimization guide covers the part-cadence / delta-vs-blocking / preload / GOP-alignment tuning lane; the MPEG-DASH guide covers the DASH-shaped counter-parts.

Next step

Want Streamwake to catch this on its own?

Sign up, register an LL-HLS source with protocol: HLS+LL, and the same ingest-chunked / prefetch / preload / GOP / manufacturing-server / delta-size / blocking-storm probes that produced the timeline above run on every cadence — and surface in a Slack channel, a webhook, or the streams dashboard.

Open the streams dashboard
Auth-gated · reads the timeline the agent wrote on every probe cadence.
  • Stream list reads from GET /api/v1/streams; per-stream timeline from GET /api/v1/agents.
  • Probe verdicts stream into the dashboard within one cadence interval — manual curl not required.
  • Self-serve signup at /sign-up — no sales call required for the first stream.
Need Streamwake on one of your incidents?
Would you like Streamwake to analyze one of your historical incidents and show where AI could reduce investigation time? (Filed under: LL-HLS troubleshooting.)
Incident analysis
  • Pick a recent on-call incident — manifest stall, edge miss, player-side stall, or peer congestion.
  • We replay it through the same reliability-agent probe cascade used on the postmortem above.
  • You walk away with a written what-could-have-been-Automated readout, not a sales deck.