Origin-shield queue saturation
on a marquee live-event spike.
A working postmortem on a weekday primetime live-event spike — a sports final or a keynote — that fanned concurrent viewer connection demand into the CDN origin-shield queue past its pre-provisioned ceiling. The shield's queue_depth_concurrent pegged to 1.00 of the slot, queue_p99_wait_ms climbed past the cohort's expected join-window profile, origin.replay_concurrency_ceiling_pct sat at 1.00 on the affected tier, while edge.egress_kbps held at-or-above expected — the cohort was delivering traffic, the shield queue was the bottleneck. The Streamwake agentic ops layer caught it from viewer-impact telemetry, ranked the failure origin_shield_queue_saturation at 86% confidence, and remediated with either a shield pull / ceiling expansion (autonomous under a confidence-and-cohort-divergence gate) or a viewer-side reconnect-throttle (surfaced to humans) — recovery is verified viewer-side, not queue-side.
Book a technical demo for Origin shield saturation
Read the postmortem — then bring your own incident to Streamwake.
Two ways to engage on this exact failure pattern: book a 30-minute technical demo where we walk through the probe cascade on your source, or hand us an archived incident and watch the agent diagnose it end-to-end.
Both routes land on the scoping intake form — no SDR gate.
What the cohort saw
The first things to read on any real marquee viewer-spike incident are the cohort-level numbers on the late-join cohort — how many sessions sat behind the shield queue, how long the queue tail ran above the slot budget, and where the symptom landed relative to the edge egress profile. Three numbers did the heavy lifting here. The figures below are simulated telemetry — the disclosure above applies to every figure on this page.
Roughly 14% of the na-east marquee cohort — session-level late-join stalled behind the shield queue on the affected cdn-A/atl01 tier, where cdn_shield.queue_depth_concurrent pegged at 1.00 of the slot ceiling and queue_p99_wait_ms climbed to 4.62s. The cohort was delivering traffic — edge.egress_kbps sat at 8642 Kbps, above the expected rate — but the reader couldn't get through the shield's backoff curve on first join, dragging time-to-first-frame and pushing cohort_join_stall_ratio to 0.142 over a 90-second window before the shield pull landed.
~36 minutes between the first shield queue variance at T+0m and cohort_join_stall_ratio clearing the baseline tolerance at T+22m. The postmortem window goes longer because the cohort-side reconnect-throttle ships configured for the next marquee broadcast window — not in the same incident window. Within the simulated window, the shield pull + ceiling expansion alone cleared the viewer-side close-out signals by T+22 m; the throttle closes the lane forward, not backward.
queue_p99_wait_ms: 4.62 on shield-NA-3 (na-east marquee cohort) — the shield tier queued behind the pre-provisioned ceiling, with queue_depth_concurrent → 1.00 and origin.replay_concurrency_ceiling_pct → 1.00. The discriminator is that edge egress stayed green — the cohort was delivering traffic; the symbol of failure was the queue slot, not the egress profile.
How Streamwake classified this incident
Three ranked hypotheses, with the top one filing the timeline and the two lower hypotheses dismissed by name to pin the lane. The edge_pop_cache_miss_amplification lane is ruled out because the warm edge POP on the same cohort reads pass on cache-hit, and the encoder_bandwidth_regression lane is ruled out because encoder profile probes read clean and origin egress is within profile. The symptom was on the shield queue leg of the affected tier, not on a cache-locale amplifier and not on an encoder-side regression that would have failed adjacent tier cohorts on the same encoder.
origin_shield_queue_saturation — a marquee live-event spike fanned concurrent viewer connection demand into the shield tier past its pre-provisioned ceiling; the shield's queue stopped accepting fan-in and the late-join cohort got stuck behind the back-off curve. Three signals line up: cdn_shield.queue_depth_concurrent 1.00 vs 0.41 pre-peak baseline,cdn_shield.queue_p99_wait_ms 4.62s vs 0.21s baseline (exceeds the 1.5s slot budget by ~3×),origin.replay_concurrency_ceiling_pct 1.00 on the affected tier (replay-origin playing out the queue at ceiling).
edge_pop_cache_miss_amplification was the secondary signal ranked at 41% — the warm edge POP on the same cohort reads pass on cache-hit; cdn_shield.segment_leg_cache_hit is pass; the affected POP and the warm POP on the same cohort have identical cache-hit posture, which is the discriminator for the shield-queue shape of the failure. The dismissal rule was "rank the cause on the broader probe pattern, not on the single symptom that landed on the player"; the shield-queue nature of the failure is the decisive signal pattern.
- Region: na-east (atl01 marquee shield tier)
- Status: resolved (window closed)
- Opened: 2026-08-13 19:54 UTC
- Spread: contained to the marquee na-east tier — apac and eu-west marquee cohorts unaffected; adjacent non-marquee cohorts on the same vendor reads at baseline.
Above the 80% threshold the agent treats as a confident top-hypothesis filing. encoder_bandwidth_regression · 0.18 was cleared explicitly because encoder.bitrate_target_delta_pct reads within baseline and origin.replay_orig_egress_kbps reads stable — the encoder could not be the regression if egress is shaped correctly.
Incident timeline
Ten events: detection on the cohort, classification across three ranked hypotheses, four autonomous acts the agent took under the governed-action gates, three acts it surfaced to humans, the recovery probe, and the resolution. The right-hand "act" tag is what makes this postmortem distinct from a generic write-up — it pins the split between autonomous agentic ops and the work that genuinely needed a person. All times below are simulated telemetry — the disclosure at the top of this page applies to every minute offset on the timeline.
Today
- T+0mDetectionby cohort agent · na-east (atl01 shield tier)act · autonomous
Shield queue depth climbing; replay-origin concurrency ceiling hit
cdn_shield.queue_depth_concurrent crossed 0.86 then pegged to 1.00 of the slot ceiling within a 90-second window on the cdn-A/atl01 cohort; cdn_shield.queue_p99_wait_ms climbed from 0.21s to 4.62s (exceeds the 1.5s slot budget by ~3×); origin.replay_concurrency_ceiling_pct read 1.00; cohort.cohort_join_stall_ratio lifted from 0.018 baseline to 0.142 in the same window; cohort.reconnect_storm_ratio reads 1.00 — the cohort reconnect fan-in matches the shield fan-in shape exactly.
Aug 13, 07:54:42 PM - T+1mClassificationby Streamwake reliability agentact · autonomous
Ranked: origin_shield_queue_saturation (0.86) · secondary edge_pop_cache_miss_amplification (0.41) · encoder_bandwidth_regression (0.18)
Top hypothesis reads 86% confidence. Three ranked alternatives dismissed: edge_pop_cache_miss_amplification (the warm edge POP on the same cohort reads pass on cache-hit — discriminator for the shield-queue nature of the failure), encoder_bandwidth_regression (encoder profile probes read clean and origin egress is within profile — discriminator for the shield-queue shape, not encoder-shape).
Aug 13, 07:55:42 PM - T+2mAutomated actionby Streamwake reliability agentact · autonomous
Governed · autonomous: pull_origin_shield_to_expanded_tier
Confidence 0.86 ≥ 0.80 gate cleared; cohort divergence active on cohort.cohort_join_stall_ratio (0.142 vs 0.018 baseline). The shield tier is pulled into the expanded replay-origin tier so the slot can absorb the queue tail within the cohort window — the affinity pin between marquee broadcast and the affected tier is preserved on the expanded tier.
Aug 13, 07:56:42 PM - T+4mAutomated actionby Streamwake reliability agentact · autonomous
Governed · autonomous: raise_cdn_shield_queue_concurrency_ceiling
Confidence 0.86 ≥ 0.80 gate cleared; replay_concurrency_ceiling_pct == 1.00 gate cleared. The shield slot ceiling is raised against the cohort marquee window profile with reasoned bounds; the slot has headroom again inside the affected tier window — replay concurrency re-anchors below 1.00 on the same cohort cadence.
Aug 13, 07:58:42 PM - T+7mStatus changeby Streamwake reliability agentact · autonomous
Slot re-anchored; cohort_join_stall_ratio collapsing behind the queue tail
cdn_shield.queue_depth_concurrent dropped from 1.00 to 0.62 within 90 seconds of the shield pull + ceiling expansion; cdn_shield.queue_p99_wait_ms cleared to 0.41s (within the 1.5s slot budget); cohort.cohort_join_stall_ratio dropped from 0.142 to 0.024 — the late-join cohort is no longer stuck behind the queue tail.
Aug 13, 08:01:42 PM - T+9mSurfaced to humanby agent → viewer-platform teamact · surfaced to humans
Governed · surfaced to humans: apply_reconnect_throttle_to_cohort
Viewer-visible behaviour change required. The reconnect-throttle configuration (viewer-side reconnect backoff + jitter tuning + connector / segment re-fetch throttle) is surfaced for the viewer-platform team to review before the next marquee broadcast window — the agent holds off emitting; the throttle comes from configuration, not from a probe-side override.
Aug 13, 08:03:42 PM - T+12mSurfaced to humanby on-callact · surfaced to humans
On-call paged for the shield queue saturation root-cause review
Page acknowledged within 88s; reviewer confirmed the agent's rank and that the shield pull + ceiling expansion had prevented the late-join cohort from doubling into a viewer-storm stall before the reconnect fan-in dropped.
Aug 13, 08:06:42 PM - T+17mSurfaced to humanby reliability teamact · surfaced to humans
Postmortem write-up assigned (this page)
Reliability team assigned the public postmortem; this page is the resulting write-up, with the ranked hypotheses, the governed-action split, and the explicit viewer-side close-out signal pinned from the timeline. The queue-normalized-≠-viewers-green caveat appears in the recommended-fix section.
Aug 13, 08:11:42 PM - T+22mAutomated actionby Streamwake reliability agentact · autonomous
Re-probed the cohort; viewer-side close-out signal clearing
cohort.cohort_join_stall_ratio settled at 0.024 (within baseline tolerance 0.05 over a same-length cohort window); cohort.cohort_rebuffer_ratio settled at 0.013 (within tolerance); cohort.cohort_startup_time_p95_ms settled at 1942ms (within baseline); cdn_shield.queue_depth_concurrent reads 0.62 — queue is back inside the slot budget AND the cohort is delivering green playback.
Aug 13, 08:16:42 PM - T+36mResolutionby Operator + agentact · autonomous
Incident resolved; shield pull + ceiling expansion landed autonomously; viewer-side reconnect-throttle uploaded next campaign window
Late-join cohort divergence cleared on the viewer-side close-out signals (join_stall_ratio, rebuffer_ratio, startup_time_p95_ms) — NOT on the shield queue metric normalizing alone. The reconnect-throttle ships configured for the next marquee broadcast window so future spikes can be absorbed before the shield queue pegs at ceiling.
Aug 13, 08:30:42 PM
- classify · ranked three hypotheses with confidence in 90s
- shield pull · emitted pull_origin_shield_to_expanded_tier under the confidence ≥ 0.80 + cohort_divergence gate
- ceiling raise · emitted raise_cdn_shield_queue_concurrency_ceiling under the replay_concurrency_ceiling_pct == 1.00 gate
- status · observed queue_depth_concurrent collapse behind the slot budget; cohort_join_stall_ratio collapsing behind the queue tail
- re-probe · verified the viewer-side close-out signals cleared over the same-length cohort window
- viewer-platform team · apply_reconnect_throttle_to_cohort (governed — viewer-visible behaviour change required)
- on-call · paged for the shield queue saturation root-cause review
- reliability team · assigned the public postmortem write-up (this page)
Anatomy of the evidence packet
The two packets on the failing source — an origin-shield queue saturation probe packet on the marquee shield tier (with the queue depth pegged at the slot ceiling, the queue p99 wait climbing past the cohort's expected join-window profile, and the cohort's late-join divergence in lockstep with the shield wait-time climb), and the agent timeline response with the ranked hypotheses, the governed-action split, and the explicit viewer-side close-out signal. The probe packet is what the agent decided on; the timeline response is what the agent emitted.
GET /live/event/stream.m3u8 HTTP/1.1
host: cdn.example.com
accept: application/vnd.apple.mpegurl
----- cycle 0 (T+0m, before shield pull + reconnect-throttle) -----
# manifest returned clean by the shield tier, queue depth climbing
HTTP/2 200
content-type: application/vnd.apple.mpegurl
server-timing: manifest-fetch;dur=42
x-cdn: cdn-A/atl01 ← marquee broadcast window, viewer storm
x-packager: pkg-prime-04
x-shield-tier: shield-NA-3 ← the affected replay-origin tier
#EXTM3U
#EXT-X-VERSION:9
#EXT-X-TARGETDURATION:6
#EXT-X-MEDIA-SEQUENCE:84217
#EXT-X-MAP:URI="init.mp4"
#EXTINF:6.000,
084217.ts
# cdn_shield.queue_depth_concurrent: 1.00 (slot ceiling — vs 0.41 pre-peak baseline)
# cdn_shield.queue_p99_wait_ms: 4.62 (vs 0.21s baseline; exceeds the 1.5s slot budget)
# cdn_shield.queue_concurrency_ceiling_pct: 1.00 (slot reads at ceiling; the shield has stopped accepting fan-in)
# origin.replay_concurrency_ceiling_pct: 1.00 (replay-origin playing out the queue — pinned at ceiling)
# cdn_shield.shield_queue_window: 12m (queue tail longer than the cohort's expected 90s window)
# cdn_shield.segment_leg_cache_hit: pass (shield's own segment leg green — not a cache miss on the shield)
# edge.egress_kbps: 8642 (at-or-above expected — cohort IS delivering traffic)
# edge.manifest_route_5xx_ratio: 0.000 (no 5xx; the shield is queueing, not erroring)
# cohort.cohort_join_stall_ratio: 0.142 (vs 0.018 baseline, 90s window — late-join stalled behind the queue)
# cohort.cohort_rebuffer_ratio: 0.078 (vs 0.012 baseline — playback stalled on the late-join cohort)
# cohort.cohort_startup_time_p95_ms: 8210 (vs 1840 baseline; cohort start-up time hitting the 8s threshold)
# cohort.reconnect_storm_ratio: 1.00 (cohort reconnect fan-in matches the shield fan-in shape)
# encoder.bitrate_target_delta_pct: 0.21 (within baseline — encoder profile reads clean)
# origin.replay_orig_egress_kbps: 4211 (stable; origin egress not the pinch — shield queue is)
# cdn_pop.partial_segment_warmup_state: primed (segment leg green — not a cache warm-up failure)
# cdn_pop.affected_region_pop_id: cdn-A/atl01
# isp.cell_load.on_affected_region: pass (cell-load 0.43 vs 0.51 baseline reference)
----- cycle 1 (T+~3m, after shield pull + ceiling expansion) -----
HTTP/2 200
content-type: application/vnd.apple.mpegurl
server-timing: manifest-fetch;dur=28
x-cdn: cdn-A/atl01
x-shield-tier: shield-NA-3-expanded ← pulled into the expanded replay-origin tier
#EXTM3U
#EXT-X-VERSION:9
#EXT-X-TARGETDURATION:6
#EXT-X-MEDIA-SEQUENCE:84221
#EXT-X-MAP:URI="init.mp4"
#EXTINF:6.000,
084221.ts
# cdn_shield.queue_depth_concurrent: 0.62 (within the expanded tier's ceiling)
# cdn_shield.queue_p99_wait_ms: 0.41 (within slot budget)
# origin.replay_concurrency_ceiling_pct: 0.74 (replay-origin playing out — within ceiling)
# edge.egress_kbps: 9014 (egress continues to deliver above expected rate)
# cohort.cohort_join_stall_ratio: 0.024 (within cohort baseline)
# cohort.cohort_rebuffer_ratio: 0.013 (within cohort baseline)
# cohort.cohort_startup_time_p95_ms: 1942 (within startup_time baseline)
# cohort.reconnect_storm_ratio: 0.32 (well-cap'd — reconnect storm has subsided)
# governed_action_emitted: pull_origin_shield_to_expanded_tier,
# raise_cdn_shield_queue_concurrency_ceiling,
# apply_reconnect_throttle_to_cohort- cdn_shield.queue_depth_concurrent →
1.00 of slot ceiling (vs 0.41 pre-peak baseline) - cdn_shield.queue_p99_wait_ms →
4.62 (exceeds 1.5s slot budget by ~3×) - origin.replay_concurrency_ceiling_pct →
1.00 (replay-origin tier pinned at ceiling) - cohort_join_stall_ratio →
0.142 vs 0.018 baseline (90s cohort window) - edge.egress_kbps → 8642 Kbps, at-or-above expected rate. The cohort IS delivering traffic. cdn_shield.segment_leg_cache_hit → pass. Segment leg green on the shield tier — not a cache-miss amplifier.
{
"stream_id": "ckmarqueelivespike7411",
"source": "https://cdn.example.com/live/event/stream.m3u8",
"protocol": "HLS / CMAF / shield-tier / replay-origin",
"checked_at": "2026-08-13T19:54:42Z",
"ranked_hypotheses": [
{
"rank": 1,
"hypothesis": "origin_shield_queue_saturation",
"confidence": 0.86,
"evidence_signals": [
"cdn_shield.queue_depth_concurrent → fail (1.00 of slot ceiling vs 0.41 pre-peak baseline)",
"cdn_shield.queue_p99_wait_ms → fail (4.62s vs 0.21s baseline; exceeds the 1.5s slot budget by ~3×)",
"cdn_shield.queue_concurrency_ceiling_pct → fail (1.00 — slot stopped accepting viewer fan-in)",
"origin.replay_concurrency_ceiling_pct → fail (1.00 — replay-origin tier pinned at ceiling)",
"cdn_shield.shield_queue_window → fail (12m queue tail — beyond the 90s cohort window)",
"cohort.cohort_join_stall_ratio → fail (0.142 vs 0.018 baseline, 90s cohort window)",
"cohort.cohort_rebuffer_ratio → fail (0.078 vs 0.012 baseline — late-join cohort stalled behind the queue)",
"cohort.cohort_startup_time_p95_ms → fail (8210ms vs 1840ms baseline — cohort start-up time hit the 8s threshold)",
"cohort.reconnect_storm_ratio → fail (1.00 — cohort reconnect fan-in matches the shield fan-in shape exactly)",
"edge.egress_kbps → pass (8642 Kbps at-or-above expected — cohort IS delivering traffic; the shield queue is the pinch, not egress)",
"edge.manifest_route_5xx_ratio → pass (no 5xx — the shield is queueing, not erroring)",
"cdn_shield.segment_leg_cache_hit → pass (segment leg green on the shield's own tier — not a cache miss on the shield)",
"encoder.bitrate_target_delta_pct → pass (within baseline 0.21 — encoder profile reads clean)",
"origin.replay_orig_egress_kbps → pass (4211 Kbps stable — origin egress not the pinch)"
]
},
{
"rank": 2,
"hypothesis": "edge_pop_cache_miss_amplification",
"confidence": 0.41,
"evidence_signals": [
"cdn_pop.cache_hit on the affected edge POP reads pass on the segment leg; cdn_shield.segment_leg_cache_hit reads pass; the prime cohort is delivering traffic (edge.egress_kbps is above expected)",
"if a regional cache-miss amplifier were stateful, the warm edge POP would also misfire — it does not; the cohort's affected_POP and the cohort's warm_POP have identical cache-hit posture, so the failure cannot be cache-locale-shaped"
]
},
{
"rank": 3,
"hypothesis": "encoder_bandwidth_regression",
"confidence": 0.18,
"evidence_signals": [
"encoder.bitrate_target_delta_pct reads pass (0.21 within baseline); origin.replay_orig_egress_kbps reads pass (4211 Kbps stable); the affected cohort is delivering ABOVE the configured egress profile on the edge leg — the encoder could not be the regression if egress is shaped correctly",
"encoder timing reads continuously; encoder profile probes are clean across the same broadcast window on adjacent tier cohorts — the failure is shield-queue-shaped, not encoder-shaped"
]
}
],
"governed_actions": [
{
"action": "pull_origin_shield_to_expanded_tier",
"type": "governed",
"decision_lane": "autonomous",
"gating": "confidence >= 0.80 AND cohort_divergence_active",
"evidence": "top-hypothesis confidence 0.86; cohort_join_stall_ratio divergence active on the affected tier",
"expected_effect": "queue_p99_wait_ms clears within ~90s; cohort_join_stall_ratio collapses below 0.05"
},
{
"action": "raise_cdn_shield_queue_concurrency_ceiling",
"type": "governed",
"decision_lane": "autonomous",
"gating": "confidence >= 0.80 AND replay_concurrency_ceiling_pct == 1.00",
"evidence": "top-hypothesis 0.86; origin.replay_concurrency_ceiling_pct pegged at 1.00 on the affected tier",
"expected_effect": "queue_depth_concurrent drops below 0.80; replay concurrency ceiling re-anchors below 1.00"
},
{
"action": "apply_reconnect_throttle_to_cohort",
"type": "governed",
"decision_lane": "surfaced_to_humans",
"gating": "viewer_visible_behavior_change_required",
"evidence": "cohort.reconnect_storm_ratio matches shield fan-in shape — the cohort is feeding the queue",
"expected_effect": "viewer-side reconnect ramp attenuates; cohort reconnects drop below the shield's tolerance"
}
],
"verification_window": {
"close_out_signal": "viewer_side",
"probes": [
"cohort.cohort_join_stall_ratio within tolerance over the next same-length cohort window",
"cohort.cohort_rebuffer_ratio within tolerance over the same window",
"cohort.cohort_startup_time_p95_ms within tolerance over the same window"
],
"NOT_close_out_signal": [
"cdn_shield.queue_depth_concurrent returning to baseline — passes geometrically, does not pass viewer-side",
"origin.replay_concurrency_ceiling_pct returning below 1.00 — same caveat"
],
"explicit_note": "'queue normalized' ≠ 'viewers are getting green playback'. Recovery is verified viewer-side."
},
"surfaced_to_humans": [
{"owner": "viewer-platform team", "task": "approve the cohort-side reconnect-throttle configuration before the next marquee broadcast window"},
{"owner": "on-call", "task": "page for the shield queue saturation root-cause review"},
{"owner": "reliability team", "task": "assign the public postmortem write-up (this page)"}
]
}- autonomous →
pull_origin_shield_to_expanded_tier - autonomous →
raise_cdn_shield_queue_concurrency_ceiling - surfaced →
apply_reconnect_throttle_to_cohort - surfaced → paged on-call for the shield queue saturation root-cause review
- close-out signal → viewer-side:
cohort_join_stall_ratio,cohort_rebuffer_ratio,cohort_startup_time_p95_mswithin tolerance over the next same-length cohort window — NOTcdn_shield.queue_depth_concurrentreturning to baseline
Detection, classify, mitigate, recover (simulated telemetry)
Four timing windows on the postmortem timeline, each read off the cohort probe cadence. The figures are simulated telemetry — the disclosure near the top of this page applies to every figure on this list. Note that the close-out window is verified viewer-side (join, rebuffer, startup-time), not queue-side.
~5 s
cdn_shield.queue_depth_concurrent crossed 0.85→1.00 within a 30 s window; the agent surfaced the detector from the shield queue variance at T+5 s.
~1 m
origin_shield_queue_saturation ranked at 0.86 confidence with three ranked hypotheses at T+1 m — discriminator is edge egress green + segment leg green on the shield tier.
~7 m
pull_origin_shield_to_expanded_tier + raise_cdn_shield_queue_concurrency_ceiling queued at T+2 m and T+4 m; cohort re-anchored by T+7 m.
~22 m
cohort.cohort_join_stall_ratio, cohort.cohort_rebuffer_ratio, cohort.cohort_startup_time_p95_ms cleared within tolerance at T+22 m — viewer-side close-out, NOT queue-side.
The loop on this incident
Three steps close the lane on a marquee live-event shield queue saturation incident. The fix is split explicitly into the governed-action branch the agent takes autonomously under explicit gates, and the governed-action branch it surfaces to humans because it requires a viewer-visible configuration change.
The probe set fans in across the shield tier's slot ceiling and the late-join cohort's expected join-window profile. The shield reads queue_depth_concurrent → 1.00, queue_p99_wait_ms → 4.62s, and origin.replay_concurrency_ceiling_pct → 1.00; the cohort reads cohort_join_stall_ratio → 0.142, cohort_rebuffer_ratio → 0.078, and cohort_startup_time_p95_ms → 8210ms. Edge egress is at-or-above expected — the cohort is delivering traffic, the shield queue is the symbol.
edge_pop_cache_miss_amplification (0.41) and encoder_bandwidth_regression (0.18) dismissed.The discriminator against the cache-miss amplifier is the warm edge POP on the same cohort reads pass on cache-hit — so the failure is shield-queue-shaped, not cache-locale-shaped. The discriminator against the encoder regression isencoder.bitrate_target_delta_pct within baseline and origin.replay_orig_egress_kbps stable — so the encoder could not be the regression if egress is shaped correctly.
The autonomous branch (shield pull + ceiling expansion) clears the shield tier inside one resolution window. The surfaced branch (reconnect-throttle) closes the lane forward so future spikes are absorbed before the shield queue pegs at ceiling — it's surfaced because it changes a viewer-visible configuration, not probe-driven rebalance.
Pull / expand the shield — or throttle the reconnects.
On a marquee live-event shield queue saturation incident, the governed fix is a two-arm branch: an autonomous arm that pulls / expands the shield tier under explicit gates, and a surfaced-to-humans arm that throttles reconnects via a viewer-visible configuration change. The two arms close the lane in the same incident window and forward.
Raise cdn_shield.queue_concurrency_ceiling and the per-request timeout on the shield tier to absorb the marquee-viewer fan-in inside the cohort window. The fix is autonomous under a confidence-and-cohort-divergence gate (confidence ≥ 0.80 + cohort_divergence_active) and surfaced to humans outside that gate so the operator team can review the false-positive rate before the ceiling lands.
cohort.cohort_join_stall_ratio clears below 0.05 on the affected cohort within ten consecutive 30-second windows; cdn_shield.queue_p99_wait_ms clears under 1.5 s on the same cadence; origin.replay_concurrency_ceiling_pct re-anchors below 1.00 on the affected tier.
NOT a close-out signal: cdn_shield.queue_depth_concurrent returning to its pre-peak baseline. The queue depth can clear geometrically while the cohort is still stalled behind earlier queue tail — recovery is verified viewer-side, not queue-side.
Apply viewer-side reconnect backoff / jitter tuning plus connector / segment re-fetch throttle so the cohort's reconnect fan-in drops below the shield's tolerance. Surfaced to humans by default — governed, because the change is viewer-visible (reconnect cadence shapes the cohort's join experience directly). The throttle is configured, not probe-driven, so the agent holds off emitting and the viewer-platform team owns the change.
cohort.reconnect_storm_ratio settles under 0.40 on the affected cohort within ten consecutive 30-second windows; cohort.cohort_join_stall_ratio stays within baseline tolerance across consecutive marquee broadcast windows; the throttle ships configured for the next marquee broadcast window so future spikes can be absorbed before the shield queue pegs at ceiling.
NOT a close-out signal: the shield's queue-depth metric normalizing on its own — the queue can normalize as replay concurrency slackens without the cohort reconnect posture having been corrected, which would mean the next marquee spike pegs the queue again on the same lane.
cohort_join_stall_ratio, cohort_rebuffer_ratio, and cohort_startup_time_p95_ms clearing tolerance over the next same-length cohort window.A fix that normalizes cdn_shield.queue_depth_concurrent without clearing the late-join cohort's join stall, rebuffer, or startup-time within tolerance is a fix that didn't reach the cohort. The audit step on this incident writes the close-out signal into the play book as "verify the late-join cohort's join_stall_ratio, rebuffer_ratio, and startup_time_p95_ms settle within tolerance over the next same-length cohort window" — not "verify the shield queue returns to its pre-peak baseline".
Want Streamwake to catch this on your marquee cohort?
Sign up, register a marquee live-event probe, and the same cdn_shield.queue_depth_concurrent · cdn_shield.queue_p99_wait_ms · origin.replay_concurrency_ceiling_pct · cohort.cohort_join_stall_ratio probes that produced the timeline above run on every prime-cohort refresh — and surface in a Slack channel, a webhook, or the streams dashboard.
Synthetic Incident — This scenario uses simulated telemetry constructed from documented streaming behaviors. It does not represent a Streamwake customer outage.
- Pick a recent on-call incident — manifest stall, edge miss, player-side stall, or peer congestion.
- We replay it through the same reliability-agent probe cascade used on the postmortem above.
- You walk away with a written what-could-have-been-Automated readout, not a sales deck.
Related writeups
The closest siblings cover the player-visible symptom lands: ISP-vs-CDN triaging, manifest-timeout storms at a regional edge POP, live-event scale-out buffering, and the CMAF low-latency cache warm-up variant that lands on a freshly activated edge cohort instead of a marquee shield tier. Together they cover the four failure-mode lanes Streamwake reliability agents are tuned for alongside origin shield saturation.