How a Transit Provider Failure to a Regional POP Shows Up in the Cohort

This is commentary on a class of public, viewer-visible live-streaming failures — failures that originate upstream of the CDN edge and surface inside a single regional audience as degraded playback, without the global cohort reporting any problem at all. It is not a claim of involvement in, or attribution to, any specific incident.

A transit-failure-to-a-regional-POP incident is the canonical case of a failure that is invisible globally and bad locally. The whole site, the whole product, the whole fleet can be at nominal — and one regional audience is sitting through a span of degraded playback because the upstream link carrying traffic into that regional point-of-presence is degraded. The signal that catches it has to be regional. The fix has to be regional. The reader-rendering loop has to converge inside the regional window, before the cohort churns and the playback stat rolls into the operator's report unexamined.

Below, the cohort saw, the ranked root-cause walkthrough, how Streamwake caught it, the three-step Detect → Remediate → Audit loop with failover to a backup transit path, and a closing note on what the incident changes for next time.

The cohort saw

The first symptom is not on a graph — it's inside the join. A long join is the one symptom that beats every rebuffer reporting pipeline a player has, because it hits before the player has even decided whether to count a stall. A viewer whose manifest fetch is delayed by 1.8 seconds has not yet asked for a segment — they're still waiting on the controller manifest to come back. The rebuffer ratio bumps only after the join settles and the playback clock has tried to catch up.

On a marquee broadcast window, three patterns show up at once:

Long join on first-segment manifest fetch. Median segment-fetch RTT from the affected region's primary edge climbs from a 38ms baseline toward 480ms within a single 90-second window. The upstream link carrying the cohort's traffic into the regional POP is the one path the cohort's player controller talks to for manifest fetches; under-deliverability there looks like latency on every first manifest refresh.
Intermittent playback, pattern-specific to the affected region. The affected region's rebuffer ratio climbs from a 1.6% baseline to 6.4% within two minutes. The unaffected regions (na-east, apac) are flat at baseline. The cohort divergence is regional — and that is the shape of a single-region deliverability problem, not a global encoder or CDN problem.
No request-side error. The HTTP 200 ratio on manifest and segment fetches from the affected region reads at 100%. The CDN edge is doing its job — it's just being asked to do it on an under-deliverable upstream link. Failures of this shape are not HTTP-error failures; they are latency failures that the player renders as stalls.

The viewer-visible shape is therefore regional, latency-heavy, and error-free. It is the inverted image of a CDN outage (which is regional and error-heavy) and the inverted image of an ISP / last-mile congestion event (which is ASN-scoped, not POP-scoped). It is precisely the shape the Streamwake probe set calls transit_link_degraded, and it is precisely the shape that does not get caught if the team is only watching error-rate dashboards.

Ranked root-cause walkthrough

The system surfaces three hypotheses when the regional-only divergence cluster fires. They are ranked top to bottom by confidence — and the dismissive work on the lower two is as load-bearing as the affirmative work on the top one, because dismissing them is what narrows the fix lane from "everything is possible" to "the upstream carrier is the move".

Rank 1 — transit-failure-to-regional-POP (confidence 0.91)

The top hypothesis is a single upstream transit link carrying the cohort's traffic into the regional POP has become degraded. The signal set is consistent with link-level deliverability loss:

The cohort re-anchors to the affected POP on first manifest fetch. There is no second route the cohort is finding; the affected POP is the only healthy edge for the affected ASN class.
Cross-region latency-anomaly fan-in flags a coherent regional divergence inside the same ASN block, while unaffiliated regions stay at baseline latency. The pattern is upstream-link-shaped: one POP, one carrier, one window.
Origin-side latency, as observed from the affected POP, has degraded (segment-fetch RTT 412ms vs 38ms baseline), while origin-side latency as observed from the unaffected regions stays at baseline. The CDN cache is doing its job locally — the latency is on the inbound leg to the POP, not on the egress leg from the POP to the viewer.
The CDN probe (cdn_pop) for the affected POP reports healthy cache state: cache.availability reads HIT, cache.freshness is within window, the manifest is being served from the local cache. The CDN is not the cause.

The carrier is the move.

Rank 2 — origin-shield saturation (confidence 0.42)

The secondary contributor is origin-shield saturation. The hypothesis is real and is named explicitly to avoid fixing the wrong lane:

The cohort's overall playback demand peaks in the same window that triggers the transit degradation. The originating side is delivering the segments, and at peak delivery ratio the origin shield absorbs every cache miss through the degraded link.
If the transit link is what degraded first, origin-shield saturation is the amplifier — it shifts every cache-miss through the same degraded leg, which means the cache-miss posture compiles on top of an already-degraded path.
This is the hypothesis that explains why a partial mitigation (raising the CDN tier to absorb more cache misses) was not enough: the miss posture was not the whole problem; the link was.
Fixing only the origin-shield saturation — without touching the transit link — would have moved segments through more cleanly, but the cohort's join time would still be dominated by the under-deliverable link.

Origin-shield saturation is the second-largest contributor, named explicitly so the audit step below knows to log it; it is not the cause.

Rank 3 (dismissed) — CDN geo-routing table drift (confidence 0.18)

This hypothesis is dismissed with explicit language so the dismissal is auditable:

The probe pattern on geo-routing table drift is edge.geo_drift failing with an X-Served-By trace pointing at a different region. On this incident, edge.geo_drift passes — the cohort is hitting the expected regional POP, not a fallback POP in a different region.
A geo-routing-table drift would have routed the cohort to a fallback POP, which would have shown up as a second POP being involved, not as the one expected POP degrading. The probe set is decisive on the dismissal.
An operator who escalates on geo-routing at this point chases a lane that is closed.

The dismissal is logged so the playbook doesn't re-test it next incident.

How Streamwake caught it

The catch is shaped by what the failure is not. A failure on a single upstream transit link into a regional POP is the kind of failure that is invisible to a global median, invisible to a single-PoP aggregate, invisible to an encoder-side metric. It is visible only to a probe set that fans in across regions and looks for cohort divergence within the affected region.

Concretely, three slots in the Streamwake probe set are load-bearing here:

Cross-region latency-anomaly fan-in. The probe set measures segment-fetch RTT into every regional POP under our measurement coverage. When one POP shows latency degradation while the others do not, the fan-in discriminates the regional nature of the failure. A single-PoP observation could be the POP being itself; a cross-region fan-in is the POP being fed by something upstream.
ASN-scope coherence. The probe set partitions the affected region by ASN class. A failure that is transit-link-shaped hits every ASN class on the affected POP simultaneously — the cohort divergence is no longer AS-bound, it is POP-bound. An ISP / last-mile congestion event would show the inverse signature: the divergence is concentrated in one ASN class, not spread across all classes hitting the POP.
Healthy last-mile / CDN-side rule-outs. On this incident, last-mile RTTs on the affected POP close to baseline, and the CDN-cache probes (cache.availability, cache.freshness, edge.geo_drift) all read clean. The probe set is decisive on the rule-outs — without those, the failure could be misdiagnosed as a CDN-side problem and the fix lane would be wrong.

The cross-region fan-in is the single signal that distinguishes this incident from a CDN edge failure (same POP, same symptom shape, but the CDN probes would be failing) and from an ISP congestion event (same ASN-scope cohort divergence, but the cross-region fan-in would also show baseline, not concurrency-with-the-affected-POP).

The three-step loop: detect → remediate → audit

The three-step loop is the same shape across every Streamwake autonomous-response surface. On a transit-failure-to-regional-POP incident the steps are:

Detect

Cross-region latency-anomaly fan-in fires on the affected POP's primary upstream transit link while the cohort rebuffer ratio and segment-fetch RTT inside the affected region diverge from the unaffected regions. The primary upstream transit link is flagged at the first deliverability degradation — the system does not wait for the cohort to start reporting. The drift between cohort divergence inside the region and baseline-freshness outside is the discriminator that closes the failure lane. The probe set is independent of the global mean: separate windows, separate cohorts, separate verdicts, fan-in is the convergence.

Remediate

Failover the affected POP's cohort to a backup transit path. The backup transit path is pre-provisioned as part of the regional POP's design — the secondary link is there for exactly this class of incident, with its own headroom against the carrying capacity the primary link is currently failing to deliver.

The cohort re-anchors to the backup transit path inside one resolution step. The first cohort member that hits the backup sees under-baseline join time within seconds; the population reproportionates as the routing table applies.
Origin-shield saturation is absorbed by the headroom on the backup path. The fix is the link — the secondary contributor (rank 2) is not addressed separately, because addressing it without addressing the link would have moved throughput but not behavior.
The fix lane is "carrier-side link". It is not "CDN tier", "player ABR ladder", "DRM shard warmup", or "encoder switchover". The loop keeps that discipline explicit and closes only the lane that the probe set has confirmed.

Audit

Post-incident capacity planning loops. Three artifacts are written into the reliability ledger in this order:

The saturation curve. The peak-window demand profile on the affected POP's primary link is logged with the delta that pushed the link from headroom into degradation. Capacity planning for the next peak window reads this curve alongside the carrier-side reroute log.
The carrier RCA. The transit provider's published RCA on the degraded link is filed, with a digest of which carrying leg failed and what the upstream contributing factors were (subsea cable, peering partner, etc.). If a published RCA is not available, the playbook escalates to a manual RCA request.
The hardened carrier-selection rule. The carrier-selection rule for the regional POP is hardened so future peak-window events do not select the same link as the primary carrying leg when the cohort demand profile carries this shape. A rule update is a playbook edit, not a code change; the loop treats it as a planning artifact.

What we learned

Three things shifted in the playbook after this incident:

Address the link before addressing the amplifier. The temptation on a transit-failure incident is to apply the secondary contributor's fix first (raise the CDN tier, shift the cache-hit posture), because that fix has shorter lead-time. The secondary contributor is real, but addressing it alone moves throughput — not behavior. The cohort's experience is dominated by the link, not the miss posture. The link is the fix.
Cross-region fan-in is the discriminating signal. Two probes could catch this incident — a single-POP observation, or a cross-region fan-in. Only the cross-region fan-in is decisive on the failure lane, because the failure is by definition regional. A cohort-running dashboard that is single-PoP-aggregate will catch the symptom and miss the lane.
The backup transit path is a planning artifact, not a cold fix. Failover to the backup transit path runs as a federated step; the path is pre-provisioned, well-known, and fast. Capacity-planning decisions are made on the assumption that the backup path is in active rotation within the resolution window — that assumption is what makes the fix cheap.

The three-step loop — Detect, Remediate, Audit — applies in the same shape to every Streamwake autonomous-response surface: the discriminating signal is what changes between failure classes, the loop itself does not.

Read the next

For the protocol-by-protocol breakdown of how to triage a live regression of the analogous shape, see the ISP congestion vs CDN failure guide. For the full editorial library of incident writeups and the Incident Lab hub, start at the hub index.

All posts