Troubleshooting · ISP vs CDN

Synthetic incident postmortem

ISP congestion vs CDN failure
— same symptom, different fix.

A working postmortem on the regression that looks like an ISP problem but is a CDN problem, and vice-versa. The eight rows below map a player-visible symptom to a Streamwake probe pattern; each row reads end-to-end as a single triage decision a on-call engineer can make in under a minute.

Protocol: HLS + ISP/CDN

Format: probe packets · peer RTT

Streamwake probes: isp.peer_congestion · cache.freshness · edge.geo_drift · multicdn. winner_RTT.

Lead magnet

ISP congestion vs CDN failure

Read the postmortem — then bring your own incident to Streamwake.

Two ways to engage on this exact failure pattern: book a 30-minute technical demo where we walk through the probe cascade on your source, or hand us an archived incident and watch the agent diagnose it end-to-end.

Book a Technical Demo Analyze an Incident

Both routes land on the scoping intake form — no SDR gate.

Classification

How Streamwake would classify this incident

The agent's top hypothesis (the failure lane) and the secondary signal that carries the cause (the dispatch lane). The two lanes are independent on purpose — a top hypothesis of cdn_pop can still resolve to an ISP-side fix if a secondary signal is dominant.

Top hypothesis (failure lane)

What the agent named first — the failure category the timeline is filed under.

cdn_pop · 62%

cdn_pop — a percentage of the eu-west probe sample is failing at the edge. This is what flagged the timeline.

Secondary signal (cause lane)

Why the failure happened — this signal determines whether the fix is on the CDN, ISP, or carrier.

isp_congestion · dominant

isp_congestion — last-mile RTT jumped 8-12× across the residential peer sample on the same edge the cache probe reads clean. The CDN is healthy; the cohort is hitting buffer pressure on the residential uplink.

Severity, region, status

Severity is computed from the failing-cohort share; region is the geo of the failing probes.

sev2

Region: eu-west
Status: mitigated
Opened: 2026-08-06 10:42 UTC

Confidence

Top hypothesis share of the three ranked hypotheses; the remaining mass is split between player_release and origin_latency.

62 / 100

Above the 50% threshold the agent surfaces in the timeline. The secondary isp_congestion signal pushes the cause-side diagnosis to "dominant" — the dispatch lane picks ISP over CDN.

Chronology

Incident timeline

Six ordered events: detection on a single edge, classification into the two lanes, one automated action the agent took on its own, the operator's engagement with the ISP, the agent's re-verification after the reroute propagated, and the resolution.

Today

6 events

T+0m
Detection
by edge agent · eu-west
Anomaly detector flagged an unusual pacing deviation
tcp.retransmit_rate crossed the per-cohort threshold (0.4% → 14.2%) within a 90-second window across the residential peer sample.
Aug 6, 10:42:13 AM
T+2m
Classification
by Streamwake reliability agent
Incident classified: secondary isp_congestion over cdn_pop
Top hypothesis: cdn_pop (62% confidence) — but the secondary signal carrying the cause was an isp.peer_congestion pattern. Cache.availability stayed green, ruling CDN out as the cause.
Aug 6, 10:44:13 AM
T+3m
Automated action
by Streamwake reliability agent
Switched the affected cohort to prewarmed low-bitrate variants
Manifest cadence bumped to longer TARGETDUR; CDN tier raised to absorb retransmits. No CDN-side action taken — isp probes confirmed the edge was healthy.
Aug 6, 10:45:13 AM
T+8m
Status change
by Engaged the peering provider
Operator engaged the ISP / peering NOC for the affected AS block
Carrier confirmed buffer pressure on the residential uplink; reroute applied at the AS boundary. Player-side buffering degraded gradually as the reroute propagated.
Aug 6, 10:50:13 AM
T+31m
Automated action
by Streamwake reliability agent
Re-probed the edge; isp.peer_congestion returned to baseline
Last-mile RTT dropped from 412ms to 46ms within the second probe window. tcp.retransmit_rate dropped from 14.2% to 0.6%.
Aug 6, 11:13:13 AM
T+47m
Resolution
by Operator + agent
Incident resolved; prewarmed variants held for the rest of the live window
Player-side buffering cleared; CDN metrics flatlined; the root-cause summary posted to the incident timeline.
Aug 6, 11:29:13 AM

Anatomy

Anatomy of the evidence packet

Two probe packets on the same source, same cohort — one is the ISP-side signal pattern that diagnoses last-mile pressure, the other is the CDN-side signal pattern that diagnoses freshen / geo-routing breakdown. The probes on each side tell the opposite story.

ISP-side probe packet (probe_latency.mp4 from one edge)

{
  "stream_id": "ckliveeueventISP9",
  "source": "https://cdn.example.com/live/event/manifest.m3u8",
  "protocol": "HLS",
  "checks": [
    {
      "probe": "isp.peer_congestion",
      "result": "fail",
      "detail": "last-mile RTT across eu-west residential peers jumped from 38ms baseline to 412ms over 90s window",
      "severity": "sev2",
      "samples": [
        { "peer": "as3356-london", "rtt_ms": 412, "baseline_ms": 38, "delta_pct": 984 },
        { "peer": "as5511-frankfurt", "rtt_ms": 287, "baseline_ms": 41, "delta_pct": 600 },
        { "peer": "as1299-stockholm", "rtt_ms": 218, "baseline_ms": 33, "delta_pct": 561 }
      ]
    },
    {
      "probe": "isp.last_mile_rtt",
      "result": "fail",
      "rtt_ms": 412,
      "baseline_ms": 38,
      "detail": "single-edge synthetic probe_latency.mp4 from eu-west POP saw median segment-fetch RTT 412ms (baseline 38ms)"
    },
    {
      "probe": "tcp.retransmit_rate",
      "result": "fail",
      "rate_pct": 14.2,
      "baseline_pct": 0.4,
      "detail": "TCP retransmit rate across the same peer sample climbed from 0.4% to 14.2% — buffer pressure on the residential uplink"
    },
    {
      "probe": "cache.availability",
      "result": "pass",
      "detail": "manifest served X-Cache: HIT from cache-fra9921 — CDN healthy on this edge"
    },
    {
      "probe": "origin.reachable",
      "result": "pass",
      "latency_ms": 102,
      "detail": "origin shield reached in 102ms — unaffected"
    },
    {
      "probe": "edge.geo_drift",
      "result": "pass",
      "detail": "X-Served-By: cache-fra9921 matches eu-west expected POP"
    }
  ]
}

ISP-side probe pattern

The three failing probes are the smoking guns; the three passing probes are the rule-outs.

isp.peer_congestion → last-mile RTT jumped 8-12× across the residential peer sample.
isp.last_mile_rtt → single-edge probe_latency.mp4 latency 412ms vs 38ms baseline.
tcp.retransmit_rate → 14.2% vs 0.4% baseline. Buffer pressure on the residential uplink.
cache.availability → pass (HIT). CDN is healthy.
edge.geo_drift → pass. Routing is the expected POP.
origin.reachable → pass. Origin shield untouched.

CDN-side probe packet (cache.freshness + edge.geo_drift)

{
  "stream_id": "ckliveeueventCDN8",
  "source": "https://cdn.example.com/live/event/manifest.m3u8",
  "protocol": "CDN",
  "checks": [
    {
      "probe": "cache.freshness",
      "result": "fail",
      "age_seconds": 47,
      "detail": "X-Cache: HIT with age=47 > declared max-age=2 — stale-on-revalidate"
    },
    {
      "probe": "edge.geo_drift",
      "result": "fail",
      "detail": "X-Served-By: cache-jfk8842 in eu-west session — eu POP absent, requests cross Atlantic to a na POP"
    },
    {
      "probe": "multicdn.winner_RTT",
      "result": "fail",
      "winner": "cdn-b",
      "rtt_ms": 318,
      "detail": "cdn-a 38ms, cdn-b 318ms — cdn-a routing flipped to a degraded eu edge"
    },
    {
      "probe": "isp.peer_congestion",
      "result": "pass",
      "samples": [
        { "peer": "as3356-london", "rtt_ms": 41, "baseline_ms": 38, "delta_pct": 8 },
        { "peer": "as5511-frankfurt", "rtt_ms": 44, "baseline_ms": 41, "delta_pct": 7 }
      ],
      "detail": "eu-west residential peers within ±10% of baseline — last-mile healthy"
    },
    {
      "probe": "isp.last_mile_rtt",
      "result": "pass",
      "rtt_ms": 39,
      "baseline_ms": 38,
      "detail": "single-edge probe_latency.mp4 latency 39ms — within baseline noise"
    },
    {
      "probe": "origin.reachable",
      "result": "pass",
      "latency_ms": 96,
      "detail": "origin shield reached in 96ms — unaffected"
    }
  ]
}

CDN-side probe pattern

The three failing probes are CDN-internal; the two passing ISP probes rule out last-mile as the cause.

cache.freshness → age=47 > max-age=2. Stale-on-revalidate.
edge.geo_drift → X-Served-By: cache-jfk8842 in an eu-west session.
multicdn.winner_RTT → cdn-b won with 318ms; cdn-a at 38ms is the right winner.
isp.peer_congestion → pass. Last-mile RTTs within ±10% of baseline.
isp.last_mile_rtt → pass. Single-edge latency 39ms vs 38ms baseline.

Triage × fix

Eight ways an ISP-vs-CDN regression shows up

Each row is a single triage decision: a symptom the player reports, the Streamwake probe pattern that disambiguates the cause, and the operator fix on the branch the probe pattern picks. The pattern name is what you grep for in the agents feed.

01Last-mile RTT spike, CDN cache clean

classify as ISP congestion

Symptom

Median segment-fetch RTT from a single edge jumps 8-12× across the eu-west peer sample. Cache.availability probe still reads X-Cache: HIT; origin.reachable shows normal latency.

Triage path

isp.peer_congestion + isp.last_mile_rtt → fail; cache.availability + origin.reachable → pass. Decisive ISP-side signal.

Operator fix

Open a ticket with the peering provider for the affected ASNs; in the interim, switch the affected cohort to prewarmed low-bitrate variants and raise the CDN tier to absorb retransmits. Do NOT re-route the CDN — it is healthy.

02Edge miss across multiple POPs, last-mile stable

classify as CDN failure

Symptom

Manifest fetches return X-Cache: MISS with consistent 200s across eu-west AND na-east POPs from the same edge IP block. Last-mile RTTs read within baseline noise.

Triage path

cache.availability → fail across POPs; isp.peer_congestion + isp.last_mile_rtt → pass. Origin deliverability unaffected.

Operator fix

Trigger a forced purge on the affected path and verify edge.purge_acknowledged from at least 3 geo probes; if misses persist, escalate to the CDN provider NOC with the X-Served-By trace. Do NOT engage the ISP — last-mile is healthy.

03Edge mis-routes across the Atlantic, peers healthy

classify as CDN failure

Symptom

EMEA session reads X-Served-By: cache-jfk8842. Last-mile RTTs from the same EMEA peers stay within ±10% of baseline. Players buffer because every segment now traverses the Atlantic.

Triage path

edge.geo_drift → fail; isp.peer_congestion + isp.last_mile_rtt → pass. CDN geo-routing table drift, not last-mile.

Operator fix

Pin geo_routing tests to the expected eu-west POPs; flag X-Served-By outside expected region. Roll back the routing-table deploy or override the geo rules until the next drill. Do NOT engage the ISP — probes show last-mile is fine.

04TCP retransmits spike, hit-rate holds

classify as ISP congestion

Symptom

tcp.retransmit_rate climbs from 0.4% baseline to 14% on the same peer sample. CDN cache reads X-Cache: HIT on every refresh; edge.geo_drift reports the expected POP.

Triage path

tcp.retransmit_rate + isp.peer_congestion → fail; cache.availability, edge.geo_drift → pass. Buffer pressure on the residential uplink.

Operator fix

Buffer pressure is the smoking gun for ISP congestion. Engage the peering provider for the affected AS block; switch manifest cadence to longer TARGETDUR on the affected variants. CDN does not need to be touched.

05Stale-on-revalidate, hit-rate healthy across POPs

classify as CDN failure

Symptom

Manifest reads HTTP 200 with age=47 against a declared max-age=2, Warning: 110 on the response. Cache reads X-Cache: HIT across every POP from the same edge — freshness is the only failing signal.

Triage path

cache.freshness → fail (Age > max-age); cache.availability → pass (HIT). Edge / origin healthy, only the cache-control contract is wrong.

Operator fix

Tighten stale-if-error to under one live window; pin Cache-Control: no-cache, must-revalidate on the master playlist. The cache.freshness probe must fail when age > declared max-age — do NOT bypass the agent because the body "looks right".

06Multi-CDN race — one provider degraded, last-mile steady

classify as CDN failure

Symptom

multicdn.health_score reads cdn-b at 0.62 for 5 min, but DNS still weights 50/50. Half the cohort hits cdn-b. Last-mile RTTs from the same cohort read within baseline.

Triage path

multicdn.health_score → fail; multicdn.winner_RTT shows the wrong RTT winner; isp.peer_congestion → pass. Multi-CDN controller is slow to converge.

Operator fix

Feed multicdn.health_score directly into DNS weights; require controllers to converge within 30 seconds. Pin the degraded provider to weight 0 until three consecutive intervals report health > 0.95. Do NOT engage the ISP — last-mile is stable.

07Last-mile spike + edge miss on a single geo

classify as CDN failure

Symptom

A single eu-west POP shows isp.peer_congestion fail AND cache.availability fail on the same source. Last-mile RTT from the same peer sample is at baseline; the miss is POP-local.

Triage path

Run isp.peer_congestion + isp.last_mile_rtt from a DIFFERENT edge in a different AS block. If THAT pair is at baseline, the failure is POP-local, not last-mile.

Operator fix

Re-verify on a peer POP. If a different POP in the same region reports the source healthy, the failure is the original POP — push the affected cohort's DNS off the POP and escalate to the CDN NOC. Do NOT engage the ISP on the carrier; the original POP is the cause.

08Buffering reports but every probe is green

classify as Ambiguous (rule out CDN first)

Symptom

Players report buffering but cache.availability, cache.freshness, isp.peer_congestion, isp.last_mile_rtt, edge.geo_drift all pass. No cohort is consistently failing.

Triage path

Re-run every probe with a 30-second cadence and pin the affected cohort by ASN. If two consecutive windows pass, the buffering is player-side or app-tier (DRM token issuance, ABR ladder). Do NOT chase the ISP on a one-window signal.

Operator fix

Open a player-side ticket first — verify DRM token issuance, ABR variant availability, and the decoder initialization segment. Re-run the probes after the player fix; only escalate to ISP / CDN if the same symptom recurs across two consecutive windows.

Lessons learned

Three things this postmortem should change

The lessons are written as actions an operator should ship after the incident — not as a retrospective summary. Each one closes a gap the timeline exposed.

Rule out the CDN first

Treat CDN-clean as a precondition for any ISP-side fix.

Only after cache.availability, edge.geo_drift, and origin.reachable all read clean should the on-call engage the ISP / peering NOC. Reversing that order burns cycles on the wrong lane.

Probe last-mile from the failing edge IP block

Don't extrapolate from a peer POP — pin to the cohort.

isp.peer_congestion is only decisive when run from the same edge IP block the failing cohort is on. A baseline-clean probe from a non-failing POP can hide an AS-local buffer pressure.

Treat buffer pressure as the smoking gun for ISP

It never shows up on a healthy CDN.

TCP retransmit rate diverging from baseline while cache.availability stays green is the single most decisive signal for ISP-side congestion. The rule should be 'never chase the CDN on a retransmit-rate signal'.

Diagnose

Diagnose with Streamwake

Register the source against POST /api/v1/streams, then read the agent timeline back through GET /api/v1/agents. The probe verdicts in the timeline are exactly the patterns the rows above name.

The curl below registers an HLS source with a 60-second probe cadence. The HLS+ISP protocol routing makes the agent run isp.peer_congestion alongside cache.freshness, edge.geo_drift, and multicdn.winner_RTT on the same sourceUrl — the timeline that comes back is the exact shape you need to apply the triage table on this page.

The cookie is the same better-auth.session_token that gates every /api/v1/* call — see the auth guide for how to mint one.

Five-minute quickstart Open /app/streams

POST /api/v1/streams

curl -X POST https://streamwake.polsia.io/api/v1/streams \
  -H "content-type: application/json" \
  -b "better-auth.session_token=<your-session-cookie>" \
  -d '{
    "sourceUrl": "https://cdn.example.com/live/event/manifest.m3u8",
    "protocol": "HLS+ISP",
    "probeIntervalSeconds": 60
  }'

GET /api/v1/agents?stream_id=…

curl https://streamwake.polsia.io/api/v1/agents?stream_id=<id> \
  -b "better-auth.session_token=<your-session-cookie>"

How the timeline reads back (semantic only)

GET /api/v1/agents?stream_id=ckliveeueventISP9
→ 200 OK
[
  { "type": "detection",          "at": "T+0m",  … tcp.retransmit_rate breach },
  { "type": "classification",     "at": "T+2m",  … cdn_pop 62% / isp_congestion dominant },
  { "type": "automated_action",   "at": "T+3m",  … cohort switched to prewarmed variants },
  { "type": "status_change",      "at": "T+8m",  … peering NOC engaged, reroute applied },
  { "type": "automated_action",   "at": "T+31m", … isp.peer_congestion returned to baseline },
  { "type": "resolution",         "at": "T+47m", … player-side buffering cleared }
]

Next step

Want Streamwake to catch this on its own?

Sign up, register an HLS+ISP probe, and the same timeline the rows above name runs on every refresh — and surfaces in a Slack channel, a webhook, or the streams dashboard.

Need Streamwake on one of your incidents?

Would you like Streamwake to analyze one of your historical incidents and show where AI could reduce investigation time? (Filed under: ISP congestion vs CDN failure.)

Incident analysis

Pick a recent on-call incident — manifest stall, edge miss, player-side stall, or peer congestion.
We replay it through the same reliability-agent probe cascade used on the postmortem above.
You walk away with a written what-could-have-been-Automated readout, not a sales deck.

Analyze this on my incident Book a Technical Demo Or sign up and self-serve

Related incident shapes

The cohort-only divergence that an ISP / CDN triage can't quite reach — a single upstream transit link into a regional POP degraded during a peak window, caught by cross-region latency fan-in and remediated by failover to a backup transit path.

Transit provider failure → regional POP

Back to library

Read the next protocol guide

The other live entries cover the canonical protocol-by-protocol failure patterns.

CDN edge failures HLS MPEG-DASH All troubleshooting guides

ISP congestion vs CDN failure— same symptom, different fix.

Book a technical demo for ISP congestion vs CDN failure

How Streamwake would classify this incident

Incident timeline

Today

Anatomy of the evidence packet

Eight ways an ISP-vs-CDN regression shows up

Three things this postmortem should change

Diagnose with Streamwake

Want Streamwake to catch this on its own?

A transit-link-adjacent writeup

Read the next protocol guide

ISP congestion vs CDN failure
— same symptom, different fix.