Tie QoE metrics to the CDN edge
spot the gap and fix it.

Dashboards read rebuffer_ratio 0.0% over the rolling-7-day window, but users report clear stalls in session cohorts that the dashboard rolls up. Vendor QoE looks green; users disagree.

QoE reporting is feature-flagged off for that session cohort — A/B partition has a fallback to no-emit, so the rebuffer events never reach the analytics pipeline. The cohort missed the QoE source flag.

Surface the rebuffer signal from player.events.rebuffer directly into the QoE probe pipeline and stop trusting the vendor's aggregate. The qoe.rebuffer_ratio probe reads 0.018 on the player-side cohort that the vendor reports 0.0% on; cross-check both on every cycle until they converge.

Agent detects missing rebuffer via cross-check vs segment_fetch_latency outliers and emits rebalance_hints=[pin_qoe_source_to_event_pipeline]. The next probe stitches player.events to qoe.rebuffer_ratio on the same window — rebuffer_ratio flips to 1.8% and the failing cell is named.

p75 video-startup time climbs +18% over the last 7 days. The SLA board stays green throughout because its alerting window is rolling-24h, which dilutes the cohort trend.

A CDN cache-control change deployed 7 days ago widened manifest fetch latency from p95=214ms to p95=287ms, dragging startup time with it. The rolling-24h aggregate washes the change out; the rolling-7-day trend is the one that surfaces it.

Re-time the SLA alert on the p75 of the rolling-7-day cohort, not the rolling-24h aggregate. qoe.video_startup_time should be probed at agent cadence (30s) and the SLA alert should fire when p75 over the cohort hops more than ~10% week on week.

Agent surfaces the cohort trend, runs manifest.fetch_latency upstream to find the delta, and emits rebalance_hints=[tighten_manifest_cache_ttl_until_rev_checked]. Streaming stays green while engineering re-times the SLA window and rolls back the cache-control change.

Global manifest_fetch_failures reads 0.04% (green) but a single geo cohort reads 1.8% (red). Viewers in that cohort see consistent manifest timeouts; aggregate-vendor dashboards stay silent.

CDN telemetry rollup aggregates per-provider, not per-(CDN, geo). The per-geo breakdown Stage is exposed by the vendor, but the per-POP cells are not; per-cell greens can hide a red POP inside a green geo.

Scope the manifest.fetch_failures probe per-(CDN, geo) cell and fail the probe when any cell reads >0.5% over the probe window. The qoe→manifest-fetch ladder must read the per-cell value, not the per-provider rollup.

Agent sees the partial-consistent cell mix on EMEA and emits rebalance_hints=[pin_manifest_to_healthy_geo_pop]. Pin next probes to cdn-B/eu-west for EMEA; the manifest path on cdn-A/fra02 gets capacity headroom while the failing POP is repaired.

Viewers in one (CDN, geo) cohort see partial stalls; aggregate CDN-segment-fetch-failure is 0.07% — under the global alerting threshold. The cohort barely moves the global rollup, but the cohort is fully degraded.

Outlier cells are diluted by healthy cells in the rolling-1-min aggregate. The aggregate reads green because nine healthy (CDN, geo) cells average out one failing cell; the failing cell never crosses the global threshold.

Floor the per-cell failure threshold above the cohort-average floor — segment.fetch_failures_by_cdn must read the per-(CDN, geo) cell, not the rolled-up CDN aggregate. The threshold should be in absolute terms (>= 0.5% per cell) not aggregate (>= 0.1% per CDN).

Agent scopes the failing (CDN, geo) cell and emits rebalance_hints=[route_segment_to_alternate_pop_for_failing_geo]. Pin segment fetches to cdn-B for the failing geo cohort; cdn-A gets isolated while on-call narrows the causal cell.

Dashboard surfaces cdn.hit_ratio below the 95% optimize threshold 4-5 minutes after user-visible rebuffer lifted. The dashboard sample window is wider than the user-impact window; the alert-by-dashboard is too late.

The hit-ratio dashboard aggregates over a 5-minute window while the live edge path is computed at a smaller cadence. The optimize-threshold alert kicks in only after the dashboard window closes.

Emit the cdn.hit_ratio probe at agent cadence (≤30s) and fail the probe below the optimize threshold (95%) within the probe window. The dashboard can stay coarse — the probe is the source of truth.

Agent pre-warms the edge cache on the next probe cycle and emits rebalance_hints=[prewarm_warm_window_for_manifest_path]. The next probe reads hit_ratio=96% — user-visible rebuffer recovered within a single probe interval, before the dashboard window closed.

Shield saturation burn but no shield fault log. The shield latency p95 climbed to 612ms and bypass rate is 11%, but the dashboard reads green because the gating metric watches only x-cache: ERROR.

The shield-error-rate metric gates on x-cache: ERROR responses — the reroute rule emits x-cache: BYPASS when it bypasses the shield (without an error). Saturation re-routes traffic but the metric never flips to fail because the status reads BYPASS, not ERROR.

Count bypass-without-fault as a shield-fail class in the probe. The origin.shield_health probe should read shield_latency_p95 + bypass_rate together — bypass-without-fault is a fail even when no ERROR has been emitted.

Agent detects the bypass-without-fault spike and emits rebalance_hints=[pin_edge_to_origin_direct_until_shield_recovers]. The next probe witnesses origin-direct serving green; shield backpressure releases; bypass rate drops back under 1% within two cycles.

A CDN provider's aggregate error rate reads 0.4% (green), but a single geo cohort reads 6.1% (red). Vendor dashboards aggregate over CDN provider, not per (provider, POP, geo) — the failing cell never reaches the dashboard.

Vendor reports aggregate CDN health, not per-cell. The aggregate-greens hide a red POP inside a green geo, and on-call never sees the failing cell because the alerting window rolls up to the CDN provider level.

Scope the edge.error_rate_per_pop probe to per-(provider, POP, geo) cell and tag the failing cell by its POP identity. Aggregate thresholds should be derived from the per-cell view, not the other way around.

Agent scopes the failing cell (cdn-A/fra02, EMEA) and emits rebalance_hints=[route_egress_to_alternate_pop_for_failing_geo]. Pin egress to cdn-B/eu-west for EMEA; the failing cell on cdn-A/fra02 gets capacity headroom and on-call is paged for the single cell, not the whole provider.

A burst of client errors reads 'MSE SourceBuffer init failure on first segment'. Init.mp4 hasn't been pre-warmed at the edge when the first segment lands; the player fetch races the CDN ingest.

Packager emits segment asynchronously from CDN ingest. The instrumentation on each side ends at the silo boundary — packager telemetry and CDN ingest telemetry lack a shared join, so the handoff wait is never timed.

Bridge the gap by registering a packager->cdn.handoff_wait probe that times segment-emit to edge-readable on the first cycle. The probe should fire when handoff wait exceeds 500ms — the budget the player's startup path assumes.

Agent bridges the packager/CDN telemetries and emits rebalance_hints=[prewarm_init_for_new_session]. Pin the next init probe to cdn-B for EMEA cohort; pre-warm the edge cache on init.mp4 so the player read races the CDN ingest correctly.

4% of sessions show +3s on video-startup time. License-first-byte latency correlates — but the regression banner reads video-startup time without naming the license-fetch path.

License proxy is a separate observability silo. Either-side metric alone misses the join — qoe.video_startup_time looks green except in the 4% cohort, and license.first_byte looks green except on assets with high keyserver latency.

Emit a probe that joins qoe.video_startup_time with license.first_byte per session and flags the correlation. The probe should fire when both metrics move in tandem above their per-session thresholds.

Agent surfaces the license-fetch path in the timeline and emits rebalance_hints=[pin_license_to_alternate_keyserver_pop]. Pin the next license probe to a keyserver POP with a healthy first-byte profile; video-startup time drops below budget within the next three cycles.

Aggregate egress healthy across CDN. One geo cohort shows egress queue depth climbing and 5xx climbing within the same cohort. Aggregate vendor dashboards read green; the cohort is fully degraded.

Vendor reports aggregate CDN health, not per-cell. The aggregate reads green because nine healthy (CDN, geo) cells average out one failing cell; the failing cell never reaches the alerting threshold at aggregate.

Surface edge.egress_health as a per-cell (CDN, geo) probe and tag the failing cell by its POP identity. The bw_ratio and queue_depth metrics must be per-cell, not aggregate; the threshold should fire when per-cell bw_ratio >= 0.85 or queue_depth grows 3x in 5 minutes.

Agent scopes the failing cell (cdn-A/fra02) and emits rebalance_hints=[route_egress_to_alternate_pop_until_egress_recovers]. Pin egress to cdn-B/eu-west for EMEA; cdn-A/fra02 gets capacity headroom while on-call narrows the causal cell.