AWS/MediaLive ActiveAlerts correlated against CloudFront cache-miss latency.
How the +Amazon CloudWatch integration pulls AWS-side metrics and logs as evidence during incident investigation, joins them against the streaming-side CDN signals on the same bus, and emits the remediation back as a CloudWatch metric so the same dashboard panel that flagged the ActiveAlerts drift is the panel that re-renders the loop-closure window.
The CloudWatch evidence pull.
The CloudwatchCorrelationsPanel on /app/incidents/[id] POSTs /api/v1/cloudwatch-evidence/collect with the active incidentId. The route signs one GetMetricData call against monitoring.<region>.amazonaws.com AND one FilterLogEvents call against logs.<region>.amazonaws.com using a handwritten SigV4 signer + sts:AssumeRole if the source row carries a roleArn.
POST /api/v1/cloudwatch-evidence/collect HTTP/2
host: api.streamwake.example
content-type: application/json
x-incident-id: ckq3xrenamexz-p95jadi-cmh
----- POST body -----
{ "incidentId": "ckq3xrenamexz-p95jadi-cmh" }
----- CloudWatch metrics pull + AWS signing (reconstructed) -----
POST monitoring.us-east-2.amazonaws.com/ HTTP/2
authorization: AWS4-HMAC-SHA256 Credential=AKIA***/20260821/us-east-2/monitoring/aws4_request, SignedHeaders=...; Signature=...
host: monitoring.us-east-2.amazonaws.com
x-amz-content-sha256: <sha256 of body>
x-amz-date: 20260821T153400Z
content-type: application/x-amz-json-1.0
{
"StartTime": "<from, 5 minutes before now>",
"EndTime": "<to, now>",
"MetricDataQueries": [
{
"Id": "m1",
"MetricStat": {
"Metric": {
"Namespace": "AWS/MediaLive",
"MetricName": "ActiveAlerts",
"Dimensions": [
{ "Name": "ChannelId", "Value": "primary-na" }
]
},
"Period": 60,
"Stat": "Average"
},
"ReturnData": true
}
]
}
----- CloudWatch Logs pull (reconstructed) -----
GET logs.us-east-2.amazonaws.com/?logGroupName=%2Faws%2Fmedialive%2Fchannel-events&startTime=...&endTime=...&limit=100&filterPattern=%7B+%24.eventType%3D%22ChannelStateChange%22+%7D HTTP/2
authorization: AWS4-HMAC-SHA256 Credential=AKIA***/20260821/us-east-2/logs/aws4_request, SignedHeaders=...; Signature=...
host: logs.us-east-2.amazonaws.com
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date: 20260821T153400Z
----- response (reconstructed) -----
{
"MetricDataResults": [
{
"Id": "m1",
"Label": "AWS/MediaLive ActiveAlerts — primary-na",
"Timestamps": ["…", "…", "…", "2026-08-21T15:32:00Z"],
"Values": [0, 0, 0, 14],
"StatusCode": "Complete"
}
]
}
{
"events": [
{
"logStreamName": "primary-na/2026/08/21/15",
"timestamp": 1755787900000,
"message": "EVENT ChannelStateChange ChannelId=primary-na State=DEGRADED PreviousState=RUNNING",
"ingestionTime": 1755787903000
}
],
"searchedLogStreams": [
{ "logStreamName": "primary-na/2026/08/21/15", "searchedCompletely": true }
]
}The Streamwake-direction trace.
The agent joins the captured CloudWatch evidence against the CDN/encoder/DRM signals on the same bus, ranks root cause with a typed confidence score, and tags the remediation actions on the same incident timeline. Every action is tagged direction: 'reconstructed' so a reader cannot mistake this trace for an actual Streamwake observation.
{
"incident_id": "ckq3xrenamexz-p95jadi-cmh",
"direction": "reconstructed",
"checked_at": "2026-08-21T15:34:00Z",
"context": "Editorial reconstruction of an AWS ActiveAlerts + CloudFront cache-miss latency fan-out to demonstrate the +Amazon CloudWatch evidence pull + bi-directional emit. No actual outage; figures reconstructed from documented AWS MediaLive + Amazon CloudFront behavior for illustration.",
"ranked_hypotheses": [
{
"rank": 1,
"slug": "aws_medialive_active_alerts_vs_origin_classification",
"confidence": 0.91,
"evidence_signals": [
"aws.medialive.ActiveAlerts = 14 vs 0 baseline, on primary-na channel",
"aws.medialive.ChannelStateChange events captured by CloudWatch Logs FilterLogEvents on /aws/medialive/channel-events (DEGRADED after RUNNING)",
"cloudfront.cache_miss_latency_p95_ms = 2,184 ms vs 812 ms baseline on iad-cmh regional edge PoPs (paired, not collapsed — same window)",
"streamwake.cloudwatch.evidence_datapoints += 1 metric + N log rows per the collect response"
],
"classification_cross_layer": {
"origin_side": "aws_medialive_active_alerts_with_encoder_profile_drift_remediation",
"edge_side": "cloudfront_cache_miss_latency_fan_out_vs_origin_shield_miss_remediation"
},
"recommended_fix": [
"retune_aws_medialive_encoder_profile_for_primary_na",
"reroute_cloudfront_egress_off_iad_cmh_affected_PoPs",
"repin_origin_shield_tier_assignment",
"rewrite_cloudwatch_logs_filter_pattern_to_routing_contract_revision_next"
]
},
{
"rank": 2,
"slug": "cloudwatch_logs_filter_skew",
"confidence": 0.07,
"evidence_signals": [
"channel-state events matched by { $.eventType = "ChannelStateChange" } — narrow but unfiltered by ChannelId",
"existing filter does not yet include the next routing-contract revision"
]
},
{
"rank": 3,
"slug": "metric_vs_classification_skew",
"confidence": 0.02,
"evidence_signals": [
"ActiveAlerts metric vs source rows in CloudwatchEvidence show consistent burned-vs-baseline delta across 4 datapoints"
]
}
],
"evidence_pulled": [
{
"kind": "metric",
"namespace": "AWS/MediaLive",
"metric_name": "ActiveAlerts",
"datapoints": 4,
"max": 14,
"captured_at": "2026-08-21T15:34:00Z"
},
{
"kind": "log",
"log_group": "/aws/medialive/channel-events",
"filter_pattern": "{ $.eventType = "ChannelStateChange" }",
"events": 1,
"captured_at": "2026-08-21T15:34:00Z"
}
],
"governed_actions": [
{
"id": "evt-2026-08-21-retune-medialive-profile",
"type": "retune_aws_medialive_encoder_profile",
"scope": "channel=primary-na",
"verifies": ["aws.medialive.ActiveAlerts trend down within window", "no regression on AWS/MediaLive encoder_health metric"],
"approval_required": false
},
{
"id": "evt-2026-08-21-reroute-cloudfront",
"type": "reroute_cloudfront_egress_off_brownout_PoP",
"scope": "deployment=primary-na",
"verifies": ["cloudfront.cache_miss_latency_p95_ms trend down on affected PoPs"],
"approval_required": false
},
{
"id": "evt-2026-08-21-repin-origin-shield",
"type": "repin_cloudfront_origin_shield_tier_assignment",
"scope": "distribution=primary-na",
"verifies": ["aws.cloudfront.origin_shield_miss counter cleared on the affected regional tier"],
"approval_required": false
},
{
"id": "evt-2026-08-21-emit-cloudwatch-metric",
"type": "publish_metric_to_cloudwatch",
"scope": "namespace=Streamwake/Remediation",
"metric_payload": {
"MetricName": "streamwake.<account>.cloudwatch.remediation.applied",
"Value": 1,
"Unit": "Count",
"Dimensions": [
{ "Name": "IncidentId", "Value": "ckq3xrenamexz-p95jadi-cmh" },
{ "Name": "Classification", "Value": "aws_medialive_active_alerts_vs_origin_classification" },
{ "Name": "Confidence", "Value": "0.91" }
]
},
"verifies": ["Dashboard panel that flagged the ActiveAlerts drift re-renders the loop-closure metric the same window"],
"approval_required": false
}
]
}Why this lands cleanly on the CloudWatch side.
CloudWatch is the AWS-side metrics + logs bus the rest of the AWS stack already writes into — AWS/MediaLive, AWS/MediaPackage, AWS/CloudFront, AWS/Lambda, plus the CloudWatch Logs log groups an AWS account surfaces as first-class signals. The +Amazon CloudWatch integration reads that bus via SigV4 + sts:AssumeRole, layers the streaming-domain classification the CloudwatchCorrelationsPanel renders under the ranked hypothesis list, and emits the remediation back as streamwake.<account>.cloudwatch.remediation.applied on the same account so the same CloudWatch Dashboard panel the operator already renders re-renders the loop-closure the same window the incident closes.
The cross-layer recovery checklist.
What to verify on the AWS side AND the streaming side before closeout, in the order the agent loop runs them.
- 1.
AWS/MediaLive ActiveAlerts → 0 across all channel cohorts; encoder_health metric back inside its baseline band.
- 2.
Amazon CloudFront CacheHitRate back inside its per-PoP baseline; p95 OriginLatency cleared on iad-cmh affected PoPs.
- 3.
Origin-shield miss counter cleared on the affected regional tier (no fallback-to-origin direct path).
- 4.
CloudWatch Logs /aws/medialive/channel-events continues to report DEGRADED → RUNNING transitions on the next window without regressing.
- 5.
streamwake.<account>.cloudwatch.remediation.applied = 1 emitted on the same CloudWatch namespace, with Dimensions { IncidentId, Classification, Confidence } the Dashboard panel renders.
- 6.
Players in the affected region back to < 0.5% rebuffer rate across a 5-minute window. POSTMORTEM posted; PRs routed to the encoder-profile + origin-shield teams.
See the CloudWatch evidence pull live. On your AWS account.
Book a 20-minute walkthrough on your AWS MediaLive + Amazon CloudFront distribution — we'll show how the +Amazon CloudWatch integration pulls the AWS/MediaLive ActiveAlerts drift + CloudWatch Logs FilterLogEvents as evidence, layers the streaming-domain classification the CloudwatchCorrelationsPanel renders under the ranked hypothesis list, and emits the loop-closure back as a CloudWatch metric the same Dashboard panel renders the next window.