Incident with Webhooks and Actions
Facts
- Source
- GitHub Status · original
- Started
- (4 days ago)
- Ended
- · lasted 1 h 50 min
- How it ended
- Microsoft marked it resolved.
- First captured by us
- Last changed
- (43 min ago)
- Affected services
- GitHub Actions, GitHub webhooks
Last text from Microsoft
On November 9, 2026, from 09:23 to 11:40 UTC, one of our job queue clusters was degraded, delaying webhook deliveries, Actions workflow runs, and Dependabot and dependency graph updates. Until mitigation at 10:37 UTC, an average of 26% of webhook deliveries were delayed, peaking at 30%, and the backlog cleared by 11:40 UTC. Some deliveries were delayed by up to 3.5 hours, and delays for a small share began as early as 01:00 UTC. Actions skips events older than 30 minutes, so some workflow runs did not start. <br /><br />The incident was caused by a degraded host that had slowed one of the cluster's Kafka brokers since November 7. The broker stopped reporting the metrics our alerts rely on, so we did not identify it until it stalled the cluster on November 9. <br /><br />We mitigated the incident by removing the broker from service. We have since added an alert that pages us within 30 minutes when a host slows a broker this way, so we can remove the broker before it affects customers.
Quoted from GitHub Status (source), captured . Reproduced for reporting and archival purposes.
Official updates
Posted by GitHub Status, newest first.
Resolved
On November 9, 2026, from 09:23 to 11:40 UTC, one of our job queue clusters was degraded, delaying webhook deliveries, Actions workflow runs, and Dependabot and dependency graph updates. Until mitigation at 10:37 UTC, an average of 26% of webhook deliveries were delayed, peaking at 30%, and the backlog cleared by 11:40 UTC. Some deliveries were delayed by up to 3.5 hours, and delays for a small share began as early as 01:00 UTC. Actions skips events older than 30 minutes, so some workflow runs did not start. <br /><br />The incident was caused by a degraded host that had slowed one of the cluster's Kafka brokers since November 7. The broker stopped reporting the metrics our alerts rely on, so we did not identify it until it stalled the cluster on November 9. <br /><br />We mitigated the incident by removing the broker from service. We have since added an alert that pages us within 30 minutes when a host slows a broker this way, so we can remove the broker before it affects customers.
Monitoring
The degradation affecting Webhooks has been mitigated. We are monitoring to ensure stability.
Investigating
Webhooks is experiencing degraded performance. We are continuing to investigate.
Investigating
Webhooks is experiencing degraded availability. We are continuing to investigate.
Investigating
The degradation affecting Actions has been mitigated. We are monitoring to ensure stability.
Investigating
We've applied a mitigation and are monitoring for recovery.
Investigating
We're investigating delays in Actions runs start times and webhook deliveries.
Investigating
We are investigating reports of degraded performance for Actions and Webhooks
What Microsoft changed
Microsoft edits the text of an incident in place. We keep every version we capture (checked every minute) and highlight the difference. Removed text is struck through, added text is marked.
First version we captured
Captured · Microsoft's own timestamp · revisionrfpl3twlOriginal text as we first saw it
On November 9, 2026, from 09:23 to 11:40 UTC, one of our job queue clusters was degraded, delaying webhook deliveries, Actions workflow runs, and Dependabot and dependency graph updates. Until mitigation at 10:37 UTC, an average of 26% of webhook deliveries were delayed, peaking at 30%, and the backlog cleared by 11:40 UTC. Some deliveries were delayed by up to 3.5 hours, and delays for a small share began as early as 01:00 UTC. Actions skips events older than 30 minutes, so some workflow runs did not start. <br /><br />The incident was caused by a degraded host that had slowed one of the cluster's Kafka brokers since November 7. The broker stopped reporting the metrics our alerts rely on, so we did not identify it until it stalled the cluster on November 9. <br /><br />We mitigated the incident by removing the broker from service. We have since added an alert that pages us within 30 minutes when a host slows a broker this way, so we can remove the broker before it affects customers.