Incident with Dependabot

ResolvedResolved1 version

Facts

Source
GitHub Status · original
Started
(14 days ago)
Ended
· lasted 2 h 30 min
How it ended
Microsoft marked it resolved.
First captured by us
Last changed
(43 min ago)

Last text from Microsoft

On October 30, 2026, between 20:15 and 21:56 UTC, an advisory-publication-driven fan-out caused an unexpected saturation of network connections to the Dependabot alerts datastore. During this period we observed a high rate of write failures on this datastore; users saw elevated failure rates when dismissing or resolving Dependabot alerts, as well as failures when linking Dependabot update pull requests to Dependabot security alerts. We also observed delays in creating and updating security alerts that extended through November 2, 2026.<br /><br />The unexpected fan-out was caused by a missing job queue throttle after the queues for the fan-out work were migrated as part of system maintenance. We have restored the fan-out throttle and further isolated and throttled these processes to guard against further disruptions.

Quoted from GitHub Status (source), captured . Reproduced for reporting and archival purposes.

Official updates

Posted by GitHub Status, newest first.

  1. Resolved

    On October 30, 2026, between 20:15 and 21:56 UTC, an advisory-publication-driven fan-out caused an unexpected saturation of network connections to the Dependabot alerts datastore. During this period we observed a high rate of write failures on this datastore; users saw elevated failure rates when dismissing or resolving Dependabot alerts, as well as failures when linking Dependabot update pull requests to Dependabot security alerts. We also observed delays in creating and updating security alerts that extended through November 2, 2026.<br /><br />The unexpected fan-out was caused by a missing job queue throttle after the queues for the fan-out work were migrated as part of system maintenance. We have restored the fan-out throttle and further isolated and throttled these processes to guard against further disruptions.

  2. Monitoring

    The degradation has been mitigated. We are monitoring to ensure stability.

  3. Investigating

    We have mitigated impact and are seeing recovery. We are monitoring for full recovery.

  4. Investigating

    We are continuing to investigate elevated errors when viewing some Dependabot alerts. We’ll provide more updates as we learn more.

  5. Investigating

    We are investigating elevated errors when viewing some Dependabot alerts.

  6. Investigating

    We are investigating reports of impacted performance for some GitHub services.

What Microsoft changed

Microsoft edits the text of an incident in place. We keep every version we capture (checked every minute) and highlight the difference. Removed text is struck through, added text is marked.

  1. First version we captured

    Captured · Microsoft's own timestamp · revision bplqj6jz
    Original text as we first saw it

    On October 30, 2026, between 20:15 and 21:56 UTC, an advisory-publication-driven fan-out caused an unexpected saturation of network connections to the Dependabot alerts datastore. During this period we observed a high rate of write failures on this datastore; users saw elevated failure rates when dismissing or resolving Dependabot alerts, as well as failures when linking Dependabot update pull requests to Dependabot security alerts. We also observed delays in creating and updating security alerts that extended through November 2, 2026.<br /><br />The unexpected fan-out was caused by a missing job queue throttle after the queues for the fan-out work were migrated as part of system maintenance. We have restored the fan-out throttle and further isolated and throttled these processes to guard against further disruptions.