MicrosoftRadarIndependent status for Microsoft services

Disruption with Dependabot Alerts

ResolvedResolved1 version

Facts

Source
GitHub Status · original
Started
(21 days ago)
Ended
· lasted 3 h 53 min
How it ended
Microsoft marked it resolved.
First captured by us
Last changed
(43 min ago)

Last text from Microsoft

We identified that an advisory-publication driven fan-out was overwhelming Dependabot alert processing on repositories by saturating the queue with its own events, delaying processing of events related to repository pushes.<br /><br />We have now isolated this fan-out and applied a throttle so it runs at a fixed rate and cannot delay other events. This remediation has successfully prevented a new fan-out process impacting users.

Quoted from GitHub Status (source), captured . Reproduced for reporting and archival purposes.

Official updates

Posted by GitHub Status, newest first.

  1. Resolved

    We identified that an advisory-publication driven fan-out was overwhelming Dependabot alert processing on repositories by saturating the queue with its own events, delaying processing of events related to repository pushes.<br /><br />We have now isolated this fan-out and applied a throttle so it runs at a fixed rate and cannot delay other events. This remediation has successfully prevented a new fan-out process impacting users.

  2. Monitoring

    The degradation has been mitigated. We are monitoring to ensure stability.

  3. Investigating

    Some Dependabot alerts continue to be delayed in closing after repository updates address a vulnerability. <br /><br />We're continuing to investigate and work to restore normal processing.

  4. Investigating

    We’re continuing to investigate delays in some Dependabot alerts closing after affected repositories are updated. Alert processing remains backed up, and additional processing capacity is rolling out to help reduce the backlog. We’re monitoring progress and will share another update as we learn more.

  5. Investigating

    We’re continuing to investigate delays in some Dependabot alerts closing after affected repositories are updated. Alert processing remains backed up, and additional processing capacity is rolling out to help reduce the backlog. We’re monitoring the rollout and will share another update as we learn more.

  6. Investigating

    We’re scaling alert processing capacity to reduce the backlog causing delays in some Dependabot alerts closing after affected repositories have been updated. We’re continuing to monitor and will share another update as more information becomes available.

  7. Investigating

    We’re investigating delays in some Dependabot alerts closing after affected repositories have been updated to address a vulnerability. Alert processing is backed up. We’ll share more information as it becomes available.

  8. Investigating

    We are investigating reports of impacted performance for some GitHub services.

What Microsoft changed

Microsoft edits the text of an incident in place. We keep every version we capture (checked every minute) and highlight the difference. Removed text is struck through, added text is marked.

  1. First version we captured

    Captured · Microsoft's own timestamp · revision 7cb4dv68
    Original text as we first saw it

    We identified that an advisory-publication driven fan-out was overwhelming Dependabot alert processing on repositories by saturating the queue with its own events, delaying processing of events related to repository pushes.<br /><br />We have now isolated this fan-out and applied a throttle so it runs at a fixed rate and cannot delay other events. This remediation has successfully prevented a new fan-out process impacting users.