Incident with Pull Requests
Facts
- Source
- GitHub Status · original
- Started
- (54 days ago)
- Ended
- · lasted 1 h 9 min
- How it ended
- Microsoft marked it resolved.
- First captured by us
- Last changed
- (43 min ago)
- Affected services
- GitHub pull requests
Last text from Microsoft
On September 20, 2026, between 21:46 and 22:24 UTC the Pull Requests service was degraded and pull request merge and test-merge commits were created late, with delays reaching approximately four minutes at peak. Merge commits were delayed rather than lost. Because some Actions workflow runs start only after a pull request's merge commit is created, a subset of workflow runs for pull request events were also delayed. <br /><br />This was due to a routine repository maintenance job for an unusually large repository consuming nearly all of the memory on a single Git storage server, which left that server unable to serve the Git operations used to create merge commits. <br /><br />We mitigated the incident by removing the affected server from service at 22:18 UTC, after which the queued merge commits were created within six minutes. <br /><br />We have capped the memory a single repository maintenance job may consume so that one repository cannot exhaust a server, and we have improved monitoring and alerting on storage server health to reduce our time to detection and mitigation of issues like this one in the future.
Quoted from GitHub Status (source), captured . Reproduced for reporting and archival purposes.
Official updates
Posted by GitHub Status, newest first.
Resolved
On September 20, 2026, between 21:46 and 22:24 UTC the Pull Requests service was degraded and pull request merge and test-merge commits were created late, with delays reaching approximately four minutes at peak. Merge commits were delayed rather than lost. Because some Actions workflow runs start only after a pull request's merge commit is created, a subset of workflow runs for pull request events were also delayed. <br /><br />This was due to a routine repository maintenance job for an unusually large repository consuming nearly all of the memory on a single Git storage server, which left that server unable to serve the Git operations used to create merge commits. <br /><br />We mitigated the incident by removing the affected server from service at 22:18 UTC, after which the queued merge commits were created within six minutes. <br /><br />We have capped the memory a single repository maintenance job may consume so that one repository cannot exhaust a server, and we have improved monitoring and alerting on storage server health to reduce our time to detection and mitigation of issues like this one in the future.
Monitoring
A git fileserver issue caused a brief delay in creating some merge commits - we've isolated the underlying server and already observed recovery.
Monitoring
The degradation affecting Pull Requests has been mitigated. We are monitoring to ensure stability.
Investigating
We are investigating reports of degraded performance for Pull Requests
What Microsoft changed
Microsoft edits the text of an incident in place. We keep every version we capture (checked every minute) and highlight the difference. Removed text is struck through, added text is marked.
First version we captured
Captured · Microsoft's own timestamp · revisionlwhskmh3Original text as we first saw it
On September 20, 2026, between 21:46 and 22:24 UTC the Pull Requests service was degraded and pull request merge and test-merge commits were created late, with delays reaching approximately four minutes at peak. Merge commits were delayed rather than lost. Because some Actions workflow runs start only after a pull request's merge commit is created, a subset of workflow runs for pull request events were also delayed. <br /><br />This was due to a routine repository maintenance job for an unusually large repository consuming nearly all of the memory on a single Git storage server, which left that server unable to serve the Git operations used to create merge commits. <br /><br />We mitigated the incident by removing the affected server from service at 22:18 UTC, after which the queued merge commits were created within six minutes. <br /><br />We have capped the memory a single repository maintenance job may consume so that one repository cannot exhaust a server, and we have improved monitoring and alerting on storage server health to reduce our time to detection and mitigation of issues like this one in the future.