Inside the GitHub Outage: What Really Happened?

A routine maintenance job took a network link down for less than a minute. That was enough. By the time connectivity returned, two data centres on opposite coasts of the United States each believed they held the authoritative copy of GitHub's primary database. The incident lasted 43 seconds in the network and 24 hours and 11 minutes in the database — a disproportion that should frighten anyone who designs replicated systems.

How the architecture looked before it broke

GitHub runs its non-Git metadata — pull requests, issues, authentication, background jobs — in MySQL. The data is split across many functional clusters, each with a primary that accepts writes and up to dozens of read replicas spread across two physical data centres plus cloud capacity. Write traffic goes to the primary; reads are offloaded to the replicas. Cluster topology and failover are managed by Orchestrator, an open-source tool GitHub had been using for years, which uses a Raft consensus group to elect which node should be primary.

That design is entirely normal. It is, with local variations, how hundreds of production MySQL deployments look. What made GitHub's setup unusual is that its clusters spanned not just two racks or two data centres but two distinct coasts — a choice that would only matter on the day the two coasts lost touch with each other.

Automatic failover did exactly what it was told

On 21 October 2018 at 22:52 UTC, engineers were replacing failing 100G optical equipment when connectivity was lost between the US East Coast network hub and the primary East Coast data centre. Connectivity was restored 43 seconds later. In between, two independent things happened, and neither one was a malfunction.

First, the East Coast primary kept accepting writes. MySQL replication is asynchronous by default, so writes committed on the primary existed only in that server's binary log — they had never been acknowledged by any replica. Second, the Orchestrator nodes on the West Coast and in the public cloud formed a quorum, observed that the primary was unreachable, and promoted West Coast replicas to primary for several clusters. The application tier, told to talk to the new primary, started writing 40 minutes' worth of new data into the West. When the partition healed, the two sides had diverged in a way that could not be merged automatically.

Orchestrator's actions behaved as configured, despite our application tier being unable to support this topology change.

That sentence from GitHub's post-incident analysis is the whole incident in miniature. The automation did not fail. It did exactly what it was built to do — promote a replica when the primary becomes unreachable. The problem is that the rules the automation was built around assumed a failure mode that did not occur. The primary was not dead; it was unreachable. Those are different events with different remedies, and the automation could not tell them apart.

The recovery is the hard part

Restoring availability would have taken minutes. Restoring correctness took a day. GitHub's team made an explicit choice, stated in the postmortem: data integrity over site usability and time to recovery. Webhook delivery was paused, GitHub Pages builds were halted (about 80,000 builds and more than 5 million webhook events ended up queued), and the site served stale or degraded reads for most of the following day.

The recovery sequence shows why. Engineers reloaded primaries in the East Coast from backups pulled out of remote storage — decompressing, checksumming and loading multiple terabytes took hours. The replication topology had to be rebuilt and the large replica pool re-synced. Then came the delicate part: capturing the MySQL binary logs containing the writes that had been committed on the East primary but never shipped to the West, and reconciling them against the 40 minutes of new writes that had landed on the promoted West primaries in the meantime. No script does that. Every cluster had to be examined by people who understood what each write meant.

• Availability can be restored by a script; consistency usually cannot.
• Cross-region automatic failover trades a rare outage for a rarer, worse one.
• Human-in-the-loop promotion is slower and frequently correct.

What they changed afterwards

The most important remediation was not a fix to a crash or a bug. GitHub changed Orchestrator's configuration so that database primaries could no longer be promoted across regional boundaries automatically. Leader election within a region remained automated; promoting a replica on the other side of a continent, where it would be guaranteed to be missing acknowledged writes, became a deliberate human decision. The sudden introduction of cross-country latency, as their analysis put it, had been a major contributing factor — and the fix was to remove the automation that could create it.

The operational changes were just as instructive as the technical ones. GitHub began aligning the automation's topology with what the application tier could actually support, and made the reporting of such incidents richer and more granular. It invested in chaos and fault-injection practices — deliberately breaking pieces of the topology in test environments to observe how the application and the orchestrator interacted, rather than discovering the mismatch during a real 43-second partition. That is the discipline of learning to swim by flooding the pool on purpose, and it is the only known cure for the kind of interaction failure this incident exposed, where each component behaves correctly and the system still dies.

There is a quiet heroics in the recovery that deserves a mention too. The reconciliation work — capturing the binary logs that held the un-replicated writes, matching them against the 40 minutes of new writes that had landed on the other coast, and folding them together without losing or duplicating a single user action — was done by engineers under enormous pressure, and GitHub's published postmortem records that no user data was permanently lost. When a database splits in two, that outcome is never guaranteed. It is the product of choosing to be slow and careful over being fast and wrong, and of having the courage to say so in public afterward.

They chose data integrity over uptime, and then explained why in public. That is the whole lesson.

The replication modes you can choose

GitHub's default was asynchronous replication: the primary acknowledged a write the moment it was in its own binary log, without waiting for any replica to have it. That gives the lowest write latency and the highest throughput — and the price, as we saw, is that a promoted replica is always missing whatever the primary has not yet shipped. The alternatives are not free either. Synchronous replication makes a commit wait until a replica has flushed it, which bounds the window of possible loss to the moment the two machines disagree — but it adds a round trip to every write and can stall the whole database if the replica falls behind.

The engineering question is never 'which replication mode is best' in the abstract; it is 'how much data am I willing to lose, and under what circumstance.' For a git forge, losing any acknowledged write is unthinkable, which is exactly why GitHub chose to suffer a day of degradation rather than drop a few seconds of writes. For a metrics time-series or a message log, losing a sliding window of recent data under a rare failover may be entirely acceptable, and the synchronous cost is not worth paying. The discipline is to decide this explicitly, in advance, per system — not to discover it mid-incident when your hand is forced by the database.

Availability can be restored by a script; consistency usually cannot.

What to carry into your own systems

First, know whether your replication is synchronous or asynchronous, and what the window of possible data loss actually is. Second, design your failover so the promoted replica is guaranteed to have acknowledged every write the old primary accepted — or accept that you are choosing split-brain. Third, decide in advance, in writing, whether you value consistency over availability, because that decision has to be made while the incident is running, and you will not think clearly then. Fourth, rehearse the recovery, not just the failover. GitHub had rehearsed failing over; nobody had rehearsed reconciling two divergent primaries by hand.

They chose data integrity over uptime, and then explained why in public. That is the whole lesson.

GitHub published one of the clearest postmortems in the industry, with the cause, the sequence and the reasoning all laid out. The value of that document outlives the incident. Every engineer who reads it inherits a 43-second mistake and gets to avoid the 24-hour price.