The AWS Outage That Broke Half the Internet

When a single AWS region degrades, the visible damage is never limited to customers hosted in that region. It spreads through the control plane. On 7 December 2021, us-east-1 suffered one of the largest internet outages in years, and the pattern it exposed is worth studying because it repeats: most of the failures were not in the services people could see. The failure was in the layer that lets anyone do anything about the first failure.

Data plane versus control plane

An automated activity to scale one of AWS's internal services triggered unexpected behaviour from a large number of clients inside the internal network: a storm of connection attempts and retries that overwhelmed the network devices bridging AWS's internal network — hosting monitoring, internal DNS, authorization and parts of the EC2 control plane — to the main AWS network. Congestion caused failures, failures caused retries, and retries caused more congestion. A well-tested back-off mechanism that had been in production for years failed to dampen the loop in conditions nobody had observed before.

The crucial fact about that day: your running instances kept running. EC2's data plane — the compute itself — was largely untouched, as were S3 objects and DynamoDB data. What failed was the ability to launch new instances, update DNS records, change IAM roles, sign in to the console, or authenticate anywhere. The control plane — the set of APIs that create, scale, inspect and manage resources — was down or impaired for hours.

Failover that requires the control plane cannot survive a control-plane outage.

The blast radius was global, not regional

Here is what makes the incident genuinely frightening, and why the title of this article is barely an exaggeration: services running in other regions failed too. The global STS endpoint that issues security credentials lives in us-east-1, so authentication problems cascaded to workloads anywhere. Route 53's ability to accept changes — DNS control, not DNS answers — is anchored there, so every region's DNS edits were blocked. The AWS console routes root logins through us-east-1, so customers could not even sign in to look at what was broken. IAM's management plane has the same anchor. A region is a fault-isolation boundary for the data plane; it frequently is not one for the control plane.

The self-healing paradox made recovery slower. Automated recovery mechanisms — recycling API Gateway servers, restarting Fargate tasks, autoscaling groups replenishing capacity — all depended on EC2 control-plane APIs that were themselves impaired. Monitoring was also impaired, because CloudWatch ran partly on the same failing network. AWS operators worked partly blind, through degraded tooling, and deliberately acted conservatively to avoid disrupting the many workloads that were still fine. The recovery tail stretched on for hours after the acute incident.

The downstream damage

Outside AWS, the damage was concrete and visible. Amazon's own retail operation slowed; sellers could not print shipping labels, and Amazon Flex delivery drivers were sent home at peak season. Apps as varied as iRobot's vacuum control, networked pet feeders, Netflix, Disney+, Delta's passenger services, Slack, Tinder and Epic Games all reported issues. Canvas, the learning platform used by tens of millions of students, went down during college finals week, and exam proctoring software with it — some exams were simply postponed. Thousands of companies had a bad day because one region's control plane coughed.

The dependency nobody drew

• Global services with a hidden home region — STS, IAM, Route 53 control plane, console login.
• Status pages and monitoring hosted on the infrastructure they report on.
• Multi-region deployments whose deploy pipeline, auth or DNS lives in one region.
• Automated recovery that calls the same APIs that are down.

The 'multi-region on paper' problem deserves emphasis. An architecture that spreads compute across three regions but authenticates through a global endpoint, edits DNS in one place, and deploys from a pipeline in a single region is not multi-region in any meaningful sense. It is single-region with better visuals. The hidden home region becomes a point of failure that defeats all the carefully purchased redundancy.

The recovery tail

AWS's own post-incident summary was unusually candid about why the tail was so long. Monitoring itself was impaired, because the observability tooling ran partly on the very network that was failing — so operators spent the early hours with a partial, lagging picture of their own incident. The internal deployment systems used for remediation were also inside the affected network, so the tooling that would normally fix things quickly was itself one of the things that needed fixing. And the team deliberately chose to be conservative, prioritising the stability of the still-working data planes over fast remediation of the control planes. Each of those three constraints is a lesson: keep your monitoring out of the blast radius, keep your recovery tooling on different infrastructure, and accept that a cautious recovery is slower but safer.

That mix of causes — a routine scaling activity, a latent back-off bug, and control-plane concentration — is exactly what engineers mean when they warn that reliability problems are usually the product of normal operations colliding with unobserved failure modes. Nothing about 7 December 2021 was an act of nature. An automated system did what it was built to do, in conditions its authors had never seen, and the architecture did the rest. The fix was not to build fewer automated systems but to make the system resilient to its own automation: bounded retries, honest back-off, and recovery paths that do not depend on the thing that is down.

A region's bad reputation is earned

It is not an accident that the canonical AWS failure region is us-east-1. It is AWS's oldest and largest region, the launch point for most new services, and the default destination for a large fraction of all accounts — which means it has the largest surface of running workloads, the most global control-plane anchors and the most blast radius. It has also, over the years, produced an outsized share of famous incidents, and forgotten enterprises. The pattern that emerged in 2021 is the strongest argument for treating any single region as the top of your dependency graph.

The practical conclusion is uncomfortable but honest: much of the industry's resilience theatre targets the wrong failure. Teams rehearse losing a data center or an availability zone, but the failures that actually dominate are the shared, logical ones — a global identity endpoint, a central DNS control plane, a common deployment pipeline — that collapse every replica at once. Uptime statistics and 'multi-region on paper' architecture diagrams both flatter us. The truest resilience work is unglamorous: map the global dependencies, move identity and DNS out of a single region, cache what you can, and practise recovering when the thing that holds everything together is the thing that broke.

What to do about it

The homework is unglamorous: enumerate what your recovery path calls, and check whether any of it lives inside the thing you are recovering from. Run game days that take out the control plane rather than just killing a region's compute — because compute can be re-provisioned if the APIs work, and nothing can be re-provisioned if they do not. Cache credentials and configuration locally so services can keep running when the auth endpoint is down. Keep monitoring external to the thing it monitors: a second-region or out-of-band synthetic check is cheap insurance against being blind exactly when you most need eyes. And set up fallback DNS and a secondary way to reach the console before the day you need them.

AWS's post-incident review was open about its own scaling activity as the trigger and about its own monitoring running inside the blast radius. The honest takeaway for everyone else is that control-plane concentration is a design smell you can choose to carry — or to engineer against. Every team that depends on a single global endpoint for auth, DNS or deployment is one bad day in us-east-1 away from discovering their architecture diagram was lying.