When the War Reached Our Cloud: Evacuating an AWS Region in 6 Hours

The attack that took down AWS Bahrain forced an emergency migration: our DR plan under real fire, and how Kiro moved 63 services in 6 hours, not 3 weeks.

#aws#disaster-recovery#migration#kiro#backup#resilience
Cover image for the article: When the War Reached Our Cloud: Evacuating an AWS Region in 6 Hours

Every disaster-recovery plan I've ever written had a quiet assumption baked into it: the disaster would be technical. A bad deploy, a failed disk, maybe a whole availability zone going dark. I never wrote the sentence "a regional war disables our primary cloud region". And then this summer, that's exactly what happened.

When the conflict between the US and Iran escalated across the Gulf, the AWS Bahrain data center was hit, and me-south-1 (the region closest to our users in Qatar, the region running nearly everything Rafeeq) degraded from "our home" to "unreliable" within the hour. This is the story of the six hours that followed, and what I'd tell any CTO in a region where geopolitics is part of your threat model.

Hour zero: the call you can't prepare for emotionally

You can prepare for it operationally, though. And barely, we had.

Our disaster-recovery posture before the crisis:

  • Backups: automated cross-region copies to ap-south-1 (Mumbai). RDS snapshots every 6 hours, continuous point-in-time recovery, S3 replication on critical buckets. We had deliberately chosen a DR region outside the Gulf. RPO on paper: 15 minutes for databases, near-zero for object storage.
  • A warm skeleton, not a warm standby: our Terraform could theoretically build everything in another region, but it had never been fully exercised. VPCs and IAM existed in ap-south-1; almost no compute did.
  • One cross-region RDS read replica for the orders database, the one system we had decided we could never lose. That decision saved us.

Here's the geography of the problem, and the escape route:

Schematic map of the Gulf showing the impacted AWS Bahrain region me-south-1, our users in Doha, and the migration of 63 services in 6 hours heading 2,700 km east to the ap-south-1 region in Mumbai

The decision: don't wait, evacuate

For the first 45 minutes we did what everyone does: watched the AWS status page, retried failures, hoped. Capacity in me-south-1 was intermittent: some instances healthy, new capacity impossible to launch, latency spiking 20×. Hope is not a strategy, and in a war, "degraded" doesn't trend back toward "fine."

Forty-five minutes in, we made the call: full evacuation to ap-south-1. Not a partial failover, not "wait and see": move everything, permanently, starting with the order path. And crucially: get out of the Gulf entirely, because in a regional conflict, the region is the blast radius.

Our estimate for that migration, made calmly a year earlier during DR planning: about three weeks. We had 63 production services, hundreds of Terraform modules with me-south-1 assumptions baked in, region-specific configs scattered across 40+ repositories, and a database tier serving 34,000 queries per second.

We did it in six hours. Honestly, I still find that number hard to believe, and a big part of it was that we pointed Kiro at the mechanical work.

Where Kiro changed the math

Kiro had been our engineers' agentic IDE for months, mostly for feature work. During the evacuation it became something else: the tireless engineer who does the boring, error-prone 80% while humans make the calls.

What we actually delegated:

  • Region-proofing the infrastructure code. Kiro swept our Terraform and found every hardcoded region, AZ name, AMI ID, and endpoint (across all repositories) and produced the ap-south-1 equivalents as reviewable pull requests. A human approved every change; nobody had to find every change. That sweep alone was budgeted at four days in the old plan. It took under an hour.
  • Config archaeology. Connection strings, SDK clients with implicit regions, queue ARNs, KMS key references buried in services nobody had touched in a year. We gave Kiro the migration spec; it worked through the repos as a task list and flagged the judgment calls it couldn't make alone.
  • DMS migration configs. For the databases without cross-region replicas, Kiro drafted the AWS DMS task definitions, table mappings, and validation queries. Engineers reviewed, corrected two, and ran them.
  • Runbooks under pressure. Every cutover got a generated, service-specific runbook: steps, health checks, rollback procedure. Four hours into an overnight sprint, tired people don't improvise well. Checklists saved us more than once.

Of the 63 services, Kiro generated the region-ready code and configuration for 54. The other nine (the payment integrations, anything touching money or compliance) we did by hand, on purpose.

Line chart of cumulative services migrated over 6 hours, from zero to all 63, with milestones: orders database promoted at hour 1, 50% of traffic at hour 3, 100% at hour 5

The database story: promote, drain, verify

Data is where migrations die, so this is where we were most conservative.

  1. Hour 1: promote the orders replica. The cross-region read replica in ap-south-1 became the new primary. Because it was continuously replicating before the attack, we lost essentially nothing. Effective RPO on the order path: under 30 seconds.
  2. Everything else via snapshot + DMS. Secondary databases restored from the cross-region snapshots, then DMS replayed changes captured while me-south-1 limped along. Two low-traffic services accepted a 4-hour data gap; we chose availability and reconciled later from event logs.
  3. Drain, don't flip. We shifted traffic in stages (10%, 25%, 50%, 100%), watching database load on both sides. The new primary had to earn each increment.

Line chart of database queries per second in both regions across 6 hours: Bahrain draining from 34,000 QPS to zero while the Mumbai region ramps from zero to 34,000, crossing at hour 3

The scariest moment wasn't the promotion. It was hour three, both regions at half load, when a replication lag alarm fired. It peaked at 1.8 seconds and recovered. Total customer-visible downtime across the entire evacuation: 41 minutes, most of it in two planned cutover windows announced in the app beforehand.

The scoreboard

| Metric | Plan (peacetime estimate) | Reality | |---|---|---| | Full region migration | ~3 weeks | 6 hours | | Services moved | 63 | 63 | | Customer-visible downtime | "hours, maybe days" | 41 minutes | | Data loss (order path) | RPO 15 min | < 30 s | | Data loss (worst service) | RPO 6 h | 4 h, reconciled | | Peak replica lag during shift | n/a | 1.8 s | | Latency cost for Doha users | n/a | +24 ms round-trip |

What I'd tell every CTO in this region

  • Geopolitics belongs in your threat model. Not as a slide, but as an architecture requirement. If your DR region shares a body of water with your primary, it shares the same risks. Ours being 2,700 km away in Mumbai is why it was still standing when we needed it.
  • A backup you haven't restored is a hope, not a backup. Our snapshots worked because we'd tested restores quarterly. The Terraform "skeleton region" half-worked because we'd never fully exercised it. That gap is where our three-week estimate came from.
  • Pay for one real cross-region replica on the system you can't lose. Ours cost a few hundred dollars a month for years. It repaid that in the first hour of the crisis.
  • Agentic tooling changes DR math. The reason region evacuation plans take weeks is thousands of small, mechanical, region-specific edits. That is exactly the work an agent like Kiro does relentlessly and a human verifies quickly. Our new DR plan officially assumes it.
  • People first, always. Our team executed this while some of them worried about family across the region. It was one intense overnight sprint: rotations were short, meals were ordered, and nobody was allowed to be a hero alone at 2 a.m. The plan matters less than the people running it.

We're staying in ap-south-1. And the new DR plan has a sentence the old one lacked: "Assume the region is gone and will not come back." I hope we never use it again. We rehearse it quarterly anyway.

Comments

    No comments yet. Be the first to share your thoughts.