What 3 A.M. Incidents Taught Me That AWS Certifications Never Did

I hold AWS certifications and I encourage my engineers to get them. They are a fine way to learn the breadth of a platform. But nothing on any exam prepared me for the specific feeling of watching order volume flatline at 2:47 a.m. during dinner rush in another timezone, with a dashboard full of green checkmarks and a phone full of red alerts.

We review every incident at Rafeeq, blamelessly and in writing. After twenty of them I went back through the reviews looking for patterns, and the biggest one changed how I think about engineering skill entirely.

The fix is almost never the hard part

Across our incidents, the median fix took six minutes. Restart the consumer, roll back the deploy, flip the feature flag, scale the pool. Six minutes.

The median time to understand what was happening was thirty-four minutes. That is where incidents live and die, and it is exactly the skill no certification measures. Exams test whether you know what a service does. Incidents test whether you can figure out what your system is doing when three services interact in a way nobody designed.

The lesson for where to invest: observability and system knowledge pay five times more than memorized facts, because they attack the 34-minute phase, not the 6-minute one.

Three incidents, three lessons certifications skip

The certificate that expired at midnight. A payment provider's callback started failing because a TLS certificate on our side expired. Every individual system was "healthy." Orders were paid; our platform did not know. The exam question would be about ACM auto-renewal. The real lesson: the certificate that kills you is the one issued manually two years ago by someone who left, and the fix is an inventory and an alert 30 days out, not deeper TLS knowledge.

The replica that lagged only on Thursdays. Stale reads caused a slow drip of support tickets: orders marked paid that a replica did not know about yet. It reproduced only under a weekly analytics job that saturated replica I/O. Certifications teach what replica lag is. The incident taught me that the query causing your outage is often owned by a different team, running on a schedule nobody in the incident channel knows exists. Now our first incident question is: what else is running right now?

The retry storm we built ourselves. A downstream service slowed down, our services retried, the retries multiplied the load, and a slowdown became an outage. Every engineer involved could have drawn the exponential-backoff slide from memory. Under pressure, in code written across three years by ten people, our actual retry behavior was aggressive, uncoordinated, and unknown to everyone until it hurt us. Knowing the pattern and knowing your system's pattern are different skills. Only one of them is examinable.

What actually makes a team calm at 3 a.m.

• Runbooks written by tired people, for tired people. Short steps, exact commands, no prose. We write them during the postmortem while the pain is fresh.
• One incident commander, decided in advance. The worst minutes in our early incidents were five smart people debugging five different theories in one channel. Now one person directs, others investigate. Cut our understanding phase nearly in half.
• Blameless reviews, without exception. The night someone can say "I merged that, and here is what I saw" without fear is the night your incident data becomes accurate. Punish honesty once and you will debug with incomplete information forever.
• Practice on purpose. We run a game day each quarter: kill a replica, expire a token, block a queue, on staging, on a Tuesday afternoon. The first one was humbling. The fourth one was boring, which is the goal.

So are certifications useless?

No. Here is the honest split I share with my team:

| Certifications train | Incidents train | |---|---| | Breadth of what services exist | Depth of how your system behaves | | The happy path and best practices | The failure path and trade-offs under pressure | | Individual knowledge | Team coordination and communication | | Confidence before the job | Judgment on the job |

Get the certification, especially early in your career. It opens doors and builds vocabulary. Just know that the engineer I promote is rarely the one with the most badges. It is the one who, at 3 a.m., asks the calm question that cuts the 34 minutes down to ten: "What changed?"

That question is not on any exam. It should be.