"In critical systems, discovering and communicating a failure in two minutes instead of two hours is worth millions. The fear of speaking up costs much more than the error itself."
The terminal responded with a line you did not expect. A migration locked the main orders table, a deployment crashed the payment screen, or an environment variable was accidentally overwritten. Your heart races instantly.
Every software engineer, database administrator, and leader operating real systems has felt this drop in their stomach. The difference between a controlled incident in fifteen minutes and a crisis that destroys customer trust lies not in never making mistakes, but in what you and your leadership do in the first ten minutes after noticing the problem.
Being paralyzed by shame or trying to fix everything alone in the dark, hoping no one notices, is the fastest way to turn a simple glitch into an operational disaster.
Why do good teams seem to make more mistakes? Amy Edmondson's discovery at Harvard
In 1999, researcher Amy Edmondson from Harvard University published an empirical study in the scientific journal Administrative Science Quarterly involving 51 work teams in high-pressure environments (including medical teams where a dosage error can cost lives).
When she crossed the teams' quality indicators with the number of recorded failures, she found a result that seemed contradictory at first glance: the most competent and effective teams exhibited higher rates of reported errors than the weaker teams.
Upon thoroughly investigating the qualitative and quantitative data, Edmondson discovered what was actually happening: the best teams did not make more mistakes. They possessed psychological safety to speak the truth immediately as soon as they noticed an anomaly, allowing the group to correct the deviation in the very first minute. Meanwhile, in environments dominated by fear and public humiliation, professionals swept mistakes under the rug until the problem exploded beyond the possibility of reversal.
In software engineering and infrastructure operations, this scientific discovery translates into a golden rule: the recovery speed of an incident depends directly on how quickly the person who pressed the button has the courage to raise their hand.
Here is the practical playbook I have learned to apply over more than fifteen years operating databases and critical systems.
1. Stop the degradation before looking for the root cause
In moments of crisis, the absolute priority is to stem the impact on users and protect data integrity. Investigating the detail of the line of code that caused the bug is fundamental, but that waits for the minute after operations are restored. In the DORA (Accelerate State of DevOps) framework, this capability is measured by MTTR (Mean Time to Restore).
Before taking any impulsive action in the terminal:
- Stop changing variables in the dark: running rushed commands in production without review, hoping for a fix through trial and error, almost always creates a second incident on top of the first;
- Trigger an immediate rollback: if the previous version was stable and there is a deployment rollback procedure or feature flag, revert to the safe state first. The rollback buys time and peace of mind to diagnose the problem in staging;
- Protect data integrity: if there is a risk of corrupting financial or registration records, activate the circuit breaker or put the affected route into controlled maintenance.
2. Communicate with clarity, facts, and zero panic
The biggest mistake during an incident is not making the technical failure; it is leaving the team, support, and leadership in the dark while tickets start coming in. Mature communication does not offer dramatic apologies or invent excuses; it delivers actionable context.
Use a straightforward four-line structure in the team's official incident channel:
- What happened: "At 14:32, during the deployment of version 2.4, the creation of an index blocked reads and writes on the main orders table."
- Observed impact: "Users receive a timeout error when completing checkout. Previously recorded data remains intact."
- Action in progress: "We cancelled the database operation and initiated a rollback to version 2.3. Estimated normalization in 7 minutes."
- What is needed now: "I need whoever is on database on-call duty to monitor the release of stuck connections and confirm the drop in latency."
This clarity instantly disarms panic. It shows the entire company that the problem has a diagnosis, an owner taking action, and a verification timeframe.
3. Isolate evidence from guesswork
When an important system goes down, it is common for ten people to join the same call suggesting ten different theories at the same time. Incident command discipline, documented in the classic Google Site Reliability Engineering (SRE) Book, requires separating roles and protecting focus:
- Numerical facts over hypotheses: look at real telemetry (CPU curve, active database connections, HTTP 5xx error rate, p95 latency, and logs with exact timestamps) before going out and restarting servers on a hunch;
- Separate those who operate from those who communicate: while one person focuses on executing the rollback or unlocking the database, another person must take over periodic updates for stakeholders;
- Record the timeline in real time: keep an open document noting the exact time of each symptom and each applied command.
4. The Swiss Cheese Model and the blameless post-mortem
After the system comes back online and the charts stabilize, the stage that separates amateur teams from elite engineering begins: the blameless post-mortem.
In 1990, psychologist James Reason, a professor at the University of Manchester, published the book Human Error, introducing the famous Swiss Cheese Model for analyzing accidents in aviation, nuclear power plants, and complex engineering.
Reason proved that in any well-designed critical system, there are multiple layers of defense (like slices of Swiss cheese placed side by side). A serious accident only happens when the holes in all the slices line up at the same time.
Therefore, if a single command typed by a developer or DBA was enough to take down the entire company, the real problem was never the person. The problem lay in the holes of the previous barriers:
- Why was a destructive command allowed to run directly in production without dual confirmation or peer review?
- Why didn't the CI/CD pipeline catch the schema lock in an automated testing environment?
- Why was there no canary deployment that limited the error to 1% of traffic before affecting 100% of customers?
As John Allspaw and the Google SRE team summarize, punishing the person and keeping the process intact guarantees only two things: that next time the team will hide the mistake out of fear, and that someone else will fall into the exact same trap months later.
Three questions before your next deploy
On your next major delivery to a production environment, before pressing the final button, pause for sixty seconds and answer these three questions:
- What is our rollback command? If this change exhibits abnormal behavior at the third minute, what is the exact and tested procedure to roll back in under five minutes?
- What number proves it worked? Which objective business and infrastructure metric will I look at immediately after the deploy to confirm the system is healthy?
- What guardrail did we leave for the future? If something fails today, what automatic barrier will we add to the pipeline by Friday so that this entire class of error ceases to exist?
Psychological safety is not about lowering the bar of requirements. It is about combining high technical standards with total transparency to resolve issues quickly, learn genuinely, and build increasingly resilient systems.
Continue deepening your knowledge: also read How to choose a path in technology without guessing the future and What to do when you get stuck on a project. Also learn about the work in Executive Engineering Leadership and the Case Studies.
References and verifiable readings
- Edmondson, A. (1999). Psychological safety and learning behavior in work teams. Administrative Science Quarterly, 44(2), 350-383.
- Reason, J. (1990). Human Error. Cambridge University Press.
- Lunney, J., Lueder, S., & Google SRE Team (2016). Postmortem Culture: Learning from Failure & Managing Incidents. In Site Reliability Engineering: How Google Runs Production Systems. O'Reilly Media.
- Forsgren, N., Humble, J., & Kim, G. (2018). Accelerate: The Science of Lean Software and DevOps / DORA Metrics. IT Revolution Press.
Good ideas keep the conversation going.
Share this article or publish the visual deck to your Stories and Feed.

Cesar Zanis
15+ years leading software architecture, database performance, FinOps, and applied AI across enterprise operations. Built and scaled an engineering organization from zero to 20+ developers.
Enjoyed this article?
I publish practical analyses on critical systems architecture, Cloud FinOps, data, AI, and engineering leadership directly on LinkedIn.
Continue Reading
Take this analysis further with data and sources
All visual cards of this article are ready above to share on your Stories, Feed, WhatsApp, or LinkedIn with full sources.



