Question 6
Handling production issues
A critical bug appears in production affecting many users. How do you lead the response, both technically and with stakeholders?
Answer outline
Start by stabilizing the system. Assess the impact, then stop the bleeding with whatever is fastest and safest: a rollback, a feature flag, or a hotfix. Name a DRI (directly responsible individual) right away and bring in engineering, product, and QA at the same time.
Use logs, crash reports, and metrics to pin down the scope and work toward the root cause. Keep the debugging group small and focused. Update stakeholders on a fixed schedule from the start, even while the picture is incomplete.
Once you understand the cause, ship the smallest safe fix, validate it, and roll it out carefully while watching the key metrics. After resolution, run a blameless postmortem. Cover what happened, why, and which changes to testing, monitoring, or process will prevent a repeat.
Principles
- Stabilize first with a rollback, flag, or hotfix before spending time on diagnosis.
- Assign a single DRI immediately, because clear ownership prevents duplicated effort.
- Ship the smallest safe fix under pressure, then validate it before the full rollout.
- Tell stakeholders when the next update is coming, and keep that promise even when there's no news.
- Run a blameless postmortem and turn what you learn into changes to tests, alerts, or process.
