5xx Errors in Production: Finding Root Cause in Minutes Instead of Hours

When a service starts throwing 5xx errors, the answer is almost always a change. Here is the archaeology every on-call engineer repeats, and how to collapse it into one question.

It is 2:14am and payment-service is throwing 5xx errors. PagerDuty has done its job. Now the real work starts, and it is probably not the work you think.

The instinct is to look at the failing service: its logs, its metrics, its pods. Sometimes that is right. But experienced on-call engineers know a dirty secret about production incidents: the signal you need is almost never in the service that is failing. It is in whatever changed upstream of it.

Incidents are usually changes wearing a disguise

Services that ran fine yesterday rarely fail spontaneously today. Overwhelmingly, a sudden 5xx spike traces back to a change somewhere in the system: a deploy that landed a regression, an infrastructure change that tightened a limit, a feature flag flipped for an experiment, an autoscaling event that collided with a concurrency cap, a certificate or credential that quietly expired, or an upstream dependency that did any of these things.

This is good news, in principle. It means the diagnostic question is not the open-ended “what is broken?” but the bounded “what changed in the window before the errors started?”

The 20-minute archaeology session

The reason 2am incidents take hours instead of minutes is that the answer to that bounded question is scattered across systems that do not share a timeline. The deploy history lives in your CI tool. Infrastructure changes live in CloudTrail. Flag flips live in your feature flag service. Scaling events live in the cloud console. The dependency map lives, if anywhere, in someone’s head.

So the on-call engineer opens six tabs and becomes an archaeologist: aligning timestamps by hand, across time zones, under adrenaline, while the error rate climbs. Twenty minutes is the good version of this. And it happens before any actual fixing begins.

Then comes the second half of the investigation, which teams skip at their peril: blast radius. Root cause tells you what to fix; blast radius tells you who else is on fire. If three downstream services depend on payment-service, the difference between knowing that and not knowing it is the difference between one incident and four separate pages.

The correlation is mechanical. Treat it that way.

Look at what the archaeology session actually consists of: gather change events from every source, overlay them on the incident window, rank by proximity to the failing service, check dependencies. There is no judgment in it until the very last step. It is mechanical work, performed by the most expensive and least mechanical resource available: a senior engineer at 2am.

This is the work we built Oscar to eat. Because Oscar maintains a live context graph of the environment, including deploy history, change events, and the dependency structure between services, the question “payment-service is throwing 5xx errors, what happened in the last hour?” is a single correlated query rather than six tabs. The answer comes back in the shape an on-call engineer actually needs: the deploy that landed at 2:02, the concurrency limit it removed, the three downstream services in the blast radius, and a proposed fix that executes only when a human approves it.

The engineer’s judgment stays exactly where it belongs, on the decision. The archaeology goes away.

The post-mortem writes itself, if you let it

One more thing falls out of doing this correctly: the timeline you reconstruct during the incident is the same timeline your post-mortem needs after it. Teams that assemble post-mortems from memory two days later produce fiction with good intentions. An agent that walked the change history at 2am can draft the timeline before standup, and the humans can spend their review arguing about lessons instead of about what happened when.

Next time the pager fires, try asking the bounded question first: what changed? If answering it takes more than a few minutes, that is the gap. The free Operator Edition closes it on your own environment, and the first 2am it saves you pays for the thirty-minute setup many times over.

Key Takeaways

Key points

✓
Most production 5xx incidents trace back to a change: a deploy, a config edit, a flag flip, a scaling event, or an expired credential.
✗
The first 20 minutes of most incidents are archaeology: reconstructing what changed, across tools that do not share a timeline.
✓
Blast radius is as important as root cause: what else depends on the failing service determines who you wake up.
✓
The correlation is mechanical: overlay changes on the incident window and rank by proximity. Machines do this better than adrenaline does.
✓
A post-mortem timeline reconstructed automatically during the incident is more accurate than one assembled from memory two days later.

Frequently Asked Questions

What usually causes sudden 5xx errors in production?

In most environments the trigger is a change rather than spontaneous failure: a deployment with a regression, an infrastructure or config change, a feature flag flip, an autoscaling event hitting a limit, an expired certificate or credential, or an upstream dependency doing any of the above. That is why the fastest diagnostic question is not 'what is broken?' but 'what changed in the window before the errors started?'

How do I find what changed in my environment during an incident?

Manually: check your deploy pipeline history, CloudTrail or the equivalent audit log, feature flag audit logs, autoscaling events, and recent infrastructure-as-code applies, then align their timestamps against the incident start. This cross-tool correlation is exactly what an agent with a live context graph automates: it overlays every change source on the incident window and ranks candidates by dependency proximity to the failing service.

Can Oscar find the root cause of an incident?

Oscar correlates deploys, config changes, autoscaling events, and audit log activity against the incident window, walks the dependency graph to identify blast radius, and returns the likely root cause with evidence and a proposed fix that executes only with your approval. It also drafts the post-mortem timeline from what it found.

Brian Kathman is the CEO and co-founder of OpsCanvas. The twenty-minute archaeology session described here is the single most repeated ritual in cloud operations, and eliminating it was one of the first things Oscar was built to do. Incidents will keep happening. Reconstructing the same timeline by hand at 2am does not have to.