Tarun Upaday.

Operating Notes

The first question in any outage: whose system is it?

When a number moves and nobody can say why, the instinct is to check two things: our code, and their status page.

Both are usually clean. The bug is rarely in your system or in theirs. It’s in the layer between them — the assumption baked into how one system’s output became another system’s input, months before anyone was looking at it.

Three places a failure can live

Any integrated system has three places something can break, and they require different diagnostic moves.

Your system can be wrong: a bad deploy, a race condition, a query that used to be fast and isn’t anymore. This is the failure engineers are trained to look for first, because it’s the one you can fix unilaterally.

Their system can be wrong: a vendor outage, a rate limit, a silent schema change on an API you don’t control. This is the failure that’s easiest to blame and hardest to do anything about directly.

The handoff can be wrong: the data was valid on both ends and still produced a bad outcome, because a field meant one thing on one side and something subtly different on the other. A status code that used to mean “cancelled” now also covers “cancelled and rebooked.” A timestamp in local time landing in a system that assumed UTC. This is the failure nobody owns, which is exactly why it survives the longest.

Check the boundary first, not last

Most postmortems start at one end and work toward the middle: “our code looked fine, so we escalated to the vendor,” or “their status page was green, so we assumed it was us.” That order feels efficient. It isn’t, because the boundary is the one place neither team is independently monitoring.

The faster path is to check the handoff before either endpoint. Pull the actual payload that crossed the boundary at the moment things went wrong, not a downstream table that summarizes it. Downstream tables are built on the assumption that the handoff worked; if the handoff is what broke, everything built on top of it will look consistent with itself and wrong about reality at the same time.

Don’t trust a dashboard that’s downstream of the bug

This is the trap that costs the most time. If a metric is computed from data that already passed through the broken handoff, the metric will confirm whatever story you already believe, because it was built assuming the handoff was reliable. A dashboard showing bookings down twelve percent doesn’t tell you whether demand dropped, whether your system mishandled valid data, or whether an upstream partner changed something and your reconciliation logic quietly absorbed the difference instead of flagging it.

The question to ask isn’t “what does the dashboard say.” It’s “what data did the dashboard never see because it was dropped, miscategorized, or silently reinterpreted before it got there.”

A pattern worth checking for

A booking count drops and the obvious read is demand. The more useful first move is to pull the raw feed from the airline partner for the same window and diff it against what your system recorded, field by field, before touching either team’s code. If a carrier changed a schedule or a routing code and your reconciliation logic interpreted the change as a cancellation instead of a reschedule, the booking count will look exactly like a demand problem. It isn’t one. It’s a handoff that quietly redefined what a field meant, and every dashboard built on top of that field will report the story the handoff told it, not the story that actually happened.

Fix the boundary, not the symptom

Once the layer is identified, the fix has to live at the same layer as the failure. Patching the downstream number back to what it “should” be solves this incident and guarantees the next one, because the interpretation gap that caused it is still there. The real fix is a contract: an explicit, versioned understanding of what each field means on each side, with validation that fails loudly the moment the assumption breaks, instead of failing quietly into a dashboard that looks fine.

The postmortem question worth asking every time isn’t just what broke. It’s which of the three layers it broke in, and whether the fix closes that layer or just moves the symptom somewhere quieter.