Okay. So — last night's load. Before we start, one thing, and I mean it: this is blameless. We're not here to work out who. We're here to work out what — what happened, and what made it possible.
David spends his first thirty seconds on the rules, not the incident. "We're not here to work out who. We're here to work out what." That sentence is the whole meeting. Without it, everyone arrives braced — and people who are braced hide things. The single most valuable thing a facilitator does in a postmortem is make it safe to be accurate.
So if anyone's sitting there with a knot in their stomach — let it go. Genuinely. The pipeline broke. Pipelines break. What I care about is why it stayed broken for eight hours without anyone knowing.
Watch what David makes the subject: not "the load failed" but "why it stayed broken for eight hours without anyone knowing." The break is normal; the silence is the problem. "A knot in your stomach" — he names the feeling in the room out loud, which is what actually releases it. Normalising failure ("pipelines break") isn't lowering the bar; it's aiming at the right target.
Good. Because I was ready to apologise for not seeing it sooner.
You found it before anyone else did, Marco. That's the opposite of a problem.
Priya, take us through the timeline. Just facts — no interpretation yet.
Right. 22:15, the load kicked off as normal. 22:19, it hit the product table and the schema had changed — a new column upstream. The job threw an error, retried three times, and then... it marked itself complete.
"Just facts — no interpretation yet." This separation is the engine of a postmortem. The moment you mix "what happened" with "why it happened", people start defending instead of remembering. Notice Priya's register: timestamps, actions, no adjectives, no "unfortunately", no "I should have". Just the machine's behaviour. Marked itself complete — the flat delivery does the work; she doesn't need to editorialise.
Alert fired at 22:19 as well. It went to the pipeline-alerts channel. Nobody's in that channel overnight — nobody's really in it at all, if I'm honest.
I opened the regional dashboard at ten past eight and it was empty. I thought it was my filter at first. Took me fifteen minutes to work out it wasn't me.
"I thought it was my filter at first." Fifteen minutes lost because a broken report looks exactly like a report with no data. That's not Marco being slow — it's the system failing to distinguish "nothing to show" from "everything is broken." In postmortems, the throwaway details are usually where the real finding hides. A good facilitator listens for them.
And I got the first call at half eight. By nine I'd had four.
And the fix — when did that land?
Marco messaged me at half nine. I found it in about ten minutes, patched the column mapping, kicked off the rerun at 9:50. Reports were green by 10:05.
A postmortem timeline doesn't stop at "it broke" — it runs to "it was fixed." Priya's recovery is fast and unglamorous: found it, patched it, reran it, thirty-five minutes. Recording that matters for two reasons: it tells you whether your response is healthy (it was), and it stops the meeting from being a list of failures. Green = working/passing, the opposite of a red alert.
So — broken at 22:19, known at 8:10, fixed at 10:05. The fix took thirty-five minutes. The finding took ten hours.
David compresses the timeline into one comparison and it lands like a diagnosis: "The fix took thirty-five minutes. The finding took ten hours." Nothing was hard about repairing this. Everything was hard about noticing it. That one line tells the room exactly where to aim — and it's built purely from facts they just heard, which is why nobody can argue with it.
How bad, Tom? Honestly.
Honestly? Annoying, not fatal. Nobody made a decision on a blank dashboard — they just called me. But it's the second thing this month that's made the regional team go "is this platform actually working?" And that, that's the bit I care about.
Tom does something disciplined: "Annoying, not fatal." He doesn't perform outrage, and he doesn't wave it away either. Then he names the real damage — not the blank screens, but the erosion of trust ("is this platform actually working?"). Separating the operational impact from the reputational impact is what makes a client's feedback useful rather than just heat.
Yeah. That's fair, and that's the right thing to be worried about. Noted, and it goes in the actions.
David agrees, names it as legitimate, and routes it somewhere concrete ("it goes in the actions"). What he doesn't do: apologise three times, get defensive, or promise it'll never happen again. Acknowledging a fair point and giving it a home is enough. Over-apologising in a postmortem quietly shifts the meeting from fixing systems to managing feelings.
I should have caught this. I set up that alert channel, and I never checked whether anyone was actually watching it. That's, that's on me.
Okay — stop. I get the instinct, but let's test it. If Priya had left the team last month, would this have happened anyway?
...Yes. Obviously yes.
Right. So it's not a Priya problem. It's a "we built a system where an alert can fire into an empty room and nothing happens" problem. Different problem. Different fix.
This is the sharpest move in the pack. Priya reaches for "that's on me" — the honourable instinct, and the wrong analysis. Instead of arguing with her feeling, David runs a test: would this have happened with a different person in the seat? If yes, it's a system problem wearing a person's face. "Different problem. Different fix." — blame produces apologies; systems thinking produces changes.
No, that's — yeah. Okay. That's fair. Different fix.
Priya doesn't cling to the guilt to prove she cares. Insisting on your own fault after the room has said "it's the system" feels humble, but it drags the meeting backwards and quietly pressures everyone else to perform guilt too. "Okay. That's fair." — and now she's available to actually think.
So let's do it properly. Not "who missed it" — what made it easy to miss? Marco, start us off. You found it.
Well — the report didn't look broken. That's the thing. An empty report and a report with no sales look identical. If it had shown an error, I'd have called Priya at ten past eight, not at half nine.
The question does the work. "Who missed it" has one answer and it's a person. "What made it easy to miss" has five answers and they're all fixable. Marco's is beautiful: the failure was invisible because it looked like a legitimate result. Notice the counterfactual he offers — "if it had shown an error, I'd have called at ten past eight" — that's a finding and an action in one sentence.
And the retry logic. It retried, failed, and then swallowed it. It's meant to retry transient stuff — a network blip. It shouldn't retry a schema mismatch, that's never going to fix itself. But it doesn't know the difference.
Priya explains a real engineering failure without a single word you'd need a degree for: it retries things that might fix themselves, this was never going to fix itself, it can't tell the two apart. Swallowed it = hid the error. This is what "concrete but accessible" sounds like — the specifics are exact, but the meaning survives without the jargon.
And from my side — nobody knew who to call. Marco messaged Priya directly because he happens to know her. If it'd been someone in Manchester, they'd have raised a ticket and we'd have found out at lunchtime.
Tom's point is subtle and important: this morning worked because Marco personally knows Priya. That's not a process, that's a coincidence — and coincidences don't scale to a regional team of forty. "They'd have raised a ticket and we'd have found out at lunchtime." Naming the informal shortcut that saved you is uncomfortable, because it means admitting the official path is broken. It's also exactly what a postmortem is for.
Can I ask a stupid question? Why did the schema change without anyone telling us?
It's not stupid, it's the actual question. Someone in the ERP team added a column. Tuesday releases. They don't know we read that table.
The most useful sentence in technical meetings, and it's almost never a stupid question — it's usually the one everybody assumed was already answered. Marco's response is the correct one: "It's not stupid, it's the actual question." Rewarding the naive question out loud is how you keep people asking them. And look what it surfaced: a whole team upstream who don't know they have downstream consumers.
Hang on — has this happened before? The schema thing specifically.
...Third time this quarter. The other two we caught, because they broke loudly. This one didn't.
Right. So it's not really an incident, is it. It's a pattern that finally got us.
Yeah. That changes the priority, honestly — that moves it from "fix the pipe" to "fix the relationship with the ERP team." Good question.
One incident is bad luck. Three is a design. Tom's question reveals that the schema change wasn't a freak event — it's a quarterly habit that happened to hit a blind spot this time. "It's a pattern that finally got us" reframes everything: you're not fixing last night, you're fixing a recurring condition. Always ask it. If the answer is a number bigger than one, the priority you walked in with is probably wrong.
Okay, so — I count three things, and it needed all three. Upstream changed without telling us. The retry hid the failure. And the alert had nowhere to land. Take any one of those away and we'd have known by half ten last night.
"It needed all three." This is the mature framing. Real failures are chains: each link on its own is survivable, and the accident is the alignment. Saying "the cause was X" always means you stopped looking. And the practical payoff of counting three: you get three places to intervene, and you only need to break the chain once.
And honestly, the third one's the cheapest to fix and the one we'd never have prioritised.
Before actions — what went well? And I'm not being nice, I mean it as a real question. Anything we should keep.
Almost every postmortem skips this or turns it into thirty seconds of politeness. It's a real analytical question: something in your response worked, and if you don't name it, you can't protect it. "I'm not being nice, I mean it as a real question" — David explicitly blocks the polite reading. The things that go well in an incident are as accidental as the things that go wrong, and just as worth understanding.
Priya answering at half nine on a Tuesday. She had it in ten minutes.
That's because we'd documented the mapping last month. Two months ago that would've been an afternoon of digging.
Priya doesn't accept the compliment as personal talent. She traces it: the fix was fast because somebody wrote the mapping down last month. That's the useful version — it turns "Priya's good" into "documentation pays for itself," which is a thing you can decide to do more of. Same discipline as the substitution test, pointed at success instead of failure.
And nobody hid it. I've been on projects where you find out three days later from a spreadsheet. This was on a screen in front of me by ten.
The most valuable thing that went well isn't technical: nobody covered it up. Tom's comparison ("three days later from a spreadsheet") tells you what he's seen elsewhere and what he's grading against. This is the payoff of the blameless frame — a team that isn't afraid tells you fast, and telling you fast is worth more than any monitoring tool.
Right, actions. And — a rule: no action that's "we'll be more careful" or "we'll keep an eye on it." If it depends on someone remembering, it's not a fix.
"If it depends on someone remembering, it's not a fix." Most postmortems die here: the actions are "be more vigilant", "double-check next time", and everyone leaves feeling responsible and changing nothing. A real action changes the system so that the next tired person on a Tuesday night can't make the same mistake. Vigilance is not an engineering control.
Okay. One: retry logic only retries transient errors — schema mismatch fails loudly, straight away. That's a day's work, I'll do it this week.
Two: alerts move to the main channel, and they page me if nobody acknowledges in fifteen minutes. Half a day.
Three — the report should say "no data loaded" instead of showing me a beautiful empty chart.
That one's harder than it sounds. The report doesn't know whether zero rows means broken or just a quiet day. We'd need a freshness check — did the load finish, when.
How long?
Two, three days. And it touches every report, so it'd want testing.
Fine — then it's not this week, it's next sprint, and it's a proper ticket. Let's not pretend it's small just because it's obvious.
Priya doesn't say no — she says "that one's harder than it sounds" and explains why in one sentence. That's how you resist an underestimated task without looking obstructive: name the hidden complexity, give a number. And David's response is the right one — he doesn't shrink the task to fit the enthusiasm, he moves it to where it fits. "Let's not pretend it's small just because it's obvious" is a sentence worth carrying into every planning meeting you ever sit in.
That's the one I want most, honestly. That's the one the regional team sees.
Look at the shape of each one: what changes, who does it, how long it takes. "A day's work, I'll do it this week." No action is a wish. And notice Tom's line — he ranks them by what the client actually experiences, which is a different order than the engineers would pick. Both orders are legitimate; hearing them out loud is why he's in the room.
Actually — hang on. There's one more, and it's the one that stops the pattern. A schema check. Runs against the source at eight in the evening, before the load, and flags anything that's changed. New column, renamed column, anything.
So we'd find out before it breaks, not after.
We'd find out at eight in the evening with a coffee, instead of half eight in the morning with Tom on the phone. Half a day's work.
Every action so far makes the failure louder or faster to spot. This one is different in kind: it moves the moment of discovery earlier than the break. That's the distinction worth internalising — detecting a failure well is not the same as preventing it. Postmortems drift towards better alarms because alarms are easy; the finding that actually retires a recurring incident usually looks like this one, and it usually shows up late in the meeting, from the person who's been thinking while everyone else was listing.
And that doesn't need the ERP team to do anything?
No. That's exactly why I like it. Ideally they'd run it in their own release pipeline and we'd never get a surprise at all — but that's a conversation, and conversations take months. This one we can do on Thursday.
Priya draws the line clearly: the ideal fix lives in another team's pipeline and depends on goodwill and time; the good-enough fix lives in ours and ships Thursday. Do the one you control now; pursue the one you don't, slowly, in parallel. Teams lose whole quarters waiting for the elegant cross-team solution while getting burned weekly by the thing they could have built in half a day. "Conversations take months" — said without bitterness, just as a planning fact.
Love that. Okay — and five, which is on me: I'll still go and talk to the ERP team about a heads-up before schema changes. That's a relationship, not a ticket — it might take a few goes.
The fourth fix can't be coded — it's getting another team to care. David owns it himself (the facilitator takes the political one), and he's honest that it's slow: "that's a relationship, not a ticket." Pretending organisational problems have engineering deadlines is how they end up on a list forever.
Good. So — three fixes this week, one bigger one next sprint, one conversation that'll take longer, and no one gets to feel bad about last night. Deal?
Deal. And I'll stop apologising for finding things.
And — for what it's worth, this is the bit that reassures me. Not that it didn't break. That you took it apart like this.
Tom's last line is the thesis of the whole pack: "Not that it didn't break. That you took it apart like this." Clients don't expect perfection; they expect competence when things go wrong. A calm, honest, well-run postmortem does more for a relationship than a year without incidents — because it shows them what happens on the bad days.
Appreciated. I'll write it up and send it round this afternoon — timeline, three causes, five actions, and the fact that it's the third schema break this quarter. That last bit's going in bold.
Send it to me and I'll forward it to Sarah myself. Comes better from the client.
That's — yeah. Thank you, that helps a lot. Anything I've missed, shout.
A postmortem that isn't written down didn't happen — the fixes drift and the next team hits the same wall. David names exactly what's in it (timeline, three causes, five actions) — a neat summary of what a postmortem IS — and puts the pattern in bold, because the recurring cause is the real story. Then Tom does something quietly generous: he offers to carry it to the sponsor himself. "Comes better from the client" — the same document lands differently depending on whose hand it's in, and he knows it.