The Behavioral / Stakeholder Round

“Tell me about a time you had to decide whether to launch an AI feature even though you knew it wouldn’t be perfect.”

We built a feature that auto-drafted customer support replies. Our eval set showed about a 6% rate of factually wrong drafts — not dangerous, but wrong enough that shipping it as fully automated would have hurt trust. Leadership wanted to launch on the original date regardless.

“What did you actually do?”

I didn’t argue for delaying the launch — I argued for changing what shipped. We launched on schedule, but as a draft-and-review flow instead of fully automated: the agent always saw the AI draft before it went to the customer. That let us ship the real value — faster replies — on time, without exposing customers directly to a 6% error rate we hadn’t solved yet.

“Tell me about a time a model’s output was inconsistent or unreliable in a way that affected your roadmap, and how you worked with your data science or ML team on it.”

We had a document-classification model that was 94% accurate in offline eval, but the errors weren’t evenly distributed — it was much worse on one document type that happened to be high-value for a specific customer segment. The data science team’s instinct was to report the overall accuracy number as the health metric.

“How did you push on that without it turning into a fight over whose metric was right?”

I brought the segmented breakdown, not just an objection — accuracy by document type, weighted by how much that type mattered to revenue-relevant customers. That reframed the conversation from “is 94% good” to “is 94% good for the segment that actually matters,” which is a question the data science team was well-equipped to help solve once it was framed that way. We ended up prioritizing a targeted fine-tune on that one document type instead of chasing overall accuracy further.

“Tell me about a time you decided not to build something with AI, even though there was pressure to use it.”

A stakeholder wanted an AI-generated weekly summary email for account managers, pulling from CRM activity. It sounded impressive in a roadmap review. But when I actually mapped the user need, account managers mostly wanted three specific numbers — open opportunities, at-risk accounts, and overdue follow-ups — surfaced quickly, not a paragraph explaining them.

“So what did you build instead, and how did that conversation go?”

A simple rules-based dashboard with those three numbers, no model involved, shipped in about a third of the time a generative summary would have taken, with zero risk of a wrong number being stated confidently in a paragraph. The stakeholder pushed back initially — it felt like a step down from the AI pitch — but adoption ended up higher than any of our AI features that quarter, because it was faster and more trustworthy for exactly what people needed.

“Tell me about a time you had to explain a model’s failure — like a hallucination — to a stakeholder who wasn’t technical, in a way that didn’t either overwhelm them or oversimplify the risk.”

Our AI assistant gave a customer a wrong return-policy detail, and the account exec who escalated it wanted to know “how do we make sure the AI stops lying.” I didn’t start with the technical mechanism — I started by reframing “lying” as “the model can state something confidently without actually knowing it’s true, the same way a person might misremember a policy instead of checking it.”

“Where did the conversation go from there?”

From that framing, the fix made sense without needing technical detail: instead of trying to make the model never misremember, we changed the design so it always checks the actual policy document before answering instead of answering from memory, and shows the source. I gave the exec a concrete number too — our target was under 1% of policy answers being wrong, measured weekly — so “stop lying” became a specific, trackable commitment instead of an open-ended promise.