The Technical / Evaluation Round
This isn’t a coding interview — nobody expects you to write production code. But it’s not a soft-skills round either. You’ll be handed a dataset, a metrics question, or a build-vs-buy decision and asked to reason through it with real technical fluency: read a SQL query, tell offline eval apart from online eval, and name a specific number when asked how much hallucination is too much. Vague answers are the single biggest way candidates lose this round.
What to expect
Six problem shapes show up across current postings and interview reports. Earlier-stage and AI-native companies lean harder on live discussion and case-style reasoning; larger companies are more likely to formalize part of this as a take-home.
| Problem type | What it looks like |
|---|---|
| SQL & data-analysis exercise | Read or lightly write a SQL query against a sample events table, then interpret what it shows — “feature usage is up 20% week over week, is that good?” tests whether you reach for a denominator, not whether you can write a join from memory. |
| Offline vs. online evaluation discussion | Given a model change, you’re asked how you’d evaluate it before shipping (offline: eval sets, golden answers, precision/recall) versus after shipping (online: A/B test, live user metrics, guardrail monitoring) — and why you need both. |
| Metrics definition case | Pick the north-star metric and 2–3 guardrail metrics for an AI feature (say, an AI search assistant) — and defend why a metric like “messages sent” is a vanity number if it isn’t paired with a quality or task-completion signal. |
| Build-vs-buy / fine-tune-vs-RAG-vs-prompt decision | Given a product requirement, decide whether to solve it with prompt engineering, RAG, fine-tuning, or a third-party API — and defend the trade-off in cost, latency, and control against a skeptical follow-up. |
| Audit AI-generated code or output | Some companies now hand candidates AI-generated code or an AI-drafted spec and ask them to find what’s wrong with it — testing the increasingly common expectation that a PM can sanity-check AI output, not just describe it. |
| Defend a hallucination-rate threshold | “What hallucination rate is acceptable for this feature, and what happens above it?” — a direct test of whether you’ll commit to a real number and a real fallback plan, instead of saying “as low as possible.” |
If you’re unsure how technical a specific company’s round runs, ask the recruiter — “is this closer to a data/metrics discussion or a hands-on exercise?” is a normal question at this stage, and the answer changes how you spend the next two weeks.
What actually separates a pass from a fail
Across sources — from AI PM interview guides to hiring-manager write-ups — the same gap between strong and weak candidates shows up repeatedly, and it’s rarely about technical depth alone:
Names outcome metrics, not activity metrics
“Messages sent” or “sessions per week” going up means nothing on its own — a strong answer immediately asks what task the user was trying to complete and whether they completed it, then pairs the activity number with that.
Keeps offline and online evaluation straight
Confusing “we ran it against our eval set” with “we know it works in production” is one of the fastest tells of someone who hasn’t actually shipped an AI feature. Both are needed, and they answer different questions.
Says “I don’t know, I’d check with the ML team” instead of bluffing
Interviewers explicitly watch for candidates who fabricate technical confidence they don’t have. A precise “here’s what I know, here’s what I’d confirm with an engineer” reads as more credible than a guess dressed up as certainty.
Names a specific hallucination threshold and a fallback
“As low as possible” is not an answer. “Under 2% on our eval set for this use case, and above that we fall back to showing sources instead of a direct answer” is — a number plus a plan for what happens when the number is missed.
Comfortable enough with SQL to self-serve a basic question
You don’t need to write a complex query from scratch, but flinching at a simple SELECT/GROUP BY, or being unable to say what a JOIN does, undercuts credibility fast in a role that increasingly expects some data self-sufficiency.
Vagueness is the most common failure, not lack of technical knowledge. A candidate who commits to a specific hallucination threshold and a specific fallback plan — even if an interviewer pushes back on the exact number — reads as more hireable than one who stays safely abstract to avoid being wrong.
How to prepare
Two weeks is enough if you already have some data fluency. The goal isn’t learning to code — it’s rebuilding the specific habit of committing to a number and a plan under a skeptical follow-up, instead of retreating into generalities.
- 1
Week 1 — drill the “how did you measure that” follow-up
For every metric you plan to cite anywhere in the loop — in this round or the behavioral round — practice a follow-up chain three levels deep: how did you measure it, what was the baseline, how do you know the change caused it. This single habit is what separates a real answer from a rehearsed one.
- 2
Week 1 — rebuild SQL fluency
Work through this curriculum’s Intermediate Chapter 8 (Evaluating What You Built), which covers precision@k/recall@k against a hand-labeled eval set — it doubles as SQL and evaluation-metrics practice in one exercise, and gives you a real, defensible example to cite in the interview itself.
- 3
Week 2 — five build-vs-buy reps
Take five different product requirements (a support chatbot, a code-review assistant, a document summarizer, a search feature, a data-extraction tool) and force yourself to pick prompt-only, RAG, fine-tuning, or a third-party API for each, then defend it out loud against “why not just fine-tune?”
- 4
Week 2 — write a one-page hallucination policy
Pick a hypothetical AI feature and write down: the acceptable hallucination rate, how you’d measure it, and what the product does above that threshold (refuse, show sources, escalate to a human). Having done this once for real makes the live version of the question dramatically easier.
Two things to have ready before you walk in
Beyond drilling problems, two concrete artifacts change how the whole round reads, because they let the interviewer evaluate real judgment instead of a simulated forty-five minutes of it.
A real eval set you built, not a hypothetical one
This curriculum’s Intermediate Chapter 8 walks through building a hand-labeled eval set and scoring retrieval with precision@k/recall@k — bring the actual numbers you got, not a description of what an eval set would look like.
A defensible build-vs-buy decision from something you actually built
One real example — even a small side project — where you chose prompt engineering over RAG, or RAG over fine-tuning, and can explain the trade-off in cost, latency, and maintenance, beats five hypothetical ones described in the abstract.
Neither needs to be polished or impressive at scale. A small, honestly-measured, actually-built thing beats an ambitious, hypothetical one every time this comes up as a talking point.