The Case-Study Round

“I run customer support for a mid-size online retailer. About 40% of our 50,000 monthly tickets are ‘where is my order.’ We’ve heard AI can answer these automatically — can you show us how?”

Before I sketch anything — where does order and shipping data actually live today, and what system are your reps working in when they answer one of these tickets?

“Tickets come in through Zendesk. Order data is in Shopify, and shipping status comes from our 3PL’s own tracking system, which doesn’t always agree with Shopify.”

Good to know now — that disagreement between two data sources is exactly the kind of thing that turns into a bad AI answer if we build past it. When the two sources conflict today, which one does a human rep trust?

“The 3PL, usually — Shopify’s status can lag by a day.”

Then the agent should follow the same rule, not average the two or guess. One more question before I propose scope: should this agent ever be allowed to issue a refund or a replacement, or only answer status questions?

“Absolutely not refunds automatically — that’s a hard line from finance. Status and tracking info only, at least to start.”

That’s the constraint I’d design around rather than bolt on later. I’d propose a read-only agent: it classifies an incoming ticket, and if it’s a status question, it calls a lookup tool against the 3PL tracking API — falling back to Shopify only if the 3PL has no record yet — and drafts a reply. No write access to either system at all in this phase.

“Drafts a reply — so a human still sends it?”

For the PoC, yes. The draft goes into the Zendesk queue as a suggested response a rep approves with one click, the same shadow-to-live pattern you’d want for anything customer-facing — you get to see the agent’s accuracy on real tickets before anything goes out under your brand’s name without a human glancing at it first.

“Fair. What would a two-week proof of concept actually show us?”

I’d pull a sample of a few hundred real, closed order-status tickets, run the agent against them offline, and measure two things against your own historical replies: how often it would have given a correct answer, and how often it correctly recognized it didn’t know and should hand off to a human instead of guessing. That second number matters as much as the first.

“And if it looks good, what changes for the live pilot?”

The pilot moves from offline replay to the live Zendesk queue, still draft-and-approve, on a subset of your support team so you can watch adoption, not just accuracy. Auto-send only becomes a conversation once you’ve seen weeks of reps approving drafts basically unedited.

“I run claims operations at a regional insurer. Our call-center reps have to dig through 40-page policy PDFs to answer coverage questions, and they get it wrong more often than I’d like. Can you build us a search tool?”

When a rep gets it wrong today, is that mostly them not finding the right clause in time, or finding a clause and misreading it?

“Mostly not finding it in time — call’s already five minutes in and they’re still scrolling a PDF.”

That’s a retrieval problem more than a comprehension one, which is a good sign for what we can fix quickly. Where do these policy documents actually live, and how do they change over time — are they versioned by effective date?

“They’re in SharePoint, and yes, each policy has an effective-date version. Some of the older ones are scanned images, not real PDFs, though.”

Good to flag now — the scanned ones need an OCR step before anything else works, and I’d want to know roughly what fraction of your active policy library that is before committing to a PoC timeline. Before I propose an architecture: is this tool only for reps, or could an answer ever go straight to a customer?

“Reps only, for now. If we ever gave a customer a wrong coverage answer directly, that’s a real liability problem for us.”

That constraint actually makes the design easier to trust: every answer this tool gives can show the rep the exact clause and page it came from, and the rep is always the one who says it out loud to the customer. I’d propose a RAG pipeline — chunk each policy by section and clause, tag each chunk with its policy version and effective date, and when a rep asks a question, retrieve the relevant clauses and show them alongside a drafted answer, never the answer alone.

“How do we know it’s actually getting the right clause, and not just something that sounds plausible?”

That’s exactly why I wouldn’t launch this on gut feel. Before any live pilot, I’d want your claims SMEs to hand-label a few dozen realistic questions with the correct clause each one should retrieve, and we’d measure the pipeline against that set before a rep ever sees it — the same precision-and-recall discipline you’d use to evaluate any search system, not just an AI one.

“And if the eval looks good — what’s the actual pilot?”

One line of business to start — say, auto policies in a single state, since coverage language is more standardized there than home — rolled out to a handful of reps, with every answer showing its cited clause. I’d explicitly leave home and commercial lines out of scope for the pilot; broader coverage is a phase-two conversation once the eval holds up on the narrower slice.

“I run field service for an industrial equipment manufacturer. Our technicians burn twenty minutes on-site flipping through PDF manuals to find a torque spec or a wiring diagram. We have a trade show in six weeks — can you get us a demo?”

Six weeks is workable for a scoped demo, but I want to make sure we scope the right thing. What’s connectivity actually like at the job sites your technicians work — reliable cellular, or are some sites dead zones?

“Mostly fine, but a meaningful chunk of our sites are in rural areas with no signal at all.”

That changes the architecture more than anything else so far — a pure cloud-lookup tool would just fail at exactly the sites where it matters most. What device are technicians actually carrying, and could it hold a local cache?

“Rugged tablets, yes, with decent local storage.”

Then I’d propose a hybrid: manuals for the specific equipment on a technician’s dispatch schedule get synced to the tablet before they leave for the site, so lookups work offline, with a cloud RAG fallback for anything outside that pre-synced set when they do have signal. For the trade-show demo specifically, is there one equipment line you’d want to showcase, or does it need to cover your full catalog?

“One line would be fine for the demo — our new industrial pump series, actually.”

That’s the right scope for six weeks — a hero demo on one equipment line, running fully offline on the tablet, showing a technician asking a question in plain language and getting the exact torque spec or diagram with a page reference. I’d be direct with your team, though: a polished trade-show demo on one product line is not the same as a production rollout across your full catalog, and I don’t want that distinction to get lost after the show goes well.

“Understood — what would it take to go from that demo to something real?”

A pilot with a small group of technicians, maybe ten, using it on real service calls for a few weeks, with feedback captured on where the answer was wrong or missing — and a real content-refresh pipeline, since your manuals presumably get revised as products change, and a stale cached manual on a tablet is a safety problem, not just an inconvenience.

“Our IP team will also ask — these manuals are proprietary. How do we make sure they’re not exposed?”

Good to raise now rather than after the pilot — the vector index and the on-device cache both need to be encrypted at rest, access scoped to authenticated technician devices only, and I’d want that written into the architecture doc before the pilot starts, not retrofitted after your security team asks.