How we build WhatsApp AI agents — our 10-day playbook
By Mohit · · 7 min read
Most WhatsApp AI agent projects fail. Not because the LLM is bad, or because the retrieval doesn't work, but because the team skips the boring parts. They throw documents into a vector database, wire up a prompt, demo it to the client, and call it done. Then the first real customer asks something slightly unexpected and the agent confidently invents an answer.
We've built enough of these to know where the landmines are. This post describes the 10-day delivery process we follow for every RAG deployment. It's not glamorous. Most of the time is spent on data cleaning, eval writing, and threshold tuning — the work that doesn't demo well but determines whether the agent survives contact with real users.
Why most chatbot projects fail
The default approach to building a chatbot is to give the LLM everything — the full product catalog, the FAQ, the company policies — and tell it to be helpful. This works in a demo because the demo query is always clean and well-formed. In production, it breaks in predictable ways.
- The LLM invents products that don't exist because the retrieval pulled a partial match and the model filled in the gaps.
- Pricing queries get approximate numbers instead of exact ones because the LLM generates plausible-sounding figures from context clues.
- The agent answers questions it shouldn't — billing disputes, refund requests, technical outages — because nobody defined the boundary between "answer" and "escalate."
- Multi-turn conversations lose context after a few exchanges because the session architecture wasn't designed for stateful dialogue.
Every one of these failure modes is preventable. But preventing them requires doing the unglamorous work before launch, not after the first complaint.
The 10-day structure
We've compressed our delivery into a 10-working-day cycle. This isn't a rigid waterfall — some days overlap, some clients need more time on data ingestion, others need more eval iterations. But the sequence matters. Skipping steps or reordering them produces worse outcomes.
Day 1 Kickoff + data audit
What data exists? What format? What's missing?
Define scope: what the agent WILL and WON'T do.
Day 2-3 Data ingestion + cleaning
Normalize formats. Deduplicate. Chunk for retrieval.
Build the catalog CSV or structured data layer.
Day 4-5 Retrieval tuning
Chunk sizes, overlap, embedding model selection.
Top-k tuning. Reranker calibration.
Deterministic bypasses for structured lookups.
Day 6-7 Eval case writing
50-100+ conversational test scenarios.
Acceptance criteria for every case.
Adversarial cases: trick questions, out-of-scope,
ambiguous queries.
Day 8-9 Iteration
Run eval → score → fix highest-frequency failure
→ repeat. One failure category per cycle.
Prompt refinement, threshold tuning, retrieval fixes.
Day 10 Go-live preparation
Confidence gate calibration. Escalation paths.
Monitoring setup. Handover documentation.Why eval comes before launch
We write eval cases on Day 6-7, before the agent has answered a single real customer message. This feels backwards to teams used to shipping and iterating. But AI agents are different from most software — a wrong answer in the first hour doesn't just produce a bug report, it destroys trust.
A customer who gets a confidently wrong price quote, or a recommendation for a product that doesn't exist, doesn't file a ticket. They leave. And they tell others. The cost of a single bad answer is disproportionately high compared to the cost of writing one more eval case.
In our experience, the eval writing process itself is where we discover the biggest gaps. Writing 50 test cases forces you to think about what customers actually ask — not what the product team thinks they ask. We've found that the adversarial cases (trick questions, out-of-scope requests, ambiguous queries) reveal more about the agent's readiness than the happy-path cases.
The confidence gate
The most underrated component in any AI agent is the one that decides when NOT to answer. We call it the confidence gate. Every response gets a retrieval confidence score — how well did the retrieved context match the query? If the score falls below a calibrated threshold, the agent doesn't generate. It escalates to a human agent instead.
This sounds simple but it changes the economics of the entire deployment. Without a confidence gate, the agent answers everything — including the questions where it's guessing. With a confidence gate, the agent only answers when it has high-quality context to draw from. The human team handles the rest.
What the client gets at handover
On Day 10, the client receives a complete deployment package. This isn't just a running bot — it's the documentation and tooling they need to maintain and evolve the agent without us.
- The eval suite — every test case we wrote, with pass/fail results from the final iteration cycle. This is the baseline for measuring future changes.
- The prompt library — all system prompts, persona definitions, and formatting instructions. Versioned and documented so changes can be tracked.
- The dataset structure — the catalog format, chunking parameters, and ingestion pipeline. When the client adds new products, they follow this structure.
- The confidence gate configuration — thresholds, escalation paths, and the rationale behind each setting.
- Monitoring dashboards — what to watch, what's normal, and when to intervene.
The goal is that the client can update their product catalog, re-run the eval suite, and verify that the agent still passes — without needing us in the loop for routine changes.
Ten days sounds fast, and it is. The compression works because we've built the tooling — the eval harness, the retrieval pipeline, the deterministic bypass layer — once and reuse it across deployments. The client-specific work is data ingestion, prompt calibration, and eval writing. The infrastructure is already there.
If you're evaluating whether your data is ready for an AI agent, our data onboarding service is where most engagements start. We'll audit your existing catalog and tell you honestly whether it's ready or what needs to change first.