Inside InboxWa's automation engine — architecture notes
By Prenit · · 9 min read
Most chatbot projects fail because the LLM is asked to do everything. Greet the user, answer the FAQ, compute the price, book the meeting, handle the objection, escalate when stuck. Any one of those can go wrong, and when it does, the failure is invisible until a customer complains.
Our thesis from day one: the best LLM application is the one that calls the LLM least. This post walks through how we built the engine behind that thesis — the tier waterfall, the retrieval stack, the confidence gate, the memory model, and the eval harness that gates every deploy.
The waterfall, in one picture
Every inbound message flows through a series of deterministic checks before any LLM call happens. If a cheaper stage can answer the message, we stop there. Only the truly generative work reaches the generator.
T0 Intent classification regex 1 ms
T2.a Pricing / product bypass CSV lookup 2 ms
T2.b Query rewrite LLM (Haiku) ~280 ms
T2.c Hybrid retrieve Qdrant ~45 ms
T2.d Cross-encoder rerank MiniLM ~110 ms
T2.e Confidence gate score math 0 ms
T2.f Generate LLM (Step) ~340 ms
T3 Handoff decision state 0 msOn a typical support query, we only invoke the LLM twice: once to rewrite the user's question into something retrievable, once to generate the final answer from retrieved context. Everything else is deterministic.
Deterministic first, LLM as fallback
T2.a is the most underrated layer. A regex matches intent ("how much", "pricing", "what does it cost") against a small set of patterns, maps to a plan name, pulls the price from a CSV, and replies with a structured answer. No LLM. No retrieval. Two milliseconds.
About 35% of the queries we see could be handled like this — the long tail of questions where the answer is literally a row in a lookup table. Operators sometimes push back here: why not just let the LLM answer? Because the LLM will occasionally hallucinate a price, and the cost of one hallucinated price is higher than all the LLM calls we save.
Hybrid retrieval + rerank
For the open-ended queries, retrieval matters more than generation. We run dense and sparse retrieval in parallel, fuse the results, then rerank with a cross-encoder.
- Dense — 1536-dimensional embeddings over 1,329 indexed chunks. Qdrant's HNSW. Captures semantic similarity (the question "how do I upgrade my plan?" retrieves a chunk about "in-place migration").
- Sparse — BM25 over the same chunks. Catches exact-match signals the dense layer misses (product codes, specific error messages, uncommon proper nouns).
- RRF fusion — reciprocal rank fusion with k=60. Merges both result sets by their rank positions, not scores. Simple and surprisingly robust.
- Cross-encoder rerank — MiniLM-L-6-v2 reranks the top 30 to top 5. This is the single most expensive step in the hot path (~110ms), but it's the step that kills "close but wrong" retrievals before the LLM ever sees them.
The confidence gate
Retrieval can succeed at finding something, but what it finds might not actually answer the question. Before we spend a generation token, we check two signals:
# tier_2_confidence_gate.py
def should_generate(rerank_score: float, dense_top: float) -> bool:
# Dual signal: rerank score measures answer-relevance,
# dense cosine captures paraphrase-level similarity.
if rerank_score < 0.0 and dense_top < 0.38:
return False
return TrueWhen both signals fail, we stay silent — we emit a handoff reply instead of a generated answer. About 20% of queries hit this gate on a new deployment (before the knowledge base is tuned); post-tuning, it drops to 5-8%.
The handoff reply has three deterministic variants (rotated by hash(session_id + query)) so a user retrying the same question sees different wording. This matters more than you'd expect — users experience the same canned reply as the bot doesn't understand me, but three different polite refusals as this is a careful system.
Per-org memory, isolated by design
Every conversation is scoped by (chat_id, org_id). Two workspaces that happen to share a phone number (unusual but possible, e.g. test numbers) don't see each other's state. The scoping is at the query level, not a trust boundary; we enforce it on every read and write.
Memory is two-layered:
- Hot cache. In-memory LRU, capped at 5,000 active sessions, 30-minute inactivity TTL. Stores the last 10 turns of a conversation.
- Cold store. PostgreSQL
chat_sessionstable. No expiry — survives restarts, deploys, and scale events. Includes not just messages but structured metadata: contact name, lead score, discovery state, CTWA ad source.
The metadata field is where most of the interesting behaviour comes from. Lead scoring is a 12-signal point system that fires as the conversation progresses. Discovery is a 5-field state machine (use-case, volume, budget, timeline, decision maker). Both are writable by deterministic modules — no LLM call required to update the state.
The human handoff that actually works
When a human agent replies inside InboxWa, the bot sets bot_paused=True on the session and records human_last_reply_at. For the next 5 minutes, any inbound from the customer is ignored by the bot. If the human doesn't reply back within that window and the customer messages again, the bot auto-resumes.
This sounds simple. It's the behaviour we care most about getting right. The naive version (bot keeps replying alongside the human) produces the most-hated chatbot experience in existence — two different tones, contradictory answers, a user who doesn't know which voice to trust.
Eval-gated deploys
We ship changes on a deploy gate: 75 test cases (growing weekly) must pass in three ways:
- Tier-path match. The expected tier sequence (T0 → T2.c → T2.f) matches the actual sequence. If a change accidentally makes T2.a fire on a query where it shouldn't, the gate catches it.
- Handoff match. Queries that should hand off, do. Queries that shouldn't, don't.
- Keyword faithfulness. The generated answer contains the expected anchor keywords from the ground-truth reference. Not exact match — anchor-match. Lets us change phrasing without failing tests.
This eval suite is not where we run deep benchmarks. It's a regression fence. When someone tweaks a prompt, adjusts the retrieval fusion, or changes the confidence threshold, the suite catches the unintended side-effects. A dropped case is treated like a failed unit test — deploy blocked.
What this costs per query
Rough economics per 1,000 queries on the hot path (Step 3.5 Flash primary):
- Deterministic tiers (T0 + T2.a): free.
- Query rewrite (Claude Haiku): ~₹15 per 1,000.
- Retrieval (Qdrant + reranker): hardware cost only, ~₹2 per 1,000.
- Generation (Step Flash): ~₹15 per 1,000.
- Total: ~₹32 per 1,000 queries. Roughly ₹0.032 per query on the hot path.
When Step times out (~35% of queries today, an embarrassingly high fraction we're fixing), we fall back to Haiku 4.5, which costs ~₹0.30 per query — an order of magnitude more. Reducing the Step fallback rate is the single highest-ROI optimisation on our roadmap.
What we didn't build
A few things we considered and deferred:
- HyDE (Hypothetical Document Embeddings). Powerful but slow. Doesn't pay off until your knowledge base is large and noisy — at 1,329 chunks, query rewrite + rerank covers us.
- Hierarchical RAG. Useful above ~10k chunks. We're not there yet.
- Multi-agent orchestration. The waterfall is our orchestration. Multi-agent frameworks solve a coordination problem we don't currently have.
- LangGraph / LangChain chains. We roll our own chain primitives because the LangChain abstractions add surface area without a corresponding productivity gain at our size.
Boring reliability beats clever unreliability. Every design choice above was picked because it was the least-flashy option that produced the best customer outcome. Next time we write one of these, we'll talk about how we plan to replace the knowledge base without any customer noticing.