How Fabiana Answers a Customer: One LLM Call per Message

Fabiana answers a business's customers on WhatsApp, email, website chat, Instagram, SMS, Telegram and the phone, in the customer's language, and then does the next thing: books the appointment, sends a payment link, creates the lead or hands the conversation to a person. I designed and built it, and I run it.

This post is about the part customers never see: what happens between a message arriving and Fabiana's reply going out.

Every message goes through a queue

Messages arrive from many places: webhooks from WhatsApp and other channels, and pollers that check Gmail and Outlook mailboxes. The first thing each of them does is put the message on an SQS queue and return.

The reason is a deadline. Twilio expects a webhook to answer within 15 seconds, and a full turn (load the context, call the model, run tools, write to the database, send the reply) can take most of that. Answering the webhook straight away and processing the message from the queue means a slow model call never loses a message. SQS also retries failed messages and moves them to a dead-letter queue after three attempts.

Customer ──► channel webhook / mailbox poller ──► SQS ──► MessageProcessor Lambda

  1. spam and guard checks (paused agent, working hours, rate limits)
  2. load the agent, its flows, skills and channel behaviour
  3. one structured LLM call; the model must search the knowledge base first
  4. route by sender: customers get a reply, suppliers and partners go to the team,
     notifications and spam are ignored
  5. run tools (availability, payment link, handoff), update contacts and leads
  6. send the reply, or store it as a draft for the team

One model call per message

The obvious design is two calls: one to work out what the customer wants, then one to write the reply. I rejected it early. Two sequential calls double the latency and the cost, the context has to be rebuilt for the second call, and the two answers can disagree.

Instead, each message gets one call with structured output. A JSON schema forces the model to return everything at once: the reply, who the sender is, the intent, any tool calls, a lead score and the language. One call takes roughly 1.5 to 3 seconds, against 3 to 6 for the two-call version.

// Simplified: what one call returns for one incoming message
type AgentTurn = {
  reply: string
  language: string
  senderType: 'customer' | 'supplier' | 'partner' | 'notification' | 'spam'
  intent: string
  toolCalls: { name: string; arguments: Record<string, unknown> }[]
  leadScore?: number
  knowledgeGap?: string   // a question the knowledge base could not answer
}

The senderType field turned out to matter more than I expected. A business inbox receives far more than customer questions: supplier invoices, partner emails, newsletters, spam. Fabiana replies to customers, hands suppliers and partners to the team, and ignores the rest. Assignment rules can match on it too.

Answering from the business's own knowledge

Each assistant has its own vector store with the business's website, price lists and documents, searched with OpenAI's file search. The prompt requires the model to search before it answers. When the answer is not there, Fabiana says so and asks the team, and the question is recorded as a knowledge gap so the business can fill it.

I considered storing embeddings in DynamoDB and computing similarity in Lambda. Loading a few hundred embeddings, megabytes of floats, for every message is slow, and a table scan defeats the point of vector search. A managed store costs per query and has no idle cost, which fits the rest of the system.

Auto reply, or a draft for the team

Not every business wants an AI replying on its own, at least at first. Every channel of every assistant has an Auto reply switch. When it is off, Fabiana still does all the work, but the reply is saved as an AI draft in the inbox. The team can send it, edit it or ignore it, and turn Auto reply on when they are ready.

No fixed costs

The backend is Lambda, SQS and around fifty DynamoDB tables in on-demand mode, deployed with SST. Most HTTP routes are served by one Hono router in a single Lambda; webhooks and heavy endpoints get their own. Realtime inbox updates go over API Gateway WebSockets.

A rule I hold to: nothing that costs money while idle. No provisioned concurrency, no provisioned capacity, no always-on servers. Growth is handled by levers set at deploy time instead, such as a cap on how many message processors run at once. A cap costs nothing; messages beyond it wait in the queue and none are lost.

Testing an AI that talks to customers

Unit tests do not tell you whether an assistant gives good answers. Fabiana has two kinds of offline evals:

• Replay runs recent real conversations through the current prompt and model, so a change can be compared against what actually happened.
• Scenarios are scripted conversations with expectations: the sender type, whether it hands off, which tools it calls, what the reply must or must not say.

Both call the model with an assistant's real configuration but never run tools or write data.

Permissions come from WardenAuth

Who on a team can see which conversations, change settings or connect a channel is decided by WardenAuth, my authorization product. Building both meant Fabiana used the WardenAuth API the way any customer would.

If you want to see it working, talk to Fabiana on her site. She answers first.