AI Checkout — Operational Runbook

For Swft engineers responding to AI Checkout incidents.

Quick health check

Per-merchant: GET https://api.swft.co.uk/health/ai-chat with x-swft-api-key.

Returns:

• status: ok | degraded | paused
• LLM + embedding configuration (just the booleans, not the keys)
• Knowledge stats
• Monthly cost vs cap

Common issues

”AI Checkout silently doesn’t work for merchant X”

1. Hit health endpoint with their API key
2. Check cost.monthly_cap_cents. If 0 → merchant explicitly disabled. If pct_used >= 1 → cap reached for the month.
3. Check llm.configured and embeddings.configured. If false → env var missing on Railway.
4. Check knowledge.product_chunks and total_chunks. If 0 and merchant enabled Mode B/C → catalog/policy sync hasn’t run.

”Customer reports a hallucinated answer”

1. Find the chat in ai_chat_sessions (via dashboard Transcripts tab, filter by date).
2. Check messages for the offending answer.
3. Check mode_b_detour / mode_c_detour events for that chat_id — props.results_count shows what the search returned.
4. If results_count: 0, the AI fell through to the “I don’t have a match” fallback. Customer concerns are likely about a product/policy that isn’t synced.
5. If results_count > 0, the AI may have hallucinated despite citations. Forward to ML team with the transcript ID.

”Stripe webhook not flipping chat to complete”

1. Find the chat by payment_intent_id:

   select * from ai_chat_sessions where payment_intent_id = 'pi_xxx';

2. Check the webhook logs (Railway → API service). Look for "AI chat ${id} completed via PI ${pi.id}".
3. If not present, check the PI metadata — swft_chat_id must be set, otherwise the AI chat handler doesn’t fire.

”Cost spiking unexpectedly”

1. Hit health endpoint, note cost.monthly_spend_cents.
2. Query ai_chat_events for the merchant grouped by event_type:

   -- running_cost_cents is cumulative per chat — use count() for volume,
   -- and sum on ai_chat_sessions.llm_cost_cents for actual dollars.
   select event_type, count(*) as events
   from ai_chat_events
   where merchant_id = 'm-xxx' and ts > now() - interval '24 hours'
   group by event_type
   order by events desc;

3. If mode_c_detour is high, Sonnet is the culprit (5× cost). Disable Mode C if needed.
4. If state_transition is high, narration cost. Mode A defaults are cheap; if elevated, check for runaway loops.

”Plugin not auto-syncing products”

1. WordPress error log on the merchant’s host
2. Check swft_enabled and swft_api_key options
3. Check swft_ai_chat_enabled option
4. Try a manual save on a product — should fire woocommerce_update_product hook
5. Check Cloudflare API logs for the merchant’s POST /merchants/ai-chat/knowledge/products requests

Rate limits to be aware of

• Anthropic: tier-1 default 50 RPM for Haiku. We’re not close in practice but a viral product could spike.
• OpenAI embeddings: tier-1 default 3,000 RPM. Bulk catalog scans throttle to 20 RPS.
• Stripe webhooks: at-least-once delivery. Idempotency is enforced via the current_state === 'complete' early-return in webhooks.ts.

Eval gates before releases

Before any production release that touches LLM-adjacent code:

cd api
ANTHROPIC_API_KEY=sk-ant-... npm run eval:classifier

Should report ≥85% accuracy. Below that → block release, investigate prompt drift.

OPENAI_API_KEY=sk-... EVAL_MERCHANT_ID=<staging-merchant> npm run eval:retrieval-products
OPENAI_API_KEY=sk-... EVAL_MERCHANT_ID=<staging-merchant> npm run eval:retrieval-policies

Should both report ≥80% Recall@5. Below that → check embedding model version + fixture data.

Killswitch

Per-merchant: set merchants.ai_chat_monthly_cap_cents = 0. Takes effect on the next request (no cache invalidation needed).

Globally: pause API service on Railway. Customers fall back to standard Swft checkout via the plugin’s 402 handler.

Useful queries

-- Top 10 merchants by AI Checkout spend this month
select merchant_id, sum(llm_cost_cents)/100.0 as spend_dollars
from ai_chat_sessions
where created_at >= date_trunc('month', now())
group by merchant_id
order by spend_dollars desc
limit 10;

-- Merchants approaching their cap (>80%)
select m.id, m.name,
       m.ai_chat_monthly_cap_cents as cap_cents,
       coalesce(sum(s.llm_cost_cents), 0) as spend_cents,
       round(100.0 * coalesce(sum(s.llm_cost_cents), 0) / nullif(m.ai_chat_monthly_cap_cents, 0)) as pct
from merchants m
left join ai_chat_sessions s on s.merchant_id = m.id and s.created_at >= date_trunc('month', now())
group by m.id, m.name, m.ai_chat_monthly_cap_cents
having coalesce(sum(s.llm_cost_cents), 0) > 0.8 * m.ai_chat_monthly_cap_cents
order by pct desc;

-- Today's funnel for merchant X
select current_state, count(*)
from ai_chat_sessions
where merchant_id = 'm-xxx' and created_at >= current_date
group by current_state
order by count(*) desc;