Corvid Cloud support AI: where the LLM bill goes, and the six changes that cut it

Executive summary

At 1,200,000 requests a month the support assistant costs $23,117. Six changes bring it to $433, a saving of $22,684 a month. Quality: 76 of 81 checks pass after (78 of 81 before), 2 fewer than before, every miss listed. Three changes do most of the work: trim prompt and rag context ($13,410), cap output and reasoning ($5,000) and route easy calls to a cheaper model ($3,689).

• Cost per request falls from $0.019264 to $0.000361. At that rate the $1,900 fee is covered after about 100,512 requests.
• Quality: 76 of 81 checks pass after, 78 of 81 before. The 5 remaining failures are listed in full; 2 of them also fail on the baseline.
• Baseline shape. The baseline is a typical unoptimized stack: one big reasoning model on every call, a 3,073-token system prompt, 10 retrieved chunks, no output caps. A leaner stack starts from a lower bill; the free 48-hour AI check sizes the saving to yours before you pay.

Baseline: what the bill is made of

On your bill the baseline is your last full invoice month, reconciled line by line against a usage export, and we report the variance. Here the baseline is the API usage field of each of the 81 baseline calls, priced at Claude Sonnet 5.5 list prices ($2.00 per million input tokens, $10.00 per million output). The table re-adds those calls and checks the sum against the analysis.

Re-added total $23,116.77 against the analysis figure $23,116.77: variance $0.00.

Per month = sample cost × 1,200,000 / 81. The volume is an assumption (assumption: the 81-request mix repeats at 1.2M requests a month (about 10 percent of volume is per-account nightly digests)). The nightly digest is 8 of 81 calls but 30% of the cost: its average output was 5,092 tokens.

Savings by lever

Levers were applied one after another on the same 81 requests. Each section shows the money, one measured request before and after, the pull request that makes the change, and how to undo it.

 def answer(question):-    chunks = search(question, top_k=10)            # naive top-10-    system = SYSTEM_PROMPT                         # 3,073 tokens: persona, greetings, repeated rules+    chunks = rerank(search(question, top_k=20))[:3]  # reranked top-3+    system = POLICY_PACK + RULES                   # 1,792 tokens: every rule still in, filler removed     return llm.chat(model=BIG, system=system, user=render(chunks, question))

 MAX_TOKENS = {"qa": 320, "classify": 12, "summarize": 220, "report": 800} def chat(task, model, system, user):-    return client.create(model=model, system=system, messages=user)+    return client.create(model=model, system=system, messages=user,+                         max_tokens=MAX_TOKENS[task],+                         reasoning_effort="low" if model == BIG else None,+                         response_format=TERSE_JSON[task])   # e.g. one label for the classifier

+SENSITIVE = ["refund", "delete", "deletion", "gdpr", "privacy", "security", ...]+def route(task, question):+    if task in ("classify", "summarize", "report"): return SMALL+    if any(k in question.lower() for k in SENSITIVE): return BIG+    if needs_arithmetic(question): return BIG            # two or more numbers, 'how much', %+    return SMALL++def answer(question):+    out = chat(route("qa", question), ...)+    if not out.confident or not out.cite or set(out.cite) - retrieved:+        out = chat(BIG, ...)                             # escalation re-run+    return out

-    system=system,+    system=[{"type": "text", "text": system,          # policy pack + label definitions+             "cache_control": {"type": "ephemeral"}}],  # prefix must not change between calls

 def answer(question):+    hit = cache.get("qa", question)          # exact, then normalized text+    if hit: return hit     out = ...+    cache.put("qa", question, out)           # invalidated when the KB changes

-for account in accounts:-    digest = llm.chat(task="report", ...)+batch = client.messages.batches.create(requests=[+    {"custom_id": a.id, "params": digest_params(a)} for a in accounts])+# results within 24 h at half price; collected by the 06:00 job

Each PR is shown in the shape it would land in your repository. In this sample the change was made in our test harness; the caption under each diff names the file. Caching, dedupe and batch were priced on measured tokens rather than run as separate model calls.

Same tokens, three price lists

The run used one model pair through a proxy. To show the answer does not hinge on a vendor, the same measured token counts are priced as each provider’s big and small tier. The big and small tier for each provider is our mapping to its commercial models.

Google’s Gemini 4 Carbon list prices rise on 2026-12-01; the table uses the prices valid on 2026-11-11.

Quality eval

Nothing ships that fails the eval. The classifier is scored by exact label match; everything else by an LLM judge (gemini-4-argon-high) with a written rubric (correct, complete, policy-safe), the full knowledge base, must-have facts and a must-not list.

How the pass rate moved while tuning

1. Baseline78/81
2. First optimized pass59/81
3. + escalation and numeric routing62/81
4. + digest counts in code, 90-word summaries72/81
5. + 90-word support answers (shipped)76/81

Every check that still fails

Risks and rollback

Each lever ships behind its own flag, so any one can be turned off without touching the others. The eval runs on every PR and nightly for the 30 days after.

30-day verification plan

Savings are measured per request, so a busier or quieter month does not change the answer. The baseline cost per request is frozen before any PR merges.

1. Day 0Freeze the baseline: last full invoice month, reconciled with the usage export, divided by its request count. Sample: $0.019264 per request.
2. Days 1–3Merge the PRs behind flags. Turn them on for 10% of traffic; compare cost per request and eval pass rate against the other 90%.
3. Days 4–7Full traffic once the reserved test set scores within the agreed margin of the baseline. Any lever that fails is switched off and listed in the report.
4. Days 8–30Daily cost per request from the usage export, nightly eval on the reserved test set, an alert if either drifts.
5. Day 30Reconcile the invoice. Verified saving = (baseline cost per request − new cost per request) × requests actually served. Sample: ($0.019264 − $0.000361) × 1,200,000 = $22,684.

The guarantee

• Shipped PRs, one per lever, each behind a flag
• An eval built from your own traffic, with a reserved test split
• A before and after report reconciled to your invoice
• A ranked fix list with dollars per fix

Method

• Workload. 81 requests: 37 support questions over a 32-chunk knowledge base (8 are repeats or rephrasings), 24 ticket classifications with gold labels, 12 ticket summaries, 8 nightly digests.
• Models. Big claude-sonnet-5-5-high, small gemini-3.1-flash-lite, judge gemini-4-argon-high, through a local OpenAI-compatible proxy. Tokens from each call’s usage field.
• Caching assumes 95% of a stable prefix of at least 1,024 tokens is served from cache; TTL and storage fees are reconciled against your invoice when we run it on your bill. Dedupe found 3 exact and 5 normalized repeats in 81. Batch is priced at half.
• Rigor. Every stage is a real run at temperature 0, scored by exact label match and by one LLM judge from the same family as the small model; the judge is a measurement, not ground truth. On your traffic we add repeated runs per stage and the reserved test split.
• Sources. public/data/results.json (engine/analyze.py), runs/*.json, pricing.json. This page: engine/build_full_report.py, which re-adds the baseline and refuses to build if it does not match.