Cost-aware AI engineering — the discipline most teams skip

The bill arrives

The pattern repeats: a team ships an AI feature in Q1, it's a hit, usage 10×'s by Q3, and the LLM bill suddenly looks like a mid-engineer's annual salary. Now the CFO wants a meeting.

The honest answer is almost always "we shipped without the cost discipline a senior platform team would apply to any other workload". You wouldn't ship a database query without an index. You wouldn't ship a hot-loop image transform without measuring CPU. But somehow LLM calls get a pass.

The teams that ship the most AI features in 2026 are the teams that treat token cost the way SREs treat latency — measured, attributed, and optimized as a continuous practice. Five habits separate them from the rest.

Habit 1: Per-feature cost attribution

You can't optimize what you can't see. The first move is tagging every LLM call with the feature it serves so you can answer "what does the AI summary feature cost us per active user per month?".

# Bad: opaque
client.messages.create(model=..., messages=...)

# Good: attributable
client.messages.create(
    model=...,
    messages=...,
    metadata={"feature": "ticket_summary", "tenant_id": tenant.id}
)

Anthropic, OpenAI, and Bedrock all support metadata or trace tags. Funnel them into your observability stack (Phoenix, Helicone, Braintrust). Within a week you'll know which feature is 80% of your bill — and almost certainly it's not the one you'd guess.

Habit 2: The token budget per request

Every customer-facing LLM call gets a written budget: input tokens, output tokens, max latency. Reviewed by the same code-review process as the rest of the request lifecycle.

A typical budget for a customer-facing summarization call:

• System prompt: 800 tokens (cached, $0.0008/1k input)
• Retrieved context: 4,000 tokens
• User input: 500 tokens
• Output: max 600 tokens
• Total budget: ~5,900 tokens, ~$0.012 per call at frontier-tier prices

When the budget gets violated in production (long retrieved context, long user input), the request either truncates with a logged warning or routes to a different model. Without budgets you don't have a feature; you have an unbounded cost surface.

Habit 3: Caching at every stable prefix

The single largest cost lever modern providers offer: prompt caching with 80-90% discounts on repeated prefixes. The catch: prefixes have to be byte-identical across requests.

The discipline:

1. System prompt + tool definitions first, never modified per-request.
2. Stable corpus or domain context next, refreshed on a schedule, not per call.
3. Variable user input last, so the cached prefix stays clean.

Teams that don't structure their prompts this way pay full price on every call. Teams that do typically see their inference bill drop 40-70% with no quality change.

Habit 4: Routing — the right model for the right call

Calling Claude Opus 4.7 for "classify this support ticket into 1 of 8 categories" is a bug. It works, the eval set looks good, the response is correct — and you're paying 30× what you need to.

The pattern is a cheap router:

1. Cheap classifier (Haiku, GPT-5-nano, or a fine-tuned 7B) sees the request and decides difficulty.
2. Easy bucket (~80% of traffic) goes to a small fine-tuned model or the cheap tier.
3. Hard bucket (~20%) goes to the frontier model.

Implemented well, this trades a 50-100ms classifier latency for a 5-30× bill reduction. The cost gap is so large that even a noticeable quality drop on your distribution can still favor the routed setup.

For the math, see the [Small models, big systems essay](/blog/small-models-big-systems).

Habit 5: Continuous cost regression in CI

A prompt change, a context-window expansion, a new tool definition added to an agent — any of these can quietly 2× the cost per call. Without a CI gate, you find out from the monthly bill.

The pattern:

1. Run your eval set with token counters enabled on every PR that touches a prompt, system message, or tool definition.
2. Diff the per-case token spend against the prior commit.
3. Fail the build if total cost goes up by >15% with no quality justification.

This is the same discipline as performance regression testing, applied to AI. The teams doing it have flat-to-decreasing per-call cost curves over time. The teams skipping it have steadily climbing ones.

The composite effect

A team that adopts all five habits typically sees:

• 40-70% inference-cost reduction vs the same product without the discipline.
• 3× more AI features shipped per quarter because the bill stays defensible to the CFO.
• Faster iteration, because cost-quality trade-offs are visible in PRs not in surprise meetings.

None of this requires new tooling. Anthropic prompt caching, provider-side metadata, and a hand-written eval-set runner give you 80% of the value. The other 20% comes from observability platforms (Phoenix, Helicone, Braintrust) that aggregate the per-feature picture.

What we built into the curriculum

Cost is treated as a first-class concept across the catalog:

• [Prompt Engineering](https://nextgenailearn.com/paths/prompt-engineering) lesson 11 covers token-budget allocation across the five context sources.
• [RAG & Vector DBs](https://nextgenailearn.com/paths/rag-vector-dbs) lesson 9 covers production cost/latency, including reranker placement to minimize wasted tokens.
• [Deployment & MLOps](https://nextgenailearn.com/paths/deployment-mlops) lessons 1-3 cover GPU economics for self-hosting and the breakeven math vs API calls.
• [AI Agents](https://nextgenailearn.com/paths/ai-agents) lesson 7 covers per-task token budgets and bounded loops.

Each one ends with a runnable beat where you compute your own per-call cost and watch it move when you change one thing. Reading is forgetting; computing your own cost-per-call on a tiny eval set is remembering.

How to start tomorrow

Pick the LLM-backed feature with the highest call volume in your product. Today, before lunch:

1. Add metadata to every call so it's attributable to the feature.
2. Write down a per-request budget — input, output, latency.
3. Check that your system prompt is stable. If not, factor it.
4. Look at one week of bills broken out by feature. Pick the most expensive.
5. Ask: could a cheaper model serve 80% of this? If yes, prototype the router this week.

The bill at the end of next quarter will tell you whether you were right.

Cost discipline isn't glamorous. It's just what separates AI features that scale to 10M users from AI features that get rewritten in six months because the unit economics don't work.