Skip to main content
Back
35 XP

Lesson 6 · 9 min

Continuous cost regression in CI

A prompt change can quietly 2× the cost per call. Without a CI gate you find out from the monthly bill. The pattern that catches it at PR time.

What to gate

Any change that touches a prompt, system message, tool definition, or context-assembly logic. The pattern:

  1. Run your eval set with token counters enabled.
  2. Sum input + output tokens per case.
  3. Diff the per-case spend against the prior commit (stored as a baseline).
  4. Fail the build if total spend goes up by more than X% (typical: 15%) without a quality justification.