How do I reduce LLM API costs?

LLM API costs fall fastest through three changes, in this order: route each request to the smallest model that meets the quality bar, cache stable prompt prefixes, and constrain output length. Together these address the majority of avoidable spend in a typical production deployment.

Editing prompts for brevity is the change most teams attempt first, and it is usually the least effective of the four. A shorter system prompt saves a fixed number of tokens per request. Sending that prompt to a model that costs a fraction as much saves a proportion of every request, which compounds across all traffic.

The reason costs drift upward is structural rather than careless. Prompts accumulate instructions as edge cases are found, context windows grow because larger context is easier than better retrieval, and the model chosen during prototyping stays in production because changing it requires re-testing.

Where does token waste actually come from?

Token waste comes overwhelmingly from repetition and oversizing, not from verbose writing. Five patterns account for most of it.

Repeated context. Multi turn conversations resend the full system prompt and prior history on every turn. A twenty turn conversation with a two thousand token system prompt pays for that prompt twenty times. This is the single largest source of avoidable cost in conversational applications, and the one prompt caching was designed for.

Oversized context windows. Large context windows invite filling them. Sending an entire document when a retrieved section would answer the question is cheap to implement and expensive to run.

Wrong model selection. Reasoning models are dramatically more expensive per token than small models, and they generate billed reasoning tokens in addition to the visible answer. Classification, extraction, formatting, and routine summarisation rarely need them.

Unconstrained output. Output tokens cost more than input tokens. A request that produces eight paragraphs where two would serve pays a premium on the expensive side of the ledger.

No caching. Where a stable prefix is reused, not caching it means paying full input rates for content the provider has already processed.

How much can prompt optimisation save?

Prompt optimisation produces moderate, reliable savings, and it is worth doing after the structural changes rather than before. The gains come from four adjustments.

Shorter system prompts reduce a fixed cost paid on every request. The saving is real but bounded, and it competes against reliability, since instructions usually exist because something failed without them.

Better structured prompts reduce retries. A prompt that produces valid output nine times in ten costs less than a shorter prompt that succeeds seven times in ten, because failed requests are billed.

Requesting structured output rather than prose reduces output tokens and removes parsing failures at the same time.

Removing few shot examples once a model handles a task reliably eliminates input tokens that no longer earn their cost. Examples added during development frequently stay long after they stop being necessary.

When does prompt caching reduce cost, and when does it not?

Prompt caching reduces cost when a large, stable prefix is reused across many requests inside the cache lifetime, and it does nothing when requests share no common prefix. The distinction is mechanical, so the decision can be made from request patterns rather than by experiment.

Caching works well in these conditions.

Caching does not help in these cases.

One design point follows directly from how caching works. The cacheable content has to sit at the start of the prompt, because caching applies to a prefix. Prompts assembled with variable content first and stable content afterwards cannot be cached effectively, and reordering them is often the entire implementation.

How does context window management reduce spend?

Context window management reduces spend by sending only the content a request actually needs, which is usually a small fraction of what is available. Four techniques apply, and they compose.

Retrieval. Fetching relevant passages rather than whole documents replaces large context with small context. Retrieval quality determines the saving, since a retriever returning twenty passages where three would do converts a cost control into a cost.

Chunking. How documents are divided determines how much irrelevant text accompanies each relevant passage. Chunks aligned to semantic boundaries carry less waste than fixed size splits.

Summarisation. Long conversation histories can be compressed into a running summary rather than resent verbatim. This trades a small generation cost for a large and repeated input cost, which is favourable in long sessions.

Compression. Removing boilerplate, redundant formatting, and repeated headers from retrieved content reduces tokens without touching meaning.

Summarisation and caching pull in opposite directions and should not both be applied to the same content. Caching rewards a stable prefix, while summarisation rewrites it and invalidates the cache. Long conversations generally suit one or the other, decided by session length.

How should requests be routed between models?

Requests should route to the smallest model that meets the measured quality requirement for that specific task, which means production traffic is normally served by several models rather than one. Routing is where the largest savings sit, because the price range across models is wide.

A workable classification looks like this.

Task type Appropriate model class Rationale
Classification, extraction, routing Small, cheap Deterministic tasks with narrow output
Summarisation, drafting, rewriting Mid tier Quality matters, deep reasoning does not
Multi step analysis, planning, complex code Reasoning Justifies the cost premium
Code completion and review Coding specialised Better outcome per token on code

Two implementation notes matter more than the table. Routing needs a quality baseline per task before traffic moves, otherwise a cost reduction turns into an unmeasured quality regression. And a fallback path should escalate to a larger model when the smaller one fails, since a cheap model that needs three attempts is more expensive than one correct call to a larger model.

Routing is also the point where cost control meets governance, because the decision about which models a workload may reach belongs to the platform rather than to each application. That boundary is covered in enterprise AI token management.

How is token efficiency measured?

Token efficiency should be measured per conversation and per business outcome, because per request cost is the metric most likely to mislead. Per request cost falls naturally during optimisation while total spend rises with adoption, and a programme reporting only the former will claim success as the invoice grows.

Four measures cover the useful range.

The last one is measured least often and matters most, and the reason mirrors the argument in FOCUS and FinOps culture. Accurate cost data changes nothing unless somebody is accountable for acting on it, and efficiency metrics without an outcome denominator invite optimisation of things that should not be running at all.

What are the most common token optimisation mistakes?

The most common mistake is optimising unit cost while consumption grows unmeasured, which produces a programme that reports improvement while spend increases. Five others recur.

Downgrading models without a quality baseline. The regression is discovered by users rather than by measurement, and the change is usually reverted wholesale rather than corrected.

Caching content that is not stable. Cache misses on content assembled in variable order cost slightly more than not caching, and the configuration looks correct.

Adding retrieval to large context rather than replacing it. Retrieval only saves money when it displaces the larger payload.

Ignoring output length. Teams shorten prompts extensively while leaving responses unconstrained, which optimises the cheaper side of the transaction.

Treating retries as free. Failed and malformed responses are billed, so reliability is a cost lever and not only a quality one.

What does a token optimisation programme look like in practice?

An effective programme establishes measurement first, then makes changes in descending order of saving, re-measuring quality at each step. The sequence matters because the largest changes are also the ones most likely to affect output.

  1. Instrument token consumption per request, per conversation, and per workload, so a baseline exists.
  2. Establish a quality baseline per task, so subsequent model changes can be evaluated rather than assumed.
  3. Route traffic to appropriately sized models, task by task, with escalation paths for failures.
  4. Constrain output length and adopt structured output where a schema applies.
  5. Cache stable prefixes, reordering prompts so cacheable content comes first.
  6. Reduce context through retrieval and summarisation, choosing between summarisation and caching rather than applying both.
  7. Review prompts for accumulated instructions and unnecessary examples.
  8. Re-measure, and set a recurring review, since prompts and models drift.

Steps one and two are the ones most often skipped and the ones that determine whether the rest is defensible. Without a cost baseline there is no way to demonstrate a saving, and without a quality baseline there is no way to show the saving was free. Guidance on where this fits within a broader cost practice is published by the FinOps Foundation, including an AI Value topic area covering AI specific cost management.