Most expensive AI-agent workflows waste tokens before model pricing becomes the main problem. Here are 12 practical techniques to reduce prompt, retrieval, tool and retry overhead while protecting output quality.
LLM cost optimization is often described as a model-pricing problem. In practice, many expensive agent workflows waste tokens before model price becomes the main issue: oversized prompts, repeated context, entire tool catalogs loaded on every turn, excessive retrieval, verbose tool results, unbounded retries and outputs that are longer than the task requires.
The useful question is therefore not simply “which model is cheaper?” It is: which tokens are actually necessary to complete the task at the required quality?
This guide focuses on practical ways to reduce token usage in LLM applications and AI agents without blindly shrinking context or degrading results. It also explains why usage and cost should be measured at the workflow level, not only per API call.
What actually drives token usage in an AI agent?
A multi-step agent can consume tokens in several places:
system and developer instructions;
conversation history;
tool definitions and schemas;
retrieved documents or database metadata;
tool outputs sent back to the model;
model reasoning and generated output;
retries, repair loops and subagent calls.
That is why a single “tokens per request” metric can be misleading. Two agent runs can answer the same business question while taking very different execution paths.
1. Measure before optimizing
Start by recording input tokens, cached input, output tokens, model calls and run-level cost. For agentic workflows, add the number of steps, tool calls, retries and subagent calls.
The goal is to identify whether cost comes from a long fixed prompt, repeated history, retrieval, looping behavior, an unnecessarily expensive model or oversized output.
This is where AI agent observability becomes operational rather than cosmetic. thaink² AgentOps exposes token consumption, estimated cost and model calls alongside agent runs, so teams can inspect cost in the same context as behavior and quality.
2. Keep reusable prompt prefixes stable
Prompt caching can reuse processing when requests share the same prefix. Stable instructions, tool definitions and reference material should therefore appear before task-specific content whenever the API and model support caching.
Avoid injecting timestamps, random IDs or frequently changing data near the beginning of an otherwise reusable prompt. A small change early in the prefix can reduce cache reuse for everything that follows.
OpenAI’s current prompt-caching documentation recommends preserving stable instructions, tool definitions and conversation history where practical, and appending new information rather than rewriting earlier context.
3. Do not load every tool schema on every request
Large tool catalogs can consume substantial input context before the agent has done any useful work.
Where supported, defer tool definitions and load only the subset relevant to the current task. OpenAI’s Tool Search, for example, is explicitly designed to let models discover deferred tools at runtime instead of receiving every detailed schema upfront.
Even without a native tool-search feature, the same design principle applies: expose a small task-relevant tool surface rather than a monolithic catalog.
4. Retrieve less, but retrieve better
RAG systems often waste tokens by sending too many chunks “just in case.” More context does not automatically mean better grounding.
Improve retrieval quality first: better metadata, hybrid search, reranking, deduplication and query decomposition can reduce the number of chunks while preserving the evidence needed for the answer.
For enterprise systems, the ideal retrieval unit is not “the maximum amount that fits in the context window.” It is the smallest evidence set that reliably resolves the user’s question.
5. Do not send the entire database schema to a Text-to-SQL agent
Text-to-SQL systems are a common source of context bloat. Sending thousands of tables, columns and descriptions to the model is expensive and can also reduce accuracy.
Use schema linking or schema retrieval to identify the relevant datasets, tables, columns, relationships and business definitions first. Then generate SQL from that constrained context.
Our Text-to-SQL production guide explains why schema retrieval and a semantic layer often matter as much as the model itself.
6. Compress tool outputs before returning them to the model
A database query may return thousands of rows; a log search may return hundreds of events; an API may return a large nested payload. The model rarely needs all of it verbatim.
Filter, aggregate or summarize deterministically before reinserting tool output into the context. Preserve the fields needed for verification, and keep a link or identifier to the full result outside the prompt when auditability matters.
7. Store workflow state as data, not prose
Agents often repeat large portions of conversation history because the system has no compact representation of what has already been decided.
Keep durable structured state for filters, selected entities, completed steps, approvals, tool results and execution status. Then the model can receive the relevant state instead of rereading a long narrative transcript.
This also improves recovery after failures because state can be inspected independently from conversational memory.
8. Put hard bounds on retries and repair loops
A workflow that silently retries the same failing tool or SQL query can multiply token usage without improving the result.
Define maximum attempts, classify errors, stop on non-recoverable failures and require a different repair strategy before another model call. For external writes, idempotency and execution journals matter even more than token cost.
9. Route simple and difficult tasks differently
Token reduction and cost reduction are not identical. A smaller model may use a similar number of tokens but cost less per token. A larger model may sometimes solve a difficult task in fewer retries.
Use routing rules based on task complexity, confidence or workflow stage. Cheap deterministic steps should remain deterministic; routine classification or extraction may not need the same model as complex analysis.
10. Constrain output length to the job
If an agent is producing a classification, SQL query, JSON object or five-line operational summary, do not ask for an essay.
Structured outputs, clear response contracts and explicit length requirements reduce unnecessary output tokens and make downstream processing easier.
11. Prefer references over repeated context
If large documents, policies or prior results already exist in a retrieval system or durable store, refer to them by identifier and retrieve the relevant section when needed. Reinjecting the same full document into every turn is usually wasteful.
For long-running agents, this separation between durable knowledge, workflow state and conversation memory becomes essential.
12. Optimize for cost per successful task, not tokens alone
Aggressively minimizing tokens can produce a false economy. Removing context may increase hallucinations, retries or human rework.
The right metric is closer to cost per successful, acceptable task. Compare token usage with evaluation quality, completion rate, latency and downstream human effort.
A workflow that costs 20% more but eliminates repeated failures may be the cheaper system overall.
A practical token-optimization checklist
Source of waste | First action |
|---|---|
Repeated stable instructions | Preserve a cacheable prefix |
Huge tool catalog | Load tools on demand |
Too many RAG chunks | Improve retrieval and reranking |
Full database schema | Use schema retrieval |
Large tool responses | Filter or aggregate before reinjection |
Long conversation history | Persist structured state |
Repeated failures | Bound retries and classify errors |
Overpowered model | Introduce model routing |
Verbose output | Use response contracts |
Cost without context | Measure usage beside quality |
How thaink² helps teams see where tokens go
Token optimization is difficult when usage is detached from the workflow that produced it. thaink² AgentOps places token consumption, estimated costs and model calls alongside agent runs, logs, events and evaluation signals.
That makes it possible to ask a better question than “how much did the model cost?”: which workflow, task or agent consumed the tokens, what happened during the run, and did the result justify the cost?
Explore AI Agent Observability to see how usage and cost visibility fit into production AgentOps.
Frequently asked questions
How can I reduce LLM token usage?
Reduce unnecessary context, retrieve only relevant evidence, defer unused tools, compress tool results, persist structured state, bound retries and constrain output length while measuring quality.
Does prompt caching reduce token usage?
It does not necessarily reduce logical token count, but it can reduce the cost and latency of reused input prefixes on supported APIs.
Does RAG reduce token cost?
It can when retrieval replaces large static context with a small relevant evidence set; poor retrieval can increase token use.
Should I always use the cheapest model?
No. Optimize cost per successful task because cheaper models can create more retries or human rework.
What should I monitor for AI agent cost?
Track input, cached input, output tokens, model calls, retries and run-level cost alongside quality and workflow behavior.
