LLM API bills have a way of quietly ballooning from a few hundred dollars to thousands per month before teams realize what happened. This guide explains exactly why costs spiral, how major providers price their APIs, and — most importantly — ten concrete strategies to cut spending by 60–90% without sacrificing quality.
Most teams start building with a frontier model — GPT-4o or Claude 3.5 Sonnet — because the output quality is impressive. They write a generous system prompt, pass the full document as context for safety, and don't think much about token counts. Then usage grows and the monthly invoice arrives.
There are four primary drivers of runaway LLM costs:
Context tokens are cheap to read but the costs accumulate fast at scale. A 10,000-token system prompt sent on every request, with 100,000 monthly requests to GPT-4o at $2.50 per million input tokens, costs $2,500/mo before you've even counted user messages or retrieved documents. Teams building RAG systems often send entire documents rather than just the relevant chunks, multiplying this cost several times over.
GPT-4o and Claude 3.5 Sonnet are brilliant at complex reasoning, nuanced writing, and multi-step analysis. They are enormous overkill for classifying a support ticket into one of five categories, extracting a phone number from a paragraph, or deciding whether a query is relevant to your product. Yet many production systems route every single request through the most capable — and most expensive — model by default.
Without semantic or exact-match caching, every user asking the same question triggers a full API call. Support bots and FAQ assistants frequently receive the same questions hundreds of times per day. Each one hitting the API at full price is pure waste.
LLMs are trained to be thorough and helpful. Without explicit instructions to be concise, they will produce long, explanatory, polite responses even when the application only needs a short answer, a boolean, or a structured JSON object. Output tokens are typically priced at 3–4x the input token rate, so unnecessary verbosity is disproportionately expensive.
LLM APIs charge by the token — roughly 4 characters or 0.75 words per token for English text. Every API call consists of an input (your prompt + conversation history + retrieved context) and an output (the model's response). These are priced separately, and output tokens cost significantly more than input tokens at every major provider.
Generating each output token requires a full forward pass through the model. Input tokens, by contrast, can be processed in parallel using attention mechanisms. This fundamental computational difference is why providers charge 3–5x more per output token than per input token. A response that is 500 tokens longer costs the same as processing ~1,500–2,500 additional input tokens.
Context window size determines the maximum amount of text a model can "see" at once — both your input and its own output. Larger context windows (128K, 200K tokens) are powerful for tasks requiring long-document analysis, but they tempt teams to be lazy: rather than retrieving only relevant content, some applications just dump everything into the context and let the model sort it out. This is the most direct path to an enormous API bill.
Anthropic and OpenAI both offer prompt caching, which stores a prefix of your prompt on the provider's infrastructure. When the same prefix appears in a subsequent request within the cache TTL (5 minutes for Anthropic, 1 hour for OpenAI), it is re-used at a dramatically reduced price. On Claude 3.5 Sonnet, standard input costs $3.00/M tokens; cached reads cost $0.30/M tokens — a 90% reduction. For any application with a static system prompt or a large shared document, enabling prompt caching is the single fastest win available.
Prices are per 1 million tokens and reflect publicly published rates as of April 2025. Always verify current pricing on provider websites, as these change frequently.
| Model | Input ($/1M) | Output ($/1M) | Context Window | Best For |
|---|---|---|---|---|
| GPT-4o | $2.50 | $10.00 | 128K tokens | Complex reasoning, code |
| GPT-4o mini | $0.15 | $0.60 | 128K tokens | High-volume classification |
| Claude 3.5 Sonnet | $3.00 | $15.00 | 200K tokens | Complex tasks, long docs |
| Claude 3 Haiku | $0.25 | $1.25 | 200K tokens | Speed, cost-sensitive tasks |
| Gemini 1.5 Pro | $1.25 | $5.00 | 1M tokens | Very long documents |
| Gemini 1.5 Flash | $0.075 | $0.30 | 1M tokens | Ultra-high volume, simple tasks |
| Mistral Large | $2.00 | $6.00 | 128K tokens | European data residency |
| Mistral Small | $0.20 | $0.60 | 32K tokens | Self-hostable, low-cost |
The spread between the cheapest option (Gemini 1.5 Flash at $0.075/M input) and the most expensive (Claude 3.5 Sonnet at $15.00/M output) is roughly 200x. Routing even 50% of your traffic to cheaper models for simpler tasks has an outsized impact on the monthly total.
Audit your system prompt ruthlessly. Remove filler phrases, redundant instructions, and over-explained edge cases. Compress verbose instructions into concise bullet points. A system prompt that starts at 2,000 tokens can often be trimmed to 600 tokens without any degradation in model behavior — that is a 70% reduction on every input token cost before caching even enters the picture.
After compressing, enable prompt caching. With Anthropic's API, mark your system prompt as cacheable using the cache_control parameter. With OpenAI, long static prefixes are automatically cached. For a system prompt sent 100,000 times per month on Claude 3.5 Sonnet, moving from 2,000 tokens uncached ($0.60/1K calls) to 600 tokens cached ($0.018/1K calls) saves roughly $580/month on that single prompt alone.
Not every task needs GPT-4o or Claude 3.5 Sonnet. Implement a routing layer that classifies incoming requests and sends them to the appropriate model tier. Simple classification, entity extraction, sentiment analysis, intent detection, and template-filling tasks are well within the capabilities of Claude 3 Haiku or Gemini 1.5 Flash — at 10–40x lower cost.
Example routing logic: if the task is "extract the date and amount from this invoice" or "classify this support ticket as billing, technical, or general," route to Haiku. If the task is "write a comprehensive technical analysis comparing these three architectural approaches," route to Sonnet. A straightforward heuristic — short input + structured output = small model — captures most of the savings with minimal engineering effort.
Typical savings: 60–90% on routed trafficSemantic caching intercepts LLM requests, embeds the query, and searches a cache of previous (query, response) pairs for semantically similar matches. If the cosine similarity between the incoming query and a cached query exceeds a threshold (typically 0.92–0.95), the cached response is returned immediately — no API call required.
Implementation: use Langfuse's built-in caching layer, or build your own with Redis + an embedding model. For a support chatbot or documentation assistant, expect 30–55% cache hit rates on steady-state traffic. At scale, this can eliminate hundreds of thousands of API calls per month. Tune your similarity threshold carefully — too low and you return wrong answers; too high and you miss valid cache hits.
Typical savings: 30–55% of total API calls eliminatedOpenAI's Batch API and Anthropic's Message Batches API allow you to submit large numbers of requests that don't need real-time responses. Both providers offer a 50% discount on batch processing. The trade-off is latency — batch jobs may take minutes to hours to complete — but for use cases like nightly document processing, content moderation pipelines, embedding generation, or scheduled report generation, this is a straightforward and significant saving.
If you're running nightly jobs to process new documents into your knowledge base, summarize the day's support tickets, or generate embeddings for a product catalog, switching to batch mode immediately cuts those workloads in half with zero quality impact.
Typical savings: 50% on batchable workloadsStreaming doesn't directly reduce token costs — you pay for the same tokens whether you stream or not. However, streaming dramatically reduces perceived latency, which makes it practical to use faster, cheaper models without frustrating users. A Claude 3 Haiku response streamed in 1.5 seconds feels faster than a GPT-4o response that takes 8 seconds to appear in full. Streaming enables you to substitute a cheaper model without degrading UX, making it an indirect but real cost lever.
Additionally, streaming lets you implement early stopping: if your output parsing detects that the model has provided all required information mid-response, you can cancel the stream and avoid paying for the remaining tokens. For structured outputs with predictable formats, this can save 20–40% of output token costs.
Typical savings: 20–40% on output tokens (with early stopping)LLM gateways like Portkey and LiteLLM sit between your application and multiple model providers, offering intelligent routing, fallback handling, and unified cost tracking. You define routing rules — for example, "route requests shorter than 500 tokens to Haiku, longer than 5,000 tokens to Sonnet, and fall back to GPT-4o mini if Anthropic is rate-limiting" — and the gateway handles it transparently.
Portkey adds semantic routing on top of this, allowing you to route by intent as well as request characteristics. LiteLLM provides an OpenAI-compatible API surface so you can switch providers without changing application code. Both tools provide per-request cost logging, making it easy to spot expensive outliers and optimize your routing rules over time.
Typical savings: 30–60% through optimized routingOutput tokens cost 3–5x more than input tokens, so instructing your model to be brief has an outsized return. Add explicit output-length constraints to your system prompt: "Respond in 3 sentences or fewer," "Use bullet points only — no introductory or closing paragraphs," or "Return only the JSON object, no explanation." These instructions reliably shorten outputs by 40–70% without reducing the usefulness of the response for the application.
For internal pipelines where the output is processed programmatically rather than shown to users, the case for brevity is even stronger. There is no reason to generate a polite opening paragraph when the output will be parsed and discarded anyway.
Typical savings: 40–70% on output token costsWhen your application needs structured data — a JSON object, a list of categories, a binary yes/no — using structured output mode (OpenAI's JSON mode or Anthropic's tool use) both reduces output length and eliminates the need for a second parsing call. Free-form responses for structured data tasks tend to include explanations, caveats, and formatting that your code strips out anyway.
A free-form response to "does this text contain PII?" might read: "After reviewing the provided text, I can confirm that it does contain personally identifiable information, specifically a phone number in the format..." (50+ tokens). A structured output response is simply {"contains_pii": true, "types": ["phone_number"]} (12 tokens). At scale, this difference is enormous.
Retrieval-Augmented Generation is often discussed as a quality technique, but it is equally a cost technique. Rather than including your entire knowledge base in every prompt (which could be tens of thousands of tokens), RAG retrieves only the 3–8 most relevant chunks, typically 200–400 tokens each. This means a prompt that might have cost 20,000 input tokens now costs 1,500–3,000 tokens — an 85–93% reduction in context costs.
Key optimizations within RAG for cost: use smaller, faster embedding models (text-embedding-3-small costs 1/5 of text-embedding-ada-002 with comparable quality), tune your top-k retrieval to the minimum number of chunks needed, and implement a re-ranking step to ensure quality doesn't drop as k decreases.
Typical savings: 80–93% on context token costsIn agentic workflows, a misbehaving prompt can trigger repeated LLM calls — the model attempts a task, produces a malformed output, a downstream validator fails, and the system retries, sometimes in a loop. Without evals and observability, these failure loops burn tokens silently. A single stuck agent loop can generate thousands of dollars in costs overnight.
Implement a lightweight evals layer on every LLM output that validates the response format and semantic coherence before triggering downstream actions. Add a hard limit on the number of retries per task (3 is usually sufficient). Use Langfuse or Braintrust to log every call and alert on anomalous token counts per session. An agent that normally costs $0.02 per run and suddenly costs $4.00 should trigger an immediate alert.
Typical savings: prevents catastrophic cost spikesCost reduction strategies only work if you can measure their impact and catch regressions. A robust monitoring setup has three layers: real-time tracing, aggregated dashboards, and proactive alerts.
Langfuse is an open-source LLM observability platform that records every LLM call with full token counts, cost attribution, latency, and custom metadata. Integrate it with two lines of code using their Python or TypeScript SDK:
from langfuse.openai import openai # Drop-in replacement
# All calls through this client are automatically traced
response = openai.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": user_input}],
metadata={"user_id": user_id, "feature": "support-bot"}
)Tag every trace with user_id, feature, and environment metadata. This enables you to break down costs by feature, identify which users or tenants consume the most tokens, and spot regressions in token efficiency after prompt changes.
In Langfuse, create a dashboard with four core metrics:
Configure two types of alerts:
Both can be implemented via Langfuse webhooks or by querying the Langfuse API on a scheduled basis and posting to Slack using a simple cron job or serverless function.
A B2B SaaS company built an internal knowledge assistant for their support team. The bot answered questions by searching a 500-page product documentation corpus and generating answers using GPT-4o. Three months after launch, with ~8,000 queries per month from 40 support agents, the monthly OpenAI bill was $820.
After instrumenting with Langfuse, the team discovered three issues:
User satisfaction scores (measured via a thumbs-up/down widget in the chat interface) remained flat — support agents did not notice any quality degradation. The team continued adding ~2,000 queries/month without any meaningful cost increase due to the caching layer absorbing repeat questions.
Use TechStackopoly's free AI workflow planner to map your architecture, select models for each step, and get instant cost estimates — before you write a single line of code.
Plan Your AI Workflow →